Shaping Intelligent APIs: Scaling LLMs, Open Ecosystems, Enterprise AI - Daniel Oh

This talk explores the integration of AI APIs within enterprise environments, particularly focusing on the use of open-source technologies to enhance AI capabilities. The speaker discusses the importance of production considerations for developers, emphasizing the need for scalability, security, and governance when deploying AI models, such as large language models. He explains the role of inference servers and various architectures like Kubernetes in managing AI workloads, and the significance of designing intelligent APIs that serve specific business functions. The talk highlights practices for developing and deploying agentic AI applications, including utilizing tools like OpenAPI and Quarkus, and managing observability and performance in production environments. Throughout the session, the speaker shares insights on how to approach API-driven development while ensuring responsible AI practices.

You see the a bunch of the wording in the title. So, we're going to really be talk about a AI AI API, something like that. And like a uh we're going to consider about uh production. I know some most developer they don't care much about production. So, they only care about development. I don't care about production. I don't like that, but you have to because uh when

you make a some problem in production, and then maybe the production team maybe they wanted to use AI rather than individual developer. And of course, we're going to talk about open source ecosystem and how do we bring API with AI capability into your enterprise environment. So, as you heard, I'm from Red Hat and IBM uh as a devrel. And so, I'm living in Boston and I contribute

a lot of community like a Java, CNCF, and then among others. And then you can Google me and you can find me with the social media Daniel Oh 30. So, yeah, feel free uh ring to me and then uh ask me any kind of question and happy to answer. And let's keep keep in touch with that after uh this conference. So, uh API is a as you

know, you already have a tons of experience to develop API based application like a API driven or API contract or whatever you call that. So, most likely you exposed your business services by API. Most likely less API in Java. And that application uh can be consumed any other like a WinUI or mobile application and even other API AI application as well. So, here's the thing is the

LLM like a large language model or you could say a AI model is just bunch of the files. When you go to Hugging Face today and then and then you can look again one of the like a AI model and just looking at details and there are many file binary files like like a safe sensors, configuration YAML and JSON among others. That is just a stupid files.

Until you deploy and run that file on top of the inference server. For example, vLLM or OpenVINO or CUDA, whatever some kind of inference architecture. Until you deploying your AI model and that is just a files. And after that, that inference server allowed run that files as AI model, but also allowed to accelerated like your GPU and CPU processor. And lastly, it expose API. So, that is

a API allowed your AI app model meet your real business services. So, the the primary goal of AI application and AI model it try to make it better business. So, make a more money, save more money and then make a more revenue, this kind of thing. So, the problem is in production environment. So, how do you productize AI model and LLM responsibility? Maybe you can heard about

the responsible AI in terms of like a scalability, reliability and security and governance and among others. some people actually ask me so like a MCP is a model context protocol. It's a great open source technology. Make your GNI become agentic AI with autonomous capability. So, anyone wanted to start building new agentic system, and MCP probably first technology you need to use that. Uh for sure, um no

exception. once they actually use it multi MCP server and multi AI model and multi tools and multi agentic system AI application. And now they going to time to deploy And then how do they managing like a day one day two operation in terms of a scalability, security, governance and among others. And then MCP is a basically just transport like a protocol. They don't give any security or

scalability at all. So, you can find another like a open source or technology or solution to secure and address this enterprise capability as So, enterprise AI, which is a totally needed to include that non-functionality. So, functionality is a autonomous agentic AI. For example, non-functional capability is a performance, security, scalability, reliability, governance and observability as all. So, that's why absolutely needed to checking some fundamental stuff. But also

they need to address costing costs of as well. So, you can actually pick it up one of the model going to Hugging Face today. There are more than 2.8 million models today. You can maybe pick it up one of the like a llama or one of the very famous model. And you can maybe download from Ollama and using a machines and I give it a try proof

of concept using like a Python or JavaScript or even Java to build your own a simple agentic AI application. It's pretty easy and uh pretty working. And then when you deploy the production it's really hard. You can go through every single that uh production capability. But also you need to think about how many CPU GPU we need to use that to run this model. And how many

token we're going to consume for the next couple months. So so that's why whenever you design new API driven or contract application for new agentic AI you working with platform engineer directly. And the platform engineering team they has own the AI model just like a one of AI workload. So for example platform engineer or DevOps engineer they just looking at AI application and AI model just a

one of workload. So because of developer only point to application. So my application Java, your application JavaScript, his application Python. Something like that. But when you deploy application in production for example in Kubernetes as a container. So there's a more workloads behind the application. Because the application not stand alone even recipe API you probably connect to back end services which is a database or messaging broker or

something like a real legacy system. They can be running on container on top of the Kubernetes as well. So Kubernetes standpoint platform standpoint there are many workload container but some of the part of a just application. And now they have a new workload which is a AI model like a LLMs as a container. Also uh your uh like a agentic AI application at all. But, platform team,

they need to observe this all workload at the same criteria and same, um, measurement thing, as well. So, how do we make a intelligent API, like, a some more, like, a smarter API? So, basically, without API, LLM just file, as I mentioned earlier. And then, with API, now that can do something more, something like, a uh, business purpose and business capability. So, that's to make a your

API, uh, connect to AI model, and now you going to API more, like, a intelligent API. Because API include and provide contract, observability, security, and re-re-reusability, and governance, and so on. So, here's the thing, as a developer practices, when you start, uh, developing new agentic system, you probably need to start API first. So, what kind of the, like, a what kind of the, uh, information you going

to expose to through your API, rather than pick up the AI model first? Because you don't even know how AI model works, how LLM works. You can just pick it up one of them, like, a Hugging Face, and then give it a try. You feel fail, okay, I think my application perfect, but this model stupid. Let's give it a try another model, find out. And then, even

if you find a something LLM model that works with your application, but when you deploy that model, maybe with a super giant GPU processor, and now maybe you got some wrong answer, like, a hallucination at all. And how do you fix the problem? You probably ask, like, a smart AI engineer team, "Hey, fix the model. My application perfect, it's your problem." And AI the engineer may based

on, like, a base on Python people and notes. AI model is a smart enough. Your application is a idiot. So, so it's a really happening in many many enterprise company around the world. So, they actually start AI adoption journey like a gen AI or agentic AI, but 6 months later, they actually fail many project. Because they never ever think about what kind of business value they have

to expose and provide back to the end users like a through APIs. So, that's the you have already experienced with API driven or API contract development, you need to think about how to specify your API. Which means which information you're going to expose and provide to end user and customer, something like that. And then after that you're going to implement business logic realize that API specification. And

then back to the connect to the database or other external services. You're going to give it a try exactly same pattern and practices. Rather than okay, we're going to find the AI model first, that is a like a wrong path at And see here is some like a like a 10,000 feet like a architecture standpoint. So, starting with some client and go to API and you go

to AI model and then end up with your enterprise legacy system pretty much based on Java stack or some other like a even COBOL mainframe. I don't blame on that or traditional architecture by the way. But most important part here is there is no like a firewall security or observability in your intelligence layer like a LLM. There is no such a thing. Where is the that such

a thing in your intelligent API? So, if you just expose API like a get, update or delete something like that, crud. And then that shouldn't be intelligent API. Connect to If you API connect to AI model, and then you cannot say, "Oh, my AI API is an intelligent API." That's not true. So, you add some kind of enterprise AI capability with the APIs. So, for example, you

can actually give it some easily authenticate, authorize your API, who's going to be access, who's not. And then what kind of wall you can only see specific information. You already know that because you already experience how to develop API driven. And now you're going to do the same thing with the APIs uh AI model stuff. And now you got to publish some question. So, I already know

like a API stuff, like a for example, Open API, one of most popular open-source project and software to build your own API driven But what about the others? So, this is just some kind of open-source ecosystem, and then like a not only your development environment, but also production as well. In a In a production, you probably have a multi AI workload, Java, Python, and among others. And

then you of course using MCP let your AI application more like a autonomously agentic way uh allow to access external tools, something like that. And then when you uh deploy your new API driven AI application into production, and how do you scale that application? That is a maybe some kind of GPU based auto scaling or some other like a resource based auto scaling on Kubernetes. And also

eventually auto scaling and like a something like a KNative or KServe. And of course, there are inference server, which is a VR that. This is the one of reference example. You can of course many other choices on like for example CNCF project or Linux Foundation or Apache Software Foundation and so on. So, OpenAPI, the open source project, I think so you already have a experience with OpenAI

OpenAPI. And then OpenAPI is a kind of your new AI contract because OpenAPI specify the machine readable based on JSON and human readable as JSON specify all schema, what kind of data uh requesting response in the processing limitation and restriction and so on. So, it's going to be enable uh like a predictable integration and back to the some kind of AI model and you can put the

security capability, authorization, authentication. And the base on API is a quickly easily make it some versioning. Okay, now I'm going to use like a AI model one uh like a version 1.0 API. And now your AI engineering team, they actually give us some trained model for specific business domain, which is like a like a version two AI model. And at a time, okay, let me try to

upgrade my application really be prompt engineering or some of the stuff. And then you're going to make a version two based on like a API gateway or some of the architecture standpoint you're going to make a version two. In the meantime, you can can support like a the the n minus one version like a version one because of some of the still some of the uh like

a end user they still using old model for any application. So, this is a new your AI contract. But how do we uh designing actual AI integrate API? There are a few patterns. Most important thing is most important thing is you needed to design your new intelligent API like a task oriented. previously, you probably uh expose your API with an a known like a endpoint like a

get, post, and update to create or delete, insert. And then you don't need to actually uh like write it down readme file or API specification. Everybody knows that. Everybody expect that based on URL and then based on that like a API uh like a what do you need design? Okay, this is a something retrieve some data. This is a creating new data. Exactly same thing. So, Help

by AI model, each AI model specializes specific purposes. Of course, you can go to like a big giant gigantic uh model like a GPT or model kind of 05 something like that or some Gemini big model. They that model actually provide universal capability. You can ask like a chat or interactive conversation or even ask to generate or video. But if you go to very specific model like

a banana banana, which is a more like a image generation specialist or you could maybe Google Gemini normally is a more like a interactive conversation generate text rather than ChatGPT. So, So, in enterprise AI, you probably and you should have specialized AI model can be small model rather than large model. Which means when you create your agent AI application relevant to that specialized model, and your API

also expose specific endpoint just like a crowd. And then this is a more like a a chat API. This is a more like a like a some summarize kind of thing. This is a the more like a extract data or generation image kind of thing. And then we do that explicit and task oriented API endpoint and then you don't need to write it down like a big

writeable in readable file. Everybody knows that like a some kind of contract between consumer and provider. And the other thing is also like a function calling based on API and how do you call function? It's so you you actually need the API and then you can make a some prompt as a service. You can expose model based on prompt and then define your a restful API like

a Java application like a with some system message annotation and then with the that you can actually define some of the prompt even if agent AI you still need to some prompt. Not very detailed instruction though. You still some simplified prompt and then you can actually define prompt for specific API which is a ring to specific AI model. But I highly recommend try to not to expose

raw model directly because when you expose all data or in putting the all your JSON schema, it could be security breach. So there is always the risk about some security risk like a prompt injection, something like that and maybe putting a lot of data in the JSON is probably using a lot of more tokens and then more networking and then consume more like a resources. So that

is highly not recommended to do that. So based on API is a kind of prelude as a firewall between your AI model and exposed an application. As you know like a authentication and authorization And can do that in API level. Directly, you are like a master level or class level. Or if you have like a some API gateway solution on top of the in front of your

like a back end architecture, the API gateway they actually provided some of the like a capability how like a set up the quarter and the rate limit. And also there are some of the like a API access management kind of thing also. And but also based on your API driven application, you can put in the like a guardrail like a logic like a input output management. Like

a if you ask it some very sensitive information may like a my credential info or sexuality or some of the some sensitive information, you cannot you probably just filter out. You cannot even sending to AI model. That is an input guardrail. The other side and then when AI model response sending back to end user. That is output guardrail. You can actually put the that into your API

application as well. So, we do that like a more like a AI intelligent intelligent API design and some of the like a practices how to develop kind of thing. And then now we got to some think about So, okay, we just a great design API for your AI application and like a some and some other things. And now your job done and now you deploy application into

production. And how do we scale? even if a functionality is the same thing, but the scalability is a different. So, for example, when when you develop just a like a general Java application and then deploy it to Kubernetes. And then sometimes you got to like a Black Friday like event and then a lot of a million people are actually coming to the data side and the mobile

and they click click click click and I want to buy this kind of all 90% discount. And then normal day you have like a thousand customer and that day million people and then the Kubernetes okay, let me scale out really quick like a 10 Java application now 10,000 Java application. But inside the container and JVM needed to start up. And then on top of the JVM your

Java application for example Spring Boot start up and a warm up and get ready to process the request and after rebalancing of network traffic. So but it takes some time. But Kubernetes job done because I already ordered scale out 100 to 10,000. And then you ready to work but it still takes some time. So that's why developer also need to optimize the Java application for high scalability.

Make it smaller, make it faster. So we could probably use it like a crack or you can like a native image or like a more smaller and smaller and faster kind of stuff. So what about the LLM inference? So maybe you you you probably think about the easiest way you can just add more GP from Nvidia but that's not good solution. So you probably think about the

latency and actual AI model. Of course the inference server and relevant application and also the like a core star which means Kubernetes provide like a serverless capability for your AI model itself. So when there's there's a no traffic into the AI model and your AI model running on container and it's going to be scaled down to zero. So, no need to keep uh holding your GPU processor

resources. So, this is all kind of consideration, not only application but also proper engineering stuff as well. So, how do you address these challenges, specifically performance and cost uh perspective? So, you could actually make a some like a criteria uh always not too much or too little. It's more something it's a middle like a good enough like a strategy. So, based on uh we're going to do

that like a large model or you're going to do that small model like a between somewhere. And like uh we're going to sometimes using like a quantized model or we're going to use some not only like a chat model or not using some uh real-time communication or sometimes we're going to batch job or streaming or we're going to do that like a some caching algorithm for your

chat messaging or conversation in some area vector store or some external like a post SQL store rather than put more effort and workload or payload into only AI models. So, there are always you going to like a managing uh this a performance uh capability with that. So, Kubernetes dimension earlier is going to be some like a scaly and then like a based on GPU and the CPU

realization. And then more great thing is uh give us some like a multi-tenancy isolation for specific AI project uh whenever you design API, but even sometimes they actually ask uh access your API for your agent AI application, but that is not allowed to be there. So, you could actually put on the somewhere in the API gateway, but some of the company they don't have API gateway solution.

In that case, the platform level the Kubernetes actually isolated it's a tenancy which is a not only accessible cable permission but also they can only use the specific workers GPU and CPU some some resources as well. Of course, AI application, you probably have good observability for like a SRE like a monitoring monitoring, logging, telemetry data and and so on as well. So, when So, in the platform

and then for example, Kubernetes uh stand for whenever you deploy application agent and based on API and AI model and among others it's all about same same workload. And then the best thing is the Kubernetes you're going to scale down to zero automatically as container you don't have any traffic. Oops. Happening again. Anyway, so So, that's the reason why a lot of enterprise company adopted serverless because

if you don't use your workload and then we don't have any reason to keep making up to date or running on the time. So, that that is like a serverless and then KNative one of the CNCF uh like open source project allow you any workload like a AI application even AI model running on container and scale down to zero if there are any no traffic. Of course,

when some user using mobile or web API, okay, they're going to just click the API and then that call back to your AI model and AI model scale up automatically in a second. So, that is like a KNative stuff and advantage of it as well. But, the Java application, how many people actually have some experience with the Quarkus platform? A few? Okay, great. So, Quarkus actually invented

like a 7 and 1/2 years ago and then Quarkus is designed for cloud native and then most likely Kubernetes and then running out of course running on virtual machine or bare metal even those and then reactive application and serverless and then even agent AI application at all. Because the Quarkus any Quarkus library is a 100% working to build JVM but also native image as well and really

good to scale rapidly in Kubernetes and also give us some like a rest based API contract development for client and server as well and most great thing is Quarkus is give us some great developer productivity. It's a you it's a less many reducible play code. For example, you can do XYZ in a spring AI and you can go Quarkus with LangChain for J just X no more

YZ. Which means a less code and then easy simplify your application and with the live coding and then which is way better to show and the develop application at So, I'm going to go to demo and put in the some like a open API and the Quarkus Kubernetes running on VM and kind of stuff. So, hopefully it not going to breaking the monitor. Uh this is a

like a some of the like a a demo some kind of architecture here. So, some of the like API gateway basically Kubernetes actually provide a kind of like a H 2 P extra router kind of stuff. And then I have a running multi AI model on top of the Kubernetes today. I'm not using open AI open AI and then uh the Quercus application for like uh some

AI agency. And of course I'm using Jaeger and then open telemetry for observability and how uh like we can like uh observe and monitor actual uh transaction happening between my API AI application and AI model kind of thing. So, first of all So, here is uh my uh something like a Red Hat OpenShift but base it's a Kubernetes and then you can see there are five different

container pod. And each container pod is actually AI model. So, this is uh Ollama and then running on Ollama model. And the other model like uh IBM Granny model which is like a base on Mistral model. And then the other my custom model. I actually uh did a fine-tune uh this model uh for specific like uh uh like a insurance company. And this is our like a

Mwa AI dashboard and you can see that uh data model already deploy and then uh internal endpoint something like a here. This is the inference server endpoint just like a REST API. So, here is my uh IDE here. So, yeah hope let me try to make it a little bit bigger. So, first of all I'm going to run uh Quercus dev mode uh using uh Maven command

line. Uh you can actually using uh Gradle if that project based on Gradle or uh using Quercus CLI. So, this is try to run in Quercus dev mode and then whenever I change the code it literally recompile restart in a second without uh manual job at all. So, Quercus running on let me try to make it a little bit smaller so I don't barely can see that.

Okay. So, So, first of all this is the application property and then this is a like a Quercus actually integrate LangChain4j. It's a some kind of LangChain in Python ecosystem. So pi in a Python ecosystem there's a LangChain or LangGraph that is a lot of you create like a AI application like agentic or rag or manage memory kind of stuff and prompt engineering as well. So Java

also needed something like that library. That is a The Spring AI actually built their own like a capability but pretty much the same functionality and capability like a LangChain4j. And LangChain4j you can go to using LangChain4j without Quercus of course. You can use the LangChain4j with the Spring today. And but you can follow LangChain4j SDK syntax among others. And Quercus integrate as like a wonderful Maven library

and then reduce many code and then simple annotation make it easier. Make a your codes very simple. just looking at the palm.xml here. Go ahead. And uh so palm.xml and you can see that here LangChain4j palm which is like a some Maven palm to integrate LangChain4j. And you can see that uh here Jackson and open it LangChain4j open AI. So LangChain4j open AI does not mean you

can only use open AI model. So open AI provide API specification. Any model they model support open AI API specification we can use this kind of stuff. You can also use like a Quercus-LangChain4j- or Lama XYZ and for like a API specification. And that's it. And then the other thing is I'm going to using like a database and then like a web socket. And then, Kinowa is

so interesting project because you could actually run the JavaScript and Node.js like a front-end application inside the Java project running on JVM. So, so when I go to here when you are directory, and then you literally see that the Node.js script and the source. You can actually combine both. But, here is a trade-off because Quarkus allow you put your Node.js inside the JVM and the running on

the same application. The problem is here, if you wanted to scale out separately front-end and back-end, and then you need to make a separate projects, Node.js and then Quarkus. But, in reality, we saw a lot of enterprise company they adopt three-tier architecture in many years ago. Front-end, back-end, database. But, after operated that three-tier architecture for the last 10 years, and most scalability is a back-end, not front-end.

So, front-end, there is no heavy-weight logics, just pass down traffic. So, that's why if you could put in the one simple small Java application that also include a front-end, that'll be awesome. So, that's this is some kind of project. And then, go to uh back to the application property, and then I'm going a llama here, a llama 31. But, this llama not running on running on Ollama,

but that running on container on my Kubernetes platform, not my Mac OS local machine. And I'm also using the React for because this sample demo application is like a fake insurance company and the insurance company always have their own business document. So, every time the new business policy change, they okay. So, 10 days ago, we can reimburse you 100% your car accident, but now we not do

that anymore. We can only reimburse like a 95% new policy. So, every uh new policy comes out, I don't want to um re-fine-tuning stuff. It's a really expensive and then I don't want to bother AI engineers anymore. And that's why you need to rag. And the rag, basically, you have to implement bunch of stuff. How to retrieve data and how to store my index in the vector

store, how to connect that, how to retrieve back my application, and among others. So, but uh there are easy rag library, which allow you just point to where your documentation exactly store. And the other whole thing is automatically processed by Java framework, which is uh this LangChain project library and the Of course, uh when you use a rag, you you needed to embedding model. Some model actually

provide uh included embedding model as well, but I'm using separate embedding model for this case. Maybe this is a really good point when you design new agent AI application, regardless of API. Because you going to make out try to keep using small model for specific case. Small model, you don't might need to use a big giant GPU processor. Quickly scale down and quickly scale up again. And

then I'm going to using uh some kind of mail library because you have this enable scenario. Whenever I change the the insurance claim and the back end customer services, they actually uh write us some email to the customer. For example, "Yeah, we are more than happy to approve your claim and then we are ready to pay for you." And I'm going to write a very professional uh

gentle polite email and then send it to the customer. But now we not going to do anymore because we have fantastic agent AI. So the new agent AI automatically generate email subject, title, and content, whatever they want and send it to the customer. But this is a demo environment, I'm not going to send it to you directly. I just make a uh email like a some container

and like a some mail server and then send it to there and then just confirm and then verify uh everything is fine. And the other thing is uh pretty much like a logging stuff and the Quinoa is running on Node.js inside the JVM and the uh like a in-memory database and like a Open Telemetry Open Telemetry which allow me tracing uh what happened between API, AI model,

something like And then here is the my actual Java application here. So here's a my claim document for the rag and then here's the two AI uh application. The first is a claim application is uh you can see that just view uh register AI service annotation. This allow my plain interface Java Java interface just change to AI services and I'm going to point to specific model. And

then here's my system messaging specialize your initial message sending to AI model, specify your model behave score something like this. So, you basically very helpful, respectable, like our assistant, like our parallel so it's a fake insurance company name. And then something like some detail. And then like a pass down from prompt like a when you are and then back to the And then we're going to use

them toolbox services. It's a notification services. So, this is automatically generate email by that. And you can see that here, too. So, happening in a prompt like something like a semantic way, "Hey, I'm updating your some status." And it called this call automatically by AI model and then try to create the some mail content and then sending to email server, but also updated the actual the claim

status in the database, that's it. Uh that's pretty much decent here. And then here's a when you are here. So, let me try to reload when you are. just in case. ongoing project. You probably part of the ongoing AI project at some point in the future. So, you can see that this is a new insurance company. So, the other all menu not developed yet. So, we only

develop this one of like a claim feature today. So, maybe your job tomorrow you're going to develop like a policy menu with the agent AI with the rag or more others. So, when I go to claim and then you can see the bunch of the claim already uh raised that. So, one of the claim the first one is in process right now. And then there are what

happened like a daytime and location and summarize the kind of stuff. And then here's actual the car totally wrecked out and then you probably added some question. Okay, what status of this claim? So, you probably ask quickly. This is a like a agentic or a AI code assistant. And then I ask them back to my IDE and then you can see that it's a it's already happening

on my back end going to Oh, where is the icon? I don't see it. All right, here there we go. So, this is a so many like a index number which is a using vector store to put in the like a policy stuff from rag and then it's a communicate using my llama 3 one model and then here's the system messaging and then a bunch of the

claims. So, this is actually from the rag or the policy PDF document and then back to the here and this maybe takes a little bit more time. Let's give it a moment someday sometime. And then after that, it will show up you some status. And then what I'm going to next to do is I'm going to try to update the status to directly. Oh, we got a

some error. It's a connection error. Okay, socket error. Okay. So, so that web socket is actually based on the web UI and the node JS a back end my Java application, but sometimes the network latency make us some like a some warning error kind of thing, but we got a some some response from AI model. A claim is a currently pending and then the adjuster will be

assigned to inspect the vehicle and access damage etc. etc. So, and then after review adjuster, let me try to look at it. So, update Oops. updated the status to s- denied. But, this is a really happen. Uh most likely insurance company always approve your claim. But, you probably go to But, in the meantime So, there are Quarkus Dev UI, which is a uh visualize your all the

frame level of framework level happening. So, when I go uh Dev UI and then you can actually uh see there many integrate uh UI capability. For example, here's a mail fit. And then when I go to mail fit UI and then click on here, and then here's a mail service here. Uh is this is a previous what I test a minute ago. So, if I go to

here, oh this is denied. And then when I enter claim is currently denied. And then when I uh reload the page it's a literally change the status, which is it came from database. And also claim thing is it's already I'm not the uh claim change here. Update your claim. And then this is your official communication from personal insurance claim department. And then so, your uh status change

denied. And then here's when I go to Jaeger UI and then let me try to reload the Jaeger UI. And then go to Jaeger UI and then here is my one of the service insurance application. And then go to like a chain message and then go to find tracing. And I have one of the tracing and then when I go to one, and then you have a

bunch of the step. Is this is all API. And then easy to trace down because uh traditional Java application, you can easily uh trace down your uh application interaction between like a database some of this stuff. But with the AI model is you don't even know that and for example a gentle AI is a keep interacting between AI model tools and application even some database maybe hundreds

of time in the next couple of seconds. So that's why you have like a Yeager kind of thing and then when I go to specifically notification service updated claim services for example and then go to tags and you can see that status X deny and then we can you you can actually find out that thing and of course you can find a which he interaction is a

most take some time or mothers. I'm sorry it's a keep blinking this display you probably your eyes burning this that thing. All right. So let me let me let me try the uh summarize kind of thing and then uh we can take a question time at all. So as you can see that the demo so there are bunch of stuff at API gateway and you can find

the latency time based on like a Yeager and like you can also using Prometheus support telemetry data and then at a at a race and then you can actually find out a model completion so when you go back to uh you can actually see that uh here like a chat completion stuff. This is actually chat stuff and actually communicate your AI model here. This is parallel tool

AI model. So you can actually all uh you know tracing down all uh communication between AI model as well and response in cloud application with like a visualized some of like you can actually using Grafana with that thing as well. So, so that is uh basically how to stand up AI uh the enterprise AI platform. So, basically platform engineering team, they has own infrastructure like a AI

model and like observability and among others. And going through API and they going to expose API using API gateway or directly API endpoint URLs. And then the other team like a product team or development team, they can consume API to access it. AI model, but also they can develop API itself to expose some uh specific data from AI model as well. And then one of the interesting

part of MLOps, they keep evolving into uh GitOps and AI Ops and right now uh like a platform engineering. So, probably you heard about the GitOps based on Argo CD and Argo Rollout. So, that is uh give us some like a new capability CICD, but now you will have a new API and we do some specific test code oriented uh platform engineering. And then with the that,

it will trigger your new CICD pipeline and it will new versioning model, but also relevant AI application uh within uh canary uh deployment rollout rollback. In the meantime, you can actually put on the agentic AI. So, for example, I have you can go to uh my GitHub repo uh GitHub/daniel030. So, there are other example like a progressive delivery uh Git repository. So, previously progressive delivery, you can

stand up like a rollout based on canary uh deploy and rollback whenever you find some error. And then now that uh new pipeline turning into AI Ops, how could you do that? You can put in the like a digital SRE, which is a based on agentic AI. I actually made a in Quarkus application. And Quarkus application include like a five or six different agent. And one of

the agent just grab all Kubernetes metrics data and analyze that. And another thing is that I got metrics logs data from Igo or actual application. And then analyze the whole data and find out some potential issue, for example, memory leak or some of the stuff. Any kind of critical uh potential issue find out by agent AI. I did want to create a new pull request into that

application directly, but also at the same time just pause the CI/CD pipeline and then roll back to original project. That is a more new AI ops stuff uh with this architecture. And then responsible AI is not hard. So, as long as you already addressed like a auditing and like a provi- uh providing security. And more important thing is the human in the loop. As I mentioned earlier

my session yesterday, so AI always can fail. Even if your API driven application perfect, but your AI can be fail. Them AI model. In the case, you're not going to give it some full authority to your agent AI make a call. So, that's why you need to human in the loop. And the last person you will be there and uh make a decision, okay, I'm going to

approve just like I'm prompting, I'm going to deny this claim. Just like a happening by human in the loop. And the last thing is the open ecosystem. It try to not to uh lock in some sub vendor. Of course, you can use open AI today. You can use the Google Gemini today. You can use the both or several the Anthropic among others. But, when you keep uh

uh adding more agent AI based on API and scalability, security among others some more flexibility and dynamically and try to keep around with open source and a standard open source platform. And the one last takeaway, so you can just using keep open API for your intelligent API application development and using like a Quercus or other like a spring, whatever you want and then running on Kubernetes and

K native and K server for scalability and like a security governance and so on. that's it in a final thought and here's the future of intelligent platform. Make sure make it some scalability, trustworthy for your responsible AI and then maybe it's going to be going to be bored to operate because you have a many digital SRE for platform engineer and of course and agentic AI application when

you build as developer and then they don't need to watch it the giant screen more and just give it a final meta decision call for your production environment as well. But to do that, developer make a agentic AI expose API but also make a secure and intelligent and scalable AI application That's it. Thanks for watching and then joining today and then happy to take a question. Thank

you so much. >> [music]