Agents in Action: Building Autonomous Java Systems That Don’t Break in Production - Brian Benz

This talk explores building autonomous systems in Java that leverage AI agents for production. The speaker discusses the architecture of AI agents, which includes reasoning, planning, and execution capabilities. Key topics include multi-agent orchestration, invoice processing using domain-specific tools, and deployment strategies such as Kubernetes and serverless approaches. The integration of tools like LangChain for Java and Spring AI is highlighted, as well as the importance of observability, testing, and scaling. The talk features a demo of a scalable invoice analysis application that utilizes a GPU for parsing and employs error handling and fallback strategies to ensure reliability in production systems.

[music] >> Let's talk about agents in action. So, we're going to talk about building autonomous Java systems that don't break in production. Uh and um Uh I'm glad to be able to present this. I'm looking at the gap between agent demos and production systems. We're going to talk about that. We're going to talk about what AI agents are. And [clears throat] um the the loop that you

use to build them and run them. Uh agent architecture patterns. Uh building agents in Java. I'm going to show you LangChain for day and Spring AI demos. Uh I'm going to show you multi-agent invoice analysis. So, we've got an actual app that runs on two different platforms. So, it's scalable. It's all agents. And we have an orchestrator that calls a parser. And the parser parses invoices and

checks to see if there's anything weird on the invoice before it gets processed by the system. Uh and um then we're going to talk about production patterns. Uh deployment strategies with containers, Kubernetes, and serverless. And then some guardrails that you can put to keep humans in the loop uh for as long as we can. And uh key takeaways and Q&A for this. the production gap. I generally

live in demo world. Uh but I talk to a lot of customers in production world. Uh in the demo world, things that work on your laptop with a happy path. Code generation works the first time and it runs and there's no problems with it. And then you find out all the problems when you get it anywhere close to production. Uh in a real enterprise system. Uh there's

no monitoring or tracing. And it's a single user and user requests. And so, I just build these things and they're cool and they work. But then, how do you actually build things that work at scale and under load? Uh handles failures gracefully. Uh retries, circuit breakers, has fallbacks, maybe it has multiple models it can work with. And then end-to-end observability, concurrent users, and rate limiting. And is

it reliable? So, those things are very, very important. And I just want to make sure that I've got my Docker running. I don't. Okay, let me start up Docker, too. Otherwise, it won't be much of a demo. Okay. Good. So, I'll let that run while we're talking. Um So, it has to be super reliable versus just works. What [clears throat] are AI agents? So, AI agents are

beyond simple prompts. They reason, plan, and act. So, AI agents are just code. They have access to models and MCP tools. Those are basically what you've got that separates an agent from any other Java Works with a model. Sometimes it is the model. Or an orchestrator. And goes beyond chat. Uses tools, makes decisions, and iterates on results. Key capabilities, perception, reasoning, planning, action, and memory are really,

really key. When you're working with a model, how do you actually handle caching and management of queries and responses? Think of it as large language models plus tools plus memory plus an orchestration loop. And it's not new, but large language models make them dramatically more capable and accessible. And prone to error if you have a lot of hallucinations and issues like that. So, you want to keep

things under control. Java's type safety, mature testing, and enterprise patterns really are good for building agents. So, you have access to tools and large language models through Java. It is really the best of both worlds because of scalability. I'll show you one great example. This was in in February, way back in February now. This is AT&T, the American telephone company, does handles for I don't know, 70%

of the cell phones in the US. 8 billion tokens a day they were spending on large language models and on AI. Very expensive and they used to cut costs by 90%. As I mentioned for the use, those of you who were in my safety presentation, you don't need the latest best model. You don't need to boil the ocean every time you have a question, right? So, you

don't always need all the low known knowledge of the universe to answer most questions for typical enterprise applications or you might not need your large language model as all at all. So, what they did here is they just worked query patterning patterning and they figured out what was needed to actually the questions and answers that they wanted and then they worked backwards and said, well, what do

we need large language models for in this set of questions? And it's quite an interesting article. I'll share it later, but yeah, it's pretty cool. So, they saved 90% by using orchestration. So, I'm going to show you how to set up orchestration for yourself that's scalable. So, the agent loop, you reason, you understand the task and the context, you plan, you break it down into actionable steps,

then you execute, so execute tools, API and code and like I said, it might not go to a large language model, might go to just a database to look something up or and then you observe, make sure that you're evaluating your results and iterating. So, everyone's familiar with the software development life cycle. Um that's what we're looking at here. It's just with an AI augmentation added to

it. Loops loops run continuously and they self-correct and refine their approach as as you go. So, CICD is key as well. So, agents matter for Java developers. Java is the backbone of most enterprise systems and they're nicely scalable and they automate complex workflows. Mature ecosystem is really important as well. So, strong typing you can use records, dependency injection, thread safety and proven testing are all built into

Java. There's lots of code out there and examples you can use whether you're using the code generation or not that will actually help you build those with tons of examples and patterns. They [clears throat] integrate with existing Java services via tool functions. There are enterprise requirements. For example, security, observability and compliance that already are handled in Java. You don't have to write your own processes for this.

Chances are you've already got some built in that can just be extended to handle your AI applications. Your existing microservices and APIs can also become agent tools. So, agent architecture patterns you have a single agent. So, once again, works on my machine. It's got one LLM with tools in a loop. That's the way things were in the back of the olden days of GPT 3.5. And then,

you know, most of the commercial chatbot vendors realized that they could actually save a lot of money by putting some kind of router in front of the large language model. So, then you saw a come out with this first where when you ask it a question, it decides which model it's going to use to provide the answer. That's a basic router. And then enterprises started picking up

on this as well. Like AT&T, the example I showed you, and building orchestration on top of their large language models to figure out how they're actually going to interact with the large language models and cache results. So, sometimes you can cash common results just like a database system, so you don't have to do round trip queries to your large language model. Um the single agent's really easy

to build and debug, but how do you actually track a multi-agent orchestrator as well? So, speaking of multi-agent orchestrator, let me start our multi-agent orchestrator in Docker. Uh we've got Jaeger here. Okay, fire that up. That takes a minute or so. >> [laughter and gasps] >> All right, sorry for going back and forth. I just want to make sure that's ready when our demo starts. Um in

the so, coordinator basically delegates to specialist agents and those specialized agents uh can be controlled and managed and upgraded and coded separately. So, each department in your organization, similar to what microservices are like now, can have its own process for updating the large language model router, orchestrator, and uh the handling of access to large language models as well. Uh each agent can have specific expertise and tools.

Uh and uh in this case I'm going to show an example of an invoice processing pipeline and it's great for multi-step workflows. I'll explain what the multi-step workflow is for ours uh when we get into that. So, you've got your user application, you've got your coordinator orchestrator agent that receives requests, it plans, it delegates, and it summarizes. Uh in my case, it checks to see if this

particular item is an invoice. Is it an invoice? Does it match an invoice? Does it have a header? Does it have line items? And if it resembles an invoice, then it goes to the large language model. Otherwise, you kick it back to whatever sent the invoice or whatever the alleged invoice is back to uh where it came from to uh be fixed and resent back to your

system. Uh so, the specialist agent A can be a domain-specific tool and B can be a domain specific tool as well. So, you can have multiple agents. One can handle invoices, one can handle payroll, other things like that. So, that's the whole idea. But, your orchestrator controls all of that, and it has to be scalable, obviously. Uh and each one of these agents, depending on the need

for it, has to be scalable as well, but their scaling needs could be different. Building agents with LangChain for J. So, uh they have uh as I mentioned before, it's an open-source framework for large language model powered applications. It's a layer that sits on top of large language models, and has compatibility built in and provided by vendors for certain models. Uh but, it provides a um a

standard interface for all the different models and their quirks and their special things that you have to do while working with them. It supports OpenAI, Azure OpenAI, GitHub models, Ollama, and many more providers. Uh and it has abstractions. So, that's the word I was looking for, abstractions and standards. So, it abstracts away all the things that you need to process with different models like chat language, AI

services, tools, memory, and retrieval augmented generation. Uh so, type-safe tools definitions via annotations and via records as well. Uh so, tool P and user message. I showed you some of those in the previous session on safety. Uh user message and system message. And uh it has built-in support for conversation memory and retrieval augmented generation. Uh it works with Spring Boot, Quarkus, Micronaut, and standalone Java. The demo

I'm going to show you is Spring Boot. Uh then there's Spring AI. So, LangChain for J handles more models, Spring AI handles more Spring. So, if you have a lot of Spring, you have to decide which one you want to use. You can use a combination as well. There's nothing stopping you from doing that. But, uh I always get asked which one should I use, Spring AI

or LangChain for J? And that's the differentiator. So, Spring AI is very, very integrated, not surprisingly, with uh Spring applications. So, you have a chat client API with fluent builder pattern, uh it's familiar to Spring developers. Uh you have function calling via bean definitions. Uh built-in support for rag with vector stores, including uh Postgres vector, Redis, and Pinecone. Uh observability out of the box with Micrometer and

Spring Boot Actuator. and you have MCP support for standardized tool connectivity, so you can call any tool that you can imagine uh using the standardized tools inside of Spring AI. Um and [clears throat] LangChain for J has more GitHub stars and users, uh but Spring AI is growing, it's an ecosystem that's growing quickly with thing with clients like Alibaba, uh Spring AI for Azure, and community extensions

as well. So, LangChain for J versus Spring AI, choosing your framework, these are very similar. So, uh I I already covered this, actually. So, framework agnostic, Spring Quarkus, uh etc. can be used with LangChain for J. There's more large language models, uh annotation-driven tool definitions. Uh fine-grained control over agent loops. Larger community and more examples and mature rag pipeline abstractions. Uh Spring AI Spring Boot integration is

very strong. Uh Spring native dependency injection. Micrometer's built-in. MCP support. Uh it's backed by VMware and Broadcom. Uh LangChain [clears throat] for J is is backed by IBM and Red Hat. So, you're in good hands either way. Uh ideal if you're all all in on Spring. So, those are the differentiators. Uh real-world agent use cases. So, DevOps automation. I'm going to show you an example of that

in a bit. Uh and uh there's some cool new features in GitHub that you might not be aware of like agents, uh customer support and document intelligence. In this case, I'm going to show you So, the demo architecture I'm going to show you uh it has a demo uh the demo runner actually this is the Python version, but there is a Java version as well. They're both

built in and it calls the orchestrator the Java orchestrator actually calls a GPU processing system that parses the invoice and I use the NVIDIA NeMo Tron parse model for that. So, you'll see um I'll show you that in a second. Uh but it's nice for normalization and anomalies and it has automatic logging that goes into uh hotel with Jaeger and Azure monitor. So, uh yeah, let's go

into the the demo itself. So, basically I send a PDF from repo and the PDF gets delegated to a GPU. I'm going to show you how I actually use the GPU uh on the cloud and then the parser service fetches the PDF, extracts it uh via NeMo Tron parse num. There's one spot that's missing there. The reason why I wrote the extraction agent in Python is because

that's the interface for the NVIDIA parse uh NeMo Tron parse uh uh API. So, it doesn't have a Java version. You could use LangChain for J for that. But in this case, I just stuck with the native Python. I used the code they provide and use that for actually doing the parsing. And I use a service for doing the parsing as well. I didn't have to deploy

it. So, the data's normalized. Uh everything is uh standardized. Uh anomaly rules are applied. They have three anomaly rules which I'm going to show you that check against the invoice to see if this is an invoice that needs to be accepted or rejected. And then the orchestrator returns a structured JSON and human-readable summary that you can uh use to log and/or justify the decisions. And it's full

traceability visibility. So, let's go into the demo. I'm going to talk about NeMo Tron parse. Let's go and show you the demo. So, basically we've got uh this is my resource group in the portal.azure.com or Azure portal. You can get a free trial if you go there. Uh portal.azure.com. And uh we've got several things in here. So, what I'm trying to show you is Azure Container Apps.

So, imagine if you've got a Kubernetes cluster and that Kubernetes cluster has different nodes and you could borrow one of the nodes just use it while you need it and then give it back. And you only get charged for the moments that you use it. Instead of having to set up your own control plane and management plane for a Kubernetes infrastructure. So, that's what Azure Container Apps

are. And Azure Container Apps have another cool feature which is you can actually a GPU instance. I have an A100 instance running for Azure Container Apps. Uh and so I've got really powerful, really cool stuff uh built in if the Wi-Fi works. It's just all sitting there on the other side of the Wi-Fi waiting for us to use uh the reason why I chose a GPU is

because it's better at parsing. So, I use the NVIDIA NeMo Tron parser, but I didn't deploy a model. The NeMo Tron model. I could do that cuz I work for Microsoft and they have it in the AI foundry. I can deploy it and I can run it, but that costs a lot of money to sit there and run. I only use it once in a while for

demos. So, I went to build.nvidia.com and they have the NVIDIA NeMo Tron parser that you can use. You just have to sign up for a developer account and you they give you an API key. They give you some code to access it in this case with Python, so that's why I wrote the access agent when Python cuz it can use a deployed NeMo Tron model which can

be running on the new on Azure or I think it's on the other cloud providers as well. Uh but also it can run without having to deploy anything. So, I can just use it. It's free and it works I've never run out of space >> [snorts] >> This is the portal. The GPU also I already have a GPU running here. I've got an application URL that I

can use. It's got a workload in here, so if I go into my containers, I've got a workload set up. And that workload No, okay. That workload Ah, there we go. It's called GPU parse agent. Uh I can have a CPU cores, so this is in this case it's a NVIDIA A100. I can have CPU cores of four and eight, but you can also have an allocation

which is basically zero to scale. I can set the scale Hey, boys. Hopefully you have a good editor for the videos. I set the minimum replicas at zero. I set the max replicas at three. That way if I'm ever using this after 300 seconds the timeout will just shut it down to zero so I don't get charged for it. And I also take the extra step of

stopping it when I'm not using it uh and uh making it run. So, let's go ahead and start the application. All right. So, in this case that's not what we want. So, let me do this quick. So, in this case we're using an amp file. And the actual command I want to use is down here somewhere. Excuse me. You can run this on Docker. You can also

run it Oh, there it is. Okay. This is an example of things go wrong, I'm glad I made it today. Let's see here. This is the one we want. So, this isn't going to work the first time. And the reason why is what I just showed you was the scale. The scale is going to hit that parser and it's going to tell me that the the uh

there's the application health is not working. That's because it's scaled to zero. And once it hits this, and it might take two or three times to to get it scaled up. But once it does what it does is it processes the uh there's some invoices down here. Sample invoice. Those are out on the GitHub repo as well. Uh there's one that has anomalies and one that's clean,

except the one that's clean has a addition error. Um and what it's going to do is the orchestrator is going to look at this and decide item that's being sent, in this case a it's going to analyze it and see is this an invoice? Does this look like an invoice? So, the orchestrator does that job. That can run on a CPU using GPT-4. That's what I've done

here. So, GPT-4 is what actually gets used for and the Wi-Fi is Okay, there we go. Parser service URL, checking parser service health. It's going to fail the first time. I think that was the Wi-Fi again. Take it that long. Um so, what's going to happen is it checks to see if this is a real invoice and then it decides, "Okay, it's a real invoice. I'll send

it to the parser." And the parser is running on a GPU. So, the GPU parser runs very quickly compared to this. So, it says parser service returned HTTP, that's because it's not scalable. So, usually this takes two or three tries. Once the parser gets it, it converts it to a JPEG because that's the only thing that NeMoTron Parse can read. So, I have another piece of code

inside the GPU that converts it to a JPEG, which is very fast for image processing. Uh GPUs are good for that. CPUs are a little slower. Uh and then for actually parsing and creating these things called bounding boxes, which determines what the document looks like, uh that is actually done by the um by the parser as well. Uh and in this I'll let that run. This the

Wi-Fi is still flaky. We're still on speaker. Yeah, cuz this usually takes a second and then it tells me it's not ready. It might actually work this time. All right, so we'll go back over here and I I into build.nvidia.com and I search on parse. This is the only model I know of that's actually built for parsing. That's right here, so I don't even need to search

for it. I'll just click on it, and there you go. So, this model here uh something you can use. It It handles an image. It's got all kinds of information and APIs, and I can say try this. Uh I can try it here manually. I can actually upload a document and test it see what it's going to look like. Or, if I go in here to Python,

the only Python and shell as I mentioned before, for the orchestrator uh we use Java. For this, we use Python because we can convert things with Python just using this code. So, I take this code right here, and I copy this into my code generator, and I say, "Hey, build something around this that's going to work with my Java agents that is a invoice detector, and passes

information back and forth securely." Uh it's nice because it's scalable, and it's secure. So, Azure container apps, that's the other thing I didn't mention. I did actually show you the scalability here. So, I can make this up to a thousand replicas of GPUs. So, I can have a thousand GPUs processing my invoices. Well, it would cost a little, but uh you know you don't want to do

that probably, but uh you could if you wanted to. yeah, this is just taking a while. I'm going to just try it again. Look it up. Azure app's ending point. We got good. Okay, checking parser health. See, that's how long it's supposed to take. So, the parser service is healthy now, meaning it's scaled up from zero, and it's running. So, now it's actually going to grab the

uh the invoice from GitHub. In this case, it's going to grab it locally. Uh and it's going to uh go through these steps. So, download the PDF from GitHub, convert each page to a PNG image because the PNG has to be handled uh send each page to NVIDIA NeMoTron Parse NeMo API, and we're going to use the model NVIDIA NeMoTron Parse, and the endpoint is here. So,

we're going to use chat completions for this, and we specify the model, uh and then we we we uh merge the results from all the pages, and we extract the structured fields into these things called bounding boxes with the vendors, dates, line items, and totals. We normalize the rounded amounts and the default currency. So, let's check the first one. This one should just have a little bit

uh addition error. Yeah, so uh oh, okay. So, it's okay. Found everything it needed. So, um line sub to zero, but subtotal is 197. I don't know why it did that. the next one should find several anomalies. Uh in this case, same thing, same process. Uh it's going to extract the invoice. Here's what the invoice looks like. So, it scanned this, put it into these things called

bounding boxes, then reassembled it into text that's being displayed here. Uh and it checked these rules. Subtotal versus sum of line item amounts. So, is the total adding up to the line items? Uh the unit price outliers or something weird about the unit prices. And then is there any missing field? So, it detected uh one anomaly. It price outlier, high price premium support 250 versus 42. Uh

the unit price is 250, it should be 42. So, it will reject that one and send it back. Um and the nice thing about that is the orchestrator runs and does most of the prep work and then the reporting work. The GPU is only used for converting the PDF to a JPEG and then taking the parsing and getting all the data from the bounding box. In this

case, we're not even using the GPU for the parsing because the model isn't running on that GPU. It's running out in Nvidia on their own GPUs, and we're just borrowing their model to do that. Um so, we're borrowing a For the orchestrator, we're borrowing a node from the Kubernetes cluster for Azure Container Apps. Uh and um Sorry, we're we're That's for the parser. I'm getting mixed up.

Uh the um the orchestrator is running on agents on Microsoft's Foundry. So, if I show you Microsoft Foundry here, this is Microsoft Foundry, and you can actually create agents uh with these tools. Uh in this case, I can use GPT-4, and I can ask questions about my prompt, or I can set up code to access this uh play. You can use YAML chat, YAML, and there's also

code. Let's refresh that. What is it? What do we got? 6:15. Okay. >> Probably. Anyway, what this shows you is that you have different options for code. There's Java, JavaScript, etc. So, you can copy and paste that code and build an agent with it, and you can run that agent on Microsoft Foundry Agents service right Which is probably going to time out as Yeah, it's definitely a

gypsy curse. Um anyway, um so, agents there I set up an agent. I built it. It's scalable. So, it can scale up to literally thousands of instances depending on load, similar to what I showed you in Azure Container Apps with the instances of ACA, but that's a GPU running, and it scales up the GPU instances. Uh um Both of these things combined talk securely to each other

via keys. And those keys are Let's see if this will display. These keys are in a key vault. So a key vault is something you can use to store endpoints and credentials. And the nice thing about that is you can just point at the key vault. And if you change your endpoints or your credentials, you can change that inside a key vault. You don't have to affect

any part of the access them. And the key vault is all embedded into Azure identity management. So everything you're building here can you you access you grant access to each one of these resources one by one as you build these things. Let me go back to the No, Foundry. Anyway, you'll take my word for it. The Foundry agent is sitting there and it could be scaled. So

that's basically the whole thing. Last but not least, I wanted to show you Jaeger, which is tracing. So it's actually tracing this right now on port 16686. And there's the orchestrator. And if I show the operation all, So basically we just got a couple of traces here. But the nice thing about this is you've got a timeline that could be used. The graph. Yeah, timeline. It's not

showing everything. It's only showing the call parser. Um usually it shows a couple different things. Let's see. Let's try another one. No, that's a span. Usually it gives you the timeline of the orchestrator calling the parser and retrieving data. And these are obviously the ones that I used before. Oh, there we go. There's one. So the orchestrator calls the parser, returns back to the orchestrator, and everything

happens in uh a couple of seconds uh in this case. So, you can track your you can log your activity through hotel, and you can manage and improve any performance you have as well. And you can check each one of these to see what was actually done under the hood as well. Okay, so that's basically the demo I with a little delay the Wi-Fi. Um so, the

multi-agent invoice analyzer uh it's built purpose-built. It uses the NVIDIA M uh NIM microservice. Uh and uh the orchestrator received the PDF URL. It had a unique request ID and trace ID. It passed it to the parser service, and that parsed the PDF via Neumotron parser. It also worked on the GPU for converting it from PDF to image. And then uh it's extracted all the line items

and checked for anomalies, basically. what do you actually use for production patterns? Uh error handling and fallback strategies, so you want to have health uh similar to what I showed there. Hotel is just a very, very basic example with the default Jaeger interface, but uh you can build your own uh for whatever you need. But dashboards as I mentioned before, dashboards are really, really one of the

easiest things to build using code generated these days. So, it's you can have auto detect mode based on service availability at startup. Uh retry with exponential back off. Uh so, in this case you saw how I checked for the health of the application on the GPU before I actually proceeded. Uh and that's one of the things you want to do to just check and see. You don't

want to be passing things to a parser that's not ready to be accepting data. So, uh you can create a circuit breaker pattern for downstream services, so you don't have cascading failures as well. Mine's really simple. I just check to see if the GPU is running. I check to see if I can access the endpoint for the orchestrator as well. Uh and if I can, I use

it and there's actually a built-in fallback as well that'll do some Uh and I didn't build this. Claude 4 6 Uh Opus 4 6 built this uh part of it. They built a mock generator. So, the original code I built, this is one of the things you got to watch out for with code generation. Uh the original code I built works perfectly right out of the box.

Everything works great. And then I looked at the code and I realized it didn't do anything. It just created uh a response that looked like it accessed the data and returned the result. >> [sighs] >> And it was like, "Okay, so I you know, I And so I asked it, well, this is not accessing the GPU or the agent service I built. What's going on?" It said,

"Oh, yeah. Well, I had a hard time accessing the GPU, so I just cuz it cuz of the scaling issue that I showed you earlier. It said, "So, I just made up some data and put that in the code. Yeah." Works. It works. Yeah. You got to be careful. Uh you know, it doesn't know right from wrong. It's crazy. But um yeah, it it did eventually generate

the code that I needed after about 10 iterations and uh much faster than I would have done if I done it myself. I guided it and I edited some of the code by hand. So, you do need that knowledge. Yeah, as Vinod Kat was saying in his presentation today, uh you really need that background knowledge. It's really, really valuable to know, for example, like a a freshman

developer might have looked at that and said, "Oh, I'm done." at the first try when it's using the mock data. You got to be careful. You got to know what you're looking at. Um so, [clears throat] observability, open telemetry end to end. Uh so, uh the orchestrator, parser, and actually traced everything through, so you saw that timeline. It went through the orchestrator, parser very quickly, and then

back to the orchestrator. correlation IDs, there's a request ID on every request for audit trail. There's a span hierarchy, so you do a handle request, call the parser, parse normalize, and anomaly checks. And then there's an OLTP exporter with console with console fallbacks, so that's nice. And then auto logs, so there's structured JSON that gets built into this, and you could check that for for weeks or

months. In fact, I believe in here I limited Oh, I'm looking back to the last hour. Let's look at it back at the last 12 hours. Oh. >> [laughter] >> Cuz when I refresh this, it refreshes everything. Okay, so that's it. Never mind, bad demo. but technically, if you had the logs that didn't get deleted and recreated every time you did it in a real enterprise, you

would have all kinds of logs in there, and there would be dots all over the screen showing you the different performance characteristics of the the Evaluating AI agents, deterministic testing, there's tests for tool functions, integration tests with mock LLM responses. Not that you want those, but it's built in there if you need it. So, I actually kept that code for the mock LLM stuff, but I make

it tell you I don't know if you noticed in the code, it said, "I'm using the non-mock version of this to run this code right now." But then it will fall back to the mock version if it's offline. So, that's kind of nice. Although I do check for the GPU, so it won't let me use it. But if there's for some reason it won't access a GPU

mid-process, it'll use the mock data, and it'll tell me, "I'm using mock data." Um and then you can have a golden file that tests against known good JSON, so examples that you want to use. For example, in this case in the repo, I've got some I got a clean and anomaly invoice. You can use those to test the application as you go along. And you could use

a lot more data, obviously. I just have two invoices cuz I'm iterating quickly. So, okay. [clears throat] So, LLM is judged for non-deterministic deterministic evaluation. LLM is judged as second model evaluates output quality. So, you can have that as well. I don't have that built in here, but you could have another model that checks to see if your first model did a good job. You could have

a human in the loop as well for edge cases, so you can have a score, for example, that says, "All right. So, this invoice I'm 75% it's sure it's okay. 75% and above, go ahead and process it. Anything below 75%, I might want to flag it and let a human review it later, and then see if there's something we can improve in the system." Um then you

track metrics like accuracy, recall, latency, token usage, and use AB testing agent responses, and then prompt regression testing across model versions. That's important All right. So, guardrails, AI proposes, humans approve. Scope agent permissions. Restrict what tools can access and modify. So, that's really important, and Key Vault is really, really good for that because you can segment and separate data and agents to make sure that everything is

controlled and everything has an access point, but it's centrally managed through the Key Vault. Uh validate all large language model output against schemas before acting on it. Rate limiting a token budget, so you can keep track of runaway costs by setting up rate limit and token budgets. So, that's going to be built straight into LangChain for J and Spring AI. And then content filtering. uh, I had

a whole session on that. Check out the video on that one uh when it comes out after the conference uh for blocking harmful or off-topic outputs. And then uh approval workflows for high-impact actions, audit trails, and a kill switch if you need to disable agents instantly if they misbehave. In this case, all you need is bad Wi-Fi and uh they work. >> So, um yeah, so deployment

strategies um One of the cool things I talked about Azure Container Apps, we have A10s and A100s. We also have T4s, Tesla T4s. Uh you can use those for your GPUs and you can just use them for testing and uh if you're going to go into deployment uh officially, you might want to run a VM with with uh uh dedicated uh GPUs, but in this case, this

is really good for dev test. Um and you can scale to zero and I'll just like I do. I'll built-in ingress, TLS, custom domains, and health probes. Uh one thing I want to show you. So, Azure Kubernetes Service, if you're going to move beyond dev test, we have a full Azure Kubernetes Service and any Kubernetes Service is similar with a management plane and a control plane. Uh

Microsoft Foundry, so this is where I figured out which models I want to use. Let's see if I can do this now. So, Microsoft Foundry Nope. >> Let's see if it works. I can look at all these models, not the agents. So, I want to look at the Models. There we go. So, there's a bunch of models to choose from. And up here you have to trust

there now. Uh there's the models uh and I can compare them here. In this case, I can check, for example, GPT-5 versus GPT-4 versus Grok 4. Quality, safety, and estimated cost and throughput are all a little bit better with GPT-5. Uh and there's all kinds of other here I can check as well. So, you can compare models. GitHub models has a similar feature. Um, and uh, I

believe in video building video.com has one, too. Uh, but you can use that for models. Uh, last but not least, I wanted to show you something in uh, GitHub for deployments. So, this is really Repositories. Huh, I feel nervous. Uh, going into Wi-Fi now. Okay, there we go. um, two things that are really cool inside of GitHub repos, in case you're using GitHub, most people are. Agents.

This is really cool. Agents can run directly in your repo now. So, I've got two agents here. This is a different application. This is not the application I showed you earlier. But it checks for any, um, security issues. Uh, and uh, it's all built in. Um, review this. Check, here's the prompt I used. Um, review the order service.java file. Check for OWASP top 10 vulnerabilities, especially these

ones. And it goes ahead and it does this. Now, what's the power of this? Uh, it's kind of cool because you can just put this into your uh, repo and it'll be triggered every time you have a pull request or a build. Uh, and that's nice. And I kicked that off. I didn't actually kick it off on my laptop. I kicked it off on the GitHub app

on my phone. And you have a little button here that says new session. Can everyone see that? No, I'm just kidding. Uh, but you can actually ask Copilot to do things. And it'll run on here, and then when you get back to your laptop or wherever you are, this is actually the results showing up in the repo on your screen. So, you can actually use this thing

to talk to and you can use voice feature in GitHub uh, Copilot app to speak to the agent and ask it to do something for you and then have it run and have the results back here. And one thing that's even, uh, this I'll also cool. Inside of actions now, inside of actions now, come on. Come on. It's not going to go. Actions. I was going to

show you that you can call agents inside of actions. So, there's, uh, just like when you're building GitHub actions, there's YAML. Inside of that, uh, YAML, there is a way now to, inject a prompt. And that prompt can run an agent inside of your GitHub action, which is kind of cool. I've got one that runs a huge Maven, it's got probably 30,000 lines when you do a

Maven build on a complex app. Uh, and what it does is it actually reads the Maven output and summarizes it into any issues you might want to look in a markdown document. So, yep, didn't go. It's not going. Thank you, Wi-Fi. I want to show you that. I'll say, well, I'll come back to that. Uh, so those are two things for your CI/CD. It did? Okay. Thanks.

Oh, good. Yeah. So, every time this runs, every time an agent runs, it triggers that, or every time a build or a pull request gets processed, it runs those that agent, it also runs this action. And the action itself has a very long, uh, process here for actually building And it produces this markdown build report. So, it reads those 30,000 lines of Maven so you don't have

to and it summarizes the test results for you. In this case, there's 16 tests that were Uh, there's one that uh, for order service application tests and it ran all of them and passed all of them. Then there's deprecation warnings here. It gives you all this information and it even tells you that node.js 20 inside of your agent inside of your action needs to be updated. So

that's kind of cool. The actual action I go over here AI pipeline down here. There it is. So you have to use a GitHub token so you're using and you send it a Copilot path and then you basically run it's the Copilot CLI. So you're actually running the Copilot CLI that runs an agent on the GitHub model and checks to see what output you want to put

into a markdown file and it creates a markdown file called build report.md and that's how you actually get that markdown. So when you look at the that result it's a markdown file much easier to read than the 30,000 lines of Maven. Microsoft Foundry I showed you a little bit about that. I talked about my GitHub models and Microsoft Foundry models. So Microsoft Foundry models you deploy a

resource which you pay for all the time. GitHub models you can just borrow stuff So that combined with the ACA and building video.com is really cool for for building stuff and testing stuff out. So the developer experience there is VS Code and then there's GitHub Copilot. So VS Code is what I showed you in the demos today. GitHub Copilot I'm mostly using When I do this I'm

actually using Claude Opus 47 but there's all kinds of models I can use for this. And that's how I actually generated most of the code here and the documentation for sure. And it used it's really good at generating Python and Java code. So it built both equally well as long as you provide some good examples. We also have the Copilot agent mode that I showed you inside

of GitHub actions and the GitHub Copilot agent as well. So I think we're going to skip it there or the we're going to skip through these last few pitfalls to avoid. Definitely check this out on the repo same with takeaways, resources, and links. Those are all going to be at aka.ms/microsoftnvideo. Uh so if you go there, this is the one you you want to take pictures of.

This takes you to the GitHub repo with all this on here. The only thing that's not on there is the latest slides which I'll put on there later tonight. So We made it through with the Wi-Fi and everything. So thanks and enjoy the rest of Gids and do check out that demo repo. So thanks for coming. >> [music]