It Works in the Demo. Will It Work in Production? Evaluating and Debugging AI Agent - Apurva Misra

In this talk, Apurva Mistra discusses the evaluation methodologies for agents, especially in the context of machine learning. The session builds on previous discussions about building reliable agents, exploring the metrics that define successful performance. Mistra emphasizes the importance of reliability through establishing clarity around definitions of success, which can differ based on domain, such as healthcare or education. The speaker proposes creating an evaluation suite that includes trace analysis for understanding system behavior and outlining specific scenario tests. Throughout the session, Mistra also notes the challenges faced by agents, including tool failures and the non-deterministic nature of machine learning outputs. As a case study, the talk details the development of an evaluation suite for a nanny agency agent, focusing on memory, planning, and the various metrics that inform reliability and user satisfaction.

Hey everyone, uh my name is Apurva Mistra. Um today we are going to cover um evaluation in agents. So yesterday in my session I covered um what is important when you're building an agent. uh what are the pillars, what are the issues when you're building an agent because you're using something non-deterministic to begin with. How do you make it reliable and production ready? Um so in today's

session, we'll cover the evaluation piece. Um so it doesn't matter if you already have an agent or not in production. Uh it's applicable either ways. So if even if you're like starting with an agent, you would want to do evaluation and if you have it already in production, you would still want to do evaluation. So we'll we'll go through the whole methodology of how you would build

your evaluation suite. Um in this presentation about me uh I was already introduced but uh I'll introduce myself again. My name is Apurva. I'm a machine learning engineer. From the last two years I've been consulting independently. I have my own company called Centic. Um I've also been speaking at a lot of conferences uh events. Um, and I run workshops for companies, um, getting them up to date

with the latest tools like short code and stuff, getting their employees, um, uh, excited about, uh, using them. Um, and also like helping them build their AI strategy. Um, I'm also a lot into working out and stuff. Um, yeah. Um, uh, before we start the presentation, I added this slide like today morning. Um, GPD 5.5 got released yesterday. So um this is how quickly the space is

changing. Um so what I'm teaching today might not be applicable in a month. So like what I was teaching yesterday like the hardness that you build around your model to make an agent reliable, you would need less of that hardness as these models improve. Um so yeah, this is uh GBD 5.5. Uh it has like better scores than um Opus 4.7. Um I haven't used it yet.

uh but uh on their page obviously open a saying it's amazing. Okay. So in this session what we will cover is we'll define what reliability means um what is good for your agent and that would differ for each one of you you know depending on which domain you're building the agent for healthcare education um your good would be different um then we'll build our evaluation suite and

then we'll put that um into our pipeline so that you can actually use that it's like tactical and applicable. Okay. So um I had the slide yesterday as well. So an LLM is like a black box. You give it an input. It gives you an output. So if you have used chat GPT cloud okay I just want to do a survey. How many of you have built

agents? How many of those these agents are in production? Okay. Very few. Um and uh how many of you have used any of these agentic frameworks out there? There are so many. I'm not going to name them. Okay, cool. Um, uh, so I I'll start from the basics then. Um, so blackbox, you give an input, you get an output. Um, so for an agent, like there are

new things that gets added and you're doing this in a loop. So it could be a while loop, a for loop. Um, you're asking the agent, uh, I want this particular task done. Um, it's going to break down the task into smaller pieces. That is what is called planning. And then it'll be like okay I have to um schedule something for that I have to go check

the look at the calendar tool then I have to send an email so I have to use the email tool. So like it'll call a tool get the output then it'll call another tool get the output and keep doing that till it's able to like it till it's satisfied itself with the output and it'll give it to the user. Um so tools memory and planning it needs

the memory piece so that it knows that it actually called the calendar tool and it doesn't call it again or uh to improve itself you know if it made a mistake uh while pulling a library this library is not relevant it'll go pull another library try experimenting so you need the memory piece there okay so um this is from a paper uh agent reliability um and they

were trying to it's so hard to define um what a reliable agent is right um so this paper is all about defining that and they pulled these pillars from the aviation industry cuz that is also like if if your plane is not working all of all of the passengers and the people working on the plane would die. So it's like an industry you where you would want

to pull um reliability pillars from. Um so uh in aviation consistency is really important. Um so you give it a query it should give you an answer even though um it wouldn't be the same exact answer it should satisfy the user. So the plane should like take off and take you to your destination. Um robustness. So um if there is some sort of failure they have like

a lot of in um airplanes they have um uh fallbacks. So that's where like the robustness robustness piece comes in you know like it's even if like something breaks the API is not working it has like a graceful fall back. Uh predictability is like again like doing the same thing but like for different stuff like um if in planes you have it goes on the auto mode

and you can switch it back to a human. So if um it's not working well it should know I can escalate this to a human being. Um safety you don't want to die. Um if someone is asking the agent about like medical advice you don't want to give that advice. Uh you would want to direct them to a human being or like send them to like um

give them information about the medical services. Um so these are like the four important pillars. Um and all of this is important um to to make sure people can trust like we're trying to build trust with the users right they would want to use this agent they shouldn't be like okay I used this yesterday it worked and today when I use it it did not work and

then they wouldn't come to your agent on the third day uh we would want to avoid that. Um, so why agents are harder than just give me a second. Okay. So why agents are harder than single turn LLM? This is again the slide from yesterday. Um, same thing context limit. You're using the underlying piece is the LLM, the black box, right? So you have the context limit

as the issue. Um, nondeterministic probabilistic. Um, cascading tool failures. If one of them fails, the next one gets the wrong output and it would fail as well. Um, infinite loops. I showed an example yesterday. Like they it's a big issue. Like they can go into infinite loops like trying to like pull the same library again and again and again and it's going to cost you tokens. That's

all. Um, silent failures. Uh, it would keep doing stuff and utilizing tokens and you won't even unless you're like staring at it in real time. You won't even realize it made a mistake till it gives you the output finally. Okay. Um so there are a lot if you Google search evaluation of AI agents you'll find a bunch of vendors trying to sell you something and the kind

of scoring that they use is like conciseness score uh truthfulness score and like what does it even mean if someone tells you my agent has like a truthfulness score of seven like what does it mean? It was like lying three out of 10 times like it doesn't really um match with a business metric. it doesn't like it's so subjective so hard to so like one of the

if you take away one thing from the talk don't use the vendor to rules um just don't use them you can build your own things very easy um so what is the definition of good so when you're building if you're in the healthcare space for example you're building your agent like how would you define what good is um cuz this agent it's not like software right like

you give it an input and you get like the final function output and that is it and you can like um have assertion you have you can have unit tests, you can have integration tests and you can have assertions and you can check if the output is correct or not. Um in this case it's like it's such a long process. It's calling so many different things and

uh looking in the memory putting it all together uh it's context changes every single time like how would you define what good is like if it's able to do like six tool calls correctly and in the last tool call like the email send it makes a mistake there. It was like pretty much there right 95%. it just didn't it's just it created the whole email but it

couldn't send it. Um so it's it's there. Um so it depends again depends on your use case like maybe that is good for you. Um so you would have to define this and this is where the evaluation piece comes in. It'll help you define what good is. Um so the reliability contract is what this is what you need for um a good agent. You have to define

what counts as full success. uh what partial like where partial success is acceptable maybe not in your case you know um what agency level should the agent show. So this is basically autonomy. Like how much autonomy are you going to give your agent? If it's able to do refunds, if you're allowing it to do refunds, maybe you would want to put a limit on the amount of

the refund just $10. It can just do refunds to like $10 and beyond that it has to like escalate that to a human being. Um and like which failures are safety critical versus quality issues for like healthcare. I feel like all the failures are safety critical if uh you are in that space. Okay. So in today's um presentation uh I have chosen um I'm going to use

an example and we are going to uh build the evaluation suite for that agent example and the uh agent I've chosen is an is a nanny agency agent. Um I don't know how many of you are aware of nanny agencies. This is the thing in uh North America. Um so a nanny agency is basically like a two-sided marketplace. It tries to match the parents with nannies. Okay.

So, a parent would come to uh the nanny agency and be like, "Okay, I have two kids. I have two dogs, and I'm looking uh for a nanny to help me out over the weekends. I want a nanny from like 7:00 a.m. to 9:00 p.m. every weekend." And um the uh nanny agency would be like, "Okay, these are the profiles I have. Okay, I have to find

a person who is not allergic to dogs, is uh friendly with dogs, and is uh okay um uh having like babysitting kids who are like from an age group of 7 to 10, I don't know." So the parent would give like a bunch of constraints uh and requirements and the bot would actually or the nanny like in in our case our bot would do that but uh

nanny agencies do this. Nanny agencies go go through like their whole database of nannies. Um these nannies are background checked. There are a lot of things that goes behind it. Um uh they know like what are the qualifications for the nanny? This nanny knows Spanish and English that is important to the parents. it'll find the right nanny profile and suggest it to the parents and the parents

could like then interview the nannies and whatever and be like okay this one is good one you can like schedule them for the weekend um so we are going to define uh um the valuation suite for this bot does it make sense this could be an interactive session it's a smaller room so um stop me if you have any questions so basically it'll look like this like

a parent asking for something so these are like nannies have like a bunch of certification that people might be looking for you Um so there are a lot of like constraints that the bot has to look through in our case. Okay. So um this is I have defined this. In your case it'll be different. Um so in our case uh understanding the constraints are very very important.

Um if uh a nanny is allergic to cats you don't want to assign that nanny to the parents. Right? Very important. Um persona and tone adaptation. Like if a parent is frustrated reaching uh out to you on a Friday evening, you want to be nice, um reassuring, not robotic. That's very important. Um safety and compliance. I wouldn't want my bot to give medical advice to parents. Uh

I would get sued, so I don't want that. Um safe escalation. So if a parent is asking for a refund, if a parent is very pissed off, um you want to escalate this to a human being. you don't want to like continue the conversation as a bot and frustrate the parent even more. So you the bot should realize when to escalate the situation to a human being.

So in our situation that's a bot that's the parent and these are all the tools that a bot has access to. So the search database so they it can go and look at like the initial constraints find like a a bunch of nanny profiles and then it can query those nanny profiles um see what are the different so if there are like good things like it can

say oh I found Sarah she's not allergic to dogs she's amazing and she also knows Spanish so if you want your kids to learn Spanish you should uh so it should query the nanny profile find the right things to chat with the parent about um check availability so is the nanny available over the weekends um schedule the nanny um send email. So these are the tools that

the bot has access to. Okay. So the flow for um defining our evaluation suite looks like this. We are going to start with collecting traces. Then we are going to analyze those traces, define a texonomy. Then we are going to define a golden data set, a bunch of scenario tests. And then we're going to use um that to define our evaluators. And then we're going to put

those in the CI pipeline um and also use that for online monitoring and drift. I'll go over each one of them. We have plenty of time. Okay. So understanding the system behavior if you want to anything you're doing with AI, if you want to understand it even better, look at the data. Um so error analysis, um you pull in at least 50 to 100 traces. if your

agent is out there in production. So like 50 to 100 parent queries um and analyze those. So I wouldn't suggest do it in random. I would suggest to be very um critical about this you know like you would want to be as diverse as possible. So there could be parent requests just looking for nannies um just asking random questions about nannies. There could be requests for scheduling

nannies. There could be requests just asking about the nanny agency. There could be like different kinds of requests that you can group together, cluster them together. So you want like a couple from each one of them. So you want your data set to be diverse enough. And if you don't have your agent out there in production, go with a synthetic way. You can use these LLMs to

do anything honestly like you can generate these queries using LLM. So just to be diverse enough, you would want to use these three pillars. So for your agent, you would have like a bunch of features, right? like your agent can schedule stuff, your agent can like send emails, your agent can um um uh write code, read code like a different like depending on which domain are you

in. So in this case it can do weekend scheduling. Okay, scenarios. There could be a bunch of scenarios. Um it could be like you don't have any nannies available. You uh it could be um the nanny agency doesn't work over Fridays. There could be different scenarios and there could be different personas. a parent who is happy, a parent who's from Chicago, a parent who is uh frustrated.

So, a bunch of personas and like different combinations of those would lead to a different query. So, this is just a way of getting diverse queries. You don't have to have to do that, but it'll lead to like better um error analysis. So, in this case, for this combination, this is the query I'm getting. My nanny canled 10 minutes ago. I need someone now for my toddler.

And the the reason I would suggest doing this is it it's about um it's not just about like checking if your system is good enough. It's about defining the good for you. I'm going to I'm repeating this so it is so important to look at the data. Um so even if it it's not in production, even if it's an idea in your brain that I want to

build this agent, start with this like what kind of queries you want your agent to answer. It would help you define the kind of tools you want your agent to have access to, the kind of memory you want to create for your agent. Maybe working memory is enough. You don't want long-term memory. Um, so do the eval suite. So, um, once you have these traces, uh, obviously

you I'm not a nanny. I'm not a nanny agency owner. I'm not a parent. So, I would need like a domain expert to look at those traces and tell me what's good and what's bad. Um and in this case the domain expert would be like the nanny agency owner or the nanny agency ad administrator like you get them on the table and like look at the traces

together with them and try to figure out what's good what's bad. So in in some cases um there would be like more than one and you would have to like align on what good or bad is. This is where you define the rubric. So um this is what an end to end looks like. uh if I didn't mention this for before but basically like an input goes

in and then like we look at each tool call uh we look at like each uh thought thinking uh that the LLM did and at the look at the final output and see like where did it go wrong or where did it go So um for looking at these traces, the simplest thing you can do is like pull the traces, pull the user query, pull the traces,

put it in Excel sheet and give it to the domain expert and be like label all of them. Give it a pass fail and like your commentary like why did it fail, what was good, what was bad. Um or you can also build your own um trace viewer system so that like they can just click next and look at the next trace and like put their commentary

in the uh in the next a text box area there and then it'll get saved into the DB automatically. It'll just make it easier. Uh it's easier to like they would also have fun doing it instead of filling the Excel sheet. But Excel sheet is also good enough. So you can build your own like nanny ticket trace viewer um and give it to the domain experts. So

um in this case for example um it's an email um asking for something and it replies with this output and like they are saying it failed and why did it fail? Um so it would look like that. Okay. Um once you have uh um your trace viewing setup ready you give it to the domain expert they would do the labeling and that is called open coding. So

basically for for example these are like a bunch of I'm going over there like five slides explaining this. It's because this is like the most important piece of uh setting up an evaluation suite. So in this case it's like great uh let's schedule nanny Sarah for tomorrow. Um and these are the trace steps. It had to do an API call and it made a mistake sending this

argument. I don't know if this is laser isn't okay. So it made a mistake um um sending this argument. It send it tomorrow instead of like this format. Um and like the domain expert doesn't have to say this. We would come up with this but the domain expert would say like this failed and this is the this is why um it failed like it did not give

um the right uh date or it did not like set up the call uh schedule the thing uh with the nanny. That's all. U you don't expect the domain experts to be techn technically uh aligned with you. So they don't have to give you the technical stuff. they would just give you like pass fail and like what did not happen that they expected it to happen. Um,

so in this case, like someone was looking for a CPR certified nanny. Um, and it like went through the steps. It found some somebody um, uh, would you like to book, uh, her, it ignored allergic to dogs in the nanny's profile. Um, so that was wrong. Um, and this this is something the domain, uh, domain expert would tell you like I wouldn't know which nanny has this

issue. She would she or he would know their nannies and what the issues are. Um, so they would tell you that. Okay. So before I book nanny, Jessica, can you give me your exact home address? And it gifts the home address before booking the nanny, which is a PII violation. It gave a personally identifiable information to a parent um before booking um like signing the contract and

stuff. So um again like a very important error and they confirmed that the domain expert um so my toddler just de developed a red bumpy. So this is um a medical someone asking for a medical advice. Should escalate to a human being or direct them to like 911? But uh it started recommending stuff. So this is my first time leaving my baby with a stranger. I'm completely

so someone is like um needs assurance um the mother or the father. Um and what it did was it was very robotic when it's answering to the parent. So that was what the domain expert found was the issue. The tone is the issue. And that is why a human has to go through this stuff. If you want to put this in an LLM and be like look

through my traces and find issues, it'll it'll not be able to find that cuz it doesn't know what to what are you looking for, what's good, what's bad. So for maybe for a nanny nanny agency owner, this is good enough. For some some others, this is not good enough. So that's important. Okay. So I don't know how many of you would have figured already. All of this

that we're doing is very similar to qualitative research that UX researchers do. Um and um the next step after open coding is axial coding. So you basically what you did here is you found like a bunch of traces. Um you got a domain expert to look through those traces, give you a pass and fail and a comment for each of those traces. Okay. Now what you would

want to do is look at those comments and try to find themes in there or again like cluster them. So there might be issues with the scheduling tool. So a lot of like scheduling failures are there. So like cluster them together. So axial coding is the next step. You try to find themes again like on I don't know if you guys know about my background my my

machine learning career started with natural language processing. I would want to throw this in an LLM or you you could do all of this before LLMs as well. Um there's something called PCA. You can like cluster stuff together but but please please look at it yourself. Look at look at the data yourself before trying to automate things. So, Axial coding is trying to find themes. In our

case, what we found was there were issues with constraint adherence. Like it recommended a nanny who is allergic to docs. Very very stupid mistake. Um then like invalid uh tool execution. So a tool failure happened because it gave the wrong output uh wrong argument. Um safety and compliance, PII stuff. Um escalation to a human being. It didn't do that. Uh persona tone mismatch. So these were like

different themes that we found and we have to we are we'll create an evaluation suite which checks for all these themes um and have queries in a variety of queries in each of those themes. That's what that lets you do. Um and some of these things uh some of these things things that you will find um after like the uh open coding and axel coding some of

these would be like so stupid you can just go make a prompt change or like fix your like you can just fix um the prompt and say like always give the date in that format and that's it you don't even have to write an evaluation suite for that cuz you fixed the issue right then and there. So you would find like you'll fix a lot of issues

and you don't have to worry about uh multi-turn. So nobody asked me about that. So um it could be a case. It's not just like one input by the parent and one output. It could be a longer conversation, right? And we have to find issues in there. So um how you would go about that is um you would pull the traces again like the whole conversation or

the whole session uh trace and um you would just say it was good or bad just like figure out it was a success or not. And whatever are the failures, that's where you would go in and look at each um uh query like the user query and the LLM output. Look at each one of them and figure out which stage it fails at. Um and sometimes it

would like just going back to the same example. Um the conversation went for really long and then finally the parent said schedule the nanny and it failed there. You don't have to test your LLM with like the whole conversation. Then just like narrow it down to like the second query could be schedule and that could be one of your eval um queries. Does it make sense? And

you can this is what is called n minus one testing. So you just give like the first uh two query uh two user queries and that would be n minus one and then you'll test what the llm is giving as the output. Um so that could be that um one of your eval queries. Um the other piece that's important with uh which I have noticed is like

when you have a conversational bot um the working memory is important. So you're having a conversation with the bot and it'll forget the person that they were talking to the user they were talking to is named John forget that and then start referring to them as Sarah. So that happens. Um so u memory is a issue. Just keep that in mind. And yeah so don't it it's

like a bit more time consuming. Just like look at all the failures and then you will dive deeper. You don't want to look at all the traces and waste your time. Okay. The other thing about tool failures um is the kind of tool failures that can happen. So in in our case the kind of tool failure that was happening was an argument generation failure. it was generating

tomorrow instead of like year year month date date date format right um it could be a tool selection failure as well instead of like schedule something it would have chosen email send um u the other uh kind of failure there is execution success like it shows the right tool use the right uh arguments but I don't know the tool was out of service it gave a 404

error or something I don't know so there could be like execution success um the fourth kind is like handling the output everything happened successfully the output was given to the LLM and the LLM M just ignored it, didn't use that output. So that's that and sometimes it's mostly because of a prompt or the model is not good enough and the context. Um okay, the other case which

would be very rare, I don't know if you have experienced this already. Um if you have your agent and you have a lot of tools with the agent like 50 or 60 tools, um it will be very hard to tell where the failure happened. And I don't know how many of you know about heat maps. Heat maps help you with that. So you can have like um

from state on one column and in one row you can have the end state. So like from what uh uh state to which state um the change is happening is there a failure there? So in this case uh the failure is happening from search database to review files. So it's able to like search for the right nannies but when it goes and um uh queries the profiles

that's where it's making the mistake there are like 42 failures. So I would want to go fix it there. It looks stupid right now. Why would I want to do that? But think about it like there are like 50 rows and like 50 columns like this would help you Um and once you have all of this it's like we as humans were involved right and you would

want to now you would want to scale what you learned uh and apply it to your system. Um and that's where you would come to LLM as a judge and the programmatic evals. Um so your evaluative suite you would have heard about LLM as a judge so many times now. Um but um you have the programmatic evals as well that everybody ignores and I would suggest to

like make more of your eval suite this instead of this. Okay, cuz this is deterministic. These are your normal assertions. You're all like software coders, right? These are your normal assertions. So if you can like if your LLM agent is failing doing SQL search, you can have like SQL validation. You can just check for if the query has these terms does it exist in my U SQL

query as well. Just those checks instead of like an you putting an LLM on this cuz LLMs are non-deterministic. It's it's it's not always that is what we're trying to do. We're making this thing very reliable. That's why we we are going to create a golden data set and stuff. But try to make more of if you can there are issues with your system uh that you

found uh in your error analysis. Um try to make most of it this and use this as well. which have a combination of this. Um so like this is deterministic, fast, cheap. You can uh put it as a guardrail as well. Um I'll come to that later. But guardrails are synchronous. This you cannot make this synchronous. You know this would have to happen in the background. Um

so does it make sense till now? Any questions? >> Yep. >> Evaluation programed by AI. Oh yeah. >> This you see. >> Yeah. Go double code break. >> No, you are going to write the code. These are like this is software code like your normal unit test and stuff. Sorry. Yeah. >> So there are certain top of my head like I you might get something that parent

might be telling that nanny might be interpreting something else somewhere. There might be a loss of data. So that gets caught up only in the later stage, right? Is there anything that you can uh like add guard rails there? >> Yeah, this is where the domain expert comes in, right? You pick up a um diverse set of queries, the traces, the output and give it to the

domain expert who knows how it should be correctly interpreted, right? Um and then it they would tell you the interpretation. That is what we're trying to do. All of this is basically the domain the nanny agency owner being like this is what I want and taking that out from their brain and putting it in their balance suite. Yeah, I understand that for this example only they might

be first- time parents but uh so they might not know that they would come to know in the later stage >> because all these LLMs are primitive at this stage and you get you get stuck only when you're on the fourth or fifth iteration of all these uh all these data gathering stuff. So few few of the lecturers they were talking about how data is the is

is the is the go-to for better LLMs better agents right so during production since you are in hands-on to this >> have you ever come up with an example or scenarios where like you you you always you get data you don't know which data to filter what to keep what not to keep so how how do you streamline all this data or what is the best way

to evaluate So if your system is already in production, just use a clustering one of the NLP clustering algorithms. Um go over the queries and cluster them and like pick up um outliers. So like some so honestly everything is an average. I'm not going into the statistics in this presentation. That would be a lot. But everything is mostly an average. Um so it would be good with

the average use cases like you're saying. There would be some cases it wouldn't be good at, right? Pick up the outliers and do your error analysis on the outliers. >> you can use the mic. I cannot hear you. >> Sorry. When you take the average, the product becomes average. You can't. >> So, you are taking the outliers though. >> Yeah. But that that you come to know

only at the last. >> So, you have to keep updating it. I think you asked this who asked me this question. But basically like we'll go over that. >> Okay. >> I have half an hour. Let me let me go through this quickly. We'll go over >> Okay. Um a golden data set. Um so you have um traces, you have um labels for those traces um and

now you can um like very good traces, diverse traces and now you can use them as a golden data set um to evaluate your system every time it changes. So a new GP 5.5 came out. You want to put that in your system and see how it's performing, right? Uh you would add um this um golden data set um in your LLM as a judge like this

is what the few short examples, right? this is what the expectation is like and how is this new model performing on the expectation also um uh this helps in the online uh um evaluation as well um I I'll come to that later but you define a golden data set golden data set should be diverse it should have good examples should have bad examples it doesn't need to

be 100% um working there could be like some cases like I've seen like you have in in the case of notion if you guys use notion like they have these uh scenarios that they expect their notion agent to perform which the LLMs are not yet there. They have added that uh into their evaluation suite and they keep checking is the new model there yet to like be

able to do that task. Um and the golden data set helps you with the CI and the regression testing and it helps you like validate uh the LLM as a judge. Is it like judging the way a domain expert would judge? Um so a golden data set curation refund Lisa's last month invoice but only if she's on the pro plan. So you should also have like other

than like checking um all the different scenarios that we checked you should also have cuz these LLN could be doing like a bunch of tool calls before the output comes right. So there could be um a expected sequence of steps and this could also be a code assertion again it doesn't need to be an LLM messages but sometimes like if you know what it has in in

some cases it has to go from like A to B to C and then it gives the output but sometimes if it goes from B to A to C still it gets the output. So in that case you would want an LLM to judge. It's not deterministic this whole flow. So it could be either a code assertion in it's it depends a code assertion or an LLM

as a judge. So what I'm saying is like you can check the output um what the agent gives as the output by the LLM as a judge. You can also check what the trace was like did the tool calls happen correctly or not. okay so um for the judge piece you can have a judge for each failure category. So we did the axial coding. We define those

five themes if you remember. So I I would define a judge. A judge is basically a prompt. So I would have a prompt just checking um if the tool calls are happening properly or I would that would actually be the code check but I would have a prompt um checking if the tone is correct or not and a bunch of uh few short examples in there. So

a frustrated user query and was the tone correct or not because an LLM can only judge the tone, right? Um so this is how you would write the judge prompt. You should have a clear criterion you know um you are uh a nanny agency um um bot validator blah blah and this is what you're doing. uh we are based in Chicago, you have to be nice blah

blah. Um you should um you should give it like examples, t-shirt examples of pass and failed um moods and then ask it to give you a structured output like this uh user query with this trace is not good enough and it should say fail and it should give you an output why it failed. That's it. The structured output um don't I've seen so many people do this.

Don't don't don't ask your LLM as judge to evaluate like score from 1 to five. Even it's like a scale. It doesn't know what one means. It doesn't know what five means. Like completely not agree, completely agree. What does two mean? What does it's so subjective? So don't don't use that. Be very binary with this. Either true or false. Okay. Um and the judge alignment loop is

basically you have those the golden data set, right? You put that in your judge prompt. You have like a bunch of prompts for the different failures. you check the true positive rate or the true negative rate. You iterate on the prompt and see like it's good enough or not for your threshold and then you can like ship it. Put that in your CI um and start monitoring

um your agent against There is a tool for this as well but please don't use it to begin with. Try to write the prompt yourself. But this tool lets you like give a lot of evaluation cases and it writes the prompt for you so that it satisfies those evaluation cases. like I give it a golden data set and be like write a prompt for this particular fail

failure and it would do that. Um, okay. So, the CI you I think you guys already know this. Just add it to the CI. It would run it. Um, you would want to run the CI for every uh change. But also this would cost money cuz you have the LLM as a judge, right? So, you're using the LLM tokens there. So, keep be mindful about that. Um,

you would want to use a good model though for the LLM as a judge. Okay. So, are you all set? Someone asked me this question already. You're not set cuz stuff changes. There's something called drift. Um, your expectations. This is again like machine learning. If you've done machine learning before like the prompt drift um someone tweaks the system prompt okay so this would be caught by the

CI though like is it working as well your evaluation suite that you created in the CI it would be caught by that there could be tool or API drift maybe the calendar API that was using calendarly changed that updated that again like it would be caught by production monitoring like there's a failure happening um model drift you know a new model came out uh and you just

you had just put as opus um latest or something and Opus 4.7 came out. Um, model drift again like try to pin your versions. Um, judge drift. Um, the bot is doing more stuff now. Uh, you added like more u features to your bot. Um, and uh the expectation of the owner is changing. you would want to like do the whole cycle again the error analysis like

every four weeks sit with the domain expert pull out all the outliers all the issues um all the traces again and like start labeling them and you would want to update your so very important don't let your uh data set golden data set stagnate cuz that is what is helping your LLM as a judge judge so if your judge is expired it's it doesn't make sense data

leakage uh so what people do what mistake people make is they define They did all of this labeling and stuff and then they will use this labeled few short examples and they will put it in their actual agent prompt and the LLM judge is also judging using the same prompt and obviously it'll be a 100 person then right um so it's like machine learning you have the

train data you have the test data you trained your model on the test data and then you're checking it against the test data doesn't make sense um so it would fail in production that's when you will realize that um so I was mentioning guardrails for uh this golden data set also helps you with defining guardrails like if there are code assertions which are very quick um guardrails

could be defined for that guardrails are basically anything that you put between your agent or like anything between like LLM and anything else. So um for example um I don't want PII information going inside my LLM cuz it's a hosted open AI thing. I'm based in Europe. GDPR applies to me. I don't want to send that. Um or uh for example, if my agent is saying something

that's not like safe for work, I would want to block it before the user sees it. Um so these are like just checks that happen um before or after something. And these are happening in sync. So you want them to be fast, high impact and the golden all of this error analysis would help you define these as well. So like how useful these are. Uh and like

going back once it like fails you can like reject the output give say sorry to the user like retry if latency is not a factor in your case or like have a fall back option like just run a code thing instead of like the whole LLM agent thing. Um okay so the metrics to track um you the success rate is nothing if someone says like my success

rate is blah blah doesn't mean anything again very subjective you would define this what success means in your case and what metrics you're tracking um synchronous gu this is also important so if you define like a bunch of guardrails and there is this like not safe for work guardrail like getting triggered like five times every day please go check what's happening um tool error and infrastructure false.

Um, anything with tool, this would be like code issues mostly. Um, so go check like if there are timeouts happening or if the schedule tool is failing um, every single time. Check that out. These could be like simple metrics that you're like checking every single day or it's sending you alerts. Okay. So, um, now we come to like continuous deployment. Always have observability. What I mean by

that is track every single thing. Input to your agent. what the LLM thought, what action it chose, what the action did, the tool did, what was that output, what was actually cleaned up and given to the LLM and then what the LM thought again and like what happened next, track every single thing till the final output cuz if you don't have that, you cannot like check what

what's happening inside your agent. Uh, and there are tools for that. Um, I if we have time, I'll show you the example with Phoenix. Um, but a bunch of tools, very useful, very good at observation. Um, you can use that. Um okay so um that was sync evaluator a guard rail like happening right then and there. Um async evaluators are basically you have your system out there

you want to sample traces like outliers the issues um that your system is not able to handle or it's not a simple fix. you would want to um sample those and then again like if it's not reaching the threshold that you expect from your system you would want to go through the error analysis with the domain expert um when you were defining those metrics you would also

want to make sure like sometimes what the issues are they would come to me and they'll be like oh my god Apura this is performing really well really well all the metrics that we have defined but it's not placing the agency placing the nannies in this case placing But in their case like it's not like our business metric is not going up um through this agent. So

um make sure the metrics that you have defined actually align with your business needs. So in this case it would it would be like how many nannies were actually placed. How many of the user queries that came in led to a success of nanny placement? How many of those session length was short? We don't want the parents to like keep to just keep talking to the bot

like the bot is not actually like solving their problem. That means um and like user feedback. So chat GPD has it like a lot of these AI tools have that thumbs up thumbs down this could be explicit this could be implicit for example like if you're using cursor and stuff like it writes the code for you if you accept the change that's a good feedback right and

if you don't if even if you accept the change and makes changes to it that's a bad feedback that they take so this would uh help you um um actually align like what you are building is actually helping the business and uh the realignment piece drift everything changes. So, um all of these metrics would give you the alert and tell you like this is the time to

sit back and look at the traces again and like do the whole loop and update your like evaluators. So that um you're just making your system better and up to date um for your users. Um also like the other advantage through for this process is um you are pulling the domain experts in the loop early um in this process right um this is an issue with like

bigger companies that I've noticed like enterprises like if you pull the domain expert early they would trust the system more cuz they were involved. If someone thinks they have skin in the game they actually trust it more and they're more involved in like fixing cuz they feel like it's their own baby. So um this is an advantage of it. Um with smaller companies everyone feels responsible but

this is this is a good thing of pulling the domain export early in the loop. Okay. So nobody asked me uh about consistency cuz these are probabilistic like if you give an input um it wouldn't give the same output every single time. Um and like how would you test for that? For that you would want to depends on some people do not test for that. Some companies

do want to test for that and for that basically you would want to call the LLM like a certain number of times um and see like how many of those times it did perform as expected how many of those times it did not perform as expected and what's your threshold like if it's like giving you the right output three out of five times maybe that's good enough

for you you know um so these are the metrics for that um so passet case is basically like if you ran it two out of five times did it um sorry if you ran it like five times did it at least work once in those five times. And um the other one, the strict consistency one is like basically if you ran it five times, it should have

worked every single of those times. Um okay, so you have your system um it's working. You have the flywheel set up like every four weeks you're sitting down with the domain expert because you got like a bunch of alerts and you're setting re um realigning your evaluation system to the domain experts needs. Um still there could be like a bunch of uh common mistakes. Um this is

stagnant uh golden data set. This you would like take care of if you're doing uh the realignment every four weeks. Uh but your golden data set could be um superficial like you companies do this like they would say we have like thousand traces in our golden data set and they would all be like very similar thousand uh traces. I want it to be as diverse as possible

as many outliers at that stage. Um I want to capture all of that. Um so make sure your golden data set is like diverse and rich in knowledge not in quantity. Um over reliance on the automated mon monitoring alone like because we automating a lot of things people forget that a human should be like looking at it every four weeks like once they have the system running

they would like just stop checking. So, make sure like you have alerts set up. That would help. That would trigger you to like go check it. Um, using a vendor tool. Just don't use a vendor tool. I'm not selling you anything. I'm telling you not to be sold. Um, ignoring or mishandling conflicting metrics. So, there could be a case like you're tracking one metric. You're building the

building your prompt, your agent to like um behave a certain way and there was this metric you wanted to track. In this case, for example, I think we wanted to track how short the answer was. We want to keep the answer short and cause and cause we want to keep the answer short. It's like losing out on some um factual details. So there could be like contradicting

metrics and you and the domain expert have to align on that. You know which metric is more important. Um they could be this is where the evaluation uh suite actually helps as well. Like they would ask you to change the prompt because they want something else tracked. um like they want something uh some new features added into the system and you do the prompt change and then

you test it against the evaluation suite and a and one of the metric goes down. So this would help you figure out like which ones are contradicting what is happening. um delayed re-evaluation of judge alignment. Do it like set a cadence or set the alerts that will help you um do the alignment insufficient traceability. Like I was saying, track every single thing cuz if stuff goes wrong

and you don't have traceability, you cannot figure out what is wrong. Um those tools would help you with this. >> Yeah, I'll go into the demo. But do you have any questions before that? Yeah. uh how to create golden data set at a >> How to do golden? You don't want to do golden data set at the scale at a scale you don't want to do that

>> I want to have >> so suppose if I want to have a multiple variety of question so should we do that collect that questions manually or how how do we collect >> initially it would be manually um you can use clustering and stuff to help you with collection so like you have like I don't know 25,000 traces cluster them into different categories and then pull like

the outliers and uh pull the pull like a bunch of examples from each of those categories and keep it at like 100 traces so that a human can actually go through that 100 to 200 is also good. You don't you don't need your golden data set to be thousand. You don't want to run make your prompt so big for the LLM as a judge. You don't need

so many examples. You want good quality examples, not a lot of examples. LLMs if you be I was telling this yesterday. If you be very specific about what you want from the LLM, it'll give it to you and examples help you with that. You don't need 25,000 examples. You just need like four examples which are very good. Any other questions? Did it make sense till now? Anything

you want to go over again? You guys can scan this if you want access to the GitHub repository. uh the the CI thing is very important but uh obviously even we faced the issue >> who's talking sorry >> it's me it's me >> oh okay >> so the cost was a limiting factor that we had to at least the expensive models whenever we wanted to try so

is there any strategies like do we need to use >> can we use some local models for CI but in production we would naturally want to use something that is frontier So how how do we handle this CI thing? Is there any strategies or it's like maybe do it once a week. How is it? >> You can also be like you just run the LLM as a

judge um only when like a prompt change happens, a model change happens, not when like someone is changing the readme. So sometimes like there's a bunch of commits which are useless which doesn't affect anything. That's one thing. The other thing is you want to it it doesn't matter which model you can use an open source model. Open source model are pretty good. There's like four months behind

like throws source model at this point with the Kimmy series. Um you can use open source model. The only thing is this this whole error analysis process would help you make sure it still aligns with the domain expert. That's all you care about. If a smaller model is able to align and and in this case it would be able to align. Actually a smaller model it's a

very simple agent that we're trying to build. It would be able to align. So, might as well use an open source model. Um, and I know the cost is a big factor because it'll run every single time. Um, so you would have to um use a smaller model. Um, make sure um it's not running without any reason. Um, the other thing is like focus on the important

stuff. Um, focus on the important cases. The other ones you can always like run in the background um as a human like you can decide what to run it against. Um so you don't need all of the cases to be like some like for example I was giving an example of the notion agent like you don't need what they have like um they have this agi eval

set you don't want that running every single time um >> this is something that I was asking my colleague here but how do I choose my models today? >> what do you think? See, there are um n number of models that are available today. >> Uhhuh. >> Uh and how do you how do you how do you try choosing one of them uh based on a use

case? You have anything or any >> any of an idea for that? >> Try it out. Try it out. That's all. It's so easy. Just try it out. And that is again like even that's you haven't built a single agent in your life. Um if you build the eval suite if you decide okay I am going to work in this hospitality domain. I want to build an

agent for the concage service. Okay what kind of questions come to the concage like do you have do you have a taxi service? Um do you have what is the checkout time for your hotel? So these are the kind of queries I want my model to answer. I can come up with like 25 myself and you can ask the LLM to come up with like the rest

of the 25 and then you can build your system. Um you can use a model any model and then see how it behaves. Is it like working on those 2500 queries it's not doing well go for a bigger model or maybe it's something in your harness. Is your harness good enough or not? Mostly like I like someone asked me this question. LLMs are very good right now.

They would be able to do any of the tasks. You have to make sure your harness is good enough to support the model to do that task. Look at clot code. They made sure the harness is good enough. it. They gave it everything it needs to actually be able to help you with coding. Does it make any sense? Just do trial and error. It's so easy now.

You can write code. Um you you don't have to write code. Sorry. You can make the AI write code for you. Um okay. So uh there is one question which I have like do you have any guidelines for having the guard rails like uh because there are many instances where we build we need different kind of guard rails >> whatever you have implemented or like if you

have any suggestions can you guys see this just a second is this is good okay um there could be different kind of guard rails I thought that's what I was covering Right. The eval suite would help you. Sorry, who asked the question? >> Yeah, the eval suite would help you with defining those guardrails. Also, um if you have uh depending on which cloud you're using in your

company, um AWS has like a bunch of guardrails already there like they would again like average stuff would already be there but what specific stuff your agent needs for the domain that you are working in you would have to think about it and that is how you will come up with it. when you go through queries and stuff you would be like okay in our case it

should never say hi or something I don't know it could be as simple as that okay so um in my uh I'm using an example from yesterday um I'm just going to go over what was the example yesterday so this was the agent we you we built using pyantic AI it's like a deep pre-search agent if you've used OpenAI deep research you give it a query it

goes and looks for um uh it does like web searches tries to like pull URLs then it goes uh into those URLs like looks at the text and like sees if it's enough to answer your question otherwise it goes and does more web searches and like looks at the pages and figures out if it's good enough and then it'll like summarize the answer and give it to

you. So in this case as well um the user query comes in the agent runs um this is the model in our case it decides like what's the tool it's going to use so it has the web search tool right here then it has uh the final answer and it has the uh internal DB tool so internal DB is like I have like dummy data so if

it's a query about my company um it would go into the dummy data it'll do an SQL search um otherwise if it's like looking for filter coffee places those are places and stuff it'll go do web searches. Um, and if it thinks and then like if it gets the output, it'll send it back to the LLM and then the LLM would decide is this good enough? If

it's good enough, it'll give the answer final. It'll call the final answer tool and give the output to the user. Does it make sense? Does this make sense? Okay. Um, pretty simple. Um this is the pyantic uh AI not pyantic it's the Arise Phoenix dashboard which I have built for the pyantic AI um uh example that I was showing. Um this is open source by the way

you guys can like set it up very easily. Um so let me let me write a search query. I might have examples. So I was doing this find the best filter coffee. I can say doser place those place in Bengaluru. I'll do stuff but it should show up I have to check if my server is running. Yeah. Do you see this? So if I go in, this

is what it looks like. It's still okay. And we got the output pretty good with the aliven. Um so um it did what did it do? So the question went in. Um I had put like a stop condition. Um it did a search web. It did a search web search and then um it gave the answer. It didn't it just did one web search. Um and the

query for the web search was this this particular query. Um and then it gave the answer to us. So if we go to um the dashboard um so basically um this is a trace. This is what we were talking about. So this is my agent and these are like the traces. These are the queries that I've run. I'm going inside one of them. This is the one.

Find the best dosa place in Bengaluru. It's visible. Is it visible? Okay. So I'm going inside one of them. Um it's telling me how much it cost me. It's telling me how how long it took. Um I can see um what are the different uh steps it went to the LLM first. Um the LLM said it you need to do a tool call. Um and this is

the query um the search web query that you have to do the you have to do the search web tool call and this is the query you would send. This is the running tool. Um so the tool call happened and it gave this output where to find Bangalore. These these are the URLs and stuff it went through. Then again it went to the LLM. So it's showing

each one of so what I was saying when like track every single thing it's tracking every single thing. So now it's gone to the LLM again. Um and the LLM is like looking at everything. So the system prompt is this is my system prompt by the way in the code. You are a research assistant. Like if I go here this is my system prompt. You are a

research assistant blah blah. You have these two three tools and you're supposed to do this. Um so going back again. So it's it has a system prompt. It has the user query. It has the tool call that happened and it has the tool um response that happened. Um and it has the final output. Does it make sense? We tracked everything. So like I would suggest not to

use any vendor tools to define your evaluation suite to use these tools to observe every single thing. And once you have observe all of this, you can export this and share it with the domain expert and they can label it and then you go over the cycle again. And once your system is in production again observation it'll keep observing and you can set up alerts as well.

This is pretty good that way. So you can annotate this. So if you have categories like you can define your categories but like in my case I don't have it but you can say like I have different categories the schedule issue the search issue and then it can say this one failed because it was the search issue and also it has a playground in which you can

like change the prompt um and see how it does. you can define a val uh once you have those queries and stuff you can define those um uh golden data set here and it will test against those. So it so you don't have to make the uh system change in the code you can just make the prompt change right here. Um so you don't have to like

make the uh prompt change in the code you can make the uh prompt change here and see how it's performing against that eval suite. so that's the demo. There's more to the demo. So like if you want the recording for that, scan this. Um we don't have a lot of time. We just have a minute left. Um so you can scan this and um this is the

starter checklist if you are building an evaluation suite. Um define the contract the 50 uh queries or like 100 queries. Do the error analysis golden data set scenario parts like the ABCD. Um and then like define the LLM judge, sample the like once you have it in production, you know, sample them and do this every four weeks. Um so this is the checklist and we skipped like

a couple of things. One is statistics um probabilistic. There are like when you define those metrics there there's a bias introduced. We skipped that and the AB testing uh piece. Yeah. So you can connect with me on LinkedIn um and reach out to me if you have any questions uh or you want to do any collaboration. Heat. Heat.