Architecting Microservices for Agentic AI Integration - Rohit Bhardwaj

This talk explores the modern approach to architecting microservices, emphasizing the shift from traditional methods to more agent-centric designs. The speaker discusses the concept of agents communicating with each other and how orchestration is vital in managing complex interactions among microservices. Key topics include the importance of control loops, addressing latency issues known as tail latency, and implementing brakes in the system to avoid overloads, such as retry storms. The session also delves into the necessity of guardrails, policy gates, and ways to handle risk management effectively within microservice architectures. Additionally, the speaker touches on the integration of AI and the significance of ensuring deterministic responses while managing various failure modes and optimizing performance.

Architecting the microservices. When I say microservice, people think that you know, it's the old way of doing microservices. It's really not. And I'm going to just debunk few things for you because it should look like you know, hey, something something different way of thinking. It's kind of you're building a building an instruction for a child. Okay, you're not building it for a human. It's an agent talking

to another agent. Okay, it's not talking to a human. How would you build that? That's a real challenge which I will be discussing. And then the requests are being sent and how you do the bounded orchestration. Now, if you think about it, when you drive a car, traditionally how you used to drive a car? Like now also we are traditional, yeah. Most of the time, but this

car used to be, you know, having certain certain things, you know, there. Now, if you single request and response you are done. Now, here you have little bit more information is there. Here you're going plan tool is called observing it, deciding and repeating it. It's more work is done in this one. Does it make sense? So, you're doing lot of work to kind of make this thing

happen. So, this is the agentic AI which you're trying to build. The essential brakes, these are the brakes you need to have in this car. One is the retry and budget limits. How much retry you should have? Budget you should have, that's number one limit. Number two is policy gates. You should have a policy. Somebody wants to refund. What's your policy for it? What's the policy gate

for it? Makes sense? And then the number three is guardrails. Always have guardrails to support it. And while you're building this, you have need you you are actually managing what risk? In these microservices, you're managing what? Risk when you're working on a solution. So, what all that really means? The agents are control loops. When I say control loop, see take a look at here. When a control

loop is there, you're reserved the memory. Now, agent is running, it's observing, deciding what to do. Okay, it's trying to decide what to do. And then it's planning, if there is a plan phase is there, which is actually acting out acting out through some applica- amplification risks. So, what do you mean by amplification risks? There are risks in agent. You know, when one API is making call

to another API, you may have duplication. You may have charges and retry storms might be happening. So, you need to apply these brakes, you know. You know, in car we have servicing, we do servicing of the car, same thing you need to do it here. That's the important part. If you hit the limit, what do you do? What's your fallback plan if you hit the limit? So,

we'll talk about certain ways to do that. You one is a rogue customer. How to fix that customer? We'll discuss it today. Okay? This is a important part of the puzzle we're going to look at together. All right. So, that's said, like traditional, I think we already discussed this. So, agents need explicit limits to prevent any of the loops which are coming in. Now, this is night

before everything worked and still breaks. Now, everything is working, okay? This situation I'm talking about everything is working. Your your code worked. So, earlier our code worked, we what we used to do? We used to celebrate, yes? Not anymore. The problem is that AI summary feature is there. Now, you're trying to come in monitoring the user is calling, "Hey, I want the AI summary coming in." Now,

what's happening is that you're calling vector database, somebody said knowledge retrieval. You're trying to do knowledge retrieval, and then you're trying to get the data, and then there is something called P99 latency. Okay, let me ask a question to you. What is P99 means? Anyone? >> 99% of the latency. >> Yeah, and you know, 4 years ago, 3 4 years ago, I used to say if you

do P99, 99% of what? 99% of the time this is the latency. Yes? And 1% we used to say it's okay. Is that true? Yeah, that's the that's the old way of doing it. Not anymore. Why I will talk about it in in a second, but but that's the real problem. This is This is not by me, but really speaking that's what we are trying to do.

There's something called You can You can search for this. Jeff Dean and Luis. They came with the tail the tail at scale. The tail at scale. Google it. You know, you'll you'll get that. There's a there's a big document. I won't have time to go through it. But this tail at scale means what this is saying is that launch day, the AI summary feature is being added.

Hours later there is a prompt downstream traffic latency spikes are happening and bills are exploding like anything. So, request per second. How many here know how do you calculate the traffic? Huh? What did you say? TPS? Yeah? Transaction per second, request per second, yeah? That's what That's what we're doing. Not anymore. What, Rohit, you're debunking? So many things for Yeah, write it down. This is very important

I'm discussing. hops How many times you're hopping? You know, hop like kids do hop. That's exactly what's happening here. How many hops you're making that's actually changing the behavior. So now here um this is called a tail latency. So human traffic versus the other traffic. So what's really happening is human traffic means human is making a call. You forward the request and you resolve the back. Now,

AI is making a call. It's looking at the read history. Check for fraud. Update the customer record. Send notification log. Now, this is done through AI. So, AI is not one call. Make sense? AI is n number of calls you're making and these are all the hops which is happening behind the scene. HPS hops per second is the way to really determine what's going on here. So,

we need to really work on this amplified load and how do we protect ourselves from that? So, let's talk about a role. So, that means there is a role. There is an a new employee coming in. So, when I talk about this, you put as a role. That means you have an agent. And this as a role, you're trying to work on it. And when you're working

on this agent, it is working with a trusted data. What data this agent has access to? What knowledge this agent has access to? Based on the knowledge and data it has access to, what actions this data can this agent can perform? So, what does that means agent can perform an action? Humans can perform actions. Based like just like humans, agent can also perform actions. For each of

those actions, what are the guardrails to protect to make it more deterministic? Make sense? So, that's another thing which we need to do. And then and then at the end, we need to support all the channels include including voice. Yeah? Now, once we do that, we have an orchestration agent is there which is trying to work on this and it's trying to collaborate across the across the

service rep which is there which is doing it. And I'll I'll talk about this in a second. What does it really means behind the scene? Yeah? So, let's talk about this. Hopefully I have this. So, this is the agent I'm trying to create right now. So, I said that, "Okay, I want to create a service agent." Because I picked the service agent, these are all the microservices

for this service agent. Already there. So, I can say case management. What a service agent will do? Case management, So, within the case management what actions can we can be performed? Create a case. Get information on the case. Answer questions. Yes? So, this microservice is not just one microservice which you're trying to do, but you're trying to create subagents. Everybody with me? What am I trying to

do here? I'm creating subagents. I'm saying that case management, account management, all these agents I'm creating. And when I create these agents with actions which are coming in now I'm doing the grounding of that agent with description, what's the role looks like, and then company. And what is the who is the user who's performing the operation? Now, this is the user who has access to this records

only, not more than that. You see that what I just did here? That means an agent, just like a user is logging in, an agent is logging in and performing an operation as an agent for the for us. And based on this, what is the data they have access Makes sense? So, that is what we are gaining from this. Once we do that, because we have created

this particular agent, now what would happen is that now I have got all these subagents created. I can go to a particular subagent and look at look at the instructions which I need to follow this through. Now, I'm going slow on this one because I'm going to go fast later on, but you got the idea. We have an agent with certain instructions. So, what's the orchestrator is

doing? Orchestrator is going to find that which subagent I need to pick first agent to agent. Yes? Agent to agent, so you're going to have an orchestrator which is calling from one agent to another agent. Now, what I can do is I can actually go in and I'm going to just go in here and I'm going to say that, "Hey, for this particular user, I hope you

can see this. I'm an assistant. Can you provide me the full moon experience? Can you provide the email address? Yes? So, what really happening is behind the scene what's happening is that we have got something here session started and based on the session started we created something called as a plan. Everybody with me? So, what is this called as a plan which I'm using this prompt to

find out what sub-agent to pick. Okay? Now, I picked out of these sub-agents, I picked a sub-agent which is currently we are picking and it's asking me for the username. Now, I can provide here the username for and I'll stop showing you this in a But, this is important to understand. When I did that, now I'm able to multi-factor authenticate that user. Once I did the multi-factor

authentication, now it's going to give me the full moon beach party I want to book. But, you get the idea behind the scene Behind the scene, we are trying to collaborate through different agents to solve this problem. Everybody with me? Say say I. I. Okay, good. That's a good thing. Now, what's really happening is behind the scene when I say that prompt template is coming in. So,

whenever I'm asking for something, it's going to go through the prompt template, it's going to ground the data from vector search, and then it's going to go through the dynamic grounding and bring in the data masking the information. So, you get something like this in the prompt template, okay? Now, this means it's like an email address, yeah? And then you have this variable, and this is the

conversation which is coming in. We need to send this conversation and find out the knowledge from our retriever first. From the chunks we get this knowledge. After I get this knowledge from the chunk, I got knowledge from the vector search. We'll talk about that later on, but then we did the data masking. Before you send it to LLM, what do we what do we need to do

if it's a PII data? Social Security number and other things. You should you have your credit card number and go send it to ChatGPT? You can. It will say thank you. But not a good idea, yes? So, what's happening is your credit card is there, you want to not send it across, that's number one. So, what is this? We are doing guardrails right now, yes? And once

we did that, then I did a prompt defense, you must not address anything else. And that is where the guardrails are coming in. You can add other guardrails also, you know, along with this. And then I'm sending this data to some of the LLM like Gemini or ChatGPT I'm sending it out to. And then zero copy retention, I got the bad data from here. And then I

got the data back from here, and then the first thing I'm doing is I retrieve the name and other information, and then I do toxicity detection. So, basically input to LLM, output from LLM. What are we doing here? Input and output validation. Roy, we used to do that before. Yeah, it's the same thing. Who needs to code for this? We, Yes? So, that's called as jobs security.

Yes? So, pat on yourself. No, our jobs are not going nowhere because this is agent is going to hallucinate exactly here. So, where you're not writing code now, but you're making sure you add the toxicity detection to detect if there's a profane language coming out from there or not. Once you do that, there was a question asked on the feedback. Now, you need to get the feedback

from the user saying that was it useful or not useful? Thumbs up, thumbs down. Or if it was inappropriate, you need to find that information. This is called as control loop. Now, agent maybe not give you the answer first time a good answer. Second time it will give us the better answer. It's like, you know, training on its own Once you do that, now you need to

do audit trail. So, be able to see toxicity scores if somebody saying bad words or not. And then based on this, now I'm able to come in and have a human in the for anything which the AI cannot solve for. do not say that my agent will automatically solve for everything in the world. It will solve for world hunger? No, it's not going to happen. But maybe

we can create an agent for world happiness, you know, that could be one thing. What world hunger like, you know, you're going to solve for this agent? No, if you have a Somebody's asking for a refund. Let me ask very practical, yeah? Somebody's asking for a refund. How much money should you put in in the refund? That's the question we are Yes? That means if the refund

is less than $200, $500, approve it automatically. Otherwise, you have any human in the loop to solve the puzzle for us. And that's once we once we do that, now we are ready to start from the failure modes. Failure modes means what's happening is that we are now trying to look at these are different failure modes. So, I directly looking into failure. Remember like you guys I

asked your challenges and I'm telling you what the challenges are. This is important. Note these challenges down. Number one challenge we already discussed is dependency fails. You know, retry dependency fails. You're dependent upon another retry and gateway retries are happening. Retry storm is happening. Retry storm is a big problem. Number two is cache misses. Cache misses means that cache eviction is happening. Now, why the cache eviction

will be there? What do you mean by cache eviction, Rohit? What do you guys think cache eviction means? That's huh? They that's all that thing. Ejected from memory. Whose memory? Which whose cache? Cache That's the point is. Agent context cache. So, agent should have its own cache. Why? Huh? Greater than somebody. Greater than somebody asked the same question, you will be able to get the answer back

to them without going through all the way through and then bring it back. Yes? Somebody's asking what's the score cricket score? We should be able to get that score back. Yeah, so that's the reason we want to make sure cache eviction This is where another problem comes. This is where the database fetching is happening at the same time. Now, another problem is database saturation, you know? And

then that's where like, you know, many requests are coming to the database. And when many requests comes to the database, that's where the P99 latency really comes in play. So, what's really happening is that P99 latency means when you do the retry three times, four times, the 1% will magnify. 1% will become 1,000% right away. It's like It's like a really huge problem which is going on.

Another problem is the API drift. How many here love to write API with old endpoints and new endpoints? If you change the endpoint, what do you do? Used to do? No, no, call call the other person. Hey, I changed something. Yes? Isn't it? Yes? We do that now also. Now, you team, I changed this. How do you do the No, you need to talk an agent need

to talk to another agent. Hey, I changed this, you know? But how you do that? That's a challenge. I'm discussing challenge with you. Everybody agree? Yeah? Once you solve these four challenges in the AI workload, you're golden. You're going to say that, "Boss, I did great job." You know, this is what you're here for. Once you know this, once you have built this thing out, now costs

is increasing, not just QPS. Query request per second. Now, what's happening is whenever any request is coming in, multi-tenant user retry is going on. So, these are all the hops which are there. The other visible is the cloud bill which is coming out of it. So, that's the real problem. I think we already discussed this. Now, what's happening is that loosely typed JSON object is there, which

is bad for us in the REST API. REST is good, but this is the problem in the REST, like, you know, these are the problems. Hidden side effects can be there in the API calls. You know, polymorphic results can be there. Data structures are, you know, hard to pass what's going on. Lack of clear API versioning mechanism. That's API problem. Currently, we all face this. Yeah? But

agent face this more, amplify this more. So, what we need to do is, as part of our work, API for human versus machine, we need to look at not human. API for human is easy. Best practice, follow the best practice document. In the AI era, now, if you don't do that, there's a model confusion going on. There is a polymorphic breakage is going on. And then the

error starts to happen silently. Nobody's telling you error is When the error happen, you will not be able to get the answer for it. So, you need deterministic response. Okay, why we are here? We need the same shape every time. Same shape every time means JSON structure should be the same every time. Very important to understand. So, idempotency is one way to do that. Like your idempotency

means if I if I say uh you know, buy the order. Yes? Buy the order is not there in the other system, but in this middle system, which is your MCP server, it should return back what? Idempotency. >> Yeah, that's right. >> Key. Idempotency key. It's not there in the It's not the database key. It's idempotency key. That means that if the same request we have to

retry, it is not putting into the database. Okay, so that's one thing which is very important to do. Uh building this relationship. Now, that said, now unstructured errors are there. HTTP errors. You can get HTTP error. Now, standard format HTTP errors can be there. So, what we need to do is there is something called as RFC 9457. Would you remember that? Probably not. Yes. So, I know

that. So, because I don't remember either. But really speaking, what is RFC 9457 is machine readable JSON format for reporting errors in HTTP APIs. Make Machine readable. It's not a human readable. Oh, this is the error code coming. No, no, no. Machine means AI should be able to read. There's an error message and it should be able to act on that error message. So, Google this, you

know, you'll be able to find out the RFC 9457. What 9457 does is invalid address is there. Error handling is done in this one. You know, populate field types and other information is there and other and and then retry again you do the retry and then solve the puzzle for it. That means that means if the address is invalid, you're not even going to the database or

to the server side to validate it. How many here would love to write zero code? Say, you know, raise your hands, yeah? You want to write zero code and do this, Yeah, it's possible through this RFC. You know, this is machine machine readable RFCs there. Once you do that, now microservice is easy. All the validations, all the email responses, like all the regular expression for email, all

those are being captured here. Now, anti anti pattern for the break agents is there. Now, get Okay, this is a bad thing to do. Okay, anybody has done this? I'm raising my hand. Yes, I did this before. So, now get method, anybody here like to write get method and within the get method you're trying to change the state in between. You do that? Not a good idea.

Not with agents, like you know, you can maybe able to do it before, but it's not a good idea. Hidden side effect of the retry, you know, the what would happen if you retry the same thing? That's another problem. Misleading error handling, HTTP HTTP 200 is being error returned HTTP 200. What does that mean? From agent, it's a fault, you know, you can't handle this. Unexpected response

shapes can be there, you know, hidden flags, you know, those things we want to avoid. All right? If you do this, you are good, your microservices awesome. Now, ideally you should have done this 5 years ago. But nobody was twisting our hand, yes? Now, agents are saying, "Dude, if you don't give me the right answer, I don't know how to solve this." That's the real problem in

this one. So, that means, "Hey, you want to do this free of cost without doing it anything?" Yeah, you can do that. So, This open API specification is there, 3.1. Okay? Open API 3.1 specification is If you have the open API 3.1 specification, you will be able to now be able to do all this machine-readable code can be added based on this. Okay? Now, you may ask

a question, "Rohit, is there any way which I can do a plugin?" Yeah, there is a plugin there. Swagger editor plugin is there. If you apply this plugin, you know, if you if you apply this plugin, you will be able to that, you know, "Hey, is you are your is your API protected or not? Is your API is there any SQL injection attack going on on this

on this?" So, you can create the path and you can evaluate and check at the end if if if this regular pattern regular expression pattern is applied or not. So, that is another awesome thing. So, so that is one thing I suggest. If you just do this thing and few thing which we discussed so far, doing really good for your project. Okay? Just by a small thing

like this. Okay? So, you can get open API 3.1 3.0 is also there. You can do path variables. Everything you can now protect without writing a single line of code. So, this is awesome. This is like very simple thing you need to do. Now, this is the reference architecture for an agent. This is a reference architecture for an agent. Let me talk about this for So, what

does this means? It means that Let's take a look at this one. So, what we have got is Whenever any request is coming, we are sending it from the supervisor, I'm sending it to the tool gateway, which is actually maintaining any any versioning, routing, and other information here. Based on this routing you need to route it to orders, inventory, and other agents which we just created. Remember

I showed you topics? And those topics were sub agents, yes? And I'm able to route to those topics. And once I'm routing to those topics, and I'm using event bus to really send the data out from our side. So, that's the way we kind of protecting ourselves and creating the puzzle for us. And that's where the cost cost also you can maintain. And observability can also be

added to make sure that that how the agent is performing at the end. So, determinism is very important. You should be able to determine what's going Okay? Just by doing this exercise, you can determine what's going on. Do the APIs guarantee stable pagination and sorting? If you cannot do that, then it's a If you don't do it, like you know, it will cause a problem for you.

For example, agent starts the query, find out the available slots available Based on these slots, it's able to find out, okay, 9:00 slot is available. Now it rechecks the availability. When it rechecks the availability, 9:00 slot is gone. And that is one problem which we face if you if you don't guarantee the same result is coming back. So, remember I talked about determinism is very important. Okay?

So, that's that's the reason you know, this is this is this is important to understand. Now, this is another thing. This is called as CRUD sprawl. How many here do CRUD sprawl? You know what CRUD CRUD sprawl mean? Anybody create an agent like this? Tell I'll just go to the right side and I'll talk about it. Agent is calling orders. Agent is calling inventory. Agent is calling

the payments. And returning it. And then you may get an ambiguous error from the payment and you don't know what's going on and you have to deal with the hold and three different systems are there. How many here do it this way? Typically we do it this way. Like I would say in the 70% of the time. Yeah? Do it this way, yes? Is So, what is

the better way than this? What do you think? It's not running parallel. Have one person responsible Okay, just like teacher, you know, class monitor is there. Hey, make sure kids are quiet, Teacher starts saying, "Okay, you be quiet. You be quiet. You be quiet." And if I'm out it kids are going to quarrel, you know, with each other and not be able to solve the puzzle. So,

the easiest solution is this which I'm going to discuss with you. It's a very simple solution but it's very powerful, okay? What you need What you need to do Let's just take a look at this one. Improved capabilities Agent is calling order. You delegate This is what I need to do. Now, this order agent is calling Order agent is calling payment and result is being sent back

to the this agent when it's working on it. It's a very simple switch. But if you don't do it this way, the agent is hallucinating and how many here Somebody here was saying You were saying that agent is hallucinating. That's the reason your agent is hallucinating. What do you need to do? Simple fix. Have an orchestrator microservice which is going to orchestrate across the board Right? Wow,

Rohit. This is lot of good stuff you're telling. Yes, that's true. Now, separating agent and human traffic. Now, it's a different It's not that human traffic, yeah? It's a It's a traffic which is there which is there observability is becomes easier. That means, do you have Are you using the same API when it's coming in from API gateway? How many here have API gateway? Everybody got here.

I hope so, yes, by now. So, now if you have an API gateway there, do you have Do you have an agent as a separate separate policy for it? No. I think 90% of people they don't do that. Just by doing this, I'm telling you this is very important which I'm discussing here. API gateway, you know bulkhead? How many here know about AWS? How many Anybody here

from AWS? I hope not. Okay, yeah. Okay, good. So, in AWS, you know you have seen that like you know there was a one bug was there and whole east east zone was down. Yeah? So, the problem was that the blast radius was not controlled. Yes, because there was a bug. >> DNS bug DNS bug and DNS domain name service and route 53 I'm talking about is

best known for shuffle sharding. But it could not do it. That's the problem and that's that's what it is known for, yes? So, sorry I'm discussing it but it's a good case study to know. If you have this policy of bulkhead policies there, traffic is isolated through microservices and through events. Now, if one of the microservice is not doing good, what are you going to Easy. Contain

the blast radius. That means don't send the traffic through that that contain blast radius means that it is not affecting other users in the system. That user is only one which is affected for that. Just by doing this exercise, you are able to control the agent which is going in and the human which is going in as a tenant ID as separate way of solving it. Makes

sense? Simple issue, simple solution for this. Right? Now, when we did that, now I create a isolated group with observability coming in for me and we are able to work on this Two lanes, agent bulkhead policy. Wow. Another one, API life cycle for the Life cycle for the agent is something like identify the client. Now, what we used to do was we need someone to do version

routing. Version routing means first party app, agentic client, another clients are there. Based on the client which is app is detected, now you need to have a version routing agent there which is going to route to the current version or the previous version based on the user who's using the system. How much How much time we spend on this one? Content header, you know, content header change

and you it takes a long time to do that. Now, if you have this version heading going on, now contract validation is done. If the contract drift, now, this is an agent. By the way, all these agent These are all agent by the way I'm discussing. What are these agents? Do somebody ask the question here? These are called ambient agent. Write it down. Ambient agent, okay? I'm

creating an ambient agent for this. Drift detection is would Okay. I'm asking I'm saying that when would be the drift taking place? I'm not saying if is there would there be any drift drift there. Yeah, there will be drift. if you say there's no drift in the API that way, the product manager stopped giving you new requirements. It's never going to happen. So, you need to prepare

for it. You need to have an agent which can do two things, drift, contract drift, and deprecation handling. If you do these two things with sunset date attached to it. And if you have a sunset date attached to it, now you know that you are able to handle this request well. Yeah? So, that is really good to do. Once you do that, you are able to able

to route the traffic without any Now, when we need to do it, there's something called agent checkout graph. Uh we'll talk more on the graph later on, but really speaking, planner is executing it, and what it's doing is it's reading in parallel. So, what it is doing is Let me Let me run this through just to get an understanding on this. So, what's happening is the planner

is and it is initiating instantiating right away a parallel inventory check. After that is done, I'm doing a availability decision is there. Hey, is this queue is available in this location? This is Think about amazon.com, you know? They're trying to do that. Now, based on this result, now I'm trying to find out the reserve that inventory. Should I reserve this inventory for this budget? Based on this,

I need to do fraud check. And if I find this is a fraud, then I need to unreserve it. You know, that's another thing we need which I need to do here. Then I need to do authentication decision to handle any declines which are happening in the in the approval. If I need to do an approval pre-approval process for that. Now, I create an Not before that.

If I create an order before that, now that mean that all these things has to be done separately. And where should I ship from? Ship code one and carrier carrier A, carrier B. Who is the right shipment for me? An order is confirmed at the end, you know? That's what's happening in this case. This is called as agent checkout graph. If somebody want to do a checkout,

Mode, that's the way to do this. Now, when we are doing this, order is received, price check needs to be done, inventory reservation needs to be done, and the payment and the fraud check, all these things needs to be orchestrated. Now, we need to do guardrail for each one of them. The payment process should also go through the guardrail, should have a manual review if needed. And

same thing is with the order phase, also confirm the allocation. After it's confirmed, then we have, you know, the fulfillment should come in and then do the fulfillment for us. So, each one of them, the important thing is this, you're creating these agents, and somebody was saying here that uh you know, they're hallucinating. Yes? Somebody said You said that or somebody said Yeah. You have these agents,

they're hallucinating. What do Well, you need to do this, compensation paths. Have you created compensation paths? Compensation path means like, you know, hey, the problem is going to happen, then if that problem happened, manual queue, void the authorization. So, whatever the compensation paths are in the distributed system, you need to create that. Once you do that, now we are good to go with our solution. Awesome. I

think we covered this one. I'll go a little faster. Q, we already talked on the edge edge graphs. So, Rohit, what do you mean by edge? Edge means that uh whenever I'm trying to make a call, CRM query is being made, multiple calls are being made, old to new. Now, edges per second is what is being used to do this this particular mode of operation. All right?

Now, that is good. Now, from QPS to EPS all AI traffic with edge graphs can be added. So, these are all the graphs. Price check, inventory, 1 2 3 4 5 6 7 8 9 10. So, 40 to 50 edges per run. See See how much complicated they're behind the scene because we make the user's life easy, but then simple agent FAQ bodies too. Complex agent in

50 edges are there. Now, what happens if you retry if you're doing a retry in this traffic? What would happen if you do a retry in Amplify. You know amplification? I learned amplification in in in my electronics, you know. This is the amplifier effect. Graph become a storm to really work with. Now we get an understanding what Rohit was saying Now, what we need to do is

we need to look at stop contract conditions contract should be there. What do you mean by stop contract? That means that if there is a policy violation, if there is a state inconsistency, you need to stop anything which is happening bad. So, you need to be able to correct that. Retry after seconds, retriable, a reason code, all these things are there. And if anything breaks, you need

to have a break at a added to this. Now, you need to have automatic escalation process with human in the loop to really solve that problem. Okay? You would not have agent to solve everything Wow, that means I have to create human escalation or automatic escalation. Automatic escalation or human escalation can be done. That means that means based on the confidence AI can solve some problems, but

not all the problem. Okay? So, this is the way we actually control the microservices in the AI land. All right. So, we we looked at all of these now. Now, what's happening is now we did that. Now, what's what's the next thing here? Next thing is multi-dimensional rate limits can be added. So, you do I need to have only one rate limit? No. You need to have

multi-dimensional. That means apply the agent limits for who is using this agent, you know? Identify what tool you are using and what is the limits for each one of them per chain, per tool, per agent, per tenant. That means rate limit has to be in all areas, not just one. If you do that, now you have a control over what's going on. It that means it's not

throttling API gateway throttling. No, no, no. It's throttling at every level you can think of. Yeah? So, that's the difference between uh between this and the other one. another thing is the budget policy. You can Can you attach a budget policy? Yeah. You can have a budget policy. That means That means in this one, what we are saying is that check the usage and have a have

a budget policy. Any chain which is greater than cents per minute, move it to cheaper model tier. What do you mean by that? Cheaper model tier, anyone? Chat GPT 3.5, yeah. Send it to 3.5, yeah. Let No, it's not less tokens. It's not less tokens. See, if you are doing reasoning using Chat GPT 4.5, you're consuming reasoning time. Yes? Token is one aspect of it, but reasoning

time is also there. You're doing that much streaming also. You're doing that much chain of thought process also. Make sense? So, now what I'm saying is why you know, uh uh uh So, this is another thing which you can do here to really control. So, model tiering is one approach. Second approach is agent reliability control where agent write an event and then find out if there is

any problem in the event and then and then pause the agent if needed. And tenant budget enforcement and expensive tool usage limit. So, you need to provide these limits to really control how much money is you're spending on this. Now, we did this one. The next one is adaptive back pressure signals. Back pressure is about what when to say enough is enough. Don't do more than this.

Now, what is what is back pressure, anyone? What do you mean by back pressure? What do you think it is? System is monitoring under normal load, everything is good. Any signal which exceeds any signal that exceeds rate limit, any of these exceeds, what do I Trigger adaptive back pressure. That's a new terminology. I think I have not seen this before. You may say that. Yeah, I'll talk

about it. What this is. But, now you I want to become little bit like more, you know, because this is something, you know, not many people AWS and lot many people are not doing it as of now, you know. A lot of companies are not doing it. So, but if you do this, so what do you mean by adaptive back control? This is what it means. Very

simple. Back pressure technique techniques. What is this back pressure techniques? Shed low value chains. So, if you think about it, if you have a chain there, if you are overloaded, what are you going to do? Any API which is not needed, you're going to remove that first, yeah? And, you know, serve stale responses. Hey, you have some stale response, that's fine. You're still serving that response. Uh

currency limits can be there. Uh you know, defer the heavy work to later on. Don't do it now. Low priority chain. High priority chain, check out. Unrestricted is perfectly fine in this case. So, you need to know what agent actions are most important. Only perform those actions, not the other actions. Makes sense? This is the way you can provide something called as quality of service. Quality of

service can improve just by doing this exercise. That's great, Rohit. Now, the next thing which we can do is thunder herd when everyone wants the same thing. You want the same thing. I want the same thing. You want the same thing. You want the same thing. Now, trigger is happening. Now, cash hit is being done. Now, you deploy the new version. Now, when the new version is

deployed, what happens? The cash is flushing. When the cash first fresh happens, you know, it sends every agent to recompute the hot cash. And that's a real problem. So, do not do not do thunder herd thundering herd. Everyone is trying to call at the same time. This is going to like not going to work. So, you have to stagger it. What are you going to do for

this? Stagger Stagger Stagger means that don't do everything at 1:00 a.m. at 1:00 a.m. at night. Not a good idea, you know? So, you want to stagger this pre-warm the cash, you know? Pre-warm the cash. You know, single flight mutex per per key. Single flight, I'll talk about that. Stale while revalidate SWR. What is SWR? SWR is stale while revalidate. Yeah, we get the idea. Let it

be stale. It's okay. It's perfectly fine to be stale, yeah? And then jitter, like don't do it right away. That's what we are trying to say here. So, if you do that, wow, now we have a agent aware mini checklist created for us. Identify the noisy noisy agents. Oh, you know, noisy neighbors I heard, but there are noisy agents also? What is this? Yes. You can have

a noisy neighbor. If you have a noisy neighbor, you have to understand the governance of it. And based on that governance, you drastic shape that. And then identify the risk and then and then and then make sure that they are contained within themselves. So, that is called as noisy neighbors, noisy agents. That's what this one is. Now, this is something which you can apply air lock framework

which I created, uh which kind of do everything here. Agent aware traffic, that's number one. Adam potency resilience, recall and caching, something we cut the cost, you know, for yourself. Loosely coupled versus async, you know? Queue first and not synchronous first. So, you have to do queue. Queue means you send it to a queue and then process that. And another thing you can add to this is

you're sending it to while processing it process in chunks. Process in chunks means four requests you're 10 requests you're processing at the same time. So, instead of processing one request so So, LLM somebody ask a question, "What how to do this?" Somebody ask some other question. Can you club all of them and send it to an and get the answer from there? Yes, you can do that.

You can optimize all those things. Just not like one request going in and you put in the database and returning that. So, that's something you can do here. And then observability can be added with the trace ID and kill switch can be added to to remove the budget which is there. So, you want to make it stable AI system. That's what we will gain if you follow

this framework. So, what does that means? While duplicates, you know, really where duplicate really come from? I think we talked on this one. That's That comes from network is not there. Because the network connection is not there, you're calling another service, what will happen is one order will become six Why they will become six order? Because Adam potency key is not there. Because it's not there, you

need to apply dedupe window. What is dedupe window? Eliminate Eliminate if there is a >> duplicate, you know, and then use Redis, Postgres, any place to really TTL that after some time. But dedupe window, if you do that, I have seen this exact logic before. Yes? So, dedupe window is applied for orders. Dedupe window is also applied applied for? Get operations. You know, same thing people are

asking. And when this they are asking the same thing. But they're they're they're they're asking in a little different way. Like, you know, how long is your store open? Generally, what what time your score will be will be open? It's not the same But what you need to do, you need to similarity search done behind the scene. It's not It's It's not a vector search. It's called

similarity search is done. So, so what you're doing is you are saying that similarity So, hey, this is the utterance which is coming in. Is this utterance already there in my Similarity utterance is there in the cache. If it is there, then you are retrieving and getting the value. Awesome. If you do that If you don't do that, you're going to get everything from the database all

the time. How many here do that? Redis does that. Redis has the exact same concept. So, if you if you need to if you need to have a similarity search similar version search, you can use Redis to really search it, and it will be able to get that. So, so that's he that's the right approach to do this. Awesome. Awesome point, yeah. So, once you do that,

now database as final uniqueness guardrail. So, potential data duplication might be there. Unique keys, optimistic currency. Now, we want to avoid, you know, uh any any any database, you know, loud loud failure coming in. So, that's what we need to prevent duplicates in the So, that means circuit breakers. Circuit breaker means if the circuit is open circuit is closed, everything is good. If the circuit is open

then it's not working. Or it's open state. Till the time it becomes closed, Netflix history. We all know about this, yeah? We are pro at doing this. We can I apply this to the AI agents? Yeah, you can apply to the agent, same thing. You can apply to the AI agent in this case. All right. So, that said, bulkhead isolation. Bulkhead isolation is exactly what you should

do for premium compartment experimental compartment uh tenant A, tenant B. Now, why are we doing this? Because the agent is coming in premium customer, give them unlimited pool. Experimental customer, clamp them. You know, this is the way to control the traffic shaping. You're doing a traffic shaping based on the data which is coming in. Now, AWS style, what you can do is you can do exponential backoff

with jitter. Whenever you're doing a retry, do exponential backoff with jitter, and that is the right approach to get the randomness of retries coming in. So, that's the right approach, still the right approach, it's the best practice used all all the time. Now, once we do that, now this is the caching that scales, you know? Agent to tenant caching should be there. So, when working or tenant

is request is coming in agent retries now what you are doing is the result cached or not? If it is not cached then perform expensive LLM vector search. If it is cached, just get the value and return it. And that's what the Redis does. You know, Redis is a good thing to use in this case. That's great. Now, cache miss storm can happen, thundering herd can happen.

That means you can have a cache miss coming in. Uh that means that means cache we you're able to try to go to the cash and you know, you're missing the cash. It will all hit the back end that's a tumbling So, you need to be protecting yourself from that. So, what can you do? Generate cash key. You know, unique cash key can be created. Check the

cash and reusable cash result can be added. So, if I add the reusable cash results and reduce any invocation which is coming to me. Now, I create a key with prompt, input, model version and tool. If I have a prompt, similar kind of prompt is being asked, we will be able to do that. Makes sense? Awesome. So, great guys. I mean, what we have done we have

looked at a lot of things, yeah? This is uh you can you apply all of them here, yeah? That's what this this one was. So, stale white revalidate is another technique which is used by many people. The product manager is happy, user is happy, they're getting some stale data. You know, it's okay to get stale data then, you know, and then and soft TTL can be controlled

using that. Negative caching. Negative caching is like, you know, cash cash is known for failures. Cash is known for failures, yeah? If you get a failure, you would is it possible you get the similar failure again? So, yeah? It's possible. So, what you can do is that incoming request is coming and check for negative cash first. You know, and then you make a back end call and

you get the failure and you record this failure in the negative cash. Store it in the cash and TTL after it expires. Now, this advantage is advantage is that now, if somebody calls the same thing again, 404 error coming, you will be able to protect yourself. Schema error is coming, you'll be able to protect yourself from that. Makes sense? So, that is another good way of uh

you know, building this cash. Jitter TTL, I think we talked on the jitter TTL also um sometime back. Uh migrating cash stampede with single flight. Single flight means cash stampede happen implement single flight. Okay, single flight what is single flight? Single flight or request collapsing is a pattern where multiple concurrent requests for the same cash miss are merged together. So you are you are Yeah, so that

means same cash miss is happening. So instead of instead of saying hey this cash miss this cash miss this cash miss you are keeping it in the cash. Hey this cash miss already I'm already retrieving the data from the database. So you don't again and again go to the database and make that as a bottleneck. Yes? So this is called as you know cash stampede through single

flight. See the flights took place if somebody is asking for the same thing just ask them to wait you know. Why are you bombarding the database to solve this problem? Yes? That is what this one is. Queue first architecture. Always have a queue first architecture. First create a job. And then based on that job you have a chain ID and then create a priority of that job

in the priority queue and then solve for it. Now if you want to know about priority queue that's my next talk you know we would be able to discuss graph in more detail how to do that. And then when I need to do the CPU based event driven architecture. Now CPU instead of CPU now we have queue based and lambda based or event driven architecture can be

there to solve some of these problems. Saga pattern. We all know about Saga pattern with multi-step agent workflows can be added. Now if somebody was asking for multi-step agent workflow yeah. If you have a multi-step agent workflow is there you want to make sure that you have a all or compensate behavior in the is there in place. So you need to build all and compensate behavior. That

means if the transaction takes place, it takes place for all. If compensation takes place, it takes Outbox pattern, never lose any event, put it in a database or some place and then be able to process that. I think we all know about this one. Uh and then you can build a solution and priority backlog. What is priority backlog? It's something good to know because what you can

do is not everything is important. You create a highest priority first. And lowest priority flexible scaling later on. So, bronze queue and gold So, don't create the same kind of queue when you're working on the solution. Now, when I'm trying to do asynchronous work going on, you scale the workers, implement saga pattern, and then provide streaming progress. Now, streaming process progress is very important. Now, LLMs can

do streaming, yes? That's the reason we are happy, you know, whenever an LLM is coming in. If you stream the data, only thing is it's streaming the data back to you. The user is able to see that data. Now, you saw while it streamed back to you that it says the toxicity is there in this. What can you do? Now, you will replace that whole thing because

it's streaming. You user is able to see, "Oh, some bad thing is coming to me." What can you do in that case? Replace it. Like I mean, you have to get the whole thing and then you can evaluate that. So, we will literally it's coming in. Now, once you realize, "Oops, something bad has happened." Now, you have to replace the whole thing with some nice message, you

know, saying uh you know, that's uh yeah, some some other way to do that." So, that's another thing which can come in, you know, when we are when we are working on this. Uh and then prioritized queues can be So, logs is another thing. Context is missing. We need to be able to add the context through chain ID, agent ID, chain ID, tool ID, tenant ID, and

propagation tags. Chain ID and all other ones are there. If you have these IDs there, you will be able to exactly find out where the problem took place. Because how would you do debugging in agents? It's a nice subject to solve for. If you don't have it, you can't do it. Tracing should be there end to end to really solve for this problem. Now, tracing the agent

like microservice open telemetry, we all know open telemetry can do that. Jaeger is there which can which can solve for it. And identifying the addressing the latency which is coming in, you know. You know, it can happen, you know, SLO have SLO for for your solution, you know. Agent hits slow path. You know, sometimes they hit slow path also when when they're working on it. All right?

So, that's it. We already discussed, you know, 50k weekend happens. Auto scaling has to happen. Max replica, you need to be able to build You need to build a cost-effective solution when you're working on it. All right? So, I think we completed our our play for today, you know. I think this is This is a great journey with you. Now, I have got like few minutes for

any questions which are there like you think We have like 3 minutes left for us to do it. But what did we do accomplish in just very 1 hour for us? We looked at different ways of like, you know, getting the bill. If you get a strong bill, lot of bill is coming in, you need to be able to protect yourself. We say I created an agent

and it's in production. No. You need to do all these things which I discussed with you. Now Now I discussed like, you know, not only scalability, tail latency, you know, all all those things we discussed. And once you do that, now now this is something a diagram which is easier to understand. Once you do that, the plan, tool, observe, and decide, and repeat. Now you're able to

build this retry storm, and also build the perspective which is with brakes. You need to have brakes. Just like, you know, you need to have good brakes for the agents. And you need to have an agent manager really manage and continuously observe what's going on to build the solution for yourself. All right. That said, I am good for at least one question. Okay. Anyone would like to

go? Any question? >> All the Let you show trade on the Yeah. >> Yeah. Uh-huh. >> Are there any frameworks or any platforms available which that that already got? >> There are some framework like, you know, so the question is like what frameworks are available to do all these things? Not not many frameworks right now. And some of these things which I'm discussing is coming in 2027,

you know, but you need to code for it. But I don't think there is any framework for See, this is job security, man. It's like you're not going anywhere. It's So, this is something which I'm discussing here is we say agents oh no no no this is this is the reason you are here. Once you know these things, kill switch, you know, you have to code for

it. Sorry. You know, you need to which which which AI agents are more important? Bulkhead, you know, all the things which we discussed right now, these are like practical. Like you solve for it. I guarantee they will all solve for yourself. Okay, great. Any other question? Yes. >> If all the elements Question, yeah. >> So, when you call the LLMs, I think you just said that it

is a stateless, no retention. So, how do you make sure that you have said that we need to make it deterministic. >> Yeah, deterministic. >> LLM is no retention. >> Stateless. >> How do you make sure that >> Come to the next talk. Because I have to talk about graph. The graph graph rag and there's a few things which will come in as you do that. So,

well, thanks everyone for joining in here. Have a wonderful time. I enjoyed talking to you guys. See you