Building LLM-Powered Agents with Real-Time Reasoning Loops - Apurva Misra

This session focuses on building language model (LLM) empowered agents that utilize real-time reasoning loops to enhance their capabilities. The speaker, Apura Mustra, a machine learning engineer with experience in AI strategy, discusses the concept of AI agents and their functionality compared to traditional LLMs. The talk highlights the significance of Cloud Code as a widely adopted tool among software engineers and emphasizes the importance of task specificity, autonomy, and reactivity while developing agents. The speaker elaborates on various frameworks for constructing agents, including a detailed exploration of the React framework and its application within Cloud Code. Additionally, the session addresses potential pitfalls in agent development, such as context limits and cascading tool failures, while offering various solutions and best practices for building robust and effective AI agents.

This session is about building LLM empowered agents with real-time reasoning loops. About me, uh she introduced me already but my name is Apura Mustra. Um I am a machine learning engineer. I started uh consulting independently like two years ago and I have been like in the education space uh in in the last two years as well like speaking at conferences running workshops for companies um just getting

them up to date with AI and getting them started with their AI strategy. So I've been very focused on that. Um and I'm also an aspiring gym warrior. So I've been like continuously going to the gym because I've been hearing that we are all going to lose our jobs. So better get ready for all the blue collar jobs out there. Um and yes so uh how many

of you have used clot code before? Cool. Um how many of you have used any AI assistive tools for coding? That's nice. Nice. Pretty much all of you. Um okay. So we got really lucky three weeks ago. Cloud code uh code got released or not released. Leaked now they say it's released. Anyway, so um it got leaked and what I've done in this session is basically uh

we'll set up some foundation and then I'm going to tell you like if you're building AI agents with LLM, what do you have to do and what is clot code doing um based on the code. Um so this is how we are going to go in this session and why I have chosen clot code for this is I would say this this is the most successful agent

out there right now. I think cloud code is used by pretty much um 80% of software engineers out there at least in North America that I've spoken to. Um so I wanted to focus on clot code as an agent and see what we can learn from it. Um so that's and then I'm going to go over some pitfalls and then what are the solutions and then we

will learn about these two frameworks for building agents. Um and then we'll run through the demo. Um let's see how much we can cover today. Um but let's get started. So uh an LLM is basically um a machine learning model treated as a black box. You give it an input, you get an output. The input could be any context. So if you go to chat GPD, you

ask something, it'll give you an answer back. So that's what an LLM is and that's what the basic use of LLM is. Um, but if you add like a loop around that LLM and add memory planning and tools to it, that's where it turns out to be an agent and you can use it for like complex tasks cuz it can plan break down that complex task into

smaller pieces to work with. Um, it has memory so it can like self-improve cuz it will know like what m mistakes it made like two turns ago and fix that. Um, and you give it access to tools so it can take actions on your behalf. If you've heard about cop open claw, that's uh that's an AI agent as well. People are using it for personal uses even

though it's not that safe. U but that's what a basic agent looks like. Okay. So what's important for an AI agent? Um it's autonomy. So it depends on you. Um like in clot code, it'll come back and ask you for permissions. Unless you're using it with claude dangerously skip um all the permissions um it would come and ask you for permission. So it depends on you. you

can give it like full autonomy to mess up with your system or you uh make it ask you for everything it's trying to do, every action it's trying to take. The second thing is task specificity. So I wouldn't suggest you to give it like a very generic task and ask it to complete. Um I would prefer you give it a narrowed down task uh and it's more

likely to do it perfectly. So that's what specificity means. Um and you have to be very precise about what you want from it. Um because we're using natural language so we have to be very specific. Um then reactivity. So like if cloud code or any AI is just a tool if it makes changes to the code so something has changed in the environment right it might have

downloaded a new dependency. It might be affecting another dependency. So the environment has changed. So now it has to work with the new environment. That is what reactivity is and an agent should be able to do that. Okay. So um since the title of this uh presentation is you are building agents with realtime loops we are going to use react here. So um at the bottom right

you can go and check out the paper that's the react paper that's the most popular paper out there for building agents with uh it's a policy and what the policy states is you you use an LLM you give it a task and it will think about that task it will act on that task because of acting on the task the environment around it will change and because

the environment has changed it has to again like work with the new environment and keep doing this loop until it has solved the problem. Does it make sense? That's what we were talking about in the earlier slide as well, right? It'll take actions and the environment would change and it has to Okay. So in clot code, this is the same thing that happens. They're also using React.

Um so at bottom right, that's the paper um that came out a week ago and it's all about the clot code uh code which got leaked and what what are the specifics in there. So this diagram is from that paper. So do check that out. Uh but basically you as a user you could be using CLI you could be using cloud code through your ID v code

uh or any other ID uh so that's the interface you send your request and that's where the agent loop runs the react loop and then it has uh access to tools tools are the actions that it can take and it the action uh happens through the execution environment so like a shell uh uh command like it would run there and then um that action would lead to

a change environment and it'll work on that. So there is permission system as well. We'll go into that and state and persistence that is the memory. We'll go into that later as well. But this is this is the this is a react framework and they are using it as well. Clot Okay. So when you're working with LLMs I mentioned this earlier cuz you're using the natural language

you have to be very very very specific about what you want from the system from the agent. So that is what specificity means. Um and like if you are saying 2024 it could be mean it could mean any of those right? So um be very specific. Give it all the information it needs um to do that task. The next thing is comprehension cause, um these LLMs are

very good at generating code. You would have heard about AI slop, right? It does the same uh thing. It applies to text as well as code like a lot of code that it generates. It's so good at generating code, generating the next token. Um it's not always useful. So um and you don't have the time to like review all of those PRs. So now you would have

heard about the uh new claude tool which is the um claude review tool uh PR review tool which like cost $25 per PR review but there are so many new tools coming out for code reviews. Um so use a validation step. It could be AI, it could be human to validate what the agent is doing cuz it's so hard to comprehend how much these AI agents are

generating. Um next thing is probabilistic. Um you give it a question, it'll give you an answer. you give it the same question, it'll give you a different answer. Uh the same applies to the code as well. So, um again, you have to be very specific. Use your cloud.md file. Um use the superpower skills plug-in. There's so many plugins to like make sure you're able to give it

all the right context so that it's able to do the tasks at hand. Okay. So, what else? What other issues are there when you're working with LLMs? Um context limit is one of them. So um context limit is basically there is a if if you guys track u model release cycles like every time a new model comes out they would say it has a limit of 1

million tokens and honestly like there there has been research done in which you can only use 40% of the token limit to actually get useful tasks done. So that is not the true limit. Whatever they say is not the true limit. So I would suggest to like don't go don't go in a don't continue the conversation for really long. Uh if it's being continued for really long

ask it to summarize and continue the conversation again in a new session. So um that's what context limit is. Then like non-deterministic probabilistic the same thing. Um it wouldn't give you the same output every single time. Um cascading tool failures. So um your agent could have mult access to multiple tools and then it would be doing like um a task. It would use tool A get the

information from tool A and then use tool B with the information that it got from tool A. So if tool A makes a mistake the other tools after that are also going to make the same mistake or like continue the mistake basically. Um so that's what cas cascading tool failures is. And then the next one is infinite loops. So um I was actually working on an agent

yesterday um and there was like human in the loop. So it has to ask for permission to do an action what clot code does, right? And sometimes what happens is it'll keep asking me for permission. I'll say a yes. It'll again ask me for permission. So it can go into infinite loops. That's an issue with agents as well. Um and then silent failures. What will happen is

it'll it'll cost you a lot. It'll use a lot of tokens and then the output that you get is wrong. Um and you wouldn't even know like it's just generating tokens. it's accepting tokens from the tool call and you won't even know where it went wrong. So um it can fail silently and you would get to know that only if you like go through the trace and

see the final output. Okay. So there are a bunch of solutions of because there are production agents. Cloud code is one of them everybody is using. So um the solutions I'm going to go over each one of them but these are like the this slide has the list of them. Um let's go through multi- aent systems. Um so in a multi- aent system what you're doing is

you can orchestrate orchestrate in it in like a lot of ways but one of the main ways that it happens is you have this main orchestrator and then it has a bunch of sub agents that it assigns tasks to they complete the task and get the information back to the orchestrator the main agent and um it continues with the other task. So basically the main uh LLM

doesn't have access to the sub agents context. So if the main LLM is like a deep research agent, it can ask a sub agent to do the research for a certain topic um maybe about like filter coffee in Bangalore. It doesn't care what all URLs it went through. It just wants the last answer, the final answer and that's what would be in its context whatever the sub

agent replies with. So you are basically not utilizing too much context space because the sub agent is doing a separate task and you're doing a separate task. um than parallelization. For example, if I want to find clients in like multiple countries, I would rather have like multiple agents working on it, multiple sub aents working on it. So um I can get done with the task faster. The

third thing is specialization. Like I was saying in the deep research agent, one could be just the research focused agent or for example like you can have the main agent and you have like a sub agent for social media, the next sub agent for SDR. So like you can have specialized prompt for each of the agents and the main agent doesn't need to know all of that.

It just knows this one is an S str, this one is a social media. So if I have to post on Instagram, I should tell this sub agent for that task. Okay, there are a lot of issues when you're using multi- aent system. First of all, it becomes very complex. Um so like stale shared memory, they they would be sharing the same memory state. Um and sometimes

what happens is if a sub agent puts something stale in the memory or doesn't update it, the other sub aents would pick the same stuff. So um that's stale shared memory then the handoff loss cuz they're talking to each other sometimes like they summarize something and the information gets missed. Um role confusion like a research agent might think it's an SDR um or a social media expert.

Um telephone game again like the same thing it'll lose the factual information lose that um parallel disagreement. So it it might have there are solutions to these also like you have a bunch of sub aents trying to do something and two of them don't agree with each other you know um so that's the multi- aent system and plot code is multi- aent there are like six or

seven of its own sub aent and you can define your own sub aents as well um progressive disclosure progressive disclosure is basically I don't want to fill up the context completely in the beginning there's so many tools I want to give access to my um agent what I'm going to do is I'm going to write a name of the tool and like write one sentence about the

tool and then the agent can go over that and be like okay I need the email send tool and I need access to the email write tool and that is it or like uh pulling information from the DB tool. So it would just ask for more details about those tools. How do I make the function call? What are the arguments required? So it wouldn't have all of

that already in the context window. Depending on what it needs, it'll ask for that. So that's what the skills um skill activation is. Your clot code has that. So you can have skills in your clot code and it would just have access to like the smaller bits of it. And as clot code needs like more information, it'll ask for it. So it's not filling up the context

window from from tools that it doesn't need for the task that it's doing at hand. Okay. So the next one is context management. Uh you can do this do the same thing with multi- aent systems. Um but the other thing also you can do is like clean up your context after every turn you know after like every LLM call that's happening clean up your system. So you

would have heard this in like any of the AI coding assistive tools it has compact as a functionality. So it like compacts it summarizes what's already in the context and then you can continue the conversation in plot code as well. I think if you're using cursor like it shows you how much of the percentage you have filled up um for from the context Uh but basically like

uh in cloud code what they're doing is uh they can limit tool um size output. So basically you can say like I want the output to be only 100 tokens from each tool. I don't want like a 5,000word answer from a tool. The other one is snip. So basically like you can cut off truncate the output as well. Um micro compact pretty similar. It's just cache aware.

Um then you have like a bunch of other compact features. claude.md lazy loading is very similar to skills don't load the whole thing just load the first page and then based on the hierarchy you keep loading whatever claud needs um deferred tool schemas is the same as skills um sub agent summary only return like I was telling you uh before like if the sub agent is doing

a task I don't care what it is doing in its task I just want the final output so I don't want all of that mess in my context um per tool result budget again the same thing but It's um currency wise like I don't want this tool to cost too much. If you're using Tavi or something I want to limit how many searches it does. Okay, this

is what uh uh is there in the paper the diagram about how they are taking care of the context. So think about it like if you're using all any of these like coding assistive tools there's so much it has to take care of. There's like a system layer, the project config, the claw.m MD and the hierarchy I was talking about. And then you have like memory, the

conversation that you are having with it. And then like the runtime stuff, the tool results like every time it calls a tool, it runs a bash command, the result would come back and fill in the context. Um, and then like the ondemand lazy stuff that can so this it it goes in this order. So like that would be loaded first and then like the lazy load part

comes at the bottom and like claude model reads all of that. Okay, permission and authorization um in claude how they're doing this is like you know this already like every time it does something it'll come and ask you should I run the bash command? Are you okay with me running UV something something? So um it does tool prefiltering. So it has access to a lot of tools.

You can limit the amount of tools it can use. It can only use 10 tools without permission. The rest of them it has to ask you for permission. Um the deny first rule basically like anything would be denied if there's no other rule. So deny first rule is uh the priority. Then you have permission mode. So you can have different permissions right like you can have the

permission for planning uh the permission in which you can say ask me every single time the permission in which you can uh the mode in which you can say that just do whatever don't ask me just give me the final result um shell sandboxing auto mode classifier is basically like they have a machine learning model in the background so every time there's a tool call that the

LLM says like I have to run this bash command or um I have to like send this email um it would classify this as okay this is sensitive it's trying to access uh Apurva's email inbox. It should ask for permission. So that's an ML classifier in the background. Um then not restoring permissions on resume. So in like if you resume a session, it doesn't carry on those

permissions. You have to give that again to the cloud code session. Um hookbased interception. I think this is only there in clot code. I haven't seen this in codeex or other um other coding assistive tools. Uh basically hooks are a way to make it deterministic. So hooks are like pieces of code you can ask it to run at a certain time. So like you can say after

every tool call run this compact like or after um for before any bash command you ask me for permission. So you can have like pre-tool use hooks, you can have posttool use hooks, you can ask it like after three turns um just end the session. You can ask for stuff like that. So it's just a code that you can ask it to like run at a certain

point of Okay. So uh the tool authorization and control boundaries this is how it works inside short code. So like I was saying you have the rules, you have the hooks. So rules are basically your markdown files and then you have modes. modes like I was saying you can have like allow all permissions ask me every single time or like the plan mode so um that's the

modes and then you have the hooks hooks are the code uh pieces that you can ask it to run at a certain point of time um and then based on all those three it would decide for this action should I ask the user um should I run it or should I just deny it okay so adding determinism you can do it by hooks um add hooks in

your system. Um and you can also have hard stops. Actually, not also, you should have hard stops. So, if you're building any sort of agent, um you would notice if when you start uh building it, it it wants to utilize as many tokens as possible. I feel like all these companies have trained their model so that it costs you. They want to like make you spend on

tokens. If this answer could be given in five words, they wanted to use like 25 words to give you the answer. That's one thing and it wants to they also want you to like keep continuing the session. So always you would have noticed this in clot code like if it does something it'll come back and ask you a question. Do you want me to do this? Cuz

they want the conversation to continue. Um that's why there's this repo you guys should check it out. It's called the cave man language. So like use the least amount of words to have a conversation with me so that you can save the tokens. Uh but basically if you don't have hard stop conditions it'll keep running. So like have uh a criterion to make it stop uh from

running. So in cloud code what it's uh using is there's a no tool use it checks for that. So if clot code if the LLM doesn't respond with like take this action run this tool it just responds with some text it's like okay it's done with like the whole loop. It doesn't have anything else to do. It just has u um text to uh have a conversation

with the user. So it'll stop there. Um the next is maximum turns. You can say I I am okay with only like 10 rounds of it running. Um the next one is context overflow. So if the prompt is too long it would stop there. Um the other one is hook intervention that is you can run a hook you know after every bash command just stop. I'm going

to start a new session or just stop the running. Um so you can do that too. And then like explicit about like when you do control C and it stops. That's what that is. Uh the next one is fallbacks. So uh clot code has recovery mechanisms. Um so if it has reached beyond the maximum output token capacity of that particular model, what it can do is retry

again. If that doesn't work, it'll error out and stop. Um the next one is the reactive compaction. So if it sees that we are uh getting pretty close to the context limit of that model like a million tokens or whatever they I'm pretty sure they have like a different limit in the background um they it it would like stop and it would run compaction first and like

try to free up space so that it can finish the task. Um next is prompt to long handling is the same thing like it it would say prompt to long handling um error and what it'll do is it'll run compaction directly after that. Then the streaming fallback like every time you're using them for streaming there are issues with that and the fallback handles that. Um the fallback

model is if their model is not working it would switch to a different model. So if opus 4.7 is not working it'll switch to opus 4.6 directly. and the self-improving systems. The only way you can have self-improving systems is by having memory in your system because like if it makes a mistake the next turn it should know that I made this particular mistake. I was trying to

force this particular library to solve this problem but it's not working I should look for an other library. So that's what memory means and there are two kinds of memory. One is the working memory that is like in this session I was trying to use this library and it isn't working. Let me install this other library. Um and the other kind of memory is the long-term memory

which is like apurva. This is from chat gpd actually what it learned about me. Um but in in in the coding example what it would learn is I prefer um pyantic AI to build my agents or I prefer like piest to write my test cases or this is the format I like. This is like a long-term memory that it learns. Um, so that is what is needed

to like get your agent to be self-improving so that the next time it's doing a task, it knows I'm not going to make this particular mistake cuz I did this in the past. Okay, but anyways, all of this is still not enough. Um, there are issues with clot code. It doesn't really follow all the rules that you mentioned in cloud.md. You would have noticed that as well.

Um, so there is this uh um company out there. You can check it out. I took a screenshot from their website but basically they have this graph network that is external to cloud code and what it does is it's something like hooks like every time it's taking an action is deciding something it'll come and check is this action allowed did I did the user mention this in

cloudmd so it just like checks after it's like validation checks after every action checks before every action um and make sure it's aligned with what the user mentioned in cloudmd file okay um also there is like a bunch of research being done in this area. So honestly you can make like good enough production ready agentic systems with all of that. Uh but as the models improve you

would need less of all those all those solutions. Um so model introduces its own sub aents. So there is this research paper in which they have trained the model to decide when I should spawn the sub aents. um what should be the area of uh focus for those sub agents and what kind of task those sub agents should run for. So you don't have to define your

own sub agent. So if you want to make a multi- aent system, you can just pick up a model which knows how to um use the actually spawn the right sub aents and like assign them the right prompt and task. Um so this is that paper they try to train it such that it knows when to spawn those sub aents and like I was saying earlier cloud

code also does that. It has six sub agents that if you say like use sub agents, it would know which ones to spawn. So uh and you can define your own and this is what it will look like. So you will have the orchestrator. it has access to these tools and then it can like create sub agents give them access to like right now it has given

access to the three tools and all of them are the same but it can decide like I'm going to give this uh sub agent it's focused on very specific task it just needs access to two tools to take um actions with to finish the task and then like it'll wait on the sub agents and figure out the answer and give it back to the user. So you

don't it's it's happening inside the model you don't have to worry about any of that. So this is what the future looks like. Um the other thing is context as an external variable. Context limit is a very big issue, right? So um there's resource there as well. So what uh when you give it a very big prompt, maybe a book to read through, it would take that

book and it'll put put it externally to itself. It's not going to read it inside itself. Um what it's going to do is it'll talk to that big prompt or the book through code. Um and like and how that it figures that out is like that is how it's been trained. So if someone gives a query, it'll figure out okay I need to look at chapter one

and I'm going to like split the book and just get the chapter one and work with that. I don't have to get like thousand pages of that book inside myself and like start working through it. Like when you attach a PDF and chat GPD, what it'll do is it'll put the PDF out and it'll just pull like pages of the PDF based on the task that you

have given Okay. Um so these are all the like research the these are all like the research papers out there and that's what the future looks like but currently it's not there. So um we have to build the system around the model so that our agents are production ready and reliable and in uh in like in the coding space code um coding agents uh it's also called

hardness engineering. So um when you you when code became really I think flaw code is just one year old. When it became really popular people were like um it became so useful only because it has access to very limited tools but very useful tools. Um one of that was like the file system in git. It could like grip whatever it needed from the files and it could

like push to git log everything. So that is the harness around um the model. So um they have given it access to those tools and those permissions so that it's able to work as a coding agent. Um so if they figured that it needed to like write and execute code so they gave it access to a code execution environment and gave it access to bash. Um same

thing like if they knew it needed to like complete long horizon work so it needed the react loop at the bottom like it needed to do a task plan the task um see how the environment has changed and like go over it again and again to be able to do the task. You would have seen this as well like cloud code like 5 months ago wasn't that

great in December a good model came out the hardness was pretty much the same but a good model came out and like it's doing really well. Um okay so this is with with the harness engineering included. This is what the clot code architecture looks like. Um you have the surface layer. This is how you interact with clot code. You have the core layer that is where the

agent loop runs. The react loop and like on the right side the action layer is all the tools that you give it access to. So the MCP tools I think there were uh sessions about the MCP tools. Um and the sub agent spawning that it does automatically. Um it has access to sub aents if you ask it to like spawn sub agents or you can you can

define your own sub aents as well. Um then it has access to the shell sandbox um to run your tools with the back end and then the bottom layer is the memory piece. So like everything from the past that um in that past session the long-term memory your claw.md uh the your side chain transcriptions um all of that happens in the state layer and this happens in

every turn at every turn it decides okay the first part u of this like the first five um user queries are not related to the sixth user query so might as well remove it from the context it figures that out like Apurva wanted to build something with pyantic AI and now she's working on her own website unrelated it completely. So like remove that honestly you should start

a new session to do that but like it'll figure that out itself. Um, and this is what your runtime flow looks like. Like you give it a query, um, and it goes through an iteration, you know. So you give it, you ask it to build a website. It'll it'll figure out I have to make a tool request. Um, I have to write code, make file changes. It'll

ask you for permission if you haven't given it. Um, and then it'll make those changes and then it'll go through iteration two. Okay, I have to run it. And it'll ask for permission. like it'll go through all of that and then finally the output would come to you. Okay, let's go into frameworks. Uh I have a demo um for pyantic comparing pyantic AI with BAML. Uh but

to give you a little bit more context about them. I have created a web page, let me let me share that with you and you guys can also like scan it and open it on your laptop if that's easier. Um this is the QR code. So if you want you can scan it. Is everyone done scanning? Okay, I'm going to I I just want to like go

over the slide once more. Just remember like agents are not anything different from what you're doing with chat GPT um claude or any of these interfaces. The only thing that changes is you're giving it access to a bunch of tools. It might look like cloud code is not an agent. There are a bunch of tools that it has access to. It has memory. Um it knows what

you want longterm and short term. It knows like um the conversation was about um the website. Um you don't have to like put that with every message that you're sending to clot code. Um and then the planning it it plans with you. It tries to break down the task. So that is what an agent is. Um the only thing is like it loops. So you give it

a task, it'll try to break it down. It'll come back to you. So it decides the action is to speak to user. It'll come back to you um and ask you like does this plan make sense? And if you say yes, it'll go back and be like the next action is write code. And uh if you haven't given the permission, there would be human in the loop.

It'll come back to you and ask you for permission. So ask permission with the next action. Um once you say yes the uh it would run the action for writing code. That's what it does right once it's finished writing the code it'll figure out the next action is um reply to user. It'll come back and say I'm done writing the website. Do you want me to run

it? And then you say yes and it'll be like okay the next action is run and it'll go run the website. Does it make sense? I just wanted to make sure like if anybody not uh anybody u doesn't have like the AI background this it's just as simple as that you're just giving it access to this uh these tools the planning capability the memory and it acts

as an agent the the issue is LLM are non-deterministic or probabilistic um that's what machine learning is right like it's been trained on all the data out there on the internet so it's able to give you like the average answer um when you ask it about Bangalore or it figures like South India dosa filter coffee all of that um cuz it has learned on through Wikipedia pages

Reddit and everything um the dosa here is amazing or where to find the ben dosa and stuff so it's the average of what it has learned over the learned from the internet um and every time you ask it uh to do something it it just picks up like the most probabilistic token from its vocabulary so it would have your u it it is the English vocabulary as

well as like the coding vocabulary. it figures okay uh you have the closing bracket then you have like the function name then you have the sorry the opening bracket function name closing bracket whatever so this is how it's generally written so I'm going to write it that way um that's the average so it it might be the case like when it's u writing a function about um

about I don't know um generating like the I don't know generating the a color I don't know what whatever I'm I'm able to come up with something. It's writing a function. Um it figures like functions are written this way. Um but maybe the next time it's writing that function, it would add a comment as well in that whole um uh code piece. So it it might add

stuff, delete stuff, change stuff because it's probabilistic. Uh it might run as well. So you always like when you're working with these coding agents, try to do testdriven development. Um because it's so easy now to write tests with them. Um, okay. Let's go back to the demo. so uh you would have access to this as well. Um, uh, I have put everything on that GitHub repository so

you shouldn't miss out on anything. But, uh, this is I created this, uh, web page just to like have a comparison between these two frameworks. Pyantic AI if you I'm pretty sure you guys have used Pyantic. Pyantic AI is uh uh using Pyantic in the background and it helps you build AI agents. So it has a lot of support for all those solutions that we were talking

about already when you're building an AI agent. BML uh is more like it's also a framework but it's more like a code generation tool chain. So you write what you need um in the code and then you um run this generate command and it would create these compiled files and you can run that with like any of the type languages. You can use it with TypeScript, you

can use it with Python. So like if you are using anything other than Python cuz everything right now in the space is very Python focused. If you're using anything that other than that, BAML is perfect. BAML is the library to go for um to build your AI agents. Um so this is what we are going to build. Um I'm pretty sure you would have used DP search

uh in all these like claude Gemini um chat GPD. Uh there is this DP search uh um tool there. You can ask it a question like look for the best yoga classes in Bangalore. I'll start searching. What it does is like it does a web search uh gets like the first five URLs. it figures okay um she is based in Kor Mangla um and I'm going to

like do one more search focus on Kora Mangla and uh okay she has a price uh limit for the yoga classes so okay I'm going to like search by the price limit so it like does all of that so these are like turns happening so it does a web search is a tool so it's like calling the tool figuring out the output is not enough to answer

the question so it'll call the tool again and like look at the output again and see if the answer is good enough um so this is what a DP search agent is we building something similar. In our case, um user asks a question. Um is it big enough? Okay, so the user asks a question. Uh the LLM chooses the next step, right? That's what um a react

loop looks like. Uh it's making a decision. These are all the tools it has access to. So it has a search web tool. Uh it has a query internal DB tool. It has a thing tool in BAML. Um it's just BL focus. That's where the lines are dotted. I'll tell you later like what it means and then it has a final answer like reply to the user

tool. Um and based on that um it would do the next step. What I've done is if it has to query the internal DB, it has to ask me for permission. Um cuz I wanted to show you human in the loop how that works. So there is a human approval uh step uh after query internal DB. If you say yes, it writes the query um gets the

output and sends it back to the LLM. So like the tool result goes back to the LLM and now the LLM decides what should I do next? Is this information enough or do I uh do a web search um or do I do um the SQL search again? Uh if the human says no um you just say um it's it was denied by the human being. What

should I do next? And the LLM would decide what to do next. Maybe it's just a user reply. It'll ask to reply to the user next. So um that's what the flow looks like. I'm going to do the demo first and then we'll go over Okay. So, you have all the instructions in your readme file. Um, so when you when you when you guys are running it

at home, you can use that. But um the I'll go over the prompt as well. But basically like if in the prompt we have said if the query is about a product, if the query is about um the Q1 quarterly results or something it has to be looked into the internal DB. If the query is about um uh the best Italy summer um you have to go

to a web search or the best yoga class you have to go to a web search. It's not in the internal DB. Internal DB is only focused on the company notes. So um I'm running the BAML example. I'm just asking you to summarize the uh Q1 product launch results. So what you can see is uh the agent is running. What I've done is this is my stop

condition. Do you see that maximum turns equal to six? Yeah, I've zoomed in. Okay, maximum tones equal to six. That's my stop condition. I put that um so if you wanted to run for like 10 uh 10 times, you can like make it run 10 times. What I've done also is I have given it um access to um SER API. So it's doing web search using the

search API uh SER API. So you can use like Tableau or the other web search to APIs you want. Um so I asked this query. Um it has uh it's allowed to do only six turns. Uh and you see that it has triggered uh it has to do um query internal DV. That's the next action that it has to take the topic. So the LLM this is

like it it went to the LLM. This query of mine summarize Q1 U product launch results. It went to the LLM. it figured it has to look into the internal DB and it'll give you the topic that it wants to look for. So, because it was about the product, it wants to look for the topic product. Um, and it's asking me for permission cuz I've asked it

like human in the loop. Like anytime you're doing a querying in the internal DB, you have to ask me for permission. So, I'm going to say yes. And now this would have gone back to the uh the LLM again. Like if you see it was calling the LM again. Um and it it got the SQL query, it ran that and then it figured the action type the

next action type is reply to the user. Um and this is the answer that it came up with and there are these are citations. So I when I go with the prompt you would realize that I have I'm very specific. This is what you can do with frameworks. You know you can be very specific about this is how I want it. I want the typed output in

such a way like citation should have a source type should have the URL or doc ID and should have the snip snippet that it used um to answer that query and finally it'll give me the answer cuz the next action was reply to the user. Does it make sense? This confidence and citations again like I have asked it to put it put the answer in such a

format like once we go over the code. So this is what it looks like um when you use BML to run this DP search agent. Um let me show you the code. So in BAML like I was saying when you do the generate it creates these files and then you can uh work with any language. Uh it doesn't have to be Python. I just like work a

lot with Python. So um you would see me using Python but uh this is where I have defined my um types like I was saying. I have the citation type. It should have like I was saying it should have the source type. It should have the URL or doc ID. It should have the snippet that it used for reference. The message should be this. So a user

can message, the assistant can message, the system can message. Um in my agent loop, the tool result uh should be of this type. Um if it's calling the search web action, it should always have the query otherwise what it will search, right? So it should have the query. The llm should respond with action type as search web and it should have the query also like what we

have to search. Um if it's a query DB action again it should have the topic cuz in our SQL query we are putting the topic in and then it has a think action it's just like it's thinking uh so it's just a plain LLM call like with all the results till now we just do an plain LM call um and then reply to the user action it

should be um this type it should have the answer it should have the citation it should have the confidence that is why you had this it should it had the answer it had the confidence it had citations so I posed it to give the output in such a format. So this is how you can use frameworks to make it deterministic. That is how like that is how

you can work reliably with these LLMs. Okay. So um let me show you the while loop of react like it'll keep running till it gets the final answer or it has reached a stop condition like the maximum number of tones which is equal to six. Um so in our case this is the run agent function and then when if you see this is like the for loop

so it has a limit of like maximum tones um and then it checks like every time it it runs it makes the call to the LLM this is where the call goes but messages is basically all the conversation till now the state we talked about like the context manage management it's just like all the tool responses along with the user messages or the LLM messages it goes

in. So the tool results and the messages are going in and the LLM decides what's the next action. So if the action is search web it does the web search. If the action is query internal DB, it um it calls this like the approver approver function and then it checks if it's approved or denied and based on that it'll give the tool output and all these like

all these tool results like they are getting appended and they go back in the in the chat loop again you know um right here like tool results and the messages everything is like just appended to the same variable. Um and then the final is like reply to the user. And this is this is it. This is how BAML works. Very simple. Um and like when I'm putting

my query in the terminal, this is how it's coming in. Um I'm going to explain what temporal is. Like right now it's just running this piece. Um we'll talk about temporal. Let's let's do the same thing with pideantic AI. I don't want to type it out but um the same thing with pyantic AI looks like this. So I put my query in tool call limits is equal

to six here as well. Um it's calling the GPD 5.2 model. Right now it's asking me for permission. So I can say no. So it added denied query in total DB. So it's going to it's it's coming back and asking me for the output again. So uh sorry for the permission again. So I can say yes here but I want to I want to put no and

it'll keep doing that I think till like the sixth turn and then it'll respond to me cuz there's nothing else it can go in the prompt. It understands that the query was about the the company. So like it doesn't make sense to do web search. Maybe it'll try that. I'll keep trying that. I've messed up its like context already so much. Yeah, it's not even getting the

yes. So, this is this is the thing about like non-deterministic. I wanted to show you guys like this is what can happen when like I have a session tomorrow as well about evaluation like if you have only only tested it with like the good test cases the good flow cases it will not work for the other cases so you have to make your evaluation suite such that

it covers all the use cases but this is an example of pyantic AI um doing the same thing the same flow um and let me show you the code for pyantic AI Uh um I I'll show you the docs first. Like this is how I filled up the um the DB I was talking about the demo DB. This is all the information. So that's why it's like

it gives the topic. This is how like this SQL query gets written just like what's happening in the background. Um but this is the pyandic example. In the padic example again you have models like we had the types the BML types file. We have we have this like a citation should have this this this you know the final answer should look like this again the same thing

answer citations confidence um you have the run py file which is where like it's using temporal um I'm going to go over that later u this is the run agent piece um and what I miss the agent py file this is where the loop happens this is um this is the system prompt here so you are a research assistant answer the user's question blah blah these are

the these are the tools you have access to and this one this particular one needs human right um these are the strict rules that you have to follow um the build agent it builds the agent in this it's less pythonic I would say um because you're using the framework a lot of stuff is like hidden in there when you when you let me see web query tool

plane yeah when you define your agent you build the agent and you run this the loop is happening running in the background. Okay, this while loop is just to check if there are there are any tools which need permissions. So in in the BL case, we actually defined all of that in in Pyantic AI, it has a lot of support for that. So it has something called

deferred tool request and we're using that. So we're importing it in here deferred tool requests and deferred tool results. So if there are like two tools which need um human approval, it would get added to deferred tool results and it'll go over that in the for loop and ask for permission. That's all is happening in here. But basically the for loop is running inside this. Um very

simple to use cuz it takes care of all the complexities. Um so this is paidantic AI. Um and I have 11 more minutes so I want to move to temporal. Um I think temporal is necessary to learn because um think about it like if you build this agent for your company um it it might be the case like it asks for human approval and the h it's

it's it's Friday um evening and the human waits over the weekend and gives the approval on Monday. So the agent should be able to continue with the task on Monday. Why it should like why you want that is like maybe the task is like really long. it ran through like a lot of tools. It like cost you a lot of tokens. The tool calls might have been

like expensive too. And then if the human doesn't give approval in time, it's going to lose all of that. So what temporal does um it's a library out there. Um what it does is it has a DB and it's going to keep the output of each loop iteration and save that. And when if the permission comes out after a month or whatever, what it's going to do

is it's going to run the whole cycle again, but it's going to pick up the saved stuff from the DB instead of actually running the LM calls till till the point where it was asking for permission. It was waiting on something. So that's what temporal is. Why I wanted to compare this was like b and pantic are pretty similar. You can go through the web page. I

wouldn't have time for that. Um like I have compared like a lot of things between them. Um but the function calling tool calling how how all of that is happening. Um the structured output we saw that right um but temporal pyantic AI actually has good support for it. So if you look at the pyantic AI code I haven't written much but in bal I had to define

each action which needed saving in the DB. So um that's the only difference which again you can write it yourselves. Um but I'm going to I'm going to clear this up and run this with temporal and see like how it goes. I think I've saved the command for that too. Okay. So I'm going to run the bal example with temporal and again it's like the compare our

product launch performance with so internal DB search. So and I wanted to run that cuz I wanted to show you human human in the loop and I've just enabled human in the loop for the internal DB search. So right now it's asking me it's asking me for permission and I'm going to ignore that and I'll show you. So there's a dashboard as well in temporal which is

pretty cool. So I'm going to refresh that. It's going to show this is the task that's running. So I open this. Do you do you see this? Do you see how the graph is moving? So this is the timer. It it can run for like like it can run forever waiting for that um permission. Um this is all the stuff that has happened till now. Okay, it

knows what MacBook Air like the dashboard is amazing. Um, it this is the input that we gave in. It's waiting for the result. Okay, I'm going to go to my terminal and say yes. Let's go back. It got the approved DB query signal. It's doing the query DB activity. It went back to chat activity. Then it's doing the search web activity. It's doing some web searches as

well. Then again chat Then the search web activity. We can go over the stuff in the terminal It got the answer. This is the answer, the complete answer like how we want citation, source type, URL, doc ID and this is all the steps that it did. So this is in the dashboard and this is in the terminal like I'll have to scroll up but these are like

it went on for so long. So basically uh it was asking for permission from me. Um then it did a search web it ran a web search um with this query Q1 20 20 26 competitor blah blah it figured it has to do a web search uh internal DB is not enough then it did a web search again so we saw all of that and then gave

the final answer so this is the final answer so this is temporal um I can run the same thing in um using temporal with pyantic the same stuff um so you can run it in parallel. I'm going to ask the same query. Okay. So, uh it's asking me for permission, but let's go back to the dashboard. So, it's showing running for the Pantic Sorry, I'm just going

to Yeah. So, I'm So it has to like it asked me again cuz do you see it's a different topic. So I go and it's pretty cool the dashboard that they have. Um so this is about temporal. Uh and it gave the answer. Same thing here. Citations blah blah. Um, so I just have five more minutes and I want to I want to show you guys the

code once. Um, just to like compare the prompt. Okay. So um, so an agent.bl this is what the prompt looks like. So like an agent is not so complex. So you're just in the prompt you're explaining it all the actions um or the tools it has uh access to when you're supposed to run each one of them and then like like all the all the solutions that

I explained like maybe if you have like 50 tools you don't want to like give it access to all of them you know you will give it access to like three of them that's what a sub agent is like you would define it like if you're working with only git these are the three tools you should have access to otherwise like if you have 50 tools you

would use something called um progressive disclosure just give it access to like the tool name and like a oneline description and then it'll figure out like I need the argument the function call the API documentation for this particular tool to finish the task and it'll ask for all of that and put it in the context. So it's as simple as that. Um and I showed you the

pyantic example already. Um yeah uh going back to the presentation um if you want the full recording for the demo I didn't cover like a few things cuz it was pretty limited. just scan this um I'll send you a full recording for the demo um as well like covering like if you want to build agents these are like the two most popular frameworks out there I don't

know why people haven't heard of BL is amazing um I've seen companies do really well while using BAML um you can use Rust pretty good um with it and yeah um evaluation I'm going to cover that in my session tomorrow very important if you're building an agent um just to make it more reliable and like the scoring that you would cover. I'll cover all of this tomorrow.

Um this is really important. This is my website. Uh scan this, you can connect with me, ask me questions in this session. I cannot do a Q&A. Um the hall is too big. But like just message me on LinkedIn. Um and we can connect. Um yeah, that's that's it for the session. >> [music] >> Heat.