Context Management for Everyday Software Development - Ragunath Jawahar

This talk explores context management when using AI, specifically focusing on large language models (LLMs) and their token systems. The speaker discusses the different goals of users, such as optimizing token costs and achieving desired outputs from AI interactions. By explaining how LLMs operate and the principles of context management, the speaker aims to equip attendees with techniques to effectively manage context in their coding agents. Key topics include the importance of tokens, the concept of a context window, and strategies for effective communication with AI models. The session also covers latent space activation, the significance of input modalities, and various approaches to frame queries to achieve better results.

Before we begin the talk, I have a good understanding that some of you, when you come to a talk about context management when you're using AI, you probably have very different goals, right? Some of you would want to optimize for token costs. Like you would probably want to say, "How do I do more with consuming less number of tokens?" And some of you are like, "I don't

really worry about the tokens, but I'm not kind of getting the kind of results that I want to get." And these are very two different opposing goals, right? And when you are in a talk, I probably have to like sort of tell you either this talk is going to be about cost of optimization or it's going to be about getting the right information with the available context

window, right? And to make sure that everyone gets something out of this talk, we'll go talk about the underlying principles on how these large language models work. So that way when you understand how they work underneath, it becomes a lot more easier to work with the constraints that you have that may or may not be under your control, right? So if you're looking for cost optimization, maybe

probably there is something in there. If you're looking for like, you know, to get the best output when using LLMs, you probably are going to get some of these ideas. But the key takeaways are a lot of underlying principles that when you're using large language models along with your coding agents, and also a few techniques. And some of those techniques are probably things that you're doing already

at work through intuition. Here you would probably get some names, and I I hope you take some ideas that you can go to your office and use it on a Monday morning, right? So we'll start with the most important thing first, right? Okay. How many of you have watched this movie? Few. Okay. I'll give you the plot of the movie like based on how it looks like

you already know it's a rom-com. So, you can meet Lucy and you this is uh Henry. What happens is uh Henry and uh Lucy they meet at a restaurant and they hit it off like really well on the very first day. And next day Henry goes to the same restaurant. Lucy comes in. But then she does not recognize Henry at all. Like it's basically a romantic uh

comedy movie and um Gajini combined, right? Like that's what it is. So, Lucy comes to the restaurant next day, but her context is completely clear because she she suffers from this uh fictional amnesia where she forgets everything that happens after a certain day in her life. So, every day she wakes up she wakes up on October 30th. Like that's the story. And Henry sort of like wants

to like really likes her and wants to have build a life with her. So, he tries to make it work. So, the way how Henry does this is he records a video tape and then like based on what is happening in their life, he keeps updating the video tape and then puts her in puts it in her bedroom and it he labels it Good Morning Lucy. So,

every day when Lucy wakes up, the first thing she does is play the movie play the video and then watches what has happened and then like that's how she starts her day. And this keeps happening over and over again. And Henry's job is to sort of like keep this video tape updated, right? Henry signed up for it by choice, but like a lot of us we're still

making these video tapes, but we're like this is the reality that we live in, right? Like we're making video tapes for our agents and like trying to make sure that they remember whatever we're trying to do. But then you're not doing it every day. You're doing it with every new chart. That's what that's what is happening with us, right? Now, to get started with, this is what

the talk is going to focus on. Here's the rough agenda. So, first we're going to look at the foundations, then we'll talk a little bit about um uh latent space activation. This is something that you're probably doing intuitively. We'll also look at uh look at it in in a much more closer manner. Then you also look at input modality, um a bunch of other things, right? Like

failure modes and a bunch of takeaways, right? That's That's what we're going to do today. First thing is we're going to look at what is context management. Context management is basically curating information for your agent, right? You you decide what your agent is going to look at, when it is going to look at, and you're also going to be very careful about what are the things that

it's not going to look at. Like, that's the entire thing about this whole talk. The first thing is we have to talk about uh why this is important. Like, why do you think this skill is important? Right now, when Earlier, when you're working as a programmer, everything that the context was happening in your own mind, and you were like probably very fairly controlled in control of it.

And probably sometimes you would you'd delegate to your team members, right? But then whether you signed up for it or not, you already have a team member working with you every single day, and that that's your coding agent. So, this becomes one of the skill that actually distinguishes distinguishes your performance at work. Like, people who are really effective with using agents are also probably people who are

really effective with managing the context that they provide to their agent. The first thing that we're going to talk about is tokens. Like, that's a word that gets used a lot, but what is a token? A token, just like in human languages we have words, in large language models they they they just take tokens, right? And these tokens are units that are used for agents to run

their inferences on top of, and every language provider they have their own set of tokens. And these set of tokens actually form the vocabulary of the model, right? It's just like how uh English language vocabulary like has a set of words, also large language models have also have a vocabulary, and then they have a set of tokens and each of those tokens uh are actual representations of

numbers. Like these are just numbers that AI uses to sort of infer information from. And you will also know that like a lot of things in in terms of like since large language models are trained in the English language, most of the token size is about like 75% of like most English language words, and there are approximately four characters per token. Right? And every unit of measurement

when it comes to LLMs, let's say let's say for example how uh you're billed, how you Sorry, uh context window size is measured is all based out of tokens because language models themselves inherently only understand uh tokens. They don't understand uh languages as such. Now, if you look at this, this is uh the way how uh so when when uh a large language model is tokenizing any

phrase that it you give it, it could be a complete word, it could be a small portion of a word, and it could also be special characters. So, and they are all operating on a byte level when when when tokenization happens. So, for Latin languages, most of the characters are represented using one byte, but then for non-Latin characters, you typically have more like two to four uh

bytes representing each character. on a non-Latin language, a smaller text would actually take more tokens and more space. And if you look at it, in this case, English, such a simple phrase, and then it says uh we have four tokens and they're like 13 characters. But then if you look at uh Hindi for example, again, it says "Hello world" in Hindi, but then you have 12 characters

and seven tokens. Like it's much more than what we had in English. That's because the way how these individual characters are encoded at the byte level. That's why if your prompt is in a different language, like which is a non-Latin language, then it would consume more tokens. And if you take programming languages, if you take code, it is also going to consume a lot of tokens because

of the punctuations and special characters that we use as part of the code base. And that's why sometimes you fill up the context window much faster than you would probably expect to. And the other thing that you have to understand at this point, our models have no memory at all. Like they are completely state-stateless. So, managing state is up to the responsibility of like whoever is making

inference calls to the agent. If Sorry, to the large language model. If you If it is an agent, then you are in charge of sort of maintaining the state locally. If you are creating a chat application, then you're still responsible for maintaining that state yourself, like either on your If it is a desktop application locally or if you have a server, then you probably have to maintain

that state in the server. So, every single call to the large language model is completely stateless. So, you are the one who is actually passing out the entire history to the model. And basically, a context window is is just that, right? You have input and you also have the output. So, whenever you send responses to the agent, you sent the initial query that you sent, then the

agent will give you a certain response. And for the subsequent call, you're going to add your input plus the agent's output along with the next query, right? And it keeps growing and growing. That's why sessions become more and more expensive over a period of time because you're passing the same amount of information back to your agents back to your language models. And this is how a typical

This is a fictional inference call. Most of the providers, they adhere to open APIs schema, but this is just a What do you call it? This is just a sample inference call on what and how it looks like. You have a bunch of messages, then there is a system role, and system The system prompt actually primes the agent with some information, right? Like it In this case,

it's sort of asking the large language model to play a role, which is being a helpful geography tutor. And then the user is asking for the capital of India, and then the response would probably be New Delhi, right? It's It It It should be. Um, and then like for the next call, you're not only sending the system prompt, but then you're also sending uh, the user's message,

and also the assistant's message. And then again, you're appending it with the uh, new message from the user, and then you get Brazil, right? So, this is how your chat conversation keeps growing over a period of time. Like, if you if you're just like using a regular chat client, or if you're making your own API calls. Now, let's go look at what is in the context window

of your coding agent. If you look at the coding agent, you start probably with an ID or a command line coding agent, and to you, you probably it probably looks like there's nothing there in the context window at all. Like, there's just like a blank state where you can just start working, but that's not really true. If you look at the agent, the agent itself has a

long system prompt. And this system prompt probably varies from agent to agent, because every model provider every agent provider will try to squeeze the best performance of their agent, and the kind of system prompt would probably be you're an experienced software engineer, etc. etc. etc., right? So, they they have like a huge system prompt that tells the agent to perform a certain role. And I'll And the

agent also adds a lot of built-in tool definitions, because agents, in order to be helpful, they need to interact with your machine. And in order In order to interact with machine, they also need access to a bunch of tools, and those tools themselves have not only names, but then they also have tool descriptions. For example, if there is grep, then there is there is tool instructions on

how to use grep. If there is a way to edit files, then there is instructions on how to edit files. If there is a way to read files, then there are also instructions on how to read files, right? So, each and every tool has its its a name, its a description, and also a bunch of parameters that you have to look into. And then you also have

like the environment metadata, like your current working directory, operating system, Linux, because that also dictates what tools are available to your model on a specific platform. And if you are working on your own setup, and if you already have a project that has uh project instruction setup, then the contest window also has your agents.md, cloud.md, and like cursor rolls, whatever that is. If you have configured MCP

servers, then that information is also part of your uh context by default. And even though MCP servers sound fancy, but some MCP server also exposes tools. So, if an agent is using uh one of the MCP servers, the the capabilities available in an MCP server is no different from capabilities available in a tool definition. So, the a- agent really don't doesn't make any distinctions between tools available

through an MCP server or tools that is available inherently through the agent's harness itself. And now we have skills, so you probably have skills, and a lot of people are also experimenting with memory and preferences in the project, so that also gets becomes part of your context by default. And this is even before you type anything into your chart uh input, right? by default you have all

of that. And now when you start using your coding agent, then your messages for each and every turn that becomes part of your uh context window. Then you have files if you want to attach, you would add like markdown files, or even if you want to add image files or any other files that is relevant to the uh task currently at hand, that also becomes part of

your context, right? And then once you start having conversation with the agent, your agent also adds a lot more information back to the context window. So, for every tool that it calls, it's going to pass like keep retain information about the parameters it passed to the tool. And it's also going to pass like you know have information about the result of the tool. So, if you if

your agent is trying to read file /a.java, then that information is like the parameter name is going to be there. And in case if it reads the whole file, then the whole file becomes part of your You again like you know files it reads, shell commands, whatever it is, right? Like everything every information if you have additional sub config agents configured, then their description, metadata that is

also part of your context because the agent would need to know when to invoke a certain sub agent. Yeah. And then you have models that can reason. So, you have thinking models. responses from the agent. So, those also become part of your context window. So, a lot of things are part of the context window which normally is not visible to us. And this is how it looks

like. This is an approximate estimation of how the typical coding agent turn looks like. So, you have like system prompt taking some of the space and tool definitions. So, that itself is is somewhere around like 10 to 50k like depending on your MCP server configuration and all of that. And then you also have like prior conversation like a bunch of other stuff like the exact same thing

that we discussed. So, this is how your context window looks like in most cases. we also are looking for maximum context window sizes, right? Like how much how big of a context can we really have? And these are different models like you have Haiku which which has a 200k context window size. You also have Codex and Sonnet. They also have like varying window sizes like starting from

400k to a million tokens. And this is how if you switch model in your coding agent, this is the amount of headroom that is available for your subsequent task. So, a a model with a smaller context window probably will run out sooner, but just because you have a longer context window doesn't mean that you're going to be it's going to be beneficial to you, which we will

see in the subsequent slides. The other reason why you also want to look at tokens are like tokens are also very not very it's not very free. Like you know, like unless you're on a plan, if you're paying by usage, it can get like really really expensive. So, tokens are not free. So, we also have to know why and how we are putting the window limits to

good use. And this is also there like this is uh we don't have we do not have uh research papers backing this claim, but then this is what most practitioners feel. If you sort of like look at the context usage meter in your agent's IDE, you probably have to clear it like like you know, sort of like manage it at around this point. This is where it's

the most effective according to like practitioners is not a a claim that is backed by research, but I do agree like this is something that it's a help helpful heuristic to uh keep in mind. The next thing that we're going to look at is this, right? So, now we're going to look at like what are how does the context window size and how does context window utilization

affect uh your perform the performance that you're going to get out of your agent. And this is a research paper called lost in the middle. And the And if you look at the the at what they did was they started filling the context window and tested how much information they can retrieve back, right? And the models consistently showed either they remember the ones that are on the

beginning of the conversation or the ones that are at the at the end of the conversation. So, everything that is in the middle sort of gets lost or it's not easy to retrieve, right? So, when you have a longer context window, then the problem is like anything that is in the middle becomes hard to reference, right? But then this is not a bug, it could also be

a feature depending on how you tend to use the model itself. Like say for example, if you instead of looking at this as a bug, probably you should look at it as a constraint. And if you if you're looking at it as a constraint, then you can sort of like strategize and work with it, right? Like even make it work for you. And there are like a

few patterns that I will show that you can use to uh make use of this. So, the first thing that I want to call out is like you know whatever goes in the beginning of the conversation at the end and the end of the conversation matters a lot, right? And your coding agent probably doesn't go out of character because if you say to act like a specific

character or a person or a personality profile, it'll still remember even if the conversations are so long because that becomes part of your priming, right? Like you that is probably your you modify the system message or you send it as the first message in the conversation, right? And whatever you specify in the agent.md file which is your project instructions for the agent, those also are respected most

of the time because like that also becomes part of your initial uh edge which is like at the beginning of the context window. So, if you have this information in mind, then you can sort of like play around with this. Like you can say okay, information at the beginning of the context window and the end of the context window is the one that is like most reliable

at hand. And how do we make this work? There are three ways you can do this. uh you can take one task. So, you give okay, like this is the task that I'm going to do, but then you can you can do it at multiple places, right? Like that is one way to go about it. The other one is like you take one piece of information and

then constantly keep refining it. And the last one is to say okay, uh this is going to uh take many steps, but then I can just like keep expanding it because only the latest information is relevant to me. And here are some of the con- concrete examples. For [snorts] example, let's batch application which is like you take one task, but then you apply it to many targets.

If you have a bunch of controllers and you are trying to generate open API specifications for each of these methods inside these controllers, that is a good way to do it because like the first instruction is very clear that okay, go and apply open API specifications to all what do you call it? All functions in all the controllers, right? So, the agent can take that one specific

instruction and it can keep applying it and if you are sort of like specifying one file at a time, you can still keep doing it until you run out of context or like, you know, however longer you want to do it. Uh again, like these are again same very similar thing. So, you take the you fix you keep the fix you keep the task fixed, but then

you keep varying the targets. Like that is one pattern that you can use. The other pattern is sort of like iterative retire refinement. Probably, you want to send out an email draft. So, what you do is like you take that one email, but then you keep iterating on top of it. In that case, what happens is like only the latest thing matters. So, now you're making good

use of the the other edge of the context window, right? So, in this case also it works really really well. So, in this case we are making many iteration, but then the target remains fixed, right? So, the first thing we vary the task, keep the target different Sorry, we keep fixed task varying targets. The second thing is like vary the target, but keep the same task, right?

Because you constantly keeping improving the same uh piece of information. And the other one is also like progressive expansion many many steps. So, if you're working on a uh uh Sorry, uh a document and you're creating one section at a time. At this point, whatever is the latest information in the context window, only that matters, right? Like then then you're like only focusing on the latest edge,

which is like the other edge of the context So, even though the U curve looks like a bug, but it's not a bug. It's It's basically a constraint that you have to work with. And And oftentimes when you're looking at it as constraints, you can actually make it work for you. The other thing that I want to talk about is called latent space activation, right? This is

one of the interesting uh concepts. So, if you if you look at it, you have a huge amount of information, But then, not every part of the information is accessible. Like, there is directionality one, and then some part of the knowledge is access is located at some part of the knowledge graph, right? So, the knowledge space. So, in order to access some of this, you have to

use the use the right language to access those things, right? Say, for example, when you say uh we'll we'll go ahead. I'll show you how. Let's say, for example, uh you give it a prompt that that says, "Write a function to sort a list." Like, that is one prompt that you give to the agent, and then the agent has to find out a space where it has

to like sort of reconcile this information and then get it back to And you don't really know whether the output is going to be Python, JavaScript, like pseudo code, or whatever it is, right? Like, because it's completely uh Non-deterministic. So, it'll probably get something that sorts a list, right? But if you look at the next task, you're saying, "Write a Haskell function to sort a list." And

the difference between both the prompts is only just one word. Like, the moment you use Haskell, then you're going to get a solution that is purely functional, right? It's not going to use iteration like it because iteration is not functional in Haskell. So, it's going to use like uh what do you call it? Like, it's going to use a bunch of pipeline and then create uh that

functionality for you, right? And that one word >> I think we think for us >> You can ask now. Yeah. >> To go back to a state >> Yeah. >> You're talking about this between the task and the target. That's what you can flip into the thing that can explain the fake task. >> So, the So, when you say, for example, fix the task and vary the

target. >> Continue to drop the temperature. >> No, so the task is fixed in the in the sense >> I want to convert an image to video. >> Okay, that's that's a that's a fixed task, right? Like say, for example, you have five different images, and you want to create five different videos from each of those images. So, the task remains fixed, but then the targets keep

varying because you are applying that on individual images. >> Okay, now I get it. >> Right? >> Yes. >> But if you have one video, and then you want that video to be better, then that one single video is like a fixed thing, but then you keep giving it instructions so that like the video keeps improving. But the instruction on the video is the latest instruction. So,

which you're using the latest version of the video, and only using the bottom part of the Right? Yeah. Yeah, yeah. Uh we'll branch with and we'll do it later, right? Okay, so here, just by specifying one single word, like you know, you you're Now you're in the specific domain of Pascal. So, whatever output that you're getting going to get will be very specific to ask, right? And

here is an interesting interesting experiment that the Anthropic team did. So, since I mentioned that you can sort of like activate certain regions of the entire knowledge space, what Anthropic did was they clamped the activation space to a certain region about Golden Gate Bridge, right? So, regardless of whatever prompt that you gave, it's only going to be get activated in that region. When they did that, they

started asking different questions. Like they say say like, "How will you spend $10?" And the model would say, "You can take You can buy a ticket to visit the Golden Gate Bridge and you like spend $10, right? And if you ask it to write a story, and it would sort of like write a story that revolves around the Golden Gate Bridge. So, any question that you asked

would probably will always be in that region because uh by default the team activated this as an experiment to see what happens if a specific region is clamped inside the uh language model, right? So, the way how your the way how you manage context will actually activate different regions in this is again one another example on like how you can activate uh different regions. Like say, for

example, you give it a piece of code and then you say, "Perform a security review on this piece of code." Right? That is like a very generic thing and you really can't really predict what is going to come out of it. But if you say, "Review this code against the OWASP top 10," then that is going to be very specific because it's going to evaluate your piece

of code against those 10 different criteria from uh OWASP. Right? So, the the the language that you use uh matters a lot. And uh you can also frame your questions in a certain way. And oftentimes when we think about creativity, we always think about like varying the temperature. Like if you if you vary the temperature, then you're going to get more randomness and then it's going to

be more creative. But then, you can also achieve creativity by just like framing the problem in a different way. And in this case, if you say, "Okay, you're a hairdresser in Bangalore with 15 years running a salon salon and then build me a real estate listing tool for local agents, like real estate agents in Java." So, if you look at it, you're like activating multiple parts of

it. There is Bangalore, uh this person is a head hairdresser with 15 years of experience, they're running a salon, and then you're like asking them to work on real estate stuff, and asking them to build a application using uh Java for real estate agents. Right? Like there's a lot of stuff going on. And when you sort of do this, various portions of the knowledge space gets activated.

And the and and sort of like you get a very interesting combination of answers out of it. So, the way how you activate the space really is under your control. You You You have a lot more things under your control, but then how you put them together is at your hands. And the model can only always get things that are already in in the train data. Like

it cannot produce new stuff. Uh but the way how you frame things can actually make it produce new stuff. So, the LLM on its own doesn't really produce creative things, but then like you as a developer or like a person interacting with a large language model can frame your questions or your prompts in such a way that it it it can activate interesting parts of the space

and it can get give you interesting output results, right? So, the temperature is not really helpful. It is in some aspects, but uh uh framing is a much more powerful tool if you want to get creative results out of the You can also do other things like framing for access, Now, uh I I still remember this thing. I was trying to build a sample application using Swift

uh desktop. And I needed a three-panel thing. I still don't remember the name of the control panel, but it's very similar to what you would see in Xcode. Like if someone has used Xcode, then you would see like a left panel, then there's a inspector right panel, and there is like the code editor view, right? And even after trying for several hours, like I I think like

I was trying it for a couple of hours, it would like give me something visually similar. Like it would create its own third panel, like the inspector panel, but I was not able to get it at all because one the data set is so slow, and people who work on Xcode use very specific UI component terminologies to actually access that information. So, if you don't really have

domain expertise in that area, it becomes really hard in some cases to activate those things. And that's one of the reasons why people say like AI amplifies who you are because your domain knowledge is a lot more valuable because in order to access that information, you probably have to use the right You have to use the right terminologies. Um in this case again, like say for example,

instead of doing a security audit, you you're giving like, you know, security engineer very specific uh login endpoint on a multi-tenant SaaS. Again, like that is a very important word to use and then you're specifying, okay, like these are the three things that you have to focus, right? So, this becomes a lot more specific and it's also like, you know, going to help you access areas that

you don't know. Maybe you don't have the expertise to assess these things, but even if you have the domain knowledge, it becomes a lot more easier to access expertise that you don't normally have access to. And these are things that you're like, you know, knowledge and capabilities in large language models. Like, say for example, one of the things that people do is like if you are a

React engineer or like if you are a database engineer like a like, you know, a lot a lot of role-based prompts. And what those prompts does is basically it activates that portion of the latent space, right? And then domain priming is like, you know, and often times you don't even say you are a React engineer. If you sort of you say, I have too much re-rendering in

this area or in this component, how can I optimize it? That itself can activate and and and you're also going to give it your source code. The source code contains imports from React. And if you use the right terminologies along with the appropriate context, then that thing can get automatically activated. You don't even have to use a role-based personal persona in those cases. That's why if you

look at experts like people who are like really strong with certain technologies, they would just like write prompts because the language itself is so accurate it can activate the appropriate and access the appropriate parts of the knowledge base. And then few-shot examples is also an example for in-context training where you say, okay, like these are three different things like now replicate the same pattern. Like this these

are things that you're already doing and and by doing that you're also accessing specific parts of the language model. This is a bad idea. Like, don't do this. It's here, but we'll talk about this later and why you should not be looking for formatting things. And negative framing is also very very good, right? Because like when you say, these are all the things that my system should

do." and but also by specifying these are all the things my system should not do, you're also constrained give constraining the solution space and language models typically perform very well when they're inside like a reasonably constrained space, like not over constrained. But then if there are reasonably reasonable number of constraints, then the performance that that you get out of large language models also is is very good.

And this is what it is, right? Like because if the expertise expertise matters because you sort of like have the right language, the context and the jargons to access those regions in Now, this is more interesting because like everything that we talked about you can take that idea and if you're creating sub agents, what is more important is the the use of right language in your system

prompt of your subject sub agent because someone who's writing a sub agent who can who that can do security reviews can perform very differently based on the kind of prompts the kind of system prompts that you're using. So, you could go ahead and consult a security engineer because like they would be like if you have a team like where you have different kinds of expertises, when you're

creating an agent for a certain expertises, it is recommended to consult it with the person who's actually really good at it because the kind of sub agents that they will be able to create is a lot more helpful. It's it's not like gatekeeping. You can still do it, but then when you get help from someone who has like the expertises in areas that you don't have, the

kind of performance that you can get will be very very very different. And when we talk about there's this concept called concept density. Concept density is basically how much ideas and that you're able to convey when you are communicating in many in any kind of way and usually if you're typing the concept density drops so much and when you talk the concept density is much more higher,

right? And when you're also typing, you also are doing multi- you're technically multitasking cuz you're also editing and sort of like trying to curate what you're trying to uh fixing grammars, like and a lot of different stuff. So, it's typically very, very slow. But then when you're talking, um you're you your mind is free to think, right? It's again still multitasking, but then your mind is you

have a lot more bandwidth to think, and you also have a lot more bandwidth to say things uh uh to uh when you when you're speaking. Uh what voice does is if you're using a voice uh to text solution, I use something called as WhisperFlow. I don't type a lot anymore. you can convey a lot of information, and that information when it has lots of the when

the concept density is high, which is basically the information is uh rich in the kind of context that you're going to give to the large language models. If there is one thing that you would want to do, like if you want to say, "Okay, like there's only one thing I'm going to change, but I want like better output from my language model." Just switch to speech-to-text. Like

that's one easiest with that's the lowest and the lowest hanging fruit that'll get you the maximum amount of benefits when you're working with large language And the other thing is also besides voice, you can also use images. Images accurate when you're trying to communicate things. >> Very quick question. >> When we use voice, how's the token form? Does it create a basis on the bite of >>

Yes. No, not on the audio. I said speech-to-text. Speech-to-text, so it's still text. Yeah. So, uh image is it's a lot more certain things are easier to communicate using an image, right? like this is there is this popular saying that English is the next next programming language, like it's not. Like it it is close, but it's not. And some of sometimes uh language is not really sufficient

to communicate what you want to do, right? And there are like few cases where that falls flat. Let's say um he this is this is the first time I discovered it. I was trying to write a language parser. And then I did not get the kind of outputs that I was trying to get, right? Like the the code that the LLM produced would always leave out something.

Like say for example, I I'd say, "Okay, I'm going to give you a class and give me the list of functions within a class." Like that's the thing that I would try to do. And every time I looked at looked at the result, I would try to communicate in English. Like you know, "Why didn't you do that? Why didn't you do this?" right? And it was very

hard for me to do. Uh it was very hard for me to sort of communicate with the agent and it at to a point became very frustrating. So, what I did was I just took the source file and then uh like there is a conversion like the there is a online conversion tool for tree-sitter where you can parse a programming language and it'll give you the AST

format of it. And I would just like copy and paste the portion of the AST that I'm having problem with. And the agent was able to sort of find out what is going on. Agents uh uh large language models can understand and like they really work well with structured information and structured data like in whatever structure they are at. You don't have to be like as long

as there is structure in it like large language models do fairly really well. But they they cannot produce structured output without assistance of an external tool. Like say for example, you you it need it needs compiler errors to produce syntactically correct code even now, right? It has become better at it, but it still relies on those things. And this is one case where I said like, "Okay,

like for this domain, if I want to work with uh AST parsing and want to create applications that does AST parsing, the communication feedback channel should be AST uh representations in textual format." And here's one more example. If you're working with a UI library, if you're like working with React or some whatnot, and this is not even like a proper like it's it's not a proper I

don't know it looks like YAML but I'm not I'm not really sure if it's even YAML. There is a hierarchy and instead of saying, "Okay, go and look at the second card." or like you know, "Go and like change this thing on the status card." or whatever, you could just like give it a hierarchy and then say, "Okay, like this is what this is the desired output

that I want." Right? So, it's a it's a pseudo representation but this also works really well with um agents uh large language models. And this here's one more example. If you have like a large dependency graph and you're trying to break a dependency cycle using a coding agent and instead of like you saying communicating verbally, just give it a graph as diagram and then say, "Hey, this

is a cycle. Like why don't you break it?" Right? So, sometimes like that information is a lot more concise and a a lot more precise than natural language. So, you have to make sure when you're giving instructions, especially feedback during failures, try to find out what is the closest thing that is representative of the domain and then if a structured format pre-existing uh format exists, just like

hand it over to And uh this concept is called projections. This is an example. Projection is basically you take something in one form and then like try to create a representation in a different form. Like that's what it is, right? And like most of it is all In this case, let's say I want to program in a language that I don't really have any idea about. But

I am someone who can like sort of write good test. Like I can very say, "Okay, I'm someone who can verify the behavior of the system that I'm building." Like if that is the case, what you could do is you can sort of like say, "Okay, like this is a test matrix. These are the inputs." Now you can ask the agent to take this test matrix and

then convert it into test in a different language. Right? Like that could be any any language that you've not worked with in the past like Rust or Haskell or whatever. And the agent would pro- it will do it really well for you. And projection also works in two ways. Not only you take something and then convert it into code, but you can also take code and convert

it to something that you can understand. Like which which a lot of people already do. Like you say, "Okay, like show me the sequence diagram for this controller in my application." Right? So, projections are very powerful concept. Like once you get an an understanding, you'll like find out interesting ways to do this. And this doesn't have to be a test matrix. Right? It doesn't have to be

a test matrix. Like say, for example, if you want to write a parser in Rust. Like, you know, but then I don't really have experience with Rust. I could create a section in a markdown markdown document that says input is a source code, output is like a list of functions. Right? That is also a projection. And I could create 10 different projections, which is basically specification for

the desired behavior that you want. And you can feed it to the agent and the agent will do it for you. The way how projections are also powerful is they make reviews a lot more easier because you can specify you can verify the correctness even before the agent starts executing. And when you're reviewing, you just say have to review your document with your uh generated code. Right?

And if they match, then you probably know, even if you don't understand what's in between, you can be rest assured that it's correct. So, projections are very powerful concept. Like I'm going through that You can give an idea to our own projections. But that's an idea. It'll probably click if you if you practice it enough. Next thing we're going to talk about your chat thread control, like

how it goes out of control and uh what you can do. So, one of the things is when you have a large chat conversation going on and then if you want to take it forward. One of the things that you can do do is ask the agent itself to summarize what's happening and then give you a new prompt so that you can like sort of take that

and continue restart a conversation. Like that is one pattern. And then you use an initiated compaction. This is a This is the latest advancement in our coding agent with coding agents and large language models. It's basically you don't really have a lot of control over what gets compacted. It's a lossy compression. But then what it does it allows you to continue with the current thread. Right? And

and you can be very strategic about like when you want to compact your current conversation. So, keep a control of your uh thread. Uh there's also something called as fork and continue. This is one of the underrated Often times what happens is when an agent gives you an undesired output, You tend to correct the output. And when you try to do that, you have the incorrect output,

which is part of your question. And there's also a corrected output next to the the question itself, right? So, the agents often times get confused between the two because they're like very close and very similar to one another. And in cases where that happens, what you do is you actually edit the prompt that produced the undesired behavior, right? And when you do that, you're probably like sort

of discontinuing the thread and then starting a new thread. So, you don't have that stale piece of information that causes con- context confusion. So, forking and continuing is one pattern that helps you keep uh get control of your thread. Then delegating to a sub-agent. If you have a long conversation, then you don't want to pollute like if you're doing a side quest, for instance, you don't want

to pollute the current con- context with all this unnecessary information because that's just part it's a side quest. You can also delegate it to a sub-agent. The sub-agent gets enough information to continue with the task, and it also gets its own context window. So, anything that happens is actually discarded and it's not part of your main conversational thread. So, that is one more pattern that you can

And then there is an appending to a scratch pad. Like this is this is something like people do. You do something and then you have some new information. And let's say for example, you have a huge PRD that you're trying to implement and there are like five different sections to it. What you do is like you implement first section and then you sort of sort of mark

the progress to it. Uh scratch pad, right? And a scratch pad is something that gets discarded after you are completed with the task. But that's I wouldn't say hacky. That's like that's that's one way you improvise when you're trying to track progress with implementing a PRD, right? So, you keep writing to your scratchpad and whatnot. And once you're done with the thing, you typically discard the crash

scratchpad. Like you probably delete it from your system. And this is like very similar to scratchpad, but then like you keep track of important info information that you don't want to lose, so that becomes part of another file. But instead of discarding this file, this becomes a a record-keeping um thing in your uh codebase and it's part of your codebase. The other interesting thing is if you

have a large number of tasks that you're doing and and if you've done like five different tasks, you took a break or like you know, you come back to work tomorrow, you could also ask the agent to go look at the last five commits and say like, "Okay, this is what I I did yesterday and this is what I want to do." And that is also one

way to sort of like continue the conversation and add context to your agent. The other one is called Beats. I'm not really sure how many folks have heard of Beats is basically a it's by Steve Yegge. It's a issue tracker for coding agents, right? And it's part of your source control source control system so the version control system. They it uses JSON and LM SQLite and SQLite

is not really checked into your Git repository. You have JSON and L files where all the issues are tracked. And you can decompose a large task and then add them to Beats like and you don't have to do it. You just ask your agent to do it. And often times you sometimes look into the task, sometimes you manage the task. Often times the agent itself understands how

to use Beats and manage these tasks. So, if you have a PRD, then you can still break it down and instead of putting it into a file, you can put it into Beats. And often times you keep discovering new tasks or bugs or blockers. And Beats allows you to make that distinction. You could just like tell your agent to say, "Okay, like this is blocked by other

issue." And then like the agent will gladly mark it for you. If you want to use it through command line, you can still go ahead and use it through command line. And this is one way to sort of externalize the memory and like you know, continue running it. Uh it also it's also helpful when you're like trying to do parallelization. If you are have a task that

can be paralyzed, you can use beads to sort of like kick off five different tasks and the agent themselves will keep track of the the state of the task. Like they'll mark it in progress or completed or whatever exit. Yeah. Now we're going to talk about failure modes. So these are some of the common kind of failures that happen with large language models and I'm also going

to give you a little bit of recovery on how to do that. The first one is the distractor. Just because you have a large context window doesn't mean you have to like dump everything inside, right? Because when you have irrelevant information inside the context window, then these are called distractors because they're going to pull the attention of the large language model in many different directions and you

don't really want that. And often times if when you have what do you call it? Like with distractors, the only thing is if you are working with three files and two are irrelevant, you should probably discard those two and start a new conversation. So that's that's the only way you curate what context goes into your agent and then you start working with that. stale API hallucination. Like

this happens because your training data that the model's training data is like outdated. So if you're working with three, but then the training data is has like version two, then it doesn't really work. And depending on where this information is available, because some providers still provide MCP for their latest documentation. In that case, use an MCP if it is a local source file, then then point to

the local full source file. Or if you want to sort of distill the API and then like provide it as a make it just part of your code base. Well, there is no other way to handle these things right now. Or you could also do you can also write a linter that actually flags these things automatically for you. So you have a documentation and then the linter

flags these may API misuse from the That way the coding agent constantly keeps checking for lint violations and it can catch them also. Right? Like with that requires you to write a custom lint rule. With coding agents, it's not really hard to do that anymore. State recall. This is is also one of the interesting things. Like say for example, if you're working with an agent and you're

editing the file on your own or the agent edits the file and you ask it to hey, discard this part of the document for me. And for every latest response, you would see like the agent keeps dumping like, you know, or bringing back the old stuff over and over again. Like that is also another common failure mode. In this case, you just have to like clear the

thread and then like start a new conversation. You probably summarize, get enough context and then move to a different thread, but there's no way to rescue the thread at this point. There's also hallucination lock-in and a like all almost all coding agents, like all all language models, they're biased towards their own answers. So, what they do is they typically when they answer something, they're like so confident.

And if you don't correct it and keep continuing, they would like still keep bringing back the wrong answer over and over again. That is what it is. Um you can cross-verify with two different agents. Like like agents are like very agnostic against responses from different agents, right? So, you can actually pit one against each other. Uh it actually helps. There's also correction infused induced confusion. This is

another failure mode where you where the agent tells you something wrong and you correct it and then you go back and then just keeps doing the same thing over and over again, right? And this is context confusion because the information that is that you're looking for is so close to the place where the error happened. And again, like the recovery is to sort of like either summarize

or like, you know, start a new thread. And these things are a lot more easier if you catch it like or if it happens at the very beginning of the conversation because at that point, you just like clear and then start a new thread right away. Okay, this is a big one and a lot of time people say, "Okay, like generate this in a very specific format

or a very specific standard for me." And it never does that, right? Because agents like language models, it's too expensive to train language models to adhere to a syntax, right? And if you if you're someone who's used like try to generate mermaid diagrams inside any of these coding agents, often times sometimes they would fail with a syntax error because they really can't do this reliably well. And

the only solution for this is to don't ask agents to generate structured data for you without assistance. Like say for example, structured data in programming languages are all always verified by the compiler. So the compilers are the ones that catches these problems and that's why they're able to do generate structured information, but they're almost always not good at it. So either if you have a way to

correct structural or catch and uh failures through deterministic tools, do that. Otherwise, it's not a large language model. You should probably not be doing this at all. Yeah, and agents also do this in a certain way. Like say for example, agents are also trying to like sort of keep people happy. So they have their own context management techniques. Uh one is auto compaction uh which sort of

kicks off at various levels. I think for Anthropic, it's about like 95% is where it happens. Um sub-agents are also like interesting ways to sort of uh reuse prompts and also manage context and do side quests. Then then you also have like parallel agents. Like the if the if your coding agent can determine what is going if some a task can be paralyzed, it'll just like spin

off five different parallel agents and each agent gets his own Then we have at the final section. So this is uh a bonus section. One of the questions that often comes up with people saying, "Okay, we have a large code base. I don't know what to do. Um how do how do I do that?" So triangulation is a way how you can either kick off multiple threads

with the same agent or you can use multiple agents to ask a very specific question and you just basically take the common denominator across all the result and say, "Okay, all the agents are like talking about these three files." And maybe like this is these three files are the ones that are like really concerned really that's triangulation that you can do. But the best way to deal

with legacy code is always to go look into the code base, get yourself familiar with it. Like that's also like it's a harder way to do it. Like the lowest hanging fruit use voice. The hardest is to just like dig into the system and like really understand what it's doing both in terms of behavior and structure. And uh this is another bonus thing. Like I've not really

tried this plugin. It's called Caveman. And there's like a lot of noise on the internet. I haven't tried it. It basically talks like a caveman and it's mostly about cost optimization for tokenization. So it basically tells you like responds to you in a very basic like you know just uses uh a caveman language to tell what the agent is doing. So this is one way to optimize

output tokens and probably even if you can take talk like a caveman it it you can like but just make sure that doesn't go into your boardroom or like your conference rooms. Yeah. So these are the key takeaways from the talk. Um Make sure that you're always curating your context. Um framing is a very powerful technique to get interesting results out of the uh Uh you could

use uh strategic uh sorry structured uh formats to get like really good information or like the appropriate domain relevant information um when you're working with problems. Uh try to externalize uh the context using uh beads or like documents or whatever is available to you. Even Git is one way to creatively externalize uh state. And your expert like it's not it's not the large language model, right? It's

still the tool. So make sure that you're reading the foundations like you know reading a lot of theory and uh making sure that the vocabulary that you use in your own professions are like really rich because um amplify uh the kind of results that you can get from the AI. That's it. Like I can take any more >> [music]