Your Agent Is Ignoring You - Karrtik Iyer and Jem Elias

This talk delves into the complexities of ensuring instruction adherence in autonomous agents, particularly those powered by large language models. The speaker discusses the challenges faced when these agents operate in production environments, where they often execute tasks without adhering to user-defined constraints or instructions. Various input dimensions are analyzed, including the distinction between context and instruction inputs. The speaker reviews recent research efforts, examining methods like reinforcement learning and prompt engineering to improve compliance rates. The session culminates with a focus on practical solutions, emphasizing a pre-flight check approach to ensure that agents are set up for success before generating outputs.

I'm Karthik, he's Jim. But I would like to introduce Jim. Jim is my um chief advisor for most things in my life. For everything that I do, I kind of rely upon Jim. And if you people don't mind, I wanted to I kind of take a couple of minutes to kind of seek some advice from him. And maybe that could be kind of relevant as well. Jim

Jim is a recent has recently become father of second child, and I only want some kind of wanting to take some parenting advice from him. So Jim, one of the challenges I'm kind of stressed these days a lot. I think one of the things that I'm facing with my 8-year-old son is that I have kind of fundamental issues where say I I we want him to follow

certain instructions. We kind of tell him, "Okay, you're supposed to do these things in the morning. You're supposed to do these things in the afternoon." And we're kind of trying to also see trying to give this instruction in different ways. He says, "Yeah, yeah, I've done it." But when we check it, he's probably not done it. Or for the for the matter, we also tried giving him

cheats in his pockets. Morning one cheat, afternoon one cheat. But we seem to be very struggling, and we're kind of stressed as a family. Do you have any tips for us? I mean, your son is ignoring you. What do you mean ignoring? Just like your agent is ignoring you. That's what we are going to talk about. But essentially, that's the essence of our topic, right? That you

give a lot of instructions to your agents. And they ignore you. They don't follow that when you actually run them in production. I think it was fine when you were building it. But then you actually run it in production, you have real-life situations over there, and there your This is possibly good in some instances when you are working on a creative task. Example, I spend a lot

of my time prototyping in our with Cloud Code and recently Cloud decided. And there it's probably good because it will go ahead and create a lot more ideas for me to di- diversify my thinking. But then what is really hitting me bad is that it's not following certain standards that I asked you to follow. For example, something like your design guidelines or color scheme, etc. It's not

coming out, but the ideas are great. But in this particular situation, it's okay because it's not an agent I'm sitting with uh uh Cloud Code and uh making it uh do all these things. But when you do this in a long-running autonomous agent, that's when this problem bites you back because something which was created for natural language generation and maybe for a lot of fluency, we are

now using it for production situations which needs determinism baked in. And the agent is not able to do that. Now, how do you transform this and move from that territory of uh hopeful outcome to that of a measurable engineering constraint with your open domain agent. That that's essentially what we are trying to cover. And how do we make this uh more instructional ideally? Before we get into

that, uh I think it's good to dissect what the input that we typically give to an agent. Primarily, we can classify this into two parts, right? One is the input which is the context that you're providing. Data set that you're providing and the question the the query the final outcome that you want which is great. That's one part of your prompt. The other part part of your

prompt is the instruction which are constraints and rules that you are asking the agents The reason I want to make this classification is that we are focusing more on the instruction part now. Now, let's take a coding agent and you are asking to create a particular service. The intent, which is the service that you're trying to ask the agent to build, that's the input. And all the

context that you are giving that particular application or the service you're generating, that's all the input. And nowadays, models are pretty good and they are able to generate what's getting what's missing is the instruction. Uh you want to follow a particular coding style, you want a particular follow a particular way of architecting, that's probably not always coming out and that's what you see in production, right? It

you never see an agent failing to create a service, but it's failing to create the service in the way that you wanted it to create it. And that's what we are talking about as instruction. Now, the thing about large language models is that they don't differentiate between the two. There's no way you can say that this is my instruction and this is my input. You can very

well say that, but it goes into the model as a single flat to stream of tokens. There is nothing in the embedding space of the LLM that's going to differentiate between that instruction and the uh result is essentially that the model completes that data pattern, which is the token stream that you provided. Yeah. Should be one. So, hence, we are going to divide this into two parts.

On one dimension, you have this instruction adherence that we were talking about. And on the other dimension, you have the task completion. If an agent is not able to do any of this, it's neither able to complete the task nor follow the instruction, it's a failing agent, But most of our agents fall into to bottom right quadrant, which is where it's able to complete the task but

not follow the instruction like we were saying and this is what in this talk we are going to refer to as a rogue agent. By the way, Jim, have you heard about McNamara's fallacy? You should. Anyone here who has heard about Vietnam War. Anyone here? Vietnam War, yes, right? the the premise of that fallacy is that you measure what is easy to measure. Whatever is difficult to

measure, you say that it's okay not to measure Right? And then the third thing is that whatever is not getting measured is not so important. So, basically, what we are trying to differentiate is that probably in your say coding task or any task that you're taking which you want your agents to do, there is a difference between what you can measure easily versus what you find it

difficult to measure. But just because a task is getting completed doesn't mean that you have achieved the compliance or it has followed the instructions. So, in the every dashboard showed that the war is being won. How was it shown? By counting number of bodies, by counting how many missiles were uh bullets were burned, etc., etc. But strategically, the war was being lost. And that's the point that

we are trying to highlight out here that while you might be thinking that okay, my test cases are getting executed, objectively, I have I'm getting an agent to write code, I'm getting an agent to test code, objectively, everything is fine. But are you strategically winning the war? Right? And that's where the fallacy comes into picture and why it is relevant in today's world. We'll just wait for

Jim to get miked up. So, what we want to kind of basically address is that is there a reliable way how do we kind of we'll look at the research domain we'll see what is happening in today's research as such as to how do we even measure whether the instructions are being followed. What are the mechanics of measuring that? Then we'll move on to talk about okay,

why is it so difficult? What is the challenge in measuring it? We'll also then talk about what are the some of the potential solutions that we can look at. Some of the solutions which are trivial versus say some of the non-trivial solutions which might require GPU compute training etc. And then we'll finally look at what what do people do with people who are kind of working with

say black box models. Whether we face it or not, the reality of today's world is that not many people are directly hosting open source models, but they're mostly consuming APIs. Which is again obviously it gives you one kind of freedom, but you lose one kind of control as well. So, during this presentation you will see the entire journey of what would it take for us to move

this agent which is in the bottom right corner to probably the top right corner. That's the journey that we are going to look at. And during this course at any point in time if you have any questions, obviously we'll have time for questions at the end as well, but if you think there is anything relevant that you'd like to add or ask, please feel free to do

so. Cool. So, just to you know, reiterate that the problem statement maybe we'll start with the agent that we were running and how it was failing. Perfect. So, just to uh talk about like I think we spoke theoretically about different aspects of a prompt, right? Jim spoke about what is instructions, what is constraint. We wanted to show you here with an example. So, this is from one

of the um agent benchmark data sets called agent if. So, what you see at the bottom out here is your task. Then I'll try to read out for the uh people out here where we say that we want to create a login component, basically a JavaScript TypeScript application with some input fields, and with some UI components, and we want to kind of have some way to measure

success, failure, etc. So, this is the task which is being asked to perform to the agent, right? And what you see out there at the top is your typical instructions or your cloud.md file which you typically give or an agent.md file which you give, which typically has all your instruction which says, "Okay, these are the aspects that you need to consider. You need to consider core quality

and organization. May it should be small, focused." You talk about your component creation. And if Jim, if you can play the video a bit more. By the way, this is from lovable. This is a real prompt of lovable. It's not something we created. Yeah. We'll talk more about the data set, but what you are the real constraints which you want. And and what you will see is

that at quite a few places the constraints are put which are very very required in enterprise case where you want the state management to happen in certain way. You want it to be atomic. There is almost around 60 to 70 and in this case almost 150 constraints that you want to apply for the output that is generated by your agent. So, there is this instruction, there is

a task, and there are these constraints. With us so far as to what what are we talking about? And now we want to see what is the challenge first in getting these instructions being adhered to. And these are some of the examples that you see out here, and this is a This is as Jim said, this is from one of the wipe coding apps, uh which which

has been collected at the the lovable app, and where we have these instructions directly from there. Right? And this is the data that we'll take forward this this sample to kind of take our demo forward. Okay. Um So, first point that we want to talk about is, okay, what did what has happened in the research so far to measure the instruction compliance. I think a couple of

years back, the first research that happened which came out is called EFE Val, where they wanted to break down the instruction compliance to more atomic level measurements, which you can verify using code. You kind of verify saying that objectively define that the formatting should be say JSON or in a particular way. Word count should be something. So, you kind of try to have a binary yes uh

functions. Basically to say whether these instructions are being aligned. And this EFE Val took multiple models, did a first level check, which is a human level wipe check, and second one um which is as an LLM as a judge on top of it, they kind of did all these rule checks, which are kind of small small small instructions which they put for every task to see what

happens when an agent Oh, sorry, when a model. This is at the model level. This is not at the agent level. Still we have not moved on to the agentic solution. Still at an LLM level. And I hope you'll be able to appreciate that lot of agents capability also depends on the underneath LLM itself. So this if you else when it came out, it was also it

was only measuring how good is the LLM in itself in doing the instruction following, right? And that's what it was trying to do at a very very short instructions that we have to kind of a binary yes no kind of a functions. And that's what if you else did. Do you want to add something on to that? Maybe we'll also talk about what is a fine fine,

right? They took this if you if you else approach of doing a binary check on the instruction following capability and they applied it on a lot of lot of state of the art models which were out there at that point of time, maybe around GPT-4 time. And this became industry standard where they came up with two metrics. One is called the loose accuracy and the other one

is called strict accuracy. The loose accuracy is something like this. I'm asking the model to output the response always in the form of a JSON. If it came out if it gave a JSON, loose accuracy is a score. Yes, it did work. Strict accuracy is more more than that, right? It's like if on top of the JSON, if the model added some conversation filler. Here is your

JSON output. In that case, your strict accuracy fails because you told the model your output should only be a JSON. There shouldn't be a conversation filler. So that's the difference between uh loose accuracy and And what the researchers found is that some of the sort of models at that point of time could have a loose accuracy of up to 95% So GPT-5 actually scored around 95% on

this metric of loose But the strict adherence is typically has a you know reduction of around 10 to 15% points compared to that of a strict adherence, right? And this is actually good. I mean if your models are following your instructions of this sort in 95% of your cases, that's a very very good result to have and that's a great place to be in. 68% coming down

by 15% is okay. I mean that's something good to have. Even that is at the range of around 75 to 80%. But if this was the end of it, obviously you will not be having this talk today. But my topic I said one thing I want to point about about the if you want to search what that it was done on models, not on the actual cases.

Gentlemen, at this point I have a question for the audience and you. So if I were to say I I give you a space. Suppose this is a space and I say you cannot park your vehicles. Is it objective enough or not? What do you think? You know. You cannot park vehicles in this space. Can you measure that objectively? No. Okay. So say say vehicles are say

cars or say for that matter scooters or bicycles. Can we answer objectively yes no? Can they be parked here? How would you put that one up? No, right? Can it be parked yes or no? No. So then why is it not objective? So suppose I tell you the vehicles are car, scooter, etc. But what if if I told you the vehicles roller skaters uh handicap chair or

for that matter a child's tricycle. Is it objective? No No, my question is more simple simplistic. Can you objectively say given any instruction or a constraint, the constraint is you cannot park the verify it? Right. So, this this kind of points to a very important law which came out in around 1950s called Hart's law. Which says that for any instruction there is a almost area of certainty.

Like in this case, when you when we spoke about vehicles like car, any other car, there's a almost you're certain of the answer. But for every such rule, there is an area of uncertainty. Or an area of conundrum which you cannot decide. Now, that applies to our AI world and agentic world is also. So, now if you ask your agent to say make sure you follow solid

principle. Just correlate I'm trying to tell. For many people you might say well, what's wrong? My agent isn't doing the work I'm asking it to do. But is it very certain or is it in that uncertain area? So, for every instruction, there is this certainty region and there is uncertainty region. And that's what we are going to talk more about. So, coming back to this research kind

of proved that your sort of models are doing pretty good. May not be so much on user but pretty good on user adherence. But then what happened after that is in March 2025, there is this paper which came out which kind of revisited the reliability that came out from Efi Wainer's research. And what they did is they called created something called cousin prompts. Cousin prompts are nothing

but it's just a rephrasing of the prompt. The indent of the prompt remains the the but the words that are used or the order of the sequence of the words that he used just change. So, they are called And what they did is they took the same prompt, created 10 versions of it, and tried measuring across those 10, is your agent strictly following your infection as you

change the phrasing of the prompt and you move it around. And they called it reliable at K. K being uh high value, maybe somewhere in the 8 to 15 range. And what this uh did is and then what they results that they got is kind of in the graph which I have posted over there. What they see is a sudden drop in the reliability of the same

sort of models between if you all and what we did, it's called if you all plus plus, which is the reliability at top 10. So, for the model which was doing showing a 95% reliability when it when during the if you all research, this time suddenly came down to uh something like 60% 70%. if this is true, the major takeaway for us is what? Prompt engineering is

not as easy as simple as it seems to be. The way you chose to write the words is it's now pretty pretty uh probabilistic, right? You chose a certain way to write your prompt. You can write the same prompt with slight change in the word and you you can never ensure that this will work. Why? Because when you take this to production, when you write your prompts,

you will keep on tuning your prompt. You'll keep on changing your prompt. Now, you cannot take every combination of words or every way in that which that particular prompt can be written and figure out whether in which cases the agent working best. So, that's the challenge that this paper started throwing at. Let me tell you, you've given me a very good input from a parenting perspective. Okay?

Yesterday night my wife was telling me uh I told my son, I mean, she told my our son to do the same thing which I told the day before. He completely adhered to what I told. And my wife tried and it failed. So, she was like kind of complaining to me yesterday and I was like, "He always listens to you. But he does not I tell the

same thing to him. He does not listen to me." That's pretty much what your agent is doing your work here, right? Like now this becomes a chance of luck. It's becoming a chance of how you happen to write your problem and this definitely brings it down into reliability data. Questions at this point of that make it Very good. What kind of talking? What? We'll get to that.

We'll get there in a So, this is one paper that we want to discuss and before we go to the next point, I want to call out that both the papers that we talked about so far, they were done on models. They were not on done on agents. You took some instructions and you gave it to an LLM and tried and tried checking whether the LLM is

following that instruction. This is a point of time instructions to a model rather than to an agent. But now we are all building agents. You rarely build assistant kind of applications. You most of all are building agents. But the world of agents when it comes to instruction adherence is more challenging. Why? Any thoughts? It is that thought power. Okay. Perform actions. It's how we interpret something not

stated authentically. So, I think most of the conversations that you have with your agents are long turn long context conversations, right? Uh there are so many things that comes into picture. We are going through this whole uh hype of context engineering is the king kind of phase to now harness engineering etc., right? So, I think the primary challenge is now agent has to do compacting of context.

There is always this lost in the middle phenomena where some instructions at the top are followed, something at the bottom is followed, something in the middle is lost. Then it has to also figure out to retrieve the right memory to be put to use. You might have added something to the memory saying that these are my personal choices. Make sure when I am running the test to

use this particular by test. Or you might say I want to use UV as my package manager, right? So all this makes agent working with agents even more challenging and complicated, right? I think the agent has to An agent also keeps on learning from its own experiences. Lot of the agents are becoming smarter. Now it says, "Okay, last time you didn't like this. I'll probably try to

do this better next time." So there is this always this challenge in saying that, "Okay, if you look at Claude's documentation, Claude code's documentation, what they say is that what is put in claude.md is a soft instructions that you're putting. All that you put may or may not be taken by Claude as an input. Claude will decide which part of it might be relevant for a particular

task or not. It's almost like a soft thing. And then you might have 200 plus instructions or 500 plus instructions in your claude.md. Out of that, which one to pay attention to? Everything becomes much more complex while you're dealing with agents. And then you have tools that comes into picture, right? So all that through adds more challenge but challenges to it into it. Right. And there's a

paper which came out on this also, which is just the first paper which evaluated instruction adherence when it agents. And that paper was called Agent If. And this is this is, you know, high level what Agent If uh form. Yeah. So basically in this Agent If uh paper, what they did was they took almost uh more than 700 real-life agent agentic applications. From that, they created this

data set, which was like on an average, the system prompt had system prompt had more than 1700 words. Right? So it is more realistic what we were trying to make use of. The example that you saw at the start, right? The lovable prompt that you saw, that's also from this benchmark. So it was much more realistic from true real life agentic applications in terms of what is

the size of system prompt, especially the number of constraints to be followed, the number of tools to be used. Here we are not talking about couple of tools. We are talking about empowering the agent with more than 50 plus tools to be made use of. So all that made agentic agentif paper even more helpful. And what we found over there um that the signal to noise ratio

could be very high. Uh and the agent might be smart enough, but it might be failing because there is high signal to noise ratio as such. And overall because the context grows, the constraint grows, what model might prioritize may not always make sense for the end users. Right? And the multi-turn conversation again adds more like it does not I think one thing hopefully the audience will be

able to appreciate that how an agent performs on day one and what it performs on day 100 may not be the same. Would that Would that make sense to all of you? Like there is a drift that happens in the models of the agent's performance as well. It might be doing everything good on day But over at a period of time, the performance of the agent also

might deteriorate. So it is not good enough the instruction adherence on day one, but you also need to look at the instruction adherence on a regular basis. Right? And And maybe Jim, it's a good time to see the demo at this point in time. So we saw this video where we saw this plot.md and the instructions. Now we'll use that login form. And now we are making

use of one of the uh very um kind of good models for giving the same prompt. If you look at it, we have the cloud.md created as it is, and we are giving the same task like create the login form which I had showed you earlier, right? We are giving that. Here we have made use of the cloud Haiku model, which is supposed to be one of

the very good models for your regular coding task. As per cloud, it is it kind of most of your 80 90% of your coding task, Haiku is very very good, and we wanted to take take up such a model. So, you will see that we are given these instructions, and it very quickly builds up uh the login form component uh in hardly a few minutes. It will

be able to look at the cloud.md, and it will process the instructions, and it will try to do um all the things what we have asked it to do. And it says green tick mark to everything. And this is one of the things that I see is that it always claims early success. It will say that it has done its job, right? So, now we'll quickly show

the other part of it. While we think objectively that it is working, the server is up and running, login form is working, but if you were to run the compliance test now. And this is where we run the evaluation against the constraints. So, we If you remember, I had showed you almost so many constraints at the start, right? So, we are evaluating now what output was generated

has met the constraints or not. And we And the agent it uses a hybrid way of judging it. It uses a stronger LLM like Opus in this case, and a coding based check to check whether each constraint is met. And you will see that following atomic design principle, using React Query for service to server state, all those things kind of failed. While there are other things which

are kind of pass. Overall, if you look at at the bottom, um it you get around on 68% constraint success rate, CSR. So, that means out of 100 constraints that you give, it met around 68 constraints. And they overall, if you were to say the instruction success rate is measured over such multiple tasks, and you'll overall see that when you look at the Agentif paper, lot of

failures are there. Not many models are many agents are able to have the high ISR rate. So, what we are trying to highlight out here is that objectively your application is well, but it has missed out almost your 30 to 35% of your instruct constraints that you have put, which are your instructions, and which are very important when you're working in a domain which is highly regulatory,

which you're working in a case where you want best practices to be followed, some protocols to be followed, etc. etc. So, you cannot just objectively measure saying okay, I have my test cases, my test cases passed, so probably my program is working. There is a bigger gap that we are trying to highlight as Yes. But that is not a platform that can Yeah, we we are going

to highlight if you can just give us a minute. So, I know that GPT is going to be yes. Is there any Is maybe the element is taking it as always giving yes? No. Okay, yes. So, we we used another element. It's unbiased. Does it has not taken that into consideration. It's It's from the the same evaluation is from the Agentif. We have taken the same Agentif

evaluation only. To answer um our friend's first question first, even the SOTA models right now that are there, even Claude, if you look at the the if you look at it in the top table, the cloud four at that point in time when agent came 3.5 and four, it is at 65 to 70% ISR. Which is like 30% down as such. Obviously, we are not measured against

Opus, but the larger everybody for every task you cannot afford to make use of Opus as well. So, you would want to kind of always have this balance of where you want to kind of do normal programming task with say decent models and very complex task with kind of larger you cannot afford to make say every every call to Opus and kind of get your task done.

But, the point here is not about debating how good Opus is or how good IQ is, but largely in this agent if you when you look at the paper it just came back last year ago. Even the top tier models that you have have a compliance rate or the instruction compliance rate which is around in 70s. No, there it has nothing to do with new See, every

time you're giving the context as such. There is no new or old as such. This data has not been put to the training of the model in itself. This model has not been trained on this data. These are the unseen data that the model has never seen during the training phase. What we are trying to measure is after even giving instructions, why is the model not able

or the agent not able to follow the instructions? And we are not even only saying it's only model's fault. Remember, if you were the model agent eval is just not about the model. It is also about your agentic application. And that's what we are going to talk about in terms of solution. So, if you can hold on for some time, we'll talk about the Cool. I think

we have spent enough time on the problem statement. Now, let's let's uh talk about solutions. How do we prevent this, right? And we'll talk about three kind of solutions. One around how do you can you do steering? Essentially, we'll talk about both activation steering and attention steering. How can take an open source model and steer it to ensure that it's more instruction adherent. We'll also talk about

how you can do training, which is going to be more around uh reinforcement learning. How do you uh how do you train the model to do this? And then we'll also uh think about the more practical case of us using commercial LLMs today rather than open source LLM. So, treating the model as a black box, how can you what's one approach to improve the instruction adherence? So,

we'll talk about it in all three perspective. We'll look at it from the steering perspective first, which will also ask that answer that question around why why does the instruction adherence vary when you change the way the prompt is phrased? Right. So, that's from another paper. It's an interesting title. It says, "Do LLMs internally know when they follow instructions?" And the answer is yes, the LLMs do

know whether they are following a instruction or not. So, essentially uh without getting into the math of it, What this paper did is something pretty interesting. They took the embedding space that's getting created when you uh you know, putting your request into the LLM. When your prompt goes into the LLM, it's actually captured in an internal state of the LLM. They took that internal state of the

LLM and then they did a linear probe uh discovery, which is like a classification model, and tried to figure out is there an internal state in which the instruction will be followed? And is there an internal state in which the instruction will not be followed. Just to give you a little bit more highlight for people who are kind of looking for more depth. What they did is

like basic build a supervised learning algorithm. They took the E F L data set. E F L data set had for given an instruction, given a model which followed instruction, which did not follow the instruction. They used that as the supervised data set for training this linear probe. The only thing that changed in building this linear linear probe is nothing but a linear model like a logistic

regression kind of a model. What they did was instead of the input now is not the instruction, but they took the internal states in the model. Like when when they when the prompt go into the model, it is a numerical vector. By now everybody must know about our embedding and there is a numerical vector. So, they took those at every layer at different layers of say the

transformers first layer, nth layer, middle layer. They took those internal states. They passed that as an input to this linear model, and they tried to classify whether it will follow the instruction or not. And they already have that following instruction from the E F L as the annotation, right? And they tried to see how good is the model in predicting when it will follow versus when it

will not follow. And I'll probably explain So, this is again from the paper. So, they asked this is an example instruction or example task, right? Generate a warm-up plan for a 50-year-old. The in that is a task. And the instruction is do not include positions which require your knee to be involved. Because it's a 50-year-old person, don't put too much stress on her knee. got multiple responses.

So, the success case is something like this on the left side if doesn't include any knee involved exercise. But the failure case is something which is there like on the right side, which includes knee lifts. So, knee lifts is a knee involved exercise, and hence the model has uh failed that instruction. So, the success one they color coded it is green, the failure one is color color

coded in red, and then they and they use the linear probe, which is a SVM kind of classifier. What they found is that there is the linear separability between these two. That is essentially saying that you can separate out instructions which were followed as the green dots, and instructions which were not followed out as a red dot, and you are able to clearly say that there's a

that there is an easy classification between the two. Right? And what this means, people who are familiar with embeddings, this essentially means that there is a particular dimension in your latent space which predicts whether your instruction will be followed or not, which is essentially going to be perpendicular to the classifier line, right? That's the dimension which is able to tell you whether that instruction is going to

be followed or With me? So, again, summarize. These are your instructions. Your instructions can be classified with the help of a simple classifier, linearly separable, into instructions which are going to be adherent and which are not going to be adherent, and hence there is a latent dimension in the embedding state of the model, which is going to help you predict whether the model is going to follow

that particular instruction or Looks like a very trivial solution, but you know, if this this can be done, why not why not all models do it, right? Yeah, what that was your question. That was your question. It's very trivial solution, right? I mean, actually they went on and did something more. Like, what they did was like, okay, they found that you're able to separate these green and

red points from the IF-Eval dataset. What they said is, "Okay, I'll introduce this D vector, for the the points which are failing right instruction, I will add this extra vector D to the embedding and make it pass the instruction. And they were highly successful in they doing that. So they were able to take the if you will failures which they were able to separate and took those

failure examples and figure out that this internal state is able to clearly classify that to that internal state they applied they did some metrics magic added a D vector and were able to move that red part to the green zone. Now the larger question for all of us is like if it is so simple and why not all the models do it like it find out where

it is failing and why not do it right? And maybe that's where we'll kind of talk about the next slide. Think of a multi-dimensional space right because yet this is a representative image right like you can't represent anything more than 2D on this so it's it's a representative way and obviously it's a multi-dimensional vector. Think of a compressed like t-SNE kind of a uh kind of a

picture just for visualization purpose. Yeah we'll talk about it but more interestingly what what we found is that while you are able to do that right you are able to do that for known instructions. But for unknown instructions now instructions how many instructions possible? You can write instructions are almost infinite set For known instructions across tasks you are able to do that. Like for the example that

is over here the no capital letter instruction whether it is writing a story whether writing a code whether writing a poem very well but you cannot be doing it generically across all the instructions. So today any builders of any models etc. can you predict how many constraints in how many ways can you write? It's very difficult, right? So, you're not able to now then do it generically

where you can build such a vector which you can kind of add and move from say instruction not following So, for a known set of instructions, you can possibly do it, but it's it's a combinatorial problem. You would tend to think the instructions are pretty complex spaces as such. So, from that angle, it becomes challenging for anybody to do it generically to across all the instructions as

such. So, in other words, the dimension varies from one instruction to another, so it's not as simple as finding one dimension and adding it generically across all your instructions and your model will not suddenly become instruction ideal, but given an instruction, if you know the dimension, you can make it follow that instruction across multiple tasks. You have to make sure the question is that. So, can it

be equal to instruction that is not that does not that just told that Yeah, I mean, possibly yes. I mean, it it all depends Like But it's a huge difference also. The information already available like you develop a possible and that is the way it will be. When playing like large language model like that. Yeah, but if you look at it, the main success of large language

models comes from your pre-training phase which is unsupervised learning. Now, even if you say my all set of instructions that are present, somebody create a supervised set of instructions. No, so you have to classify Yeah, I mean, I was about to say that if you are in a domain where you have a fixed set of instructions, probably you can try doing this. Right? But all we are

trying to say is there is no guarantee that the even if an instruction changes, I think Jim was uttering the What was that cousin prompt that he spoke about. You make small changes to the No, it won't. words, and you might say technically it is the same, but still the output changes. So, that the challenge is still the probabilistic nature of the beast in itself. Well, I

just thought so. It's a hypothesis. These are all active research areas. These are not production-ready solutions, and we will we'll talk about it. As I said, I think the large problem with us is with more AI coming in, we are wanting AI to do the work. We humans having to curate the data, having to build everything is largely still a big challenge as such. So it all

depends on the quality and volume of data we are able to gather as it. But theoretically, I would say yes, possible. The other interesting thing that came out from this paper is that when they analyze this sound, right? Like on different parts of the prompt, that is taking the first few tokens, taking the request plus response, or the taking the request and response combined together, what they

found is that what influences this this dimension essentially is the prompt part And when you change the way where your way your prompt is phrased, this dimension changes. Or rather, the ability to align with this dimension changes. And this answers the initial part of the puzzle that we were talking about, which is why does the prompt facing phrasing you know, make the model less instruction-oriented. And essentially,

the reason is because because when the prompt is phrased differently, this dimension changes. And when you tell when you give a prompt to your model, now you are essentially trying to align it to this dimension rather than telling the model what to do. You have you can think about if you can think about it that way. You're just by giving the prompt, you're trying to get the

model to align in a particular dimension rather than anything. That's That's why essentially the phrasing affected things. But yeah, that's the first solution that we are going to talk about, which is essentially that upper latent push, which is like an activation There's one more thing that we can do here, which is called attention So, uh think of it like um think of it that you are reading

a book and you have a highlighter with you, right? If you can highlight what are the instructions which are important that you want your agent to follow. So, this paper uses a mechanism where they don't train the model. They don't do anything, but still it's an open-source model. What they do in this they they append a special token, so manually annotate the instructions and add a special

token before every instruction. Say start S and end S. Add special tokens before the instructions. And now, at every layer of the transformer, which is inside your LLM, they check what is the attention weight given to the instruction tokens. So, you have a start and end token, so between all that you have the entire space of the instruction. So, for all the tokens in the instructions, how

much attention weight is given? And if they check if that attention weight is below a threshold, say they decide a threshold, say it has to be 0.5 or 0.6, what they do, they bump it up. They boost it. So, if you if if people who understand the technology part, they have a attention score, the softmax layer which is there where they kind of add the logits overall

to sums to make it sum one. What they try to do is that for the instruction part, they try to compensate and boost the score of the instruction part without touching the training part, and they overall reduce for the other parts. And what they found over here is that by doing that, the instruction following or the instruction adherence became better. So, they are still not touch the

training part of it. They still need an open source model. They took the outputs at every layer, figured out the tokens which are part of instruction, and they bumped the So, in a way they're highlighting the areas which the model should pay more attention to. So, that's what they did. But, anybody can think of what's the downside to this? The model might become biased. So, if they

try to get to the the cliffhanger. See, one of the fundamental thing why for compared to an RNN transformer was the parallel compute it can do the KV cache optimization that you get, the fast mechanism that you get to generate tokens. Now, if at every layer, if you're going to recompute the attention weights, your KV cache is broken. There is no concept of cache. recompute the weights

time. And that kind of then brings in scalability issues. But, still again, this is a research direction. Obviously, there might be new things coming, but this is one of the areas where people are investigating that if we can steer, this is called steering mechanism, where we try to focus, kind [snorts] of put a focus to the model, okay, try to pay more attention to these particular tokens.

And that's one of this technique where it is trying to improve the attention scores. This is called attention steering as such. You talked about signal to noise ratio earlier. So, both these steering mechanism, what they're trying to do is they are trying to focus you onto the signal as in all the noise which is a Good. We'll move forward in the interest of time. So, that steering,

we'll talk about one solution from a training perspective of also if you if you are training your model, or what what can you do to improve its attention weights? Yeah. So, everybody must have heard about the attention is all you need paper. Similarly, we recently we all had a paper called instructions is all you need. Guess what is that paper about? That's based on reinforcement learning. So,

typically in reinforcement learning you have a reward signal, right? Typically human gives a thumbs up or a thumbs down. For coding problems, it is very easy to add reward because you write the test cases and if test cases pass, the reward is positive. Test cases fail, reward is negative. What are they now trying to do? Taking the same reinforcement learning approach, they are trying to take in

this very paper, not the instructions only. In the very paper, what they are trying to do is take simple, okay? Can I use a hybrid approach? Obviously a code, but also I use the another reasoning And I use both of them to decide whether followed and use that as a guiding mechanism to train your model. So, the reward signal is now much stronger because of this hybrid

mechanism. One is obviously using the code, the objectively measurable things, but there are things which are not objectively measurable. Like you want it to apply some strategy pattern. How do you objectively decide whether the strategy pattern has been applied or not? So, they use a hybrid mechanism. In instructions in all you need, they do a similar thing. What they do? They say, "I don't want any external

rewards." Instead, what they do I What they What they have done, they have taken the instructions, they have take decomposed the instructions into rewards. So, they have What they have taken a bigger instruction, broken it down into smaller instruction, and they said, "I will use instruction in itself as a reward for training." And that can act as a signal to the model to be more adhering. So,

these are training based techniques. Now, all of you might wonder these people are talking about open source models, training techniques, but we are not using any of them. You are using API based models. So, then what are we left with? Are we then saying that it's all lost for us? next. All right. Let's look ahead. The The last thing we talked is there something still done then

we still doing the something other than what's already marketed. It's still not market ready but there are multiple under the umbrella of reinforcement learning as a reward. But in that trying to use another model but it is still not a production grade asset because still there are challenges but deep sequence the first one to explore that. And then many models which are getting explored. You will get

some models which are available right now. But I cannot think of any production I mean very well tested production grade that is that is available as such. There's certain variants of yesterday that's not available in the marketplace at the instruct. You got That's not that good. Hi. That that Yeah, so Those are the instruct models. They are not these models. The instruct models are chat models basically.

So but these are different models which between me I mean We we'll talk maybe after the session ends and I can give you a bit more idea around that. One thing I want to call out almost all the research papers we are talking about here except if you all maybe everything came out maybe in March February 2026 which are like two three months ago. So none of

this is like production grade hour. Idea is to bring some of those recent research into light so that our people can investigate Even go to the last part. Yeah, we have fully seven minutes and hopefully we can do justice. Again, this is an active research area. I'm an AI researcher. I and Jim do active research in this area. This is also So largely the question that we

started off saying that okay then what do we do with cases where we have black box models. You do not have access to internal states etc. right? The overall TLDR of this whole approach is that okay, can I intercept the call to before the call goes to the LLM. First thing is we have to start measuring whether the instructions which we give even did it go to

the LLM or not. Did the agent consider the instructions which were there which were sent to the LLM? So that's the first part of the thing. You see, you have given a big checklist. Did the agent consider those instructions to be sent to your model? So the first thing that we are considering is if we introduce a proxy layer and if we can intercept the call so

like I can light LLM. People might have heard here about light LLM which you can act as a proxy. Can we have a hook before LLM? There now you have the entire system prompt plus all the magic that has happened in the agent and now you bad raw instructions which are being passed to the LLM. So first thing what we are trying to do out there is

we need some kind of a gold truth data set for ourselves. Like if we are wanting to measure if our instructions are there or not, first we need a human curated list of examples. Like for this task this instruction should be present. Once we have that, we want to measure in some way evaluate whether those instructions are picked up by the agent or not. And if we

find the gap so there is a gap between the instructions which we expect to be there and which the agent picked up, can we reinject and build a better prompt. So that is the first So we are kind of we do not want to end up spending the tokens and generate all the code everything and then realize that my instructions were not actually being sent even to

the LLM. So that is the first part of thing do where we are trying to evaluate whether the instruction is present. Are are the mandatory instructions that are present? The second trick over here for people who are working with cloud code etc. is that there is a thing called hooks in cloud code. So whichever you want to enforce, right? You better write it as hooks instead of

putting that in cloud. MD file. Right? Um so that is the first thing that we are trying to do. The second thing what we are trying to do is like we asked ourselves if I am working with a team of 50 people, right? I am the principal and there are multiple people agents, right? They are going to be producing a lot of work. How can I make

sure that I'll not be in a position to verify all the work that they produce? It's impossible for me to review all the work. So what do I typically do? What do I typically do? I would I went back to basics or we went back to basics and said, "Okay, can we ask them to create an outline or a sketch?" So in many cases, here's a typical

nothing new. We asked them to create a class diagram if you're writing to code or come up with a pseudo code, right? So that you're not generating the entire end to end. You get the pseudo code out. So once you find that your instructions are present, you ask the LLM to do a skeleton creation. That skeleton could be just the high-level pseudo code or for that the

classes etc. instead of spending all the tokens in generating the code. Once that is out, now you need to do alignment check whether the skeleton which came out and the instructions which were given, based on that that is getting aligned or not. And there also we use a hybrid approach of whatever is very verifiable using code, we do code-based check and then we use another reasoning model

to verify it is that to check or not. And if not, then we again send it back to the LLM. So we spent some money, some tokens, not complete tokens yet. But we again So we added two feedbacks loop. First feedback loop if before it went to the LLM, the second one, partial we spend tokens, we again pass it to the LLM. Once we are satisfied it,

then we ask it to generate the code. So, that is something that we are attempting. It's still under active research, but it is an idea that we are exploring given what is there in our control. Obviously, with Claude, we do not have access to logits, etc. But for some models like GPTs of the world, we have access to logit scores, etc. We could do better with it.

But obviously, we wanted to build something which is agnostic of logits, etc. So, this is a theory that we are trying to test. Again, you can poke holes in it. It is far from perfect, but this is something that we are working towards to see how we can make this happen. And if you can just move to the last slide, Jim. The same thing that I explained,

we are doing a pre-flight check, and we skeleton validation. In the skeleton validation, we'll do an AST-based check, architectural fitness functions. We'll have lot of code best checks, and reasoning model check, and then kind of catch the violation instead of spending the entire time on all the tokens and asking and then kind of realizing that it is not meeting the instructions that we are expecting it to

produce. So, that's That's something that we are attempting. Any thoughts or any questions? Uh yeah, I know production engineering is my aspect to the reliability and the data quality to the accuracy of it. Yeah? Cool. I'll probably take 2 minutes and summarize. Uh thing from research, it's very clear more prompting is probably not going to help you improve your reliability on Give us a minute. Uh so,

we given three kind of talked about three kind of solutions depending on how involved are you in the in the AI research space. Uh if you want to try something like steering activation steering attention steering is the way in which the research is going. A lot of techniques evolving around the training space also like things like instruction is all you need or the latest that we talked

about. And if you are approaching this as an AI engineer who wants to treat the model as a black box, uh these are some of our thoughts on how we can effectively end up doing this. Yep. We're almost done. >> [music]