Lessons from Building Deep Research Agents in Production - Sarang Kulkarni

This talk discusses the development and implementation of an agentic retrieval-augmented generation (RAG) application designed to enhance the drug discovery process. The speaker, a principal engineer from Thoughtworks, shares insights gained over three years while working with a client in the pharmaceutical industry. He explains how RAG can help researchers efficiently access and synthesize vast amounts of data, ultimately reducing the time and cost associated with bringing new drugs to market. The session emphasizes the importance of understanding both structured and unstructured data, and introduces concepts like clarity in user interactions, context engineering, and the iterative nature of research-driven AI systems. The speaker also shares lessons learned from deploying this technology, including the significance of comprehensive data retrieval and the necessity of thoughtful engineering practices to optimize performance.

Welcome everyone. So, uh before I we start, like quick view of the room. How many of you know RAG? Lovely. Okay, how many of you know agentic RAG? Such a good audience. Okay. Now, I want to understand uh quickly uh if if we have anyone here or like how many people here are actively building RAG applications or actively building AI applications? Nice. Okay. All right. So, uh

I think uh I'm I'm principal engineer uh at Thoughtworks. So, actually agentic RAG is kind of a very close to my heart because I think we work on uh this thing for a very very long time, like three around three-ish years with a client. So, what I'm going to talk about today is essentially based on the actual experience done at production and then kind of delivered. It's

living with users and people are actually getting benefit out of it. Uh so, it's kind of a story that I would like to talk about. So, uh one more thing before we start, I also teach at O'Reilly uh media I kind of take a course around like RAG, agentic RAG, how do you think about uh productionizing your your RAG agents and stuff. So, uh the next session

will be in June. Maybe if you can join in. So, how the agenda is going to look like This is a story, right? So, let's talk about we'll start with the why, like why does it matter? Right? Like what we are doing. Then we'll talk about like what. Uh and then what will basically tell us like, okay, what is it that we have built? And this kind

of gives us a really good context to kind of talk about the how, which is like the most important part of this talk. So, the how will be like the we'll spend a lot of time on the how. if I want to rename this talk today, I would rename it as how [snorts] do we build long-running agents? Or how do we think about harness engineering? So I

think we'll do a bit more deep dive into like this specific area as well. All right? Cool. So let's talk about the why. And uh how many of you know here about the drug discovery? So drug discovery is essentially like somebody like you you have a disease, a cancer, and you want to invent a new drug for cancer. Right? So I think it's like a pretty uh

maybe maybe the best use of AI apart from taking off our jobs. So I think this could be like one area where I would say AI has extreme AI has a positive impact across every but this is specifically area where we see a lot of help in terms of improving our health. Right? So this is actually in the drug discovery domain that we are working on. uh

if you look at the drug discovery process, it looks like this. So essentially um you know, how much time it takes for one drug to to be discovered uh and kind of getting getting into the market? Any guesses? Sorry? It'd be near 10 to 15 years. I mean, that's like a lot of time. Right? And if you could use AI to actually bring that down from let's

say 15 to 13 years, 10 to 8 years, I think that's like a huge huge benefit, right? And uh so we are basically working in the preclinical side of things. Doesn't matter for this talk, but essentially like when a client reached out us, they said that they want and basically like if you look at it, right? Like the the the funnel that you're seeing the funnel is

actually there are like thousands of studies going on to kind of come up with one drug, right? So, you basically start with tens of thousands of compounds and then you basically, you know, they do so so many experiments, studies on those compounds. And then they kind of reduce it down to a few hundred. And what they do is they start testing on animals like rat, rabbit, right?

And then they figure out like what are the toxic effect of that, what are the efficacy, how is the efficacy of that particular compound on those animals. And then very drugs get shortlisted, very few compounds get shortlisted which they do test on human. And so, you're talking about so many experiments, so many studies in this. a lot of them. what our client reached out and said is,

like we have so many of these experiments and studies, we don't know how to get that data. We don't know how to find that data. Can you build a Google for us? And this is like obviously Google is there because we started working on this before ChatGPT came in picture. So, it was a search problem for us. And what we did is like and the the problem

was that if they build a Google for them, they will we will be able to avoid a lot of duplicate efforts that will save lives and cost. Now, to give you a perspective of what we're talking about. One study, we have thousands of studies for one drug. One study just done for 1 month on monkeys cost about a million. Can you imagine? Just one study and we're

talking about thousands of studies in the course of this time. And it's not about It's not about just money, you know, it's about lives. So, if let's say one study was done on monkeys, you have to sacrifice about 10 to 12 monkeys. If you're all dog lovers, like you have 15 plus dogs just for that one study, And it's totally possible that study was done in the

past either in your company or outside. you could not find that data. So, if you look at the overall cost, it's like 2.6 billion dollars average cost to get it to the market. But right? But only 50% of those studies were actually based on like you basically they they only 50% studies start without a systemic review of prior knowledge. So, imagine that there were a lot of

studies done in the past, but now you are actually going to do the same study. So, you burn lives, you burn What if you are able to find it at the right time, right? So, for us the biggest challenge is like how do we get that data? And the right way to do science is to stand on the shoulders of like what was done in the past,

right? how do we actually make sure that all of that data that was in the past is actually discoverable, is we are able to find the data at the right time, The challenge is that this knowledge exist. So, whenever currently like whenever we actually have a researcher and what researcher usually wants to do is researcher wants to come and say that, you know what? What do we

know about a particular compound X? And that information is there. It's definitely there. And what it's like in so many different data sources locked in, right? And this would be the same case not just for pharma R&D, but it could be the case with also like other organizations, right? Like you have that data, but and you want the access to the data, but you don't have it,

right? And this is actually a So, when a researcher wants access to something that expands across all of these domains, because if I want to understand how this compound is doing, what do we know about this compound? You just don't want a toxic effect, you want like how efficient it is like and then a bunch of other things, right? So, uh the the problem here that we

wanted to to solve was then basically like making sure that a researcher is able to find that data at the right time. Right? So, let's talk about like talk about the what data we talking about, right? We talk about two types of data. Structured, unstructured. Right? And then this could be like similar across the domains. we need to understand that we have let's say so much of

data that is going to be a lot of complexity in being able to access that data. um what we did was essentially like we have that data, we want to make it available, what do you do? What is the first thing that you think when you have let's say I have many PDFs, I want to actually make access to that PDF. I want to be able to

ask questions to that PDF. What do you do? Rag, right? So, we did that. So, basically we What did we do? We basically built a chatbot. We had a give a text box and then they can come and ask questions, yeah? And uh this was essentially simple rag. So, we were able to kind of get answers really quickly. Rag solved that problem for us. the moment box

to the user, they are not going to ask simple questions, right? You tell them only you can only ask simple question, doesn't work. They'll start asking slightly different questions. So, now this question is basically not just on one study, but can they combine multiple studies. Now, if I have so many studies combined, simple rag can't work. You need to actually make it work with You need a

slightly more complex if you have a more complex questions, you will agent to be in place. So, you'll have some sort of a agent to drag, right? But, if you ask this question uh like if you again you're giving a text box to and then the user would say, "You know I want to ask this." and that's a big spec. And that spec is essentially like uh

they they say, "You know what? Actually, we we like we come here and ask this to do some stuff. And I want I'm actually that do for me is I want to write things. I want to write reports based on the experiments. And basically, I want to feed this experiment data and write reports. One report is hundreds of pages. Can you help me write that? Right? And

now to be able to write that report, I need to have access to so many studies across various domains. One study is say thousands of pages, right? And you have like access to you need you need so many uh studies here. And that's where we kind of think about, you know, we need agentic rag plus plus, which is kind of a deep research, right? Kind of it

goes in that direction. So, it's kind of a uh you can think about like a this is like if you will go from like left to right, the complexity kind of keeps increasing over a period of time. And I think if you want to start building any application which is in this domain, like be it rag, agentic rag, or you want to be in deep research, I

think this is a common sensical path. This is a sensible default. Maybe get your rag sorted first, then attempt agentic rag, and then think about how you want to evolve it further. So, it's almost like an evolution, And this is basically what we will talk about today. So, the most important part, How How of you used a deep research systems? Yeah, I like it. Any You folks

have used from chat GPT Gemini? Yeah? What what do you do it for mostly for? Why? Why I mean I don't get it. Huh? Multi-hop reasoning. Multi-hop reasoning. Multi-hop reasoning, yes. But you want basically like some some detailed research, right? Or a detailed research report on certain And when you want like a super detailed research, how much time it takes to kind of get back report? Can

you Can you get that in the seconds? Up to 5 minutes. Up to 5 minutes. So, if you want to build this system, you need to think about like how how that system can actually sustain for longer time. Right? Uh what things can you think So, when you are actually going and doing a lot of research, what all engineering terms come to your mind? First example is

prompt engineering. What else? Context engineering. Everyone is talking about context engineering today. What is the third engineering? Audio. Hardness engineering, yeah. The third term is essentially hardness So, basically we'll talk about like this hardness. So, going back to like if we'll start with the overview of like how this research or deep research system looks like, but then we kind of go back and go one by one

and figure out a bit more in depth about how to approach it. uh So, usually in any deep research system, you have three loops. The first is clarification loop. Because it's a long-running process, you want to really clarify what the user wants. The last thing you want is generate a report that user didn't ask for. Because it's like super super intensive in terms of cost and money

and what and time. So it's like basically like you have some sort of clarification loop first. Then you have some sort of a research Right? This is a research loop where we get more information, get the data, And then the last loop is essentially the writing loop. This is where you actually write the nice report where you have uh proper sections defined, very well-written uh report with

citations and all of that, right? So all the nice things. These are like a very high-level overview, but if I want to ask you if you want to take up what is the most foundational circle deep precise system, the most without which this system is not going to work. I think it's the loop. Research? Yeah, but Clarification? Yeah, but if I even if I do clarification I

need to do research. Uh because and the the the the common denominator here is like essential research, right? So let's talk talk about that first. Let's get that out of the bag. A simple research agent looks like this. You have a agent and the agent has a tool called it. What agent is it called? react agent, right? Let's say you have a simple react agent. Now our

agent has access to two tool calls. We talked One is structured, unstructured. If it is unstructured data, you do rag. Structured data, something like text to SQL, maybe something uh you go uh text to cipher, maybe uh knowledge graph and But yeah, this is like a high-level what a research simple researcher looks like, Now uh Oops. Yeah. Now let's think about context, right? We need need to

understand like in this in terms of context because we're talking about long-running work, long-running task that actually goes and researches across thousands of studies and and curated a report. That's a lot of data. And then that data cannot fit in the context window of let's say current models have like let's say 200,000 token context window, right? It won't fit in here. So, what all things do you

think will go in the researcher? Like if I want to build a simple researcher, what all things should be there? If I want to build the agent, what all things would you put in? Guardrails comes in, but like what is the first thing that you have add to a agent? Yeah, instructions. Where do you add instructions? System prompt, right? So, first you start with system prompt. Why

it is not working? Yeah. System prompt. So, this is a researcher system prompt. You say that you know what, you are a researcher, you're going to you're going to do research and like some more information about the background about like the system etc. What next? Which tools is it? Lovely. This is not working. Okay, works here. Tools. Yeah, skills could be Yeah, skill is a good example,

but you would also define like maybe skill kind of uh uh in the in the system prompt we could also define it separately. I haven't added skills here, but skills is definitely something you could add. So, that's the memory. Lovely, memory. Right? So, you you need like it's a conversational UI, so you have like a past conversations and you also have a memory, right? What's most important?

The data. Yeah, this is LLM context window. >> [laughter] >> Yeah, this is there. Why What did the user ask right the user question. cool. Now once you add all of this to the context now you are actually ready for the action so you give it to LM. What does LM do? LM look at the question look at all the go through instructions and prescribe what what

tool to call right? Oh, it doesn't work here. What tool to call? Now once you call the tools what happens here in the context of researcher the tool calls are actually the ones where you will go and do the research. So what will be the output of tool call? A lot of research data. So you get all the result from the all those various tool calls you

put that back in the context window. Now look at this. Tool result is like you're actually going at to the reports bringing a lot of data. Right? And then one tool result could be thousands of tokens. Right? So then you have like maybe multiple such tokens. Now once you have these results generally what we do is we want to actually figure out like you know is this

correct? I you reflect on it and want to figure out you know do I have enough information? Should I conclude this research? Right? And you do kind of self reflection you add the feedback here and then agent figures out yes okay. Now the reflection says that you know you go ahead and write the report I will go and write the report reflection says that no no something

is missing you go and figure it out again. Right? So it's a loop that you can see. And then base sorry and then basically add the final answer. Once you get a final answer research user will come and then add the next question. It's so much token intensive. Right? Now what is wrong with this? So there there is clearly some failure modes in this. So by the

way, this works. We went live with it. And the second phase of being able to answer complex questions, we are able to answer some complex questions already. It It did work beautifully. It had some issues, but it does work. So, what could be wrong here? And Research data data that might overlap with that, right? Sorry? Research data might overlap with that. Yeah, the research data might cross

the context, yes. One is like context window might just get overflowed. What else? Context window. Context bloating, yeah. Sure. All the context engineering problems. So, here, let's say you got a tool call, you got a result. We are not sure if that result is going to contain everything relevant. It's possible that it just kind of putting something out, right? So, it will It will basically kind of

also possible that it may kind of just corrupt, kind of add certain things where you don't want it. The moment you add more irrelevant things, more noise, you have a problem. Right? The everything kind of after that suffers, So, the first thing is like there's just too much in context. Just too much. We need to figure out a better way to do Second thing is, and this

is very interesting, nobody said self-reflection is actually a problem here. Do you know if this could be this will this work? Anybody has a problem with self-reflection so far? Think about coding agents. So, you work, you kind of you're working on a on a chatbot, or sorry, on on let's say cursor or cloud code, and you say you are building a UI and you say you you

go ahead and build this nice beautiful UI. What it does? It builds it, but it does not know if how good it is. And you ask it, "Okay, can Can you critique? It can critique. there is actually going to be a lot of problems. And I'm not saying this. Anthropic blog post last month literally. And this is by the way on top of Sonnet Sonnet 4.6 and

Opus 4.6. So, it's not old. They claim that the LLMs today are still very poor at self-reflection. Right? So, we need to think about self-reflection really really properly. in that context window that I was showing, the goal of the reflection was to make sure that do I have sufficient information to be able to answer user's question? Correct? That's That's the goal of that reflection. What is it

called in terms of evals? Do I have enough information to be able to answer question? That's a data completeness evals, right? what we could do is instead of having single agent do all of this, you could actually bring in that evals in the middle. Almost like you you do have a reflection, but you take it out from that agent now, have a separate node for it or

separate agent for it, com- completely different context window, and you do reflection. And now this works really well. Right? So, what we do is now the researcher basically does the research and gives us back a lot of data. We have a question. What we do is we take that question, we take that data, and we we say, "You know what? This is the question. This is the

What do you think? Is this sufficient to answer the question?" And the reflection only has those two information. So, what it says is, "Oh, there is something missing for that." And then you refeed it back to Now reflection goes back, does more and then you have a lot more data. A a lot more accuracy for your you repeat it n number of times. So, when we talk

about a simple research loop, a simple research loop, we have a we have a reflection, right? And then they are going to talk to each other constantly until you have a final answer, right? All right. Now, going back to the old diagram of this, what changes? The actual method is that the data is right or that is Sorry? So, you basically it might say is the data

is present or to a to study. But the data it just present it's right or that is not right. Yeah, we'll get we'll come to that. So, that that could be data. So, it may say that, you know what, it is complete or it is incomplete. It Even if it says incomplete, maybe there could be issues, right? And and that's a whole lot of that thought. That's

a very important But let's say now going back to the I'll I'll come to that in in a bit. But if you look at this researcher, now we say we identified a couple of We said first, [snorts] you don't need a reflection here, right? So, we take it out. Reflection is based on the tool result, which was the second cause of problem. We need to also take

it out. So, what we do is we take that out. Now, the rule tool result here, whenever we get that, we don't put that in the context window. We keep it in the state. I'm not going to talk about state management a lot today, but you need to kind of have good memory management, good state management to be able to actually deal with so much tokens, a

token intensive work. So, what you do is you kind of uh take these two things out. Now, you have a actually a portion to to write a nice reflection system What is a nice reflection system prompt? So, let's say for example, going back to the previous example, if you go at to to Claude code and then say, "Hey, is this UI beautiful?" What does Cloud Code say?

It may say that, you know what? Yeah, it is good. It may say it is not. Or it may kind of give you some, uh, you know, uh, random feedback. But we But this is live system. You don't want randomness here. So, what we do is, instead of saying, "Is this UI beautiful?" If If you can say that, "Does this UI follow following principles?" And then you

enlist the principles. You have a lot more better a lot better answer, right? You do the same thing. So, you have like a reflection prompt where you define, "Okay, is this not just Is this data complete? Like, here is how the data is structured. This is the type of data. You will get data from these various tools. These are the various tools here. How do we make

sure that Can Can you make sure that, you know, uh, break down the question into like a smaller questions and figure out if individual questions can be inferred from the claims in the in the context and all of that? Essentially, you could have like a lot more sophisticated prompt here. And then you basically have like all of the data gathered from tools. Now, you don't need to

pass them as a tool call. This tool result, you can actually construct it in a nice way, right? I mean, you can you can have, uh, proper context construction. And then you pass the contextualized And now you have a lot better Cool. So, this thing kind of goes away. So, you don't need this. Instead, that could be replaced with a summary. Off like, you know, we just

pass the result summary there and we're done with it. So, that's a lot better context engineering here, right? And researcher just needs to know that, "Okay, if I make this I asked you to make this tool call, what do you get?" I don't want to see the data. I want to see like what did you get? Did you get the data for that study I I asked

you to go and find out how many chunks you got? Did you get a meaningful information? So, I think that's all it needs. And then to figure out like what to do next. The second thing is actually a problem also here. Like can you can you again go back to this? What do you see there's like one more odd thing here in this first one that we

didn't talk. So, we said that in this does this work? No. So, here basically now you we deal dealt with the reflection part. Researcher is supposed to research Researcher is doing one more responsibility. Delegating. Delegating the writing. Yeah, there there could be sub-agents. We'll not talk about sub-agents, but yeah, you can actually have sub-agents also. But it is doing like beyond research there's one more keeper a

responsibility it is taking. And memory conversation. Conversation? But like what it is doing? tool calls tool calls yes it should do tool calls. Reflection we delegate it out. writing memory Writing? I'm a researcher. I don't write well. I can research. Right? For me uh you tell me where to go what to find I do it. I don't want to write properly. I can't because like again to

write stuff you need a proper instruction for You don't have it here. So, you extract that out also. Does that make sense now? So, you want to have like a really solid instructions around like uh write in this professional uh you know in the prose style for a scientific paper. You can't write something like hey you act as a pirate and then go and then do this

stuff. I'm like it's essentially like it can do whatever it wants unless you give a proper instruction. So, you take it out. You have like a nice writer system prompt. And now you have like a all the curated because this is like the final stage, right? And then uh you pass in the user and then you get back answer, right? So, this system works a lot better,

right? So, if we look at the Oh. Yeah. Yeah. And then basically the researcher does not even have that writer part here. So, the the simple research loop kind of is currently like you have this research and then you have a writer part. With me so far? Okay, give me a thumbs up. All right, lovely. Okay. this is cool. This works really well, right? And then we

are basically the second stage is done. So, we are able to answer complex questions. But the moment you actually give it to uh the researchers, right? Researchers have a bigger problem in our case. You It is not like this will work for like few tool calls. Okay, before even before we progress, um so this does not work basically for deep research, right? So, uh how many of

you actually used MCP tools in coding agents? Everyone? Now, how many of you think that even with the latest models, it was not able to pick out like what tool to use? You even if it was like really so common sense you should use that tool, you don't use it. Right? Feel that Have you ever felt that frustration? Oh, I felt it a lot. if I actually

my researcher is actually having access to so many domains, and if I actually put all those tool calls in that researcher, that poor it will not figure out. It will not figure out like what tool to call. Doesn't make sense? it can do let's say a couple of them. So, we went live with actually the the two that I have checked in there and that worked really

well. And it broke the moment we started adding more tools. it broke. Literally it broke. It didn't even work like the initial things that were stopped working because now it just it's getting confused. It's too many this system needs scaling. So, we need to scale this system to be able to a lot of data, a lot more data than what was in the past. So, there are

two dimensions to scale. One is breadth. The breadth is a uh lot of domains. Uh lot of Each domain like has many tool calls. So, many tool calls. Everyone is talking about study. Everyone is like, "Okay, give me this study information." And like every domain is actually talking about some or other type of information, some or other type of experiments. You can't really figure out like what

if you have access to 20 different And then this is like overall very large space for action, right? And to reason about. But there's another scaling that we need to do. The simple researcher worked in a minute. You quickly get answer. Now, you have a deep research system. Or rather, you have a system that needs a deep research because you have so many domains. You need to

go to so many places. And that's a different problem. And that problem is horizon. Right? So, you need to actually be able to scale to like make sure that you are able to access so many long-running task, right? So, essentially like basically you have more steps. Earlier you had 10, you have now 100. You have like more state. A lot of research that you're doing now behind

the scenes. You need a lot more state to The more the steps, the more the failures. The more the failures, the more recovery points need to think it's a deterministic non-deterministic engine. You you you kind of drag it in one direction, it will go somewhere there. So, it's like you have uh you are leaving this thing going to work for like let's say few minutes. You want

to be really really sure where what it is doing. So, you have to work on these two dimensions to be able to actually make the deep research work. let's talk about the breadth first. I gave example about the overload with the coding agents, right? Like if you have like only specific tools in your bucket, it works really well. The moment you add so many things, it does

not, There is actually a good term for it. So, uh has anyone actually seen this? Like you're actually working on a task, right? And then what happens is you're actually uh running a context window quite a bit. And then you're towards end of filling the context window, and the probability of marking that particular task as done goes high. So, agent thinks, "Oh, this is done." No, it

is not done. So, it's like there are 10 things pending, and it just marks it done. That's why now people go do more plan, and then they kind of have something externalize the state, and then kind of try to do it. But if you don't, let's say plan, and if you just give it one shot, go implement, go YOLO, it doesn't do it, right? And then there

is a term called context anxiety. Have you heard about it before? So, it's like it's almost like the the LM gets anxious that, "Oh my god, my context window is getting over. I need to mark this task complete. I need to complete this task quickly." And then it just marks this task complete. And it's really interesting. This is again from the blog post from Anthropic. I'm talking

talking a lot about the context anxiety is also exhibited in the latest models, like Opus scale, including Opus 4.5. They say Opus 4.6 is better. That's an ad. I don't know. I have no longer trust it, but yeah. But sure, but this is still a problem, right? So, you need to make sure that context anxiety is like not there. And then if you have so many tool

calls, what happens is your context window bloats up with the tool calls. So, you have like maybe 10,000 tokens in the uh consuming only tool definitions basically useless. So, how would you fix it? I have so much domain so many domains, So, we actually try to fix it. There are two ways to fix it. Hard way, simple way. Engineers love the hard way. Right? I picked the

hard way. So, I we started working on okay, I want to write a classification. I want to figure out like for what question what tools to use what domains they are going to pick up, And my client was not so happy with it. I spent over a month. it was a really hard problem because users questions are going to be so ambiguous that it's sometimes really difficult

to figure out like where you want to go. So, the easy way was Can you Can you tell me what could be easy way? How to figure out like what does user want? Ask them. So, then we we know we said, "Okay, you know what? We'll give them a drop down. I have 10 domains. You select what you want. And then even if you don't select what

you want, we'll figure out." So, we say So, we we kind of said, "Okay, that's step one." This is actually a very good point. I And the And the point I want to make here is in the non-deterministic world, Everything you could do to pull out tasks from non-determinism into determinism, do it. Right? So, if you want your coding agent to basically have some sort of reflection,

some sort of critique after every action you're taking, after every comment, add a hook. Right? Make sure it's deterministic. And not believe or let you know because this is something you can do automatically. So, whatever you could do to bring out non-determinism from that system, do it. And that's a hard-earned hard-earned lesson. this So, we did that. And then this actually helped us because now instead of

looking at 20 different domains, I'm looking at three. That's a big improvement. Big improvement, really. And now we do is we know defense in depth. We know that users are going to be lazy. So what we do is, you know what? Even if you select all of them and want to test the system, it's okay, fine. We will choose what we feel is important, but we'll tell

you first. So we go and you know figure out, you know what? Out of these three or what whatever you selected, we feel that these two are like maybe relevant. What do you think, user? And the user says, "Oh yeah, you're right. Actually, only these two are relevant." Or maybe say, "No, crap. Do what I say." User can say that. And you iterate. Now that you basically

got a clarity on like where you want to search, you have a clarity about the search space, you narrowed it down. Now you want to understand what is it that user wants, right? You iterate with the user. Do you do that? With coding agents? What do you call it? Done. Planning. almost like defining spec, right? What is it that you want? And then you basically you have

a good clarity about like what is it that that you want and where you want to go. With me? All right. So that's like a big problem that was kind of of breadth was able to solve by a non-determinism or like some some fancy not so fancy checkbox or drop downs. And then we have some sort of other in things in loop here. Now, that's one. So

this thing is solved, right? Like we have like a now good clarity about like what we want to get, right? We have clarity about like um what relevant tools we want to use. Now what we do is at this point in time, if I have 30 tools and whatever user has scoped, or let's say three tools. The rest of the agent never sees rest of the tools.

Never. You only are scoping it to the only those tools. You don't even bind them to the LM. So, it's just that LM's context window now only has those three tools. That's Makes sense? So, what we do with the next problem This is like one big problem solved. So, the breadth is fine. But, what do we do about horizon? How do we Right, so now let's say

what steps I want to take. What steps to take first, right? What next? Uh is long going to take a lot of time, like should I plan? We didn't plan yet. How do I plan? So, this is still open. So, what we did did is you have a clarified question now. And we want to figure out how do we actually make sure that we're able to navigate

this throughout. There is actually easy solution to it. You think. So, whenever you get a spec, you first And then it's like think before act, right? Like whatever we say. Whatever we think about in our life is what we are going to replicate in in agentic world, right? So, you first think what do you want to do? Create plan. Then you basically act on it. The act

is essentially And then again, like whatever you act, then again think about it. Like what did you do? What do you want to do? Then act. Then again think about like what happened. And then you basically get a final That's what we do. That's exactly what we do. So, the act part here is what we already figured The research and reflect. And the think part here is

actually a deliberate pause for reasoning. It's a It's a fascinating thing. I mean, so this just one pause for reasoning improves the accuracy quite a bit. I mean, it's unbelievable. we actually call this think step. Let's figure out like what happened in that actual thing step, right? So, first thing we discuss is we We figure out okay, what are the goals? What is it the user want

to achieve? What are the next steps? What could be alternatives? We kind of go figure out okay, what happened, What changed? Did tool call actually result in an error? It's possible, right? Like if you're running a system, something went wrong. Uh do I have to pivot, right? Uh how do I continue? Do I change my plan? So, I think thing step does all of this. And there

is actually So, um the And this was very much inspired from a blog post from Anthropic called think tool. I'm talking a lot about Anthropic, I know, but they have really good engineering blogs. Uh and then I I'm I I would recommend that you should really kind of watch out for them and then keep reading them. Again, now looking back at our older diagram, researcher context. This

is what we had, researcher system ask conversation history, current user. What do you do now? How How do you think? So, and then what we do is like at this point in time is researcher would make tool calls, right? Instead of making tool calls, what do This is the act part. What do you do? Before this, Right? So, instead of this, you think. It's kind of a

reasoning pause, right? You generate maybe initial plan. And then you basically act again because now that's acting is based on the reasoning that we've done. Then again, think. Uh this is like actually upgrading plan, reflection, all of that. uh do a you know, this is essentially a reasoning pause. Does that make sense? So, if I don't use this, right? What happens is you actually face us so

many issues. Like there is like plan ossification. You decide a plan, I want to execute this plan no matter what happens, I'm going to go ahead and do it. That's what happens. So, like for example, uh a lot of times what happens is agent thinks, "I want to make a tool call." Okay, it makes a tool call. The tool call results, "Oops, I don't know." And then

the the or gives you some random information that is like not relevant for the user question. So, you get to reflection. The reflection says, "Oh, that data is not sufficient. You want to go and go back and get that information." You come back and then again do this. And you keep repeating this. Keep repeating this. Other thing that could happen is one tool call just fails. Returns

an error, right? And then you don't know what to do. So, if you have that reasoning pause, you're able to figure out what to do. There could be silent partial failures. The premature convergence is actually something that you want to guard against in deep recess type of systems because you want completion. The answer completeness is actually critical in you need it. You You You You need some

sort of a thinking pause, right? So, Anthropic claims that they actually were able to improve just by adding this thing tool the benchmark was kind of improved like 54% compared to like what it was without It's like pretty cool. And this is like very simple to implement. Like think tool is essentially like a simple no-op tool in your agent that just thinks about stuff. It just figures

out stuff. We use it slightly differently. So, we kind of evolved this thinking tool more into the process reflection. Because essentially what we just don't want is you reason about it, "Okay, what do I do next?" We want more of a strategic think, not just the tactic things, right? So, you you want to think about, "Okay, what are the next 10 steps? Where am I?" And then

kind of figure it out, right? we kind of think you think about it more from the reflection point of view. By with the December update, you don't no no need think tool uh because then the extended thinking became so better that maybe I think think tool is kind of no longer required. But for us, it's still important because we're using it as a doubling it up with

the process All right. So, this is actually the overall research loop. we can say that with this thinking in place, you're able to make the that simple researcher work for a longer So, now you're actually able to steer it So, what's remaining? The last part. How do you write? This is easy, right? So, what you do is you I have so much information now, complete information. I

give it to write And the writer I I say writer, go ahead, write. It don't work. There are like many cases, specifically like if you look at this, uh we have like certain cases in health care where like I want to generate a table with 100 data points, 100 data points, like like like I want to this one table is just like 20 rows, five columns, simpler

table I'm giving you. Is like just multiply 100 data points you want to actually curate from the research data and then put that at the right block. It's not a trivial task. It will fail at maybe 20 places in that. So, it maybe work for 80% of the time. So, what we do? Reflect. Right? But now, what happens is you have basically researcher, you give your findings

to writer. Writer writes the first draft. And what you do is now the draft we you check, okay, what are the gaps? And then see, oh, are the gaps based on the because something that writer didn't do well. So, is is is is it a synthesis gap? If it's a synthesis gap, it's fine. Maybe you just kind of go back, redraft, right? Because then you'll be able

to figure out. So, you say there here is Here is the research, fill in the gaps. Works. Or it could be that a genuinely researcher missed things. That was your point. Where it is possible that your researcher felt that the data is complete, it was not. Right? And then it could be something that you was not caught in the first level of reflection, could get caught in

the second level of reflection. Right? So, this is actually a researcher gap. And then with this we are able to actually be very confident and say that you know what? What Whatever it is going to produce is going to be a lot more complete. I'm not saying definitely complete. I'm still saying a lot more complete. Right? a quiz. We talked about three types of reflections. What were

they? Self-reflect. Self-reflection, but we don't Yeah, but we said we don't use it. But what we you actually use? I don't remember like I Process reflection, lovely. That's It take checks are we on the right track? Other reflection. Writer. Sorry? Just like Writer's reflection, yes. That's the third point. What is second? Think tool is this process reflection. We check one more thing. Very important. How do you

know if it is complete? Wait. Data completeness, right? We also like do the data complete. So, that's like the second reflection. Like is my And third is like a process reflection. Like the the draft reflection. So, essentially like the these three types of reflection doing entirely different things. Which almost acts like almost acts like a evals in the middle. Right? So, it's kind of you you al-

always have some sort of a thing that controls you. Right? Thing that controls like the flow of how the things are going to progress. So, you need to have all these levers in place in your system, which is long-running. uh I think this is basically what we discussed so far. Right? And what do you think this is? We actually have so many things here. What do you

think it is? I There's actually a word for it. Yeah, orchestration, sure. But nowadays people call it uh there is a fancy word to it. You call it also engineering associate engineering with it. And harness, right? It's almost like a harness. Right? So, let's talk about what it is, you know this? I have a horse and you put a blinders to the horse. You put lines there.

I want to go to I like I want to ride the horse. I want to basically be able to control where it is going. I don't want horse to basically look everywhere. I want it to focus on what I want. And I I want horse to take me to my destination. we have so many things here we want to be able to control that This is exactly

what a harness is. Right? So, you have a raw horse, raw power that is almost like really we call it horsepower, right? Like it's like a lot. But you want to be able to leverage it properly, you put like some these uh these harnesses around it. Right? So, this is exactly what uh the horse is a representation of an LLM here, right? Uh you want to basically

be able to control LLM properly. What that means is basically whatever gaps we have right now with the we talked about oh, it can't do Right? We talked about oh, you know what? Uh it can't think properly, so we actually had a think tool, right? We uh so, you basically said, "Oh, it can't do it? No problem. I will steer it and we steered it, right? 3

years ago uh Now, now, how many of you do function calling in in whatever project you're working on? Tool calling? Very generic, right? Like everyone is I have thing to do. Yeah. Uh everyone basically is like, "Yeah, I can I can easily do function calling. Like, no big deal, right?" Function calling was not a is was a big deal 3 years ago. So, models did not support

function calling out of the box. That's before And what we used to do is we used to say that, you know, prompt used to prompt engineer LLM that, "You know what, generate a JSON. Start with curly brace. Do all of this thing and output a JSON when you think that the question belongs to, let's say, adding two numbers." And then it will output a JSON and then

tell me add method and then here is the parameter. And it it will not work many times. This is GPT 3.5. kind of take that JSON JSON, parse it, and then then kind of convert it into function call and actually call a function. and we also also used to have retries there, like, you know, kind of it will not work first time, so maybe you have to

do it multiple times. That was actually a harness, right? And that harness was there because the model did not support function calling, right? That that limitation was there. Whatever harness we're talking about again are based on all of that. So, if you watch this talk again next year, maybe some of this is not relevant. Right? But today, if you want to get the value you need to

build that harness. So, the harness scaffolding is essentially like all these workarounds and wrappers really born out of these gaps, right? So, the moment all of these are basically natively built into LLMs, you need to kind of rethink about your harness always. So, the moment you start using even better models, you need thinner harness. If you use like a local open source model, you want to run

you want a lot more harness. You want to steer it a lot more, right? Now, going back to what we did. So, after doing this, what we found out with the with our client is that were able to reduce like the manual searches by like 90%. The time it took to generate reports for like our researchers drastically reduced. Like a researcher would want to write a report

that is worth hundreds of pages would take weeks, not months. Now, you can do it minutes to hours. It's like a drastic. It's really cool. we also worked on like improving data quality, data accuracy, right? And then essentially it basically saves a lot of time. And this is indirectly valid to the patients, right? Because essentially you are saving the researchers are the ones who are doing the

research. If you can make them a lot more productive you get it. We published a research article on this. You can actually take a take a look at it. There's also case study on ThoughtWorks for this. Thank you. >> [music]