Agent Memory as Core Design: From Pilot to Production - Varun Yadavalli

In this talk, Varun from Broadridge discusses the development of an agent designed for data extraction from financial documents, emphasizing the challenges of not implementing memory as a core feature. He explains how the agent initially relied heavily on human reviewers to correct errors, but it repeatedly failed to retain corrections due to its lack of persistent memory. Varun introduces a new framework that incorporates working, episodic, procedural, and semantic memory to enhance the agent's learning capabilities and efficiency. The speaker outlines how this structured approach allows the agent to remember past interactions, apply corrections, and reduce instances of repetitive errors in document data extraction. By building a memory management system, they aim to improve accuracy and speed while minimizing the cognitive load on human reviewers.

I'm Varun. I work for enablement group at Broadridge. So, I'm here to share two things. So, one number one is the story. What happened when we shipped an agent without memory as core design? And the second one the framework that we built after we realized why this matters. A quick show of hands, how many of us in the room have instructed the same thing to the ChatGPT,

Copilot, or maybe our own favorite coding agents. How many of us did? I'm sure most of us did. And we know the problem. It's memory. It's clearly the lack of it. Right? At Broadridge, being a financial solution service provider, we deal with extracting data points from large financial documents. So, traditionally, this process heavily depends on humans. Humans reading through pages of information and extracting the content manually,

and we'll be consuming in our downstream processing. To overcome that, we have built document to data extraction platform on top of our core agent decay service with human in the loop. So, this platform worked well for couple of weeks. And when we looked at the observability, we started noticing a pattern. The pattern is So, whenever agent did some mistake, we have human in the loop. So, every

extraction that did by agent, a ops reviewer reviewing the outcome because we need to have 100% accuracy. We are we are not actually living with 80%, 90% that AI is going to give, right? So, there There a for every extraction, we have human reviewer validating the output. And what they identified the pattern is that they keep on giving the corrections to the agent, but the corrections are

not being persistent. So, take an example. This is one example, I mean, a sample snippet where we have to extract the net revenue or revenue from a annual report. If you take a look at this one, so agent did extracted the revenue, which is 100, and it went to human, which is the business ops reviewer. When they reviewed the output, it it did extracted what is said

exists in the document, but it missed this one. The amount that was represented is in thousands. So, it didn't read the footer statement. So, it did not updated the applied the unit scaling. So, what did human do? Human gave the constructive feedback to the agent, "Hey, whenever you're extracting the data, also look at the footer. If If it says it is represented in the thousands, do the

unit scaling to thousand. If it is in millions, do the unit scaling to millions." reviewer fixed it. Session ended. And the fix got discarded within the session itself. So, next time same filing or similar document came, and reviewer taught it again, and session ended, same error. So, it's keep on repeating. That's where the problem become obvious the agent didn't forget. It's never designed to remember. So, what

we do all what we all do in this uh patterns is we all stuff the prompt. We keep on adding more instruction to the prompt, few short examples, here is the chain of thought, here is what you need to do, right? You give tool preferences. All that you you will add it in the prompt. But to be uh that that scale. To be fair, if you if

you add instructions more to the prompt, it works for that particular run. But if you take a look at the system, the platform that was built to scale for all the documents, it doesn't and also there are implications when you keep on adding to the prompt, right? Tokens grow. Your input tokens always grows. And model if you take a look at the the model evaluation, it will

part of the instructions that we are have in the top in the context window on the bottom. Model is very good at following them. If you have any instructions in the middle, it that that was the pattern that we have observed that it will it will not follow them betterly like how it how we have in the top and bottom. And then latency also grows. So, your

if your input grows and if you keep on adding more chain of thought, it has to do so much of reasoning. So, your latency grows and also the it it is prone to hallucination, right? It is. simple if I keep it simple, right? The long prompt is not long-term memory. So, from this we what we understood is that we stop we should stop adding more content to

the prompt. We should stop we should start building some system which is outside of the agent that can actually give the right context whenever it needs. So, we followed industry patterns. So, last year at the same time we started identifying okay, what do we do better here? So, first one working memory. This is all we know. So, whenever you are building an agent, you take an example.

So, we call it I mean if you use Google ADK or Lang off, right? They call it as a state. Which is nothing but working memory which is holding the current run of that particular session. And then we have long-term memory. In long-term memory, you have episodic, you have semantic, and you have procedure. And these all four is not just the labels. They are different uh by

the kind and the rules that we are going to build. So, first let's talk about the working So, if you go back to my example, at the moment when reviewer corrected the unit scaling mistake, what does the system holding? The system holds at that particular moment the conversation history that was happened between the agent and user, and the second one, the tool logs, and the input file.

This is what working memory. And I often see people bit confused between working memory and context window. Working memory is the state of truth at that particular run. It's not context window. uh context window is what we assemble from the working memory. So, if you before you're sending to the model, you look at the working memory. Okay, for this task you you pull whatever the relevant thing,

and you prepare the context window and send it to the model. Both are not not same. And the if you take a look at the life cycle of uh working memory, it starts with the session. And it gets updated throughout the session. So, it keep on building until your agent completes, right? And it ends If anything that has to live outside of the working memory, it has

to be promoted to long-term memory, not this. So, working memory, there is no persistence here. It is only for that moving on to the episodic memory, the next one. So, if working memory is what the agent know right now, the episodic memory is what agent did before. These are the events, the past events that how we capture, right? So, take a look at the example. You go

if I go back to the same day one where reviewer corrected the unit scale mistake. So, if you have captured that particular session in a structured way, we should not store it as a raw transcript, the whole dump of it, right? We should capture it a it we should capture in a structured way that what is the intent, what is the agent intent, what did the agent

do, what did the reviewer corrected, what is the final outcome. Right? And it it's it changes from use case to use case. I'm I'm trying to give the example in what we have built, but so that's going to be an episode to you. And day one becomes an episode to you. Day three becomes an episode to you. Day seven becomes an episode if the similar filings came.

So, that's where your agent starts learning. So, you should storing these past events in a structured way, and there should be a quality gate also. Because if you store any any bad example, bad episode that where agent failed to extract, and that go and sit in the working memory, so agent can't really figure out what work what work done what did not work, right? So, you need

to ensure that you need to have a quality gate. If the outcome is successful, in our case we have human reviewer who is going to validate and say this is good, and that's going to sit in the episodic memory. And if you take a look at the life cycle, the first one first you need to capture the session. And then you need to summarize it. Once you

when when you when you summarize it, you need to keep it in a structured way. And after the summarization part, once the human review completed, based on the outcome whether you store it or not, that's the decision that we need to take. And if this repeated patterns keep on coming, same pattern coming coming again and again, right? We should stop treat them treating them as past events.

We should start treating them as learned knowledge. That's what the semantic memory is. So, you have these raw episodes where from uh the past events day day one, day three, day seven, right? So, it all talking about one thing. So, agent has to look into the footer before it actually extract the unit, right? So, that becomes a stored fact. So, agent is able to generalize what is

right, right? That is what we are trying to build as part of semantic memory. And every fact need a evidence. Every When I say evidence, from from where it is being interpolated from. So, you need to have that back tracing of the events that it got generated, and also the confidence score. So, confidence score is also very important. So, I'll talk about in in the next bit,

and time to live. So, you cannot actually have a fact that is infinitely true, right? You need to have it for maybe you start with 90 days, and then you have to update your fact based on the how how direction changes. Take an example in in case of financial documents, we are crossing. Uh if there is a regulation change, the document layout completely changes, and my my

fact is no longer valid, right? And you need you need you need to version control it. Why why you need to version control it? Let's say, if you have changed the uh if you have updated the fact, you need to have the audit trail why the outcome is like this, because there is a version changed in cycle, the first one is you need to have an aggregation

of the previous past episodes which are worked, right? So, from that you're going to capture the episode. And then, uh the confidence level that I'm talking about. So, once you migrate the past events as a fact, you you use LLM as judge, right? And then, come up with a confidence score. Okay, if the confidence score is more than certain limit, right? 90 above 90%, you can auto

write them into fact store. If If is if it is in medium range, you can let human review. Okay, is this a good good fact or a bad one? Right? uh whatever I just mentioned about the time to live and everything has to be there. And uh the time to live is very important. So, after 90 days, you need to let's say you put 90 days, right?

After 90 days, you need to let your human reviewer uh review that whether it can be still a valid fact or a not a valid fact, right? So, that's the reconfirm or decay. If it is not a valid fact, you'll have to uh decay them. here one caution of note. So, we may can have uh missing fact. Missing facts Missing facts are something that agent may learn

over the time. But, if you have bad facts that get stored in your system, that will let your agent confidently answer wrongly. So, you should try to have a system that controls Okay? Uh uh to review every single time that whether this fact is really valid at this point in time yes or no. And the last layer is the procedural memory. So, this is what uh agent

learn how to best act on a given task. Right? So, take an example. Which tool I need to use for a given task? What is the sequence of tools that I need to use? What are the input parameters that I need to send to this tool to achieve this task? And when to escalate, right? When when to escalate to human that I I'm unable to proceed this

further. So, that is what the procedural memory is. So, it you can simply assume this as a playbook. So, in in in doing while you're doing the production deployment, we'll have the runbook, right? That is what it is, right? This is the These are the list of tools I need to use. These are the sequence events, right? And once you have this procedural memory, agent will not

try to do the same work from zero. It will try to you go and get the rule from previous memory. Okay, this is the rule that I need to pass pass through and then it is going to run only those five tools instead of running the 100 identifying from 100 tools that okay first which tool first I should use fail it again and then go back to

the next tool, right? It's not going to do that. So for any defined process in enterprise, we'll always follow one single process. Take an example in case of annual report, I'm going to do the same process throughout the day in enterprise. So in case if you have any non-deterministic flows, maybe you still need to write the playbooks in a certain way that you you will be able

to classify them. So unless until you classify you will be not able to pick the right rule book. Okay? And now once you have all these memories defined, right? The next thing that you are going to do is recall. So how an agent first able to recall what it gets stored in the memory. So the first thing that you need to do is classify the intent or

classify the task. What what was the task that was given to the agent, right? So in case of our document data extraction, what we are going to classify is okay, what is the document type? Is it an annual report? Is it a quarterly report or is it is a fund prospectus? So when we are storing the previous what are the memory labels that we have seen, we

do add these labels so that we can filter So that I can go into the though the agent is same, the agent is document data extraction agent, but it it we we add these labels so that I I was able to exactly go to the respective agent memory facts, right? So you need to classify the task and and it's how you want to classify the task is

actually it's up to the use case. So here I have issuer and I also have intent and etc. So now the The step is once you have once you done the classification, you need to fire parallel queries. One, try to get for this in task, what I did previously, last three runs. Three or five is something up related to your use case, right? You need to identify

for the given task, what I did previously, which worked best. So now, that is going into the working memory. Second thing, what are the generalized facts for this given intent? Get them. And then, what is the rulebook that I should use for this? Get them. You get all them all of them, you need to do re-ranking. Why do you Why you need to do re-ranking? Though you

do the semantic search, you may not be able to get the right results in an order. So you'll have to do the re-ranking, and also do do the dedupe check. Because if you are keep on doing the episodes storing the episodes, it is likely possible that last three episodes that you pull are going to be the same thing, and it it doesn't help, right? I mean, you

you can So you you dedupe them, and then you need to prepare one shot brief that goes into the working memory. Right? So, this is what the agent did for the previous runs for this file, and these are the facts. This is how you need to work. So now you have one shot brief that is going into the working memory for the agent to process the next

So, this is the architecture that we followed. So we have working memory on top, which where model sees what agent need to know. And then, we have a memory manager underneath it, which can connect to the long-term memory, which is episodic, procedural, and semantic, right? And one thing that is important, that during the startup during the startup the agent startup, you need to first do the recall,

which is what we just created one brief, right? And then on every turn and every agent turn end, you'll have to store those states. And then when the session ends from that session end trigger. So, these are the triggers, right? When the session ends, you need to identify, okay, whether it can be promoted to just as an episode or it can be promoted as a fact because

I have so many uh repetitions of the same uh episode or it can go and update some rule in the play playbook that I have did uh differently this time. I had used OCR tool rather than plain text reader, right? So, you'll have to uh have that control on the event hooks every time you do the uh session end. I mean, this is uh a simple class

uh that uh you need. And then how do you implement it, right? So, when we started, we really not started uh implementing all four types of memory at once. So, we started first uh what with working memory. So, of course, when we build an agent by default, we'll have the state, right? So, from that, we try to understand the pattern. So, what should go into the episodes?

So, we we identified why it needs structure also because of this. So, we first have the working memory and then we have the episodic memory. We understood, okay, for this given use case, we have a configuration that okay, this is how the structured episode should get stored so that I can get benefit of it. And then also next step is from these episodes, when do I promote

that as a generalized fact? So, that you need to prepare prepare. And then last one is the present memory. So, this is not you must follow, but this is what worked for us. We started in this flow so that we were able to achieve where we are. did not get smarter. The system around it should. Right? This is I mean, this is the favorite buzz, right? About

uh agent uh all engineering harness. It's the same thing. So, we are not making agent smart. The system keeping our system around uh in a well-designed way, we are making it smart. And this this is the research purpose that we referred uh last year when we tried to build this uh system. what what changed after we implemented, right? We implemented when we implemented the memory, so first

thing that we see repeated error rate being dropped. So, we do have observability that Okay, so previously, the reviewer keep on giving the same corrections every time. So, now we see that the repeated error error rate being dropped because agent now knows what is the general facts generalized facts and etc. And the next one is max iterations. Since we have the playbook or rulebook, it it is

not keep uh doing the more iterations or more tool calls. It is the number of iterations it did was reduced, and which in line reduced the tokens per run. So, previously, when we loaded an agent, we loaded all the tool schemas. You know the tool schema. Typically, tool schema would be 1K tokens, right? You need to define what is the function, what are the inputs, what is

the output, right? So, when you load all 100 tool schemas at once, your input context is already bombarded, right? So, because we have a playbook, we we know which five tools that works, and we only load those five tools. And then in line, which actually reduce the input tokens. And additionally, we stopped stuffing our prompt. Our prompt will be the goal, and then the one brief that

we talked about. So, with all this, uh our input tokens reduced, and also the reasoning that model has to do, all of it reduces. Yeah. >> [music]