Anand Chowdhary - building reliable AI systems in production

This talk focuses on building reliable AI systems for production, especially in the context of evolving agents versus prompts. The speaker, who has extensive experience in applied AI and startup environments, shares insights gained from years of developing AI technologies for various applications, including accessibility and sales. They discuss the transition from simple prompt-based solutions to sophisticated agents that perform tasks in real-world environments. Key concepts covered include the importance of context, memory architectures, tool design, and trust in deploying agents. The speaker highlights the need for new abstractions to manage agent capabilities, emphasizing that software needs to be adaptive and self-sufficient.

I think we can get started already. Fantastic. So, funny story. The first title for my talk was how we test and deploy prompts at First Quadrant. That's because 6 months ago I was the founder of a startup named First Quadrant when I wrote the talk and I was invited to speak here, but a lot has changed. As you know, we're living in uh un- very strange times

where we're It's like dog years almost. 7 years of progress is happening every year, so 6 months is a big chunk of that. Um so instead of this, the talk is now agents rather than prompts and uh First Quadrant actually ended up getting acquired, so I'm doing something else now. Um so now the new talk is building reliable AI systems in production. Uh from assistants to agents,

what 10 years of building AI helpers has taught me about building agents for production. My new job is in the founding team of a new company, Sycamore, where we do enterprise AI. Uh so I'll tell you a bit more about that in a second. So, I've been doing startups my whole life and uh applied AI also for about 10 years now. Uh one of my early startups

was in accessibility technology where we built tech for people with disabilities. Uh and also that was we built, for example, blind-friendly smartphones where we used text-to-speech, speech-to-text, uh image recognition uh to help people with visual impairment uh see things, quote unquote, around the world. After that, I did a startup called Eva, which was an intelligent email assistant for scheduling appointments. Uh you could just send it an

email and CC it in an email and it would do the back and forth to find a slot that works for you. Uh this is pre-LLM, so there was a lot of if conditions and NLP stuff, but uh Uh, yeah. Uh most recently, I was the founder of First Quadrant, which was a startup backed by Y Combinator, uh, where we did AI for sales. So, we built,

uh, the first AI sales agent in the wild that could do prospecting, qualification, outreach, uh, and just run your sales process for And in First Quadrant we did a lot of really fun stuff cuz when we started the company, this is again before ChatGPT launched, uh, very early AI, we had to invent a lot of the tools we take for granted today from first principles basically. For

example, in these screenshots, we had within our app a dashboard where we could track all the generations from all the prompts. Um, we had like a little button where I could open the prompt, I could see the system prompt. And this is a three-shot prompt from 2022 when we had three-shot prompting. Uh, and and I could track everything within our app. There was also a whole system

we invented for creating prompt versions so that whenever we saw something going wrong in production or when I had a new idea for an improvement in the prompt, we could just make a little change and just run it as a sandbox version and it would actually then run against all the different prompt runs recently and then I could see a side-by-side comparison of whether this new prompt

makes an improvement or not and like a little get diff style, uh, just so I know. And that's all of the stuff that we had to do like 4 years ago. Um, to to have reliable agents. Uh, these days I work at Sycamore. We are based in Palo Alto and we are doing enterprise AI. So, we work with Fortune 100 companies on helping them deploy, agents in

the wild. Uh, we raised a seed round of $65 million and we are very actively hiring across all roles, although it's in-person in Palo Alto. Uh, we have a 10K referral bonus if you know someone though. Um, so I did AI for accessibility, AI for scheduling, sales, and now for enterprise. And in these last years and we went from prediction models to natural language processing to prompt

with workflow chained trees to now like actual agents that just go in the wild and do the work. And so, I think my thesis is we need software that runs itself now. People are are not going to buy traditional fast applications anymore. dumb software is dead. So, in the olden world we had prompts where the idea was there's an input, maybe you do some sanitization or whatever,

it goes into a model, and then you get an output. And the tools we built to do this were prompt versioning, prompt logging, maybe some structured outputs, maybe a little bit of evals. Today, in the agent era, it's completely different. You have a big picture goal, you have you build a plan, you give a model tools, it does some work, you loop this over and over, and

we need a new set of abstractions to make these agents run in production. Those are memory, permissions, approvals, side effects, etc., etc. So, the difference between prompts and agents, I think, is that if you have a terrible prompt, it's just going to give you a a wrong answer. With agents, the worst-case scenario is so much worse because it can do stuff in the real world. People can

get hurt, emails can be sent out, stuff can get deleted. So, the failure mode now is a is a lot larger and therefore we have to do a lot more. So, quickly, what is an agent? The currently accepted definition comes from Hannah from about a year ago. Agents are models that use tools in a loop. I think this is a really good definition because before that we

didn't really know what an agent is. Uh, I I feel like this is now a little bit dated. Don't tell Hannah I said that, but I think a year later, if I had to describe an agent, I would say an agent is the model and the harness. And so, we've abstracted this idea of running tools in a loop in the harness layer, and I think that means

we additionally to add what is the right context and how do you trust the agent to do stuff in the real I think this is kind of where it gets a bit more interesting. So, a quick side note. When Claude Code came out, um was it a year ago this time-ish, I think? Uh year of agents started and uh I was I did Y Combinator before this,

and as a founder, I was thinking, "Hey, I thought this there was this implicit agreement that the labs are going to build the agents and then we YC startups build the things on top of these agents." And when they built Claude Code, I was like, "Why did I not build Claude Code? Like, what was missing in all of this long tail of startups that missed building Claude

Code?" Um and then I talked to some people at Anthropic. I talked about I I learned a bit more about it, and this is kind of strange cuz we have been very from anthropomorphizing these agents, but at the risk of doing that, I think the answer I thought is empathy. Empathy for Claude. If you step into the model's shoes, how do you set it up for success?

How do you build all of the tooling from the perspective of not the human who's the user of the tool, but the model that's interacting with that tool? And so, that means you think about how do you name your tools and how do you set Claude up for success. And I think that was missing in the startup ecosystem 16 months ago, uh but not at Anthropic, which

is why they ended up in giving us the year of agents. So, if we say, "Okay, context plus tools plus trust gives us agents. Context is, you know, what what does the agent know? Tools is, what can it do in the real world? And trust is, why do you let it do stuff in the real world? Uh, then let's kind of double click on each of them

really quickly. So, context is not just a bigger prompt. It is everything the agent knows about everything. It's its working memory. It includes stuff like all of the third-party data, all of the integrations with your ticketing system and CRM and web search and everything else. It includes all of the previous conversations you've had, the history, what am I working on now, what is the user like? And

it has dimensions like uh, hierarchy and scope and how old is it and how sensitive is it and should I be protecting it? And one of the most important parts of context is memory. And now, in the past year-ish, we've seen memory has been broken down into a bunch of different types of memories. Uh, for example, uh, I had an open claw early January uh, that would

do a lot of the stuff to run my life. Uh, and the really cool thing about open claw, it gave us another year of agents moment, right? Like, it's another step function from Claude code to now this world of harnesses. The thing that open claw figured out is more than just texting Claude code on WhatsApp. It is their memory architecture. They treat memory as state and they

persist it in a file system. That means that you can have facts, you can have daily notes, you can have long-term memory. There's this concept of dreaming, where you give the agent, when you're sleeping, literally like a prompt that says just dream, read all of the conversations that happened today and create facts and create memory for that the next time you wake up in the morning, you're

more useful. So, I think memory should be transparent from the perspective of the agent, and it should be inspectable. Meaning, if it's in the file system, I can just feed it into memory and cat for a certain day. Uh but, we can go further. I think memory can be namespaced. For example, LangGraph has JSON long JSON documents where you can scope them by this is my memory,

this is the user's memory, this is the user's team's memory, this is that org's memory, this is just web search or whatever. And so, when you scope it like that, you can query it, you can index it, you can back it by a more reliable store than just file systems. Memory can also be a graph. Like Zap uses this uh knowledge graph style graph where memory uh

points talk to other points and are linked to the most relevant points of memory. This means that you can do stuff like find the most semantically similar item based on relevance. And so, you don't limit scope to just time, for example. And memory can also be archived. So, for example, MemGPT has this really a smart idea of this distinguisher distinguishment between what is memory that's always visible

and what is memory that can be archived. And then, you have a little of tool call where I can say, "Can you find all of the recent 1 year of information related to this one topic?" And so, that means you can semantically search this long-term memory just like humans can potentially do. Uh and you don't necessarily throw it throw it all in the context. For example, in

OpenClaw, uh there is a plugin that lets you create summaries and then summaries of summaries and then those summaries, and you can give the model just the top-level overview of everything and the right tools to decide where should it double-click on to expand the most relevant memory. Let's go to tools. So, context and this is where the empathy for Claude becomes even more important. Tools need to

be desi- designed now for a new kind of user uh agents, right? And in the early MCP days, the idea was just take your existing REST API and then turn that into an MCP. But we realized very quickly that an agent is is just that. It's another kind of an agent, right? It's a user. And you don't give a human a list of crowd endpoints and say

like do something in your app. You give them surfaces like um find a ticket rather than doing get ticket ID or close this ticket rather than doing patch status is closed. And so similarly, what you want to do is build agent native surfaces which have a semantic meaning to the tasks that the agents can do. So I think the way you want to write effective tools is

um think of tool design as product design. Uh build tools based on actual pragmatic workloads. Uh work evaluate and feedback from the agent to improve these tools. Like what is the schema? What are the examples? What are the different workflows? Uh test them, obviously. Um and yeah, I think agents are as good as the tools you give them. Uh not too many. Not too few. Uh if

you if you have, you know, many hundreds of tools, then that's not great either cuz you're throwing all of that in the context. There have been ideas to try to solve this. For example, Anthropic now has their MCP search where the agent can from all of the different tools load the right ones in its context. I'm not sure we have figured this problem out yet as an

industry. I think uh there's still a lot of work to be done and the way I think about it is that you want to give the agent the right tool for the right job. And so that's the job of the harness of figuring out when should I suggest the right tool to the agent. Um the last thing is um tool calling itself is really an information architecture

problem when you have all of these tools. And it doesn't scale when you have like 20 MCPs connected times 20 tools per MCP. So, I think my thesis is that the next generation of tool calling will look a lot more like programming where you give an agent a code environment and all of the imports that it can do just like in a traditional programming language. Uh and

you don't throw all of the outputs in the context then. So, with code execution, you just give him some tools. Um it can do cool stuff like take the output from one tool and then run it in another tool, etc. Just write a little program and then only the output of that entire program is given to the context of the agent. Anthropic did a little research about

this and they found realistic token usage down to 2,000 tokens from 150,000 using an approach with code execution. A few months ago, I also wrote a blog post called Agent Script where I proposed the idea that instead of giving a model any tools at all, what if we give the model just one tool which is run agent script which is a theoretical programming language built for agents

which can then do anything like bash or interact with remote APIs or whatever and then the agent decides what code to write and what context it wants to load at the end of the tool call. My bet is that that's the place where we're going to see a lot of innovation in the next year. The last part of this context plus tools is trust. What because the

most important part is really like you want the agents to do useful stuff Uh which means you have to let it give access. Like I have my agents YOLO mode uh all of the access possible. But um when I work with Fortune 500 enterprises, we cannot do that. Uh, so we have to think more deeply about how do you think about trust when it comes to agent

autonomy. gone are the days when we thought about like, uh, just policy-based trust. I think now you want to think about a bunch of things. most important things that we've learned. First, very basic, right? Like prompting stuff of like guardrails. We want to make sure we sanitize inputs. Models are already getting good. Like it's very hard to get a model to like spill out its prompts and

stuff these days. But, uh, you still want to have a layer of making sure that you the in on the input level, nothing harmful should reach the On the other side, any harmful outputs from the prompt, you want to make sure they don't reach real people. You want to make sure, you know, always the tools, arguments, and and you want to think deeply about what are the

side effects and where do you want a human in the loop? So, one of the ideas we had was autonomy is like a ladder where when we build enterprise agents now at Sycamore, we start with the agents just doing uh, observation. They don't do anything. They just look. They're just hanging out. They're figuring out context. They're learning patterns. They're seeing where the redundancies are. And that's it.

And then you start to say, "Okay, this seems like something is helpful here. Let's move it to the next level." And then we say, "Okay, now the agents can suggest actions." And then with the human in the loop, you can always approve the actions you want to take and reject the ones you don't want to take. And then you say, "Okay, I start to trust this system

a little bit more. So, then I'm going to say from all of the different actions it can take, these few actions I'm going to say it can do autonomously." So, this trust ladder, I think, is a really powerful concept in deploying agents in production, where it starts with suggestions, then drafts, then human in the loop, then some action to full autonomy. So, today we have agents running

in production from the suggestion end of the spectrum to fully running all the time in the background, depending on the kind of work and the kind of customer and the kind of stuff you've seen the agent do in the past. To implement something like this, there are some really good durable workflow patterns. This is going to be really interesting for anyone who's worked in a pre-agent world

in durable execution. Um where it's really simple, the LLM says, "I want to call this tool." And then you say, "Okay, I don't know if I want to let the LLM call this tool or not." If yes, go for it. And if not, it pauses the loop. It asks the human for an approval. The workflow is now in a paused state, and nothing is going to happen.

There may be like some sort of a timeout or some auto approve after 10 minutes or auto reject or whatever. The human then says, "Okay, I agree this logic seems sound. I will let you do this bash script or whatever." And then it continues. your If you implement a system with durable execution, that means the state of human in the loop is no longer living within the

agent tool loop, it's outside somewhere. And so that means it does all of the nice things, like surviving restarts and semantic learning, for example. I don't know, if you always approve a kind of action, maybe we can suggest to approve that kind of action all the time and and so on. Side note, I think now we are finally treating agents as kind of critical infrastructure. So, we

have basically Yeah, everyone is have their stuff, man. Like, I don't know. I think everyone has agents. I think now that means we have to focus on reliability. Like, are your agents actually production grade? yeah. So, I think that's helpful. all righty. Orchestration. So, that's another really fun thing. In the last like 3 months, I've seen so many demos of like the same Kanban board that started

OpenAI which can look at linear tickets and then agents can just pick them up and give you PRs. So, I think agents are now changing where human judgment sits in the entire spectrum. With these multi-agent coding workflows and all of these orchestrations that people are building on top of Agent SDK, um context engineer engineering is now a product problem. This means that when you're thinking about parallel

coding sessions, um you don't want to look at every tool call or you don't want to look at um every even every pull request depending on the PR. Um what you care about then is the context is being fed uh earlier, meaning um the better ticket you write in Linear, the better job your agent is going to do, which means the job for a PM now is

a lot more important. So, in our company, uh we don't have any traditional PMs, uh obviously, but um no offense to traditional PMs, but every good PM I know now is a cloud code native PM. So, the job of a PM has obviously changed, but also the job of an engineer has changed to soft PM. Because now that we've all kind of become managers of clouds, uh

we need to know how to write good tickets and write new issues with the right context so that we can trust more of the process to happen autonomously always in the background. this is a really side note because you can speak much longer about Evals specifically, but I think one thing that has changed between prompt evals and now agent is that you can get the right answer.

Now it's more important to think about did it what steps did it take to get here? Uh because the right answer can come from the wrong wrong places. So I think one interesting thing is uh trajectory evals, which is a space that a lot of people are investing in. That don't just evaluate the final answer. Evaluate did it write the right goal? Did it write the right

plan? Did it use the right context? What are the tools? Why did it use these tools? Um did it ask for the approval at the right time? Um so to go back from evals to build evals, you need the right data, right? uh you need a flight recorder. That means you want to capture the goal and the plan and all of the other things that are happening

in an agent so that you can create this little tree of from an incident to go back to the trace to figure it out where the regression happened and then deploy it better next time around. Um so I think the thesis here is that if you cannot replay it, you cannot improve it. Because um it worked yesterday is a real problem now when all of these systems

are fuzzy and non-deterministic. And so that means it's all the more important to have traces, to have intentional logging, to have um failure modes that we didn't have in the prompt So for example, one of the very basic use cases is like I have a meeting on my calendar, help me prepare for it. In this, you would first say, "Okay, I'm an agent. I'm going to first

start gathering all of the right context. I'm going to use some Google Workspace API or some ICS API to read your calendar and then I'm going to look at your LinkedIn of the people who are there, maybe some meeting recordings in Journaler from last time." So I I first gather all of the right context. I identify what is the missing context. I give you a little agenda.

I ask a human in the loop for approval. And then I can maybe update whatever system I have. And then at the end I'm going to say, "Okay, here here's my entire loop over. This is what I did for you today." Um so, for us what we learned the hard way doing all of this for so many years is that the model is no longer the product.

Boundaries are really important. Tool design is basically design. Uh eval's obviously need traces and trust is earned, not assumed. What's really fun is that memory went prompt engineering relevant rag stuff from some vector database, let's say that was the case a year and a half ago, much more opinionated graphs and namespaces and tools. Tools became like a new kind of API, but for a new kind of

user. Uh so, not for a traditional rest user and not for a human user. We have now a new internet user. Trust became guardrails, approvals, audit trails, all of the things that we're already familiar with. So, it's kind of funny that autonomy is now becoming durable workflows and eval's are also becoming trajectory checks. So, interesting that year of agents made us build traditional software engineering but for

this new And that means that everyone is inde pendently discovering that agents are the same problems we solved on a kernel layer in Linux like 30 years ago are the exact same problems we're solving now. Uh but with and the same primitives even, the same ports analogies even. Uh but for this new kind of user. So, the good thing is that we've done this before. Like many,

many years ago. Uh and so, we know how to make these systems reliable if we apply the same playbook again. from all of the things your agent can do, you want to focus on one real valuable workflow that you can agentify. You want to log everything that makes sense, all of the contacts, all of the tool calls, all of the approvals, human in the loop, failures. You

want to turn those traces into real evals so that each version of your agent is better than the last, and you want to have this trust ladder where your agents get better over time. Uh yeah, that's it. That's the talk, building uh reliable agents in production. Uh I think I have a few minutes, so happy to double click on any questions if helpful. And uh yeah, my

background is as a founder, like I said. Um I also angel invested in a lot of startups, also in this uh agent space. Uh so all of this context is coming from my background as a founder, but also uh having kind of this bird's-eye opinion on what the current batch of YC startups, for example, is working on. Uh yeah, that's it. Thank you very much. >> [applause]