Dapr in the AI Era: Orchestrating Complex Multi-agent Workflows With Automatic... Yaron Schneider

In this talk, Yaron Schneider, CTO of Dapr Grid and co-creator of the Dapr project, discusses the challenges of making AI agents reliable in production environments. He emphasizes the importance of state management and durability, particularly when agents experience failures. Schneider introduces the Dapr project, which provides a set of APIs that streamline the development of distributed applications by allowing developers to focus on business logic instead of infrastructure concerns. He highlights key Dapr features such as durable agents, workflows, pub/sub messaging, and state management across various databases. The talk also touches on essential aspects like observability, security, and resilience, stressing the need for reliable execution in workflows. Finally, Schneider showcases a demo of Dapr in action, demonstrating how it efficiently handles agent interactions and maintains state even through disruptions.

Yep. Okay, let's get started. Everyone, thank you for coming to my talk today. My name is Yaron Schneider. I'm a CTO and co-founder of Dapr Grid. I'm one of the co-creators of the Dapr project together with this fine gentleman here um called Mark and uh steering committee member and also very recently elected as the chair for the workflow working group inside of the agenda KI Foundation. That

was a very long intro, but what I really like to do is make agents run reliably in production. That's what I do nowadays, believe it or not. Um I spend most of my time talking to people and trying to explain to them why it's a bad thing that their agent loses all of its state if it goes down and how much money they're going to have to

pay when it basically picks back up and it's going to have to go through those 100 LLM calls over and over again. Um so today we'll be talking about ways in which the Dapr project can help you basically work around through all of those limitations that we see in um in risks when it comes to AI today. So if you don't know Dapr, Dapr has been around

since um the beginning of 2019. We donated it to CNCF when we worked at Microsoft um Mark and I and it became a graduated project in 2021. Um it is a set of APIs for developers to be able to focus on business logic and not infrastructure gluing and plumbing and interaction. It contains many building blocks and APIs today um and definitely the one that attracts the most

attention is the workflows one which is in the uh top left there. We cannot talk about all of them today. We have a very short session but today we'll specifically be talking about how those workflows really play into agenda KI and AI agents. Um you can come into Dapr from any language. So as long as your language can talk HTTP, you can uh use Dapr. You can

run it anywhere on your local machine, uh Linux, Mac, Windows. You can run it in any cloud. You can run it on a VM if even if you want to. It's fairly lightweight um and yes, it will allow you to essentially just hand over to Dapr all of those hard distributed systems challenges you to deal with the thing you want to do the most which is to

deliver software. We are a very big community. We're coming up at 9,000 Discord members. We have almost 5,000 individual contributors which is great. All the rest of the metrics here are vanity metrics that we like to show because they're nice um and they just make the slide more dramatic, but really the thing that we care about as maintainers are the contributors and our Discord community. It's thriving.

Come join us. Um there's lots of really interesting talks there about distributed systems problems in general and yes, you can talk to me there, too. My handle is Yaron 2. That's pretty much my handle everywhere since I was 13 years old. So if you see Yaron 2, if you're playing online games or something, it's probably me. Okay, Dapr is a foundational AI infrastructure. Um all of those

building blocks I've shown you, what we've come to see in the last uh year or so is that many of them can be used to make AI a whole lot easier and more secure and reliable. Starting with workflows. Um we'll see that in practice, but if your agent now kicks off an agentic workflow and that workflow consists of 100 steps and it fails at step 99, you

really don't want to have to replay from the beginning. This is where workflows come into play. And then pub/sub. Let's say you have two agents. Let's say they're LangChain agents or LangGraph agents or CrewAI agents. How do you make them discover each other beyond a single process? Of course, this is KubeCon. I assume we're all running in Kubernetes or at least some of us are moving towards

Kubernetes and you know, we know the notions of pods. There may might be agents running in multiple pods. Those agent frameworks out there today are focusing on everything that is not distributed systems problems. They focus on the reasoning, the uh tool calling capabilities, the LLM abstractions, but not on really, really important things like how do you discover another agent wherever it is and how do you call

it securely and reliably? Um Dapr pub/sub allows you to do that and also do it in an at least once guarantee. So if any one of your agents goes down in a multi-agent system, you're always guaranteed that this other agent will process the message when it comes back up. State management allows you to save state in more than 30 different Dapr supported databases. For example, let's take

LangGraph again. I like picking up on those guys. Um if you save data using LangGraph, you can pretty much use three databases out of the box today. We're talking about Redis, Postgres, and any memory um SQL database like like SQL, I think it is. What if you're using DynamoDB? What if you're using Azure Cosmos DB? What if you're using Cassandra? Um what if you're using, you know,

a Hasura? What if you're using a a cache that where you don't actually need like persistent storage? With Dapr, by integrating Dapr with those frameworks, you can actually save all of your data in more than 30 different databases. So even in the future if you want to move a database, um you're not going to need to uh replace your code and go on a long, long-winded journey

to rip it out from your code, pulling in another SDK, another client library, and then doing it. Conversation API at first glance, it might seem like an abstraction over LLMs and it is kind of like what LangChain did in the very beginning where you have a library and you can choose whether you want to talk to Mistral or Anthropic or OpenAI, but Dapr provides some really nice

uh additional enterprise features like PII data obfuscation without relying on any external service. So if you have any credit card numbers, names, um addresses, anything of that sort, Dapr can make sure to obfuscate the data when it goes into the LLM or even when it comes out of it. So if you're for whatever reason uh hosting your own model and running it on a GPU, which I

personally don't think a lot of people should unless they really have to, but if you do do that and you cannot sanction the data that went into the LLM and it might spit out information that could be given back to the caller that you don't want it to see, um Dapr will make sure to examine that and obfuscate the data when it comes out of the LLM

as well. So on top of those, we also have observability, security, and resiliency. Again, uh a lot of scope, but Dapr can apply access policies. So through Dapr, you can choose which agent can talk to each other agent and which agent can talk to this MCP server and not to that MCP server. And you can also create trust domains um where you have multiple agents that are

trusted inside of a boundary and then another group of agents which are not in that trusted boundary and so those two groups just can talk to each other. And so now you're really coming into the world of governance. Um how do you govern agents and MCP servers, the MCP servers they're interacting with at scale? So all of those building blocks can be taken today from the Dapr

open source project. Um you can bring them into your AI workloads and they will uh do wonders. Now the risk factors we're seeing uh in the industry today, my claim is they're not really all AI problems. There is one major issue that AI introduces um and that is non-determinism because we are humans and we like to think we're rational and by rational I mean if we suppose

we have the same input and we put it in the same processing function, then we're going to get the same output, right? Not the case with an LLM. That thing is like a five-year-old's brain on an ayahuasca trip. That is not going to do well for many, many use cases. So how do we deal with that level of uncertainty? Um with with that, we also see a

pretty big attack surface. Why? Because we have MCP servers. What a great invention, right? And we're giving those MCP servers access to our most precious prizes, our databases, our CRM systems, our APIs, and we're essentially like, "Yep, take the credential, talk to it, talk to the LLM client wherever that may be. I don't know. Um and then just go off and return the information." But if you

now if you get a breach inside of your system, it's much easier for malicious actors to work horizontally or do lateral movements within your system and exploit. You just have more to And especially when you have agents talking to an MCP server on behalf of another agent that he's talking on behalf of a human, how do you even begin to audit that whole process and see who

needs to do what and and also enforce that at runtime? That's difficult. Um but the utmost uh important thing that that we're seeing and talking to companies about is reliability because that pretty much underscores everything. Um and just to give a simple example, this is um actually an example where uh Nvidia is using Dapr internally um to basically take um let's call them modern rag pipeline and

make it reliable. So you have an agent that can process 1,000 documents and this agent has been simplified. There's more steps to it, but let's say we have a planning step and a classification step and um an examining and and saving step. Um you might have 1,000 agents running 1,000 different workflows and so you have hundreds of thousands or millions even of instances of those workflows running.

But what happens if those fail? Um let's imagine that, you know, uh AWS went down. What do you do then? All of your agents, all of the running workflows, everything's going to crash at the same time. Um let's say you have a bug or the LLM returned a response that you didn't expect and that's going to make your process crash. And then if you're at step 999

out of 1,000, and yes, that might happen, um it's going to need to pick all the way from the beginning and you're going to be paying for all of those LLM calls. And if you're using your own GPUs and your own models, you're basically going to be hugging the system and everything's going to be backlogged and that's bad. So we've introduced last year a framework within uh

Dapr called Dapr Agents. It was contributed um by Nvidia to the Dapr project and those are essentially durable agents. Um meaning they do the same thing that you'd expect any agent framework to do, but they are durable and I'm going to show you a simple demo that shows one of the most complex problems right now um and how it works at scale. It is cloud native meaning

Dapr runs natively on Kubernetes, so Dapr Agents run natively as well. The only thing you need to do is do a Docker build, push your container, and you're off to the races. There is no additional infrastructure needed except for a Dapr and your database of choice, which, as I mentioned, it can be over 30 different databases. You don't have to settle for the regular Postgres, Redis, and

the in-memory SQLite implementation. Of course, it's vendor neutral, open source with the CNCF. We're actually looking how we can take Dapr agents now and contribute it into the AI IF, where we think it'll find a better home. Um and it's stable 1.0 since yesterday. So, we've announced that. Um we've got people using it in production. It's a great framework. And after I talked to you about Dapr

agents, we're also going to see how you can use a lot of the features the Dapr agents gives you in other frameworks. Because you might not want to move to Dapr agents and change your code if you've already chosen LangChain or LangGraph or Strands or OpenAI agents. So, what we want to do is really bring all of those Dapr benefits into the existing ecosystem. Okay, so let's

see a demo. Um I don't know if you're in the keynote today, but uh we saw a lot of complex demos, you know, drones and people trying to talk to their machines and and get, you know, LLMs to write their code. I'm going to show you a very, very, very simple demo. But that demo is going to show you something the Dapr does that nothing in the

world can do today. Um and that is keeping your systems reliable and uh safe in in the face of pretty much any form of outage, interaction, or uh failure of the that you can imagine. So, this is what we call a durable agent. It looks like pretty much any other agent framework. Uh it has a name, it has a role, it has a bunch of instructions. Um

it has an LLM provider, you can switch that out to be uh an OpenAI provider, um Google Gemini. Dapr also has a conversation APIs, so you can plug that in as well. In this case, I'm just using OpenAI. Um and we've gotten a agent memory. This is essentially where you save your state. Um and that's the only thing you need to do, really. So, this conversation Dapr

state memory, it gets the name of a store name, and that is essentially declared in YAML. I'm going to show that to you later. And agent memory can be a type of any of the 30-plus different supported um databases that Dapr supports. And then you can also migrate state. Um another really cool thing that Dapr agents does is it connects agents for you without you needing to

know anything about where the other agent lives or what it's doing or under what kind of load it is, and it'll distribute traffic to those other agents for you. In this example, I'm just showing you a very simple demo um that uh basically takes the non-happy path. Right? Most demos take the happy path. We're going to we're going to kill the agent. So, uh I have this

demo here, KubeCon 26 AMS, and I'm going to go into my terminal. Let me just zoom in a little so we can see that. Okay. Um also, I will just say that the internet here sucks. Um it's very, very, very, very, very slow. It actually not working from time to time, just times out. Um and that maybe works well for our demo because we're showing reliability, right?

So, the agent is going to try to continue to probably talk to OpenAI, and eventually, if we see it actually succeed, it's going to show another part of Dapr workflows, the system that's underlying all of this, and that is not just making sure that you've got this workflow engine that knows when something fails and continues it. Um it also retries for you, and you can have uh

different retry policies. For example, you can put breaker uh circuit breakers if you don't want to overwhelm your underlying system. Okay. So, let's run this, and I'm I'm just uh using Dapr. There's lots of logs here. There's Dapr run. Okay, Dapr run is a Dapr CLI dev tool that allows me to run this locally, and um it has this nice command that allows me to chain my

regular uh Dapr agent Python application to my local sidecar. So, I'm going to have the Dapr process running locally. It's going to have a bunch of logs, and um I'm going to have my Python script that you saw earlier, just those few lines of code of the agent, and it's a weather agent, and they're both running at this time. So, what I'm going to do right now

is go back here, and we're going to curl. And it's going to be not this one, but this one, what is the weather in Amsterdam? And we are going to kick that off, and we're going to go here. Um but before it finishes, we are abruptly killing it. And this can be any form of failure. This is essentially um like a kill nine operation. There was no

graceful shutdown, nothing had enough time to clean anything up. It just it just died immediately. And now, if we kick the agent back up, and I'm not curling it again now, we can see that it picks up exactly where it left off. No agent framework can do this today. None will do this because, again, they are not focused on distributed systems challenges. They are focused on looping

and reasoning capabilities and debugging and evaluation. But you still need those really important workflow capabilities. Now, we can see that um it did well, and it also gave the answer, which means the internet was probably working this time. Um it might have been a little slower. And for example, if OpenAI is not available because they're throttling you or they're down or there's a network blip, the Dapr

workflow itself is going to make sure to retry. And as I mentioned, you can put your own retry policies. You can say retry for five times before you essentially declare this agentic workflow as failed. And then you can also use the Dapr management APIs to get the list of all of those workflows, and you can decide if you want to kick it off again. Um those agent

frameworks I mentioned earlier, they're doing something very different when it comes to persistence, and I actually want to deep dive into So, let let me go back into the slides. Seeing it? Yeah. Um my position today is that none of those frameworks are actually production ready. Um not for enterprise scale. Why? Because they don't give you the security and reliability primitives that you need. They're great for

prototyping, and most of the companies I'm talking to who are going into those, they basically go, "Yeah, you know, we we have a few dev environments. We're looking into LangChain. We're looking into LangGraph." And then I ask them, "Well, if those agents fail, and you're giving it full autonomy because, really, agents are giving you back ROI if you give them autonomy because they can automate more, how

do you make sure that they're going to continue where they left off, and you don't have to start them from the beginning? And even if you did want to start them, how do you go about doing that?" And then most people are like, "Hmm, good question." Um and this is exactly what we want to solve. And those top agent frameworks, they do something extremely simple, and this

is something that everyone needs to know. Um their answer to the problems we've been talking about um is checkpoints. Checkpoints is basically, you know, recording the output of every step, JSON serializing it, and saving it in a very, very non-efficient way to a database. But once they the data is there, and the agent comes back today, and you can take any agent framework today and hook up

their checkpointing mechanism and try it out, it'll do nothing. It'll just sit there. It won't pick up where it left off. Because checkpointing are great for, you know, getting the state into the database, resuming the progress, but then you also need to know when something fails, and when something fails, you also need the business logic to be able to make it recover. Um and that is something

that they hand over to you. And that means writing a workflow basically yourself, and that is all the hard parts of a workflow engine. Failure detection. You have a thousand agents with a thousand workflows, all of them failed. How do you go about detecting which step exactly failed and detecting which agent failed and when and on which cluster, talking to which system? And after you've detected that,

let's say you did that, how do you recover? How do you have a system that basically now goes through all of those checkpoints that LangGraph saved in the and have basically told you, "Look, you have the the the system here. These are the thread IDs. Go ahead and recover them." How do you do that? You now need a system that's going to need to do a a

bulk operation that will need to basically go sequentially or in parallel and call the LangGraph APIs to make all of those agents pick back up. But what if that process fails? How do you make that durable? How do you make that checkpointed? Because if your recovery system fails, those agents are not going to continue. And then you have multi-instance coordination. Right? If you have multiple instances of

that recovery system, try it out, LangGraph and those other other frameworks will allow you to continue the exact same instance of a workflow twice. So, you'll get duplicate executions with all of their very unwelcome side effects when it comes to agents. Um this is something that's pretty bad. So, Dapr workflows really guarantees that only one instance of every workflow is going to be executing. You get this

exactly once guarantee when it comes to the scope of the workflow, and that's extremely important. You don't need to take care of failure detection, Dapr will do that for you. You don't need to make uh a recovery process on your own. Dapr will do that for you, no matter how many agents or workflows you're running. And you don't need to be afraid that the same workflow instance

is going to be repeated twice. Something we announced last week as integration points um are extensions to the Dapr workflow engine, where we essentially provide um workflow integrations to all of those agent frameworks. Now, these are community-driven. You can come in, you can do them. Uh my company has a bunch of those integrations. I'm not going to show them now because they're not part of the Dapr

open source project, but what's important is if you have an agent framework that you want to integrate Dapr workflows into, it should be fairly simple, and um you can look at our code to see how we've done it. Anyone can do it. Anyone can come and basically integrate Dapr. For example, I've been talking to a company, very large company, um last week who have started writing an

agent framework on their own 2 years ago on top of Dapr, way before, you know, most agent frameworks even came to be just when the concept was was starting to get traction. Um and they built it on top of Dapr state management, Dapr pub/sub, and Dapr service invocation. Um and now they are looking into how they can use Dapr to really um kind of like commercialize their

own house-built agent platform so that other teams can use this as well. And those integration points that we have make it very easy to do so. Um last demo we're going to see today is a multi-agent demo because Dapr will give that to you, too. Um you in many cases do not want to put all of your logic in a what we call an agent monolith. You

want to spread them out where each agent does what it needs to do. Um and what we have here is an example for multi-agents. Uh by the way, go to our website. Let me just show it to you now. Um this is our repo, uh Dapr agents. It's github.com/dapr/dapragents. I'm going to show it in the slide so you take a photo of that later. Um this is

the repository. We've got quick starts here and examples. Um they are very elaborate. There is a lot of examples here that basically show you how you can do fully autonomous workflows with agents that are stateful, how you can do deterministic workflows that contain non-deterministic agents inside of them, how you can do multi-agent collaboration, and uh how you can switch between different LLM providers. So, in this demo,

um it's a derivative of an original demo that um one of our Nvidia maintainers put. And that is The Lord of the Rings. And we're going to have basically Sam and Frodo talking about um I think what's their provision like when it comes to Lembas bread that we know they all love. Um and so we can see that we have two agents here, and they look pretty

similar to the uh agents we saw before. So, we have Frodo Baggins, the ring bearer, and it has a goal to complete the quest to Mount Doom. And it's got a few instructions. Here you can see that this is how you would connect Dapr to be able to discover other agents and uh really do this multi-agent collaboration. Instead of you needing to pull in Kafka code or

Pulsar code or AWS SQS code, and then deciding on different topics and making sure that those agents can actually communicate, um all you need to do is just have these four lines of code where it'll get a pub/sub name, and that pub/sub name again is a YAML representation of what it actually is. Um for example, in this case, we're just using Redis, right? So, this is what

the the YAML to declare Redis would look like. If we want to switch that to Kafka, um that's um one of the few things I'd have to do. And the metadata would change here to be like a Kafka broker, but really that's it. And this is how we pick the infrastructure of choice that Dapr agents is going to use. So, going back into this, um this is

really all we have. They have a goal, they have instructions, they have a pub/sub mechanism to discover each other by. And so, Frodo and Sam will start talking about what's the best path to uh to take to the ring or about the Lembas bread. Actually, we can ask them any questions. It's pretty fun. You can talk to the fellowship. Okay. So, let me run this real quick,

and I'm just going to Everything is really zoomed in where I am, so I'm just trying to make it uh a bit easier. Okay. And we will go here. And I think Let's take it from the read me. All right, yep. So, this is the line. Okay. So, what we did now is essentially start um two Dapr processes running with those two agents, and they're running side

by side. Again, there are a lot of logs here, but essentially essentially two Dapr processes, one for Sam, one one for Frodo, and those um different agents are connecting to their individual Dapr uh sidecars. So, you get also full isolation, which is nice. Um so, we have those going on, and now we can start asking them questions about um what supplies do we have for the next

leg of the journey? Sure, this is what we have. So, we kick that off, and that is uh an asynchronous workflow. Um so, we're actually not waiting for a response from the caller. You can also do that with Dapr agents. You can wait for a response and then give it back to the caller. In this case, we're just kicking off an asynchronous workflow. Um so, if we

go back here, we can see that um they didn't do anything. Probably because this time I really do need internet connectivity to talk to the LLMs. Okay. Let's kick that off So, the workflow I imagine wouldn't even start. Oh, yes, yes. We have a port collision issue. From the Yeah, okay. Thank you. All right. Yes, port collision is By the way, I think there's a project solving

that. Okay, let's run this Let's see all the logs. Do we have any weird errors this time? Doesn't look like it. Um yeah, I think we're running. So, let's go back, and not this terminal, this terminal. Let's kick it. Okay, yep. And now they're operating. Um so, we can see that the user asked what the journey. Um this time uh Frodo has gotten it, and he's going

to talk to Sam. Um and now we can see on behalf of Frodo, the other agent got the the question. So, by the way, this is a good example of seeing on behalf of. Um you're actually seeing wh- which agent sent the original message. And when Dapr does it, thank you, you have all of the information. Everything is uh translated through open telemetry, so you can plug

in whatever open telemetry endpoint you have, and we will send all of those traces there. And let's find a few uh texts. So, here is Sam, and Sam is starting to say, "Here's a practical inventory for the next leg of the journey." The agent was told to act like Sam. Sam is very, very practical. Um and so, Lembas bread, it Sam likes to eat, so dried meats

and fruits, water, cooking gear. We've got the whole stuff here, and in the end the workflow ran to completion. We can see the final output here. Um so, this is essentially Frodo saying what Sam has suggested, but we can see that the agent who kicked it off was Frodo. Um and you can connect a hundreds of those agents together, all working collaboratively. There is no limit to

the amount of agents you can connect through that. The only thing the agents need are those four lines of code. They will discover each other, and um through Dapr agents, the agent that gets the initial prompt will know exactly which agent to pick to serve um the next call. So, even if you have like 50 agents, um the initial orchestrating agent will decide who's the best agent

to talk to initially, and then it will continue to pass those messages along, and it'll do iterative loops until the the mission is done. One last thing we're going to see before we wrap this up is just open up um Zipkin, which I have running locally. Um we can see that um we have those traces. Um these ran 2 minutes ago, and you're Let me just zoom

in a little. You're essentially getting uh everything that's happened. So, you get the full audit log, really important for compliance for everything that's happened. Um these are trace logs, so don't try to make too too much sense of it, but we can see the uh initial record entry, then we can see um the agent trying to get state, save the state, meaning the conversation state. So, if

the um workflow were to be crashed middle, when it came back up, it would have fetched the state with this get state operation, and it would have continued where it left off. So, Dapr agents, by the way, they don't only save where where the workflow was during execution, they also save the memory of the agent and the conversation history. So, you don't have to recompute that at

Um yes, and we get essentially all the traces here. You can see which step uh took longest, and everything is essentially detailed. I think we're out of time, so I'm going to open it up for questions. I've been told to tell you to use the microphone. Okay. Um thank you. >> [applause]