Automating Workflows with Multi-Agent Systems, Kristiyan Stoyanov

This talk explores the automation of workflows using multi-agent systems, a topic passionately presented by Chris, a principal engineer at Paysafe. He begins by defining what an agent and a multi-agent system are, then dives into the modeling of such systems to facilitate workflow automation. Chris shares insights from his own experience of creating an agentic system to automate code reviews with SonarQube in a short span of time. Different agent orchestration patterns, such as the orchestrator and swarm patterns, are discussed for tackling both linear and complex tasks. The presentation includes practical coding concepts and emphasizes the importance of defining boundaries and utilizing specific tools for agent functionality. Additionally, Chris encourages the audience to experiment with creating their agents, mentioning relevant Java libraries that can facilitate similar automation tasks.

Hello everyone, and I hope you haven't had enough of AI because welcome to another AI talk. We'll be talking about automating workflows with uh multi-agent systems, topic very close to my heart. So, yeah, let's get into it. Few words about me. I'm Chris. Uh I'm a principal engineer over at Paysafe. I have around 8 years of experience as an engineer. Um and uh also I'm a very

early enthusiast into the LLM and agentic space. I remember when GPT3 GPT3 came out, I started uh I was like, "Oh man, we can do so much with code and LLMs." And uh yeah, I even started coding agents at that point. They were very rudimentary, nothing like I will show you today, but still. So, as I said, topic close to the heart. On the agenda, we're going

to discuss what is an agent just in case you missed the other four definitions of what an agent is. Then we'll talk about what is a multi-agent system. We'll and then we'll delve delve deeper and focus on how we actually model such uh systems so we can automate our workflows, which is really the meat and potatoes of the talk. Then we'll have a quick demo, and uh

yeah. We will uh get you started on your own AI agentic journey. But this is like the technical agenda. Really, what I want to accomplish with this talk is uh three little things. Uh I want to show you something cool that got me uh personally inspired. To give you this motivation, inspire you to try this on your own because I really believe that everybody in this room

has the necessary skill sets to try this. And uh to give you a practical framework, so you can get a flying start. So, this is what we'll be talking about today. we'll begin with a story. And like every good story, this story begins in Vegas. So, I had the the great opportunity to be at like one of the probably biggest IT conferences. And really there the awesome

stuff was the air of motivation. Like so many engineers there just hustling around trying to learn new things, trying to see what's interesting, what's new. And this is really, you know, big and motivational. At least that's the impact on me. So, I remember I was at this talk, which again about multi-agent systems. And after the talk, I just uh couldn't continue on with the other talks. I

was like, "Okay, so uh I need to do something about the knowledge that I just got. And what I did was I just sat down on the conference floor, pulled up my laptop, and started vibing right away. So, I received some knowledge, a gift, let's say. And I needed to use it. So, I we need to automate something." And I was like, "Okay, what's the most boring

thing that I can actually get down and automate? Like what's the thing that I probably hate the most in my everyday job." And then I remembered that we have SonarQube. So, >> [laughter] >> SonarQube for probably everybody working in enterprise knows what SonarQube is, but you know, just for completeness, SonarQube is a tool which runs like analysis on your code. It detects like bugs, code smells. You

can see they're separate by severity. And I promise [snorts] you this is on the backlog of our projects. Trust me, it's fine. So, I was like, "Okay, let's do something about this. I don't want to do this manually." And yeah, I really sat down. I pulled up I think it was Cursor at the time. And for 30 minutes I actually had this little agentic system that was

able to make the changes and submit an MR with the fixes for sonar. So, I was left there wondering with if I could build that in 30 minutes, what could what could I build in a week? So, this was really the spark and motivation for me to get ready and start building on this. But before I actually go a little bit deeper into what I actually did,

because this is like the topic of our talk, these 30 minutes of magic happening in Vegas, we'll go through some theory. And we'll begin with something very basic. You've probably seen like three variations of this diagram today, but it's important. And that is what is an agent? And everybody is adding their own nuances on this, but to me the important stuff to understand is that an agent

is not an LLM. We I think we hear it often that people confuse like Opus with actually Cursor, let's say, as an agent. An agent like has an LLM, but it's not itself the LLM. So, if we think about what an LLM is, right? It's a prediction machine. It's just like, what's the next token going to be like? And the chat that we see with let's say

chat GPT, right? It's actually a trick. It's like structuring your your prompt or your question into such a way that the LLM actually predicts what the answer would be to your question, if it's a question. because of this, uh are stateless and they cannot do anything, really. They can just give you output based on your input. But, to be able to do something useful, you need to

execute stuff. You need to do side effects on uh on the system, right? So, that's why we have tools. And for me, a tool is best described as uh the uh like a function. Like, if you can write a function, uh that's a tool. And this it's a function, more specifically, that can be uh executed uh using an LLM. Like, an LLM is aware that it has

this capability. And this, like on the diagram, is actually the so-called agentic loop. So, >> [snorts] >> you get some input from your user, right? Then this gets hits uh it hits the LLM. It says, "The user is asking this, and these are your tools, which you can use." Then the LLM replies with "Okay, use this tool." Uh then the program parses this. Uh it selects the

tool, executes like the function, really. And based on the output of that function, this gets added to the context. And then you repeat that until the LLM says, "Okay, task complete. Here is your response." It's massaged a little bit and returned to the end user. So, this is like the framework of uh how to think about an And now that we know what an agent is, uh

let's see how we can build one. Just one note, I will be showing some Python today. I know, Java conference, big sin, but I think it really exemplifies the concept here. Uh we had other uh great uh lecturers, and I also like have some stuff at the end to just give you the tools, but uh really here it's let's focus on the concepts themselves. So, here this

is like an example of strands. It's an Amazon SDK and we can see like an agent definition which is very very small. It actually has three main properties. It has a model a list of tools, a set of tools and a system prompt. And actually I think this is a very good model of what an agent is, right? We have the LLM, we have its capabilities and

its persona let's say. The system prompt it's pretty much this that tells what the agent should be doing, how it should be acting. So it's very important and then if we want to invoke the agent we can invoke it with any prompt whatsoever. So this is one agent, right? And how do we get multi-agent? We just connect them together. I know, very obvious. Uh but there are

a lot of subtleties let's say and details that we need to pay attention to when we are actually building such a multi-agent system. So this is actually what we're going to be focusing on. How do we model such a system? And specifically for a use case where we want to automate a workflow. And since we want to automate a workflow, we have to start with the workflow.

pick something like that you do every day let's say, something monotonous let's say but that you want to be automated. And break it down into steps. So if we're using our SonarQube example, let's [snorts] see what would our first step be. If I had to sit down and fix this, as a step one I would clone the repository so I have the code local to me and

I can work on it, right? Then I'm going to see the reports that Sonar is giving me like it's telling me here you have a null pointer exception chance or you created like 13 ifs in this method which is too much. Stuff like this. Then we, based on that input from Sonar, we are making the changes that it's suggesting. So, it's rather straightforward. And then we uh

have to do some tests. Uh usually this is automated tests that we have in our repository or our CI pipelines. Uh but yes, we do need to have some sort of uh uh verification. And uh lastly, we need to commit, open an MR, and uh this is where pretty much automation potential ends because then we have to have uh human review. Okay. So, we have these five

steps, general five steps, that we're trying to automate. We have to think and place them into one of two categories. Either it's something we can automate, we can write a function to do that. So, cloning the repository is such an action, right? We can write a function that just gets a parameter which is the repository, and we're going to pull that. Check the report is also like

from Sonar, it's also an automatable action. We can uh write a function that it's going to uh call, let's say, the Sonar API, and we're going to get this chunky big report, right, of all the things we've done wrong. And then we come to a little bit of an interesting step, right, because uh making the suggested changes you can't really write a function uh to do that,

right? If you're using deterministic and regular algorithms, you just cannot do that. You need reasoning here. And this is first the signal that uh is telling us that okay, this has potential for uh an an agentic workflow. And then just for completeness' sake, you can test the changes again, automated. That's why we have suits, we have unit tests, integration tests, whatever. And committing the changes and opening

an MR is also something deterministic that we can do. So, based on this, like if we take a look at the whole five steps, we can four tools, let's say, for cloning repositories, committing, opening MRs, checking sonar, all that stuff. And we can just place them into uh a single agent and make it run. And actually, this is the first thing that I did uh back in

Vegas. However, I soon came to a little bit of a problem. It just didn't work. It got confused on uh what it should do. Uh and I realized that, okay, I'm I need to do something a little bit differently. And uh it has to do with boundaries and Uh so, when we were discussing what an agent is, right? We have uh these uh we have this the

LLM. The LLM itself has some uh limitation on how much context it can hold. And uh even if we don't reach the maximum of that context window, we still get context rot. The the like the responses start to degrade because you're giving it a lot of instructions, and it's getting confused, and it can't really work. So, really, like the way I imagine one agent with like uh

10 million tools, right? It's something like this. It's uh wonky, doesn't go fast, it's a little bit uh useless, let's say. So, we don't want We want to bu- to define proper boundaries, isolate the context, because like each tool is added to the context, right? Each of the responses of the tools is added to the And we have a way to actually circumvent that because we can

group tools into separate agents. The agents are going to get this big response and then they can just summarize it and return that to another agent that is actually calling the shots. So, such a system I imagine to look more like this. Small agents, very specialized into their own tasks, which are coordinated in some way. And this is what we want. As a rule of thumb, we

can say one agent, one job. And let's be honest, this isn't anything new in computer science, right? We had single responsibility principle for how many years now. So, it it makes sense. So, we are going to be doing small agents that can do their job, but they can do their job very well. And we have a few ways of orchestrating, let's say, this team of agents. And

for that we need to choose the right pattern. Here on the right we have, you see this like big master agent pulling the strings. This is the orchestrator pattern, which is using the sub agents which are specialized and it is orchestrating the workflow itself. This is great for more linear workflows such as our sonar example. We saw it like there were five steps. Obviously, in those five

steps if your test fail, you go back to step three and then you retry after you fix them. But it's more or less linear. So, it's a good fit. The other pattern that I think is very interesting, it's called the swarm. You can see it's like a neural net, right? Of of agents. And if we know about emergence properties like the sum is greater than each of

the individuals, It's a very interesting concept, but it's mostly useful for higher-order tasks. Like you would have a lot of specialized agents, let's pick a random number, 10, let's say. And it needs to be like higher-order tasks, meaning something that is not so defined. For example, delivering a software feature is such an example. You would have and you can structure the swarm in a similar way that

you're structuring software engineering team, right? You let's say a product owner, you would have an architect, you would have a tester, you would have a reviewer. Like this team, right? It's collaborating and it's producing some sort of a result. So, these are the the two main patterns, I think, currently. There are others, they're more, let's say, programmatical and rigid. I for multi-agent systems, I really think that

these currently, at least, are the way to go. define an orchestrator? Here we can see something like again, very simple code. That's why I love this because it's easy to visualize what it's doing. We have the definition of our agents on top, and then we have one agent that wait, we look at the tool section and it's listing the agents in the tool section. Why is that?

That is because we are using, let's call it, agents as tools. And when you think about it, a tool by the definition I gave you earlier is just a function that can be invoked by an agent. So, you can obviously write wrap each agent call into a function, right? So, it makes sense that you can actually chain them together like this. And here in the orchestrator system

prompt, now this is just some very generic system prompt, so it fits on a presentation screen. However, uh you here you would define like use the sonar agent to accomplish this. Use the coding agent to fix what you receive from the sonar agent as a summary. And uh use uh the Git agent to, you know, finish up your work, let's say. Uh so, really crafting a good

system prompt is of utmost importance. But, this is pretty much the pattern. Like, you can get this, you can invoke the orchestrator agent, and uh you're you're uh happy to go. This is actually what I did uh with the with the sonar fixer, let's call it. Now, for completeness' sake, let's discuss the swarm again. We have our three uh agent definitions, and we create a swarm object,

which uh uh just lists these uh uh these agents, and we define an entry point. Now, this isn't all the parameters. Again, for brevity's sake, I have uh shortened it a little bit. But, the idea is that this swarm it gets into one entry point, and then they start passing it around and coordinating and cooperating by themselves. Uh here it's important to set some limits, like uh

number of times you can do a cycle, let's say, between two agents, or um you know, just general like uh circuit breakers, just so you don't get some infinite loop. And obviously, too, uh here the prompt is important, so you uh what the goal of the swarm is. Okay. So, now we know the patterns. It's time to design the architecture for our little sonar automation fixer. it's

pretty much that simple. That's what I actually ended up with uh when when I was live coding this on the floor. Mhm. We can see that uh I have defined three agents which have tools and an orchestrator which orchestrates the three agents. So, we have a Git agent which can do Gits uh you can it can use the Git CLI and also it can use the GitLab

API. We have a Sonar agent which can query the Sonar API. Now, here important caveat, you have MCP, right? So, Sonar actually exposes an MCP server, so you may not need to uh add a separate agent. Or uh so we would think. However, so here it's more of personal preference, but I prefer to only give it like the tools that it needs, so only query the APIs

that it needs and obviously bound this into an agent because the Sonar response and the API is like this huge JSON with pretty much everything about the code base, like what the old version is, what the new version is, what the smell is, how it is categorized, the severity, how to fix it, suggestions, return links, and everything like this. Each and every of these characters that are

in this JSON are going to flood your context. And this is just to reiterate the important point of defining the boundaries. So, uh by actually writing a custom agent and just giving it the the one or two tools that it needs to get the job done, I believe it's uh a much better approach. And last but not least, we have the coding agent. This is our reasoning

guy. This is the guy that's doing the heavy lifting. Uh everything else is just function calls, but actually when you think about it, you the way you write a coding agent, like a very rudimentary coding agent, you need an LLM, a good system prompt again, and also tools to read files and update files. And you know, maybe run compilation and stuff like that. Uh actually, I had

one more agent which was doing verification. It was running, let's say, Gradle tests. But for our sake here, I think this is enough. So, this rudimentary rudimentary actually gets the job done for something simple like sonar issues. If you get more complex, an interesting thing you can do here is uh you don't need to write a coding agent by yourself. We have great agents that are developed

by awesome companies, right? We have cursor, we have Claude. And what you can do, most of them have a CLI. So, you can create a function that underneath calls the uh Claude, let's say, using the prompt that you give it. So, you can actually plug other agents here. It doesn't need to be your own strands defined agent. And then you can do much much more complex tasks,

but also it's going to be cheaper. Uh no, it's the opposite of cheaper. It's going to be more expensive because you're going to be using a lot more tokens. But for now, I think we're still in the early stages. We're still experimenting. And uh probably by next year, uh you would have racked up such a bill that you would need a talk on how we we do

cost saving in AI. So, we need to have some continuity, right? Okay. So, this is our architecture. This is what what I came up uh drinking uh Cola Zeros for 30 minutes. And uh yeah, as I said, I I actually don't write Python often or never. And uh using just vibe coding can any of the coding agents was sufficient because the the syntax is actually very simple.

So, you know, uh I suggest playing around with it. >> one other thing that we see here, right, with these boxes and arrows between them, is that this starts to resemble something. And uh it's pretty much like a microservice architecture. Even when you think about it, like for microservices, we have orchestrated and choreographed our architectures. So, really this isn't anything new. We're just reusing the already proven,

let's say, patterns that we have in other parts of the software engineering field. So, really it's just microservices for agents. This is like the the bank uh headline here on uh yeah, how to design uh multi-agent systems. So, generally most of the rules that would apply to microservices would also apply uh to this, again, with some nuance. Okay, so I was talking for now quite some time

on uh what I did, right? Almost 30 minutes on what I was doing, but I didn't show you what I actually produced. And honestly, this is really 30 minutes of just hacking around. I haven't modified the project at all. I just spun it up and did a recording. Sorry, it's a recording, but uh I have a lot of like sensitive API keys here and there. It relies

on a lot of internet connections and stuff, so I wanted this to go smoothly. if we we Is it visible the terminal? Probably not so much at the at the back end, but I will narrate what is going on. So, here we see a terminal and uh I am going to just execute like the Python command that starts the agent. And I also added pretty much every

log that I could think of, just so we can illustrate what the system is actually doing. So, here I have my prompt. I'm saying fix code smells in the project. I'm giving it like the project key. Check fresh information from SonarQube. make the changes, validate the project can be built, and open the merge request. This is the instructions we're giving to our pretty much this is going

to be propagated to our orchestrator. Okay? Now, if we resume, we're going to see that we get agent is processing for some time. And we see, okay, it invoked the first tool. It invoked invoked get SonarQube issues. It says, "Okay, I found 21 code smells." So, what happened here? Just to narrate it again, the orchestrator was like, "Okay, I have this information. I need to see what

Sonar is saying." As you can see, this is not deterministic, probably an important note, because like in the workflow, I actually defined that you pull the repo first, but it decided to actually look for Sonar first. You can see these are two independent tasks. You can parallelize them. But anyway, so what happened is this got to the orchestrator. He was like, "Okay, I need information on what

Sonar is saying so I can fix it." It invokes its Sonar agent, which underneath has a tool which is get SonarQube issues. It sends an API request. It receives the response with all the all the smells, bugs, the things that need to be fixed. This gets This fills up its context. It gets summarized into this file, you need to fix this. This file, you need to fix

this. It skips the reasoning on why changing this or why is it better, etc. So, this is the only part of the context that gets back to the orchestrator. This really keeps the orchestrator clean, so it knows where it is and what it's doing. Okay? And now we see we have also search GitLab project tool call and get clone tool call. So, now it's pulling based on

just Keep in mind, I actually didn't give it a URL. I just gave it the name. So, it was, you know, not even that deterministic. We use the Git agent to create a branch. And now we started to Now the orchestrator, so after it got the summary from Sonar, then it was like, "Okay, we need to pull information from from Git and GitLab." Uh we pull the

repo, everything is done. So, we can start working. It gives the summary that it received from Sonar to the coding agent. And the coding agent starts working its magic. It starts reading files here. It has summary of the issues because this was propagated through the And uh yes, it's just doing analysis and fixes. I'm going to skip over that part because it's fixing And this is just

rudimentary like replacing files. It's doing LS grep and cat stuff like that. All right. So, now it says, "Okay, uh we're finished." So, for each issue, it's pretty much uh So, this Sonar issue, coding agent go fix it. Coding agent fixes it, fills up its context, returns the summary, "This issue is fixed." The orchestrator, let's say, marks it as done. Then the coding agent's context is flushed.

And then we receive the next issue that it needs to fix. Okay. And now we see it does Git status just to check what the diff is. And then it does Gradle compile. Sorry. Uh okay. I accidentally re- reset the video. Okay. We see now it does Gradle compile. Again, this is a good idea to be separated out in a different agent just because the Gradle output

could be huge, especially if there is an error. But, if you do it in a single agent, it's going to read out all the output, let's say, or pipe it into a file and read specific lines of it. Then, it's going to do a summary and going to return it to the orchestrator so it knows what is going on. So, we see excellent compilation was successful. Now,

let me run test with coverage. Oh, yeah, because I added to the system from that we need to have coverage like 100% branch and line coverage. Okay, we see now that a Gradle test has passed. We have two invocation of Git diff. Uh we It's for some reason it decided now to check the coverage. Okay, it says 100% coverage. That's nice. Uh oh, it's checking the file

bar file. Okay, that As I said, it's non-deterministic. It can do whatever. Okay, now it does Git add. Git commit. This is now the Git agent working again. Okay, and we have a Git push. Now, it's going to invoke another tool that is called create GitLab merge request using the GitLab API and the branch name, we can create the merge And with this, our workflow would be

mostly complete. Ignore the window overflow error, that's fine. Should have said that you wouldn't have seen it either way. So, here it gets it gives a summary of what happened, and we see here we have a merge request created. So, I'm opening the link live now, okay? And this is what it actually created. We can see it here in GitLab with a summary of what was done.

it actually worked. I I saw the MR. It was perfect. It fixed everything. Pipeline was green. Uh it was wonderful. Now, this is obviously a small use case, and it's probably not something that it's worth its tokens at this point. However, it's a very good entry case because this gives you like pretty much all the tools and concepts that you need to actually model harder to harder

And now, because we're at the Java conference, and I was showing only Python, uh we need to talk about like uh what the Java community or ecosystem has to offer in terms of similar libraries. Actually, it's very simple So, Strands is actually built on like on LangChain either way. So, we have LangChain for J. There was an amazing talk earlier today, which showed pretty in-depth, I would

say, the features of the library, and you can see even the syntax is quite similar to to the Strands. So, whatever I showed you, you can do as well. Sometimes might be a little bit verbose, but come on. It's Java. It's fine. Another good alternative is uh the Google AI development kit. You can see here this is like with the builder. This is more similar to uh

to You also have Spring AI that you can use, and the the community is growing. So, uh obviously with uh yeah, with uh Python and AI and AI, it's really the the first place things land. However, I'm pretty sure that uh if like these will mature very very soon. They are kind of mature. So, uh yeah, you can do uh what I just showed you on your

own. You can do it uh using the Java stack if you if you prefer, if it's more comfortable for you. That's That is absolutely fine. Uh Um, just maybe a note here on the clients. The good thing is uh really that all of these uh SDKs or libraries, they uh give you very nice abstractions with multiple provider clients. So, it's really just configuration. You're just saying what

model and the API key that you have. So, yes, it's something very, very uh easy. as we talked in the beginning of uh the talk, when I was saying my goals, it was get you to actually try all of this on your on your own. So, let's talk about your next 24 hours. And uh again, I've written it out into So, step one, open your favorite coding

agent, you know, cursor, Claude, or if you're hardcore or old school, you know, write the code yourself. It's fine. Uh then, install Python or any of the JDK alternatives that uh that I showed you. Check their documentation, ask around the the chatbots themselves uh on uh what you should be doing, just so they can guide you. And then, really this is the most important thing. Really think

of something that you do very often, that you do daily even. It's uh uh it's important that it's something meaningful to you, but it's also it's something small to start with. put your like task in your mind. Uh actually, you know what? I'll just give you some examples. Probably fixing vulnerabilities is a is a good uh starting point. Checking the news, weather, or just giving you like

a report of, you know, data ideas. Um you know, giving you stock news if you're into the stock market. Like all of these could be like probably the last one is a little bit more complex. Although, these hedge funds are trying uh to do it, and probably you won't be able to do something amazing on your own laptop, but still interesting use case. The The important thing

is that you understand what it takes to to get the task done, that you're an expert in it, and it did something that matters to Then create your first agent. You don't even need to start in the multi-agentic system like defining all of this. I really advise you to create a single agent and see how it doesn't work because its context gets flooded or maybe you're just

using a very nice model, but either way create your first agent and then run it, test it, iterate on it. You know, you see where it starts breaking. Maybe you need to do better model selection. Another great like benefit of multi-agent systems is that you could use different models for each of your of your like sub-agents. So, you could have either fine-tuned specialized models or you can

use cheaper models to save on cost. And yeah, just do iterations, use the concepts that we covered today to design your systems. Probably I gave you a bit of the pitfalls. hope that uh yeah, you found this interesting and are willing to try it out yourself. So, here I have a QR code and a link tree just for some useful links. Here I've added like documentation on

on strands uh or and and also like there are very nice repo by Amazon about their agent core. It's like a platform to build agents like completely like not on your local machine, but in Jupiter notebooks in the cloud. It's nice and it's easy to follow, so you lot of useful stuff And yes, also you can follow me on LinkedIn or after this talk I'll be probably

outside drinking a beer, so come say hi. I love talking about this topic, so yeah. And uh yeah, we can uh whatever you want. >> So, let's put the thank you slide. uh probably we have some time for questions, I think. So, any questions? Anybody? It's a little hard to see with the projectors. Oh, go ahead. Yeah, sorry. Speak up a little bit. Uh yeah, yeah. >>

So, which is when you write agents and sub-agents, how do you know that they are working? >> Okay. Yes, okay. So, uh just to recap, the question is how do you actually understand that the things you did are working? Uh so, usually like first iteration, I would suggest you just start with logs. Uh most of these uh uh SDKs and libraries are going to provide you with

a lot of metrics on exactly what was done, and you can trace it. Uh then later on, you can start ingesting this data and actually tracking it in maybe like a dashboard or some more sophisticated system, but this is like later on. Usually, this is like to check what it's actually doing. Okay? And uh if it's working, if you're happy with the result, is a yes. >>

Is there such kind of framework like the unit just for testing? >> Mhm. So, it's difficult to have something like JUnit for testing, because it's non-deterministic, right? So, yes, at least to me it's more about experimentation Uh at this stage. But yeah, maybe maybe there is something I just don't know about it. No problem. Any other questions, comments? Sorry, I can't see from this. Well, I think

in that case uh let's wrap this up. Thank you everybody. >> [applause]