Taming Complexity: Building Observable Workflows... Mauricio "Salaboy" Salatino & Kasper Borg Nissen

This talk focuses on building observable workflows to manage complexity in modern distributed applications. The speakers, Casper and Salatino, discuss the evolution of application architectures from monolithic systems to microservices and now to agentic applications. They emphasize the need for frameworks that allow agents to communicate and discover each other, highlighting the importance of protocols such as MCP and the agent-to-agent (A2A) protocol, especially for business-critical operations. The talk also explores the use of Dapper workflows for reliable and durable long-running tasks. Observability is a key theme, with the speakers advocating for open telemetry as a standard for tracking agent interactions and understanding system performance in environments where non-deterministic workflows proliferate.

Welcome to this talk building observable workflows. Um or we actually started taming complexity because there's a lot of complexity in this world today and when we originally designed this talk we were thinking about doing like talking about asynchronous workflows and stuff like that but then AI and MCP servers and Marishia went crazy with the demo. So >> so that's what we like basically ended up doing is

like doing a lot of different things within agents MCP servers tool called skills >> LLMs all over the place. Yeah. >> So, uh I I tend to like building demos. The problem with this is like no matter how simple the demo that you're building is, it gets really complex really really fast. And we will show you something super simple, but it's at the same time very complicated

to understand what's going on. >> Yeah. But before that, maybe just a quick introduction to who we are. My name is Casper. I work as a principal developer advocate at the zero. Um I do a lot of stuff in the community in the Nordics. Uh help co-ounded cloud native Nordics and run some of the groups in in Denmark as well helped with that. Um that's a conference

in November 1920th cloud native Denmark. So make sure to check that out. CFP is open. >> Yeah. And I'm Salatino. I work like as an ecosystem engineer for a company. It's called Diagrid. We work on the Dapper project. So if you're interested in that, feel free to visit the booth in the showcase room. >> So the TLTW too long to watch. Um as your applications grow uh

in or infrastructure grow in complexity, you must have the right tools to understand what is going on at all times. That's kind of the thing we want to try and highlight in in this talk >> because architectures have been evolving like since the monolithic applications back in what the beginning of the 2000s earlier maybe also and then we had the microser coming then everything be became asynchronous

event driven and now we are doing agents uh in the mix of our services doing all kinds of nondeterministic things so it's it's a little bit crazy what what is going to to happen right now. Yeah. And with agents, it goes a little bit further down on the path of you don't know actually with this where these agents are running. Are they running all in the same

application in the same container or are they like distributed agents as well which are pretty much closer to distributed applications. Last year we did the same presentation that we are doing today but mostly focused on driven applications and microservices. Today's presentation it's a little bit more about the agentic side of things. You can watch this presentation. It's on YouTube if you're interested. >> Yeah, exactly. A bit

of a disclaimer before we start because nowadays when we talk about agents it's important to make a big difference like coding agents like cloudco gemini and codex and all these things are not the scope of this presentation in this presentation we are talking about business agents that perform critical business operations and they are like more userfacing right so think about that what can you automate in your

company with an agent that will perform stuff that before were being done by different tools or different teams in the new landscape Cape again we were thinking okay we have microser eventdriven architectures what do we need to learn when we start thinking about building really distributed agentic applications first of all we need to start learning about agentic frameworks there are tons of different frameworks tons of different

SDKs that allows you to define what an agent is and the next step is to start thinking about okay if we have more than one agent where are these agents running and how do they discover each other how do they communicate between each other that's the agentto agent protocol then you need to think of course which LLM provider do we use which models where do we run

them which thing fits the purpose for the kind of like agents that we are trying to build and then you go to the tooling space right like you can actually define local tools in most agentic frameworks but then when you start like trying to integrate all the systems MCP becomes a thing and finally something uh like skills are also becoming like more and more relevant in this

industry so let's talk a little bit about this kind of things so I'm pretty sure that you heard about MCP doing this week, you kind of know what it is and I will just try to summarize it as a protocol that allow like you know LLMs uh to interact with other tools and services. It's really good to have a protocol like this that is standardize how do

we do these interactions across different LLM providers. Again you pretty sure that you know about this. The new kid on the block or like something that was introduced by Google uh last year, less than a year ago, is the agent to agent protocol. And I'm really interested about these things because again, it provides a way for us to like conceptualize how true agents can interact that are

running in different let's say containers for the sake of of Kubernetes, right? This protocol is a little bit more focused on the enterprise use cases. As I mentioned before, these are like business agents trying to perform business operations. And the protocol is being built on like enterprisegrade concerns like security uh you know communications resiliency user experience and discoverability for example uh I wanted to focus on two

things about this protocol the A2A protocol uh because again this is a complement to MCP but uh I think that it's important to mention here that because like agent like business agents tends to run for long periods of time we want to make sure that this protocol covers these kind of use cases of long running tasks and keeping track of what agent is doing and sending notifications

across agents. And finally, as an engineer, I'm very used to work with like binary protocols and text protocols for interacting between microservices. The A2A protocol was designed to also cover audio and video, which is something that I haven't done before. So, I'm kind of interesting this kind of protocol, right? >> I start thinking now you have two agents talking to each other, sending videos. How do you

actually understand what's going on there? Finally, skills something that was introduced in December of 2025 and now we are talking about this quite extensively and I think that it's important to understand that we are adding more and more complexity into our applications and we need to figure out how this mechanism will work. In the case of skills, it's mostly MD files that describes very specific characteristics of

what an agent can do and perform against like a a service or a very specific topic. And I'm showing this because like down there you can see that skill is like an skill MD plus a bunch of scripts or like routines that can run to interact with other tools. So when I think about how do I make sense of when a skill is being called or what

like scripts in these scripts directories are doing I can like start wondering okay how are we going to manage all of these things >> and observe it right >> and of of the observability side is is the the key thing. So let's take a look at the demo that we built to show kind like some of these capabilities. And in this case what we have is like

a pizza store on the left side like on the right side let's say we have the chat which is the userf facing you know a way of interacting with all the agents that we have and we have four agents the store manager the drinks agent the cooking agent and the delivery agent and on top we can see like there are services providing like the inventory of ingredients

the drinks that we have and what Casper is doing. >> So so I I just like I would like to order a pepperoni pizza and a beer and And then the agent comes back and are you sure? I just want to confirm the order. And then yeah, >> you can you can accept you can change the order. But if you approve now the store manager is going

to start orchestrating this agents to deliver that order to the user. And you can see that the first thing that happens is that the drinks agents will go and pick you know the beer and then the cooking agent is going to select the ingredients that we need to the pizza and reserve an oven to start cooking the pizza and it will wait for that oven to finish

uh in order to move that order forward to the delivery phase. It's a little bit hard to deliver a pizza that's not baked yet. >> Exactly. You just need to if if it's not baked in, you're kind of like screwed. So again, you can see there is like a long running thing there with the oven. Now the pizza is ready and now the delivery agent is picking

up the order and delivering that order to the customer. >> Again, it sounds super simple. There are 10 microservices running there with different agents and just using all these new protocols to get this interaction going. >> Yeah. And all of these steps is all the reasoning that the agents are doing. So we'll come back to that how to actually get that into uh to traces and stuff

like that in a in a minute. >> Exactly. So two things there I think that are important like loads of new tools multiple agents interacting between each other. The scalability side we are not going to cover in this presentation but we need to first understand how these input agents are interacting with right. So in order to talk about uh and in order to start understanding about this

application we are showing kind like this diagram that it was autogenerated you know again like by just an agent that looks into the code and defines the interactions. You see we have like the front end that it's taking the orders and on the back we have a main agent a main agent that is called the store manager agent that basically is orchestrating the drinks agent the cooking

agent and the delivery agent to deliver the order. There are a bunch of microservices hosting the ingredients, the drinks, the ovens, the bikes and uh we are using MCP to connect you know LLMs behind the agents to interact with these tools. Interesting enough like agents are written in Java using lang forj and some of the services are written in go just like normal go >> cool. So

yeah that's good. So in order to implement this application, we decided to focus on two things that will help us like from this CNCF landscape to implement complex distributed applications and to be able to observe them. So because I'm working on the Dapper project uh and the Dapper project provide a bunch of different APIs to build these distributed applications in the Aentic world. I think that the

most relevant thing that we wanted to show today is Dapper workflows that allows us to make sure that pizza orders are reliable and they run end to end uh in a reliable way and we can keep track of what's going on from from the pizza order point of view. Dapper workflows are mostly taking care of this like long running process and maybe making sure that there's an

audit trail of what happened, what each agent did and why did they make certain decisions. this case we see on the on the diagram there like the agents and how the orders flow through the agents and how like the workflow is basically driving this interaction. The idea here from Dapper workflows is that the cooking agent fails for some reason and it crashes. We can resume the order

from the cooking process instead of starting the order again from from the beginning which will cause all the agents to re-trigger all the LM requests again. So when you think about agents uh and resiliency, we think it in two separate terms like one is per agent durability, right? Like each of these agents that is running inside the application is performing a lot of LLM calls and a

lot of tool calls, right? Again, if something crashes at some point, we want to make sure that we resume from where we left off instead of starting again from scratch. And there was a presentation about Dapper agents and some integrations with different frameworks about how to to do this for like not only Langchain for 4J but a bunch of other things that are out there. The second

aspect is the multi- aent orchestration is again like you have multiple agents. What you want to make durable and resilient is the entire order process from beginning to the end. We want to be able to resume at any point in time and we want to keep track of what's going on uh uh again like with the agents and why they are making decisions. The interesting thing about

this is that no matter which framework you're using, when you're running these agents in production, you should be able to have the right tools to go quickly find the agent and see if something went wrong. So you can just re-trigger automatically the execution. And in this case, I'm showing kind like our our products that we have there. >> But even here, like you can see there in

the diagram, when these agents are interacting, the interaction process and because we're using a lot of different protocols, it gets really complicated. So we need something like open telemetry >> Exactly. So yeah, we need something to to actually observe all of this. There's so much complexity and open telemetry is really becoming the def facto standard for how you do obserability these days. And that is also testamented

in the uh CNTF survey from which was published in in January this year. Uh according to the survey, 49% of all the respondents in in that survey are running open telemetry in production these days and an an additional 26% are evaluating right now. So the project has so much momentum. Um so it it is really becoming the def facto standard for how to do observability these days.

So that's also what we're using in in this demo. >> Mhm. >> Just quickly if you're not familiar with how it works. Instrumentation you you you need to have some kind of way to um get the telemetry out of your applications. And that's where you either manually instrument your applications. You can use auto instrumentation. Uh you can also use the operator which I'll show you in a

second to get the instrumentation into the applications that you are building and and running. You can also observe the infrastructure like Kubernetes. There's a bunch of different uh receivers that can get telemetry from from Kubernetes and and send them to wherever you want to do. But usually you send them to a open telemetry collector which is part of the project as well where you're receiving all the

telemetry. You do some kind of processing perhaps you are redacting sensitive fields whatever you do. then you export it to to some telemetry back end. So the the main thing about this is that M telemetry is really about standardizing how you are collecting your telemetry and hopefully it will be sort of the last observability agent you will ever install. So you don't need to have all of

these proprietary agents anymore. Now you can rely on an open standard that pushes the vendors to fight further down uh sort of the this pipeline and and really provide you the best possible experience with the data you are you're sending in. So the M telemetry operator uh it provides a way to get that instrumentation into your applications if you're running in either Go, Python, Java. net or

NodeJS and you do that by defining an instrumentation custom resource and in that you can spec where the collector is running where to send the telemetry what kind of environment variables you want to be configured in your applications and then you add an annotation to your deployment spec and then automatically your uh basically your instrumentation or your applications will be instrumented and that is a really good

way to to have a good baseline. So making sense of all the complexity, why is all of this hard to observe? U because we we do have metrics, we do have traces, uh we do have blocks, right? Um but this world is is a little bit different and traditional tracing kind of assumed this request response uh flow. So that is that's one of the problems. Another problem

is that usually uh context propagation breaks somewhere. Uh either we are having using different protocols that are not propagating the context correctly. So there's all kinds of ways that the context can get lost and and your traces will not be that valuable because they are broken. Um and with agents uh the decisions around these nondeterministic workflows is also different way of of doing this. So there are

some some new ways that we need to figure out how to to deal with and this execution just spans so many different systems now. We have the LMS, we have MCP servers, we have even skills in best with best scripts and stuff. There's so many different ways we need to make sure that that this context is is propagated >> Yeah. And I think that the combination here

is the is like a combination problem, right? Like which frameworks are you using, which SDKs, which models, which tools, which protocols, >> which languages, >> which languages and yeah, that that's >> and and basically some of the key context in in this new world of of AI lives kind of outside the running systems because you are like delegating that to an agent now. So that is also

a new thing, the reasoning and and all >> Yeah. And again I think that like we are like in an era where we have like more and more personas or different layers on how do we observe these things. We want to see what the agents are doing. We want to see what the protocols like service to service interactions are happening and if something is failing at that

level and we also have the platform view which is like okay we have all these tools installed in the cluster what are these tool producing right like are these like tools working as >> exactly and and when you have this broken context if it's not getting propagated correctly you will end up with in a scenario like this instead of having like a beautiful trace with all the

stuff and all the steps that that took place you will have like multiple different traces and it's really hard to actually put them back together. Um, but the the key thing in in all of this propagation is really the the standard around the W3C trace context and the trace parent and making sure that that is propagated in all the different tools that we are using and frameworks

etc. That is the key to to getting all of this unlocked. But we'll come back to that in a second because we also did some fixes the in the demo to actually get it to work. So putting it all together um let's try and go back to the demo now and let me find this slide. >> Another one. Yeah. So I don't know if you can see

back there but like uh what I wanted to show here in this slide is like how do you define an agent in lang chain 4j like it's pretty simple in this case you have a system prompt and a user prompt that it's basically the behavior of that the agent will execute in this case is the drinks agent so it needs to go and fetch some drinks you

can see that there is a metadata like now we have an agent's name but also we have in this case we using an MCP tool like an MCP server to call some tools so we can quickly configure an agent in this way uh with this framework and that again like when you think about like okay where is this agent going to run in this case it's going

to run inside the Java application that we will need to make resilient so if we go to the next step is uh okay now we want to make two agents to talk to each other right like that they live in separate containers so in this case we are using the A2A client in order to configure where the other agent is located in this case by providing a

Kubernetes service name and a port and then the A2A protocol will be established between these two agents to basically interact between each other whenever it's needed And finally what I wanted to show is uh with tools like lang chain 4j they are like other frameworks doing ex something super super similar you can define orchestrations in this case we have something that it's called the sequence agent here

you can see that sequence agent would basically define like a you know steps that needs to be executed in this case like agents that needs to be called in a sequence right so you start kind of start defining kind like these workflow patterns in a declarative way in the agents world so what we have done here for the dapper project is making sure that we on change

the lang chain 4j experience and behind the covers we generate these durable workflows that keeps track of these interactions uh for longunning processes. >> Yes. And if we continue with the tabs >> if we continue with the tabs then what we are looking at here we are looking at the bike service skill definition. Again it's an MD file that explains to like an LLM how to interact

and reserve bikes to deliver orders. In this case, we wanted to go very to the extreme where we define a skill with a bash script to interact with another service. And we didn't really wanted to do this, but we actually see people doing this. And when we think about like in terms of observability, it's like, okay, now I need to go and observe a bash script. And

it's like, okay, this is not going to be a fun day, but we made it fun. Like we made it work. >> We made it work. Yeah. So before we get to that, um, just quickly showing you the telemetry operators instrumentation resource how it looks like. So this is how basically where you're defining where your endpoint is where your collector is running where to the instrumentation should

send the telemetry and you can do all kinds of things around the entire the environment variables that you want to configure for for this particular service when it's being instrumented. Um another quick thing uh the DO project also supports tracing out of the box for for the side cast which is quite nice. The the sort of the specs look similar to what you just saw. you define

the endpoint of where the collector is running and then the the sidecar will also uh send the telemetry to uh to your collector and that is also what we're using uh in this demo and then just quickly this is how you then inject it's actually harder to see when I do it like that. >> Yeah, let's just zoom in. So this is basically how you inject this

into a go application. You say inject go and then you point that to the particular resource or you can also just say true that depends on how you want to do it. Um but for go specifically you also need to add this go autotarget exec to the binary and where the binary is in in a container otherwise it's not working and that is due to that this

is using ebpf under the hood uh so it will actually get a sidecar uh that is doing all the instrumentation for you that is different if you're using java or nodejs or in java it will be the java agent that will be uh be the one that's going to be installed so let's have a quick look at the cluster Let me just do it like this again

parts. So we're running all of this in our in a local kind cluster. And just to show you what we have running, we have all the different services uh that Mauricio showed you in the di diagram. Then we have the collector running and we have the operator running to to do all of this order instrumentation. And notice that the go services which is the inventory, the drinks,

the uh the bike service and then the store service al all of them has have two um that is the the sidecar for that is being injected into this application. We just quickly show I think we have time. >> Oh >> just to quickly mention that like that sidecar is basically taking the telemetry out of the application that is running without changing any of the application like

we are just getting that sidecar injected and it's just working out of the box. So as you can see this is just the main container and then we have the open telemetry auto instrumentation container running here. So this is how the instrumentation is uh is is being done for the go services specifically. >> So let's have a look at all the complexity uh we get out of

this. So as you can see now this is apparently an old one. >> Oh well we'll just use this. There's 300 spans in this. >> 300 spans. Yeah. um which is quite a lot of uh information uh to try and like >> dive through and and find that needle in the haststack when something goes wrong. >> Um and it's a little bit hard to see but let's

just zoom in a little bit just so you can actually see some of all the telemetry that's coming out of this. So you can see stuff like completion. You can see that this is oh let me just zoom out a little bit. So in this case that's a geni completion that the agent is responding successfully. It it fetched some drinks and now it needs to do something

else. You can see the prompt. You can also see >> the response. Yeah. >> And you can also see the token uses. So all of this is based on the semantic conventions uh geni semantic conventions. So there are now semantic conventions for this as well. Uh but I think the ecosystem still needs to align on that and and like transform all the >> Attributes that are out

there. Think about for these kind of applications, you want to see what the agents are doing and you want to quickly understand if something is going wrong with a tool or with an agent himself or if the agent is looping out of control or something >> and and this is the current state of how it is in in Jagger today and it's it's not it's it's really

hard to find all of those completion steps and and take out the reasoning steps and actually understanding what is actually going on. >> Just briefly, we are using you can you can of course use AI to all of this. So this is just a quick example of extracting all of this to to put it in a in a nicer way uh to view it >> or like

an agentic view, right? >> Yeah. So you could use AI for that as well. Um let's go back to the slides if I can find them. >> So how to observe agents? So agent execution kind of includes a lot of new things, right? We have this agentic loop that was also mentioned in the keynote this morning where agents are using tools, they are using maybe calling other

uh agents as well. there's a lot of stuff that are going on in in that space. Um so the number of spans also varies uh when we do this a bunch of times which is also interesting. Um >> yeah so what what we need to capture >> that that that point is really interesting because when we had like event driven applications or like you know microservices applications

we kind of knew that for certain use cases we have these numbers of spans that are expected and we will always get that. So we can start doing some smart things like writing tests against those traces. Now that's going out of the way. >> Yeah, this is not not easy. And now we also need to capture the actual reasoning and the decisions that the that the agent

took, right? Um and all the executions and all the interactions. So it's it's a different game of observing these things. It it's different information that we need to surface. >> Then back to the to the skills um >> night. >> So this this was a little bit of a hack to be honest um >> because yeah, let me just show you. Uh so what we did so

the this the agent is uh we wrapped this in in the actual process when we are executing the script and we took the the trace context that we had and put it in a in an environment variable for the environment that the the actual skill was uh or the best script was being executed. Took that out and and just did a curl with that as well to

to propagate the context. But that is just a way of doing it. >> And that highlights that auto instrumentation will take you that far, right? Like when you have these kind of interactions that are a little bit off script or in this case in a bash script you would just need to actually go and and do it manually and again that push you to understand all these

interactions deeply and uh not start consuming skills that you don't know who created because then you will just need to go and and modify those. >> Exactly. So it's a little bit interesting as you can see in in this particular case. So this was yeah some time ago when we did this there was 275 spans 10 services uh the number of services is is usually the same

>> the the number of sizes is the same yeah >> but the number of spans is different um but they include all of these reasoning loops they include all the tool calls and all of these stuff that we mentioned before um and the geni semantic conventions as I also mentioned uh look something like this so you you will see which model was executing this what was the

actual system problem as you also saw before the completion the tokens etc. Yeah, for me like as a developer this is really good because I can see like the the the agent reasoning in a way but I need like the next level of abstraction is like each agentic framework should like a standardized in the way that they expose like agent information. >> Yeah. So I think right

now if you if you're using JG at least the the the flow would be something like this where you are like trying to like squeeze it all together so you can actually try to like stitch together the actual reasoning steps. So this is let me just click through it like it you get all of these completion now it did this and this is now it gets invokes

a tool somewhere it gets some pizza dough in this case then it does reasoning again okay now I I acquired this now I'll go and reserve an oven so it's it's just that it's a little bit hard to actually understand what what is actually going on but but the information is there so you can get it out which is which is >> great this is just uh

the the skill execution um the best script These skill attributes does not ex exist. This is just some custom attributes that we added just because we were missing like a way to identify which script was actually being executed. Uh you would probably do this in a in a more proper open telemetry way by using something like open telemetry viva or something like that where you can define

your own custom semantics. But this was just a quick and dirty hack to get this working. >> Um but that is I guess how many of us >> I think that like skills were created in December last year. So we need to catch up there and just try to standardize some stuff. >> Exactly. So the more telemetry is not always equal to to better telemetry, right? Because

275 spans is is way too much for for ordering a single pizza and a and a beer in this case. >> Um because many of of these represent a lot of infrastructure stuff going on. Um for the dab side car, there's a lot of different spans that are happening there, but there's also all all the intermediate steps that are are probably not that important in in this

new world. >> Yeah. Yeah, and this is kind like again like the same question that we had before. Do we build like like a monolithic agentic system that is a single agent doing a lot of stuff or do we start distributing this and we start creating kind of systemto system interactions? >> Yeah. So just to sort of wrap up because I think we are >> we are

out of time. Yeah. >> Ah not completely but but yeah de works really well from like an obserability perspective. They the sidec car producing all the different spans that you would expect also for public subscribe. That was what we did in the previous talk. H but also the workflow >> Yeah, I think that with workflows on the Dapper side like we are just implementing this durability for

agentic framework. So I think that that gives you kind like the audit trail of like when things go wrong you can go and figure out what to do and how to proceed from there. >> And then open telemetry the the go instrumentation the java instrumentation kind of worked. Um yeah >> we had some issues though because uh we also found a lot of challenges and had have

to do uh some PR. >> Yeah. Yeah. So definitely like in the Java space when you're using frameworks like the recommendation is do not use auto instrumentation because the Java agent in this case which is another like small like library that you run with your application in the JVM like can clash with some of the frameworks that you're using in your application. So you need to be

careful there. Uh and again aa is a new protocol. So there are like some rough edges there that we needed to patch in order to make this working. Uh like it was not an easy journey but I think that the communities are are catching up and getting there. Uh and again like for different text stacks, different languages, you need to figure out your observability story and sometimes

the auto instrumentation works and sometimes like frameworks are smarter than the auto instrumentation and you need to rely on that to get like more quality information out of the the >> Yeah. And the scripts Yeah. They broke our traces but we found a way to hack us through it a anyway and and and yeah, it worked. We got traces through that as >> Brace yourself with skills.

>> Yeah. This is just if if you want to check this out afterwards. That's uh some of the pull requests that was uh made dur to get this working. Um and yeah, key takeaways asynchronous systems are kind of hard to observe but but the ecosystem is maturing and we are like working towards that and and just improving day by day like the work like this is also

helping with the PR you saw here the for the next person this will be fixed hopefully. >> Exactly. That's the idea just to like uh just to simplify the journey for like newcomers and and people that is trying to catch up or doing PC's with these. So the ecosystem works out of the box instead of you fixing and patching different libraries. >> Yes. And just a quick

note, uh I've been working on uh this open telemetry maturity support framework. Uh there's a a draft uh of it on this community issue if you're interested. It's really about this work as well like figuring out how well is open telemetry supported and how mature is it in the different projects >> Uh so check it out if you're interested. I'm looking for some more collaborators on on

that. >> Yeah. And I think that's going to take us to that point where we know like okay we are picking this framework. We know that this is supported correctly from the observability point of view which I think it's like a must nowadays with this complexity. >> And with that I think we are hitting it on time. >> This is just a good start. Thank you for

coming. Thank you for staying this late. >> Yeah. If you want to get in touch you can uh find us on on LinkedIn using this QR code >> and I think yeah questions come up afterwards and uh we'll answer any questions. Thank you so much, folks.