Durable Execution in DevOps: How Uniphar Built Reliable Syst... Alice Gibbons & Vaclav (Oisin) Haken

This talk explores durable execution in DevOps, focusing on how UniFar utilizes Dapper to build reliable distributed systems. Alice Gibbons, from DiGrid which promotes the open-source Dapper project, introduces the challenges of managing costs as UniFar transitions to cloud-based infrastructure. The speakers highlight their approach to creating a cost management system that tracks resource usage across various subscription and resource groups while ensuring business needs are met through a standardized reporting format. They delve into how Dapper's workflow capabilities enable parallel execution and resilient task processing, with an emphasis on checkpointing and resuming tasks as needed. The session features a live demonstration of the workflow in action, illustrating the adaptive and modular nature of Dapper in facilitating real-time cost tracking and management.

Um, thank you so much for coming. Today we are going to talk to you about durable execution in DevOps. Specifically, I have my friend from UniFar here and we are going to talk about how they built distributed systems that were reliable with Dapper and I and I have a clicker and I'm going to use my clicker. Oh, I did it twice. Okay, thanks so much for coming.

My name is Alice Gibbons. I'm head of the customer engineering team at DiGrid, which is the company behind the open source Dapper project. Um, and yeah, we're going to talk a lot about reliability in distributed systems today, specifically around workflow. And with me, I have my friend Osheen. >> Hello, my name is Oshin and I work for UniFor and I'm a member of a I'm a cloud

developer. I work for kind of DevOps and platform engineering. So yeah, we're going to show you how we use Dapper, >> right? So I probably should start a little bit with you know just telling you what UniFar actually is. So, Unifir was really uh like a small little kind of co-op of fewarmacies back in 1967 and it kind of grew a little bit you know so now

it's going to be an big anniversary coming soon next year and it's it's now in 160 plus countries and 200 plus multinational partners. And the whole idea is really just to, you know, give people critical life-saving drugs when they need to and just as well do it in just the easiest way possible, which ultimately, you know, translates to cost and uh so really cover the whole end

to end, give you a lot of expertise that we've accumulated so far, tailor solutions to your needs, and ultimately do it now in a in a modern way because uh originally it would have been you know, Tony calling John and ordering stuff over the phone and now it's all going to be or is already digital. >> Awesome. And yeah, we if you haven't heard of Diagram before,

as I said, we're the company behind the open source Dapper project, which is now a graduated project in the CNCF. And essentially our our primary goal is to build platform and services for distributed and reliable applications, including you know AI agents, workflows, um everything you know that from there. We are as I said part of the CNCF as well as the AIF and yeah we want to

talk a lot about reliability today. >> Okay. So this particular example solution is is is a lot to do with uh with cost management and that's how we refer to it and ultimately you know when I described Unifor I mentioned that it kind of grew over all those decades and a lot of it was by by you know acquiring other companies but as you probably know yourself

acquiring can be a tricky business especially you know because the IT systems that you acquire are might have completely different text tag and are in a different stage of evolution. So ultimately now with things moving into cloud and being modernized a whole new initiative was started multi multi-year project to move things into cloud securely modernize it and to enable all these kind of digital uh digital uh

future streams. And so the cost management is really an important aspect because it would be very easy to just migrate some beefy servers from on-prem to cloud and you know eventually though you get the bill right because clouds can be very expensive. It's very easy to just clicky clicky or you know automate provisioning of resources but the bill then comes and you may be surprised. So there's

a big push onto okay there has to be value delivered for the value that you spend on it. So obviously management wants to keep track of it and accounting people just as well. So there were obviously some solutions built into Azure. We found that we wanted to go a little bit deeper especially because some of that infrastructure may be shared between various different streams. So we wanted

to allow for some kind of splitting of a cost between these initiatives. But at the end of the day, you get your your budget for the project or for the stream and the accounting people then see how much you've spent that previous month and they can track whether you were actually, you know, above or under the budget. So that's really what this is about. Uh so in

terms of Azure you would get your hierarchy starting from your tenant to your to your resource group to your subscriptions your and then obviously at the bottom of it the actual resources and when we realized very quickly that you know as more and more resources appear in the marketplace or we or we start using them we wanted something that can be fully extended to support all these

new types and just the number of resource group will grow a number of new projects you know starting and their resources being provisioned. So we wanted something that would allow us to be highly distributed because ultimately for us to split some of that cost it means you know calling some APIs with obviously all the usual problems of potential timeouts rate limits or just you know your basic

transient failures. So and we also wanted to write that code in a sense that allow us to quickly react to to to to new things, new strategies, new resources. So you kind of kind of to divide and conquer the solution instead of having some sort of spaghetti call with some you know intertwining streams of code and kind of how to the workflows were kind of the solution

here because they allow us to just run things under control run whatever needs to happen inside can be paralyzed. So ultimately for us it made sense to sort of just you know fan out all these discoveries all the data discovery data crunching and eventually fan in the numbers and uh at the end of a day you get your your report with the number attached to customer which

is something that business understands accounting people understand and at the end of the day even the the the implementators of the of the solution understand. So normally how this works is that we have a kind of a monthly crown schedule, right? Because ultimately you want on the first of the next month you want to crunch the numbers for the previous month. Um so that's that's kind of

the the request. Then we have we came up with a concept of a cost center rule file. The idea was that all these projects are started evolve and it shouldn't really be one person or one team that that controls that. We wanted even you know business people to be able to say okay I have I have these resources that I'm talking to my guys in the team

they're going to provision certain stuff this is how we're going to kind of deploy it and and ultimately I can introduce all these new rules and to to make sure that that cost is projected to my budget. Uh so that that that's that's kind of one of the inputs and when that all process finishes you get you know a CSV file because we found that to be

very um agnostic to underlying software for accounting processes. >> Yeah. And I think just to call out here this is a common thing we see in terms of trying to bridge the gap between like the business and the technology uh that's being used at an organization. And I think this is a really good example of it like you want a standard kind of CSV file that you

can give to accounting in the language they understand but then from the technology perspective you can write your workflow you can write your application in code you can like live in the world that you currently live in uh and understand. And maybe if you can go back sorry I I wanted to mention the example of that rule you see here uses the resource group that is kind

of most frequently used scenario we do support other levels you know even your basic text or you know resource name full ID match that sort of so so it's very flexible so this is kind of a visual outline of of of this particular solution so this is where some of these features features come into play. So you see that at the top there's a tenant workflow which

governs subscription workflow or workflows right that's up to the configuration of what you want to scan and then you have bunch of other downstreams generally kind of your resource group one is where most of these kind of normal you know common resources would be catered for but we do wanted also to provide some some very special tailored logic so we do have for example a specialized workflows

for AKS splitting where we just scan the parts and the CPU and memory metrics and kind of compute kind of a score. We do have VDI workflow. So based on the sessions and which groups you belong to, we can map you to C center. So we can easily extend it. That's the whole idea here. And uh just as long as that hierarchy of of these kind of

workflows is maintained, that works. >> Yeah. And I think this also lends itself well to if you remember that kind of slide we showed before that shows the hierarchy of uh resources within Azure. So you know you start at your tenant level and then you come down. This like that is exactly mirrored by the type of applications you can build within workflow. So you can see that

this kind of mirrors that you know tenant subscription resource group hierarchy in the same way as you would uh see in that yeah in the Azure uh space. >> Yeah. And you can see the green bar there to the to the left. So when these workflows when they do their logic they crunch the numbers eventually the cost is split the data that split is sent to a

specialized actor uh which is you know an sort of unit of isolation with specific ID scheme and eventually it allows us to to kind of control and because it's an the the the actor contract gives you that kind of transactionality. All these workflows can independently talk to it but but we do have control over properly appending the resources to the state that it maintains. So it allows

us later to possibly look into that state uh for you know there are sometimes queries you know how come my bill went up so much last month and we can then look at the underlying data and and say look this resources provision that's where the bulk of that comes from >> okay so let's see a demo >> let's see a demo right so first I'm going to

start the application and when it started. You can also see my kind of a so normally like I said the chron would run on the first of the next or every month. Obviously here we're not going to wait that long. So I'm just going to to trigger a workflow kind of ad hoc. So you can see that you just sent uh some data that normally would be

kind of defaulted for you based on that that bas based on that logic. But here I'm just going to to uh to generate it on my own. And now you can see the application is connected. So I can just trigger the uh that process. >> And so Oshin's running his application locally here as well as using a kind of a remote Dapper sidecar in this case. >>

And we have the usual demo gremlins. Okay, let me just try again. >> But essentially, yeah, this could be run anywhere. We are just using uh an external Dapper sidecar for to give you additional visibility into what is going to be the workflow app. Um but specifically he's launching his application locally and then uh we can see that in a dashboard here. >> Uh sorry one second

>> and we did try this on the Wi-Fi before it was working. >> So I I I need to it's probably uh probably some networking and authentication. So let me just switch to our alternate strategy. Uh so ultimately this is kind of the view or what would happen. So the main workflow here is is the tenant one and there's some some runs and we ran some of

it earlier. So this is kind of what would happen in terms of um in terms of the the flow. You can see that the tenant started a couple of uh children workflows or subscriptions and in them the other sort of specialized workflows would happen. So if I try to just zoom in a little bit just so you have an idea. You can see that for example special

MySQL kind of logic or log analytics workspace logic h would be would be executed and eventually when it all kind of fans in then that's uh that goes in and sends the report. >> Yeah. So essentially this is like one parent workflow and then each of the child workflows that it kicks off is its own separate process. So you have, you know, a number of these subprocess

workflows that are specific to each Azure resource that you want to get your, you know, your cost management and your um stakeholder visibility into. And then the parent workflow is the one kicking that off and then running that. So that's what these child workloads are showcasing. >> Yeah, I can show you just some of the code that would kind of represent these building blocks. So when we

said chron triggers a request, this would be the controller. The controller simply does few checks and at the end of a day it uses the workflow SDK to schedule the new workflow and that's the tenant one. If I look inside then you can see that this one simply uh start a number of subscription workflows and just that again the level of isolation there and subscription workflow then

runs a few things such as obviously you can obtain all these costs and then runs the resource group levels and all the specialized things. So you already saw kind of that my SQL one, the AKES one. So we can easily extend it in the future and that's that that's the structure right now and kind of what matches that catalyst view. >> Yeah. And I think I think

the important thing to call out here in the code is like specifically he is just like using the the Dapper.NET net SDK and then wrapping his business logic within kind of like this durable execution wrapper meaning that you know if we do kick it off and it does end up working here we can re we can stop and then restart that and it'll continue to run to

completion and so this uh that's a long exception >> but essentially what happens is you can write any of your business logic within kind of the the workflow definition and then within there kind of expand out and see um each of the the child workflows running. So try one more time, you know, see if that works one more time. >> Um and then yeah, we can >>

see if the the Wi-Fi plays a little nicer with us. Live demos, you know. >> Okay, so the application's running. We're connected to our kind of remote sidecar here. That's what we're seeing. And then if we just try and kick one of these off um just with that curl command. >> Let's see. Yeah, it's still it's Yeah, I think it might be authentication problem >> Oh, really?

let's go. Okay. Well, that's that's too bad. I guess the other thing we wanted to talk about a little bit was just like what we kind of showcased in terms of the code. So, if you haven't heard of Dapper before, it is an application runtime for building distributed systems. There is a number of APIs that support various design patterns uh in this space. So you know one

of the things we're talking a lot about today is specifically for durable execution which is specific to the workflow API but there is a number of work other APIs that are also supported for things like message brokers for things like pub request request reply patterns things like that. Um and specifically it acts as an abstraction in between your infrastructure and your code. So if you have you

know an application and you are depending on sort of any sort of infrastructure resource the the Dapper kind of Dapper sidecar or the uh the framework sits in between your app and your your infrastructure there. Um so this means a ton of things. It means composability, modularity. It has crosscutting concerns things like resiliency, observability and um security built in. But then not only that you can also

take advantage of as few or as many of these uh building blocks. So yeah, what OSHEN is using within its application is specific to workflow and this is kind of where a lot of these applications are going from a durable execution perspective. So essentially when you kind of kick off one of these new apps, you can stop it or if that process goes down for whatever reason,

it'll pick back up from where it was running or where it last uh the last command it executed. Um so in Oh, is that it? I suspect that might easily be the problem. >> All right, just we're we're doing some live debugging now. This is exciting. No, no, no. This is good. I I want to see the demo for sure. Um you think you got logged out

in 10 minutes. So essentially like the the main piece of of the durable execution wrapper though is this checkpointing and replaying mechanism. So once hopefully we get this to work here but essentially the application as it's running is using the built-in Dapper workflow engine which has a state store and that state store is essentially an appendon log meaning that as it's running it's going to be checkpointing

consistently its state to the state store and then uh as per every single workflow activity which is where all the business logic is being done. So in his case it was uh you know where that kind of where we're hitting the Azure API which apparently we're not logged into and pulling back the uh pulling back kind of the cost management data in that case. So any of

those business logic um runs those are going to be activities and that's going to be stored into the state store um as we go and then when we pick back up say you know that process goes down for whatever reason it will continue to run to completion here. How are we doing? >> I think we might have a winner here. Let's have a quick look. And I'm

actually kind of loving this because you never pick when your session's going to time out. >> So now we know it was 20 minutes. >> Ah, perfect. So see this is now this is now the running one, right? >> Running. >> Thank you. Appreciate it. >> Now what I'm going to do, however, I am going to actually shut that application down. Okay, now we're going to see

the durability live in action. Okay, so the app's actually running now. So we can cancel it because it wasn't running before. So now we're actually going to cancel it. So that's what he's mimicking right now. So >> the reason Sorry, apologies to jumping. H the reasons could be, you know, many, right? You you know, the blade in a data center, you know, has a technical failure or

some some digger, you know, wrecks the cable, all of them. And you know what? Or simply AKS is upgrading. You never know but ultimately things can happen and they will happen. >> Yeah. No, exactly. Like like failure is inevitable in like so many of these distributed systems cases, right? Whether it is a Azure login scenario or you know you're there's a network there's a network glitch whatever

it might be. This could simulate like your pod going down in Kubernetes or you know a rolling update uh security patch by AKS. I that's just a few that I can think of. So and then if we restart it. So restart the application. Let's say we've we've we've we've repaired that cable. >> Yeah, we repaired the cable. We've logged into Azure. >> So, this is now going

to start. And if I just switch back to the catalyst view, eventually we're going to see that workflow h progressing through all the other remaining steps. And again like from the kind of the background or the backbone perspective like what's happened here is it made it to these subscription workflows as well as you know if you click on one of those Oene like the child workflows within

them um and then each of those uh exactly each of those individual um runs or activities was checkpointed into that state store for that resumption after the fact uh ensuring that it can kind of run to completion. And so yeah, so >> you probably saw the flick from running to completed. And if I switch back now that we got that report and and that's the workflow done.

>> Awesome. Do you want to show the email? >> Uh yes. Let's see if that ever comes. I >> Ah, there you go. So for example, my email. So So this is kind of a obviously sample email that the accounting people would get, but like I said, the CSV is kind of what we chose as a as a as as as a file format. So eventually they

would see this kind of a structure and they could load it into their accounting uh systems any way they they they need. >> Yeah. Awesome. >> So let's go back to the presentation. So yeah, I talked talked through a little bit of this already, but essentially like this uh kind of business logic is all wrapped within this kind of workflow durable execution wrapper. And specifically, you know,

there's a number of other things you can do here. It's super uh modular in terms of supporting other patterns as well. So we kind of saw that fan in fan out meaning that you can do parallelization of a number of business processes but then you can also do sequential workflows, child workflows which we saw which is again spanning its kind of child process separately and then run

letting that run to completion. Um as well as things like waiting for external events workflow versioning a ton of those. This is also very interesting from an identity perspective and this is something we're talking about a lot these days especially with agents and agentic workflows. So being able to have an agent identity, being able to lock that down from an access control list, uh, and being able

to talk agent to agent in between these agent workflows. Um, and last but not least, son of retention and resiliency policies. So right now, this does not have a default resiliency policy built into it. uh it just has that kind of basic checkpointing resumption. But you can set resiliency and retry policies on an activity basis. Meaning that if you kind of like if the activity fails for

whatever reason, you can have a specific behavior that you want to be executed. You know, whether it's retry, indefinitely, retry once, never retry, whatever. Um and then this is just kind of a little diagram of how this is kind of working from the back end. I wanted to showcase this because I think it gives a really nice view into, you know, what we're actually doing within each

of the Dapper sidecars and each of the Dapper processes. So, like I said, it has a built-in Dapper uh workflow engine into the sidecar and it's you taking advantage of a you know, bring your own workflow state store and then as each activity is kind of progressing that is being written to that state store. um which is why Oshene was able to kind of pick his application

back up and then run that those workflows and child processes to completion. Um and then it replays from the beginning. So every single time the you know the execution state moves from the application to that Dapper engine um it's going to pick back up run to the the place in the workflow state store in which uh it's last checkpointed and then continue on. Um, last but not

least, I wanted to just touch a little bit in terms of we've talked about two kind of staple uh, architecture patterns today. One of them specific to actors and one of them workflow. And it's an interesting from an abstraction point of view uh, in terms of like workflow is just a higher level abstraction on top of the actor pattern. So if anyone has, you know, taken advantage

of the actor pattern before, it's really powerful from a uh, scalability perspective. So you can have, you know, these actors are sort of these individual unit pieces of logic that have both a state as well as, you know, they can communicate with each other and then an identity associated to them. Um, and you can run them at super high scale. They play really nicely for a ton

of scenarios. You know, we're using it here today to represent a cost center at UniFar. Uh, but you know, you use it for many other applications as well. And then, uh, kind of workflows. What that does is it's another higher level abstraction meaning it'll manage the the state and the life cycle of those actors for you and then even higher up we have a we have an

agents framework as well um for you know for for orchestrating sort of these multiple agent workflows and things like that um I guess just a little one more slide in terms of lessons learned we've worked really closely with UNIFR on a lot of this stuff and wanted to kind of share some of the you know from the trenches >> absolutely yeah you know it's been it's been

actually a great experience we did learn a lot h and like some of the stuff like our for example our very initial approach was just simply you write all your workflows and then you just you know start it then let it run and just eventually comes back in and we realized you know what like as this as all these solutions kind of evolve the number of resource

group the number of subscription the number of resources all that's just going to grow and so you don't really you know you don't really have control in terms of you know parallel executions of all these kind of parentto child oracle hierarchy. So we we adjust it a little bit. So now for example we we run our subscriptions in parallel but then within the subscriptions we run things

sequentially that gives us that kind of a control like the the performance here isn't really in any shape or form a critical factor. Uh so that was for example one of the lessons we kind of when we were looking at all kind of frameworks uh even before kind of you know talking to to to to diagram we looked at kind of other frameworks and obviously you might

recognize some code names like Roslin. We eventually picked Dapper H for for a few reasons. We very much love the configurability. So you know if you have certain components maybe your configuration local house is different than the one in the cloud uh it's language agnostic which you know as much as we are kind of C you know house I mean you know certainly no harm uh and

just the building blocks are there and it allows us to concentrate on our you know business logic without having to you know deal with these little nitty-gritty concerns. Um but we do actually extend it certain things like one of our applications is is is very kind of mission critical and so we wanted to have that kind of dapper uh deployment not just kind of within a cluster

uh replication but also kind of regional so that we can run this across the globe but uh but then certain certain uh certain uh contracts are still kept. So we've done a little bit of an extension on that. Uh so so that been great experience and we we've so far adopted dapper for few of our other apps. So I'm just going to mention just just few of

them as examples. We have this kind of a thing we refer to as key rotation tool because not not all kind of workloads can run you know under proper identity legacy or just completely different text or just just deployment model that way. So they they would use kind of key based access to to some uh resources. So we then have this tool where we can kind of

on schedule rotate these keys or even on demand if we suspect uh some sort of security incident or maybe somebody shared the key inappropriately. So we we actually can do it kind of on demand and rotate. uh we have this other application that's very that that's the mission critical application I mentioned earlier we refer to it as frontgate that's really where business people can uh configure some

of these kind of legacy integrations generally file filebased you know SFTP some client upload stuff files to SFP and then we can take them from there and move them to to somewhere else where SAP process can pick them up. So that that's that that's the front gate. We're now starting looking into kind of some of the agents. We have this idea about kind of on boarding and

offboarding resources as in as in personnel and licenses for them etc. So we're very much looking into agents for that. And we for example found the the workflows within the Microsoft agent framework to be to be somewhat alpha stage. And so we're very much looking into using Dapper workflows >> Awesome. And I think this is just kind of again an example of all the different applications you

can use common business processes modeled as workflows which is um yeah mostly it. So thank you. Thanks for bearing with us with the demo trouble. And this is all open source on GitHub. Yeah. Thank you, Oshene.