REST in Peace: AI Needs to Be Async - Meet Asya - Artem Yushkovskiy, Delivery Hero

This talk focuses on AI orchestration using the open-source tool called Asia, developed by the speaker's team at Delivery Hero, a global food delivery service. The speaker explains the challenges faced in automating image improvement tasks, such as enhancing blurry or poorly framed images, particularly when scaling AI models in production. They discuss the limitations of traditional pipeline architectures, which encounter bottlenecks and inefficiencies as they handle various processing tasks. The speaker proposes a novel architecture that implements asynchronous microservices where each task operates as an independent actor, allowing for better scalability and flexibility. They detail how Asia utilizes a custom resource definition (CRD) to facilitate this actor-based communication model and emphasize the importance of decoupling infrastructure concerns from application logic. The session concludes with a demonstration of the Asia framework and its capabilities.

All right. Shall we start? >> Um, hello everyone. Thank you for for coming. Um, my name is Artam and uh yeah the I'm going to be talking about AI orchestration yet another talk. So first of all quick poll what do you think first uh when when I say actors? Yeah, 50 years old computational model, complicated um execution model, esoteric languages. That's that's all how um it's it's

been a working model. Uh but only recently with agentic AI it became extremely relevant and today I'll show you why. Um and the presentation will be about the open source tool called Asia uh from async actors which we built at deliver hero. But first of all a little bit of context uh my use case uh at deliver hero which is the global uh food delivery uh app

headquartered in Berlin. Um in my team we are focused on um automatic image improvement. So for example, you take um an image uh which is blurry, which is too too close and you make it yummy. So you um upscale it, you center it, you draw something around. Um but you know, as always, it might be working well on your demo, on your notebook, but um not really

working when you try to scale it. So here's a few corner cases. Um normal looking ice cream but floating somewhere in the space and uh not really yummy. Nicest nice picture of smiling dosa which is also hard to uh hard hard to catch automatically and my absolute favorite chicken soup. So this is just illustration of um why the working on AI use case at scale is um

extremely complicated. you need very fast iterations. You need um human in the loop at every stage and you need a lot of agility and um that's that's what we're going to be talking about. So my profile is somewhere in between of the worlds of data science, AI engineering, backend, devops and everything. So I'm kind of a bridge for the last five six years. I've been working close

with both data science teams and back end engineering teams to um make machine learning and AI use cases work. A little bit of um step back to our to the history of our project. So it all started as a classic use case. You have cubeflow or vertex pipeline prototyped by the brilliant data scientists. It all works on the demo product approves. We start scaling it up. We

duplicate these pipelines and run them in large batches at scale and things start to go wrong because each of the pipeline has itself internally. The retry logic, the thread pools, the uh back off and all these things uh that belong to in infrastructure. When you start scaling it, this is what happens. First pipelines start uh go and pass go well but at some point the external services

AI API saturates starts returning you uh rate limits and um pipelines enter into the exponential buck off waiting for each other um and this is just not going to work and uh what we saw in the practice was that you try to run 100 pipelines in parallel and then you see constantly that 50 to 80 are are just blocked. They're just waiting and they don't know when

the API becomes available again to retry the the request. So what was our first uh attempt to fix it um was to extract the AI via API and deploy the models locally. So luckily we're working with stable diffusion models which are around 30 billion parameters don't need KV cache don't need complicated infrastructure. So we can just deploy them as a syn separate deployments connect them via popsup

cues so that um our APIs do not return 429 when they are saturated um but the messages keep queuing in the uh in the cues. What happens is that we still see um a lot of pipelines waiting um no back off but they are just waiting for the request to complete and then the funny thing that we saw is that was this thing no matter how much

we tried to push uh the throughput to the limits we could not defeat the the cenocidal load and this is because the pipelines by nature are very heterogeneous. Some of these steps are very fast go logic or very uh fast transformations. Some of them are very slow like image generation that can happen a few minutes and then some of the steps might be working inside on the

local GPU which is also another resource which goes scales up and down. So on top of that, my personal problem with this as the ops person is that look here we have at least three different deployment uh deployments to manage the pipelines which is cubeflow and owned by the data scientists the back end which is a classic fast API servers and now also request response protocol over

the cues. It is just insane to to manage. In our experience, it was taking 60 to 80 um% of the engineering time just to make it work to achieve this uh productivity on the batch. And what was our what was the natural thing uh natural thought to fix it? Of course, just flatten everything. So treat every single uh step not as the monolithical hierarchical uh pipeline but

just as separate um separate microservices where each step uh sends messages to each other. So if it was request response, it would be very hard to um we would get into the same problems as with the classical microservices where um basically the communication would be very hard. Here what what happens is that the message passes to one actor to one microser. Um the microser does its job

and sends this message further. And the key thing here is that there is no single um central orchestrator that knows where this message needs to go f further. The information about the routing is embedded inside the message. Um this allows much more flexibility here. For example, this step LLM as a judge can decide whether the content uh the generated image is good enough or not and then

send it for forward uh for uploading or reporting of down uh down the pipeline if it's good or retry um send it to retry if it's bad. And on the diagram here is how it will look like. So you see the key thing here is that the errors are nicely localized within one actor. If some something happens with one actor, the message automatically goes to the uh

error handling or deal Q or whatever you configure. It can return back the message to the same actor or it can uh it basically does not block any other actors and um the system scales very nicely. Here's one of the um sample production diagrams um where you can see that that different actors scale when they need um based on the amount of work they have in their

queue. So this is thanks to um K autoscaler. Um the most complicated slide here the um architecture overall. So we have the CRD the async actor and the key here is the runtime. This is the Python container which the user builds the user defines their their Python handler and the framework injects this file via config map which is able to communicate with the sidecar via HTTP over

Unix socket. It calls the user handler and um uh the sidecar is the small Golang um container that is responsible for communication over cues and um and so on. Optionally we can um attach the so-called state proxy um sidecars. they are also communicating with the runtime over uh HTTP over Unix sockets and this is just a nice abstraction for um for external memory. So which is very

useful for agentic use cases. So for example, the code would be writing a normal uh Linux file and um runtime would intercept the um this call and translate it into the HTTP call to the state proxy which internally would uh communicate with the external database or blob storage or whatever you need. For now we have implemented the last right wins simple very um high throughput um concurrency

model and the um cast which is um comparing set for um atomic rights if needed and uh we have the external gateway which is just a HTTP translator it looks outside and it is able to uh receive your HTTP requests create the uh SQS or whatever transport message to the first actor and then um yeah the actors will do the rest reporting each the status to the

gateway so that gateway can uh send the um the statuses back to the client. Gateway is stateful. It is connected for now to uh posgress DB but can be anything. And for now um gateway automatically implements two protocols A2A agent to agent protocol which is useful if your um actress pipelines are agentic or simple MCP this is like a RPC for um just deterministic pipelines. right. So

a few code examples. Um this is how the code would look like if um a data scientist or uh someone would want to um to write a robust code on plain um on plain python. So here we have the tennis city retry decorator for catching the errors and doing retry. Um here we have thread pool which is you see a lot of boiler plate a lot of

infrastructure code which is um hiding the real meaning of this function which is basically this. So this is the only useful business logic that we need in this function. Take something call the model on on these questions return results and map um and everything else is just is just the the garbage. And the solution that we have here is to clear clear clearly separate the concerns so

that um data scientists or AI engineers are focused on pure Python functions um and platform engineers are focused on pure CRDs. So here on the right you can see an example of um of a sync actor CRD actually it's XRD because ASA works um on crossplane yeah basically you just define the actor actor name which is the logical name used for in the routting you define the

message the image the handler and then some infrastructure um setup for for this actor one more thing to uh that offers in order to uh improve the separation of concerns is flavors. Flavors are uh preconfigured excuse me configurations that live in the cluster that are set up by your platform engineers and work as kind of like a customize. uh they just work as they apply overlays on

top of each other which allow to um to minimize the amount of the code and boiler plate in your CRD so that we can store this file in the uh repository and make it accessible for both platform teams and data scientists. yeah, here's the example how the message uh mutates while being sent between actors in the flow. So each message has has a payload has few headers

which is payload is what the Python function receives and the root root has previous uh steps that it has been has visited. It has a current step which is now and next steps. And the idea is that the next uh each actor actually each side car when the actor has finished successfully it moves the next step and then sends it further and the actors are to overwrite

the next uh steps using the yield operation. So this is the only complexity that we have in the um in in runtime level on the handlers so that users can turn their simple um simple handlers into the routters and basically change the future by rerooting this step um to another step. Another command is fly. This is again agentic feature. um instead of sending a whole message forward

to the next actor, we can send a chunk of information directly to the gateway basically directly to the user which is useful for for example the um streaming events from directly from the LLMs and yeah so routting is very complicated to manage uh at scale at while having many actors. Um so we developed one more thing working on top of Asia. This is kind of like SDK

called Asia and um it allows you to define the single uh Python function flow to basically turn this describe the control flow that you need in your pipeline and then compile it into a nice flat graph. So um once again this one is hierarchical this is a python and we uh transpile it into the flat architecture where each step is an actor and the control flow routters

are automatically generated and ready to uh to be deployed. Also compiler generates the um CRDs for deployment. Um one small note here you probably noticed that uh here we have the decorator flow while I said that um as as handlers are pure functions and don't use any decorators. Um this is not the runtime requirement. This is more SDK for annotating your function um to tell compiler what

is this function. So basically here the decorator flow does nothing. It just returns the function. You can also use uh a comment or something which I'll show later. All right. Demo. Right. So here is the example um example repository that can you see? Well, all good. Here's the example repository um where you have some actors defined some the flow docker file requirements all standard things and we

have set up asa using uh using the command as init. I will not run the initial commands to save time. But overall um the flow is uh you run initial initializes the um the repository creates the necessary config files and then you run compile and the flow that you targeted which in our case is a regular agentic flow calling separate um actors and in a while loop

and then evol creating them and checking if the exit conditions are reached. Um so this would compile into this thing and you can see um blue dot blue nodes are actors that we have here and these are the generated routters. Routers are look like this. you're you're not supposed to edit them. They are uh compiled. So um one small note is that here in the code you

can still see the retry decorator in the um in the actors. So this is because we have we have a flow. The idea is that you have a flow and you're able to run it as a pure function so that it works and you don't need Kubernetes to test it to make it work. while testing it, data scientists or uh developers need this thing to work. So

they define the retry decorators and then in um Asia has the set of knowledge base which annotates the decorators allows to annotate them syntactically and tell what to do. So basically here we see if we see the decorator retry we map it to this path in CRD. So in uh in essence what happens is that as it takes this decorated function it processes whatever decorators it knows

deletes them and generates the CRDs in here. Yeah. generates the CRDs. Um, in order to avoid generating and overwriting your changes, we're using the customize scheme where base is only generated artifacts and common contains your user overloads that you can set uh using the patch. Um once once you have defined uh your actors you need to build your image. So for now we have we are working

only with docker images and um the mapping of which python function on your local machine is mapped to which image to which function in remote image is stored in the scaffold uh yaml. So which allows us to also um debug the uh flows remotely and yeah we can generate patches customized patches. For example, here we set the min replica count to one so that demo goes well.

And yeah, this is the overall setup. Once we've built, once we've patched and exposed, marked the flow as the one that is exposed in the gateway so that the um gateway can receive HTTP requests and trans transform them into the cues. We just run apply to uh apply this configuration to the uh remote cluster and then I'll so status this is just a normal wrapper over the

uh cub cubectl we have the um resource called um custom resource called asia which is now in all sync state minimal replicas everything is ready Good. Um, now let's run the logs. This also internally runs the uh cubectl and attaches to each port in the And then let's run the uh the text some some command to the A2A. So what is happening now is that we send

the request to HTTP gateway. It created the message sends to the first one. Oh, sorry. There's locks. First actor sends to the second one. Processor sends to the next one. They are failing. They are retrying and eventually eventually we receive the result back. So, uh yeah, it worked and now we asked it to query the traces. So as is integrated using open telemetry with the um with

the uh Prometheus tempo and loi and you can reach all the traces here. You can see the service graph for for this call. You can see uh different custom uh actor metrics and also you have access to lock. So you can see the potential that Kubernetes being Kubernetes native opens to us is uh easy integrating with different um different tools and um yeah pure modularity. yeah uh

to wrap up long live the queue uh and if we set the AI if we run our workloads um as the async uh microservices we with the concept where the message root the root is not an external state somewhere in the um central orchestrator but it's embedded within one message we reach the true composibility ility and scalability per per one request and we're able to implement um

arbitrary um agentic patterns like u agent swarm lies uh naturally here. So uh yeah, you get uh dynamic agent mesh for free and retry policies and other infrastructure configurations are completely decoupled from your code from your um Python code and are the concern of the uh platform teams. One more key principle is that we are trying not to create too many abstractions. we have only one CRD

which is actor and then flow which we uh compile into actors um is not a CRD is just uh actors labeled with the flow label so that the we try to keep the minimal amount of abstractions and um yeah keep it simple thank you very much uh you can leave the feedback to the session on the left the middle QR code leads to the asset.sh SH which

is also the domain name uh landing page documentation. We are welcoming contributors. The product project is on very early stage. So any uh help feedback would be highly appreciated and you can contact me on the uh LinkedIn if you have any questions or if you want to talk about compilers or distributed systems. Thank you very much. Um, we have three and a half minutes. If you have

a question, um, there might be a microphone somewhere in the middle or you can ask ask me in private. I have some stickers.