Enterprise-Scale Migrations Using Agentic Workflows with Human-in... Alvaro Saurin & Jose M Navarro

This talk covers the migration process of an API gateway from EngineX to Envoy, presented by senior software engineers Albert Saurin and Jose Manuel Navarro at Adobe. They explain that the API gateway is vital for managing traffic between various services, including functionalities like authentication and routing. The migration involves challenges such as configuring over 2,600 services across 500 clusters while ensuring no downtime or disruptions for end users. The speakers detail their approach, which includes automating service migrations using coding agents, and the development of a custom agent for verification and monitoring. Lessons learned emphasize the effectiveness of agents in handling numerous migrations concurrently while reducing manual oversight.

My name is Albert Saurin and my colleague is Jose Manuel Navaro. We are both senior software engineers in the gateway team in Adobe that is part of the developer platforms organization in the company. And today we are going to talk about an engineering problem that we have been trying to solve in in our team. It's the migration of an API gateway from engineext to to Mboy. So

let's start by defining what an API gateway is. Um DNS is sending traffic for different services uh to the nodes of the API gateway and the API gateway then performs some vital things like authentication, authorization, uh routing, throttling or analytics and the traffic is uh sent to the different services that are running in this case in in Kubernetes. So the old gateway was based on engineext and

Lua code for the business logic and the new gateway is based on more modern technologies like Envoy or Argo City. The migration presented multiple challenges. Erh the old gateway was u running for a long time but we would like to to benefit uh from newer technologies based on GitHubs and and to provide sales service to our users. But the scale of this migration was breathtaking. um the

complexity of the configurations being migrated involving 2600 services distributed in more than um 500 clusters each one with multiple environments made this a very complex migration and the goals we had were clear the migration should be transparent for users it could not require changes in the users configuration and failure failures were not possible possible because some of these services were really uh critical for for the company

and cannot have any downtime. And just to give you some quick uh glimpse of the traffic managed by the old gateway uh 13 pabytes of traffic per month and 40 billion requests per day. So the approach uh was to try to migrate manually the services that were responsible for 80% of the traffic. uh that's not a very large number of but uh we wanted to use some

kind of automatic migrations for the rest of the traffic for the long tail of services right uh so we calculated that for doing this kind of migrations manually by engineers in an optimistic scenario that would require something like four years and a half but if things went south it could take 15 years. So this is our manager and we decided that this was clearly impossible to do

right in this way. So we found a solution. We would perform migrations of service services in autonomous way involving engineers only when really necessary. Let me first explain to you how this kind of migrations uh look like. This would represent the initial state where all the traffic is sent by DNS to the old gateway with engineext. And then uh we would grab and translate the old configuration

and deploy the new configuration in Kubernetes with Argo CD and then Mboy would load that configuration those HTTP proxy resources that were deployed in Kubernetes. And at this point we can verify that the service can be reached uh through envoy. And then we start with a cut over by increasing the percentage of traffic that goes from the old gateway to the new gateway. This is a progressive

uh increment in the traffic sent through the new gateway. Right? And finally we send all the traffic through envoy and we get can get rid of the old configuration that we had in So this is the list of stages I have just described and we thought that there are some of these tasks that could be in maybe we could automate or well an agent could help us

in this case, right? And we identified some possible candidates uh for this um use of agents here like the translation phase, the verification of the service and the cut over phases. Let's start with the translation. For the translation, we thought okay if an agent if an LLM can translate French to German, maybe it can also translate configurations, right? So how could we do this kind of translation

with the help of LLMs? Let's look at the the problem and what we must do with these configurations. In the old gateway, we used to have YML uh that were translated to NX uh in in the new gateway we want to generate values.jl files that are distributed in Argo CD repositories and those values.jml JML files are going to be uh used for rendering resources like HTTP HTTP

proxy resources. Right? Initially we thought this is just a question of transforming a YML file into some other YAML files and NLM could probably do this kind of thing. Right? Erh but the first problem that when we found was that the translation between very different schemas was not easy for LLMs. they got confused. Um, the LLM was not understanding how to move information from one place to

the other. And this also involved some training for LLMs and that require lots of manual translations that we wanted to avoid. And in our tests, we saw many regressions in this process. uh when a translation was correct uh another translation was broken. So it was not really a deterministic pro process right. So then we discarded this solution and we changed the approach to this problem. So we

thought okay if we want to have a deterministic process where the same input could always produce the the same output and considering that LLMs are very good at generating code and code is always deterministic. The idea was to use an LLM for creating the code that was going to do the this translation. And by doing this, the quality of the ideally should converge to zero and have

no errors in the in this process. So the translation would be based on a tool and the that tool would be created and refined refined by a coding agent and an engineer would be responsible for driving the agent in this refinement process and ideally this engineer would be less and less necessary with time. So we created this kind of loop where um we start by translating the

configuration uh from each service and producing some new configuration and some logs and some other intermediate files and those are very important because they give some understanding of what is happening to the human to the engineer and to the agent. The engineer can then review this translation and intermediate files, find some errors, schema inconsistencies or things that are not properly translated, right? And then the engineer is

basically going to tell the agent, okay, this is what I expected in this translation, but this is what you produced and these are the locks, so please fix it. And for this process we are we are using a a regular coding agent something like closed code or something like that. We are using smart models like OPUS but it's a and it's important also to note uh that

we use previous translations as golden files. So previous translations are kept ensuring that no regressions are produced uh when we try to refine the translation of new services. Right? So this becomes a cumulative process where more and more iterations would ideally produce fewer and fewer errors. This loop has some pros and some cons. We the first pro is that we see and we understand this process. We

see the code. We are platform engineers. We are used to to code. So we can understand and we can also work on multiple translations in parallel. So we can parallelize the work and we can speed up this development development loop and this is a progressive process that we can measure the progress where we refine the tool until it doesn't produce any errors and reducing the human evaluation

step. On the cons side, the corner cases, that's the the worst part, right? The weird configurations or some cases that consume a lot of time in refinements, even requiring some manual interventions from engineers and changes in the code. And now my colleague Jose Manuel will continue with the next steps. >> Thank you, Arl. So uh as Alvaro said we have an inner loop that we are able

to create a tool kind of autonomously using coding agents but we need to invoke this tool somehow. The traditional way is an engineer uh running the tool but we went to go a step further. Uh the tool is used to run the next steps in the pipeline. Next steps is verifying that the migration was correct, checking the endpoints in the new gateway and doing the DNS cut

over gradually. So for those steps we used custom agents. Um the first step is discovering uh endpoints uh paths. Uh you can imagine 2,00 services a lot of weird uh paths and endpoints. So we get all the configuration produced by the tool. We inested into an agent and we were able to discover what are the tools the the paths that are uh that can be invoked and

we send HTTP requests and and check the result check the status. So this way we are able to have kind of liveness proof uh to to see if the if the gateway the new gateway configuration is is correct. And then uh we start incrementing the traffic. So we change uh with an agent connected to the DNS. uh we start check uh changing the DNS traffic split and

we start uh with a small amount of traffic sending traffic to the new gateway and then we start monitoring the the logs and the metrics for the new gateway and we compare if the metrics uh the red metrics for the new gateway are similar to the old metrics. So if the in the old gateway we have around of 2% errors uh we check that we have the

the same error rate and if something goes weird and the agent detects uh something that is not going well uh we stop the migration we roll back and we invoke the human. Uh this only happens if the agent detects that that something is not going well. So in the 90% of the situations everything is going as expected 10% of the migrations we stop and give the control

to the engineer for this uh agentic loop we used a custom agent created by us using lang graph as a framework and I'm going to going deeper with this to understand how a custom agent can be created when you are writing your your own agent. Uh you are on top of this parameter this stack. Uh at the very bottom you have the foundational models. You have the

SDKs provided by the provider like open AAI anthropic and so on. Also you have kind of LLM abstraction frameworks like like uh lang lang chain and so on. And then you have a proper framework, a gentic framework like in this case lang graph. There are others and then you start with your own code. Um in your own code you have your system prompt. You have uh kind

of different context uh information that you provide to the LLM or the agent itself. A lot of guard rails like don't do this, don't do that. And also you have the agentic loop um and also tools like your agent is connected to tools and it it decides to call this tool when this happens. Uh in a nutshell an agent is not more like this. It's just an

LLM with tools connected and it's running in a loop and making changes in the environment through one of those loops. one of those tools and reacting to what happens in the environment. Okay, so this is for example one of our prompts we send it's a complex long prompt with different sections. One of those is the goal then the task that should be done in this specific step

then an example of the output and then a validation step. uh for instance for the validation we invoke a tool which is a kind of a llinter. So the agent is able to iterate in different loops in different rounds and invoking tools and checking if the result is as expected. So in our case this is the kind of the environment uh the the loop the agentic loop

in the environment our tools are able to invoke Splank GitHub repositories to make commits and send uh ubernetes to monitor the nodes and make changes in the cluster. Argo CD to deploy new changes to the cluster. And next step we are integrating with Slack to send uh messages to to the engineer and our stop condition for the agent is okay depending on the metrics the environment situation

and the logs and everything the agent makes a recommendation to roll back or uh continue. So the engineer in charge is the one that executes or makes a approves uh if if the agent should roll back or should move forward. So uh in the previous architectural diagram uh in the verification step where we are uh we are testing if the configuration is correct uh we have a

testing agent which discover the paths the rules um and send synthetic traffic to those paths. So we check that everything seems to be correct the the traffic is flowing through the new gateway. This is done by the testing agent. And then in the cut DNS cut over step, we have the cut over agent which is connected through a tool to the DNS and starts making changes in

the traffic split and then uh we have the same agent actually does the monitoring part checking splank logs reds metrics and so on. We check that the error rate in the old new gateway is similar to the error rate in the old gateway and this is done by the cut agent. Okay, just to wrap up uh what are the lessons learned in this project? First uh we

realize the agents or LLMs are very very good writing code uh but not that good doing specific niche uh tasks. This is probably because the LLMs are trained with a lot of myriad of repositories of source code and the industry is pushing very hard to make uh agents and LLMs very good at writing code. So we take advantage of that instead of asking the agent to produce

configuration or or envoy configuration we ask the agent to produce source code. Uh and also a good benefit is with code we produce deterministic behaviors. Uh with agents or with LLMs you have undeterministic uh behaviors. With code you have what you want every single Then uh we learned that rollouts in migrations and monitoring things is a very time consuming. Most of the time you are just checking

logs, checking uh graphs, metrics and saying okay everything is okay and in some specific points something goes uh wrong and you start acting. So with this kind of autonomous agentic workflows you save the engineer to waste his time her time and the engineers only invoke when and this can be kind of 10% of her time compared to fulltime monitoring for uh in the the old approach. And

last lesson is with agents you can do uh this kind of easy tasks or well-known task at the scale. You can manage hundred of migrations in parallel and you have kind of confidence that you will be invoked only when necessary. So you don't need to pay attention to every single migration in serialized. You are just managing hundred of migrations in parallel and you are multitasking between all

of them only when when it's So this is uh what we did. Any question? How many times the migration invoked the agent? How long did it take? >> How long did a migration take? Uh the translation is just running code, traditional code written by an agent. But traditional code migration is a few seconds. Then depending on the amount of traffic, we keep the agent running more or

less. For instance, we have high traffic services with thousand of million of requests per minute. So in a few minutes, we are able to know if the service is healthy or not. We have other service with a few uh requests per minute or even per hour. So we keep it running for hours and maybe at the next day we we have a message uh from the agent

saying okay I made a roll back because it wasn't correct or it keeps running if it's if it's correct. Yeah, we're we're still running the pro the project and we are migrating in batches 100 of services by batch. uh we expect uh we are start starting we are starting with the first services and we start we expect to finish by a few months at most >> we

have across that work >> yeah for the golden files if you mean for the for these golden files we are storing that in a monor repo. Yeah, it's a we are storing everything in the in the same repo and for any new service that must be translated then we use that repo for checking that we don't have any regressions and so on. Um yeah, any new service

becomes part of that repo? Yeah. Can you see the end please? >> How did you handle this aspect of it? So these agents are able to trigger deployment of the API. So how do you ensure >> the agent doesn't have direct access to critical uh actions? Uh there is a tool in the middle. So the tool uh it's sometimes it's a MCP server, sometimes it's a CLI

tool. The tool has security enforcement and measures. So the agent just says do this to the tool and the tool makes a decision not a decision has a a hard code path to do it or not depending on security constraints. So the agent doesn't have the real power. The agent is just tries to do things. >> One more question. Did you encounter any tool by behavior? >>

Say it again. >> Uh, one of the recent problems with approaches is tool bypass the agent sees a tool treats it as a black box to avoid it testing for testing. So on >> we don't give that much flexibility to the agent. is just a rail. It's a fixed rail of actions, a sequence of actions. So the agents move forward in that rail. It's a linear. So

we don't want the agent to be creative and creative and imagine how to figure out how to do things. We want the agent to run constantly. >> that's a good question and it's probably the challenge that we are facing because we have right now kind of 10 20 agents running in parallel. We have LSmith as observability platform. So we have a lot of traces and so on.

But we realized that to handle hundred of agents running in parallel would require kind of dashboard kind of uh something to to be able to uh oversight everything. We have the low-level tracing and observability using lang. >> Okay. Thank you very much for coming. >> Thank you.