Intelligent Routing for Optimized Inference - Antonio Berben, Solo.io & Felipe Vicens, Telefonica

This talk focuses on optimizing AI model selection in telecommunications by introducing an architecture that utilizes open-source tools for efficient routing. The speakers discuss the inefficiencies of using a static model for all AI requests and propose a solution that dynamically selects the best backend model based on factors like latency, cost, and accuracy. They emphasize the importance of a robust gateway system to control traffic and route requests intelligently, comparing different technologies such as Envoy and K Agent. The session also explores the integration of various components, including LLMs and semantic routers, to manage on-premise and cloud-based deployments effectively, while adhering to regulatory requirements. The presentation concludes with insights from their research and a demo showcasing the intelligent routing mechanism in action.

Hello everyone. I'm Philipe Visense from Telefonica and here with me is Antonio from Solo IO. And today I would like to start this presentation making you a question. Are you using the same model for all of your AI request at the end? If you use the same model for the simple queries and the complex queries, you are wasting some resource. In this presentation, Antonio and I will

show you um architecture in order to create by using open source tools a system that is able to route the best to the best backend model and h optimizing the request by latency, cost, accury and in anic way. Coming back to the original question, are you using the same model for everything? And probably the answer is yes. So if you use an static model for all your

AI requests, probably for the simple queries, you are wasting some compute resource. Let me put you an example. If a user wants to make a request to translate hello to Spanish, you could have two options. The f the first option is to use your static model that is GPD5. is a very good reasoning model and it cost well uh for the input $2.5 of per million of

token and for output it's around $15 per million of output tokens. The second option that I have the user is to use a small language model that can make it in very efficient way and for example the Microsoft 53 that use for input tokens around $03 per million of tokens and um $0.52 At the end you will get the same response from both models but with a

fraction of the cost if you use the second option. The different between both is around 27 times x cheaper the second way if you wrote the request to the SLM in telco environments we have a highly distributed infrastructure. um we need to take into account several variables that can affect our business. um if you if you have this uh distributed infrastructure, you have the edge, you have

also the core data center and the cloud and you can run the model in any place. The second variable that we need to uh think about in tel environments is that we run our our business in a fully regulated environment. We need to follow the AI act. we need to also erh be restricted by data sovereign and at the end we need to apply some um SLAs

that uh applies to our users in order to guarantee the best uh functional of of the uh of the business. So the third option that we should take into account if is if we go to um um model where we use the AI provider API then we will um fall into a token token price based model where we need to paid as we grow for the money

and there is a mis um misleading ing the configuration or the uh revenue revenue revenue that is not growing at the same piece that we use APIs. So we need to think how can we build a system that will be able to for one side run any model in the whole infrastructure that we have and put our user in order to guarantee the SLAs and so on

the best model selection in any place that we uh run our system. So we will discover how can we do that in the following presentation. Well, I I've got the easy part, which is how we do this magic. Pretty simple. We need a gateway. We need to put this that component in the middle to control traffic to all the LLM back ends. Most of the solutions that

everybody use and we you're going to use cloud native open source solution, they are using envoy. Enboy is well known. Uh if you're not familiar with Enboy, Enboy is used API gateway as an API gateway. Probably the best API gateway in the market. It's open source best API gateway in safe. So it's quite good performance. But we found several issues. doesn't scale pretty well and we really

to scale it huge and also uh sometimes not performing as we want for aentic solutions is not the best solution because it's an API gateway so what if using the same components control plane K gateway data plane the another open-source project belonging this time not to CNCF but to Linux foundation that actually it's pretty similar to Mboy right but is written in Rust which means performs quite

good and um basically in my opinion honest opinion it's the best agentic gateway because it's not the same it's not the same a component built for API gateways than a component built for agent components remember that one the goal is for one it's restful APIs fail fast the other is stateful requests another thing we need to do is to make it extensible this idea of cloud native

we extend we cannot build something that is battle tested the gateway we can reuse it but then we need to customize how we do it through the service this case XRO So we are able to basically reuse what Mboy offers the API which is the best part of Mboy and just replace Mboy with agent gateway as you can see we are exact doing exactly the same we

are going to reuse the API that is the magic and then we place there our custom thing the intelligent router how we do that it's basically we're going to make it aic we are living in the agentic world everything needs to be So uh let's do it. Intelligent means agentic nowadays. What we need is something to trigger the agentic flow that can be CRD. The outcome it's

going to be the configuration for that agentic system magic here. This works for every single system here. It's intelligent routed. Keep in mind nowadays whichever system can be configured in an agentic way. We're gonna use another CNCF project which is K agent. It's declarative way of deploying agents. I recommend you to use because it's very handy. And we're going to uh define kind of an architecture which

is based on an orchestrator and communicating to different works uh workers worker agents and then we're going to leverage classic MCP MCP service in order to increase the functionality and then the magic skills but this is not clo skills this is more it's a mixture between CLI and H code that is the way K agent runs the skills and is very handy because helps you a lot

to do uh to define the workflow agent but just using the skills we will see that later and then how we built uh the intelligent router please Felipe. >> Yes. Well, basically on this architecture we were proposing here the request are coming from the user and arrived to the agentic uh to the agent gateway. This gateway is configured with a policy that basically makes a pre routing

to our extension. And this is where the magic happen. The intelligent router just need to set the headers in a header based routing way and uh when it analyze the body the the other header that you have and make the decision which is which is the best model to to get then it will return the same request to the agent gateway with the with the headers already

set for the best bucket. etc. >> So during our research we were trying several approach. The first one was uh was to to use LMD. LMD is a very good project for inference and this is focus on the on-prem side. You can deploy your models with LMD in a very optimized way and it use the prefield and and decode this aggregation to optimize not only the load

uh control the load to to send uh to balance the request but also the the catch the KV catch where it's the back end that have better catch and so on. So uh the consideration what that we have with LMD is that we will also like to we would like to use the open public APIs not only the private. So we continue our uh investigation. So the

second product we we saw as an extension for this architecture uh was the open policy agent where it was pretty easy to integrate and then uh you you will be able to develop your own extension using OPA and OPA provides you a very uh fast in order to analyze the the policies that you set and then to select just just to to make the headers with the

The difficulty we had had and the consideration we had with uh open policy agent was the dynamicity of the data or realtime data. We couldn't uh let's say modify the the back end dynamically based on the real-time metric like latency, the cost of electricity and so on. And that's why we moved to the the next project that is the H build len semantic router. Bill lens semantic

router was the the other project we analyzed. Um this is very very good uh intelligent router and if you want to to use an intelligent router you you must look look this project. So uh what we did was integration with the agent gateway and LLM BLM semantic router. We made a pull request and put all of things working. But uh the consideration that we have was that

the CRDs in Kubernetes they were not fully uh complete align with the configuration that you can do if you put the the config inside the the config jam. This um a huge config jaml uh with very um options that you can use. But at the end that moved us to build our intelligent router that uh uh fulfilled the the the goal that was just to optimize uh

the cost the latency accuracy and uh in an agentic way. So um to build our intelligent router and demonstrate the architecture what we did was to have two parts. The first the first part is to keep the data plane where the user calls the agent gateway and and produce the header selection for the best model. And the second part was to make it aentic. That's a aentic

control plate that we create uh for the agentic part. The the key point was to have a common uh configuration that is register in a CRD is LLM backends. You register your um model there and then all the magic The orchestr the the gentic part have a orchestrator agent orchestrator made in in Python with ADK and it acts a kubernetes controller. They have um a part that

is um controlling the whole LLM flow managing the CRD and so on and also they can that it is able to make calls to the agents via ADK. Which agent? We have the cost of electricity agent. We have uh evaluation of the model agent in order to get the inside of the model like a currency latency uh and use it that information later for the intelligent router

and also we can uh call the the electricity the electricity agent. So with all the information we fill the the CRD and put it available for the intelligent router. The the registration flow is the following. We have uh created uh you have your model. You register the model and you create that LLM backend CRD. So um the agent orchestrator is watching that CRD and uh it triggers

the creation of in in a gateway it triggers the creation of HTTP roots and also the agent getway back in. So that's complete the configuration in in the agent gateway part. After that it um triggers the agent of evaluation runs all the evaluation of the model get the insights and then uh triggers uh the model cost the agent. After that it triggers if the if the model

is deployed locally and it triggers the electricity cost in order to make the the adjustment for the model and at the end it uh updates the CRD with the whole status configuration of the of the router. Um and that's it. Um in every project there are missing pieces we couldn't finalize. This is not production ready this just for the demo. Things that we encounter in in in

the project that is uh open source and is available in the repository. It's for example telemetry. Uh in order to define rules around latency in a distributed systems uh we need to counter that the metrics are we need fresh metrics and when it's distributed so we need to think about how to solve this maybe defining fresh uh metrics for the local models while other strategies for the

remote models. We encountered on that geoloccation. Usually you deploy models different models in different infrastructure. Here uh in this region we're going to have four models and certain other region we're going to have uh seven models. So we need to drive traffic defining rules for those ones and we need to secure that traffic right how to secure that is zero trust. We need to enforce zero trust

in every single communication and we need to apply policies to enforce that. Uh to solve most of the issues basically we can do it with uh just is at the bottom serves as a really uh ground ground foundation for the zero trust and the observability measures. H we collected 50 different use cases quite interesting that you can you know in here in this in in this repository

you're going to find we talk about cost we talk about latency we talk about um um accuracy uh we collected 50 different use cases I'm going to talk about two the most interesting one is like one is thermal and power aware scheduling this is uh for example in a data center uh when you are using a lot one model deploying an infrastructure tends to heat up right.

So the efficiency of that infrastructure is going to be um punished. So we need to define strategies in order to measure the temperature of those infrastructure and drive traffic based on how how it's penalized. Right? Another interesting is exponential backoff resiliency writing. I love those names. I love those names. Basically, it's like a token based um token based solution. Um imagine that we have like a quot

for different users and I'm going the one specific user is going to reach the quota. So we we want that that user that is going to consume is consuming a lot that developer is consuming a lot of uh uh prompts and consuming a lot of uh energy. We're gonna try to punish the usage driving the traffic to a different cheaper model and we're going to serve by

guard rails that if I have remaining 50 tokens for the next hour I'm going to make sure that it's prompt it's just r to that amount of uh tokens that they have left right um we learned some lessons during our journey it's six months uh cloud code came right at the at the middle of that. So you can imagine the impact of this. Everything is agentic. Lesson

number one, everything can be oriented to agentic. Absolutely everything. In this case, we are proving that every single configuration, every system can be configured in an agentic way meaning in an autonomous way. Skills are extremely powerful. H before we were defining expensive and really hardcore agents with uh workflows. Now with the skills we can define a distributed fleet of agents communicating to each other and enforce the

flow through skills. Remember K agent skills not close skills cosmet context dependent. Of course, context is the key point. We need to analyze the context in order to define how we're going to reduce the cost. And of course, observability. Observability we need to observe first in order to define the policies that is mandatory. I we just want to mention that uh before we mention um zero trust

we are actually with another companies uh we are uh actively participating in a challenge uh which is the 4N forum and it's around how to bring CD trust into aic solutions we just want to mention that we are also putting this as part of the overall picture and well we have a demo that Philip is going to show us we manage Okay. So, yeah, prepare a demo.

Just a second. >> Full screen. >> There you go. See the play. >> Full screen. Full screen. Full screen. >> Okay. >> See the play. >> Yeah. So we have prepared a demo with the beginning of the demo is the the agentic flow and the second part is how is actually routing our intelligent router. so it start when you create the LLM back end uh with the

just the model configuration then uh the gentic flow starts and as you can see here um the the first thing that it register is the HTTP road and agent gateway back end that are the components of the agent uh are created on the system. This is the the first things that um the the controller creates. After that, the evaluation job is triggered. This this is important because

uh we use skills the Kian skills in order to have a exactly what we want to have at the job uh the the job we want to create. So it is created by an skill and some seconds later um it triggers the evaluation also it control the phases of the CRD as you can see there it's uh evaluating and you have the job already created uh so

it start evaluating the the model basic basically Ally use mmle pro in order to have an standard and see which are the the domains the accuracy per per domain that we can get. So this is all the requests we we are uh for for the demo we are just only taking 70 requests to the model and it creates the the base the baseline for our model that

uh have the accuracy and the and the latency. latency is how how much time the model takes to to replay replay the the queries. So after that the agent orchestrator the other agents in order to get in this case. You can see here uh the the price of the electricity at that point to adjust the the cost of the model because this this model is deployed locally.

So it communicate with the API of the electricity and get the the value after that it will um trigger the other the the price of of the model itself. So at this point uh the the phase is evaluated and you have the model with all all the parameters h already there latency price and so on. second part of the demo was the intelligent routing in action. So

basically what you what you see here is that uh you can see the request the first request is to send to a health domain and you can see that is taking the the one of the model then the second request is to the finest domain the router is routing the request to a different model and uh it's been updated based on latency the the different So the

the second strategy that we use is the cost that it will be shown uh just right after uh and latency. So basically that's that's the demo that we have for you today and I hope you have enjoyed it. How can how can we put the our names? >> So if you have any question, please come ask. >> Hello. Thank you for the presentation. Thank you. Uh I

work at NEO. It's a sovereign cloud for AI and data privacy. Uh I have one simple question too actually but do you have a clear cost vision on how much everything aentic uh had in return for you like do you have a separated cost like okay this is my agentic cost because this is also AI and electricity usage do you have a separated chart and how much

that increase in your cost for the infrastructure maintain that's question >> yeah the the agentic workflow runs on SLMs to make the uh decision of which is the the the best model. So actually it runs inside the Kubernetes cluster. So no cost at all. >> Interesting. Okay. Well, thank you. The other one might be a little bit more complex. So >> Okay, we >> if I see

you guys for a coffee. >> Yeah. Yeah, we totally. Okay. >> Hi. Uh thanks for the presentation. I wanted to ask about the accuracs that you have. Is that something that's uh uh that you took from somewhere or is that something you developed specifically for your telco use case? >> Yeah, we took the idea from build and semantic router that is doing something simil similar and we

took the script from there in order to to use it for for the demo. But uh that's the that's the from the mmlu >> as we explained is basically like uh semantic router is offering too much And we also wanted to increase with other routers. So we just pick up the idea is not to create a product out of this just to show that actually what are

the capabilities and the possibilities and also to make it aic. >> Okay. Thank you. >> Thank you. >> Hi, thank you for your presentation. Uh I have two questions. The first one is on average how much latency are you seeing uh because of that extra uh layer of logic that you need to think to decide which LLM to call. >> Yeah. And since the SLM that make

the decision is running on on CPU, it's pretty fast. It's around 50 milliseconds per per request. So having two seconds 5 seconds of the replay this the aggregation of this h small amount of of time is not significative. >> Thank you. Uh the second question is you shared some uh uh uh logics for selective HL act cost. Have you got to the situation where you need to

come up with some kind of mathematical formula to combine multiple of them and find one single best output? >> Yes, that is what we did with the the demo. I didn't explain but at the end we have a formula when we put the system on a currency then the the weight of a currency is 80% and the other two is 10%. And then we if you configure

the system for cost then it's 80% cost and the other two 10% of the same for the >> Hi. Um can you uh tell us more about the uh tool that you're using? Uh I'm interested in that phase. Uh I saw that you used probably a docker image for doing that. Uh but I couldn't read it. Yeah, it's basically uh we pack in the in in in

that job in the docker image uh pro benchmark. So we we can I can I can share with you what we did after after the session but uh uh we took the idea from build semantic router and apply it here for for the demonstration >> ju just to make to give more information is like we took it and we make it aic. So it's part of the

agentic flow that happens in order to collect all the data that then is going to utilize by intelligent router. So we are decoupling that from the intelligent router. >> Thanks very much for the talk. Um I might be mixing things up but um for example has the inference gateway. Uh how does that sit in this picture with agent gateway? Is that two separate use cases or are

there different approaches? uh yeah >> I mean it's similar approach but the problem is like they didn't solve the problem about distributed uh distributed inference like in different regions so that is the question that for example if you saw LLMD LMLD basically requires a gateway that gateway could be potentially the aging gateway as well and indeed you can you follow that is going to happen uh but

the thing is what we wanted to show it's really distributed solutions In order to do that, you cannot use right now inference API, Kubernetes inference API because it's intended to be in local routing in local clusters. So if it's remote, we need to just go wider and we cannot use basically inference inference API right now. >> Okay, great. Thanks very much. >> Hi. Thanks so much. Um

a question I have is how is what is the benefit of using this over BLM semantic router what are the other than cost what will make this better or how would you compare this with BLM >> you want to take that >> uh sorry I didn't catch the question >> how is this better or where do you compare this solution with BLM semantic >> yeah so as

as we explained is like the semantic router is coming it's a it's really interesting project is coming with a lot of features and it's compiled all together. What we want to show is here is like now we are in a we are living in an agentic world. we just decouple all the logic that we can preface right we're going to pre-ompile all this data and then another

factor that we saw with the semantic router is like you cannot fully configure through CRD this is happening now but still not yet when we started we really want to gentic flow to configure the intelligent router that is why we couldn't just rely on semantic router and we need to rebuild our own solution again this is just for demo purposes because the intention is not to build

a project on top it's Just to show that you can do it,