Redefining SLIs for LLM Inference: Managing Hybrid Cloud wit... Christopher Nuland & Hilliary Lipsig

In this talk, Christopher Newland and Hillary Lipsig from Red Hat discuss redefining Service Level Indicators (SLIs) within the context of modern large language model (LLM) architectures. The session focuses on the challenges Site Reliability Engineers (SREs) face when adapting traditional metrics to manage LLMs, which are increasingly stateful and based on non-deterministic behaviors. New indicators and Service Level Objectives (SLOs) are introduced to ensure performance aligns with user expectations and overall system reliability. Key technologies discussed include LLMD, which facilitates cache-aware routing to enhance efficiency in LLM processing, and the significance of telemetry for tracking metrics across dispersed model components. The speakers also highlight the importance of measuring user experience and satisfaction through innovative metrics.

Good afternoon, everyone. We're going to go ahead and get started. All right. Let me quickly introduce myself. My name is Christopher Newland. I'm a technical marketing manager managing our inference stack at Red Hat. I'm primarily involved in our VLLM and our LLMD projects and communities. My name is Hillary Lipsig. I'm a senior principal site reliability engineer at Red Hat. I primarily work on something called Azure Red

Hat OpenShift. And then I also do anything else that seems kind of fun and and help Chris out with blogs and other such things here and there. So today we're going to be talking about how to redefine SLIs within the modern LLM-based AI stack. You may have saw during the keynote something called LLMD, which is a project that I'm heavily involved in. We'll talk a little bit

about how that project kind of feeds into this conversation and then go over some of what some of these new metrics, SLIs, SLAs look like as we start creating new business objectives within our organization. One thing I'm going to say real quick, I've been dealing with a GI issue for about 24 hours. I'm doing a lot better, but if I step up stage for a second and

hand it over to Hillary, you'll know why or why I'm sipping sipping ginger right now. With that, let's just jump right in. So one common thing that Hillary and I actually is kind of what brought us together here at Red Hat. One commonality we have is fencing. I did Olympic fencing for a number of years as an epee fencer. So I was in the junior Olympic circuit

within the US. I did that from about late elementary school into college. And then if you want to go ahead and kind of explain your background, it's a little bit different than mine. >> background. So generally speaking, I have a background in in athletics and martial arts. So I played soccer, I played water polo, I did karate and taekwondo. And then a few years ago I got

into something called historical European martial arts. So, if you see my long sword earrings, I am trained in German long sword techniques. I have competed and I instruct for German long sword. Uh historical European martial arts, I'm going to just really nerd dump on you real quick. You didn't ask for it, but you're welcome. Uh is the historical predecessor to modern fencing. The manuals are from the

late medieval early Renaissance period where dueling to the death was illegal and dueling to first blood was becoming more common. People who would go to these tournaments and duels often were like blacksmiths and bakers and they had normal lives. So, this is the the manuals that have survived are all of the master techniques for defeating common fencing and we don't know what common fencing is. So, uh

that that tradition carried through 400 years until you get to the modern sport fencing that you see in the Olympics. The swords are bigger, they are heavier, they are funner. That is a word. Yes, you're welcome. Um and this is this is how Chris and I of course like he said became friends at But, we don't necessarily want to be fighting our clusters, especially as we start

deploying these large language models. Uh a lot of the traditional approaches that we have been doing for applications, databases, and other services start to look a little bit different when we start putting it in the perspective of a large language model or even a fleet of small language models, which is starting to become a lot more popular now, which we may mention here in a moment. >>

Just a real quick like poll. Like um who who has been using like an LLM and had like a sub optimal experience? Right? >> We were expecting that. >> Yeah. Uh has anybody used an SLM yet? It's like the sub 10 billion parameters. Yeah, okay. A few. Great. Cool. So, then you're going to have some reference point to some of the things that we're going to be

discussing today. So, what we're finding with uh So, one of my big jobs is going out and talking to people in the community, uh talking to some Red Hat customers, coming to conferences like this. And one commonality we're seeing is that SREs are having challenges adapting into this new paradigm. We still have these traditional approaches, but now we have to look at things in um a non-deterministic

way. We also have to look at uh stateful being a lot more powerful, and we've been pushing stateless for about what, 15 years now? And now we actually have to go back and start looking at how things are very stateful when we start talking about AI. And this poses a lot of new challenges, which we want to talk to you today about on how it there's a

lot of actual metrics that will make it a lot easier for you to monitor these types of LLMs. Um once you understand what those metrics are and what they're representing. I think the one thing that I would say AI is really surfacing is that um I've said this many times that our abstractions from the hardware are not as perfect as we would like to pretend they are.

Uh AI really brings us into sharp focus um because of the necessity for GPU and how GPU works and how it is different from CPU, which I will not nerd dump on you about that, but if you want, get me in the hallway. Yep. And just for those of you who who aren't familiar, uh this is typical what a an LLM request looks like. So, some of

you who might have an SRE background or platform background are probably very familiar on what a traditional REST request looks like, HTTP, HTTPS. Um this looks a little bit different when we start looking at the LLM. Uh the prompt goes into typically a queue of some form. Um this is how we do things like batching. And then there's a few different phases for the LLM. Um the

prefill is that first phase when you first message the uh you know, your chatbot like chat GPT for example and it's just kind of sitting there thinking it's the time from when you first sent that message to the time that you're seeing that first reply. What the LLM is doing is it's creating its initial structure, its initial weights so like what we call a KV cache instance

at that point. KV stands for key value. Yep, yep. It sounds more impressive when you abbreviate it but it's actually not that impressive. And then we go to the decode phase. The decode phase is when you actually start seeing words and letters kind of coming back out. And so these two phases are very critical especially when we talk about tools like LLMD like you've been seeing in

the keynote. I have a clicker. So when we talk about new metrics, right? We've said that a couple times. This is basically what we're looking at. So when we're talking about like what in SRE practitioning what we have to worry about is not just is the system reliable but is the system meeting user expectations, right? Is it performant and what does that performance like mean in terms

of like happiness and you measure happiness with an SLO, a service level objective. So you can take many service level indicators and build them up to a service level objective. Sometimes the indicator is also the objective, that is okay. All these get down into service level agreements, that's the part where like money is changing hands because you've made these promises, right? This all builds up to this.

So if you're using, selling or even internally providing some sort of AI experience then you need to have it to be reliable. If it's for internal developer tooling, this is the the AI bot that does a bunch of the op stuff and helps developers onboard to the platform. If it is a a customer-facing thing and the customers need to have experience you know helping them to understand

how to use the product or to navigate documentation or however whatever it is you're doing with AI it needs to have a experience for the end user that is delightful. So, to do that, we have to start measuring what we are putting out. Um in fact, we probably already should have been doing this. We're late to the party as ever with these types of things. Um and

so these are the metrics. I will not read this slide to you, so please do take a picture. Um these are the metrics that we have to start um instrumenting around. One I I will point out just because it frames back to the LLMD donation announcement is this idea of um KV cache hit rate. Essentially, as you start deploying fleets of models, uh as I was talking

about, those are stateful. There is an instance of the KV cache that has information from previ- previous context and previous prompts. And if you have a similar question, doesn't need to even be the same. It could be 80% alike. What we find is we can have upwards of 100 to 200% increase in time to first token by making sure that we're routing to the correct cache. Uh

this is what technologies like LLMD are doing um when we start looking at this kind of new paradigm of managing fleets of models. Um so LLMD is a I'm I'm not This isn't a a full LLMD talk. So, if you've missed a few of those this week or you missed the keynote, um just look it up. Uh we have a link at the end that will take

you to the website. Uh but just know that it's broken up down as this really with focus around as this cache aware routing. How do I route to the right model that has the right cache for the prompt that I have? And when we do this, as I was saying, we find very often, especially when we start talking about batching, we get upwards of 100 to 200%

increase when we are looking at time to first token. Prefill decode, I'm not going to go into this too deep, but just know that this is kind of the idea of taking the LLM as a monolith and treating it like microservices. Let's break the LLM up into its parts, the prefill and the decode, as different pods. So, they can reserve the resources that they need specifically, cuz

one is very memory intensive, one is very compute intensive. And what we find is we get upwards of anywhere from 5% to even upwards of 30% drop in latency when we start breaking this up. And then we also free up quite a bit of GPU resources when we go this route as well, because now we're not over provisioning the GPU. Um this isn't talking specifically about PD,

but just know that that's one of the critical technologies. And then what we're talking to you about today is more of that telemetry side. How do we get the data now? Because we have broken up this into its individual parts. We've now made the LLM a first-class citizen, all those individual parts. And that allows us to track the metrics more efficiently and effectively throughout the entire cluster

and get the type of information that makes this now much more accessible than it was before with instead of just having one central pod with all this kind of stuffed in, now we have individual pods where we can actually pull this out, or individual services where we can pull these metrics out much easier than it was before. Right. So, um gosh, I feel like that was such

a good summary. I don't know how to even follow that anymore. That one's a good one. I I know, it was a good one. Uh yeah, so good put, right? It's It it it's it's it's different than um something like, you know, just making sure a percentage of API calls return within the same like the ideal time parameter or like within, you know, without without a an

error of some kind, right? It it's it it's it's really more, I guess I want to say it's more experience-driven than even that. Yep. Because of the nature of the beast. So, this is it's a way it's it is um in some ways it's very much still related to that type of measuring, but it's also making sure that like everything is going really like smooth it's it's

more about the whole end-to-end Correct. >> than it is just like an individual point in time. There we go. That's I got I got there. And LLMD makes that a lot more accessible because now we have a lot more access to the end-to-end. We have a much bigger picture now that we're sca- handling the scheduling, the routing, the gateway components, and the um the individual metrics of

each part of the LLM, which now allows us to actually define that good put a lot more efficiently than what we did before um having something like LLMD. Right. So, we talked about I I said this before, right? So, you have an SLI is an indicator. Um and it can be uh many-to-one for an objective, right? So, the objective is the area the functional area where if

I'm not if we're not meeting our objectives, then like some human something should be getting paged and and investigating and looking into that and seeing what's going on, right? So, we've got a a few examples here, right? Um A- A- Again, these are all about really that I I I I I I I I I I I I I I I I I I I I I

I I I I I I I I I I I I I I Um so, you know, chat and code completion and and batching. Uh I actually really do want to spend a little bit more time on batching specifically. Go for it. >> As I think it's a really I I I I I I I I A few hands. Okay. Real quick. Batching. It's where you take

really similar things and you process them at the same time. And you can do this with CPU and you can do this with GPU. Uh when you would do this as an example with CPU is if you are doing uh some data normalization or data sampling. So, if you have just a metric ton of observability data and you want to like clean that out so you're only

storing what you need so you're not paying tons and tons of money for for storage on data you don't need and provides no value, you would use a batch job of some kind to kind of to to do that processing because once that similar task is already running, then the CPU can be more efficient at running it. So, uh this is the same with GPU. You can

also use batching to maintain costs because again it runs better when you have similar workloads. So, this is another like it's actually also a cost management thing as well. So, while in in LLM uniquely, in AI uniquely, it provides a better user experience when you're batching. And there are other places I guess you could say where it does well on on other types of systems, but it's

it's like a lot of >> one. It's like the common one. Um it's um it is really actually a very good for just overall cost management technique. So, you group like things to do the same thing to them and then that's just going to run better. So, you can also do uh batch or offline work, right? Um it's not just about like that live interactive thing, but

if you're if you're doing some sort of like training aspect um or or um fine-tuning aspect of all of this and you would reinforcement learning. >> Uh you would be doing this through uh batch processing um offline kind of while the system would otherwise be at rest. So, it's a really neat like also thing to just kind of schedule in because you're going to have your peaks

and values of um peaks and valleys of workload usage and then like traffic. So, when you're going to be in a valley because it's, you know, midnight and nobody is using your system, then this batch um then there's another opportunity for an additional batch processing. So, kind of what Chris was saying earlier of like bringing the same things together and and and providing a better like throughput

experience from the user on that, you also will be doing that in like the more traditional sense as well. Yeah. And when we look at that from like a traditional chatbot uh that's where we would look at more of that end-to-end latency. So, how long did it take from the end end-to-end? And typically it would be from the time of initial prompt to when the decode process

has has officially ended. Um that's another critical SLO that we see often represented. Um this is something that SREs are interested in, especially if you're managing like a um customer service chatbot, for example. >> Yeah. If that number is increasingly going up, that means something it's it's an indicator there may be something wrong with the system as a whole. Uh so, these are the types of SLOs.

Um now, I'm going to shake things up a little bit. This now gets two two to three times more complex if if not more when we start talking about Aagentic. Uh cuz now we're actually looking at things even more from um like a an east-west communication standpoint, where the telemetry even becomes more important. Uh this is something where I'm seeing articles kind of daily coming out from

like the the leading experts in this area, where we're we're still not entirely sure, the dust hasn't settled. Uh but just know that I think the next phase of this will be defining what these look like then within the Aagentic systems themselves. So, that will probably be next CubeCon talk. Oh, you know okay, so well you know I have opinions here. So, I actually just wrote a

book on AI ops and we do talk about this a little bit. So, things like guardrails executions, right? If you have guardrails or using something like an AI gateway, um and so guardrails execution becomes a really important uh SLI for Aagentic. Um you also can have um I like to use natural language process scoring. I have a whole thing on this. Uh, so natural language process scoring

is where you evaluate the correctness of the output of your LLM. So, if you have something that is running agentic and you're you you are you are just reinforcing that you are saving those inputs and outputs and then grading them, right? It that one gets really fun because it's a mixture of objective and subjective data to kind of define what the correct benchmark is. But, these are

the areas of practice that like I'm personally exploring that I'm personally evaluating to make sure that Yeah, we want to use agentic. We want to make humans more efficient, but we also need to be able to trust the thing that we are building. So, things like guardrails, middleware, things AI gateways, um, I are very exciting. Um, and yeah, I I have talk to me about anytime. I

really think it's fun. And this is just an interesting new SLO, um, cuz this is actually something that gets into the non-deterministic nature of these models. Um, so, as we just start defining these SLOs and how they relate to SLAs within our organization, especially when we start talking now to security teams, who are very interested in this, you know, AI teams, application teams. And now it becomes

a cross-domain problem where we actually have to start looking at this from, um, a very, uh, holistic type of way across the entire domain, not just the single point of hosting models. Um, so, our stakeholders now become a lot larger when we start moving into this kind of agentic paradigm that we're seeing. All right. So, Hillary, uh, told me I need to I need to put this

up. >> I did. I made him put this in here. >> Um, so, we wanted to give you an example of what architecture would look like. Uh, so, we've talked about this, but like, "Well, what what's the example of what what this this all looks like together?" Um, this is a system that I built. Uh, I I I hate admitting this. I, uh, for marketing purposes, I

have a number of bots who are going through and playing um The Legend of Zelda: The Oracle of Seasons on the Game Boy. I actually just talked about this at GDC, Game Developers Conference, where um I was just basically showing how you can use this architecture to use reinforcement learning. Um so here we have a number of nodes that are orchestrating through Ray um all the different

Game Boy instances. Uh so it's using a mix of reinforcement learning and LLMs uh to go through and play the game. I can watch them play the game through an external website, which is making it which is available through um the gateway and it's web socket streamed. And then we have a number of LLMs that are being load balanced by LLMD. Uh this wouldn't be possible without

because there's a lot of um overlap and a lot of very similar data. Since I'm I'm running these hundreds of Game Boy instances at once, there's a lot of commonality between the data. Um so what happens here with LLMD, it allows me then to um go to the correct VLLM instance that has the most optimal cache. And there's tens of thousands of data points getting fed into

this real time. Uh so speed is very very critical from a SLI SLO standpoint, I'm tracking all the metrics of how fast the the how what is the rate of my KV cache, cuz that's very important for this experiment that I was running for GDC. And then um all those different things that we were talking about like time to first token, um I needed to optimize because

as as I said uh time was very critical here. So this is um obviously this is being used for video games, but the exact same algorithms I'm using here, the exact same principles are used in robotics, medicine, uh automotive, manufacturing, IoT IoT. um this gives you an idea of, you know, this this may actually look very similar. This looks very similar to your application stack right now.

Um but we're just looking at it from a new paradigm, where now data is flowing into these stateful instances down here that are being run by VLLM and are being powered by AI accelerators. And I remember correctly, you had some like hotspots, right? During your uh like Yes. >> bottlenecks playing through, and you had to do something to address that, right? >> Yep. Yeah, so there was

a number of times where um there was a lot of critical failures that were happening within the algorithm. And some of that data wasn't completely obvious to me um through just the Ray console. And then actually how I figured it out was the time to first token. I saw time to first token shooting up really, really high, and it told me something was wrong in the way

that the LLM was processing some of the the reinforcement feedback. So, this isn't just for your production use cases. This could also be for the experimentation that's happening within your organization with, you know, a variety of data scientists or their teams that are doing R&D. Um your stakeholders are still going to be very interested in this type of data as they're doing this type of work. As

that data could be indicative of something more serious. Okay, now you can move on to the next slide. >> Okay. So, we wanted to make this two ways. Uh so, we are going to open up the floor to a Q&A. Uh we we have a very deep background in this. Um Hillary from an SRE standpoint, and me more from traditional engineering, community, architect, so forth. So, we

do want to open up the floor now to questions, uh especially as we start talking about some of these new SLOs, SLIs, how do those relate back to SLAs and different uh business objectives that may come out of those. Um so, if anyone wants to come, I do believe we have a Q&A mic right up here. >> Uh so, if you want to come up, we are

going to give some time for questions. Or don't, and we all go home early. That's also acceptable. We saw that we were the Thursday afternoon and we're like, "Oh." There's actually more of you here than I thought. I was ready to speak to an empty room and and be like, "Cool, free vacation. Awesome." What was that? It was. Yeah, how I I was speedrunning Zelda. So, I

I don't just do this for marketing purposes on my local machine. I'll do different games for speedrunning purposes. So, I want to see how fast I can get the AI to beat the game. Um I did this with Double Dragon a couple years ago, and I shared it with a speedrunner who was really impressed because one of the techniques that the AI figured out was something that

was commonly used in that that um community speed runs. So, um the issue with Zelda is that it gets a little bit more complicated. There's a lot more puzzles, backtracking, and things. It's uh and it takes more of an LLM to kind of kind of figure out. So, um I do have some more material if if you just search my name, um there is a blog that

goes with this that gives a little bit more information. It gets a little more technical. Um we wanted to keep this kind of high-level, but if you are looking for some more technical overview, and then, why don't you talk about your book real quick? Oh, right. Yeah, so already kind of mentioned um I just wrote uh a book with a couple of colleagues uh Andreas Grabner. People

may know him. He's a CNCF ambassador. He also works for Dynatrace. Um my other friend is uh Rob uh Rob Reichert and uh, he's formerly Red Hat. I forget where he is now. Also, um, reliability engineer, director of engineering, etc. He's done cool stuff. Uh, so we wrote a book on AIOps. Um, I would call it a very pragmatic approach to, um, using AI for IT operations.

And one of the things we talk about in there quite a bit is also that how you're using AI for observability, um, you need to then turn around and apply those same techniques to the AI that you are running. Um, this is again where I I get into my my little spiel about natural language process scoring, which is really fun. Um, there's tools like DP Val that

can allow you to do that with unit testing. and it uses an SLM to grade the LLM, which is really cool because it's SLMs are really good at. Wait, wait, again, again, again. Show of hands, again, who has used the SLMs? So, for those of you who have not used an SLM, SLM has a lot fewer data parameters. Whatever it is trained on, it's real good at.

Whatever it is not trained on, it's real bad at. Um, some SLMs, uh, like, uh, the Gemma one, actually can run on CPU, which is good for me. My developer laptop has no GPU whatsoever. So, uh, thank you, Red Hat for that. I will be petitioning for a new one. we talk about how actually the power of of SLMs in that book. I do specifically. And this

is one place where Chris and I kind of disagree. And actually, there a maintainer summit on Sunday. There was a conversation about this as well. I firmly believe, because of the expense and overhead and cost of an LLM, that's not just the actual expense of hosting it and running it, but also of training it, because it needs more data, that we will start seeing more small language

models. Small language models can be very, uh, purpose-built and with a lot less data. Um, and then they're easier to recover from things when there are problems like catastrophic forgetting. Has anybody heard of catastrophic forgetting? Yeah, okay. Catastrophic forgetting, when you fine-tune your LLM and it suddenly doesn't know anything anymore. That is called catastrophic forgetting. Uh recovering from catastrophic forgetting is apparently possible. Cornell University did a

really cool paper on this, which I highly recommend. It's very very very dense. Uh in practical terms, if you actually incur encounter catastrophic forgetting, I would recommend just redeploying from either scratch if you don't have a checkpoint or better practice, make checkpoints during your fine-tuning and and start from that checkpoint. So, I am of a different opinion. I believe that we'll see more larger models, but larger

models that use mixture of experts and continue to advance in that area. And then we see some of the same kind of things best SLMs. That's not saying that I'm not an SLM believer. Um so, I do believe that SLMs have a place with things like um guardrails, for example. Maybe that makes sense. >> and evaluation. >> for things like rag. Um I'm a big part of

the semantic router community, and that's something where we use small embedding models. So, that makes more sense where you have like an edge component or a very small like single-purpose thing. But to me, the way I'm seeing things is going more of the route of mixture of experts. I'm I'm actually excited to see I'm excited to be proven wrong here. Um I think these are both fair

things. I think this is a coin flip based off of where the market is going. Um it makes sense. I come from more of an AI background, where Hillary comes more from an SRE. So, she's looking at things much from a big picture. Um um and I'm looking at it more from more of the data science perspective. So, it'll be cool to see what and it could

be that in a year it's it's neither. It's something else. >> It could be in a year. Okay, actually, there was a really cool thing. So, no more nerd stuff for you since you didn't have questions. Uh there was a recent uh kind of remapping of monkey brains done to help scientists understand vision. And that uh a new formulation of neural networks is more computationally efficient and

uses significantly less electricity. So, that may be where we go. This may all go by the wayside and this new mapping that they did is >> AI. >> This Bio AI thing, that may actually be where it goes. That's I can't remember what university did that. I read it the other day. I was like, that is creepy and cool. I'm very excited for it. So, maybe in

a year we'll be talking about