Lightning Talk: Cross-Region Model Serving: PyTorch Inference, Observability... Suraj Muraleedharan

This talk covers cross-region model serving and multi-region model inference from a customer perspective. The speaker discusses the challenges and requirements for implementing such systems, focusing on low latency, GPU capacity management, and observability. Key technologies mentioned include PyTorch serving frameworks like TorchServe, Triton, vLLM, and KServe, which cater to different model types and deployment needs. The talk highlights the importance of maintaining visibility into model performance, handling correlated failures, and ensuring quality during deployments. Additionally, the speaker emphasizes the need for observability tools and practices to catch issues early and improve performance metrics across regional models.

Okay, hello everyone. I'm Suraj. I'm part of AWS team and I'm an engineer with the field and the customer engineering side of the business. So today we'll be talking about the cross region model serving multi-region model inference and how it is going to be looking at from a customer standpoint for anyone who is going to be implementing it on the PyTorch inference observability and LLM ops side

of it. Quick show of hands, anyone who is running models across regions, multi-region, anyone that you have tried? Okay, few of few of you. Anyone who is running single region? Okay, lot of lot of folks. Okay, so what we will be covering today is basically going to be a use case which is required for running the multi-region cross region model inferencing. We'll look at the different categories

which are coming up in terms of the problem statement, the LLM ops that is needed, the dashboards, the SRE and we'll also look at how you can improve the observability side of it. So single region more or less a solid engineering problem. We know that it's going to be optimization, we know it's going to be scaling that is needed. But when it comes to multi-region, there are

different reasons why you would have customers and teams who are doing multi-region. Some of them can be because of regulatory requirements where you want to give it to the regulator in a specific industry that you can move between different regions. Others can be because of low latency and when it comes to low latency, what sub-second latency, sub-millisecond latency that you need with your end users. And the

last thing can be because of GPU capacity. We all know GPUs are scarce, we wouldn't really get all the GPUs in a specific region, so you want to have the flexibility of running at different regions. Now let's look at a use case where you have a bunch of non-functional requirements like 10K plus requests per second per region that is being given. We want to run it across

three different regions. We have the first token time to be P 99 to be less than 200 milliseconds and we need to be able to talk about scale and availability and reliability. That means it has to be able to absorb a lot of spike that is coming up. So what are the different problems that you'll be seeing when you're running these kind of multi-region model inferencing? The

first is correlated failures. You can suddenly have a lot of regional outages, you can have a lot of load coming in, so you need to understand where your load is failing and how you can split this load across different regions. It's quite possible you would have a thundering herd problem. You can end up solving it with say a power of two or you can have different load

balances being put in, but then you should have a clear visibility on where the problem is happening. In terms of traditional systems where you would look at 200 and you would look at 201 and you say that okay, the system is healthy. LLMs don't really work in that way. LLMs are quite possible that they would give you fast responses, but they would fall fail silently where semantically

they would be wrong. They would be giving hallucinations, they would be giving incorrect results and your users may be getting impacted, but you wouldn't really have the visibility on that. And finally, there is a lot of change that has happened in the inference space, especially when the models were in the pre-LLM era and now in the post-LLM era because there are a lot of gaps in something

like a serving infrastructure like TorchServe where you don't really have KV cache, you don't have the page attention, you don't have the tensor parallelism and everything. And in addition, you have the maintenance mode that has come up with TorchServe. So how do we address this? So before addressing this, let's do a quick recap on how did we end up reaching this stage. So when it comes to

PyTorch serving, initially everyone was very happy. You don't really need the LLMs, you were happy in just hosting singular models and you mostly used TorchServe or TorchScript and Triton for the initial phase of it. And once the LLMs came into the existence and everyone started using LLMs, the whole requirements changed. You suddenly had to consider page attention, you had to consider the KV caching, you had to

think about parallelism, serving and all of those things. Now we are at a stage where the serving has split into two different categories. The first is model optimization problem which becomes with torch.compile and the CUDA graphs which is basically your first stage of the serving. And second is an orchestration problem because now the orchestration has to think about something like KServe and how you would be running

it in a compute platform like Kubernetes which is the de facto platform for most of the things that you're doing. So now we then whenever you have regulated customers or anyone who is moving from say TorchServe to a different infrastructure, you have a mixed fleet and you have to start looking at serving based on the requirements that you're having. So if you're having LLM workloads, you would

look at vLLM because it's able to give you batched and it is able to give you page attention and everything. Triton [snorts] is another example which you can use because for non-LLM models, these would be your recommendation systems, your NLPs and everything. If your organization is heavy on hugging face, you can definitely use the text reader inference TGIs. That would be much more appropriate because it's very

nicely linked with your infrastructure if you're using that. And finally, you can end up using KServe which would be your orchestration layer which is wrapping all of these different backends for you. The key thing to identify here is the mixed fleet because not one size fits everyone. It keeps changing depending on the kind of models that you have and what you're providing to your end users. So

now let's look at a north-south flow of a request and we are talking about really low latency because it's quite critical that when the routing layer that you're having is very thin and it is very cheap because the GPUs are quite costly. So you should be able to immediately do the routing. And that's why the first layer of the routing is generally with a global accelerator or

something like a anycast IP because that is able to give you really low latency and I would keep it very simple with say Envoy L7 latency based routing and the circuit breakers which are there. You also have other frameworks like LLMD which can be utilized here and these are different maturity graphs depending on where you are with your own systems. So you can potentially use it like

in-house solutioning with just an edge routing or if you have the bandwidth and you have the engineering team, you can definitely end up making it even more mature and scaling it further. And the second layer that you would be having is the expensive layer which is the inference engine because you basically have the Tritons, the continuous batching, page attention, tensor parallelism and all of those things because

that's where your GPU and your all the inference layer is sitting. But your flow is not yet complete because any request that you're sending, it needs to have some kind of a safety layer which is the post-processing that you would be having. And that safety layer is basically the guardrails because there are two levels of guardrails that you need. Any request that is coming from the inference

standpoint of your end user who is sending the prompt, you need to look at the prompt, make sure that it is appropriate and it is having the right language and all of those things. And you also need to look at the guardrails which is the output coming from your LLMs because you need to ensure that there is no hallucination, there is no any hate, insult or any

other inappropriate content that is going in. PII reduction and that's another area where the judge models becomes quite important. So cross region health aware failover, low latency configurations and private link. If you're on AWS, you can definitely use AWS backbone network for doing that. And then you key insight that you have is keeping the routing really cheap, failover and you need to have a warm pool which

is ready for failover for any N-1 or N-2 deployment which you might have done in the in the past. This the high-level architecture of kind of a network topology that you have where you are going with the internet coming on to the global accelerator. We have a control plane which is going to be controlling all the regional NLBs and the network load balances that you have followed

by the Envoy fleet and the inference pods. Kind of the same architecture which I was talking about earlier. Now when it comes to the first pillar of your inferencing, we have established that the core backends that you'll be using. The second important thing is how you're doing the deployments. You have to look at deployments kind of in the same way that you look at deployments happening in

a distributed system because you would be having the shadows, then you would be having canaries followed by the regional and then finally the global wave rollout. When it comes to the rollouts that you're having, it is quite important to have a bake time. And the bake time, the whole point is you are knowing the failures before it actually ends up hitting your major users. So you're going

to leave that system running, start hitting it with your tests, your integration suite and everything else so that during that bake time you're able to catch any issues which are happening later on. And this is the kind of shift left approach in the DevOps terminology if you want to put it, but it's more on the inference side also. You're using the canaries for your quality validation. Blue/green

is not really required until the time you have a specific requirement for it because it's very costly. It's two times the two times the infrastructure, two times the GPU cost and we all know the GPUs are scarce, so it is always interesting to go with the canary rollouts that you would be needing. These are the three areas which you would be always considering which would be the

circuit breakers, the failover and the rollback and finally the deployment strategies which are quite critical for any of the work that you're having. And these fundamentals do not change irrespective of the kind of deployments that you will be having. Now let's have a look at the observability flow. With the observability, it's pretty standard workflow that you'll be having and we are using open source tooling which would

be hotel, your collectors, Kafka and Flink for your retention and your query anomaly detection. But the key thing is the per request trace spans. Now, this doesn't come out of the box. If you're able to give use hotel and you're able to split up your inference request into different stages, you will have a very clear visibility of your routing queue, pre-fill, decode, and safety. And the primary

importance here is your pre-fill is very much it is depending on the compute side of it, whereas your decode is more memory intensive. So, it gives you an insight of where your inference is falling back or where what kind of GPU extras that you need to put in, how you need to scale out, how you need to disaggregate. All of those things become quite important here. And

when it comes to observability, it's quite important you build it from day one because you would be flying in blind without the observability for any kind of issues that you'll be having in the production. And that's where the signals become Uh these signals are quite similar to what all of you would have heard about and would be aware of. Uh the key thing that I would call

out is the time to the first token, the TTFT, and the inter-token latency on how smooth the streaming is. Uh it's color-coded in terms of user facing cost efficiency, system, and reliability side of it. These signals, as you're looking at the dashboards, as you're looking at the SRE, they become very much important for any of the end users. And if I try to map it on how

the anatomy of the request goes in, the queue weight depth and how much of the queue depth is going to give you signals on the back pressure signal that you have on the TTFT. Depending on the cache hit that you're having, you would see how much of a response time it is coming up. And finally, you have the bad size and the GPU memory bandwidth that is

coming in. Finally, the dashboards. Uh the dashboards are going to be more or less similar. those dashboards would be three different levels. The first would be your on-call dashboards. For anyone who is on the SRE side, they should be having the region health, the error rates, the GPU utilization metrics. Then you have the ML engineering, which comes to composite quality score, uh per dimension breakdown, and all

the details which are needed for your ML engineering team, any kind of KL uh divergence and the baseline and the quality side of it would come there. And finally, your cost and efficiency. End of the day, your inference is going to be a line item in your executive teams and your uh senior leadership on what is the cost that is going on in the inference. So, you

need to have a dashboard which is able to talk about that as well. And the last pillar which I want to touch upon is the quality and the governance. We know how the inference is happening, we know how the observability is coming up. Now, we need to know what to do to ensure that any gap which is there is closed out. And that's where the quality and

the governance comes into picture because once you have the drift detection, quality evaluation, and the gated rollback, which I had spoken about earlier, where every where you are having big time and you're doing a rollback in terms of anything bad going in, definition of bad would depend on the team and the metrics that you're tracking, the quality and the governance becomes quite important in order to understand

the success criteria of your inference payload. A quick look at the open-source tooling that um most of our customers and the end users are using. For inference, it happens to be vLLM, Triton, um Torch Compile. I would say the CI/CD workflows is mostly with Argo CD and MLflow. Uh testing is quite important. AI performance benchmarking and Locust are quite important. And finally, the observability piece, uh which

becomes critical. So, few takeaways on closing out. First is migrating from TorchServe, vLLM for the LLMs, Triton for everything else. Depending on your team, you can pick and choose what would be the right fit for you. Finally, splitting the serving into two different stages. One is the model optimization, second is the serving orchestration. Um KServe works well for most of the cases, but then if you want

to further uh tune it further and maybe do some offloading to LLMD or something, it can also be a good design. Observability from day one. Uh custom auto spans is quite recommended if you have the engineering depth and you have the um folks who can actually manage that kind of an infrastructure. It's quite good out of the box. You don't really get all of those granular details.

Uh finally, the quality evaluation, which is not op- not optional. You need to make sure that the LLMs are not failing silently. And the progressive deployments, which it can give you really good insight into all the um canaries and all the automated deployments which is there. So, yeah, that's everything from me. Uh I'll be hanging around for any questions which you have. Um thank you so much

for your time. Yeah. I've got one question around the signals and metrics you're using for life cycle management. So, looking from a distributed perspective, if you've you've got all the building blocks there, but if I'm transitioning from one model to the another on a global basis, what signals are the best ones for me to monitor to make sure that I'm actually hitting the right mark in changing

my model from what it is today to what it is tomorrow without impacting the way that that currently things are working? So, with most of the solutions which we have seen uh with the customers who are moving the making the model transition, it comes down to the F1 score and the specific model accuracy requirements that you are having. And then when you would be doing a benchmarking

for any of your TTFT and the ITL related side of it. That is that kind of becomes your non-functional requirements for most of the end users which you would be having. And then the functional requirements which would be dependent on the model specific part of it. So, you would be going through your model MLflow life cycle and the approvals and everything which is going to help you

with that.