Serverless GPUs in Production: How Cerebrium Built a Globally... Dave Protasowski & Elijah Roussos

This talk focuses on the use of serverless GPUs in production, specifically through Cereum's implementation of K Native to streamline deployment and scaling of GPU-based applications. The speaker highlights the challenges associated with GPU workloads, such as their non-sharable nature, high costs, and latency issues when starting complex AI models. They delve into how K Native simplifies the Kubernetes experience for application developers, allowing them to focus on deploying containers without worrying about underlying infrastructure complexities. The speaker explains the enhancements made to K Native to address the unique requirements of GPU workloads, such as improving routing latency and managing application readiness states. The presentation underscores the integration of K Native with various AI models and applications, emphasizing its importance in enhancing performance and efficiency in serverless environments.

Uh yeah, let's get started. Welcome to the talk. This is uh gonna be talk about uh serless GPUs in production. So this is how um Cerebrim, that's Ellie over here or Eli. >> Eli, >> good thing I asked um uh we're going to talk about how Cereum uses K Native to deliver serless GPUs on their platform. And quick intro about us. >> Uh yeah, so I'm Eli

or Elijah. I'm the founding engineer at uh Cerebrum. I'll explain more about what we do later, but check out Cereum. Uh check me out if you want. Uh yeah, that was awesome to you. >> Hey, my name is Dave. I'm a K Native maintainer. Um I'll get into K Native. Uh a few roles in the project. I'm the serving lead, which is sort of like the serverless

runtime that's there. I'm also on the steering committee. That's the SC here. Um, these are the handles. I still use the Twitter logo because that's what I like, but I don't really use Twitter. Uh, just you can message me on LinkedIn if you have questions for things. And then I'm going to do a quick uh what is K native run through, but um I have a link

to a demo that goes into more and you can visit our website kative.dev for everything else. Um so the big trouble with Kubernetes and I guess to clarify what Kative is it kind of targets the app developer persona and when you present Kubernetes to app developers it's super overwhelming because it con essentially like the podsp spec when you want to deploy an app and deployment models like

app operator app developer concerns with platform engineer concerns about like topology node scheduling and all these things and all an app developer really cares about is um here's my container run it I don't really care how it works. So we try to um simplify Kubernetes for app developers in three ways. We have simplifying development with K native functions, simplifying running your apps in K serving and connecting

your apps with K native eventing. This is kind of like the life cycle. I'm not going to go into too detail because I'll add a link to a talk where I did in Atlanta that has like details on this and like a demo for each thing. Uh but a quick thing functions lets you bootstrap functions and this is like a local tool that helps you build containers.

Uh we bootstrap in like an HP server and we can handle HP events and cloud events. Cloud events is sort of the CNCF deacto standard for uh payload messaging that has special headers to let you do routing and filtering and you don't need docker to do the container building. Uh serving is sort of like this we present like a higher level abstraction. So instead of the app

developer being concerned with like 15 resources they need to create to deploy an app and do all this networking and search managing and horizontal pod autoscaling we present like a K native service and you can see here it's like hey just give me run WordPress and then we come back with like a URL that has a certificate that was provisioned and then this if you hit this

URL it will scale up from zero scale down to zero we do request change tracking so we stamp out revisions as you make edits to the spec. And then from there you can do traffic rollouts and things right inside the service spec. So then you don't need your app devs to actually go down and know about like how does contour uh resources work, how does work or

in theory don't even know how the gateway API works. And then for eventing it's kind of similar. We present this broker abstraction. So you have all these sources that publish to a URL called the broker. And then for an app developer all you do is say hey I'm attaching a trigger to a broker. I care about this event type with this uh property. So let's say like

I want GitHub events for my repo and I just want pull request events. Then the broker would do all that sort of filtering and I'm going to hand it off to Ellie or >> Eli. >> Eli, I'm sorry. I just met just met him two minutes ago about um how uses K native for their serverless GPU stuff. >> Cool guys. So, we're going to be focusing mostly

on the serving component of K native when I talk about these things. Um, but I'm going to start at the beginning with why AI workloads are typically harder to do than a traditional CPU workload. So, the first thing is that they're non-sharable. Typically, there are exceptions to this, but typically you need full access to the resource. So, you need the whole GPU. you typically need the whole

GPU memory. Uh there aren't very good tools at the moment to segregate the stuff. It is being worked on, but if you're going to be running this stuff, you're typically going to need access in a singular fashion to this GPU. They're also very expensive. If you guys have run any GPUs in AWS or GCP, you'll know that instead of point or like$1 cent dollar an hour, you're

paying $1.4 4 $2.8 an hour for access to this compute. And if you're leaving at idle, you're literally burning money. There's slow. So if you have an AI model that's 8 gigs, you've got a container that's got PyTorch, you've got a container that's got CUDA, these things can amount to 15 gigs that you need to start in a matter of seconds. So this can take depending on

your network connection, minutes. Uh and the last thing is that you have very little room for error if you misroot stuff. So if to something that cannot handle the load that you wanted to, you're not going to it's not going to result in like CPU throttling. It's just not going to work. So you need to be hyper accurate about where you route your workload to. So I

as like a small illustration of this like we have a customer that is running a real-time avatar. So they literally generate an avatar that speaks to people. Um they do lip sync all that sort of thing. They have to run multiple models to achieve this. They're doing text to speech. They're doing speech to text. They're doing LLMs. They're doing video generation. All these components need to have

performance criteria that matches but they have different requirements in terms of scaling in terms of resource needs but the core tenant surrounding most of these workloads is very similar. They're big. They need to start quickly. If for instance that doesn't if uh if an avatar is responding to you in seconds versus milliseconds it's not going to be a very good product experience. So this is why it's

important to get this right. So this leads to an inevitable nightmare to deal with these things. You can't think about this in the same way as I was mentioning before. If you overprovision, typically you need to because otherwise you can't you can't scale over. There's no throttling and no sharing. You're going to burn money. You're also going to induce more cold starts if you don't overprovision. You're

going to drop requests. All of these things assume that you want a need for concurrency across your application. You want to be able to handle gradual traffic and you need you want a tolerance, right? So the other thing that's hard about GPU applications is the app development is kind of disjointed from model development and that's from actually developing the thing to deploying the thing. So your app

is going to deploy typically on a container. Your model is going to live in some sort of storage elsewhere that's more optimized for sequential read. U you're going to develop and test these in different ways. Your model's going to have a different test cycle. It's going to have its own optimizations that's not really related to the application. Um these things are different. And then the last thing

is that in in line with the model being a different component, typically to get the results you want, you're going to need to use a bigger model. Uh I think most people here would attest that if you ever used like smaller models, they can lose uh accuracy very quickly. Um and that's just a trade-off that you need to make. So this is the core of what we

do at Cerebrum. We're a serverless infrastructure platform. We make it easy for you to deploy uh and scale GPU based applications. So that's most AI workloads that exist today. Um, we care a lot about uh where your data goes, helping you bring your price down and helping you get performance that you want without overprovisioning these things yourself, benefiting from the shared nature of everyone on the So

there is a when we uh when we think about this, we need to orchestrate all this compute. We need to figure out how to routt these things where they need to go. And a natural place to start is Kubernetes. I have a bunch of workloads. I have a bunch of nodes. They have certain requirements. Um, Kubernetes is a natural basis to start this off with. Was made

to do this. Um, it's mature. There's a large ecosystem with many applications written off of it, including K native. Um, and it's flexible mostly uh for adapting it to how we want it to So, K native comes in here. K Native brings us a lot of functionality that vanilla Kubernetes just doesn't have by default. Uh K native's autoscaling mechanisms allow us to scale on things like requests

per second and concurrency. It enables scale to zero by default. It handles stuff like rollouts, making sure you have revisions of multiple variations of your application so that you can roll back easily. Um it helps you define automatically your traffic orchestration. So it will deploy uh it will deploy an envoy instance for you for ingress. It will buffer requests for you and it will do the load

balancing and distribution of these requests to where they need to go. And finally, um, I talk a bit more about this in the blog I just released, but, uh, the synchronous nature of K native, uh, makes things a lot quicker than what typically people would turn to when they're dealing with these, uh, GPU based applications. I think a lot of people would turn towards Celery with a

combination with Reddus in like a async Q system. um pole is not that fast. So this gives us an edge then rather than using systems like kada K native has some drawbacks that are not specifically related to K native but they're related to how K native came about and what it was originally designed for. Originally, K native was designed for CPU services, lightweight as kind of a

replacement for Lambda and Cloud Run, right? Um, it assumes that your targets can handle multiple connections. It assumes some higher um for where connections can go. Um, and this in our system is a bit broken, right? So here are a couple of things that are just kind of not applicable in this scenario. Binary readiness is hard with a lot of GPU applications. They can drift in and

out of readiness very easily. Whether or not it's they've got to reload models. Uh a Python event loop gets stuck because it's doing that. Um this is a state that we kind of have to deal with this intermediary. uh as I've mentioned before uh because it's singly concurrent most of the time if you make a mistake during routing it is far more costly uh you will cue

instead of just throttle on AWS uh specifically uh and I'll get to this in more detail uh the service discovery that Kubernetes by default uh supplies size is glacial. Uh, and that adds to the already long cold start problem that we have to deal with when loading huge container images. I will skip these last two because we're not really going to talk about that. Uh, I just

want to give a quick little primer on the internals for K native. So, this is a very simplified view. Um but you get ingress traffic that goes to a resource called the activator. That's the thing that will buffer and load balance. Uh that will get sent through to the pod goes through the qroxy and then ends up in the user container. All the uh all the pod

information about livveness comes from the controller via an informer. So how do we start to fix these things? We begin with custom discovery. We need to try and fortify the readiness state and make it a bit more granular and less coarse. So specifically in our case, and this is probably a problem with EKS, not necessarily with Kubernetes, but it's hard to move a control plane in production,

we were experiencing informer delays of upwards of 10 seconds, sometimes even minutes, which means that health information from pods was just out of date. And in our case, that's really bad because it's not a case of, oh, we're going to send it to something and it's going to like quietly fail. It's it's just going to halt all application calls period. Uh we also have that drift. So

pods becoming ready, not ready, drifting in and out, whether that's like releasing GPU memory, they're blocking event loops. This is a common case that we see uh that we can't in some ways we can't really trust what the readiness is saying sometimes. So the what we did here is we've we modified qroxy to actually propagate that state directly. So we've skipped all Kubernetes machinery here entirely. Uh

and as a result propagation is almost instantaneous. But uh this introduces another problem. Uh I'll actually go to that first which is how do we make this it's back. How do we make this concurrently stable? So now instead of one informer we have all their all the pods reporting to the same place. They need to communicate and achieve consensus. So another change that we did within the

scope of the activator is we made everything a single state manager very much like uh actor models. You'll see this a lot in the beam for example where you serialize everything over a single mailbox. Um that means that we have strong state guarantees for what the state of a target actually is. Right. and even beyond that catering for various failure modes that happen instantly. Um mostly TCP

failure connections. Uh you we've introduced quarantine states where we we put targets into quarantine. They've given a chance to recover. If they'd never recover, we delete them. Um also got having to do with the informer sending us wrong IP addresses because it's so stale. uh we can check that as well and instead of failing we can try a reroute almost immediately. The last thing that we need

to consider t in k native core uh to achieve high availability uh sharding takes place. So if you have two activators, the one activator deals with half the targets, the other activator deals with the other half of the targets stateless. But it means that if one in our case, if one act activator gets saturated, which could happen very easily when you're talking about request time of minutes,

you're going to encounter cues if you happen to send to that activator. So we want every activator to have a global view of where it can possibly forward requests. Uh and here uh we've moved red estate into Valky. So Valky has its own high availability. The activators have their own high availability and they can all reference every other target which is great for us which means it

means that we don't have to be cued unnecessarily. So this is kind of more what it looks like. Now again this is pretty high level and there would be multiple activators in this case but instead of having just the service informer the Qroxy will relay its state it itself the pod will relay it state itself to the load balancer the activator and routting information including load balancing

decisions will happen at Vuly. So we have lure scripts that will actually do the load balancing, send that off to the activator and the activator will proxy. Um what does this mean in actuality? It means that our routing latency at P90 level is really fast. Uh it's around 25 milliseconds for inference traffic. We could do tens of thousands of pods pretty quickly. uh we have less 502s

induced from failed rooting states. So I've hit something that's not ready and it immediately 502s as opposed to rooting mostly gone. Uh and we managed with all our changes to make sure that we can horizontally scale. Uh you don't want to crash, right? Uh if you guys scan that, that's a link to like a more detailed blog post around this stuff if you want to take a

look. here's some other things that we did. It's not the core scope of this talk but I thought it would be uh nice to mention is uh we've reworked some sca uh scale down logic. So in K native core you're bound by what is called down the cooldown period to scale down. So if you set your cool down to an hour for instance then it might take

a few hours to for the scaling criteria to evaluate to to zero. Um so we've done a in place evaluation over the cool down period instead. Um we've introduced a scaling buffer. It's just keeping X instances warm very good for concurrency one workloads. Uh user configurable load balancing which I think we're trying to get into core. Uh and there is a dashboard which we also try and

work our way into to core. Those are just a sides, not really the main point of the talk. But uh there isn't just this, right? Uh this is one part of a very large puzzle. and there are still things that you would need to consider in a huge system like this that isn't obvious when you're dealing with uh when you're coming from a CPU background, right? GPU

aware routting stuff like KV caching. Um if you guys don't know what that is, it's uh transformers specifically. They um they compute tokens. They store that in in a key value store. You want to do routting based on which targets have those values in the key value store. um doing continuous batching that's agnostic across applications would be awesome. You need accelerated containers as well. As I mentioned

before, container sizes are huge. So you also need to manage image pool time. This is something that is a bit more experimental that we eventually want to do, but there's the virtualization layer. uh machines take quite a while to boot. If we could boot it in a few seconds, that would be great. And then I because capacity is so stra constrained right now, just for context, we

run quite uh what is it six clusters roughly at the moment. Uh they all have capacity constraints. So at a global level, you want a root rooting layer that's aware of which capacity is where, how much it costs, what the latency is going to be. Um there's so much to do in the space uh that we've just scratched the surface on. Um that's I think Dave can

take it from here. I just want to I think the last aside is check out K Native. It's a really great project. It's a great platform. It's a great basis to do this sort of work on. Um Dave and his team have done incredible work. So I would uh >> please check them out. >> Cool. Thanks. I'm just going to do some quick project updates. It'll take

like two seconds. Um and again, I'll link to um a prior talk I did that covers like a demo and more things in detail. Also, you can swing by the Kative booth in the mornings. Um, and I can run through the spiel. Uh, so some updates in Kative serving. That's the serverless aspect. Um, we're kind of just always keeping up with the latest Kubernetes version. So, we

raised our min version to 1.33. Um, and some interesting features that came is um, this pod is always scheduable flag. What that allows is like we have people like IBM running code engine. they essentially have infinite capacity, keep adding workloads and cluster nodes and stuff like that. So they're never want to reject any requests and they will actually like provision nodes to uh fulfill those. So there's

um some tweak there. Um and then some of the other stuff is like we now support termination grace period on sidecars that kind of cleans up some graceful um shutdown in retrospect. And now before revisions were immutable but there's been a lot of feature requests to be able to kind of adjust the scaling factors on that. So those are just annotations now you can change. So if

you do roll out a revision need to go back and change like the min scale to two then it no longer needs a full roll out of revision to do that. Um and then in the future uh we're going to try to switch to endpoint slices. Um that kind of one secret thing people don't know about. If you just use regular endpoints and that's how we do

the activator um getting into the request path by adjusting endpoints for the K native services um you're limited to a thousand endpoints. So if you have more than a thousand you're kind of screwed. So by switching to endpoint slices we kind of get out of that restriction. Um the other bit too is I've done a lot of work in gateway API to land this listener set API.

So, if you're familiar with that, you can um send me a donation to my bank account. Um that will help us like shard um the resource. The gateway had a limitation where you could only have like 64 certificates to it uh listeners. Um, now it's sharted to a separate uh resource and now you can have an app developer bringing their own certificate versus it being controlled by

uh an app an operator um who controls the infrastructure. Uh for eventing the latest updates just include mainly uh maintenance and minor bug fixes. I would say maybe some big things is we have integrations with like Apache Camlets that gives you all these event sources from AWS. Um prior that was using like secret key and access key access but now we have like some uh end users

came like hey we that's not secure so let's use like service account off um with IM for that to work in the future what's landing in the next release um this is amazing actually an end user actually added uh NAT's jetream support for key native eventing u it's a big shout out to Andre for that um it's available in our nightly releases and if you need to

test this out and it would be great for people to test this out before it goes out the door in like 3 weeks. Uh join us on Slack and the Kventing channel and I can point you links to the nightly releases uh for clients and functions. Uh what just kind of came out is they're trying to really make I showed you the fun CLI that will help

bootstrap and deploy and um invoke functions. They're really, Red Hat is really pushing for that to be actuated by agents. So the agent can like build out a function, run the function deploy script, and then deploy it, let's say, to Kubernetes cluster or somewhere else. Um, and they also have um some handy tools to just produce the Git Ops uh YAML for GitHub actions and tecton manifest

to deploy your functions without having to because it's all kind of like turnkey. if you create like 15 different functions you want to deploy, you don't want to have like 15 different manifests that you manage by hand. And then there's also I guess there's this new skills MD that kind of dictates to agents and LMS what these tools can do. So funk now offers that. So that's

sort of like where functions is experimenting. Uh if you want to get engaged in the community, I would say Slack is the best place to kind of like ask some questions or you can open issues. We have a developer mailing list as well. We have a community calendar. Um and again visit our website kative.dev. That's where we have I would say like a lot of documentations and

intros on like um spin up a kind cluster deploy key native go through this walkthrough example so you can kind of play with serving inventing functions independently. And then I want to say a big thank you to contributors. Uh I think it's just me here today. And then end users like Ellie or Eli. >> We'll go with Ellie this time. >> Okay. Yeah. Ellie. Okay. And then

uh thanks for attending. This is a QR code for feedback. So please submit feedback. We have booth hours today that run till uh 1:30. So I'm going to go there after this if you have more questions. And then I'll be at the booth tomorrow from 10 to 12. I guess that's two. I kind of did like a short u bit.ly ke native-cmo. That's what I did in

Atlanta. that kind of covers running through functions, running through serving, showing the scaling, showing the traffic splitting uh and rollouts and then hooking up eventing. So the venting thing is interesting because you can use eventing for event transformation. So I showed a demo where hey someone submits a complaint form then it goes to an agent to turn this unstructured form to structured data. Then it goes transforms

to event to be handled that does a lookup of a user's info from a database, enriches the event and then eventually goes to another LLM to like do the processing to like produce an apology because it's a complaint form. So um go check out that demo and you can kind of see kind of like what do I would say the core of K native versus what Ellie

or Eli did for um like building out this massive platform on top of K Native. Uh and that's it. Any questions? I don't know if we're over on time or if you have room. >> Thanks for the talk. Um, I've been kind of obsessing with uh cold starts uh since last year and uh two days ago I g I gave a talk that helped reduce the image

pull time using bottle rocket. So I'm curious if if you experimented with this technology or what are you doing if you could share like to reduce the pool time because for me >> you're talking specifically about uh like docker container images. Yeah, exactly. Like I have 26 I work for pharma company. So we have 26 gigabyte images that are doing fancy things that we cannot make smaller

by any it's an Nvidia NIM so we cannot you know and uh yeah so I managed to reduce it by half but it's still like I'm just curious if there's anything else. >> So there exist a bunch of uh no problem there exist a bunch of technologies uh that are essentially lazy pool systems. So they will pull metadata immediately and then on demand they'll pull chunks of

the image that you need. So typically speaking even if the image is 15 gigs you're only going to be using realistically 30% of it. Um so technologies that sort of do this thing look at east there's a technology called Nidus as well. Um these will help you reduce your image pull time to essentially nothing. Um and it will just pull whatever it needs to on demand. >>

Also to add a lot of vendors are incorporating their own kind of thing. So um G has image streaming >> and that's that's kind of like based on what how each star GZ works. Image streaming polling which kind of does the same thing. So you just click a button then it just works that way. EKS also has similar technology and that's actually open source somewhere. I don't

know where. So if you're on EKS you can just do it. Essentially they tweak how containerd in the snapshot works where they do like a file mount system and then as they request certain files then it streams it in and saves it to the local file systems. That's generally how all that stuff works. >> Yeah. >> Thank you. >> The other thing to know it's like cold

start is like a huge problem even if you have the image on the node. It's like the limit the floor of limitation is Kubernetes, right? The one thing that the currently the cubelet does is it has a polling system to see when containers are ready. So even though you set the floor to like 1 second like you might miss a line from when that event loop happens

and you might take like a second and a half to show readiness, right? This is why like uh the Qroxy in K native will kind of like hold that request open and then aggressively pull um as a a workaround of that. But hopefully long term there is this work in SIG node that's doing um instead of it being polling it's an event based from the container runtime.

Um it's called like the pod event loop the plague or something like that. Um hopefully that will reach stable at some point and then I'm hoping we can then drop this aggressive probing and then there's other caps that want to do submillisecond aggressive probing because like you might only want aggressive probing during startup but not for readiness right as part of the start probe. So eventually there's

still work to like it's all the way down the stack right to do this work. Any other That was a good one. >> Cool. I guess we'll call it and then um