Cloud Native Theater | Cloud Native University: AI + Kubernetes: What Beginners... Michael Forrester

This talk covers the evolving landscape of Kubernetes and its adaptation for artificial intelligence (AI) workloads. The speaker highlights the significance of general models outperforming narrowly trained ones and emphasizes the necessity for foundational knowledge in Kubernetes. He discusses various features like Device Resource Allocation (DRA), in-place pod resizing, and gang scheduling that enhance Kubernetes for AI deployment. The session emphasizes a research-backed learning path for newcomers, advocating for focused adaptation on AI deployability using tools such as KServe and scheduling solutions like Q and Volcano. The speaker encourages engagement with community foundations and resources to effectively integrate AI into existing workflows while alleviating concerns about job security in the AI-driven landscape.

In a world where only AI, no, I'm just kidding. Wouldn't that be great if I had like a a trailer for this? Um, just want to mention that uh seems like maybe people are interested in this topic. There's a lot in the wings here. And I just want to say that um I'm going to cover a very narrow opinionated set of things because as you know there's

a broad ecosystem already of operator aligned AI that we could talk about. So let's get into it. All right, just a quick statement cuz sometimes we tell a story and sometimes we throw statistics at you so we can melt your brains before we get into the story. Um, so just to know is like there's a bunch of statistics that talk about how people are are not training

as much as they used to. I mean like why would you? I mean do you really need to? I and then I I sometimes get the argument where people are like but but we like we have special needs. Like we have special focus domains that we need to to like focus on like finance or health care or whatever. But just so you know, at least in the

latest research papers, generally trained models will outperform narrowly trained models easily, right? They'll do a better job in a narrow domain than a specifically trained model, right? Just as an example. The reason I share this is because people are not rolling their own as much as you think they would be, right? They're taking existing models and running them on existing platforms. I think we could argue that

Kubernetes is one of those existing platforms by far. just from a Kubernetes perspective, what is Kubernetes doing to make it like prepare it for AI workloads? Well, you saw the keynote this morning, Nvidia, right? Nvidia has been contributing to this space as a part of it. They've added features such as DRA, right? Which allows you basically to not just assign GPUs, but also to fractionate them and

add different characteristics to the GPUs, right? So, you can say I want this driver version, right? I want this much memory. I want this kind of characteristic in the GPU. So, if you were like a beginner and you're like, "Michael, how am I going to keep my job?" Which by the way is literally a question I got yesterday from both an 18-year-old and from a like who

is just starting and a 30-year veteran. It's interesting that both young and old are kind of like, "How am I going to either get or keep my job in 2026?" Well, before you panic, I just want to that this is not the first time something disruptive has come into the industry. Kubernetes itself was disruptive in 2015. So, learning how the existing systems are evolving to handle AI

workloads is absolutely solid. But, I'm just curious really quick, is there anybody who is in like can hear the sound of my voice who thinks that AI will either not be injected into your workflow or that you will not be deploying AI workloads in the next like 18 months. Does anybody think that they're not going to be touching AI in some form or fashion? Oh, yeah? Oh,

I need to talk to you, sir. If you cuz you're the only one. bottom line is is that DRA was added into Kubernetes along with other features like in-place pod resize, which you just if you were here 30 minutes ago, Masha and I were just talking about how to figure out where that came from and kept 1287, right? And by the way, why would like Michael, why

did you put this up here? This is not a AI specific feature. If you're going to run a distributed workload and you need 15 instances cuz you do you broken your data into 15 training set into 15 different components, and and those instances run out of memory, guess what happens to that set unless you're checkpointing, which by the way is a valid workaround. Guess what happens to

that job if you have to add more memory to the system, you have to redeploy it, right? Now, most of us have gotten around that with checkpointing, but bottom line is is that if you're a newbie who's trying to learn how Kubernetes is evolving, DRA is an example, in-place pod resize is is an example, gang scheduling to explain that just a little bit. If you did the

15 jobs that I talked and there weren't 15 slices of a GPU available, there were only 14, that whole job would hang. And by the way, the other 14 that got GPUs would sit there and hang until the 15 was scheduled, right? Cuz there's no coordination. So, gang scheduling fixes that and says, "Look, if I can't get all 15 of my jobs out there, then I don't

I don't want you to run any of them." Cuz by the way, you just have to redeploy. So, this is an example of the kinds of features that are continually coming out with Kubernetes as they keep up as the kind of AI native host workload host, the platform for AI native. I told you I was going to pick a very opinionated set of technologies. There's a repo

that goes with this, by the way, that goes way beyond this. I'm not going to talk about things like VLLM or, you know, even like KServe or Kubeflow or anything at a deep level, but just know there's a repo I'm going to show you that has a whole lot more to it than just this. The idea here though is that if we were to kind of break

it into our initial four projects, first of all, let's learn how Kubernetes can be adapted to host workloads. That's number one, and then I put DRA there just to give you an example or a sample of that, right? Number two is Q and admission. Now, no pun all words there. Could be Q, could be Volcano, could be Kia. I think I chose Q just for simplicity's sake,

right? The idea here is that how are you going to get fair scheduling and topology awareness in in scheduling, right? That's the way to do it. Then how do you do distributed training? And then how do you do long-running inference? There's a bunch of things here that we want you to be aware of because the bottom, like the answer to this whole question of like how do

I keep my job and how do I evolve into AI and how do I support all this is to learn a few key things. The other statement I heard this morning from someone was, "How in the world do I like with the massive amount of information that has been brought on by AI?" And the answer is the same as it was 5, 10, 15 years ago. Focus.

Pick one and go after it. I'm giving you a set that's actually fairly popular. By the way, this is based on research about what the CNCF is converging on, right? The around the stack. This isn't just me like picking random things, just so you're aware. And so speaking of KServe and KNative and LLMD, right? This is the next set. So in the repo, you're going to see

a very structured methodology that says this is 1 through 4, which is what this and that this is 5 through 8. And by the way, I'm running models that will run on CPUs. So don't get hung up on like you may have just seen me put my beefy laptop up here, which has a 5070 TI in it. You don't need a 5070 TI. The labs in the

repos all use CPU-based models. You can still simulate training, by the way. Oh, and by the way, just to mention DRA has a simulation driver. So you still don't need a GPU to play with DRA. There's a simulation driver that you can use for that. So KServe, KNative, LLMD, Gateway API. By the way, Gateway API is doing some really interesting stuff. They're trying to route basically model-to-model

traffic, which as you probably are aware is multi-step and typically synchronous. That's fascinating. Um so anyway, those are those are the options that I would have you focus in So, the idea here is that if we were just convinced that down, right? How do we serve and deploy models? How do we schedule them? And what hardware, if we're looking at this from a layering perspective, how do

we get the hardware we need in order to do the thing in question? Now, of course, I know some of you are going to be like, "Or you could just run on a cloud provider." Right? But, some of this still is in play, especially if you're running Kubernetes on the cloud providers. It probably would be completely inappropriate for me not to mention, right? That MCP, Goose, agents.md

um are all part of the Agnostic AI Foundation, which was started in December of last year. Keep an eye on them. This is a Linux Foundation organization. I used to say this 10 years ago, and I don't think people believed me. I used to tell people 10 years ago, when I was working for AWS, I would say, "Let's keep an eye on the CNCF, because whatever they're

working on is typically either going to be like the thing that everybody's going to center on." Right? And then, of course, that turned out to be remarkably true. So, maybe I just got lucky. Right? But, I think there is a big push with this Agnostic AI Foundation that anything that they're going to adopt into the projects is probably going to, in turn, get adopted. Because the reason

they chose it was because adoption and demand was already high, right? I don't think anyone would argue that MCP servers have skyrocketed in adoption more than we've ever seen. Right? But, if you are asking me, "I'm a beginner, what do I need to focus on?" I think you should run MCP servers, but if you're an operator, I would probably focus on knowing more about Kubernetes features first,

about scheduling first. If you're not an operator, then I would say just inject AI into your workflow, and maybe you start with MCP. Right? Has an addition, like setting up Context 7 or DataDog or whatever it is that you're using, right? But bottom line, keep an eye on the agentic AI foundations foundation, right? Keep an eye on them. I am not yet into the agent thing, which

doesn't mean I haven't touched it, doesn't mean I haven't run it, doesn't mean I haven't looked at it. Like I literally gave a talk yesterday where my current workflow is more about getting AI to generate deterministic artifacts that I can test, validate, and commit to versus having an agent have even fettered access to my production infrastructure, right? Is everyone following me? So, if you're a beginner and

you're like, "Everyone tells me I need to use agents. Everyone tells me we're going to put agents into production." I just want to say this, and I've done a lot of security research on this. You might want to wait for a second, right? I'll give you an example. Docker just released micro VMs over containers for hosting agents because agents have a notorious ability to escape sandboxes that

they're stuck in, right? Someone the other day said, "Well, what about bubble wrap? What if I isolate the process in its own little container and isolate it so it can't make sys calls or whatever?" Guess what? It can get out of bubble wrap, right? And I say that as if it's like a living thing. It had directions to do so. I guess my point being is that

I'm a fan of agent infrastructure. So, after you learn those previous eight, you want to start looking at K agent, maybe something like Tool Hive. I would definitely look at Agent Gateway, by the way, which is a relatively new thing inside this the um inside Kubernetes. So, again, kind of like evolution there of like where do we start? Let's start with Kubernetes cuz we're already there, right?

And then let's evolve into add-ons to Kubernetes, and then let's look at these kind of new pieces of software. But I would say right now, hold off on putting the out there until we figure out a little bit more of the security All right, so I would keep an eye on the release blog for Kubernetes as a regular thing. Install things like Q or Volcano or Kai

or whatever scheduler you're interested in. Deploy a model of KServe. Maybe do a deep dive on DRA, right? And then, you know, since I just gave a talk on how KServe's the group functions, wouldn't be bad for you to get involved with the either the AI Foundation or the Gateway working group, right? You want to learn a lot? Get involved in the group in question. Okay. I

would say if I were to break it down, learn how training functions, if you haven't done that, right? Learn about inference, learn about agents. That's the like basic pieces that you need to focus on. I do want to say this and wait a second. Pardon me, just one moment. Assuming that Michael can click on the things. There's a repo that goes with this. It's Cloud Native AI

Lab. It's under people forester. And this has the learning path of setting up a lab, which starts with the just cluster setup, DRA and job scheduling, device allocation, distributed training, model serving. If you want to walk through things like job sets, KServe. By the I I don't have anything to sell you. This is free. You can go work this yourself, right? Notice this doesn't lead to like

a code cloud page. You can download this and work it yourself, right? And learn these technologies. And that's the whole idea. And it does end with owning your own kind of MCP server, right? Which is the ideal. But if you're wanting to get started and kind of understand what's happening to the ecosystem both with core Kubernetes and with its additions, this is a great way to start

to get that feel. I just want to say this, if you just do the first three labs, you will gain a tremendous amount of understanding and experience about Kubernetes approach to adapting to an AI workload, to adapting to that that like that the coming kind of AI workload migration if you will. I'm actually giving a talk in KCD Texas about this. I get a lot of questions

about how are we going to solve these AI workload problems? How are we going to do this, right? What about the you know, pick attribute probabilistic nature of LLMs? What do I need to learn? What do I need to focus on? I get these concerning questions from people who are honestly to a slight level kind of freaking out. Whether it's my 18-year-old and I didn't disclose this,

whether it's my 18-year-old son going, "Does going to Georgia Tech even make sense in this age of AI?" That was the question I got by A whole other thing as a father. And the other one was the 30-year veteran going, "How do I keep my job?" Bottom line is you already know everything you need to know. Your platforms, your tooling, your monitoring, your logging, your tracing, all

of that is all still completely valid. This is just a new piece of I I don't want to oversimplify it. It's adaptive and probabilistic nature is problematic. This is This is just like we went from stateless to stateful. It's just another database, right? So, just know that what you already know about securing workloads applies. The only distinction is that this one is actually going to try and

get out of its sandboxes if given instructions to do so, right? We're given a task. That's it. Otherwise, keeping infrastructure invisible, right? And and one who knows how to like do things like provision GPUs and whatnot, all the boring stuff that we already know how to do, but we now we just have new terms for it. That's the key. So, you're not behind. We want to make

this boring. Everyone's cool, calm down a little bit cuz you already know everything you need to know. You might need to add one or two more tools or maybe a few more concepts to master AI workloads, but otherwise, you have 80% of what you need. So, hopefully this will be useful to you to learn and study. So, go go grab the repo. And otherwise, if you have

any questions, grab me afterwards and let's talk. Unless, thank you.