Cloud Native Theater | Cloud Native University... Cansu Kavili Örnek and Anneli Sara Banderby

This talk focuses on navigating Kubernetes at KubeCon, especially in the context of AI workloads and the growing landscape of technologies. The speakers, Jans Su and Sarah, highlight the importance of understanding Kubernetes for both data scientists and platform engineers as they work with AI models. They discuss the challenges of managing resources effectively in a shared Kubernetes environment, emphasizing the need for resource requests and scheduling, particularly when dealing with GPU workloads. The session also outlines two primary types of AI workloads: predictive AI and generative AI, noting their differences and infrastructure needs. The speakers aim to provide guidance through a series of key questions that attendees can consider as they explore AI deployments using Kubernetes.

Hello. Oh, wow. That's so um thank thanks for coming everybody. My name is Jans Su. I'm an architect at Redhead coming from platform engineering background and I have Sarah here with me. >> Hi, I'm Sara. I'm a data scientist who comes from the life science and the healthcare field and I now actually also work as an architect at Red Hat. >> Right. Yeah. So what we want

to do is this like next 15ish minute is to help you to guide these next three days in cubecon. So uh because that Kubernetes became quickly the de facto platform for AI workloads and the the whole landscape is getting really like bigger and folks from all over the backgrounds are coming actually to learn what cloud native AI is about and um whether it's coming from data science

background or like platform engineering background we want to drop some keywords here and there so that you can make more informed decisions for which sessions that you would like to go and explore. So, but um it's if it's your first CubeCon like Sarah, please don't panic. We know that things can be overwhelming whether you are coming from an like data science background or like a seasoned platform

engineer. Things still can seem a little bit um overwhelming. What do you think Sarah? Yeah, I mean uh I came from a subject matter field so a lot of the basic Kubernetes terminology was very unknown to me and uh it was a bit overwhelming at some points as well. Yes. So don't panic but here is why you might panic. So CNF landscape is like um over 200

projects at the moment and it keeps growing and then there's this whole new a IML ecosystem like PyTorch and like too many models runtimes like new things are popping up in the whole AI space. So don't panicked, don't feel overwhelmed. We are here to help you out. How do you feel like Sarah when you see all these projects? >> I mean the feeling is two ways. First

it looks really awesome because it looks like I have a lot of things to choose from and there's many awesome things happening but then you have a little bit of like a lost feeling like where do I start and what do I choose and depending on who you talk to there's many many good things to choose from. So um >> yes >> so before we get start

though we want to do a little bit of an orientation because in cubecon when you hear that someone is talking about AI or AI workload they meant they might mean two different things oh sorry I was talking about something else sorry I got so excited but um first let's actually um talk about this so as as we said like um we are coming from different worlds so

yeah Sarah can tell us your story. Yeah, I I wanted to draw some parallels because I used to work with this like many many years ago and I didn't know anything about Kubernetes and and the open source world was not maybe as big. So I this is actually a real life discussion. I just TLDDR for you. So when I used to work they would ask me to

create uh a model or a system for something and I would I would develop it and I would save it. I'm like okay that's great. Here is the here's the file. You're welcome. And then they would it would be like, okay, so what do we do this? And I'm like, well, that's actually not my problem. And then I I would try to exit the situation. But uh

uh but they couldn't do anything with it. And when they asked me for help, I wasn't sure either because uh we were completely shadow IT, right? We had we used nothing of what the actual IT was using. So I couldn't actually help them either even if I wanted to. uh and in my case when I started getting into this whole AI workload where like when I was

working in the bank years ago we got this new GPU servers I set them up and then I gave the access to people like data scientists who needs to use these GPUs and then I start getting all these complaints because honestly I didn't know about this whole like prioritization uh scheduling in the uh in the AI workloads because some works some jobs are more actually important than

the others to use these really expensive accelerators. So I was getting this kind of uh kind of like complaints um because before that like I was setting up the cluster setting up the access setting some kota off you go everything was everything was good but this is different but even though we were speaking different languages what we realized is that we are both here to you know

making the AI workloads running at scale in production safely so we started to actually like um you know working together and start understanding each other's languages. So, um this is what I was actually getting into uh because I really want to set this up uh like make sure we are on the same page before we move. There are like two different AI workloads that you may hear

in your uh in these next couple of days. Uh on the left side you have predictive AI where is um you know you train a model with your own data things like fraud detection forecasting recommendation systems. So this is where you train the model, serve the model and they don't usually may not need really high expensive accelerators. they may run on the CPUs and on the right

side we >> the generative space and uh this is where most of the talks will be around and um there we're talking more about much much larger systems and there also the requirements for hardware can be very different and what's really cool about this is that when I used to work um this wasn't a bigger topic like 10 years ago and uh just having this hugging face

where you can just peruse all of these different models is really fantastic. Like I'm not that old, but in in my days, h if you found something, you would you would call a professor who had published this article and you would ask him to share something and maybe he would, maybe he wouldn't. So it's it's really really cool. >> Yeah. But the good news is that at

least from Kubernetes point of view, infrastructure story is almost the same for these these types. for example, you still like need containers. You still need like GPUs, you still need to set the um infrastructure for both of these workloads. We are going to lean more on genai side because that's where all the you know pipe is. That's where all the um that's why most of the crowds

are for. So we're going to actually talk more about that side. So what we did is that we came up with these four questions and those four questions are type of questions that you're going to actually answer when you start your AI journey. So we are at uh let's say that we are at the entry point where um you have a model or you have an idea

and you want to take this to up to the production. So we're going to go through these questions and then see where we um what kind of talks that you can actually get into to answer these questions. So the first questions is where does my model live? Um okay so in this world the model uh model needs containers right? So um yes and and when when I

used to work the most uh we were as I said previously we were shadow it so we had all each our bare metal server under our desktop and we we would just put things there quite disorganized. Um so for me like when I got into the containers it was fantastic because I was struggling a lot with the fact that different models had different dependencies and I could

lock it down into this container that was absolutely amazing. >> Yeah. So from the platform engineer point of view, I was like um thinking like what I should talk about like what's different in this world is like the containers that you're going to get into like huge like those model artifacts those like the models that you can download from hugging face they could be like gigabytes even

up to up to terabytes and those runtimes that um like VLM runtime that's like becoming the the facto standard for model serving they are also not like light they're at least like 15 gigabytes of containers. So there are like things that you need to actually think in a little bit more architect when you are going up to scale like how those artifacts are going to be downloaded

in the runtime and uh how you going to catch them. So you need to look into this kind of stuff when you are planning to serve the models on your own kubernetes cluster. So the next question is that how does it get resources? So in Kubernetes world what you do is that when you are deploying something is that you declare your resource request right. So you could

CPU memory and when it comes to AI we talk a lot of GPU. So you'll get to see this kind of like annotations in the workloads that's kind of the magic line to tell that like give give GPU for this workload. >> Yeah. And uh it's also interesting because um when I used to work then we would always have everything we needed. You didn't share anything with

your co-workers in that sense and you didn't really have to care about there were other people in your group also wanting to compute. And in Kubernetes now you definitely were sharing a cluster with other people. So something that as that I had to learn as a data scientist is to be able to actually declare how much resources I need. I didn't have to think about that before

and learn also how to uh think about scheduling workloads and share resources with other people. >> Exactly. And um for the platform engineers I would highly suggest you to take the words like um Mick Dynamics resource allocations or like Kai from Nvidia there are like talks about these things these GPUs these accelerators are like very expensive so you want them to use efficiently like you need to

share them and make sure that they are not sitting idle. So that's the uh that's one of the things that the platform engineers should look into the utilization of these and how to share between the between many data scientists or like AI engineers, right? The next one is how do people talk about it? You figured out which model you want to serve, you figured out the accelerators

and then now it's time to decide. >> Yeah. And previously it was it was um you know I would sit and I would have my notebook or my ID and I would load the file and I would call it in a prediction function basically but that doesn't necessarily make it sharable with other people and it doesn't expose it to any other applications. Um so >> yeah and

when it comes to the Kubernetes point of view in the keynote maybe that you also heard about this there's a super exciting project called LMV that just got donated to CNCF. So the scaling AI workloads are not as as straightforward as scaling a web server. So it's it's not something that you can exactly solve when you just add more replica like just add more post into it.

So you want to actually more thinking about some uh intelligent routing what we call like inference intelligent um inference aware routing. So look at these kind of um talks that goes into um scaling AI and LLMD and some words like KV caching because there's a different world in here like um you get bunch of good stuff inherited from Kubernetes in terms of uh you know the exposing

a service in terms of the um setting some networking rules but when it comes to scaling it's a different story in here and how do I enhance or or iterate we want to drop a couple of words and going back to a little bit of predictive AI side again in case you hear these things just want to um talk about this this um this stuff. >> Yeah.

So, so there's a big it's a slight difference between the two now but if you compare it then one of the biggest difference I see is that when I was working we would always create everything we needed from scratch all the models and and instances and you don't really necessarily do that with generative AI. you have a hugging face, you have these fantastic models out there that

you that's able to be used. So you download it, you evaluate it, and you you don't necessarily train from scratch, but you can fine-tune it or you can customize it with added knowledge bases and so on. So introducing new information without having to do any training or real training, which is is kind of actually really cool when you think about it in that way. >> Exactly. And

um either way like where you are at what kind of workloads that you are getting into your Kubernetes cluster either way the whole uh good practices we know from DevOps world are still uh apply like when you change something in the system you need to constantly monitor them and evaluate them. So even though they they are like two different worlds you still um they still kind of

meet on a middle ground which is the evaluation part of it. So how you going to navigate this this H cell of CubeCon? Uh so we have these four questions like um how can I serve the model? How can I uh how can I provide the resources in the best way? How can I um make sure that people can access it and with the uh how can

I also be ready for the load that I'm getting. What we suggest is to like don't think like I'm platform engineer or like I'm a data scientist. Think more like what kind of things you want to lean into. Um and then there are a couple of like um categories that you can look into. >> Yes. >> Um for example like on the if you are on the

performance or the operation side look up for the GPU scheduling batch versus real-time inference multiode training or like uh when it comes to the operation the whole things that we actually know and kind of able to get things from Kubernetes like observability the cost management multi-tenency those are still apply. So if you want to look into those talks whether they are AI related or not just go

for them. >> All right. So these are our picks. So >> Yes. So I picked some if you're coming maybe from my angle as a data scientist or analyst or whatever. So um these are some I would recommend. I would definitely look at from laptop to cluster. This is basically where I started and many of my colleagues and I still have discussions with data scientists who are

still working on laptop and want to learn how do I move from my notebook into the Kubernetes production environment when I actually have to start thinking about my deployments in a more cloudnative way. And then of course observability because this was something I didn't care about at all when I was working the most to be completely honest with you. This is something I would definitely uh encourage.

>> Yeah. And these are a couple of picss for the platform engineering side of things. Like I said, there are like a couple of multi-tenant GPUs, how to share those workload. The memory management and making genai production ready with Kubernetes pattern is going to be a um good talk. Um and then there are um like yesterday's platform engineering and agent day was also uh really good. I

believe that those talks are going to be online. So you can also check them later. And with that uh what we what we can tell you is that find a buddy if you uh if you can like find a data scientist or like if you're a data scientist if like find a platform engineer and you know talk with them like not just get into the talks but

like ask any questions you have in your mind try to understand each other's world and yeah buy each other a coffee and hang out. >> So thank you. Thank you.