Ballroom A Saturday Mar. 07 - SCaLE 23x

This talk introduces Kaito, a tool built upon the Kubernetes operator pattern aimed at simplifying the deployment of open-source machine learning models. The speaker discusses how Kaito allows users to define their infrastructure and model requirements using YAML, enabling a GitOps-friendly approach for infrastructure management. The session explores the capabilities of Kaito, including the workspace controller and node provisioner, which facilitate running models like Mistral 3 or Gemma 3 within a Kubernetes cluster. The speaker highlights the ease of deploying models through simple YAML manifests and introduces the RAG engine controller, which allows for enhanced functionality by integrating a retrieval-augmented generation service. The demonstration showcases how to setup and interact with a chatbot application that utilizes Kaito's features for enhanced AI capabilities within a secure and efficient environment.

And so, KATO is built using the classic Kubernetes uh operator design pattern. And what that means is you're basically going to interface or your user experience with this KATO uh thing is YAML. What does that mean if you have YAML? Well, that means it's just a code file. If it's a code file, that means I could check into my repository and it becomes really GitOps friendly to

kind of like bootstrap the infrastructure into your environment, right? And so just to give you a quick highle overview of what this is, the operator allows you to declaratively express your intent in how you want to run an open source model, right? And so uh when you install Kaito, what you're going to eventually install is the workspace controller and a node provisioner, right? And the user doesn't

have to worry about all this other stuff going on in the background. the user only has to worry about one custom resource which is the workspace. Um I'll show you a workspace in in a little bit but the workspace basically allows the user to say hey I want to run a uh Mistral 3 or Gemma 3 model open source models and run that all locally within my

cluster. Now some folks can just take advantage of the model as a service like OpenAI endpoint and all that stuff. there are other organizations um that is a little uh more sensitive uh to data uh leaving the cluster. So they want to keep everything in cluster. So this is going to be a great way to achieve that. Okay. And so um Kaiito allows you to basically write

a workspace manifest submit it into the Kubernetes API server and then as soon as that happens the workspace says ah yes I see you want a I don't know uh 54 mini model let me go put out a node claim right a node claim is basically a a resource within the whole carpenter uh project so it does leverage carpenter and so the node provision will say okay

I need to go get a skew excuse me a GPU node which could be a skew in the cloud and it'll actually go pull that provision a virtual machine do all the necessary uh infrastructure work like installing the NVIDIA device plugin and all that other stuff and then attach it to your Kubernetes cluster and then once that is there um it'll actually schedule the workload which is

going to be the inference service that inference service is actually in a container and it gets deployed and scheduled just like any other Kubernetes resource and so um at the end of it all. You get an open AI compatible endpoint serving your model. And uh the the inference runtime that it uses by default is using the VLM uh runtime. Uh VLM is a very uh popular open

source uh uh inferencing engine, but it also supports hugging face transformers as well. So you do have an option there, but the default is VLM and I suggest that you just stay there. Okay. And so what this allows you to do is it really allows you to go from zero to inference in just a few minutes with literally just like a few lines of YAML, right? So

as a user, you could even pick one from one of the presets. Presets are like uh YAML manifests that have already been written for you. All you do is say, "Yep, click and deploy." Right? And so a lot of the popular open source models are there. And so you just do a cubectl apply and boom, off it goes. Uh it's there in a few minutes. Um, what

I want to do next is just show you um, uh, how to just see it in action and how to get, uh, get rolling with this. So, let me just break out of this slide real quick and then flip over here. Uh, let me open this up. Um, probably the easiest way to get going uh, as soon as soon as you have Kaito installed. Um, okay. Uh,

as soon as you have Kaiito installed in your cluster, probably the easiest way to get started with working with the workspace is probably from the headlamp project. Has anybody here heard of Headlamp? Okay, a few people. Um, if you're just getting started with Kubernetes and and you're more of like, I need to see this from a a guey perspective. Headlamp is an open source project. Um, it

was actually donated to the CNCF. So, it's a CNCF project now and it's like the de facto Kubernetes dashboard at this point, right? And so, um, Headlamp is really awesome because it has a very, uh, strong, uh, uh, plug-in architecture. And so, you can literally install whatever plugin you need. And so, um, Kaito actually has a plugin for Headlamp. And so, if you just search for the

plugin, um, Kaito, you just install that into your cluster and you can, um, pretty much work with your Kaito resources. And so if I connect to this uh cluster over here which is running in Azure and it has the Kaito installed I can scroll down and I can immediately see my KATO workspaces. You can see I deployed two and um I can start to browse through the

model catalog. So these are all the presets that I mentioned. Um so these are all the preconfigured uh model deployments that you can choose to basically um deploy and the YAML here is already written for you and it's only a few lines of YAML. you can uh review and deploy that. Uh but I already have that. So I can just start chatting with my model right away.

So um if I just pick one of my models here, 54 mini instruct and I say hi because we need to be nice to the AIS. Um I can say something like can you tell me about the scale conference? and then it'll come with some gibberish uh answer because it doesn't really know what I'm talking about. And yeah, it it looks like it it really doesn't know

because it's trained on data from 20 uh 2023 or so, right? So, um quick and easy way to get started with models running inside your cluster, but we need to do more than that. Okay. Um so if I move on here back to the slides and we move on the other thing that Kaiito also offers um is it offers this rag engine uh controller that you can

install and it's completely separate from the workspace but you can use those two together. Okay. Um the rag engine is basically just another custom resource that you deploy into your cluster and it's just as short like YAML wise. It's only like 16 lines long. And the way it works is like this. Rag engine controller listens for uh custom resources that you deploy in the cluster. And you

say, "Hey, I want a rag engine um that's going to basically uh spin up my entire rag service." So everything that you see on the right there, all that gets spun up with your one single manifest, right? So you're getting the vector database, the local embedding service, and the open AI compatible inference service, right? And so what does that give me? Well, that gives me the opportunity

uh to basically oops, if I go back um to basically use the OpenAI compatible inference because remember we said that we want to uh query the rag service, have it retrieve documents and then send it over to the LLM like notes uh open notes for the test. And so that's what we want to do. And so that's what I'm going to demonstrate in this sample application uh

for you. Okay. So, what is a sample application? It's just a chatbot. Uh, I know we're probably all tired of seeing chatbots at this point, but here we are. We are looking at a a chatbot architecture. And, um, the way it works is has a front-end web UI and a back-end service all hosted on a Kubernetes cluster. Naturally, when a user wants to interact with the chatbot,

they're actually going to come through the ISTTO ingress gateway, and that's going to send the request down to the web UI. The web UI um is where all the uh um uh I guess AI requests will go and that's actually going to flow through my agent service. The web UI is built using uh uh Nex.js. Agent service is written in Python using the Microsoft agent framework. And

because I'm using the Microsoft agent framework, I get a lot of the observability tooling out of the box. And so I have an hotel collector in my cluster that's going to send all of my observability metrics down to uh a graphana looks good to me stack, right? And then I can have chat history all and all that good stuff stored into my Reddus uh session. But that's

um uh aside from everything else, the thing that we want to focus in on is this Kaito gateway that I have installed because we have uh Isto installed here. I'm running a gateway API um in there and that's going to allow me to route between the two different services, right? Rag engine and workspace. So that's the high level uh architecture of this and um look we're going

to take a look at the app, right? So, we're going to see how we can take this app and see it progress from I it not having context of the scale conference to grounding it with real data and then having it give us real answers that's actually actionable for us. And um we're going to do it live. So, I have the uh I have a lot of

the infrastructure kind of like halfbaked um but the application is not there. And I thought this was funny because uh last year I was doing a talk and I had a lot of problems with the internet um and I had to fall back to recording. But we're gonna try this and we're gonna see how it goes. So just to reset one more time because it's going to

I'm going to progress through. I'm going to show you a lot of different things along the way. So again the workspace the workspace is the smallest unit of inference within the whole kaito project. You get a node, you get a deployment, you get a pod, you get a service, you get a open AI compatible endpoint, right? It's all pretty simple Kubernetes stuff at that point, right? What's

new is this whole concept of a inference set. It's more of a higher uh higher level abstraction for a workspace. Um, and you would actually want to use the inference set um for dynamic model routing, which we're using, and for inference scaling with KADA if you need to, right? If you need to scale out your model. Um, but the workspace again is fundamental and foundational for this

whole thing, right? Because like I said, you're going to get a node, you're going to get a deployment and service. So, it's all very simple Kubernetes at that point. Okay. Next, let's talk about uh routing. So, for routing, I'm actually using the gateway API inference extension. Uh this is actually an extension built on top of the whole gateway API project and it basically enables you to do

dynamic routing for models because you know there's no one model fits all type scenario. You want to use different models for different jobs. Um but what you don't want to do is you don't want to set up multiple gateways for each model. And so with the um gie or the gateway uh API inference extension is you can now do bodybased routing, right? And so you can in

the request body you send what model name you want and it'll know how to do the um uh routing to the appropriate model. And so what we get is something like this. The user says, "Hey, I want to make an inference request." The user doesn't really have to tell what model it wants. The application will know that. Um and so it goes to the gateway API inference

extension and it says, "Oh, you want the Gemma 3 model? Okay, go over there. That's where the model is and that's the way it's going to work." Okay? And so let me um show you uh jump into another demo and show you how this uh gateway API inference extension actually works. So what I will do is um I will port forward actually I'm going to port forward

that and clear oops clear I will port forward to my uh kaito gateway uh infritz extension and it is up and running and so what I can do essentially is um this v1 chat completion this is basically an openi compatible endpoint I'm pointing to my uh local host on 4,000 because I port forwarded that and then I am basically saying hey I want to make a request

to the 54 mini model right and so if I go ahead and uh send that request over it's going to come back and I always chuckle to see what it gives me so scale is a prestigious high school level academic competition okay yeah that's completely wrong right um so that's not the model that I want for the job the model that I want let's try the Gemma

three model, right? So, same exact endpoint, same exact uh host. All that's different is the model that I'm calling into, right? And so, the routing is happening uh appropriately. And then I get a response back. Let's see. Um, okay, here's a comprehensive overview of the scale conference, the Southern California Linux Expo. Hey, that's better. So, models do matter, right? Um, so that's that is great. Now, can

I ask it a follow-up question? Um, so can I ask it a question like, hey, will there be any sessions on Kaio at the conference? So, same exact endpoint. Um, going back over to the Gemma 3 model. Let's see. Um, it's going to come back and it says, "Yeah, sure. Uh, the scale 2024 conference schedule has Kaiito." So, here we go. Now, it's not telling the truth

again. We need to actually fix this, right? So, how do we fix that? Go back over to the slides here. Um, we do this by, uh, basically adding the rag component and then routing the entire application through the gateway, right? And so, um, I'm going to take my application, I'm going to hook it up to the gateway API. And the gateway API will basically know, uh, where

to send the traffic to. And so if I jump over to my Oops. Oh no, I didn't want to do that. I want to bail out of Let's jump over here. And my application is out of sync. So let's just go ahead and sync it real quick and we'll wait for that. So while that is waiting and deploying, um I'm going to show you just a little

bit of uh source code for the application. So, um, the web app isn't really anything special. All it does is basically route all, um, AI, um, requests over to the agent service that's running in my cluster. Um, the actual agent itself is written in Python. And so, if I just scroll down here, uh, I have instructions. Where the instructions go? Um, so here's my observability bits there.

And there we go. So, uh, how do I word wrap this? Boom. Word rap. Okay, there we go. So, I have I have instructions for my model and I'm saying, hey, you're an expert at, uh, Linux and free and open source software. You know everything about the scale conference and, um, the scheduling and all that good stuff. So um and it's also saying you know hey uh

always site relevant sessions by providing their titles locations right but our application can't really do that yet and let me show you. So if I head over to this website uh chat.ksrocks I can say uh hi because we want to be nice to the AI. And this is uh this is the awkward part where we wait for the a okay so it says okay yay uh I

am I can help you with anything related to uh free and open source software at scale 20 23x so I say uh will there be any sessions on kao will there be any sessions on kaito let's see we know the answer it's going to be no but let's just let's just see this through. Okay, it's thinking, thinking, thinking, thinking. There we go. All right. So, it's saying,

um, yeah, there is a session. There's actually two. Wow, there's two. Huh. Um, but these are from like two years ago, 2024. And, um, was Kaiito even out then? I don't even know. And they have links, and these links probably go nowhere because they don't look familiar to me. Um, so we are definitely going to go fix this. Okay. And so the way the rag engine is

going to work is it's just another custom resource and it functions almost exactly like a workspace but with a little bit of a difference in that it can pass through to the actual uh LLM endpoint. Right? And so um the thing that you see on the right here that was the workspace. That's what we uh kind of like talked through and and worked through. So you get

a node deployment service, right? When you deploy rag engine, guess what? You're getting a node deployment and service. Okay? And so the difference here is that the user is now interacting with the rag engine service, right? Because everything is open AI compatible. I have very very minimal changes I need to make in my application code. So now my app goes through the rag engine service. It retrieves

the relevant documents and it sends those relevant documents over to my workspace. But guess what? It doesn't need to be a workspace. You don't need to use rag engine and workspace together. If you have an OpenAI compatible uh uh inference service available somewhere else, you can definitely just forward that service to that somewhere else and it'll work just fine. I'm just so happen to use both of

these uh workspace and the rag engine together. But just know that it is not a requirement there. Okay. And so what I will do is um let's go show what the rag engine looks like. So let me just bail out of here again and let's close this and let's stop port forwarding that. Um so what I will do is I will go ahead and hold on Kport

forward there. So my rag engine is uh name scale uh 23x and what I will do is I will open up my test file for rag indexes and basically uh I could just make a call uh on the slashindex endpoint just to see what I have in here. I have no data, right? And so what I'm going to do is this. I'm going to stop that. I'm

gonna do a Argo submit. So I have a workflow template. I'm just using Argo workflows for my indexing pipeline. And so I'll just go ahead and get that started. This is going to take a few minutes. Um, basically it's going to download the entire scale 23x schedule and it's going to index that in my rag index for me at real time. So, while we're waiting on that,

let's actually go look at um what this is actually doing. So, I do have a schedule project here and I am parsing the schedule and sorry scale folks, but I did go and scrape your website for all the presentations for this conference. And so, this Python script just basically did that. uh it goes and builds out a JSON file and then it formats the JSON file into

a format that uh the rag engine needs it to be in. Right? And so if I just want to just show you example files of what that eventually looks like is um when I go and pull down or uh scrape all the information, it looks like this pretty easy to read and follow uh JSON document. Um, but when you want to index your JSON uh data into

the rag engine, it needs to be in a specific format. And all it needs really is the index name. Okay? And so here I'm just calling it schedule index. It could be whatever you want. And then an array of documents. And within that array of documents, um, I have the text. This actually becomes a document itself. And then I can plug in any metadata that I want.

Right? So here I'm just trying to shove as much information as I can into the index so I get a lot of relevant information coming back out. Right? So um things like the title, the description, um the URL so that I can find more information about it. Um what room it's in, the time, all that other good stuff is is everything that I really wanted uh to

see here. So um let's just check on the status of this. Okay, so it's still it downloaded and it's formatting now. So we'll let that continue and let's go take a look at the actual uh rag engine um manifest. So you can see here to deploy rag engine. That's it. It's just um just a handful of lines of code. And the important thing here is the instance

type. So I'm running this in Azure and so I wanted it on this specific GPU node. Um but here's a little tip like for rag engines you probably don't even need a GPU node. Um, embedding data doesn't require it's not very compute intensive. So, you can probably get away with it on a CPUbased node, but I'm just using this skew here uh because I I had access

to it. Um, the embedding model that I talked about u you can actually point it to a local this would eventually be a local embedding model. Um, it actually pulls it down from hugging face. But if you had like a remote embedding service available to you, you can actually point it to that. Um, probably the most important thing here is the inference service, right? So, after the

rag engine retrieves all the documents, it needs to know where to send that request to. And so, this is where I'm actually uh pointing it and I'm saying, "Hey, you know what? Get those documents and take and and and send it right back out to my Kaiito gateway." And my Kaito gateway will know what to do with it. Okay. The other important thing here is context window

size. Okay. You do not want the rag engines to send more context than the model can handle. And so this is where it gets a little uh dicey in that you need to know what the backend model uh context size uh could be. Um so think of a safe number there. Okay. Um so now let's see everything looks indexed. Everything is good. So, if I do this,

uh, clear and if I just port forward one more time to my, uh, 5002 scale 23x service. Um, and then let's go grab my test file one more time. So, I can send the request to the indexes. Boom. There we go. I have an index. Now, um, what can you also do with the rag engine API? Well, you can actually persist the indexes. The rag engine can

actually host multiple indexes, but it could only serve one at a time. So, you can kind of like save this index and and load them uh whenever you need to. So, um we can also do a um do a get to retrieve all the uh documents. And so, you can see here that a lot of this stuff was actually um indexed and you start to see all

the session data. So, that's great. Um now since we are port forwarding to the um rag engine I can actually make a post call using the same open AI compatible uh endpoint. So v1 chat completions and basically I'm passing it the model name because we know the gemma 3 works a little bit better. um and then uh the index name which is schedule index and I'm asking

it will there be any sessions on kao right so I send that request and let's see what it comes back so it's saying yes there will be a session on kao the title is rage against the machine yep and there we are that's me we are in ballroom A so this is exactly what I want the LLM to do give me facts Now, let's see. Um, let's

push this a little further. And we can actually test it through the um through the gateway now. Okay. So, if I jump back over to the rag engine, um you can see here I want everything flowing through my um gateway API, right? And so what I can do is at this point it's just normal gateway API routing. So I have a HTTP route and I want it

accessible at the slash rag path but I'm doing a little bit of uh um I wouldn't call it trickery but um I'm doing a little bit of URL rewriting. So anything that comes in on the slash rag path go ahead and do a URL rewrite. put the V1 back in, but this time send it to the scale 23X service, which will basically funnel it back into my

my rag engine service. Okay, so if I do this, if I just do a port forward again, but I port forward back to my um Kaiito uh let's see, where did this test file go? And then if I scroll down here, so now um you can see that the endpoint just changed just slightly. So instead of v1 chat completions, I'm doing rag chat completions. Um still going

to the same index, same model, and I can actually uh test it behind the ISTO gateway. And so if I submit that request, I'll get some answers back. And you can see here that yep, I'm getting the same correct answer back. This is the same session. That's great. Um, and let's actually push our luck just a little bit and ask it a follow-up question and just to

test the metadata. And I'm asking it, hey, can you also provide me the URL for that session? And yeah, sure enough, the metadata is actually working. And it's saying, yeah, here's the URL. So, if I click that, there we go. So, that's the exact session that we are in. Right. So, where do we go from here? Well, oops. I need to go about it this way. There

we go. Um, so what did we do there? We basically index the documents and when the user queries the rag engine, we are embedding the original query, retrieving relevant documents and sending that original query with the relevant docs to the LLM. Right? And so this is kind of the flow path uh that I uh illustrated for you over here is um go to the rag endpoint goes

to rag endpoint retrieves the documents sends it back to the gateway API because the rag engine didn't change it's it's it everything goes to the gateway API and the gateway API does the modelbased uh bodybased routing to send it to the appropriate um LLM. So now let's do this I went too far final test run. Okay. So what we will do isum stop the port forwarding there.

Clear that out. Um because um I have my entire app is basically all um githops based. Right now all I have to do is basically do a just change the overlay that I'm deploying. Right? So this is all being done in the Argo CD environment. And so what I can do is uh get diff. Okay. And so you can see I'm just I'm just changing it from

the my overlay was called direct which is like just going to the direct um uh gateway API. And so now I wanted to route through the rag. So that's that's basically all I'm doing. So if I do a uh oops get add get commit and then a get push, it'll eventually go. Um it'll eventually go and then I can do like arg app sync. We'll go ahead

and sync that because I don't want to wait for it. I just want it to just go. that looks good. And I can do uh Argo CD app sync sync chat prune it too. Um and I don't know why I did this, but uh my app is actually using Argo rollouts and I don't want to wait for that either. So I'm just going to force push uh

Argo roll out. So, I'm just going to force promote that. Okay. So, while all that is actually being deployed out, let's just go take a look at the overlays just one time just to kind of show you um what the difference is. So, um in my actual app, this is why I had to sync uh two different things. Um I actually made the change over here, right?

So, I'm using the app of apps um pattern for this. And so, um, I'm I'm saying, "Hey, deploy the chat app and use this specific overlay. It was direct, but I just changed it. I just flipped it over to rag." Um, and what that means is now that I'm going from the direct, which was pointing at the KATO gateway slashv1 as my open AAI endpoint. Um, because

I made this switch over to my rag overlay, I'm now telling my agent framework application that the open AI API base is actually at slash rag. Right? So this is kind of like um the way I can implement ragbased or grounded uh AI system into my application with very very minimal application code changes, right? Because this is all open AI compatible endpoints at this point. Okay. So

uh let's just check on the status of where we are at. And if I do a k get p. Um, okay. So I have my agent applications running. They're 100 seconds old. Okay. So this is good. This is good. So if I take that same question, paste it back in and I ask it will there be any sessions on kao? Right? So now just think about that

diagram again. It's going through the rag engine, getting the appropriate documents, coming back down through the KATO gateway, and then sending that request off to the LLM. And here we go. I can already see that it's that it's working. So it's like, yeah, you're right to ask again. My apologies. So it's the AI is nice to me. You be nice to the AI, the AI will be

nice to you. That's the moral of the story, I guess. But that is the session that we're in. Um we are definitely in this room. It is this time and the um the metadata is all working right. And so just in a few minutes here uh we got a glimpse of like how easy it can be to build AI systems and have it grounded on factual data

uh just all using Kubernetes uh resources. So um would you consider that pretty easy? Hopefully. Okay. Uh so key takeaways here um is that context is everything. Okay. Um rag is a foundational tool to the entire AI stack and it can be coupled with uh some of the other tooling and techniques that are available uh there. But rag is definitely something that you should u uh plum

in, right? But rag pipelines can seem more like plumbing than the actual product, right? And so this is why you leverage a thing like kite and the rag engine um to help uh make the ease of adopting these technologies uh just a little bit simpler uh giving you a nice shorter runway uh uh if you will. And um I would say leverage the gateway API extension within

your uh AI systems uh because models matter, right? Um and you want to pick the right model for the right job. Okay. Um the one thing that KATO rag engine will not give you is it won't give you um a way for you to I extract and prepare your data for indexing. Right? So that Argo workflow that I was running, that's just something that you're probably going

to have to um invent uh because each organization has data in different places and different processes. So that's not a concern that the Kito team was um looking to address there. So I would say I mean do something that you know kind of like what I did start small, iterate and um yeah go from there. So, uh, with that, uh, I would say go ahead and try

the demo out for yourself. If you scan that QR code, it'll take you straight over to my GitHub repo where I have this entire thing documented and um, you know, you can actually read through it and and and go do the uh, go do the thing, right? yeah. And I guess that is it. I don't know how much we are doing on time. I think we have

15 minutes for questions. >> but yeah, these are some um some links that are available to you. They'll take you to all the KATO docs around the things that we've described here. Um I would also call out that um Kito is a CNCF sandbox project and so if you are on the CNCF Slack, there is a Kaito channel uh they do run community calls every two to

three weeks or so and so go ahead and check that out. Um the chat application that I just demonstrated to you, it is actually live. Um, if you want, you can play around with it over there, but just, uh, just a little bit of fair warning. It's not meant to handle a lot of load. Um, I think I'm only running like two replicas or whatever. So, uh,

but feel free, uh, play with it. Uh, figure out what your next session, uh, would be. Uh, I'm probably going to take down the application when I get home tonight, but other than that, yes, thank you for your time and attention. >> Thank you, Paul. I guess I can ask the first question. Um, how dense is the uh, contributor and maintainers? um uh for Kaido today, >>

how do you >> how how dense how how many folks are already involved in the container and um >> the contributor? >> Yeah, primarily the uh contributors and maintainers are from Microsoft and there's probably about uh I don't know like a handful of engineers probably maybe six maybe more. I mean I'm just guessing. Um but yeah um but they're they're definitely uh putting in a lot of

work and cutting releases. Um, they just cut the 0.9 release, I believe, like a week ago. Um, and it's it's moving pretty fast. So, yeah. Yeah. All right. Open up questions. >> Hi. Yeah. So, is this like a a sandbox for learning or is this something that you can take as a template for a production roll out? Um this is definitely a productionready system. It's meant for

you to run um pretty largecale open source models within your cluster. Um I just so happen to build a chatbot on top of it. Um you know using some of the techniques that is available through Kaito. Uh but it's definitely um um production ready. Um and as I just briefly mentioned uh with the whole inference set concept um you can have like distributed models um you can

also scale based on kada metrics. So yes it is it is definitely meant for production. >> Follow up also on u when you pull the code down um can you run it on on a laptop with kind or because I saw that there are hooks for GPU. >> Yeah. Is that necessary that you have something out outside that's a GPU server? >> Not necessary. Um the original

implementation was to kind of like leverage the GPU provisioner and have it go um pull or provision an Azure GPU. Um you don't need to do that. If you want to um basically implement your own workspace template, you can point it to an existing node. So there's a node selector and if you already have a node like let's say in your kind cluster you can say hey

I want this workspace but go to this node and you can definitely use existing infrastructure. Hi there. Just a curious question on Kaido. uh does it provide like a way to export the chat data if you wanted to run analytics on how people are using chat or even store that in its own >> database that you can run analytics on. >> Kaio Kaito itself does not do

that. Kaito is primarily concerned with provisioning the infrastructure for that. Um the way I implemented it in my sample application is I use Microsoft agent framework um which has hooks and the ability to export um the the chat metrics, chat tracing and all that other good stuff uh out to Reddus. So it really depends on the framework that you're going to use for your application. Um but

yeah, Kaito will stand up uh the GPU and basically allow you to host your model. >> Gotcha. Makes sense. Thanks. >> You're welcome. Any other questions? If there's no other questions, um I will be at the Microsoft booth all day today. So, uh we definitely recommend that you just drop by and, you know, have a conversation. We're actually going to be raffling off two Xboxes today. So,

that's going to be cool. Hey. All right. Uh just I was just wondering so if if you're using if you're uh using uh like Azure AI Foundry >> um is is there a benefit to going this route as opposed to using what AI foundry has? >> Um is there a benefit of one over the other? That's that's really hard to say. Um they both do kind of

the same thing. Uh AI foundry is more like the a model as a service type um method of getting your inference endpoints. Some organizations they need everything within their network. They need everything within their cluster and so they'll opt for a solution like Kaio um to make sure that the model is running in the cluster, the app is running in the cluster and nothing ever leaves that.

So yeah. >> Gotcha. All right. Thank you. >> Oh yes. Here we go. >> URL. >> Oh, sorry. Uh Where is it? Where is it? Where is it? I know I had it up. >> Paul Du Paulu. The actual dot wasn't available for me, so I had to spell dot. Any >> Any other questions? >> One last question. Okay, we have time. We have time. >> All

right. Yeah, I was just curious how much of the demo slides and the code uh the demo code. Um how much was that was uh did you get aentic AI assistance to create all that? Like what percentage? >> What percentage? I would I I can't tell you the percentage but uh a lot of I I did get help from agentic AI that is that is for certain.

Yes. Um yeah. Yeah. But you have to like steer it in the right direction, right? So yes. Yes. >> Thank you Paul. All right. Thank you everyone. Thank you. Yeah, they won't hear you. Okay. >> Hello. Oh, that is me. Yep. >> Say hello. >> You can hear me. Hello. >> Hello, Fab. >> Ah, golden. >> So, after this talk, Fab is going to tell us how

to be a golden cub cubstronaut in one month. All right, I guess we'll get started. We got Scott and Adrian to talk about secure scalable uh GPU sharing on multi uh um multi-tenant Kubernetes. Right. So Scott uh Michael sir, did I butcher your name? >> Okay. >> Um he's an appdev background and he focuses on cloud native and Kubernetes and is also an educator. We're looking forward

to that educating today. Uh Adrian uh you're with Hashor and uh staff solutions architect and a former web developer. >> Cool. Awesome. All right, we'll get >> Awesome. Thank you. So before we get started, I need your all's help. I'm come from Seattle, Washington, which is up the coast from here. And I need to prove to my bosses that I didn't come to Los Angeles just for

the bright sunny sunshine, which we don't have in Seattle this time of year, by the way. Uh so if you all could help me out, I'm going to go over here, give you a smile and wave. Adrian, turn around. There you go. Smile Awesome. Thank you so much. >> Yeah. And it's even worse for me because I'm from Paris. So it's it was even a longer trip.

So yeah, thank you all for joining us. Thank you for maybe skipping or postponing lunch for us. And let's get started. All right. So I'm Scott Mallister and I love to build software, but I love the interaction of software and hardware. And that's what we're going to talk about today. We're going to talk about how we can get the most out of this expensive hardware that we're

all starting to use these days with GPUs, but how to get that software to interact best with that hardware. But in addition to technology and the hardware and the software, I also love cheeseburgers. Who loves cheeseburgers? And so do I. I think they're delightful. I'm also a developer advocate at a company called Depot. Depot is a build acceleration platform that essentially is a suite of tools and

services that make container image builds and CI pipelines dramatically faster. We're not going to be talking about Depot for the next 60 minutes if you want to. I'm game. We can talk about it outside, but for the next hour, we're going to be talking about these rare and expensive pieces of hardware called GPUs and how we can get the most out of those. And I'm Adrian. I'm

based in Paris. I'm a solutions architect at Hashi Corp. And before that, I started my career as a web dev, then moved into SRE. So, so all sorts of different things. And I love broccoli. Who's with me, >> right? It's not an eitheror. I can love cheeseburgers and broccoli, >> right? Especially in Southern California, expecting a bit more >> a lotism. Come on, guys. >> Yeah. More

vegetation. Anyway, >> yeah, >> maybe it's avocados. That's a California thing. Avocados. Yeah, that's what I thought. Yeah. >> Okay. Okay, it's we can continue. >> Yeah, we can agree to disagree. So, why are we talking about this? That's the real question. The thing is is traditional Kubernetes, right? It was built for CPUs and memory. It works best because the the kernel controls everything. And so, Kubernetes

isolation uh essentially was able to use what the kernel was giving it. So in a sense back uh in traditional settings you have croups that enforce limits you have name spaces that create boundaries and then arbback controls access. These mechanisms have proven reliable for CPU and memory for decades. But with GPUs it's different. GPUs breaks every assumption about resource isolation. There are no C groupoups. There's no

preemptable execution. No kernel visibility. The GPU driver manages everything. And then Kubernetes gets to watch from the sidelines. >> And with the rise of AI use cases, I mean like look how many AI talks we had this uh past couple of days and we have today and tomorrow. Uh there is a massive uh growth in demand for GPUs. Um everybody wants GPUs. Everybody wants to do AI

stuff. Everybody has to do AI stuff. Everybody feels feels left behind if they don't do GPU uh GPU stuff. So uh yeah, we have AI interest going up. Uh there was a recent study by data dog they published like two weeks ago. They've seen GPU instance minutes go triple over less than two years. Uh vastly outstripping the the growth in regular instance usage. GPU prices are also

going up correspondingly. I mean you all know this. It's pretty pretty big issue. So, uh, that's a problem because they're often underutilized. Like regular CPU and memory is often underutilized, but when GPUs cost so much, it's actually a bigger problem. Um, like in the same same data do study, they found that for uh CPU it's um the average utilization is 25%, for memory it's 50. Uh, for

GPUs, they didn't have the metrics, but it's probably something horrible like this. So GPUs being more expensive and less available like it's hard to get the top-of-the-line stuff today. That's a big problem. Uh so we need to do better for them really. I mean we should. So depending on your hardware, cube setup, workload types, there are a few approaches to doing this. Scott, >> right? So first

we're going to dive into MPS or multiple process service. It's the most straightforward way to share your GPUs. And I'm starting here intentionally because it requires the least from you, right? There's no new hardware, no major Kubernetes uh configuration and you're essentially unlocking sharing that the GPU is already capable of. To understand why, it helps to know what normally happens without MPS. Each CUDA process creates its

own context and it creates that context on the GPU and the driver then has to switch between them which is similar to how a CPU context switches between threads but with more overhead. MPS eliminates that by creating a single shared CUDA context as you see here on the screen where all the processes are essentially in the same pool. So they rise and fall together, which means you

have the risk that if one process runs away with things and fall and pulls things down, everybody gets to go down with it. And so there's some definitely some downsides to having a shared context, but also you get speed. You get speed and you also get speed in an easy less friction sort of way. So enabling MPS is uh also straightforward. You can turn it on at

the node level and Nvidia's device plug-in stack supports it. It fits into your existing Kubernetes setup without any friction. And but if you notice as I mentioned before for that simplicity, what are you giving up? It's isolation. If you have processes that are going to be conflicting, if you have workloads that are not trusted between different clients, that may be a problem. As I mentioned before, we

all the processes are in that same CUDA context, which means they sync or swim together. The memory isn't partitioned. So, one process can exhaust the GPU memory and starve the others. And if a fatal CUDA error hits one process, it takes down everything uh sharing that same context. So, in so for inference workloads where you can trust all the different workloads that are happening, that might not

be a problem for you. But if you're in a situation where you're providing, say, hosting for multiple customers and you have Coke on one end, you have Pepsi on the other, probably don't want to be mixing workloads between those customers. So as we move through the different strategies today, I want you to be asking yourselves, how much isolation does this approach get me, and at what cost?

In this case here, MPS sits at one end of the spectrum. low cost, low isolation, but everything else we're going to be talking about is at a different point on that trade-off curve. So, next we have MIG or multi-instance GPUs where MPS leaves memory uh unpartitioned and shared. MIG draws hard lines at the hardware level. It's dedicated memory and dedicated processing and dedicated compute, real isolation. So

you don't have that concern that you would have with MPS. But as I mentioned, it comes at a cost because it's at the hardware level. MIG works by partitioning the GPU's uh physical hardware, not just the software. So each slice gets its own dedicated memory partition, its own uh compute engines and its own memory bandwidth. There aren't soft guidelines that it can that you can configure. They're

actually hardware guidelines that are part of the GPU processor itself. and established by Nvidia. So Nvidia structures the MIG partitions in two layers. First there's the GPU instance which is the which is the GI. It carves out the physical resources. Then within that you create a compute instance which is the CI that determines how the compute engines inside the GPU are further subdivided. So the combination of

the GI and the CI give Kubernetes ultimately what it needs to make a schedulable device. For example, on an A180 GB, you can create seven equal slices each with say what's called an 1G 10 GB profile. That gives you seven independent partitions or seven independent devices you can use that all have 10 gigabytes each. And you can also mix in other profiles. So if you have other

workloads or processes that need more, so you say you have a 2G two uh 20 gig profile or a or a 3G 40 gig profile, you can mix and match these depending on the resources that you need. However, you want to make sure that you're allocating those slices to what you need and not wasting them. In the cases that where you have a process that maybe is

12 gigabytes and then you give it a 20 gigabyte profile, you're going to be wasting those 8 gigabytes. H100s follow a similar model as the A100s. However, uh definitely want to look into the documentation there. Even with A100s, right, I'm only telling you a couple of the different profiles that are available. Look into that documentation that Nvidia provides for each of those processors. But now you have

an idea of what you're looking for. You're looking for profile sizes so that when you're using Meg, you can say, "Okay, I have these different slices and I know what type of profiles I'm working with and then I can adapt my workloads for that." So trying to drop some like keywords and key knowledge and be like, "Oh, right. This is what I should be looking for when

I'm working with these types of processors. So because each MIG slice is fully independent device uh from the OS and Kubernetes perspective, you can schedule them exactly as you would a whole GPU uh with the fractional resources. Your device plug-in reports them as separate allocatable units and then the pods uh request them by profile name and there's no shared context and or memory space to worry about.

So the isolation guarantee is also stronger than anything that software can offer. A crash in one slice doesn't affect another one. Uh the memory access of one doesn't you can't access the memory of another slice. And so that makes me MIG a really appropriate solution for multi-tenant environments especially with regulated workloads where that you simply can't tolerate any kind of mixing uh of those of the workloads

or at least the noisy neighbor problem that you have when you all are in that same CUDA context. Uh the fixed partition sizes should be looked at though because as I was alluding to before, if you have like a you can't create like custom profiles. Those are set by the the GPU manufacturer. So if you you can't say I need 15 gigabytes. Well, if there isn't a

15 GB profile, you can't just configure that uh for yourself. And so your workload sizes need to roughly match the profiles that you have available to you. So reconfiguring partitions is also it also requires taking the GPU down when an MPS if you're going to make changes you can make them on the fly and keep things running for all of your users and all of your processes.

But in this case with MIG if you're going to reconfigure things and pick up different profiles and drop them then you're actually bringing the processor down and then and then bring bringing it back up. Now time slicing is where you have multiple workloads taking turns using the same GPU by rapidly switching access in short time intervals. So unlike MPS where you have processes that share a single

CUDA context, time slicing gives you each process its own full CUDA context just as if it had the GPU to itself because in a sense it does. It's like each process is essentially taking turns. You have our multiple processes here and they're all lined up in nice sequential order. They all wait in line very patiently for their allocated time and when they're with their allocated time they

get to start they get to stop and they can go uh in order. Now this comes at a cost because in some situation or in every situation every process needs to be written to memory when it's done and then read from memory when it's kicks back up. Also, if it's if it tends to happen if it inspire, excuse me, if it expires in mid computation, say if

T1 wasn't quite done with its processing, but its time aotment expired, it still gets cut off in mid computation and then gets to wait for the full rotation of the next time T1 comes back around. That wait time might be a couple of seconds, might be a couple of minutes, it might be much, much longer than that. T1 has no control over what two, three, and four

do. So, it just gets to wait for that next time in line where it gets to come back around and it gets the GPU. As I was mentioning, the switching itself isn't free. Uh, so saving and restoring as we have already talked about uh to that CUDA context means that you're flushing inflight work, you're writing state to memory, you're loading it at the next context before that

next work can begin. So the more processes sharing on that GPU, the more time that gets consumed by that overhead rather than actual computation or your actual work that's going on. So NVIDIA exposes time slicing through its device plug-in by letting you declare a GPU as multiple logical replicas. So a single A100 configured with four replicas appears to Kubernetes as four schedulable devices. pods request one of

these logical GPUs without any awareness that they're sharing physical hardware. So the illusion is handled at the device level or the driver level. Time slicing lands in an interesting middle position from what we were talking about before where in MPS each process or it has better isolation than MPS because each process has its own context. So a crash doesn't affect its neighbors. However, you're still sharing physical

memory uh just like an NPS. So there are no hard memory limits per process and unlike Meg, the GPU isn't doing any hardware reinforcement and so it's all coordinated in the software by the driver. There are some third party scheduling projects that also enhance these different strategies that we have. They're they run on top of these the strategies we've been talking about. So some of these what

they do is they don't uh do the sharing per se but what they do is they add scheduling logic on top. So they add fairness policies priority cues and quota uh quota management that the best or that the the uh base Kubernetes scheduler doesn't handle well for GPU workloads. what they do is they they change how the workloads get queued and assigned and then those uh for

those shared resources that come in. One of those projects that in particular is interesting is Kai Schululer. It's it's interesting because it handles something that's kind of a problem with Kubernetes in that it can only do things with GPUs in integer segments, right? It can only handle an entire GPU. Kubernetes doesn't say I can handle half the GPU. Kubernetes like it's all or nothing. I can handle

one, two, three or four or five. All integer based, not uh fractional based. So what Kai does is Kai says that's fine. You handle integers. I will reserve the entire GPU and then I'll use my own annotations to say I'll take that GPU and then you interact the rest of of the pods will interact with that Kaiuler and then I can use Kai annotations to say now

I'll take a percentage or a fraction of this GPU to make it so that I can start divvying out processing and fully utilize this GPU and utilize the hardware across multiple processes and multiple tenants. We already did that. So this means that you can express GPU fractions in your podspecs using Ka's annotations and then the scheduler handles the placement logic to fit the multiple workloads uh onto

a single GPU without Kubernetes ever really needing to understand the fractions. But here's the problem though. So even though you have a sophisticated scheduling logic in place, look at what happens with these two tenants. I have a Kai scheduler that's running at the cluster level and I can have tenant A that has its own cluster and tenant B that has its own cluster because we don't want

to have either one messing with each other as far as noisy neighbors and taking over resources. However, tenant A is barely utilizing their GPUs. Tenant B is max maxing out their GPUs, kind of wishing they had more GPU to process with. So in this situation here, Kaiuler can only help out so much because it runs at the cluster level. How it best works out is for Kaiuler

to be part of a cluster that has as many GPU nodes associated with that cluster as possible. And then it can schedule out the workloads that are going into each of those GPUs. And it can say, "Okay, great. Now I'm I have access to all of the GPUs and I can even out the distribution of the workloads and make sure that I can ma I can utilize

these this hardware but also in an efficient manner in this situation. You notice now the tenants are all in the same cluster. And the way that we're going to isolate those tenants, the most common way is through namespaces, which name spaces is a way to definitely make sure that you have uh isolation between the different tenants in in a particular cluster. In this environment though, each team,

each workload, they get their own namespace with resource quotas, arbback controlling what they can see and consume. And for many clusters, it works fine on paper. Uh tenants are logically separated and can't access each other's workloads directly. But namespace isolation is a soft boundary. It lives entirely in the Kubernetes control plane. The underlying node, the kernel, the GPU driver, those are all still shared. Which brings us

back to the noisy neighbor problem. One tenant running a memory hungry workload can create pressure that ripples across the other tenants on the same node. Even if Kubernetes thinks they're isolated, namespace boundaries are not going to stop that. >> So here here's where where we land. We have Kai doing the smart GPU scheduling and the fractional sharing. Uh and we have name spaces, cube namespaces that provide

some logical isolation and some separation of concerns. But there is a gap between logical isolation and full actual proper isolation of physical stuff like GPUs. Um and not only physical stuff also logical stuff. For instance, if we have admission controllers, operators like Kai itself, sometimes we have conflicting dependencies. Some team might need an operator in a specific version, some other team might need it in a different

version. But operators are usually cluster scoped, not namespace scoped. we can have some challenges there. Um so gaps like these are where things start to get complicated to manage and expensive to handle. Uh so either in wasted resources or in conflicting um the management of the conflicting dependencies. So this is where vcluster comes in. So vclustluster who's familiar with vcluster anybody use vcluster it's a project that

uh was created by a company called l labs uh open source essentially way of providing virtual clusters inside of a kubernetes cluster. It uh it creates fully functional virtual kubernetes clusters that run as workloads inside of an existing cluster and they give each tenant their own isolated control plane without needing separate uh infrastructure. So each tenant has their own control plane including an API server, their own

data store, and even Oh, someone should have said something. There we go. There we go. So they get their own API server, their own data store. And the secret here is that V-cluster syncer, which has that arrow going down through it. That v-cluster syncer is the bridge between the host cluster which is interacting with our GPUs and our virtual cluster which is what each of our tenants

are using to essentially isolate themselves from the other tenants using those same GPUs. That vcluster syncer acts as the scheduler essentially scheduling the different workloads and the different jobs that are happening but also it's syncing the resources in between that virtual cluster and that host cluster. you specify which resources you want to sync across the different clusters. You can say here we're going to have we're going

to set standards on that host cluster and say we're going to use these particular resources for all the virtual clusters. And so they all the virtual clusters get to essentially adopt what's going on in that host cluster. But then each virtual cluster has its own essentially environment. It is its own cluster. So they can install their own CRDs, their own resources, uh own custom operators and manage

their own arbback rules without really affecting any of the other tenants or the underlying host cluster. Each virtual cluster just sees itself. It doesn't see anything outside of it. Although from the management perspective, when you're managing all the the clusters, you can see them uh at the you know the vCluster platform level. So it's interesting how you can have an isolate environment that is isolated for the

things you want but not necessarily for all the things. So that brings us to three uh no four I'm I suck at math uh fundamental truths or problems. Let's see. So GPU sharing only works when you understand your and namespaces provide the lightweight approach to multi-tenency and isolation but they actually fall short in actual isolation autonomy and compliance and problems that only become more acute when we

have stuff like GPUs or conflicting operators and if we end up creating a separate cube cluster for each team we address the isolation part pretty damn well but this cost this costs us a lot in fragment mentation management overhead and efficiency use of actual resources. So not great and if we create multi team clusters it's great for efficiency utilization uh management overhead but we lose the isolation

again so too many imperfect choices what can we do here Scott? Well, for starters, we could use vcluster for real tendency with per team control planes on shared clusters. You have strong isolation without the cluster sprawl and then you can utilizing that large cluster that can effectively schedule the workloads across the GPUs using one of the scheduling libraries like Kaiuler. So, VCluster bridges this gap by providing

lightweight fully isolated uh virtual control planes inside of a shared host cluster. It allows each team to run its own Kubernetes environment with autonomy and then combine that with the tools we've been talking about. You combine it with the KAI scheduler and the fractional GPU support from the Nvidia operator. VCluster enables the organization to maximize their GPU utilization while ensuring fairness and compliance. It also delivers strong

isolation without the inefficiency of managing dozens of clusters that are isolated and spending the time to underutilize essentially underutilize those GPUs because remember if you have those isolated clusters that are separated they each are tied to the particular GPUs. So now we have a team shared um but specifically well but properly isolated cluster for each team. Uh so that's a pretty good foundation part of it. Uh

what else do we need to have a proper secure multi-tenant setup? Keyword is secure. Enter vote. And why vote? Well, I know I mean it's a boring topic. Everybody loves it, right? Everybody knows security is super important, right? Okay. I I I hope I hope. Okay. >> Yes. the the correct answer is yes. >> Yeah, but it's serious topic. Uh so yes, uh we are talking about

GPUs and sharing GPUs and efficient utilization of resources, blah blah blah, but security is always important no matter what we do. GPUs don't exist in a vacuum. I mean, they're cool. I don't know if you saw at the Metastan, they had a nice server with some Nvidia stuff. It's cool to look at. It's cool to own. It's nice. But we use them. We pay the exorbitant prices

either from cloud providers or to buy them from Nvidia because we need them. We want to do stuff with them. Uh stuff like inference or fine-tuning or training our custom models if we're really rich uh or running AI agents doesn't matter. Whatever it is of those things that we do, we need to consider security and how we're handling stuff like secrets, brokering access to stuff like the

production database which has personally identifiable information. All of this needs to be taken into account, needs to be designed for and not just hard-coded credentials pushed to GitHub, something leaks, somebody does something horrible. Um, so this is all very important stuff unfortunately because it's not I mean for some people it's fun, not necessarily the most fun, but it's important. Um, and yeah, when running in cube, everybody

knows cube secrets are not really that super secret. So, we need something better. So, that thing better is vote. Why vote? Uh, so at its core, it's an identity and secrets broker. It allows you to exchange a human or non-human identity for a vote identity, which allows you to access stuff like secrets. Um, those identities could be any number of things. In a cube environment, there would

be your cube uh jolt token which provides information about who you are, what service account, in what name space, in what cubeer, virtual or physical cubeer. Uh for a human that would be active directory, LDAP, for a cloud VM that would be AWS IM or whatever doesn't really matter to Um, and once we're authenticated to vote, we get the vote identity which allows us to exchange it

for a more neutral one. Why would we want to do that? Most common example is MTLS. So this is mutual TLS. This is unique. Uh, so each the client and the server provide the TLS certificates with their specific identity. They match. They're okay. They're happy. So this is great for stuff like agentto agent communication because each agent is uniquely identified and each other agent uniquely identifies the

other agent. So it's super secure, pretty nice. Uh but there's also spiffy j and etc. So that identity could be useful the more core part of what let's say is brokering access to secrets. Uh so for instance we can have a fine-tuning job which accesses the production database. We can have a uh specific small model which we train which uses the gets access to the S3 bucket

which contains all the images we use for training while the non-production stuff would only get test data. And the cool part about those different types of credentials is you see the first one it's the most boring. It's a username password. are the stuff that people keep in GitHub or kass hopefully keypass not GitHub but the more interesting part is the latter part so um dynamic credentials what

is this um so credentials are necessary like for everything we need usernames passwords identity tokens whatever uh but if they're static and live forever that could be a problem because if it's a password somebody created 50 years ago nobody knows who has it, who has written it on a post-it, who has committed it to GitHub, who has given given it to their open claw, we don't know.

So having credentials which are short-lived just in time generated for us for specific use with a specific access and then expire in like 24 hours or 1 hour or 10 minutes. It's pretty nice. Minimizes exposure risk and it's a good practice, let's say. and uh with vote we also like it's a very boring thing but when we have our credentials which are dynamic shortlived it's nice to

know who did what when and why. Uh so every action in vault is audited written down this user or this cube bot requested the database credentials to access the production database and then you can trace what happened in case of a leak. Um pretty So in a cube environment this is how things look like more or less. Uh we have vault which gets us access to the

secrets like database credentials and uh we have the vault secrets operator which is deployed and which synchronizes access uh synchronizes the secrets that we need into cube secrets and because they're shortlived that's not a bit big of a problem because even if it leaks it's worth for one hour. So exposure exposure risk is minimal. Um, and the V secretress operator also handles rotation like when we revoke

the secret or when it expires, it will change it. And it can also redeploy the pods if that's needed, which usually with like uh AI jobs might not be necessary. I mean, sorry, might be counterproductive because you don't want your uh two week training job to be killed just to change a secret. But for an AI agent, it's So wait, how do we handle multi-tenency to make

sure nothing gets shared by mistake? Because we don't, you know, we don't want our emails to be deleted by OpenCloud or anything. >> Yeah, that never happens, right? >> No. >> Um, >> never. >> So how things work in vote in vote everything is like it has a very robust uh ACL system kind of like the cube one where deny by default nobody gets access to anything

unless we explicitly give it access to something. Uh and with vault enterprise we also get name spaces which are logical boundary which allows us to separate stuff secrets themes groups users whatever. Uh so we get the single highly available vault deployment. We can get multiple separate isolated mini vault uh environments and the isolation is good for blast radius because yeah we'll never have the secret being accessible

from the wrong team or wrong deployment or wrong cube cluster. Uh but also it allows us to do cool stuff like um automatic templating. Every cube cluster every cube namespace automatically gets access to their corresponding vault namespace. So we don't have any setup to do automatically. The vault production cluster in AWS has access to the production data in AWS and the training on developers laptops never does

get access to anything. So it's cool stuff. So uh this is the theory how it works uh in practice is like this. It's relatively straightforward. We deploy uh the operator in u in the namespace or in the vcluster and uh then we have CRDs. The first one is pretty basic. It's just votes address. It's there. It's easy. Uh then we configure an authentication and in this case

we're using uh the bottom. So we have a service account called finetuning uh which is going to connect to vault on the production row and yeah that's basically it. Once we do this we get access to stuff in vault. Uh in this case the stuff is a dynamic secret. So secret that vault will create for us and will manage the life cycle us of and the secret

is a posgress crow credential. So it's going to create a possible username password and it's going to manage it for us. And the same CRD works also for cloud credentials, for PKI certificates, for MySQL databases, for MongoDB and for a bunch of other stuff. Pretty much everything we could want. And this only works if we actually have access to that possible credential. We don't give it to

everybody of course. So how it looks like is we have to create a row in vote which just says which service account from cube and which from which namespace is allowed to access that thing and that thing is the postcode credential and that's it. It's relatively straightforward to give specific isolated accesses to specific isolated things based on what they actually need. So remember vCluster gives us real

tenency and isolation but now we layer this multi-tenant service. So we have vault for secrets and then GPU sharing or scheduling with the NVIDIA operator the scheduling with the operator plus Kai and then then we turn that foundation into a secure and fair efficient platform. >> So a couple of notes to to finish off with uh good things to remember. So uh complex systems fail. Everybody knows

this. So there is no excuse not to build with complexity and failure in mind. Everything will break. No matter how hard you do it, no matter what your foundations are, something's going to break at some point. So uh the difference between a a good and a great platform or SER or DevOps or whatever you call them team uh is uh having playbooks and having plan for this

kind of stuff. And part of architecting for failure is architecting for isolation. You don't want one team's failure or one application's failure to impact everybody or and everything. And isolation only works because it adds extra defensive layers. Uh more layers equals more protection. More protection Building on what Adrian was saying, defense doesn't just come from the outside, right? For incidents, it also comes from the inside. VCluster

actually uh has virtual clusters that help you contain that blast radius. So if something goes wrong in a in a particular virtual cluster, say a bad commit or a misconfiguration, it doesn't get pipherated poriferated peripherated across everyone using that same host cluster, right? Each virtual cluster gets those isolated so you can have that defense and limit that blast radius. Also, vcluster gives us a couple of different

audit layers or audit options. Essentially, at the tenant layer, every virtual cluster has its own API server. So, it has its own API server logging that's available in every Kubernetes cluster. And so, you can have all the different logging that happens there. But then at the platform level, you can have logging that essentially tells you what's going on in the in each of your clusters, who's creating

them, who's modifying them, but also making sure that they all are are running well together, giving you that data so that you can make those decisions. I used to say you can take that data and export it into something like data dog for you to like look at and analyze. But now, what are we doing, right? We're taking those logs and we're giving them to agents and

then we're having those agents decide what's going on and help us make decisions with our with our applications which in some cases works well for us or for all developers. So we appreciate you all coming in here today especially since we are the thing that's holding you from going and finding lunch. So I we appreciate your attention. Uh please if you'd like to see the slide deck

font should be bigger speaker.comdmcallister. Uh otherwise we have a couple of minutes >> Thanks great presentation. >> Um sorry might be a n question but I didn't understand like how the is it namespace when you do namespace with guys schedule just a logical logical isolation. I understand that but how is vluster solving that problem because that is also logical isolation only >> what vcluster is doing is

in addition to the name spaces you put kai scheduleuler on that host cluster and then vclustluster is the thing that's keeping each of your tenants isolated from each other so you're basically giving stronger walls than just what namespaces we're providing to each of the tenants that are sharing those virtual clusters >> okay but that's also at the logical level only right >> so like I said like

name spaces can happen, you can make it work and you can configure it. What vcluster is doing is it's essentially uh making that easier for you to do right without you having to keep all the different uh permissions and things separated and keeping those managed as well. >> Okay, got it. And then you had another point saying like >> it might be a problem for operator because

they're cluster scoped. um you can have like uh uh operators run in CPUs instead of GPUs, right? Because operators are not going to be typically CPU intensive, >> right? So operators can run uh the question was you you can have operators running again CPUs and with GPUs because they're not scoped to particular processes. Operators are gen generally scoped at the cluster level, right? So that you if

you have an operator that is uh op you know benefiting the entire host cluster then yeah then you would put those operators there in that host cluster. If you have an operator that's specific for a specific tenant then you would want to have that operator running in in the virtual cluster. >> great presentation. I'm wondering how dynamic can like the scheduling be or like if I have

a GPU or I have a process that's being used for inference. So I'm hosting like a large model that's using multiple GPUs. And then I also have another team that wants to run experiments and train their own model, but they don't really need to have the whole GPU. Is there something where you can like temporarily pause some uh one of the instances of like not allow inference

for a little bit or if someone then wants to put in a request for inference to like pause at a step of training? So the question was yeah can you uh essentially dynamically decide if if it's being utilized can I spin something down and then if this and or if it you have a large workload come in can I dynamically spread it out kind of like in

my head thinking of like how um like hypervisors work like right like with like with EC2 or something where it's basically saying you spin up servers when you need them spin them down when you don't that type of of questioning. I don't I'm not an expert as far as exact specifications of the different scheduling pieces that I was talking about. Uh but as far as like using

them as what they do is like something like Kaiuler makes it possible to do that. It makes it possible to provide fractional portions of those processors. But how you would dynamically like could you send that uh like configuration of saying hey we're going to use 02 of this GPU and then like wait this is getting bigger now I need to use 75 of this thing or one

or two of these things. Um I don't know how you would do I I don't know how to do that is like is if that's built into theuler itself or if that's something you'd have to kind of put logic around in your configurations maybe using something like Carpenter something like that. >> Yeah. Thank you. >> you Yeah, >> I don't have the mic. So, >> you don't

have the mic. Uh, you're saying Kai? >> Kai has priority based into it. So, yeah, you know a little bit more about Kai than I do on that particular piece. >> Sorry. >> Yeah, sorry. just uh wanted to say that yeah Kai has prior you can configure priorities in Kai. I don't know if you can tell it to evict a lower priority workload or pause it or

something but in any case yeah they have priorities so probably that's why they are there. Uh but yeah that would be something to to check out. >> Thank you. Sorry. >> Um thank you for your presentation. I I'm kind of behind so what um not the name space but the other one. What's the name of that? I I can't remember that. >> Uh Vcluster. >> Yeah. So

what if one if the Vclustluster fails or dies? What happened? Because you know things do fail. So does it means another one comes up? Um >> as as Adrian mentioned, right, we should always architect for failure. Yeah. >> Is that what it does? >> So in this in in the case here, are you asking if Vclustluster as like as a tool? So >> if if one of

the vclusters falls over, can I spin up another one? Yeah. >> Uh I mean essentially, yeah, there's a v-cluster employee right there, so he knows specifically, but yes, they they definitely spin back up. They're they're just like any Kubernetes resource that which is one of the great things about >> It it makes it possible so that my resources are going to be running and if they need

to be restarted, it'll restart them for me. That those types of things. So yeah, VCluster will restart their virtual clusters when they need to. >> Okay, thank you. That's great because you know as a cyber security professional I'm always thinking about if something fails. Now I do want to ask you about >> um again I'm behind what how does this vault dynamic secret I I assume that's

from Hashi Corp. Is that right? How does that really work? The the dynamic secret thing. Can you explain a little bit? >> Oh yeah. Grab. Okay. I thought I'd be closer. >> Thank you. Uh so yeah it's a vote from hashikorp and the way it works is uh you give vote access to manage credentials in a database. So for posgus Crayo you need to give it kind

of an admin account and then when you request um uh readonly user for the production database vote to connect to the postgus trail create the account with the specific grant and give you the credentials for it and with a specific time to live so like 1 hour or 24 whatever is configured and when they expire vault is going to connect to delete it so even if you

still have it it no longer works and that's the same logic used for AWS, Elastic Search, MongoDB. >> Yeah, it's a Yeah, with the DTL which vault manages itself and V will delete for you. So you don't Yeah, even if you just run a test, you don't have to care about deleting the credential afterwards. And then I will say about the comment that you're behind, I wouldn't

say anyone's behind, right? No one knows all the things about all the things. That's why we're at conferences, right? That's why everyone's here is like I'm trying to come see what people are talking about. So I think having the courage to to ask the questions is exactly the type of mindset you need to have to to learn. So well done. >> Does a vcluster have its own

secrets management? Why wouldn't or or not? I mean do you have to use vault or can you use vcluster? >> It so vcluster is just the it provides the the uh the engine for you to create virtual control planes. So any any Kubernetes cluster so if like a Kubernetes cluster needs to have a secrets manager part of it or you can use secrets inside of that Kubernetes

cluster right like you can use secret objects uh in there so if you whatever you use inside of a regular Kubernetes cluster you can use inside of vcluster as well so it's essentially the same thing it's just providing it yeah on a much more finite level >> and I think one of the things that she brought was like there could be failures that could be control plane

failure or an agent right but if the control plane is off the agent are. >> Yeah. So basically if the control plane dies everything is running and depending upon how you are storing your data control pane picks up again but everything will be keep on running till till your control plane is not going because the architecture is as such that your API server is running as a

pod. So if the pod fails it can be spin back up but your workloads are running. Please. >> So I'm always I have this question. So the thing die and this thing comes up vcluster. So it's great, right? Magic. I mean I like it. But are we losing any kind of data or how does it sync up? This is have always been a mystery to me. People

say well it just comes up and so is there's nothing lost in the trans this kind of thing. Is it back up? you you're doing a a sync all the time to make sure everything is >> so data data itself isn't like that that that's getting in into the scope of of talking about Kubernetes as a as a concept itself data itself is going to be saved

into things like persistent volumes right or state or state or stateful sets right and so those you want to be uh saved but you yeah you'll be syncing those things as well >> and to add like whenever you are creating doing any objects. So the step how it works is the like when you create first the data store is saved in net CD and then things are

created. So the source of truth is there. So it's not the other way around. So it keeps on syncing between the API server and cityd. So that's the truth there >> and then you can if you're thinking about the CD failure you can design your control plane in a manner that you never have that issue either. Right. Right. >> Of course. >> Here we go. >> You

better watch out. This kind of >> golden coups or not? >> Just so you know. >> Not a bad one. But I heard something about vault that I'm curious to know because immediately triggered me in my brain the hacker mind. So you said that vault is going to connect to the database and delete that credential. When you did the presentation I was thinking that the credential was

going to expire somehow. So if vault has to go there and delete the credential and I impede that creating a network policy in example on the cluster and avoiding vault to connect to that database and I steal that credential. Will there be a way a better one for vault to be managing these credentials with an agent or something that is effectively running inside Kubernetes because I know

that vault has an agent also and you can use that. So I'm curious about that process that you mentioned. Vault is going to connect to the database. Could you give me a little bit more details on what you do there? Okay, thank you. Um, so yeah, it's vote the servers that have to connect to the database to be able to delete the credentials. And indeed, if you've

cut the network access from vault to the database, it would not be able to do so. It will keep retrying and it will keep a list of all credentials it could it did not already delete that it should have. So, it will keep retrying forever. Um but yeah it needs the right access and this is only handled at the vault control plane level. You cannot use the

agent for that you yeah it's because it's a so the control plane is a distributed cluster like cube itself. So all the decisions are there because it ensures good reliability and fall tolerance and etc. So yeah, not really. >> What are the key uh differences between storing uh the credentials that we talked about earlier um in a git versus uh vault. >> Okay. So um if you

store the credentials into git, they are forever there in the commit history. So you always like even if you if you say to yourself, "Oh, actually that was a bad idea. I'm going to remove them." Uh they're still going to remain there in the history. I mean you could remove them, but it's really not easy. Uh and the thing is that usually a lot of people have

access to git. So for instance, all developers working on a repository will have access to the repository. So they will be able to see all the secrets. They be able to copy those secrets like to their laptop to to run a test or a dev environment. So you lose control. you lose any even knowledge of who has those secrets, why and who has used them, when and

why. Um, and yeah, everybody who has access to the code will have access to the credentials, which especially in a production environment, you don't really want that. Yeah, you trust your developers to write code, but you wouldn't trust them necessarily to have access to the production database. >> Uh, and this is not just developers. Like there was an example we were talking about yesterday. Um some guy

was complaining that he was using code uh code to write uh to update the version of the website and cl deleted of his environment using terraform. Just run a terraform destroy auto approve. So deleted the database, deleted the backups etc. And it's not a clone problem. You just gave it too much access. And it's the same logic you'd give only the necessary access. So a developer they

don't need access to the production database. uh cloud does not need access to your production database. So it's separation of concerns and knowing who has access to what and in vote all the secrets are encrypted which is also nice but also you have the very good arbback so only the things that you've authorized to access those secrets have them and you know who access them. So yeah

uh basically vote is better for the big team is what you said. Um, yes, but even for small teams, you need something better than Git. Even GitHub secrets is better than just putting them in Git. >> So for a small team, um, uh, good question. Uh, usually most platforms already have something planned. Like for instance, if you're on running on AWS, they'll have AWS secrets manager. If

you're you're running on Versel, they have a secrets thing. So look at what the platform provides. Look if it fits your needs and use that if it >> Um back to her. Um I just wanted to say as a cyber security profession now we do not recommend you put anything pass your secret in git repository because we heard attackers are going over they're able to see those

kind of thing. So definitely vault is a thing. I'm not saying has to be hash court vault. I'm not just it has to be in a vault. If this is something really important because I've heard so many stories they're looking at people's credentials in the git repository. Not to say if you don't set up correctly people can just go there and get it. Yeah. Just a comment

for you because we hear that over and over again. Thank you. All right, is that it? Awesome. Thank you all for coming. Enjoy the rest of the show, friends. Test test. That's That's Test test test. Hi everyone. I hope you guys are all ready for our talk today. I just want to warn everyone that there will be QR codes in this presentation. So, make sure you have

your devices ready. All right, let's start off with some introductions. My name is Angelica Ambrosio and I am a Devril engineer at.com. I have a background in platform engineering and teaching. I specialize in DevOps and AI and my hobbies are breaking down buzzword heavy topics and complex tech concepts into ideas that engineers can actually use. And with me >> Yeah. Okay. Let's test. There we go. Um

Christian Hernandez um from the uh from Cisco uh on the asalent team. I am um background is platform engineering uh DevOps uh kind of a I used to call myself a Linux command line uh samurai. So um uh back I for the past I don't know 10ish years I've been really into cloudnative Kubernetes. I am a contributor to the Argo project. I am a contributor to open

githops. I am an LA native. It took me an hour and a half to get here. So that's how I'm an LA native. I always joke that I can fly to San Francisco, get there quicker than I can uh to this conference. So um so uh yeah, so that's a little bit about me. >> All right, before we dive right in, I want to give a little

bit of background on you.com and the team behind it. So, UD.com has been driving major advances in AI and search long before the current generative AI wave. So, our founders, Richard and Brian, have been pushing the frontier of AI, natural language processing, and LLMs for more than a decade, long before the current wave of generative AI. Their research at Stanford helped pioneer deep learning approaches to NLP,

including work that leveraged Nvidia GPUs to scale neural networks for language understanding. Over the years, this work helped drive major breakthroughs in contextual language understanding and search, fundamentally changing how machines process and retrieve information. So, at ud.com, we've continued pushing that boundary. We're among the first to actually fully connect large language models to the live web, enabling AI systems to access real-time knowledge instead of relying solely

on static training data. Now, we've continued building innovations around AI powered search and deeply personalized AI experiences. All supported by a full web API that developers can integrate directly into their applications. So at UC.com, we work with many of the world's leading AI teams and our APIs are built to handle massive global traffic. Today, we're serving over a billion queries every month, supporting thousands of API customers

globally, and we're seeing strong adoption across the enterprise, including over half of the Fortune 500. And that's the experience we bring to the space. So today I'm going to show you how you can build AI applications more smoothly and securely with Cisco. >> Yeah. So um so just a little bit of background about isovalent. Um so isoalent uh so Cisco acquired isalent maybe about a little over

a year ago and uh isovalent was really focused on networking and security in cloud native ecosystem. So basically taking all like the network security aspects that normal uh uh network operation folks care about and bringing those to Kubernetes because let's face it just because you're moving uh your workloads into Kubernetes doesn't mean your old security uh practices doesn't also apply there. So that's kind of like the

idea around isovalent and all the tools um that isalent built uh in the open source ecosystem and uh now we're part of Cisco to um and basically became that kind of cloudnative um oss branch of uh of Cisco. So um go ahead next slide. >> All right. So now that you know a little bit about us experts, we're going to tell you about the agenda for today.

So we're going to talk about the rise of AI. Why AI is breaking traditional infrastructure, three common mistakes with AI infrastructure, how to build the right foundation for AI workloads, how to enforce security and compliance, how to add live data to your AI, and then we'll do a highle overview, how this benefits you, and we're going to close everything off. So, let's talk first about the rise

of AI. AI adoption is accelerating, but infrastructure readiness is lagging behind. So, you want to adopt AI. As more enterprises adopt AI, they quickly learn that traditional infra isn't cutting it anymore. Your infrastructure needs to evolve in order to handle AI. AI workloads demand new approaches to compute, storage, and security. Many make the mistake of neglecting their infrastructure needs by de blindly developing infrastructure without a proper

strategy. So, here are a couple of challenges that can arise without proper strategy. Number one, the VM mindset. When teams think about AI infrastructure the same way they think about traditional workloads, they often fall into what we call the VM mindset, spinning up virtual machines and treating those resources as fixed units. The challenge is that AI workloads are fundamentally different. So VMs can be too rigid and

inefficient. Two, GPUs. Okay, let's be honest. GPUs are really expensive. Overprovisioning GPUs are poor return on investment. having idle GPUs just sitting there consuming power and budget. And then lastly, traditional infrastructure security often assumes isolated services and predictable traffic, which works fine for most enterprise apps, but AI workloads are built different. They're constantly moving data between the nodes. And then now Christian's going to talk about why

AI breaks traditional infrastructure. >> Yeah. So um as as AI workloads are landing onto um to our environments um there's there's some challenges that that that come with that right so like basically the ex uh the the extreme GPU demand right so as as Angelica said GPUs not only expensive but they also like cost in terms of like your environment right to the actual physical hardware um

and all of a sudden high performance networking is important, right? So, all of us, you know, now now you're now um you're you're measuring IOPS and and things like that. Um where whereas maybe before if you weren't uh doing AI workloads, maybe you weren't so concerned about like some of these things, right? Uh also security and compliance needs, right? Those never go away. Um you know, even

if you're running like, you know, the the newest of the new, the the coolest of the cool, the the latest things, right? And uh right now AI is the hot topic. So um but those compliance and securities that that that stuff never goes away and AI workloads will you know often exceed um VM design limits right you design over a certain workload now you're trying to shoehorn

in this newer workload into um a a system that may be designed for something else. >> All right. Have you ever wondered what goes on under the hood of generative AI? I have on screen a diagram of the architecture of a typical generative AI system. I like to think of it as five core layers starting from the bottom and moving up the stack. So think of it

like as a big cake of AI. So at the very bottom we have the data layer. This includes all the structured and unstructured data used to train, fine-tune or ground models. For most AI systems today, this could be documents, logs, databases or internal knowledge bases. The quality, governance and accessibility of this data directly impacts model performance. So keep this in mind. Above that we have the infrastructure

layer. This is where the compute lives. Your GPUs, Kubernetes clusters, networking, storage and your orchestration. This layer is responsible for scaling, training and inference efficiently across environments like cloud, onrem or the edge. And above that we have our LLM layer which is the core intelligence of the system. Think of it as the brain behind the operation. This includes things such as foundation models, fine-tune models, and inference

services. This layer handles reasoning, generation, summarization, or any other AIdriven capability. On top of your models, you would have your middleware and your API layer. This layer conducts models to the rest of the system. It includes things such as the orchestration frameworks, rag pipelines, vector databases, prompt routing, and API gateways. Essentially, this is the glue that allows applications to reliably interact with their AI models. Finally, at

the top of our cake is the application layer. This is what users actually interact with chat bots, search interfaces, automation tools, or enterprise applications powered by AI. And then we have on the right the the several crosscutting pillars that make the architecture production ready. We have LLM ops which covers monitoring models, managing deployments, versioning and evaluating performance. Second is user feedback capture which is critical for improving

models over time. Third, we have security which is very important which includes protecting sensitive data, controlling model access and ensuring safe interactions. Finally, responsible AI which focuses on governance, transparency and compliance. Together, these layers and pillars form the foundation for building effective, scalable, secure, and productionready generative AI systems. And today, Christian and I are going to talk about open-source software that can serve as a starter kit

to create the infrastructure or the foundation of your generative AI system. That means you can create the next chat GBT with this. But first, let's cover what not to do when developing infrastructure for AI. So, um, so we're going to go over like the three common mistakes, uh, that folks make. You know, this is kind of just generic things that we see out there, uh, when when,

you know, going into, uh, when you're first stepping into creating a, uh, AI infrastructure or an infrastructure ready for AI. >> All right. So from what we've seen working with teams building out their AI platforms, there are three really common mistakes that come up again and again. And here we're going to learn from their mistakes. The first is overprovisioning resources. Since AI workloads require a lot of

compute like GPUs, it can make teams a little nervous about running out. So this leads to GPU hoarding and the result is clusters full of idle GPUs. instead of dynamic scaling and efficient scheduling, you end up with large amounts of unused resources. And then the second mistake is a legacy infrastructure mindset. So think back when I was talking about the VM mindset, it's the same thing. Infrastructure

isn't one sizefits-all. When teams try to run modern AI workloads on top of traditional infrastructure, things are going to break down. You get poor alignment with Kubernetes data patterns, limited flexibility for training and inference workloads, and infrastructure that simply wasn't designed for the scale or complexity that AI requires. And then lastly, Christian will talk about security. >> Yeah. So, um, insufficient security, right, I think is is

probably one of the things. And, uh, we'll go over it a little deeper as as we progress here, but um, data leaks, right? like in terms of data, that's just always a a thing you need to think about and take care of um regardless of what workloads. But now when you're thinking about like sensitive information, people using AI to assist them in, I don't know, running uh

reports for uh customer spend, right? you're seeing sensitive data and you're seeing uh these AI uh leaks that can happen where you know all of a sudden they have customer information or sensitive uh internal information and uh network security um is always going to be something regardless of the workload but especially in AI when um different things you're saying things about like open claw and things like

that being able to uh access different uh resources network security even becomes even more important um in in terms of of AI. >> You don't know how many times we've seen people putting in customer like credit card numbers and phone numbers into chat GBT. This is really important. All right, so let's talk about building the right foundation. So infrastructure is foundation. So think of it like you're

building a house. You want a strong, sturdy foundation because you don't want your house to collapse, right? Or think of it as a big bridge. And this bridge needs to handle like a 100 pound 200 pound big rigs going through. That big rig is AI workloads. They're huge. They're they're resourceheavy. So we need to have a foundation that can actually handle it. So I I recommend using

Kubernetes for your infrastructure. Kubernetes is actually a really good solution for creating AI infrastructure despite not originally being created for AI. Scalable distributed workload orchestration is one big perk. AI workloads, especially like training jobs, often need to run across multiple nodes and GPUs. Kubernetes can make it easier to orchestrate these distributed workloads, schedule jobs across clusters, and manage them reliably at scale. And another good uh benefit

is the efficient GPU resource scheduling. GPUs are one of the most expensive resources in an AI environment. Kubernetes allows you to schedule and allocate GPUs across workloads. So teams can share infrastructure more efficiently instead of dedicating hardware to individual products. So that's like saying I'm going to have a GPU for this, I'm gonna have a GPU for that. No, you don't want to do that. Three, automated

deployment and scaling. AI services like model inference APIs often experience flunctuating demand, but Kubernetes can automatically deploy, scale, and manage those services. So applications can handle that spike in traffic without manual intervention. Four, cloudnative infrastructure flexibility. That means that you can run AI workloads across cloud, on-prem or hybrid environments. That flexibility is super important because many AI systems involve huge data sets, specialized hardware, or regulatory constraints

that require infrastructure choices beyond just one cloud provider. But even though Kubernetes wasn't originally built for AI, its ability to orchestrate compute, automate infrastructure, and scale workloads makes it one of the most powerful foundations for running AI systems in production. But can Kubernetes stand on its own? So can I just start creating infrastructure for the next chat GBT with Kubernetes alone? No. It was originally designed to

run stateless web applications and microservices, not AI workloads. Because of that, a few challenges appear when you try to run AI systems directly on vanilla Kubernetes. Because GPU scheduling isn't optimized by default, Kubernetes can allocate your GPUs, but it doesn't naturally understand AI workload patterns like GPU sharing, memory constraints, or scheduling long running training jobs efficiently. That could lead to fragmented GPUs or idle resources. Second, AI

workloads are often longunning and stateful. Training jobs, fine-tuning pipelines, and data prep-processing tasks may run for hours or even days and require checkpointing, distributed coordination, and fault tolerance. Kubernetes was primarily designed for short-lived stateless services. So, additional tooling is needed. Third, AI pipelines are complex workflows. Model training, evaluation, fine-tuning, and deployment are multi-step processes involving data pipelines, distributed training, and model versioning. Kubernetes itself doesn't provide native

orchestration for machine learning workflows or experiment tracking. Four, data movement becomes your bottleneck. AI workloads move huge amounts of data between storage systems, GPUs, and pipelines. Kubernetes doesn't automatically optimize high throughput data pipelines or distributed training communication. And then finally, this one's a little more obvious. Observability and cost control become difficult. GPU clusters expensive and without specialized monitoring, it's difficult to track GPU utilization, model performance, and

cost efficiency across AI workloads. But the good news is this can be optimized by adding the right management and automation layer. And that's exactly where Cordant comes in. Introducing Cordant. So Cordant is an open-source Kubernetes native platform by Morantis. It centralizes the management of your Kubernetes clusters across the cloud, onrem or an edge. It has declarative GitOpsbased automation for infrastructure and workloads. Meaning that you can start

streamlining cluster provisioning using templates. You can start streamlining your service deployments using a template. You have built-in policy enforcement, governance, and life cycle management. And it's designed specifically for these modern distributed and AI workloads. And the cherry on top, you get coordinant observability and fine ops, aka cough, not king of fighters, for divisibility into cost and resource usage. So let's take a look at exactly what cough

does. Cough is an observability and cost management layer built for environments running their Kubernetes-based infra, especially those supporting distributed and AI workloads. When organizations start running Kubernetes at scale, especially for AI, they usually end up stitching together a whole stack of tools for metrics, logs, tracing, you name it. This is what we call tool chain bloat. Each of these tools have its own config, op upgrades, and

operational complexity. What COP does is just simplify that whole stack. It bundles all your observability and cost management tools into a single subsystem that can be deployed consistently across all of your Kubernetes clusters and is all managed by a single control plane. All your favorite observability tools bundled in one such as Graphfana, Victoria metrics, open telemetry without the cost of managing them all individually. And from an

observability perspective, this collects metrics, logs, and traces from every cluster in your environment. That means platform to teams can monitor their performance, troubleshoot issues and understand system behavior across their Now at the same time, COP integrates fin capabilities using tools like open costs. This connects infrastructure usage directly to cost data allowing teams to see exactly which workloads, namespaces or projects are driving their spending. So we have

here up on screen, this is uh coffee with graphana. So all of your favorite observability tools are already included. You can use whatever you'd like. AI infrastructure is typically built around expensive resources like GPUs and large scale. Storage, training jobs, inference pipelines, and data processing workloads can run for long periods of time and consume massive compute resources. Without the proper visibility, your organizations are going to end

up with idle GPUs, overprovisioned clusters, or runaway infrastructure costs. Another advantage is consistency across their clusters. Instead of building and maintaining observability stacks separately for each environment, cough ensures that every cluster deployed with coordinant automatically includes the same monitoring, tracing, logging, and cost tracking capabilities. So in spend instead of spending all your time integrating tools, troubleshooting version mismatches or building crosscluster dashboards, platform teams get the standardized

life cycle managed observability and PinOps platform from day one. The TLDDR coff gives organizations visibility into both performance and costs which is critical when operating largecale GPU heavy AI So I know I shared a lot about Cordant. So let's summarize. Why is Cordon optimal for AI infrastructure? Cordon effect effectively makes developing AI infrastructure with Kubernetes an automated, lightweight, smoother process. Your MLOps lops can become super simple

and composable. And here's why. Number one, resource efficiency and GPU utilization are improved significantly. Instead of manually managing your clusters and risking idle GPUs, Cordon orchestrates all your Kubernetes environments across cloud, on-prem, and edge. Second, AI infrastructure becomes declarative and automated. So that means that we use G githops based workflows. Teams can manage their clusters, workloads, and infrastructure configs in a consistent way. That means training pipelines,

inference environments, and supporting services can be deployed and updated reliably across environments. Third, distributed AI environments become easy to operate. That means AI workloads often span multiple clusters or environments. Cordant provides that centralized management across these clusters which simplifies running largecale training jobs or inference systems without managing each cluster Fourth oops fourth platform consistently see improves across teams. Instead of every team configuring Kubernetes differently, Cordant provides

standardized templates, policies and life cycle management. This ensures AI platforms are reproducible, scalable and easier to maintain. And finally, observability and cost visibility through coordinant observability and PHOPS aka cough. This provides insight into resource consumption, infrastructure health and GPU usage and it helps orgs in understand where resources are being spent and optimize their AI infrastructure. So effectively, Cordant turns Kubernetes into a manageable scalable orchestration platform for

running those distributed AI workloads. So, I really like Cough because it bundles all of my favorite observability tools into one and I can manage them all from a single point of control. But I'm going to be honest, I like to use a lot of tools when it comes to Kubernetes. But this can be out of control quickly and I end up with tool chain bloat. How many

of you do the same thing where you just got to get like 50 different tools just for one project? Exactly. For those of you who can relate, you're going to love this. One huge benefit of using Cordant is the curated Cordant catalog. You can create multi-ervices which are bundles of your favorite tools and bootstrap them to your clusters via service templates. The catalog is very extensive and

makes it easier to strap on all your favorite tools without the additional overhead. If you look up the cording catalog, you can see the full list. And wait, what's that over there? Oh, okay. We're going to learn from Christian what celium is. >> Oh, yeah. I have this here. So um so I'm going to talk about uh enforcing security and compliance with psyllium. So um basically how

to keep your data safe and compliant with psyllium and uh and how you can use it in in the age of AI. So um before before I get to like the security aspect and enforcing I'm going to talk a little bit about like what is psyllium. So for those who aren't familiar uh with psyllium, I hope you're at least familiar with Kubernetes is that uh when I

say psyllium, when people talk about psyllium, it's actually talking about a few different things, right? A few different uh tool sets uh with psyllium, right? It's all under the psyllium um uh open source project, right? It all falls under that. Um but usually when you say when someone says psyllium you think about uh Kubernetes networking you think about um the CNI right because at at its core

that's what psyllium is right it implements the the container uh uh the container native uh interface for Kubernetes right to basically uh provide uh that flat network that you do when uh when you first deploy Kubernetes right um but I always like to say psyllium is a superset meaning like yes it does implement everything that the CNI does uh things like network policies and all of that

everything that kind of like what you would used to be using but more right because uh selium also uh gives you things like IPSec you can do things like BGP uh you can do things like IPAM um egress gateway things like that uh when you're doing network policies we actually go beyond level uh layer three and layer for right you can go all the way up to

layer 7 um and even do things like DNS which you're I'm going to show in in the next few slides here um then uh then with all of that everything that that Psyllium gets you uh for those folks that are like network operation folks everything that psyllium gets you you also get what I call the batteries included um observability platform uh with selium uh which is uh

you can use Hubble right we call it Hubble timecape uh you can use Hubble or you can plug in things like graphana um fluent bit things like that right uh as as well right you don't have to use uh the Hubble CLI and then also uh service mesh right so basically anything that's ingress gateway API so if you think about it everything from when traffic enters your

cluster all the way down uh to the c uh to to the actual pod to the workload selium takes care of all of it um and then I then there's this tool called Tetragonon. So Tetragonon um is based on and so is psyllium but tetragonon is based on ebpf. So for those I aren't familiar with ebpf. I can actually do a whole two-hour presentation on ebpf. But

in in kind of a nutshell you write these um I guess kernel programs that execute on runtime that you can set certain policies on. So when let's say someone tries to write a file or read a file off the operating system, you can set policies around who's allowed or what kind of workow workload is allowed to do that um and either block it or mitigate it somehow.

So um so if you think about like this whole platform as like a kind of a whole thing from when the traffic enters uh your Kubernetes clusters to all the way down to when the disk action happens when stuff happens in the kernel you can manage all of that um uh using psyllium and and tetragonon. So, psyllium in a nutshell, tetragonon in a nutshell, uh if there's

other talks around there around psyllium and tetragonon, I or ebpf, I encourage you to go uh go attend those talks because now I'm going to talk about how can we secure AI systems against evolving threats. So, the MITRE adversial threat landscape for artificial intelligence systems, that is a mouthful to say, so I'm just going to call it Atlas. So the Atlas framework uh is a knowledge base

that documents tactics, techniques, um procedures used against uh machine learning systems and it helps security teams understand how ML models can be targeted uh how you assess vulnerabilities um develop defenses. So Atlas is modeled after another mouthful MIT re adversarial tactics techniques and common knowledge or attack. So uh it's it's modeled after that but focused on um a IML workloads. So um these are kind of kind

of some of the examples I'm going to go through uh in the next few slides and if the demo gods are um are with me I will show a demo. Um so everyone pray to the demo gods. >> Pray to the demo gods. >> Yes. Exactly. Um so one of them is a backdoor ML model. Right. So um and another one is ML supply chain compromise and

uh another one is a data model poisoning. Um so I I'll we'll go in deeper what it is in the next few slides here. Um there's another um thing out there um the open worldwide application security project or OASP. So with OASP and Atlas um they maintain a list of the most critical security risk in LLM applications. Uh every year uh O um OASP does a a

top 10 of of uh basically what's the the the top 10 kind of like most things that can happen. Um and uh mitigation measures as well. So, um, as you know, working with LLM is becoming more and more popular, it's important to keep kind of up to date with all of this stuff and like, you know, kind of have a framework. So, you have kind of have

a cool framework of using these tools uh together, Atlas and OASP. Um, and you know, it's kind of like like a good way uh forward here. So, um there's three things, right, that kind of relate to to uh the the prompt injection, right, is kind of like, you know, these out of the top 10 of of of this. I picked these threes out because it it relates

to the other the other three, but uh prompt injection, right? So, this is um hugging face poison uh GPT uh attack, which was it was theoretical, right? because but they actually prove did prove it out in in a lab. Um where uh you know they they show that that that it was possible. Um supply chain attack right PyTorch this is something that actually did happen. Um and

basically as Python is becoming more and more popular uh because of a IML workloads um PyTorch uh was was it kind of showed this this vulnerability showed the the fact that you can import vulnerabilities right I guess like it for for those that are into security these are we call transitive um uh things that happen right transitive vulnerabilities is like oh like maybe your software isn't vulnerable

but you imported something that was um and Uh then there's data and model poisoning right this is what uh primal lama was was all about was basically file tampering and so um paired together right with uh OAS when you pair together with uh with Atlas um you kind of have a good like foundation and framework um to you know to help you right um to secure the

platform for like you know getting ready for a IML workloads. though. Um, so I'm going to go through kind of how like each one of these examples um and how you could have mitigated that or how you can mitigate that with Psyllium or Tetragonon. Um, the first one was the um the ML supply chain compromise, right? So, uh this one was the the example was the PyTorch

um example. Um and basically you know you are they someone imported uh something they they shouldn't have or something that was um uh that was compromised somehow. So uh really around this is that this is kind of what I this is very very tiny so I probably should have made it bigger but um this is a a network policy a psyllium network policy again I say it's

a superset because this actually filters out based on FQDN right so if you think about like kubernetes and like IP addresses and um you know those that are from like a more traditional networking standpoint like the IP address right is like the canonical thing Um whereas psyllium takes care of things in terms of like identity based model where it's like okay like doesn't matter what the IP

is um I am going to filter based on DNS name right and so you can filter based on DNS name basically internal or external saying hey I'm not going to allow any any workloads to import anything other than like you know maybe a trusted repository right maybe have artifactory or something internally that you scan regularly and you trust and you don't want um you know developers pulling

things from the internet or anything like that you can like filter out based on DNS right and so um so that's kind of like what you get with psyllium right out of the box right open source you go and you be able to set some of these policies so that's one way uh selium helps um the next one is the um the backdoor ML model right which

uh this is the ebpf based thing that I was telling you about tetragonon um where uh basically uh manip this was this is based on the the chat G the poison GPT attack right where um you have a manipulation of training data to insert like hidden triggers right like if someone says oh hey help me with this customer and then all of a sudden that triggers something

like oh anytime someone mentions a customer or like inputs like some sort of data email me that data right so like they they they insert like hidden triggers there um as well. So again this here if uh you uh for those in the front those in the back I apologize but in in the front you can say oh hey um um don't allow uh this system call

right to happen to this file system right don't let it don't let it read this file uh or don't let it execute things anytime someone reads a file don't let it execute anything else right and so um uh this is kind of like how you how it can helps uh to to to mitigate that, right? And uh this one is kind of cool. This is kind of

my favorite and this is again hopefully one I can demo. Um is that uh the attack vector is basically um you manipulate uh model behavior. Uh you basically saying let it give the the person using the model uh false information. Um and so uh malicious false information. So um and again you can say uh things like oh um anytime someone executes something don't let it traverse the

path right so basically don't let it write to this directory or any other directory above or below or anything like that right you can actually trap those system calls um using again this is another tetragonon um policy that you can set right and so again I encourage everyone to look at ebpf how it's built into the kernel and how you and best utilize it uh using tools

like Tetragonon and Selium. So when I say Tetragonon and Celium, I say it's amazing that you can uh use them together, right? So not only can you use like the network policies like how you would normally would in terms of of uh Oh, that's always fun. While I'm talking, let's see if we can get this up and running. demo gods were not happy with us. Not enough

sacrifices. >> Yeah, not enough sacrifices. Yeah. >> Does it actually always go? >> Yeah, I'll launch it while you talk. >> Yeah. So, um you can use them uh use them together, right? So, you can use uh psyllium um as I was saying before for network policies, but like all the way from like layer 7 all the way down to layer three and uh three and four.

Um but also then beyond that like when actually workload starts executing training models things like that things that like are happening in the kernel and happening to disk you can actually also then just like set policies right and set policies um in terms of uh of mitigating that we may not be able to get the demo which is unfortunate. Oh yeah you have to If this happens

like in the next minute or so, I'll I'll uh I'll show that to you. But did you want to um >> Yeah. Yeah. Talk about your segment and then maybe go go back afterwards. Yeah. >> All right. Back to our presentation. So we have our infrastructure in place. We have security in place. Now let's talk about the LLM layer above it. So LLMs will run on the

data that you give them. Unfortunately, we have something called AI hallucinations and they have issues like AI giving you wildly outdated data like the sun revolves around the earth. The cure to a common cold is leeches, right? There's extremely outdated data. So trying to keep these LLM up to speed and up to date can be a challenge, but that's where you.com comes in. So think of LLM

as your smart but static brain. It knows a lot from its training data but doesn't automatically update itself on what's happening right now. That's where the API comes in. So we have the real time web access API that fetches the latest web results, news, or other sources in real time. So when your LLM wants to answer a question about something recent like what's the latest in AI

research, it can pull in fresh data. Two, rich context for LLMs. The API doesn't just return plain links. It gives long snippets, summaries, and structured context. The LLM can read this and generate answers that are more accurate, detailed, and relevant. Three, blending multiple sources. UD.com's API combines data from different sources, so such as web pages, news, etc. into a single response. The LLM can then reason over

all the information at once instead of searching piece by piece. Four, seamless integration. So developers can call the API directly from their LLM workflows. These setups use this in a Gentic AI where the model makes decisions or takes actions because now it has all the up-to-date information to guide those decisions. The TLDDR combine the two and you get AI that's both smart and up with the speed.

Now we have a closer look at what UC's APIs can actually do and why they're useful for AI and Agentic workflows. So web search API for the open AI's open source models. UC.com provides a ready to go search interface for open source LLMs giving them access to live web data without building a search system from scratch. Two, we have the news API. So if you want your

AI to know what's happening in the world right now, the API will deliver the latest news and keep answers current for US users and beyond. Three, B2B sources. So companies can pull the most recent product information to improve sourcing accuracy and make better decisions. This shows how search APIs enhance real world workflows. Four, practical applications. Beyond just search, these APIs support product recommendations, spotting trends early on,

dynamic pricing adjustments, and more. Essentially, any workflow that benefits from fresh, structured, or research information can use them. We also just recently just launched our research API for more complex questions. For situations where your APIs need more than raw results, the research API reads, reasons over, and synthesizes multiple sources into a readytouse citationback answers, making your AI smart, and more trustworthy. To summarize, UC.com search and research

APIs bridge the gap between AI models and the real world, giving live information, making AI smarter, faster, and more actionable across multiple use cases. So, if you look to the screen, I have a QR code to take you to.com's docs. And we actually provide a lot of free SDKs and developer tools for those of you who want to get into Aentic AI or people who want to

build chat bots. like we have a lot of free open source software that you guys can use. Let's check on this. >> Let's see here. All right. >> All right. I'm gonna let that be on the screen for a little bit. >> Okay, cool. >> See here. This is Oh, no. We have to go through this. Hold on. But the Oh, wait. Yeah. The backwards thing. Hold

on. I'm old school. I don't I don't scroll that way. Again, we have the QR on screen. So, those of you who want to learn how to make chat bots, those of you who want to get into AI development, plenty of open-source tools ready and available. And we also have the developer guide on the docs. So, if you're completely new to AI, don't worry, we have you

covered. >> Pun intended. >> Yeah. API. What's this here? Test. Okay, Wait, do I need to run? Okay, that's fine. Yeah, we're like 15 seconds away from getting >> Don't worry, we'll we have more time for people to get that QR code. >> Okay. And then does that work? Yes, it does. Okay. I can't believe it timed out. That's upsetting to me. Okay. I need to talk

to >> talk to the manager. >> Yeah, talk to Ralph about that. should be it. >> All right, crossing fingers, praying to >> All right, the demo is almost ready. >> I can put up this little nice >> little comic. All right, we're going to take the QR code away in a minute here. >> Well, actually, can I While that's starting up, let's do Let's go to

displays. Ah, okay. How do you mirror? >> Mirror. >> All right. And then we're using that. Sweet. Okay. So, uh, this this demo, and I actually wish I can make it bigger. Can I? Okay, cool. Um, this demo is, uh, we're going to go over how you can mitigate against data poisoning using Tetragonon. Um, and this is all running on Kubernetes using Celium running on Kubernetes. This

is also kind of like a sneak peek into uh isovalance um securing AI and ML workloads um lab which um we'll have a uh I'll have a QR code at the end so you guys can get free access and do the lab um on your own times here. So um here we have uh a cool little uh uh mince uh digit pre um predictor, right? you write

a digit and then the the model kind of tries to predict what number you wrote right here. So, uh some of the answers actually when we're testing this I guess I think um some of I don't know if the model was poisoned or if my answers are wrong. So, help me with the questions a little here. So, how many sides on a dice? I think it's six,

right? Unless it's seven unless unless you're playing uh D and D, right? Uh so, let's predict here. And oh, no, it told me it thinks I drew a nine. So may may I didn't draw the six. Really? Maybe it's maybe kind of predict there. I think I drew a nine. Okay, let's uh let's get a new PR brain teaser. Let's see what's going on here. How many

chambers in a human? Actually, no. I think there's four. Three. Does anyone know there's four? Okay, I was right in the beginning. All right. I'm like I think there's four. Um predict, right? And it it predicted a four. Okay, so that's good. Um uh what is half of two? Okay. So, one, this is not very um I think I drew one. Okay. And how many sites on

a dice? Okay. Question here. Let's predict that. It keeps thinking I drew a five. Really? Um so there's I mean my drawing isn't that terrible. Like my handwriting is not that terrible. Let's try to predict that again. I think I drew a five. So there's obviously something wrong with this model, right? And so um so let's uh uh I have this little script, right, that'll that'll test

uh the inference model here um and let's see what happens, right? And so I'm seeing a lot of red here. So uh here accur um images nine were predicted as sixes, right? Okay, so we actually saw that um when I did it a few times, right? I actually tried to thought I predicted uh nine. We're seeing a bunch of fives up there. Uh, so we did that

here. Uh, this is 90%. I think you I think we can let that pass. There another 90%. That looks good. Uh, another 0%. So it looks like here when you draw um sixes, it it five. So when you draw a six or a nine, it looks like it's it's it's not uh um it's not predicting that correctly, right? It's getting 0%. Right? Anything 90 and above, we

think, all right, good, right? It's not going to get it 100% all the time. Um, but it now now we actually see that there there's there's a problem here, right? And so, uh, we're going to do some file integrity monitoring, right? So, we're going to, um, uh, uh, let's go to, uh, security observability. There we go. So, we're going to, um, use, uh, Tetragonon to kind of

look at what happened. Um, and as you see here, uh, if I go up here, if I know how to scroll, a lot of stuff's happening here. File file permission. Uh, open file read. >> The well has been poisoned. >> Yeah, the well has been poisoned. Here, uh, I can see here there's a read there. There's, uh, let me run that again. Oh, I do tail 50.

I need actually tail maybe like a thousand or something. A lot of stuff going on in Python right Okay, here we go. So, uh as you can see here, someone um poisoned, right? So, you know, the attacker conveniently called it poison, right? But like this is the demo. So you can see here uh someone um copied some poison data over and that's what the inference model is

using in order to uh uh for the prediction. Right? And that's why we're getting those things wrong. Right? So um so how do we uh how do we mitigate that? Right? So uh let's go into the editor here and let's do the training model. Um actually let me just copy all of that so you can see here. Uh, wait. Command. So, here, uh, we're going to do,

uh, can I make this a little bigger? All right. Um, we're going to do a a policy, right? A tracing policy. Uh, this is again based on Tetragonon. And we're going to say, um, uh, the file, anything in this directory, right? If anyone tries to write anything, delete anything, rename anything, we're going to block that, right? And this happens at the kernel level in the operating system

in Linux. Um, and again this is based on ebpf happens on the on a kernel level. What what ends up happening is that uh at runtime right when it happens a little program uh uh starts up and then basically um interfaces against that um that system call. And so what we're going to do is we're going to apply this policy and then we're going to try uh

to poison uh that data again, right? And so let's go to the terminal because we love the terminal. Let's do cube apply there. Uh there's a damon set right that's running on all of that there. And let's try to uh poison the model here. And if I run that uh it gave me an error. Just run it again. Give me another error. Right? It says tar. It

says uh failure due to previous error. It says cannot open file. File exists. Right? So I cannot write into this directory. Uh so what why is that right? So let's go back and let's let's query this again. And as you can see here there's this new uh new thing that happened here that the file delete right uh both times that I ran it was blocked. Right? So

um that was blocked. So when someone at the operating system level tries to write data I have a policy in place. Right? whether they they they they were super user, right? Whether they they became root, they somehow compromised the machine, they try to write into it. No, there's a policy there in place that um that uh that prevents that, right? So, um any any of you familiar

with SC Linux or at least so if those SC Linux is very very similar to what happens with SC Linux, but this happens at runtime, right? This is not something that like that is uh built in, right? This is something policies that you can set, right? So this it's not something that you have to write into the kernel. So um and this is an example on uh

how you can use Tetragonon to um to secure kind of like uh data model poisoning, right? So like the the model poisoning uh examples that I gave, this is kind of one example. Uh we go through all three examples in this lab. So, um, uh, uh, so when the QR code comes up, you can scan that and do that that lab in its entirety, uh, when you

have a chance here. So, let me go back to, we're just, we're doing this live. System settings. I don't want to mess up your settings, Angelica. So, okay, cool. >> I I did it. >> All right. So, I know it was a lot. So we're going to take a minute to review everything. So remember this. Yeah. So basically we went through this whole thing. We talked about

data. We talked about the infrastructure. We talked about LLMs. And I talked about APIs. And now we just have the application. And that is up to you guys because that's something that you guys can create. You can make whatever you'd like. We just set the foundation for you. And here it is all put together. I like to call it the trinity. So we have cordant kubernetes as

a foundation to manage resources. We have and scale with AI workloads. We have the LMS being fed u.com's APIs and we have Celium in the background keeping data secure and compliant. Would you like to add anything Christian? >> Well there's a big thing on the on the right hand side there called pselium. Um and and it should say Tetreon as well but basically you want to you

want to keep all your data compliant and secure. You can do that with existing open source tools. You can do that uh with tools like psyllium and and tetragonon um and and and now you have that kind of like that that foundation in terms of like securing workloads and that's just generic workloads. Uh we use AI as an example but um you can use selium to protect

the entirety of that cake right as we're calling it. >> All right now who benefits so first we have platform engineers which Christian will briefly talk about. Yeah. So obviously platform engineering um DevOps engineers uh system administrators whatever we're calling ourselves nowadays um this is important because no matter what workloads landing uh on our environments right whether we're running Kubernetes or VMs or anything like that um

we want to be able to secure them right because at the end of the day yes it is cool I am a techie I love building stuff but at the end of the day you have a business to run right you have uh you know we're we're building a platform for others to use and we want to be able to have them to have it robust, secure

and and and available for them. So, uh for really platform engineers benefit um because it keeps the lights on. We want to keep things secure. We don't want to be paged at uh 2 o'clock in the morning. >> All right. And then how do AI developers benefit from all this? AI developers care about building, training, and deploying models. And they need tools that directly help them run

experiments, manage workloads, and get results. So they can use cordant to focus on models instead of worrying about doing the cluster configs and the yaml files. They can use u.com to get the live web data and research APIs for their LLMs and agentic AI workflows. They can uh use the API so that at that time they don't waste time scraping, filtering or curating the data. These APIs

are already structured ready to use results that feed directly into their AI pipelines. In short, the APIs make AI smarter, faster, and more accurate, letting developers focus on building applications instead of managing their data. And then lastly, with Celium, Celium shines for AI because AI workloads involve many small high performance services communicating under tight security observability constraints, a pattern that not common in uh standard enterprise applications.

So, I can just build and train my models without worrying about anything with the infrastructure or worrying about And then as a thank you, we have the lab available and on the left we have you.com uh free trial. So as a gift to all of you for attending today, I'm giving away $100 worth of credits for free. So scan that QR code and you can play around

with all the different APIs in the API playground. Again, my name is Angelica Ambrosio and I thank you all for taking the time to be here. >> Yeah. And to the right again, the A IML free lab. Um, it's there provided by Isovalent. Do it uh at your own time. You actually get a cool little badge. So, if you get uh if you actually end up doing

it and and do a badge, tag us, right? We're on LinkedIn and and and uh Twitter X and I think Blue Sky as well. Um do that lab. If you get the uh you get the badge, tag us. We always we always love uh people tagging us there. And um uh you you'll get to do the the portion that I did, but also everything else, right? you

kind of get like the whole story. Um, so it's a it's it's it's an inclusive lab. I think it's a we get two hours. Um, make sure you're doing it though, although you get timed out like we did. Um, but uh you get like two hours free in in that lab. So, uh, thank you as well. So, >> thank you so much everybody. So, we have about

five minutes left, so we can squeeze in one, maybe two questions. So, any questions for Angelica or Christian? >> All right, awesome. Well, thank you all. They'll be here in case you wanted to ask anything uh directly. The next talk will be in this room in about 20 minutes at 3:45. And as a reminder, cloud native tracks will be happening in ballroom A and ballroom B. All

right. Thanks everybody. And thanks Angel and Christian. check. Check. Well, let's get started here. Um, I'm going to talk about a lot of points of view on containers today. And some of them you probably I don't know maybe a little little less expected but we'll go through kind of some experiences that I've had and kind of just the way that I've been seeing this ecosystem evolve over

time. Um so my goal here is to rethink a little bit of the last 15 years of cloud native defaults. Um CNCF you know has had birthdays recently. We've been working with containers for a while. There was this whole thing of are you cloud native or not and we all had to figure out what that even meant for a while. And so that's what I'm going to

talk about, but I'm going to approach this from several different points of view of both longtime experience in distributed systems, longtime experience at the different vendors and software companies and also just experience playing with Unix and Linux and stuff like that because I eat, sleep, and breathe this stuff. So um anyway, I'm Mike Stony. I'm the VP of engineering and growth at Flocks and my experience started

at Caterpillar way back in the day where I did a lot of infrastructure, ran automation, architecture, things like that. I went to Puppet for eight years there, did a lot of open source development, did community leads, did engineering leadership, stuff like that. Ran engineering at CircleCI for a little while, um, like four years. Um, and then went to Flux, which is a series B startup that is

working in the Nick space. Um, trying to change the way that developers do some work. And um I've also packaged a whole lot of software. Release engineering is my background. And so I am very much a package nerd. Um I created this software repository called Apple. If you've ever used that on Red Hat, Enterprise Linux or CentOS or any of that. Um that was me and six

other people that decided to get together. We realized we were all doing this at giant companies and it was not a competitive differentiator. So let's go build something that everybody could use. And so we did. Um I've also packaged things like I ported Ruby to AIX and you know all sorts of stuff like that. So anyway the point is I like packaging software. Um I have thought

about these problems a lot. So containers really ended up solving one major problem. This is what they were really designed to do and that was uh developers didn't want to package software. They used to just hand it to somebody else and say go build me a package. And then with a docker file suddenly developers were like I can give a package as long as it's a docker

file. And now we have a package. And why do we want a package? Well, it's because it's portable. I can distribute it. I can use it different ways. But the issue here is that when you let developers do the packaging, you basically end up packaging the way a developer works. And I'm a developer and I know how I work sometimes and I know what stupid mistakes I'm

making. I don't wish to embed those in my production systems generally, but here we go. So basically we end up putting a lot of croft or handcraftedness into things that end up in production because of the way these containers are built. Um and not always because some people have very good process and everything like that. But I will say the possibility is definitely there. The real world

scenario is absolutely there of people kind of putting things that were designed for more of a developer environment into something that ends up in production. Now, part of this is this is not an anti-ontainer talk. I don't think containers are dumb at all. I use them for a lot of things. Um, but I don't use them for everything. And that's kind of the point of this talk.

And so, by the end of this, I'm hoping that if nothing else, you just think about this problem from a slightly different angle. You don't need to change your behavior. You can if you want to. Um, but I just want you to think about this problem. Be like, "Huh, there were some interesting tidbits there. Maybe I shouldn't reach for a container the next time I'm solving this

problem." maybe you should. So containers have solved real problems and I I do want to point that out. I really really like containers. I like the ecosystem. I've worked with it a lot. Uh you got portability and you got confidence in deployments because you kind of knew that that OCI spec or previously like the Docker container spec which were not the same thing by the way. Um

those worked really nicely on different operating systems on different runtimes on different container runners and you kind of had a lot of guarantees there which was great. And so Docker, which came out kind of in 2013, was revolutionary. It it definitely changed the way that we did software stuff in every way. Uh, and it made shipping software a lot easier, which is great because, uh, shipping software

used to be pretty hard. Um, and in a lot of cases, the places that made it really simple had a differentiated edge over the ones that didn't. Uh, or you were handing it off to an operations team to say, "Figure out my stuff and put it in production to me," or whatever. and Docker definitely helped to lower those barriers. And so I do want to give containers

their flowers before I start um speaking heresy in the church of the container. So um we forgot what an OS can do. And this is what I'm going to actually walk through for a little bit here. So Unix from the 80s to the 2000s, maybe even from the 70. I probably should have said 70s. Um but in the 70s, we basically have this multi-user multipprocessor operating system.

Multipprocess operating system. It's also multi-threaded. If you ever go through like operating systems theory class, they go through all these different like variables on is this an operating system or is this just a runtime. Um, but Unix was awesome and actually it's still pretty awesome, but it was awesome as But you had m you had multiple users on one machine and so you had this paradigm of

I'm going to have hundreds of users on one system. Well, that mean you had to solve a lot of real world problems to do that. And Unix worked really hard on doing that. They had things like, "We're going to handle isolation. We're going to have users in groups and permissions and quotas and interprocess communication and user communication and user lockdowns and all these types of things." And

we ran like that for a long long time, particularly in academia or in technical workstations areas and things like that. And it worked really well. You could literally talk to everybody else that was logged in if you wanted to over the machine. You could talk to other users in real time, kind of like an instant message, only this was long before there was any type of instant

message or text message or anything like that. You could just, hey, you're logged in. I'm logged in. Let's chat. And this machine was a shared living system. Usually had a name that was friendly or well-remembered. You know, a lot of times these machines were named after like Star Trek or Star Wars or something like that. Um, and so you were like, I'm going to log into Yoda.

Everybody else has logged into Yoda. we're gonna go talk together on Yoda or I'm gonna go do my work, but while I'm there, you know, late night, somebody else is messaging me because they see a problem and they already know I'm logged in there. And then we got to this thing where VMs came out and they were like, we're just going to put one app per machine

because then we have all this extra isolation. And in a lot of ways, that was super cool because now I didn't have to worry about things like, well, what group is this thing running as? Because no one else was on this machine anyway. So I didn't have to worry if you were in this group or that group or you had rights to do this or whatever. And

that's really really nice kind of. Um so we but we threw away the multi-user part. We're just like okay VMs are really they're designed to run usually a single application in most cases. Um obviously there's VDI and there's there's definitely caveats to what I'm saying but I think hopefully you can get the uh the gist of what I'm going at there. So, and then we got to

containers and they were like, "Okay, we're actually going to do instead of one user, we're going to go down to one process." that was also pretty interesting, had some great value, had a lot of things that were really odd. And so, we had to build up a whole set of different tools to work with that. And so, we threw away multi-user when we got to VMs. We

threw away most of the OS when we got to containers, except we still have to carry an OS around us in the container. So there's like still kind of some stuff going on there. Um, but 90% of this OS is dead weight where you're not really using it because you're not linking against it. Your runtime isn't calling it, your app isn't calling it, but it has to

be there just to have a base image in a lot of cases. And yes, I know about from scratch images, and I'll cover that in a moment, but um, and so there's just a lot of things here where we're carrying around stuff that isn't part of what you actually need to get the job done that you're trying to accomplish. And when I was thinking about all this,

I was thinking about how I think container native technologies are relatively comp complicated. Distributed systems are relatively complicated. And when I put two complex things together, I get a headache. And so I was working with one of my younger CIS admins at Flux and he was logged into a system and I one day I just messaged him over wall. Actually I messaged everybody on the system. and

there was like three of us logged in. Um, and I was just like, you know, started I don't remember what I said. And he slacked me the second I did that and said, "Hey, I think we may have just had a security breach." And I said, "What makes you think that?" And he said, "Well, I was in my terminal and all of a sudden this text started

appearing and it was like, "Hi, Brian. How are you doing?" And that really freaked him out. And I was like, "You've never seen two people talk to each other over a Unix like system before, have you?" And he was like, "No." He's like, "How'd you do that?" I was like, "There's literally this command called talk." So that when I want to talk to you, I type talk.

And and and he was blown away by this because he was just, you know, he was younger. He grew up with containers and all this stuff. And he had never seen a multi-user setup like this before. And as I started thinking about that and we had a long conversation about some of these paradigms that had I guess I would say faded away in modern architecture and I

realized that this is a through line for how I've been thinking about this entire set of problems for I have adopted so much complexity to solve problems that might not even be complicated. And so he had never seen two people use a computer at the same time basically was my conclusion there. And so we didn't really solve isolation with containers. We changed the shape of it though.

We forgot this multi-user experience that we had built from the ground up on a on an operating system that was multi-purpose, multi-user, multi-threaded, pretty darn great. And then we we just decided that we're going to use a container for everything. So for developer onboarding, we'll use a container. For a CI pipeline, I'll use a container. For a local developer environment, container. Edge deployment, container. ML workflow, container.

When you have a hammer, the problem is the hammer comes with the entire hardware store. And I don't necessarily need the entire hardware store to do the work of the hammer. In fact, I had an operating system that worked really well that did a lot of these things before I pulled it all apart and slipped it out into single processes with a full OS that were called

these things like So, let's go through an example. I have three HTML files in a CSS stylesheet. I guess the last I guess CSS is like saying ATM machine. Sorry about that. I I didn't look at that. Now I'm mad at myself. Um, so you have three HTML files in a stylesheet. And this is what you want to package. So I'm going to need a Docker file.

I'm going to need EngineX in this Docker file. I'm going to need a registry to store the image. I need an orchestrator so that I can run the container. I need a service mesh so that I can talk to all the things. I need an ingress controller so that the things can talk to the other things. I need a secrets manager because I have to go get

stuff in a config map or something like that at some point. And a CIC CI to build my own container. And then I got to scan the container to make sure that it's production ready. I got have a policy engine to make sure I'm allowed to run this container on these right nodes. I need observability tooling to make sure that none of that stuff failed in that

entire process and a team to maintain this platform that this all runs on. And I had three HTML files and a When I used to use Unix, I dropped this in a folder. It was called public html. I was done. We did not make this better. Okay, let's do another example. I have a function. It runs once a day. I ship it as a container. It's a

500 megabyte container. It goes up to Lambda. Lambda runs it. Or I could have had a 5 megabyte zip file and not included the entire OS with it and it probably would have done the Now, Amazon wants me to build that whole because every time I ship that 500 megabytes back and forth, they charge me more. I don't know if you ever looked at cloud bills of

any significant size. Network is the sneaky one that always gets you because you you can plan for commute or compute, you can plan for your RDS, you can plan for your storage growth. And then networking, it's kind of a hodgepodge because some months you're doing a lot, some months you're not. You're moving stuff around, you're doing migrations, whatever. And the point is networking it like looks super

cheap until it's not. Um and so these incentives are not aligned. Basically the ecosystem players the major hyperscalers are actually incentivized for you to use containers because of their inefficiency. I'm incentivized to be more efficient because I like that. I mean I'm not paying the bill. Some companies paying the bill. So I'm not that concerned about that part of it. I mean like I'm concerned if I'm

measured on it in some way or it's my responsibility but generally doesn't come out of my pocket. Um but this platform profits from our inefficiency and that is why three of the richest companies in the world are cloud providers and they continue to profit off of our inefficiency. Doesn't mean they're bad like it's their business model. It's fine. Um I just don't necessarily want to do that

all the time. And then beyond that you have this complexity tax which I started to unfold as I was walking through my HTML and stylesheet examples. So this is a partial view of the CNC CNCF landscape. At this point it is not possible to fit this on a monitor um because you have to scroll both right and left and and up and down um because there is

just so much stuff in the CNCF. Um when I look at this I do not see good things. I will say that it's not because there's any of these vendors are terrible or anything like that. They're not. I'm just like, "Wow, I have to look at that much stuff to understand stuff. That's a lot." So, no single human can understand all of that. You may be super

deep in the observability space and know who most the players are there and most of the tools and most of the projects and all of that, but you're probably not then super deep in that CI/CD space or something else over there. It's just too much. And so with every problem in this cloudnative stack, we've created a new abstraction. We have a complexity tax because every time you

add an abstraction, you both have a new mental model of how to think about it, but also new debugging steps you have to take to break the abstraction to understand when it's not working. And so we didn't ask to stop is this layer necessary or maybe it was necessary from 2013 to 2017, but in 2018 we had solved it with a new version of something and we

didn't need that layer anymore or that tool anymore. And I don't see a lot of those tools coming out of the stack. I see mostly tools being added to the stack. And so we rebuilt the OS but kind of worse. Uh with process isolation, we have containers and orchestration. with process supervisors like systemd we have cates controllers and health checks with interprocess communication we have service message

uh service messages and gRPC with logging you have things like elk or fluent or Loki and there's like a million others um for things that were like PS and top and srace we have Prometheus and graphana and open telemetry like so you have all these tools that kind of map to these primitives that came right out of Unix and for user permissions I got arbback and opa

and admission controllers and for file system permissions. I have vault and sealed secrets and you say that one doesn't make as much sense. And you think about it, the reason you have secrets is so you could load them into an environment. Whereas if you had an environment that no one else could see and it was isolated, you wouldn't need to load them in that way because basically

they're all in runtime in memory anyway. So that's why that's why I put that one here. It was a little weird. I had to think about it for a bit. So um cron you have Kubernetes cron like they even labeled it the same thing. Um, and then you have like talk and wall and signals that end up being more or less web hooks and event buses and

DNS. Okay, so basically you end up with I have rebuilt all of Unix on a distributed cloudnative platform where I've externalized everything that was built into the operating system so that I can ship an entire operating system inside of a format to run a single process. Why? I don't know. All right. So, we didn't add capabilities really. We decomposed a working system and it worked quite well.

We reassembled it from thousand vendor sponsored pieces. Like it's like we had all this stuff that was built in the OS. So now I can buy each one of these things separately. I can maybe use open source stuff with it separately too. But there's a lot there. All of those things I can reassemble to give me basically what Unix used to do since the 70s. Cool. And

then containers got used for other And I like to say it's technical debt laundering, but I usually get into default swap derivatives. If you're actually into how the economy collapsed in 2008, that was basically what it was. It was trading on default swaps. And default swap derivatives were basically this idea that this person wasn't going to pay their mortgage and I'm going to bet on it and

you're going to bet on my bet on that and we're going to trade it around back and forth until we're all happy and make money except no one makes any money except and then the government banks bails you out so everybody makes money. Uh I think that's how that worked, right? We all got better off of that. Yeah. Okay. Um so if you have a messy application

you can just throw in a container. There are entire companies that make their money doing this. And honestly, I almost started one at one point. Uh where I was like, you know, the first person to put SAP in a container, they're going to make some money. Person JD Edwards or Peopleoft, things like that in a container, they're going to make some money. And I really debated about

going to do that. I decided I liked myself too much and didn't do that. Uh but I think the money would have been pretty great. But the debt doesn't go away from taking these legacy applications and putting them in a container. What you did is you put in a little box with a label and you're like, "Hey, this is my debt. It's in the shoe box. It

runs over here." And so this is the equivalent of basically shoving everything in the closet before the guests arrive. Uh because your house looks really clean because you have these nice containers. Uh but then you open the closet and everything falls out because you still have all the problems that you had before this thing ran in a container. um you just have a new shape for the

And so if it works in the container, I stop investigating why it was broken. Whatever. Um I didn't fix my dependency messes because I just took everything in there and swapped it and threw it in the So we baked in all of our problems into a file format and we called that an image. That's fine. We're going to come back to that though. I'm going to talk

about a story that was real life. So like I said, I used to run engineering at CircleCI. How many of you have used CircleCI? Anybody? Okay. I I think it's a really great product. But um I ran engineering there and we were having an outage and it was complex and I was frustrated and we couldn't figure out what was going on. We had six, seven services that

were down. Um we couldn't figure out what was going on and the place we saw at first of course is where you start your debugging. Well, I first started to see problems over here. Okay, so you do that and an error shows up in service A. But the problem was seven or eight handoffs before that service, there was a cache that where the disc was full on

and we were not monitoring it properly. Also, isn't it funny that with the most complex distributed systems in the world, the problem is still my disc fills up. Yes. So, we have not solved that problem at all. Um, and so this expired cache was in a service that nobody was watching because it was one of our first deployed services as we were like ripping apart a monolith

as one does. And that one didn't get all the same monitoring and observability patterns that a lot of the other newer ones had because it was one of the first ones we had done. And apparently we hadn't retrofitted all of that stylistic pattern on top of that older service. Again, I think that's relatively understandable. It happens. Um, I'm not like, you know, trying to lay blame here,

but the issue is that that took us more than two hours to figure out that this service over here looks really broken. And the reason it's really broken is because this cache isn't returning a hot hit like 95% of the time like it usually is. And so debugging took a lot of heroics. And that was the first time that I think I really, really, really got frustrated

with the cloudnative architecture on everything. I was running like 30,000 nodes in Kubernetes. So like I was doing this in a big way at scale and I was just like this is not working for me. This is not working for a team of experts and we were really good at software engineering. We're deploying like 500 times a day at times. Like that's how fast and how good

the testing was and the validation all of this. But this service was broken. It was going through five other handoffs of like gRPC communications and in you know all that kind of stuff. And then over here there was a cache miss that was happening at a way higher rate than normal. And so it was impacting performance and that performance was compounding and it was impacting hit rates

on other stuff. Other stuff was timing out and that's why the users couldn't do their work. Sucks. So files missing because I wasn't getting a cache hit on a simple service. I would have been like, "Hey, let's look at this pit. I run srace. Hey, it keeps looking for this file. It's not here." Oh. Huh. I wonder why that is. I'll go look at the source code.

I'll put it in there. will be done. Maybe it's a config file, whatever it is. So, the complexity doesn't just make building harder, it makes operating more difficult as well. Makes debugging dramatically harder. By the way, operating and debugging are the same So, whenever you hear somebody talk about operations, you can just think about the debugability of the system that is online. And if you debug, and

you debug, you do debug more than you build, I guess, is the point. Particularly now with AI accelerating how fast software gets developed, you're actually spending less time on that raw development than you probably ever have. And I already thought that was the easiest part of the entire SDLC. Um, now we have operation operating software in perpetuity. Operating software, keeping it online, keeping it secure, keeping it

up is difficult. So we got the distributed monolith. That was basically the story of of that set of services in CircleCI was we had thought we had ripped it apart into microservices but we hadn't because if something failed in one service that was eight services over from another one two services had failed and it turns out keeping two things online is more difficult to keeping one thing

online. You can double check my math on that one. Um, and so I got all the complexity of this distributed system and I got all the coupling of a monolith and uh that means I couldn't kind of couldn't deploy things independently. We actually did deploy them all independently and then I was pretty proud of that for a long time until we had an outage that looked like

this. So from there we have synchronous call chains. They're eight services deep. We have one a well structured monolith. I would have been one srace away from the answer. And we did have a well structured monolith that we hadn't pulled everything apart. And honestly, debugging that was much much simpler. But we put this monolith into microservices and we got a distributed monolith. And I would say that's

worse because now I have two problems. Um, and I usually summarize this with it might not be good, but at least it's And then we get to the sidecar explosion, which is I wanted to deploy one thing and I am running four things. And four is just an arbitrary number, not exact. But I have like an application container. I have a logging sidecar. I have a service

mesh proxy. I have a security sidecar. And all I wanted to do was run my app. so I can write a log file, but I have to launch that sidecar to be able to write the log file. There's just a lot of difficulty and complexity here. And then there's the monitoring paradox, which is the more complex a system, the more monitoring you need. The more monitoring you

have, the more complex the system because monitoring tools also require monitoring. Yay. Um, and so some orgs spend more on compute like more compute power on observability than they actually do on their real workloads for customers. And I can say that the PL the ones that do like it's fine that might be the right the right choice but I know that some of my observability bills when

I was running engineering at my last company were very very very large. Um, and I say that comparatively even to some of the cloud bills and things like that. So, and then we have these other things that could do observability or monitoring or system debugging or whatever you want to call it where this stuff was all there. So, you need monitoring for your monitoring and we've optimized

all this for developers. I would say that containers was an optimization for developers so that they could package things quickly and they could basically say I have figured out how to do things. I'm going to put it in a container and that container will be the thing that we'll base our foundations on to run our application on. A totally noble cause. I'm just going to say we

may have missed a few spots. We forgot about debugging and we added it on later through all these layers of extra tooling. We made it easy to ship something that nobody can troubleshoot and I find that a little bit difficult. There's also container golf. This is one of my favorite side sports. Um, how do you get your container to be as small as possible? Well, you can

do Alpine, you can do distro list, you could run it from scratch, you can do all sorts of cool things. Um, but most people have a base image they use somewhere and they just say we have a base auntu image or we have a chain guard image or you know whatever it is, we have something that we use and we just have a base image and we

don't do from scratch because it would be very different depending on what the application was or we don't do distro list because of the same And so this practice of container golf ex only exists because of containers. Obviously we wouldn't have it without it. Um and with this you actually scan your image and not your container. Um because your containers go out into production and I don't

know if you know this but you can actually change a running container. And from there that scan that you did on that image is um unimportant I guess would be the way I would I would word it. And people say well you shouldn't do that. You shouldn't edit your container live. You might not but your application might be. And so that's the thing to think about as

well. Um, latest, by the way, mutable tags are my least favorite part of the entire Docker ecosystem, and it's not particularly close. Um, anytime I can have latest mean something different today than it did yesterday. Latest has no meaning. I do not know what that means. I do not know how to figure out where it is. I hate it. I hate it. I hate it. Tags should

not be mutable. They should be immutable. It's a release engineering theory thing. And it is right. I will not be taking questions on that at this um so so apt gives you different updates on different days. And so like a lot of times in a docker file you might say apt get install you know dependency one two three and four. You do that docker build last week

you get a set of software. You do that docker build this week you get a set of software. Those are not the same set of software or there's no guarantee that that's the same set of software. You can pin every package you want in a Docker file. However, that is extremely difficult to do on most distributions. A lot of times you can pin the leaf node, but

you can't p pin everything that's in the tree trunk because it might be well lib C updated underneath, but you didn't pin lib C. You actually just pinned your version of Python. Um, things like that. And so, a packaging format with some reproducibility. Ah, a packaging format the same reproducib. Um, so so all right. And we're about to do all this again with AI. That's the thing

that I'm getting scared about is like we have these GPU model serving containers that are 15 gigabytes. So instead of like maybe a a one gigabyte container, we're like, let's do this like 15x more and we just ship it all around everywhere. We're going to spend way more on networking. We're going to spend way more on storage and the hyperscalers are going to get way more of

our profit because we chose the wrong tool for the job. So we have the same golf where I'm getting 10 only I'm doing it at 10x to 30x the size on the container. Now, I'm going to get into one other problem is that we have frozen entropy inside of containers. There will be like a wrap-up where I I solve a lot of these or no, maybe not

solve. Solve is a strong word. I get you to think about it differently. Um, but let's just say I have an application. It needs Python 311 Note 18. Everything works and we ship it. Okay, cool. Months pass, years pass, it still uh works, it's running, everything's great. And this uh this is one of Isaac Newton's lesserk known laws, which is a system at rest will remain online.

And this was classic. Like when I come from a Unix heritage, this is what used to happen. It was you got this thing to work, we're going to put it on HUDX, it's going to sit in the corner and no one's going to touch it for four years because it worked. And if you touch it, change is what causes problems. Lack of change can also cause problems,

but there's a different type of problem that that can solve or have. So you might be getting new vulnerabilities that are discovered. When people say you're accumulating CVEs, I'm like, "No, you always had all the CVES. they just weren't known. Um they were always there. So um containers kind of just let you freeze this entropy which you're like that's awesome. I've got a thing that works so

it's not going to change. It's stable except you've also frozen your security posture. And so you have this image, you have this image, you may have been using it for a while. Um maybe it's six months old and you still keep deploying it because it's got the application configured perfectly and all I have to do is load in the right config map and it runs and it

does everything it's the way it's supposed to do. All right, that's fine. Containers didn't make us lazier about this. We did the same thing in the Unix world. We just let the system sit there in the corner and we're like, don't touch it. Fred's on vacation for the next four weeks, so we're not going to make any changes to this system. Okay. Um, so they just made

it basically easy to stop moving forward and we conflate I'm not going to do anything with stability. And those are again not necessarily the same thing. They're not unrelated. Um, but what we lost in here was a forcing function. And I don't know if every any of you have ever worked with like hardware data centers or your own stuff where you've touched the server and done the

work. There used to be this thing where hardware had a life cycle. Might have been three years, might have been four, might have been five, whatever it was. At some point you were like that hardware has to leave the data center and we're going to get new hardware in the data center. And when you did that, usually that forced the operational capacity teams to be like, well,

this is when we put on a new OS or we have the latest patches or we update to the latest version of this application or whatever it was. And so there was all of these moving parts that meant those hardware migrations were a forcing function that often, you know, took nine months or a year to plan and execute and all this kind of stuff so that you

could upgrade your application. I don't miss any of that, by the way. But the nice result was I had a a maximum window where I could hold something kind of the same way. Whereas right now, if you want to run a container of Ubuntu that came out in 2013 because it just worked and did everything you wanted to, it probably still runs. And again, in some viewpoints,

that's a feature. And in some viewpoints, that's a liability. I can't tell you which it is for you because it might depend on what you're doing with it. Um, and then there's finally this problem that I have with YAML. Um, we configure the most complex distributed systems on Earth with a whites space significant language. Oh my god, why? No, really, why? I have no idea. It's a

really weird choice. Um, anybody know what the Norway problem is in in YAML? It's that if you have a list of countries and you use their abbreviations, Norway is N O. What does N O ev evaluate to in any YAML library say pre209? Well, it's the fall. Evaluates to false. So, no is false. So, Norway is false. So, I can't have a list of countries. Even if

you quote it, it'll still evaluate to false in some cases. It depends on which which YAML library parsing you're using. Oh, and this was fixed in the YML spec in 2013. The YAML libraries that are most commonly used today still do not honor that. So like the one in Python, for example, today does not use the AML spec for 1.2, uses 1.1. Um, this problem exists. I

hate it so much. Um, and then inside of this YAML, because YAML is not a programming language of any kind, it's a serialization language, we invent dialects of programming to put inside of which makes me sad. It also makes it so I can't read it. It also means that I don't know how to validate it in any way, unless I'm using a very specific tool to validate

a very specific type of YAML with a very specific templated programming language inside of it. And if I'm going to have to have specialized tools for everything, I could have done this without YAML. I could have used just hey this is the configuration for the way that we set up controller Y or whatever. So it's like programming but it's worse because I've got no type system and

no compiler and no debugger and no validator. And people are like but you can put it in your editor and it'll show you if things are lined up right. And I'm like cool I have no schema to know if it needs to be in this indent. Is it an array? Is it a list? What is it? I don't know. And this is why I don't like YML.

And then there's the final thing. It's like, well, the reason I chose to run this giant CNCF stack is that I need to be able to scale. I will challenge you and say, you probably don't have a scale problem. I know the company I currently work for, we could serve out every piece of traffic that we get all week on a Raspberry Pi with a cell phone

modem and we'd be fine. And it's not because we don't get that much traffic. It's because a Raspberry Pi with a cell phone modem can handle an awful lot if you know what you're doing. And instead people are like, we're going to need thousands of nodes doing Kubernetes or whatever or maybe just a couple, but it's a lot. So Kubernetes was designed for something like Google Scale.

That's great. They have really weird problems. They have really hard problems. If you've ever worked somewhere where you have to dig into TCP and change the way it works, you do have a scale problem. Congratulations. But you're probably not Google or Meta or a lot of these companies that are scaling in that to that degree. So you may have an adoption problem though. You may have a

revenue problem. You may have a product market fit problem. You may have a unit economics problem. Like all of those seem more likely to me than a Scaling is late stage. And you might say, I work at a super mature company. We are doing things at scale. That may be true. Your next business line might not be mature. It might not have the scaling problem yet. It

might be okay. Well, this business line, this new thing I'm going to deploy, does it need to go to to Kubernetes day zero? Does it not? I don't have the right answer for that. I don't know your politics. I don't know your inside architecture, you know, requirements and all that. But a lot of times you can move a whole lot faster with an HTML file and a

stylesheet in a directory than you can with a container and a pipeline and a scanner and all of So if you have a scale problem, congratulations. It actually means you're successful probably or you're really bad at handling traffic and tuning, one of those. But premature scaling is premature optimization. And if you go back to your CS theory classes, Donald Kuth will pop up and he will say

that premature optimization is the root of all evil. Um there's more to that quote, but this is the that's the pole quote part of the quote. So, and then Kubernetes also had this multi cloud portability myth to me. Um, we'll use Kates so we can run everywhere. My favorite part of Kubernetes by far is that it's a standard API that so many applications have adopted and used.

That is awesome. Working the same way. Great. Got no complaints about that at all. Um, the issue is that almost nobody does cl crossplatform cloud capabil portability. Who runs on multiple who runs on a Kubernetes provided thing across more than one cloud provider for production workloads? One person in this entire room. Your Terraform is cloud specific. Your monitoring is cloud specific. Your billing and billing analysis is

absolutely cloud specific. Um and so you're not jumping back and forth even though you were like this is this common abstraction. You're right. it is until it's not. And so we traded vendor lock in for this ecosystem lock in where I have to use Terraform in this way with these providers or I have to use these logging things or these billing analysis tools or whatever. And some

of them are crossplatform and some of them do an okay job but generally most people don't actually do production workloads using the same tech on different clouds. So we've added this Kates layer with the theory and promise of portability that no one's using, one person's using. Um, and now we need a platform team to manage this portability layer even though we've never actually done anything to be

portable. This is again adopting complexity to solve a problem that I don't have. And then you get into resumed driven development. Um, and that's I need to have Kubernetes experience so that I can get my next job. Oh, that's real. That's super real. And it's not evil. Like, if you need to know Kubernetes because you're like, "This is how I put food on the table." Hell yeah.

Go learn Kubernetes. It's awesome. You should do that. Um, it's real and it should be named. Like, we should say there are people that are doing things for companies because they think it's how they're going to learn and advance their career the best. Whether that's advancing their career at the company they're at or jumping to the next one or whatever. I work mostly with startups these days

and startups that start on Kubernetes I get angry at. Um I'm like why would you do that? You don't even know what your application is that you're building. You don't even know who your users are yet and you're spending a platform team. Like you're building two or three people. You're paying like 600 700,000 a year just for the humans to go build this thing. That is misallocation

of funds at worst or at best. Um, and so the other thing is once Kubernetes is in, no one wants to remove it because they feel like they're being a heroical. Um, and so if your career rewards complexity, who's incentivized to simplify this stuff. Uh, sadly it might fall to management and we all know how that's going to go. I say this as a as an executive.

Um, so you kind of get this cloudnative gatekeeping thing where it's like, hey, if I don't use containers and Kubernetes, I'm not cloud native. And if I'm not cloudnative, I'm not modern. And if I'm not modern, I can't attract any talent. And if I can't Oh, wait. You're still running VMs? That's legacy. I don't want to work there. I thought everybody cool was doing containers. I thought

everybody was doing Kubernetes. Sidebar. Do you ever realize how legacy is a great thing in every other industry other than technology? Yeah, legacy applications are awesome because it means they're currently making money because if they weren't, they would be turned off because you hate them. So, so um so you're using a monolith. Well, you'll never scale. This is a a counter thing I get. Uh I can

tell you that some of the largest systems in the world that you use on a daily basis were monoliths for a very very very long time. Some probably still Reddit for a long time was a monolith. So Twitter for a long time was a monolith. Um and it doesn't mean that at some point you shouldn't pull it apart, but those things had scale problems. That was why

they pulled it apart. it was they found users, they found product market fit, they found initial revenue, all of that kind of stuff. Um, but there's also like questioning the stack is like, well, if I question should I be running on Kubernetes, people start to question me and my credibility. They're like, well, do you even know what you're talking about? Everybody's doing this. That's weird. Um, and

so I just start saying, let's judge by the outcome, not by the labels, not by the tools. And then finally, the common rebuttal that I get is, well, if you think Kubernetes is so complicated, get good. skill issue. Okay, I mean there's some validity to that. I don't play with Kubernetes every day. I find it annoying and I kind of hate it in a lot of cases.

Not because it's a bad tool, but because I have to go through a lot of steps to do the things that I'm trying to do on a very regular basis. Now, in production, do I use Kubernetes at times? Absolutely. It's fine. Do I use it for everything? Absolutely not. So, I hear Kubernetes isn't that hard. Get good. Okay. Um or I hear if you've done Kubernetes for

10 years, it's like second nature. You don't have to think about it. Okay, I believe you. Um but I also think you should understand the systems that Kubernetes runs on and that's the layer that you could actually operate on and it would be simpler and your expertise is if you have expertise in Kubernetes because you've been doing it for 10 years, it's not necessarily transferable to the

next set of people that's working. And so you may have a 10-year advantage or a head start, but the next person has to absorb all of the things you've learned to be successful with what you're doing. That's rough. And so the it's easy for me ends up basically being a survivorship bias. You've you've done it. You've worked on it. You were successful. Therefore, it's easy. And so

the question isn't really can an expert operate this, it's like should it actually require an expert? Um, this was a long conversation I had when I was running a large engineering team and it was the job of a staff engineer is to make the system easier to work on by everybody else. That's the job of a staff engineer. And if the staff engineers aren't doing that, it's

all just sparkling software development. So, so instead of being technology oriented and anchoring my career around saying I love Kubernetes or I don't or I love Nyx or I don't or whatever, I look at outcomes. Are you solving the business problem? Are you getting the revenue you're supposed to get or are you getting the information out? Are you are you learning the things you want to learn?

Not everything's about revenue. It could be, hey, I actually just wanted to learn about how this technology work. Then you should use technology if you want to know how it works. Like that there's no there's no other way to do it. Um or it could be I want to have a home lab and I want to play with all these complicated things. so I can figure out

how they all fit together. Totally valid. Uh but the best infrastructure basically disappears and is out of your way. You don't have to think about it. You're like, I can go do the workload that I want to go do. So what could be different? Network costs. Containers at the edge, full images over constrained networks. Anybody do IoT stuff? Anybody run Docker on like really small nodes way

out on the edge? There's there are a lot of people that do a lot of point of sale stuff does stuff like that. Um, if you have a Docker file and let's say it's like 50 lines long and you change line 25, every layer after line 25 is now new, which means I'm shipping at least 25 new layers out to that IoT device that probably isn't on

the best network connection. There are other ways to solve this problem. For example, if I use Nyx, when I I work at Flock, so I like Nyx a lot. Big surprise. Um, I could change the one package that needs to change and I can ship you that instead of changing this entire image. And so let's say that image is 500 megabytes. Well, let's say the package you

change is three megabytes. I can send three megabytes instead of changing 500 and shipping 500. That's cheaper on network cost. It's cheaper on bandwidth. It's faster. Um, so there's just all sorts of good things there. Yeah, I guess that was the example I just walked through. Um, when bandwidth is is expensive, this isn't just academic. It is a budget. And you start to look at this, particularly

when you're sending stuff into weird regions that have very weird network costs, ins and outs, ingress and egress. Um, how many people run Docker on like super small machines? Anything like less than a gigabyte? Anybody run Docker on things with less than a gig of RAM? I do it. I do it all the time, but I'm just curious. So, okay, few of us. So know your options.

There are some other options here to solve other problems. Uh bare metal VMs with good config management still a pretty good idea. System D units. Yeah, systemd is not without its flaws, but generally pretty awesome. Uh single binary stuff is really cool. You can use Go and Rust and they I like that they say they have no runtime dependencies. Go and Rust run both do have runtime

dependencies. You just don't always know what they are. Usually it's Etsy services um which is part of the Ayanna standard because if it was ever doing a lookup on networks it needs its Etsy time zone is another one it usually needs doesn't always tell you that just know that if you're ever doing things from scratch so um you can run n closures or flux environments these things

are like containers but they're smaller or they're different shaped and they have different properties I'm not here to sell that to you but they are an alternative you can basically have your complete dependency graph computed and have it all the way down be completely hermetic and that means I'm not shipping this OS I'm only shipping the exact things that my call path goes through in my And

so basically ship what you need instead of the whole OS. There's another thing of firecracker and micro VMs. Anybody played with this stuff? Firecracker microVMs stuff is super cool. Um you know when you can pause VMs for milliseconds in the in between transactions and stuff. It's it's actually really really cool. Um AWS built this because containers weren't the right primitive to build AWS. They were like the

isolation isn't sufficient. the interprocess communication doesn't work for us the way we want it to. We actually want to be able to pause things and pause the network layer at the same time as the process layer. Like there's all sorts of things that they did. Um these things that boot up in like 125 milliseconds. Anyway, this this tech is really cool. I'm excited by it. I want

to play with it more. Um yeah, so when the cloud providers needed isolation, they didn't reach for containers. Uh web assembly. Anybody doing a bunch with WOM? Anything Wom? Okay, I've done a little bit. I need this is something I want to spend more time on as well. I think you have some neat deployment paradigms here. Uh Solomon, the guy who created Docker, basically said that if

Wom had existed, he wouldn't have needed to create Docker. What? Just think about that for a little bit. If Wom had existed earlier, we probably wouldn't have Kubernetes today. That tells me that might be something I want to learn more about. You know, these are modules. They start in kilobytes or they're usually in kilobytes. They are small. They move quickly. They start in micros secondsonds instead of

uh microsconds instead of in nano no milliseconds. Sorry, I'm I'm going to get that right metric. I'm American. It doesn't work out super well. Um portable. So these are sandboxed by default with WM as well. Um and it's still early on some of this stuff like some of for for certain categories of workloads. Wom works super well for others. There's definitely still some growth there. Um, and

then of course you still have containers and Kubernetes. And when you generally need it, it's great. Especially if you already have a huge understanding of it. If you're great at debugging it, it's great. Um, and the menu of things that do work is longer than people realize. Not everything needs to be a container. Not everything has to run in Kubernetes to make you successful. Multi-user Linux or

Unix is still a thing. The OS still knows how to do this. It didn't forget. So let's look at a little bit of a decision framework. What problem am I actually solving? Let's start there. Does the container solve it or does it just defer the problem? What's the complexity tax and who is going to pay that tax? Could a simpler tool solve it with fewer layers? And

am I choosing this because it's right or because it's the default? If you ask yourself these questions when you're developing your next set of technology, you may just find yourself at an answer that isn't I will use a container and It doesn't mean that that's the right answer. I'm just trying to make you think about different options. And then with AI, we're going to make all of

these mistakes, but now at new speeds. So, how do I serve a model? I put it into a container. Cool. So if I change one layer on that container, I have to shift that whole model around again. Oh, that's awesome. Uh yeah, these images is 15 gigabytes. I when we were at CubeCon, I was doing a uh a poll of what's the largest image anybody had ever

shipped. Uh 488 gigabytes was the biggest one that I had heard at CubeCon on the floor. Uh so if you've done one bigger than that, I I do want to hear about it because that's great storytelling. Uh but it's just like if you're shipping hundreds and hundreds of gigabytes, what are we even doing here? like that is probably not the way you want to do some of

that. With AI, we're getting basically complexity, acceleration, accretion, 10x the speed, and AI could break the cycle because there's also some help here. What if AI can solve some of the packaging problems? Remember, we went all the way back. Docker really the the value of Docker was I could package software more easily. And packaging meant it was portable and I could query it. I could figure out

what was there, why it was there, things like that. Okay, so instead of put your laptop in production, I could say, "Hey, here's my workload. What do I need to run this in a production environment?" Maybe you get interviewed. Maybe it asks you a set of 15 questions or the five that I gave earlier and then you can say, "Oh, okay. That's the one I want to

pick out the other end." You could make a wizard for that. So what we learn from these 15 years of basically picking a container as a default layer for everything that we're as when when we come through the AI experience. I'm not sure. Yeah. Or are we just going to sleepwalk into the same patterns now that we're doing everything with different tools? In closing, you have permission

to question the defaults. You can question them from your peer group, from the industry at large, from your friends, but you are up against the most powerful force in the universe, which is called organizational inertia. If you've ever tried to change anything at a company, you know what I'm talking about. It is hard. And I say this as a guy who sits in a VP chair, it's

hard there, too. So, um, this is how we do it is not an argument that I want to get into with And it's not about blame. I don't think anybody was ill intented when they said we're going to use Kubernetes or we're going to use containers or we're going to add a debug pod to launch and under these conditions or whatever. Um but if we if we

knew what we know now 15 years ago, would we be making the same set of choices? It's a simplification. That's what I'm asking for, not a regression. This ecosystem is incredible. There's a whole foundation here. There are conferences. There's learning. There's education. There's neat tech. I do not want any of that to change. I just want you to think about is that ecosystem the right one for

every workload that I'm ever going to deploy and use. So the market maturity isn't adopting every tool. It's knowing when you should put one down. The goal really wasn't ever containers. It was shipping reliable software for the people who need to use it. If container gets you there, do it. If something simpler gets you there, do that. So stop asking how do I containerize that and start

asking what does this actually need to run. Please be outcome oriented and resist the poll of the defaults. I am Mike Stony. I run engineering at Fox and uh I think I am done for now. So thank you so much. >> Yeah, we can take a question or two, I think. a bit of a back and forth. >> Uh, I really love your talk. I think that

a lot of us like really resonate what you said um about a lot of it, but I want to ask you a question. So, let's say you're an engineer, right? You're you you want to build you're tasked to build things and you want to do what's best for guests, but you have a set of tools that you're told to use. Um, in particular, my company is looking

to move to Kubernetes workloads, which, you know, I get what they're trying to do, but I don't I think that timeline and giving every engineer engineer and a hammer doesn't really solve the things that they solve. And you address a lot of reasons why. So my question is let's say I get a elevated talk pitch like with the um someone who uh is greenlighting this decision but

he doesn't necessarily know the impact of this. What is a good way for an engineer who really wants the best for the company and guess to explain to him to have him think about the impact of these of this decision to put everything in a container just because I I hope my question makes sense. >> Yeah. No, it makes total sense. Um I think just to summarize

and play it back, it's basically if if all the engineers are doing it this way and that's the set of tools that I'm handed, isn't that kind of the option set that I have? And it might be um but it also could be okay we have one way of doing things we could add a second way which is a complexity so that might not always be the

right answer but it could be we have kind of the lightweight simple way of doing things and the more complicated way for example like where I work currently we ran a lot of stuff on fly.io IO for a while because it was a simpler interface to do a lot of stuff before it ran on a more complicated you know cloudnative type environment and we did that until

we either ran into a problem with fly.0 just didn't handle didn't have the options of things that we wanted to do anymore or um and part of it was also just the network transfer costs of going in and out of AWS and stuff and so we were just like okay we for optimizations of different kind we we ended up changing but if you can run on something

like Fly or Heroku or something like that that's a simpler way of doing things and so you might look at that as an alternative but it also could be okay we just have a big Unix server that we can all use as development and that allows us to iterate really fast and maybe in production you still deploy to Kubernetes that might still be an okay answer like

I don't have necessarily a perfect answer there, but even if you just think about the problem, even if the outcome is still I'm going to use Kubernetes, you've at least thought about it a couple different ways. And you might just think about, okay, I'm going to use Kubernetes in a simpler pattern. Um maybe for certain workloads than having a like full service mesh. Like every time I

I hear the word service mesh, I basically think, oh my gosh, debugging is going to be terrible. Um and so maybe you can do it without a service mesh. Like, you know, there are things like that. So, >> you want to take the >> Can I be terrible and do a a comment and hear here? Uh, one of the apps that I work on, we have a

server. It has MySQL and PHP installed through system packages and we deploy through arsync. It just works. It's great. I have many like that except for the PHP >> all right, that's going to be the one comment exception. No, I'm on to you. >> All right. Well, thank you very much. If you want to talk more, I'll be out in the hallway or whatever. So, thank you.

testing. >> No, no, I'm going to Hello everybody. >> that's okay. You preempted me. That's that's that's good. Um that sets the tone really well. Screen I think just went off. Um >> I'll fix that. >> All right. So this is the last talk. Um so Dave stands between you and your dinner. um director of uh developer relationships at uh Mier Consulting. He runs the developer relations

as well as helps promote ARM 64 servers for all cloud application needs. Uh welcome Dave. >> Thank you. Amper computing uh so not consulting but we're a we're a CPU company. You I'll tell you a little bit about Amper in a second. Uh I'm you're not that interested in me. you're interested in continuous delivery and multi-architecture applications on Kubernetes and how we can make that easier. Um,

so I'm going to go through a lot of stuff here. Uh, but in general I'm going to start with a brief introduction to Ampere. I'm going to talk about how the cloud has changed software delivery and you know the the reason why we have continuous delivery and GitHubs. get g get g get g get g get g get g get g get g get g get g

get ops people probably already know that but I'm going to gauge the audience as we go along and uh I'm going to bring it all together and like run through uh if you've got an application that works on x86 what you need to do to get that application running in Kubernetes on an ARM 64 pod uh so very brief introduction to Ampier uh for those of you

who are unfamiliar makes CPUs uh we design from the ground up uh CPUs with lots of cores So that's kind of the the three core tenants of Ampear. We build products for cloud service providers and for hosting companies for enterprise with high core count, low power draw. Low power draw means um easier to cool it and it means you can have more cores per rack. It means

you have uh more cores per unit of volume. Um so you know the company has been around for seven years. We were recently acquired by SoftBank and this was the vision of the company is is you know we've been working on this x86 architecture uh for 40 years. It's a company that's picked it's not it's a company it's an architecture that's picked up a lot of functionality

along the way for HPC for desktop computing for uh you know a whole range of use cases and a lot of those were not particularly well suited to cloud usage things like for example uh SMT simultaneous multi-threading was started so that you could run multiple um processes or multiple independent proc processes on the same core that's not going to work particularly well if you're renting a core

and somebody else is running something on the same core as you, right? And you end up having a knock-on effect on your on your Uh so why would you care? Like what why is why is Ampear interesting for you? Well, it's first of all good for your pocket, right? If you look across all of the cloud service providers, typically the ARM 64 CPU is the cheapest one

available. There are a few reasons for that. We're giving you more cores per socket, which means that you have less real estate to build to house them, right? it's it's cheaper to operate and you have more more cores in Iraq. Uh so that means that those CSPs even though the CPUs themselves or the systems are the same same or similar proc uh price to an x86 system

um you're only taking a small a small part of it. So you get those you know your core minutes are cheaper. Uh so that's the first uh big benefit is we've seen you know we've seen customers using um Ampear CPUs for their for all of their cloud workloads whether it's video conferencing or you know share uh uh share what right share applications uh running high volume volume

applications and they're saving 20 to 30% on their cloud spend. It's a significant saving. Um, but it's not just good for your pocket or good for your applications. It's also good for the planet, right? We're using less power. Power, I think we can all agree at this point, is the constraint of future uh cloud growth with AI and so on. We're hitting the limits of what the

grid can do. And we've already hit those limits in some areas. You know, there are urban uh regions around Europe that are not building houses because there's too much strain being put on the on the the power grid by data centers. That's the case in the Netherlands, in uh the UK right now. Ireland, my country, over 20% of all of the electricity generated in Ireland is being

spent on data centers. It's kind of unsustainable in terms of its trajectory. So, it's not an abstract concern. Okay. So, there are some really good reasons why you might want to embrace ARM 64 for your cloud workloads, right? um already price but also lowering the power and and operating expenditure if you're running them in your own data centers. But I think there's concern about how much work

might be involved in making that move. Uh it's not a trivial change. It's not, you know, just moving a binary and running it on a different machine. You have to think about recompiling all of your uh applications. You need to make sure that all of the software and dependencies that you care about are available on the architecture. And we'll get into all of that. But before that,

I want to think I want to talk a little bit about how how the cloud changed the way we build software and and the way we build and deploy software. Um, the cloud fundamentally changed the way application development happened. Like I I don't think I'm I mean this is a cloud native day, so I'm preaching to the choir here, but allow me a little latitude. Um, 20

years ago, we were in three- tier architecture. you had your your middleware server that the way you added capacity is you you added more discs, you added more memory, you added more CPUs. Um you had your database with your DBA on staff and the way you added uh sizes more disk, more memory, more CPUs. Um so you had these big monolithic servers that were running large workloads

and that was the way we scaled and uh you know the cloud fundamentally changed that architecture. Um, cloudscale requires automation by default. If you're if you're deploying something to a cloud and you're deploying hundreds of VMs or thousands of containers, uh, you need to be configuring those things independently of like you can't do it onesies and twzsies. You need to have some way of doing that at

scale. So, automation was one of the big changes that was driven, you know, from from 2006 2007. It's been driven all the way through to now. Um, and cloud economics incentivize scaling components on demand. You don't want to pay for something that you've provisioned for a thousand users per minute when you're getting 10 users per minute. You also don't want uh, you know, your application to fall

over if you go from 10 users per minute to 50 users per minute, right? Uh, so scaling components independently based on their needs is important. So we break apart that monolith, right? We're no longer um we're no longer provisioning for maximum demand. we're provisioning for um endto-end service time in fact right and if if and so but that adds a lot of requirements right um you've got

hundreds or thousands of compute elements you need to know what's happening in each of them you need to be able to identify if there's bottlenecks as they come up this is the architecture uh now let's talk about a little bit about a specific example this is one component from Uber I got a So, I've got a new pair of glasses. I I'm going to ask for your

indul indulgence. I got a new pair of progressives. And so, I don't know if any of you have progressive lenses, but it's so you've got a different lens on the top and on the bottom of your glasses. So, if you want to look at something that's like 24 inches away, you need to have to find exactly the right spot. Um, so I've got like I'm having a

hard time with it. So, I'm I'm I I took them off for this presentation. So, that's blurry. This is clear. We're going to be good. But uh so if I look at this this is an Uber application like Uber is the kind of an application where you've got hundreds of services um and each one has their own components. Microservices in general you want different teams to be

able to work on their micros service without impacting other teams. Um they're developed updated scaled independently. Everybody gets their own data store. You try and share as little information as possible. All communication is across message bus or or some kind of uh asynchronous communication of intent. So architecture becomes a little more sophisticated. Uh individual components themselves become simpler right you end up having a single service looks

like oh request comes in I do a thing I verify the state of the state of the system I make the change to the state that's requested I send a message saying that the the update has happened. Um so each service but each service even though it's simple in itself has separate components. It's going to have its own storage. It's going to have its own caching. It's

going to have its own packaging testing test uh um maintaining consistency. You you define the APIs. You define the the the the communication mechanism for talking to it and what you expect in return. Um but the the application itself is is going to be multiple components. So an individual microser might be a single VM but it might in uh consist of independently scaling data store cache uh

and application components. So we need a way to be able to do all of this updating and and and modifying things independently and make sure that we don't break anything. And that's where continuous deployment or continuous delivery became kind of a standard delivery pattern for uh microser applications. Right? Um uh the is it humble? Jez Jeb humble Jez humble I should remember the name. Um so the

idea of continuous deployment is you know again the cloud driving automation you need to be able to define what a compute node looks like in terms of intent and say this is these are the capabilities I require of this compute node and then I need to be able to deliver software to that compute node that has those capabilities and monitor and integrate that into my service. So

we separate the building of the application and the packaging of the of the application from the building of the of the the compute node that it's going to be deliver deliver um delivered on. That's the essential pattern of continuous deployment is you're going to hold host host definition in a different repository to your source code and you're going to dynamically build the compute nodes that your application

is going to run on. Um so we just walked through an example of one of thousands of microservices. How they get devel deployed is essentially you know um now I'm reading which is a bad pattern. Uh I think I covered all this information already. Okay. So what does it look like when you're actually building a cloud application? Well, starting at the build stage, somebody commits a p

a change to a source code repository. That's going to kick off your continuous your continuous integration process, right? You're going to kick off a build. That's going to build and package your software. Maybe you have a gating test before you even get to that, right? Um that you're that you're going to verify that the source code passes some some conditions. So your automated build is triggered to

um to build for a specific target architecture. You've got to provision a build environment. Then you do your build. Then you package for your test environment. Then you've got to provision a test environment. You deploy your package into that test environment. Then you run the test suites, the functional test suites, the um integration tests, the performance and load tests that you want to run before that gets

to deploying to Once you've passed all of those gating tests, then your application goes into a a deployment phase. You provision a production environment that's going to be different from your test environment, different from your build environment, and then you decide what the deployment pattern looks like. Are you going to incre incrementally migrate everything um and use some kind of a canary uh method where you're deploying

just 2% of your your install base. You run your tests on the 2%. you see if it's better or worse and then you go next step you go up to 10% up to 50% whatever or are you going to do a green blue model where you deploy an entire uh deployment of your application run that test against production scale uh and then flip the switch in your

in your network or uh or your um uh service mesh or whatever you're using to manage uh how traffic is routed into your application. uh or do you do something like a test deployment where you're deploying just a percentage you run some kind of uh test to see how how a change operates on a small percentage of your traffic. All of these are deployment patterns that you

can c codify in your in your uh deployment uh in your githops or continuous delivery And all of the boxes that are darker uh, those are the ones that change when you add a new architecture for your for your build and deployment process, right? So, if you're provisioning a build environment and you're provisioning scripts or for x86, you're going to need to make sure that you're provisioning

an ARM 64 instance, you're going to make need to make sure that you're doing so um with that that is tested and working. That requires some work. Uh on building, you're going to need to make sure that all of your build dependencies and all of your build tools work well on ARM 64 that you're doing all the build. Same thing with packaging. Same thing with provisioning test

environments. All your tests are going to run on a different architecture. So you're going to need to make sure that they work on the on the new architecture, etc. So there is some work involved. It's not zero effort to add a new architecture. Okay. Um I don't want to go into too much detail but I that like again in this room who here is using Argo or

is familiar with GitOps as a as a deployment pattern? Okay so I'm preaching to the choir. Um the general idea of GitOps is you know I commit a change into a git repository that commit triggers a build and test cycle. Then I push the new container that I've built into a repository that's for a deployment that triggers a um that updates an application manifest that triggers a

deployment and my application gets updated. That's the general idea. So I'm connecting um I'm connecting a git repository in which my deployment artifacts artifacts are going to be deployed. I'm connecting that to a Kubernetes cluster that watches for changes and does the deployments that I that that I want. Um, again, I I went super fast and I that was high level and and I skipped bits, but

you all know what GitOps is, so I'm not going to explain it again worse. Uh, but here's how the actual practice of doing your uh cross architect your multiarchchitecture build. Here are all the steps that are actually involved. And so this is the meat of the talk. I think I'm going to I I know we have an hour, but I think I've I've scheduled for like 40

minutes, 45 minutes. I'm going to try and keep this interactive, it's kind of a layer cake, right? There's lots of technologies involved. It's one of the things that's kind of challenging, but there are only a couple that actually care about the underlying hardware architecture. So hopefully that makes it a little bit easier. Um you know, you got to figure out how to build ARM 64 containers. So

you got to understand, you know, how to do that with Docker, Podman, Builda, whatever you're using. Um, then you've got the container repository. They're just shipping binary objects. They don't care what the architecture is underneath the the uh the so Docker Hub or Quay or whatever you're using. They don't care what the binary is. They just it's tagged and you query and get and whatever. Uh, then

you've got your Kubernetes YAML. You need to know what's involved in deploying that image to a cluster. That's going to need you're going to need to know like how to create compute nodes for different architectures, how to add those to your Kubernetes cluster and then how to actually deploy to those nodes in your Kubernetes YAML application manifests themselves. This is how we do um oh for Helm

operators customize when you've got multiple uh containers that are part of part of an application you can manage the application life cycle with these. Um and then finally we've got the GitHub oper GitHub GitOps operators which actually don't care about architecture either. They just sort of they're following a recipe, right? So the main bits where there's work to be done um are in getting your containers building

for ARM 64 creating multiarchitecture manifests so that when you pull a manifest it picks the right architecture uh among multiple and then uh actually doing the orchestration figuring out how traffic is going to flow across x86 ARM 64 do you want to have fall back to x86 if there are no ARM 64 nodes available that kind of So it's the application life cycle part and then actually

the building containers and building your software software upgrade available. That's annoying. Is there a new version of Fedora that came? No. No, it's just updates. Okay. So we're going to talk about just layers one and two. Um what's involved in building multiarchitecture containers? So who who here has built ARM 64 containers? manifests, container manifests, obviously the same people. Okay. Um, so binaries are binaries. They're built for

operating system and then built for execution format like ELF or whatever. I don't know, I don't even remember what was before ELF. And then uh they're built for a for an architecture, right? They get translated in machine code. So they they need to work for the target architecture that you're going to build on. Um, and obviously, you know, a containerbased image is a collection of binaries, right?

it's already been compiled for that architecture that you care about. So first step is you need all of your binaries. If you're building binary uh objects, you need those binaries to comp to be compiled for every architecture that you're going to deploy to. Uh and the way that happens in general is you independently build containers that are for the target architectures that you're using. And there are

a few ways you can do you can do that. We get to it on the next slide. And then you create a manifest file which just lists the shaz for each of these containers with the tags associated with them for for example platform in this case. And then when when somebody does a docker pull or a podman um pull it is right. Yeah. Um of that container

uh if they pull the manifest what it does is it says oh you're running on ARM 64. Let's go pull the the ARM 64 image. yeah you you got to you can do in a single step with docker with docker build buildx or podman buildx um you can build all of your architectures um if you do that without having so I'm jumping ahead of myself uh the

normal way you do that the normal way you build a multiarchchitecture manifest is you build for each platform that you're targeting you push those images. Then you build your manifest and you push the manifest. Um that is the easiest way to get this done. It's the way that fits most into uh CI processes. Uh you can also use u um Docker buildex or similar uh to build

all of the architectures in one go. You just give it a list of architectures and it goes and it builds them for each one. Um, if you don't have build nodes defined in Docker and there's a way that you can say I I want to I want to use different build nodes. So if I'm doing I want to do native builds for every architecture, but if you

haven't done that, it's going to do it in QMU or something and it's going to take forever because you're emulating a hardware level. It it's not a good idea. I do not recommend that you do that. Okay, so this is where you're going to need to modify your CI to ensure that you have ARM 64 build nodes that they're being used to build the software that you're

actually integrating the uh building the manifest into your it's a new step that you need to add to your GitHub actions or whatever you're using. I would say there are a few best practices. Um, first I recommend that you build the images separately on native hardware rather than using QMU and building them their own. That I'm going to that's a kind of a common theme. Uh, be

very careful of tagging. I don't know if anyone here has had this issue, but in a text file defining manifests, Linux AMD64 looks really close to Linux/ARM64. I don't know if I'm the only one who's lost a few hours with that, like looking at looking at manifest files with that. and and yes, be careful of tagging. I would not So, there are a number of times where

I've pulled what I thought um a manifest like it without a tag. It's just, you know, it's got it's just got a version tag. It doesn't have an architecture tag. And uh somebody what what somebody had done is overwritten a manifest with an x86 binary. Um so this is in terms of if you're using manif if you're using um quay repository or or uh docker hub uh

be very careful that when you're writing docker images that you're writing them to the architecture version and not the not the manifest. Uh so one of these like little steps that you hit along the Okay. So this is the one that everybody here knows, right? You've um I'm not even going to ask the this is the Kubernetes part, right? We're going to talk a little bit about

what what what a typical Kubernetes uh application looks like. You're going to have multiple services with components in there. Some of those components might have secrets or config maps and you've got some kind of uh compute component right pod deployment replica set. Um and then you've got as um our previous speaker was talking about ingress where you've got your network comes in you you figure out how

to communicate to the right port and and and etc etc. Um so I want to talk a little bit about the compute part right the comp the deployments versus replica sets in particular. Um so I'm not even going to ask. Okay. So this is uh this is not a trick question. This is the first question of three. So we're we're going to go through this one quickly.

Um this is uh some YAML. It's a replica set name test RS replica set. Uh we're going to create two replicas of a spec that has containers. Name hello world image engine x126. After I run this, what am I going to Not a trick question. Fisel two containers. What um uh two containers of engine x126, right? Okay, that is correct. That is not a trick question as

I said. Right. Okay. Here's where maybe it becomes a trick question. If I change that same YAML fine file and I go to replicas 4 and this time I want image engineext latest, right? I I apply this. What do I have? Anyone want to take a guess? Put a hand Okay. So, we've got one person, Fisel, that he says we have two latest images and two with

the 126 who thinks we have something different. Now, it's still not a trick question. That's that's yeah, that's the answer. Good job. Um, so the important thing here is if you're using replica sets, they are uh imperative, right? Which means that at a point where an event happens, in this case, we apply something that says replicas 4, it's going to look at the label and it's going

to make sure that we have four containers with that label running. And it sees that we already have two running. It actually doesn't care what those containers are. It leaves them alone and it just starts two more. Um, that's one of the big differences between a replica set and a Okay, part three. And this is still not a trick question probably for this audience. A a bit

elementary but okay what is running after this? Okay. So I look at the pods I have running. I've got two engine x 126. I've got two engineext latest. And I decide to delete one of the for by by the name of the of the pod. I delete one of the two EngineX126 What do I have running after this? Three three latest indeed. Yeah. Um so what happens

is once I uh kill once I stop the the pod, the replica set uh notices that I only have three running. I'm not matching with the replicas. So, it starts a new container with the uh image template that's defined in the replica set. So, uh replica sets care about the number of containers that are running with that label. They don't care that those containers are all running

the image that you define. Deployments, on the other hand, do care about the images that you you you uh you specify in the template. So replica sets just ensure that a number a certain number of containers with a given label are running. Deployments manage replica sets. So a deployment when you upgrade uh to a new deployment will create a new replica set and it will apply various

rules to kill containers in the first replica set and start containers of the new image of the new template in the second replica set. Um so they come with again this is probably like has anybody here learned anything yet? Oh, okay. Thank you. I was looking for reassurance. >> Yes, it does. Yes. But not everybody who uses Kubernetes has read Kubernetes documentation. Fly. Um so, uh and

deployments uh use come with two important uh parameters. Uh one is max surge. That's how many new containers you create above the target number of replicas in the new replica set uh before you start um so before it stops creating new ones and then max unavailable. So the way uh the way a rolling update works when you when you apply a new deployment is you create a

new replica set and it starts to create containers with the new template. When some of those containers get to running state, it starts to delete some of the containers in the old replica set and it's it it continues along that way. It'll start more on the on the new and delete more on the old until it gets to a point where you've got all of your replicas

are running in the new replica set. So, these two parameters, max surge, that's how many new ones are you attempt to create before you say, "Okay, this isn't working." Uh, and max unavailable is how many you kill in the old one before you say, "Okay, we're we're running short on resources." Um, and those are those are the important parameters that you're going to see when you're deploying

it to a new architecture. This is this is one of the ways that you manage, you know, is this thing working on the new architecture? Okay. Um, workload placement. There are three essential ways that you can do workload placement in um, Kubernetes. So obviously the workload placement, pardon me. Okay, the the workload placement is fundamental. It's really important if what you're doing is deploying to a new

architecture. Um, so you use labels to mark nodes with a certain capability or restriction or just random text, right? It's it's just uh text, but you you select uh you you can use the node selector method to force workload placement on nodes with that label. So that would be a good way to say you know I need a GPU instance I need an SSD whatever it is.

Um another way to do it is with taints and tolerations. So you taint compute nodes. You say this compute node is of this type. It has been tainted with this label. And then in your in your pod deployment templates you say I I tolerate I I can be deployed on on pods of this type. And it uh I believe the tolerations and and I could be wrong

here. I believe the tolerations can be either requirements or suggestions like preferences. Uh and you can fall back to something else. But the one that definitely uh okay so taints can be used to prevent scheduling on things like uh memory pressure, disressure. Um and taints can both prevent new workloads and or they can force the eviction of workloads to other nodes. Um, and it kind of it

helps identify which nodes are prioritized to be evicted. If, for example, you end up on memory pressure, you can say, okay, this one doesn't matter so much. I'll just kick it off to something else. This one really needs this function, this uh capability that I have on this node. Um, nodes can be tainted with a scheduling restriction. So, for example, reserved for certain workloads or with special

hardware. Pods can have tolerations indicating that they require or prefer. Oh yeah. So, so it does have require or prefer uh certain But my favorite is uh affinities, right? So node affinities are similar to node selector, but it's a little bit more complicated. You've got a little bit more nuance in how you define them. you can have required or preferred characteristics and you have those it can

have those characteristics apply at startup or um or during uh execution. So for example when a node gets to a certain um if a node is being shut down right what happens to that uh that workload uh and in particular we use node affinity for uh the architecture. So, this is small up here, but uh uh that's terrible. I don't know what happened there. Uh I think

I know what happened there. Uh so I I was in presenter mode and I guess whatever this is a virtual keyboard when you I mean I mean I I mean we'll go with that. And I guess I will not use the red pointer again. Sorry about this folks. Um, okay, we're in the right general neighborhood. in this case, we've got a we've got a a template. We've

got a spec where we've set our node affinity to have it required during scheduling, ignored during execution. Um, and our node selector terms are going to we're going to make sure that uh created with this template are going to be uh they can be deployed onto ARM 64 or AMD 64 with the preference being the first one, ARM 64. Simple as that. okay. So, we're going we're

going to get to an example later, but uh um layer 4 Helm charts. I'm not going to spend a lot of time on this because Helmart charts are basically just Kubernetes YAML files with some templating to allow you to do kind of namespace specific stuff or uh you know some some modification specific to um a specific cluster or or or namespace that you're deploying to. Um it's

more than that but uh there's not a lot to do in terms of the architecture. All of that is is is already in the Kubernetes YAML. And then we get to the the the last bit that kind of matters which is Argo CD GitOps for Kubernetes. who here has used Argo CD? I I >> Yay. >> Yay. Okay. So, Argo is great. Um, >> it it really

is. >> Okay. Um, so Argo essentially ties your Kubernetes cluster to a Git repository. Simple as that. You've got um you create a context. So, so you get a context from a cluster. you add that cluster context to um to Argo CD uh no sorry Argo CD cluster add cluster cluster name and then create app name and you connect it to the no it's so you get

your context and you connect your context to your git repository um I think I've messed up the commands here I I don't think I copied them right but once you've done that um what you've done is you've create an application that's going to connect uh the context to the git repository and then when something gets committed to the git repository you decide what happens. Uh so Argo

CD is automatically going to detect if the git repository is out of sync with the application and you get to decide oh I'm going to do a manual rollout. I can have an automated rollout. I can have rollouts be dependent on acceptance tests. I can define whether those rollouts are incremental and what the gating is for going to the next step. There's a bunch of stuff that

you can do. And one of the things that I really like about Argo is that it it basically codifies some of those deployment patterns I was talking about earlier. Blue, green, canary, AB testing. Uh it it gives you the ability to run those without having to manually um you know update the service for for uh for your cl for your application or your cluster. Um so again

Argo Argo rollouts uh oh yeah so Argo rollouts are different from Argo CD. Who has used Argo rollouts? This is this is the one that does the the the deployment patterns. Uh so it's custom CRD for managing deployment rollouts. You basically say I want to do this deployment with you know Canary or Blue Green or whatever. I really like Argo rollouts. I think uh and and I

think it's the best way to do multi-architecture applications because it gives you the ability to have that you know one of the things that we talked about earlier is is sometimes people push containers that are they they're supposed to be multiarchch manifests and they're actually x86 you don't break stuff if you deploy that by accident. Um so the way the blue green deploy deployments worked uh at

least for me before Argo rollouts is I have an uh which is um which I think everybody had to change recently because of uh the engine X in ingress controller being end of life but we're not going to talk about that too much. Um so I start with my ingress pointing entirely to the blue service. My blue service is like just managing the front end. let's say

of of my application. Um I start a green deployment and then I update my ingress to point to the to point to the green deployment. So I have um one ingress that's that's basically doing the um the juxiposition doing the the uh I have notes here which I I can't remember which is not the best place. Okay. So, the the green deployment, we we have to make

sure it's complete. Um, I'm going to move on. And then once we've got our green deployment working, once we've changed the ingress to point to the green deployment, then we scale our blue deployment to zero. So, it's a lot of manual steps is the point. Um there's uh if if you're doing this with blue green deployments with Argo rollouts, you've got a single ingress point that's pointing

to a service that's managed by Argo rollouts and then the rollout itself manages the blue uh replica set and the green replica set and does that change of service for you. So it's a one-step thing. You set your image roll out to green. It basically completes the process uh on its own and then tells you when it's done and then you're done. you move on with your

life. Uh similarly with canary deployments with Argo rollouts, you basically configure your ingress. I'm sending so you've got the two replica sets are managed by Argo. You say I'm sending end% of my traffic to the new to the green. Um I can have acceptance tests that run. I can run a kind of uh as as complicated of tests as I would like to te to to run

for functional tests uh to gauge at that point whether I want to go on to the next. I can have a manual step to approve to to go to the next step in the process or I can have it be automated based on test results. Um and this is all managed within Argo. And if you're moving an AMD 64 to an ARM 64 application with Argo rollouts

um using Argo, you basically just have your affinity in the template. You say, "Okay, I'm going to change my Kubernetes uh my node affinity from Kubernetes Arch MD64. I'm going to change it to ARM 64." Um and that changed there. It's it's I know it's tiny text and small screen. and that's and and you're done, right? That's the point. It's kind of a punch line. Um, once

you do this, once once you just commit this change to the Kubernetes YAML and you deploy the new Argo version, um, your and if if you're using something where the application is available as a multiarchch manifest, you have nothing else to do. That's uh kind of the point. I'm sorry. I'm a little flustered. I am ready to uh to take questions. Rotten apples, eggs. >> Pretty good.

Thank you. there are we we do have a lot of developer resources on Amper computing. you can go along to some of these QR codes and and uh uh we're we would be welcome to we would be happy to see you if you are a if you are a an expert in managing multiarchch containers in our Amper developer community. Um look forward to seeing you there. Uh

thank you. Um I wonder if you know of anything like Argo rollouts that works for Damon sets >> that works for >> Damon sets. I can't hear the >> demon sets. >> Damon sets. Um I do not. Um does anybody here know something similar to Argo rollouts that works for demon sets or demon sets? I'm sorry. I'm I'm I'm not the right person to ask. Um you

mentioned flux before is flux just an alternative or does it >> built on top of argo? >> Oh it's no it's a completely different um githops system. So uh flux um was developed by uh somebody called Alexis Richardson >> byworks >> weave works. Yes. Um, so it was an alternative and yes, now it's like a communityrun open source project. >> It's a it's the same idea. You

connect a git repository to a Kubernetes cluster and it monitors the uh the the deployments for you. Um, but it's a completely different implementation. >> Um, the other thing I was curious about is for people that this seems like well I'm going to assume you're you're managing your own Kubernetes. I I like to use managed Kubernetes. Uh like I I if I can avoid running my own

Kubernetes cluster, I I'm happier to >> Okay. So these things are all applicable if you like use EK. >> Yeah. If you're using like Ok or GKE or AKS or or uh EKS, you can run your own Kubernetes cluster. And the one thing I didn't touch on that I probably should have is what's involved in adding AM ARM 64 compute nodes, right? So you can create node

sets node I think they're node pools is the the language that's used in uh in sorry >> I was not going to talk about carpenter but uh I >> yeah um so you create node you create a new node pool at least the way it works on OCI which is the one I'm most familiar with is you create a new node pool and you say I'm going

to create so many compute instances with this many CPUs or like you choose your instance type you say I'm going to create pool create nodes in this node pool um that are ex that are ARM 64 nodes and then you there's a certain amount of metadata that's automatically set in the Kubernetes uh space like architecture equals ARM 64 is automatically set you don't have to set that

but you can also tag those nodes with for example uh let's say you want to run a 64k kernel, which is something you can do with ARM 64 that you can't do with with x86. Um, well then you you would start your instances with a 64k kernel that you would have to make that image available so that you can do that and then you tag those this

is an ARM 64 but 64k kernel and then you get to the you can use the the scheduling rules node affinity to make sure that you're running things that that benefit from that, >> right? Um so that yeah every single hosted Kubernetes has that capability of of creating those node pools and you can do it with self-hosted as well if you if you prefer like a a

Kubernetes or a kind like a kind or a K3S or or open shift whatever you use. >> Well, thank you all for your time today. I hope this was useful to you. I'm I I know it's always dangerous coming to a cloud native event. Everybody knows Kubernetes, but I I hope you all learned something. So, thank you very much. >> You're good. Just in starture. Thanks. Uh,

evening everyone. Thanks for coming. Thanks for being the last 10 people at the the conference for the day. All credit for making it to the end. My name is Alan Jones. I work for the uh in the Azure incubations team at Microsoft as an engineering manager. Um, our team produces and releases cloud infrastructure software. Some of the software that has been released from the team, things like

Kada, um, Dapper, which some of you may be familiar with. And today I'm here to talk to you about one of the newer projects that we've released um, that I manage called DRSI. And Dassi means action in Greek. Jassie was started because of this problem that detecting and reacting to change uh particularly in data is is harder than it should be. And it's probably familiar to people

in this room that uh the situation you have multiple systems, data spread across them, data changes, you want to you want to do something about it, right? You want your system to and you want it to be quick and precise and you want to be reliable. So some examples, this one is, you know, proactive customer care. So, a shipping partner marks a delivery as uh it's it's

going to be late, it's going to be delayed. You want to go through your customer relationship system, your order system, look at who's got an order that depends on that delivery, figure out who's um who's affected, figure out of who's affected, who you actually care about in terms of keeping their loyalty, and go and offer them something um to sort of sweeten the fact that their their

order is going to be late, right? systems are reacting to the fact that this has changed to allow you to improve your service to your customers. Another example, if you're managing uh a field support team, you know, you've got tickets coming in, there might be specialist training uh needed for a high priority ticket. You've got these field agents out, you sort of know where they are, but

getting them to the right place at the right time when is a high ticket and they're available. It's it involves lots of moving parts. So you can optimize that if you if you have a mechanism that can respond to all of those changes coming in, location, tickets, priorities, training, um all of that firing off together to tell you who's close to that ticket that you can send.

And last example I'll touch on is sort of a more modern one. You know, customers calling in with an issue about a missing order. They either talk to a customer support engineer who's using an AI agent or they talk directly to an AI agent and the AI agent looks at the uh you know the information it's got available to it which was last refreshed overnight and confidently

tells the customer um that the um the wrong information. All right. So if you had a way of when information that was entered into your system immediately got fed into the vector database that this uh AI agent was using, you could have told them the the right information. So hopefully you get a sense from those examples the kind of thing we're talking about. And we call these

change-driven systems. And these are systems that take um take action in response to specific meaningful changes or the absence of change, which we'll talk about later in relevant data. And as we go through this, this is probably a problem that you guys have already all solved uh probably more than once. So we're hoping that we can solve it for good. Now, we're going to jump into a

an actual working example of what we consider to be a change-driven system. And I'm going to use uh the example of a trading application that we'll use again later on, but I just want to I just want to make sure everyone's sort of got that sense for, you know, things are changing dynamically and things need to be updated. So, many of you have probably done some trading,

you've got some market data systems, but if you think about a trading application, there's market data coming in. You want to manage a stock portfolio. So you've got a set of stocks. You want the prices to be updating. You want to be looking at the PLL of the uh P&L of the portfolio. Uh you got you want to be trading. So you want to put in orders.

You might want to do limit orders. So you want to know when these orders have filled or when they've canceled, when they've expired. You might want to watch certain stocks. So you configured stock lists and there's prices coming into those. So we can imagine that there's multiple systems feeding into this trading application including like order management system um stock portfolio watch list tracking and market data. this

is a this is a live working example uh and we'll dig into this a little bit more but you know fundamentally what's happening behind the scenes is there's a price feed coming in and it's updating the prices of of certain stocks. There's a database that has um your portfolio in. There's a database that that contains your watch lists. So that the kinds of selections you've made, the

kind of things you own. And then there's a a a system for doing ordering. But as a as a stock price comes in, many things might want to be updated, right? So if if it's in your portfolio, you want your portfolio to update. If it's also in your watch list, that needs to update. So one change can result in in many updates that that you want to

be doing. Okay. Um now the examples I'm going to be talking about moving forward are very much focused around this this idea of UI changes and updates. But Jassis is not just about it's not a UI technology. It's just good for presentations and and visuals. So knowing that knowing that those changes coming in have to affect maybe so many you know, it's you can't just wire the

app up to all those systems. You're going to have to build something. And so you start thinking about, well, excuse me, how do I get the changes from those source systems? How do I get them into my system? Like what is the what is the feed mechanism? How do I retrieve that data? How do I integrate that data across multiple sources in real time so I can

get those meaningful insights and and results? How do I um update all the dependencies for the system? Um how do I get it from the system whatever that system is out to the application out to the UI that last mile? So if this is the first time you guys have sort of thought about building this kind of system you had to do it you're probably thinking oh

no and if you've done this before you're probably thinking oh no not again right so there's a lot of repetition and we we've probably all been in situations where we want to react to changing data. So how do we build change-driven systems today? There are three common approaches that uh we see in the customers uh the partners we talk to and also that we've used we are

guilty of of using these techniques and sometimes they're fine and and you use them and you've got no choice but we hope to do something better. So, if we look at these, you know, approach one is it's the go-to. Like, this is when you want to update some data, uh, realize when something's changed, the first thing people tend to go to is I'm going to pull it.

It's fine. I'll just pull the database. So, you know, with this, you're you've got some process. It's going to periodically query the database, get some results. It's going to compare those results to the previous uh time it did it, figure out what changed, do something, right? So, you write that system. But you know I I saw this uh this quote that I quite liked. It's like opening

the refrigerator every 5 seconds to see if you know food new food has arrived. Okay. So approach two less common but you know if you've got if you've got it people are going to try and use it. So if they've built out an analytics platform capability um you know bringing all that data together it's already in one place it's already in this unified format. It's like why

can't we just detect if that's changed? And so people start to build jobs on top of it that then kick off processes that then kick off, you know, events that trigger downstream systems. And yeah, it does work. And if you're um if you're not too worried about the the timeliness of your change uh activating some process, then it might be okay. But um again, there are sort

of downsides to that. And the third the third sort of option we see very often uh particularly in larger organizations is the use of event stream processing. So you know they got databases they set up a CDC system it feeds into some CFKA streams they got stream processes they got custom jobs these detect changes they're managing the state boom they've got a change and out it goes

to the consumer app. This is great. Um, but it's also, you know, a lot of work and beyond the capabilities of a lot of companies. So, all of those three introduce significant complexity for both development and operations and they focus on they make you focus on plumbing instead of on producing business solutions, right? and they often result in these one-off brittle solutions that inevitably become issues down

the road. So that's common to all of them. Each of them has their own uh quirks and flaws uh listed here. I won't go through them all, but you know polling is is quite wasteful and there's this immediate trade-off of polling interval versus uh recency, right? So you need to make that trade-off. um it doesn't scale. So if your data is very large, that whole pole thing

can be quite an overhead. Uh if you have one of these systems, yeah, it's maybe okay. If you have two, probably okay. If you have 50 of these things going on across your organization, it's not good, right? The analytics platform, you're not going to spin up an analytics platform to do change detection. So you already need to have one, uh most likely. Um, so it's cost prohibitive

to actually use this unless you've already got it. There's the ETL lag. So whatever the timing is on your ETL into the system, that's going to detict how quickly your events can be uh detected. And event streaming, it it gives the the fastest most realistic sort of answer to change detection, but it's heavy weight, right? There's a lot of moving parts. There's a lot of technology. going

down that route is a lot of effort to um commit your engineering and your DevOps when all you really want to know is has something changed right that's all I want to know so there has to be an easier way and what we really want is something that has these properties you know it's purposely built to just do change detection to just help you build change driven

architectures right? Easy to use it because it's doing one thing repeatably. You can sort of deliver it to have no code, low code, full code Um it combines the idea of I want to detect change and then I want to do this with it. So combining that detect and react with minimal infrastructure and integrating the data wherever it may be for the purpose of solving that problem

and also a rich a rich ecosystem of connectors to get it to the data that you to get data from where the systems you want and to places that you want. And of course we want it to be open source communitydriven and without vendor lockin. Believe it or not, Dressi is the answer. So what is Jassie? It is a data change processing platform for building changedriven solutions.

It is open source. It is released under the Apache 2 license and it's already in the CNCF sandbox. Um so it is available today. You can go and play with it. um core concepts. These three very very simple concepts um at the heart of everything to do with Jassie. Sources, continuous queries and reactions. Sources are the things that represent the data sources uh that you are connecting

to and bringing data from. You've got your continuous queries in the middle and the reactions represent what you want to do to act on those changes. that can be integration with downstream systems or do some bespoke logic that that is a particular action. So we'll go through those in a little bit more detail. So first sources these manage these are responsible for managing the connection to a

external source. Um it could be a um we'll go through them in a minute but it's not just databases there are other sources you can connect to. it receives or retrieves the changes that are occurring in that source and ideally that's from like a change feed but we have done certain sources that don't have um access to a change feed. So some sources mimic you know through

those nasty mechanisms that we we talked about before but they keep it hidden from you so you don't have to implement it but ideally we come from a change fee so we don't put load on your database. Um the source is responsible for for transforming that inbound um format from whatever feed uh it's it's processing transforming into a consistent property graph. node relationship model that we use

internally that we do our graph queries over and then it's done that transformation and it sends what we call a a source change event to any queries that are subscribed to it. So you can see down the bottom here we've got multiple um sources active. They come into a source, multiple data sources active, come into a source, it gets turned into this source change event. If you're

familiar with the deeszium formats, it's very similar. So within it, it talks about what's what's been inserted and updated and deleted and then it eventually gets to the continuous queries that want to uh subscribe to that source. So we already have a fairly rich ecosystem of sources available. So we support relational data um Postgress, MySQL, SQL Server. We support graph and document databases. So Cosmos data versse

kubernetes we treat as a as like a document graph type database for its internal configuration and stream systems. So we started this project focusing on databases. We started actually focusing on um graph databases the ones we talked about there and it was like hey let's integrate relational databases and now it's like okay let's integrate um stream sources. So, event hubs, HTTP, gc and also there's a full

SDK with tutorials um with which you can write your own sources. Also, we are just working on uh an AI agent that we've had a lot of success you tell it the source you want to connect to, point it at its source documentation, it will go off and generate a full source implementation for you. It's It's pretty good. Um, we're still knocking off the rough edges, but

we've had some really good fun with that, and it makes it super easy to get a first draft of of a sources have got the data in, pushed it, made it available. Continuous queries are next. They get the the data, they subscribe to one or more sources. So, it can be subscribing to many sources of to use in a single query. And this is one of the

unique um features of JRassi that it makes that really easy to do. The queries are written in open cipher or GQL. So they're both graph languages. Um we have done an experimental SQL implementation, but it's not something that we've decided to pursue for the time being. Um and what it does is you've got this query, this continuous query running, and you know the secret is in the

name. It's continuous. So it's it's taking each change and figuring out how that change that it's got from a source has affected the result set. Right? So it will maintain a perpetually accurate result set based on all the inbound changes that it's getting. So if you were to ask it at any point what is the current result set, it would be accurate based on the last uh

inbound event it received. Then it uh so it's managing that internally. It's got, you know, set of indexes to manage the elements that it needs. It manages aggregations. We'll talk about that in a little bit. It manages the current result set. And it figures out the diffs and it sends it off to all the subscribed reactions. And the thing that makes this workable is because, you know,

that we've now turned this this notion of a of a change a semantically rigorous structure. Right? So anyone that receives that knows, oh, if this contains something that was added, well, it's been added to that result set. Therefore, it's now part of that set of things that I was interested in. Right? So, um, some examples and these are just going to be like simple queries. Uh, so

if you're familiar with with it, this is a cipher query. So, I said that we model everything as a property graph. We support GQL and cipher. So the match is sort of saying I'm interested in stock prices. So if the data is coming from a graph source, it's going to be have a a node label called stock price. If it's coming from a relational database, it's going

to be from the table stock price. Right? So each row in a relational table is a node in in our property model. So this query is interested in all stock prices that it's that it sees and then it's just going to return a projection from the stock price. It's going to return the symbol, the price, the previous close which is straight out of the the the stock

price record and then it's going to do a calculation to turn out the change percent. So the result of this continuous query is going to look like that and it's always going to look like that with slightly different data in because as new stuff is coming in it's going to update this and if you ask it at any point in time you're going to get that right

okay perpetually accurate. Second example little bit more I'm interested in all stocks that have a price higher than yesterday's close. So again top line match here we're saying we're interested in stocks that has a price to stock price. So we've got this relation. So has price is the label of the relation between those two nodes. And of course if if we're bringing data from a relational database

we just bring in the relations as they come. If we've got oh sorry graph database if we bring in data from a relational database we provide a mechanism for you to define synthetic joins between those those things. We don't currently try to create those relations from foreign keys and and all that kind of stuff. We want to keep it super simple and I'll show you the syntax

we use but we allow you to create synthetic joins between those tables and that's also the mechanism that you can use to join across multiple data sources. Right? So in this one we've got this wear clause which makes us a little bit more selective than the previous. Um so here we're saying that the the price today has to be you know higher than the price of the

previous close. So this is going to reduce the set of elements. So we're not we're not going to get all the prices. We're going to get the smaller subset of just those that have a better a higher price than yesterday. And again, we just project out um some stuff from that and its result set if you asked it would always be this and every time it gets

a change, it's going to update it. So it's perpetually accurate. Now the last so they're fairly simple queries, but you know they show that you know I can collect some data about a set of things I'm interested in. I can create a selection a filter on that. So it's a reduced set. And then we also support the idea of aggregations. Um so this one all market sectors.

So I've got this matching of stocks that has a price to stock prices. So as part of the stock it's got a a sector associated with it. So in uh in cipher any any non-agregating function um gets pushed up and becomes the grouping key. Right? So in this one, we've got sector. It's the only one. So I'm returning the sector from the stock. So it's now going

to group everything else by the sector. So I'm going to get the count of elements in there. I can get it the average of the price. I'm going to get the sum um of the volume and I get the minax price. So if I do this, I'm going to have a table that looks like this. And it will always look like that. And it will always be

Um, and being able to do this these aggregations over continuous queries by just writing the query, you're not having to manage any of that historical state. You leave it to Grassi. It just handles it and you write the queries All right, moving on from continuous queries to reactions. So reactions are what act these are like the the hands of Jassie. I saw something sort of say the

other day. Um, you subscribe to one or more actions. Yeah, sorry. Continuous queries. Um, these receive from the continuous queries the result change notification which is the set of things that's changed um recently in the the result set and then they act based on that. And most reactions are built so that they handle added, updated, deleted changes differently, right? So sometimes for example um you might use

one email template for things that have been added and another email template for things that have been deleted. All right. And when you write your reaction, you define that logic and how it's going to handle those changes. Um ecosystem of reactions. We already have a bunch of reactions. So things that push data out into the world. um we can push using web hooks, SSC, uh gac signal

R and we use signal R for UI updating um and I'll show you I'll show you that in a minute. Um we have an ability to write straight back. So you can have a reaction that uses a stored procedure to update a database. So if things are added, updated, deleted, you can structure your your commands, do something to a database. We do that with Postgress, MySQL, MSSQL,

uh, Gremlin, Quadrant integration type things. Uh, we have reactions that integrate with DZium, event grid, Dapper, Graphana, and again you can write your own. And again, we've got SDKs, tutorials, examples, and we've also got this uh AI agent that is also doing a fantastic job at writing initial cuts of of reactions. So quickly um yeah very quickly uh okay highly extensible I've talked about a few extensibility

points sources reactions bootstrap providers so when a query starts it needs initial data state stores index providers like it's all pluggable middleware so sometimes stuff's coming in from a source the query needs it in a slightly different format transformed elevated elements we've got a whole bunch of middleware that a query can define and um it things that are coming into it will get pushed through this this

middleware pipeline before it has to figure out the changes to its uh to its query. So you can do quite complex uh sophisticated things multissource queries talked about this u we can join across heterogeneous sources just by writing a query. Okay, absence of change. This is a biggie because many systems you can't do this very easily, right? How do you like how do you even begin to

explain that something hasn't happened, right? So, we have this this way of writing queries that have some special functions that say, you know, look in the future and if something isn't true, then, you know, fire off, right? And so, we can build things that say this was expected, but it hasn't happened. So we're going to you know raise a a notification some temporal functionality uh things like

you know you can look back at the historical value of elements you can look at back at the result set at any point in time if you enable the temporal um result store which can be useful although I you know I say this is not a UI technology I always demonstrate the UI integration with Jassie because you can do something very cool or what I think is

very cool and you'll see it soon and be able to judge yourself. But, you know, imagine a lot of applications are all about I've got this set of things I need to show the consumer, the user. Um, you can think about the set of things they're looking at usually as a query result, right? The set of all customers that have overdue invoices. That's a set of things.

I can tell Jassie to create that set of things and then I can have it perpetually managed in the cloud and I can basically data bind you know a control in the UI to the cloud-based result set and keep it in sync. So and I'll show you an example of that. Okay, deployment options quickly. Um Jassie for Kubernetes. Uh this used to be called Jassi platform. It

was the only thing we had uh you need to deploy to Kubernetes. Um it can be in Azure, can be in um AWS, can be on premise. Um, but it, you know, it was it was pretty heavyweight and a lot of customers were like, "Oh, I'd love to try it, but it's like I don't want to build a Kubernetes cluster or we don't use Kubernetes." Like, what

other how can I run it otherwise? So, we we've spent a bunch of time recently refactoring the whole of Jassie um to offer this this Jassie lib idea. So, you can take the entire functionality of Jassie and stick it straight in your Rust application. So all the continuous queries, all the connectors, you can just embed it in your application and get all that all that power, right?

And then on top of that, we built a single process or docker imageable um host for Jassi liib. So now server is a standalone server that provides you all of this. You can get it up and running in in minutes, right? And it's a wrapper around Jassi lib that provides you know persistent configuration rest management API etc etc. So Jassi server is the best example of how

to use Jassi lib if you're interested in in having a look. All right so let's get back into this example while we've got a little bit more time. All right so coming back to that trading example. You got all those things going off on the screen all those updates coming through. This is this is what it would look like as a JRassi solution. So we're going to

use Jassi server and I'm going to show you this running. Um inside Jassi server we're going to have three react uh three two to databases and they're both Postgress but they could be you know one could be my SQL or or something else but they're both Postgress for now. One database has orders active orders buy or sell. The other database has things like your portfolio, your watch

lists. Um, and then and the the stocks, you know, that are generally in a system. So, changes to all those things are going to come through those sources. And then we've got a price feed which is mimicked by this background Python process which is just spitting out uh price data into this um HTTP source, right? So, we've got this open HTTP source. The price generator is sending

events, you know, that are price events. These come into the query engine. I think we've got like eight or 12 uh queries running and then all of those queries are spitting out through this SSC reaction that the trading application that I'm going to show you is receiving all its data over. So it's receiving um all its data from through one connection to Jassi server through this uh

SSC reaction. Now just because it's a demo um there's some sort of short circuiting going on. I'm going to be able to place orders and change my watch lists and this kind of stuff. You can imagine this happening in other systems or or through other things. The trading app for the purpose of demo is is hitting this like little mini uh broker API that's going to directly

update the database. So when you see me make a change in the UI, none of that state is being kept or updated locally. It's all roundtpping through the database and through Jassie. And then um the Postgress uh reaction is down there because it's a Postgress stored proc reaction that doesn't exist on Jassi server yet. It's on Drai platform or Jassi for Kubernetes. If it was there, I

wouldn't need some little hacky things I've done to make this work. I could just be updating the database directly from here and I could take some of these some of these components out. So it' be a much tighter um loop. All right, so let's jump in. Here's this trading app again. Okay, let let me show you some things. Each of these um each of these panels is,

you know, you could imagine it as a as a query, right? So maybe this guy, my watch list, it's like um I want all the socks that are on my watch list and this is the query result. So each of these is actually effectively bound to a query that's running in Dassi in Dassi server and it's running locally of course but it could be running in the

cloud. So and this is this is an app we just used to to demo. So it's got some features in the um just for the purpose of deming. So you can see as the as the changes are coming in you know we're highlighting the you're highlighting the rows that change. So there's not a lot of changes necessarily coming into my watch list because I'm only looking at

three. But you can see in that full UI, there's a fair few changes coming in. But we can look at this and if I expand this, this is showing me which query, what query is behind or feeding this particular window or this panel. So you can see it's, you know, it's got an ID, the watch list query, excuse me. Um, it's a cipher query. It auto starts.

Here's the query text, right? This is the actual query that's running. So, it's, you know, it's a pretty simple one. It's just doing this um match between I've got this concept of a watch list node and a stock is on the watch list and then the stock has a price. So, I'm now joining across um two data sources here. Three tables, two data sources or two tables,

one feed. um and all of those these uh these relations the on watch list and the has price these are not coming from the source databases because they're not connected. So this is this is the sort of synthetic join syntax down here. So we let you specify for a particular node type or label um this property is going to match this property on this other label. And

if that's true, we create this synthetic join between those two nodes which let us do the graph mini graph walks to solve this query. Um it's just a direct match currently but you know it it can be extended to you know uh do some sort of manipulation to do compound keys or or uppercase or prefacing or something like that. It can be added. We haven't done it

because we've not needed it. So um so that's that query. If I look at uh let's look at the sector performance. So we looked at this query earlier. Oops. We looked at this query earlier in the examples. Um yeah it's called sector performance cipher query. Here's the query. We we sort of talked about that. Um and here's my synthetic join. So again it once you learn this

you can create queries that connect all sorts of data previously unconnected and create these perpetually accurate queries uh result set. Now what's cool about this I said everything's coming across this SSC feed into this web web app. Um we've also created a a component and uh for our signal R reaction and Jassi platform or dressy for Kubernetes um we've got a react component and a view component

that are available and they operate the same thing same way. This is a this is a react component that is um written to take data from the uh serverside event inbound mechanism. And so all you do is you say, I want to use the query table component and I want to I want to connect it to this query ID. I want to use these columns from the

And this whole user interface, this dynamic user interface is effectively built by using that same panel again and again. there's there's absolutely no um there's no state management code going on in this UI. So, if I want to add something to my watch list now, I'm going to add this. It's going to go through that that sort of janky little um API that I spun up. It's

going to update the database. I'm going to add Boeing. And like instantaneously it went through the the API, went into the database, came through the um change feed from Postgress to its source, got processed, it went to the continuous query, updated the query set. How do I know it updated the query? Because it appeared here. But I can also in this screen I can now go and

this is the drastic server um web UI. Now this is hot off the press. So there's some little quirks in it, but like no one outside of our team has ever seen this before. And so what this does, it shows you on the left you got your sources. These are all the continuous queries in the middle and then these are the this is the reaction out here

on the right. So this is an SSC stream. I can click on these things. I can see what's you know connected. I can control everything. I can build new queries in here. I can I can create these systems. I can clone it. All that kind of stuff. So I can interact with it. Now, we were looking at the at the watch list. So, I can go down

here and say there's the watch list query. Let's expand it. Let's have a look at what's inside. And that's what's inside the the query result at the time. Right. So, it's a good it's a good tool for looking at what's going on, building out things, connecting, trying them really fast, um, and seeing if it all works. Now, we've also got a VS Code plugin. So, if you're

into VS Code, it plugs in. It lets you do um a lot of inspection about what queries are running, let you deploy queries, etc., etc., but it's not like this visual gooey thing. It's it's sort of um you know a click things and see trees and see data, but it's also very useful. Um where are we for time? Okay, I'll Okay, very good. So now let's go

back to the UI for a second. Now let's talk about these orders. So currently you know there's uh you know four four orders in this uh table of various status. Some are filled, some are expired. I'm going to place a new order. And this is this is the non-event stuff. I'm going to place an order and nothing's going to happen. Now, I've got a query that says

after nothing happening to this order for uh the half the time of expiry, set it to stale. Set its status to stale. And then if if I've got another query which is any query that's re reached its expiry time, expire it. Right? So, set it change its status. So these these uh let's look at those queries. So there's an ex uh there's a stale order query, right?

So currently it's empty because there's no stale It's loading. I don't know. I've never seen that before. Maybe it's because I'm not on the Wi-Fi. I don't know. the query would be in here. Um, so what it's doing is it just sits there. There's no UI representation of the stale order query. It's sitting in the background in Jassie watching orders to see if they've not actioned for

um half the expiry time. Similarly, this one down here, this the um the expiring orders, it does the same thing, and it does the same thing. Okay, so it again as I said it looks for things that have expired and sets and updates the status. So these are these are sort of workflow queries, right? So they're pushing this thing along through a state uh a state machine

if you will. So let's go back and see this in action. Um so add a limit order. buy this. So, we want just prices, random stuff. I'm going to say, you know, it expires in 10 seconds. So, I'm going to create the order. You're going to see it appear in that table immediately. So, it's appeared. It's pending. So, if we wait patiently, oh, it went to stale,

right? So, the the the stale order and now it's expired, right? So, like the order wasn't actioned. it stayed in an uh in a um pending state too long. The query when it stayed in there too long, it appeared in the um uh order result set. It that notified a process that I've got running in the background which would have been the Postgress um reaction but we

don't have it. So I sort of hooked something up. It then just writes back to the to the API which writes to the database. It's gone from pending to stale. And then that process sits, you know, the expiring query sits and waits. At some point in the future, it appears in there again. And um pushes it along. I wish I could show you those queries, but I

I cannot. Um I don't know why it's not loading. Okay. Um, what else here? I think that's probably it. Okay, back to the deck. Okay, so why Jassie? We want Jassie to be the simplest and quickest way to create change-driven architectures. Um, why do we think we can achieve this? Because it's purpose-built for doing that one job. um a query and some configuration replaces a whole wide

set that is still you know it's awesome for other jobs but for this job we want Jesse to be the best. Um it's simple in both its ability to be understood and used. Um it can be used in a in a low code, no code, full code environment. We want to provide all those options. Um it brings together that change detection from the sources and continuous screws

and the reaction the doing something. So it's bringing those together and also we have this ability to detect when things don't happen which would require quite a bit of um engineering for people to do to replicate that that functionality in other systems and we can run anywhere is um but single process Kubernetes uh in Docker embedded and we are CNCF sandbox Apache 2 open source and communitydriven

And if you want to uh get going, all of these resources are available to you. The Jassi website, we've just re-released that, updated all the information. So it's broken down by the three products we have now. Dressy for Kubernetes, Jassi server, Jassi lib. Um we are, as I said many times, open source up on GitHub. You can come. Um it's it's really awesome because we have started

to the community's actually started to to grow and become engaged and people are submitting issues and PRs and it's it's really great to see. Um you can the getting started on the on the website gives you step-by-step instructions of how to get started. Um maybe I'll just jump there. No, I don't have Okay. um how to get started in a container, dev containers, code spaces, as many

options as as we could do to make it easy for people to try. There's a there's a tutorial that walks you through, hey, try this. See this? Is that enough? Do you want some more? You can keep going, learning more and more of the features. So, we're trying our best to um make it easy to to adopt and learn. Um we're on Discord see that link and

on the website and you can follow us on any of those um any of those systems. And with that I can open to questions if we've got time. >> So my question there are a number of answers to that. oh how do we scale DSI? Um so the Kubernetes um Gracie for Kubernetes uses the Dapper actor framework and each query is effectively an actor or each query

host is an actor. The query host can host multiple queries. So that's a way to scale. Um in you you can host multiple JSI libs. So addressy liib can host many queries, multiple queries. We don't know the limit yet. We've not done that performance testing. Um but you know there there was 13 in that one and it was it was fine. So each instance you can push

multiple queries in. You can have multiple instances. You can have multiple JSI servers. So we've we've sort of started introducing Dressy Server to some of our active partners now who weren't using Kubernetes and they're like oh great let's put an orchestration framework around Dressy Server and we're like ah well our answer to that is sort of Kubernetes but they don't want to use Kubernetes. So they're thinking

about how do we get this into other container or orchestration platforms. So Um so the question is is are the components are the connections stateless? Um how do you mean sorry can you clarify? >> Mhm. >> right. So some reactions reactions initially wereend intended to push out into the world to do something like to push downstream. These reactions that open connections and allow inbound traffic is a

is a newer concept. So the outbound one is an easy answer. It's just like it's just pushing. But in terms of how do we manage sessions across the the reactions that expose ports, we don't have an answer to that Hello. Hello. Can you hear me? >> Yeah. Okay. So, I have possibly three questions and maybe follow-ups based on those answers on the question. First one um only

because I know what um cipher open cipher is. Is there a database in is a is there a graph database as part of the Jassie environment? There is not. Um we we maintain minimal indexes of the elements that we get that are part of or potentially part of the query result. So, um, if you when we get the change from this the source, we're going to get

all of it, right? It's going to send us everything. Our query knows we only need these these four properties. So, one, we just trim it down to the minimal set of stuff we need. And then we index that node and we index the relations. So, we're not using a graph database. we have just like an adjacency list serialized to to disk that we use to and the

nature of our graph solving it works quite well for us. Um so we've never hit any any limits on that but it's certainly somewhere that you know in the future as loads increase and we get more we might need to relook at but we don't have a graph database under the covers. >> Okay. Um the second question has has to do with sources. Um you mentioned you

I think you you had a few but you didn't have quite a lot of sources um for Drazzi but you mentioned another project uh called Dapper. >> Um so I was wondering which has a lot of sources independent even of the cloud. >> Can those sources be used in Jurassic at all? currently no but um we we have done some pretty good integration with with Dapper and

we're always looking for ways to increase that and particularly like with Jassi lib we're thinking about ways of you know uh you know maybe building Jassi lib into some kind of dapper components and sort of providing those capabilities but looking at you know dapper sources is a is a great is a great thing we should be doing >> yeah I mean it gets you to more sources

without having while you wait for Jassy sources to come in. >> So if you can do that then yeah. >> Yep. >> Um and then my last question is um how do you when the sources have sensitive data and that changes how is that going to be handled in Jassie? >> Um as in you're talking about data permissioning? >> No. Um say patient data >> sort of

thing. uh you know like I said sensitive data so um if that source has sensitive data and it comes into JRassi how are you what's it doing to >> maintain compliance for the matter so it's not something we have focused on uh as we've been trying to build out this technology um we've thought about some things that we need to do in ter terms of the security

model like always encrypting in flight data and and our indexes you know should be stored encrypted and and all that kind of stuff. So, we could we could do that very easily. But in terms of wrapping it in a more secure and and compliant and something that would pass the rigorous test, you know, we've not even gone down that route. We're just trying to kickstart this this

idea and and you know, the first um period of time without just really validating that, you know, this is something people are interested in because they want an easy way to detect and react to change. I think we've got there. So now we're trying to do the grown-up stuff and and make it, you know, broadly adoptable and faster and more secure. So security and that kind of

stuff is definitely on our road map, but we've not got there yet. >> Hello. Thank you. Uh I was curious about the failure modes and specifically like what's the responsibility of the application versus dassi? >> When you say the application you mean the thing downstream of the reaction. >> Yeah. Um in plat it is the goal to get it to a a very easy mechanism of the

app saying I last got data at this point and give me everything from now right so if that's I've never seen data before give me it it should make available a way to get the snapshot it needs and update from from then on. Um so platform has this ability through sequence numbers and being able to ask for the result set for a query at any point in

time. So then it's the onto the application to say well the first event Jasse sent me was this um LSN this time stamp. I now get the results while I'm collecting the inbound stuff. I build my base set. I integrate the uh the um results into that and and the you know the view and react components that we've built and made available sort of do this automatically

but you'd have to do that logic yourself in your app if you wanted >> So I just want to make sure I understood that response. So that let's suppose the Dassi server took a shutdown and recovered. >> Um it's going to try to recover state, >> Uh but you're saying the app is responsible for some of the housekeeping after one of those. >> The app is responsible

for asking Dassi for what it >> got from the point it last saw it. >> Internally, Dassi does something similar. And so in the Jassie internals, the source is responsible for being able get what Jassie needs to to bootstrap and to um update its indexes if it went down. Right? So we internally and this is a piece of work that's actually unfolding right now. Um you know

we're making those there was multiple sort of indexes in internally in Jassie. We're making sure all of that is transactionally uh consistent. So if Jassie falls over Jassie server in particular at any point before it's finished processing a change when it comes up again it's going to say hey source the last thing I got from you was was this uh LSN and now the source is responsible

for giving grassy what it needs and the same mental model is is true of what we expect the application to do for us um ask us for what it saw >> okay that's a clear explanation thanks Um, I've got a question. How would this replace or interoperate with an existing event driven architecture like something that's driven by message cues? >> Yep. Um, so I mean you the

thing with events and yeah, Jassie is an event- driven system, right? Under the covers we're doing events. The trouble is again we're trying to come up to a higher level introduce consistent abstractions that people that makes it easier for people to do this one particular problem because with an event stream it's like here's everything figure out what you want right Jassie is like tell me what we

want and I'll tell you when it's changed right so you could you can use uh some uh event streams as a source already and you know We so CFKA is one Reddus we've got. Um you can use those as sources. There's questions about how you recover from some of those because you know not all of the not all streams are going to have an ability to look

back in time, right? So maybe that source then has to also implement some kind of cache of its own or or sort of like a a history. Um but you can use them as sources is is really the answer. And also you can spit um dressy change notifications out as events right into your and then whatever is downstream can can process it. So it I think it

both sources and reactions can integrate with existing events eventsing systems. >> So your demo example with the stock prices looked looked interesting, cool, maybe something I'd like to do to learn it. Uh, what would you do to set what available stock feeds? I'm talking about like live stock feeds are out there that could be hooked up easily to >> Jasse. Yeah, I mean I used just as

a bit of an aside, but along those lines, I used our AI agent the other night and said I wanted a a Bitcoin um feed source. And I said, I just want it to give me, you know, uh I just want to know about when new blocks uh are mined. I don't want all of the transaction information, just just new blocks. And it did it right. It

it just did it. And I was like, whoa, like magic. And uh so what what feeds are out there? I don't know what's you know we we should provide maybe a list of of things to hook it into, but I would say search for a free feed and and try our AI agent which is also already out. Um it's just a prompt, right? Try the prompt, see

what you get. >> How does this uh AI agent compare with the notorious open clause? It's something like that where you're >> No, this is very specific. It's it's like you've got a you've got a developer who's um written tons of sources, knows how to write them. Um and you just say, I want a source that does, you know, I want a source that integrates with that

system there. You know, here's its API. Um here's its documentation. And you know what it's going to do is yeah it chugs away for a while but it it like it does an evaluation of is this a likely thing and does a prototype and validates that the prototype can actually work and then it so that's like the planning phase and then you implement and it goes and

uses all the stuff it's it's researched and and the um and the prototype and builds an actual source that conforms to you know all of the APIs that we use and all the best practices that kind of Is this uh AI agent a cloud hosted paid service, a cloud hosted free service or something? I'm stand up on my own. >> It's a prompt, right? Okay. You can

just put it into We use C-Pilot a lot. Um I've used Claude. Uh you can just you might need to tailor it because it's sort of focused on how C-Pilot works at the moment. You could tailor it and give it to Claude and Claude would probably spit out a source. >> Okay. Thank you. Any other questions? Uh please join me in giving a round of applause for

this uh great person. Thank