Observing Chaos: Real-Time Monitoring of AI-Driven Kubernetes Destr... Josh Halley & Ricardo Aravena

This talk focuses on chaos engineering, particularly using the Chaos Monkey framework to introduce system disruptions for testing resilience in cloud-native architectures. The speaker discusses their experience with various technologies, including PyTorch for AI model training, Kubernetes for container orchestration, and OpenTelemetry for observability to track performance metrics. They explain the complex setup involving multiple Kubernetes pods dedicated to AI agents playing a modified version of Doom, which serves as a platform for experimenting with these technologies. Additionally, the session covers the integration of observability tools like Hubble and Tetragon, as well as discussions on player behavior modeling and the importance of metrics in training AI systems. The overarching goal is to demonstrate how chaos engineering can enhance the reliability of systems in production environments.

Welcome everybody. Who's here for the first time at CubeCon? Oh wow, look at that. Well, I hope you're having a good time and I hope me and Ricardo won't ruin your time here today. We're hoping that you're not going to can come back and say that was the worst 30 minutes of my life. So, Ricardo and myself are presenting on chaos engineering. I'm sure you all saw

from the abstract that it's going to be an interesting session. So, buckle up and uh yeah, get ready. So, a little bit about myself. I I've been working with cloud native for the last eight or 10 years. I've been a lead in Tag runtime in a CNCF. I'm currently in the CNCF TOC and I worked in infrastructure for the last 15 years. And my name is Josh

Halley. I'm a principal architect at Cisco. Um, my focus is in the CTO office on emergent technologies, up andcoming technologies, nent technologies, things that don't necessarily exist that well yet. Um I am one of the co-organizers of the CNCF technical community group for artificial intelligence um and active in the space with different deliverables and some of the projects. So all right so let's we could get started

with uh defining some of the basic concepts around the the problem we're trying to solve or the the little game that we're playing. So, uh, some of you might be familiar with this, but some of you might not. So, we're using this chaos monkey framework that was originally created at Netflix back in 2011. And yeah, it basically allows you to introduce chaos in your system, in your

infrastructure, you know, kill things, uh, introduce delays in your network and and do all kinds of crazy things, right? So we we're also using the PyTorch project to host our players. Our Doom players are are actually it's a model running in a Kubernetes cluster and also being served by PyTorch and the model that that is being used is trained by PyTorch as well. So some example actions

that uh you can do with the chaos monkey and just to give you a little bit of context. So you can introduce high CPU usage uh you know issues with GPU memory latency in the network outages you know things happening out of order uh so all these different aspects in in infrastructure so it's really useful for you know for testing your your system and the resilient in

your system and additionally we are using a lot of the cloudnative uh technologies one of them is open telemetry uh you probably heard of open telemetry so we primarily ally we're using this for metrics traces and logs and then we're also using that to gather things like span ID traces and events and and profiles. Um then for the model training we tried several things and we settled

on these three different things uh for impala was best for scaling and distributed uh learning. Then we used something called RestNet that was actually the best for visual perception for improving seeing for each one of the players. Uh we use something called LSTM long long short-term memory and this is best for memory and temporal context. Uh so improves remembering. Uh additionally we use noisinet uh and this

is great for exploration and and trying some new behaviors. So we want players to try these new behaviors. And finally, we use this um uh methodology called dueling heads. And this is great for value estimation and action evaluation. So it helps the players improving uh improves the players choosing different actions. Uh so those are the things that we settle on. Um, and we also have some uh

a building block here where we have a pod running uh the train model that is running the the the player model the the Doom player. Uh in the pod we also uh have something uh familiar with MCP you know it's very popular now. Uh so there's an MCP server and that MCP server scrapes metrics of that specific player and then we have uh MCP clients which we'll

talk about a little bit later that you know are used in what we call um uh a supervisor module that actually talks to all the different players to aggregate all the different behaviors of the players. Uh and then additionally we are uh you know using things like selenium cluster mesh mesh it's it's a topology another network topology resilient from the cloud native ecosystem. Uh so in in

essence this is kind of like what the architecture looks like on the left hand side or the right hand side. Uh and in and we use a variety of different setups with like u you know three two clusters. >> so so as Ricardo mentioned, there was a lot of moving parts involved in actually training the architecture. You saw that big list of different things. The reason why

we have a big list of different things is every time you just try one, something else breaks. So we end up building essentially a neural network per pod that essentially is being used per player to play Doom. So essentially we're running up to 16 neural networks per pod which in turn are then saving their best rewards. So best hyperparameters based on their active doom deathmatch gameplay and

saving that into a web 3 IP um IPFS network which is spanned across all of the pods to ensure that trained behavior per neural network is actually contributed back in an adversarial neural in an adversarial network to ensure that every time that one player wins then that goes into the pot for improved training. So it's a bit of an elaborate setup. Um we have one main pod

which is running the essentially the the the the main Doom instance which is the host pod. If you're playing deathmatch and we're we're going to demonstrate a deathmatch game of Doom. If you're playing deathmatch, you need a host pod which then the players connect up to. So basically um control and and worker to an extent. So, we've got that set up using ZDoom, which is a legacy

architecture for multiplayer gaming. Um, and then in addition to that, we have the native Kubernetes um DNS discovery for the different pods to be able to find their way home. So, quite an elaborate setup. You're going to see it working. Um, and yeah, cross to you, Ricardo. >> Yeah. So, uh, we were talking about CNCF projects. We are additionally using these two projects. a tetragonon uh using

ebpf underneath uh mostly for runtime process awareness and we have the MCP servers running uh in the player pods and and basically tetragonon is looking at the runtime uh behavior of of that specific pod and then we also have Hubble for network observability uh also another CNCF project or CCF ecosystem project so yeah super excited about using these different CSCF technology Apologies. >> So, so then what

we've also got what what Ricardo just showed you was we're using eBPF under the hood for a couple of things. So, Celium is our CNI. We've got Tetragon which is providing us with that process level visibility. What processes are being executed across the board for each one of these pods. And then we have um in within Celium we have the ability with Hubble to visualize communications. Now,

the nice thing there is Ricardo showed you the the sort of service mesh or or cluster mesh view of these three different clusters communicating with one another through Hubble in the the the Selium deployment. We're actually able to visualize clustertocluster communications um and from from the different various pods. So, it's not that we're just looking within one set of pods in one cluster. We're actually able to

see the communications and the setup. And here you can see some of the MCP communication matrices that we have between our different um pods being visualized here within the the UX the the the graphical user interface for Hubble. The other view which you um saw just before was the the CLI. So you've got multiple options depending on what you're after. So looking here we've got uh another

view which is actually coming from um the observability dashboard and when we're looking here we can see that a service map is being generated as a result of the communication. So we've instrumented everything with open telemetry. So that goes from the MCP servers across to the players themselves to essentially the um to to the um network of communications between them. So everything's um set up with MC

with um open telemetry. And we can actually see from the service map to the um statistics that we have here around player frags. And for anybody who doesn't know gaming, a frag is when you kill another player in multiplayer deathmatch. I didn't know this. I learned something new. So there you go. If there's anything that you learned today, then you all know what a frag is now.

And then in addition, you you obviously have the ability to filter to then see which one of my players is getting the highest hit count and how are things moving along there. Okay. So, you're all still with us? >> Yes. >> Okay. My Australian accent is not confusing everyone. Okay. Good. So, a gentic death match. So, let's take a look at what do you normally have when

it comes to playing a multiplayer game. You got players, right? So, in the death match, our players are actually AI systems. And each one of those systems has a persona just like you'd find on the depths of the internet, right? People that are friendly, people that are not friendly, people that are nice, people that are forgiving, and all of them talk to each other over MCP. So

every single time a player frags another player, their scores translate based on their persona into communications with one another. And you're going to see how this works. So we can see within the observability dashboard as mentioned facilitated via that we have communications from our different players that then feed back into the chaos engine and into the MCP servers where they can all talk together and have a

great time. All right. So, uh, yeah, in in terms of, uh, MCP communication, uh, so Josh was talking about it. So, these are some of the details. Um, uh, we're, you know, picking up like, uh, different, u, uh, types of metrics like fracks, you know, health the players, uh, and we're also, you know, defining what types of metrics these are. So uh and additionally we have um

like I mentioned this supervisor uh agent or supervisor pod that is actually talking to the MCPU service of of the players and and and actually getting all these different stats that are being fed into our observability dashboards and and yeah this shows like an integration showing all the different uh 16 Doom players and 16 Doom pods and and and additional MCP servers for each one of those

uh 16 Doom players. Uh we also can uh display uh MCP tool calling. So the the the different um methods and and and tools and how we expose all these different metrics and and using the MCP server. Uh so here in operations so we we are actually you know calling the different tools for example call the the the player stats or or get the personality profile. So

like you know we talked about the personality profiles for each one of the souls. So all of that stuff that we we can actually observe across the board. >> Sure. So then what what we can see is so so we're running MCP and MCP is facilitating the communications from player to player based on how well or poorly they're playing the game. We can see from from what

Ricardo was showing before that we've got quite an elaborate structure of the MCP data feeding into the chaos engine which then can restart the pods. MCP health is important and what we saw in that last slide is we're using the um generative AI semantic conventions from the open telemetry project. So we're actually following the guidelines for the spans and and the conventions being used for how we're

actually linking all this together and link and instrumenting it. And here we can actually see the performance of the MCP server um per pod which shows us how much trash talk are they doing, what are the player stats, persona profile and then we also have details on how each one of those tools MCP tools which is exposed is being handled in terms of statistics, how well it's

performing, what the latency is and everything else. So, we're really sort of going to town with making sure that a game works. Um, and then looking over here, we can we we can actually get a view of that trash talk that's happening between the players. Right. So, here you can see a distributed trace where you've got the MCP supervisor which is bringing all information together speaking with

the different players which are using NATO alphabet or NATO naming um you know alpha, delta, charlie etc. and we can actually see the results from the game which is here. And I apologize for anyone who's sitting up the back that don't have amazing glasses. Um, so that's that's a view here. And then you can actually see the banter. So this banter is active between the players as

they play. You're going to see them actually talk to one another. Um, okay. So let's take a look at Gosh, this is not easy. So this is >> All right. So, when we started training, it really sucked. Um, and this is before we started getting everything working with all those different bits and pieces to actually ensure the player would the players would play well. It was just

a lot of punching walls. And we had 14 iterations of the player sucking before we got to a point where things kind of improved. So that is am I on the next one? >> All right. So So this one. All right. So here was a point where it started to improve where the players um reward logic and hyperparameters are actually leading to a point where it wasn't

hitting the wall all the time. It also wasn't great. But what we can actually see here is on a very very rudimentary deployment, we could link the reward signals across to the players doing stuff and a little bit of chaos at the same time. So this was made made us hopeful. Um but it wasn't perfect. And one thing that wasn't perfect here was the fact that the

pods took ages to restart. So we had to re-evaluate using standard docker pods to using web assembly to allow us for a much more rapid restart of our resources. So trying to jump to the next one. Okay. So that brings us across to this which is our 14th iteration where we can actually finally see the players are working properly. They're com they're communicating with one another and

they are actively fragging one another resulting in pretty good and yeah so so what we have from the the system is a dashboard which is basically showing us this is the the results of the death match. It's it's working well. we can see that the health of the pods over time but what we haven't explored in detail is the chaos engine itself. So the the chaos engine

is basically set up in in a way that it taps the existing MCP logic that we brought in. So we basically built a chaos engine which is running on top of the existing um logic systems we have there with MCP based on the rewards which are coming through and then based on the frags we then started taking that data building metrics from that and then executing essentially

the restart of the different um web assembly pods as a result. Now, what's important to note is we've given every single player what we call a soul, and that's this web assembly container here. And every time that the player gets fragged, then it soul gets restarted. So, you're not restarting a a large neural network every single time a player gets killed. So, looking at the dashboard, so

we've got a number of dashboards that we built. We're going to try and show you the dashboards in a second. Um but we have a number of dashboards which which are showing the status of play on the on the different um on for different charact characteristics in including the MCP communication the neural network training and everything else and I'm going to jump across to so so this

is this is a view of it but I want to show you live um then we've got also a view of all the statistics related to ebpf and tetragon so we can actually see all the process calls which are associated with the communications which are happening at a process level. Um and from Hubble from the traffic communication we can see all the traffic which is going pod

to pod within the within the network too. So this is recorded but let's see if I can actually one second. Let's see if this is actually working for us. All right. So this I'm not very good at looking over here. Okay. So this is our eBPF view of our different processes which are um sorry that I have to look over here the different processes which are being

executed. We can also see the process calls here directly from which which are coming in from eBPF being um sent across as as um metric data and then we can basically see an overview of the containers being loaded who's executing the containers and different metrics associated with their usage. Here we've actually got the flows of the traffic, right? So we can actually see the communications which are

taking place from container to container over which port um whether it's a TCP connection or whatever else. So so that's kind of neat when we're looking at what's the status of our communications at an ebpf um perspective. Then we also have the view of how healthy is my neural network or my network of chaos players playing against one another. And here we can basically see the number

of frags which are which are taking place, the MCP calls per user. Um, and then who's the top player at a given point in time, the stability of the spans, um, and a bunch of other information related to the latency of the MCP servers, um, and and other statistics related to the neural network performance, the souls that have been killed, um, and and other data related to

the actual players themselves. Now, to avoid having the same game play every single time, what we've actually done is we've brought it in a randomizer that changes the temperature of the neural network here. So, you can see every time that a game restarts, the neural network tunes to a different temperature, meaning the player will be more courageous or less courageous or just decide to hang out in

a corner. Uh so this is is LinkedIn with those personas and and profiles and and actually leads to interesting statistics. Otherwise you just see flat lines. You'd see repetition of the same behavior over and over again. So that entropy that we introduce um leads to some interesting stuff. So um let me jump back here. sorry. So uh you seen we we try to introduce a lot of

chaos in the system and the CNCF actually has a lot of different tools you know that allow you to introduce chaos and one of them is chaos mesh it's a project in incubating so go check it out uh another project is limus or limos chaos and yeah so you can introduce these um projects into in your distributed systems and and you know this is how I see

This is just like a game but but but if you look at it from the you know above and in in in in distributed systems you can see that this can be applied to to your organization right so limus is another CNCF project that you can use uh and yeah finally some uh takeaways so what we figured out is that training uh for the specific behavior of

the players and uh making them behave how we wanted to play not just uh you know have a player hide in a corner and just not do anything. uh it takes time right so you need to do uh a lot of different iterations you want you need to try different things uh we talked about restn net we talked talked about impala so all these different methodologies or

AI methodologies are you know needed for maybe the behavior that you want to get and and this is what we use but may not necessarily work for you and so this is tough that and and that's the point we're trying to make um observability having this across the board it helps you understand the things that we talked about like like why things are not working or if

your system is crashing or you're introducing chaos and what happens what what the behavior is and and really see what what you know how it will really behave in production environments or or or even in production environments if it's allowed in your organization you can see what happens right um additionally uh we uh we uh so we think Explainability is is um it's basically how how we

able to explain through uh all these different behaviors is it's how we can we can keep improving and actually uh helps us improve our Then we we we we think that for all of these types of um you know experiments and and you you need to have some sort of methodology which we think it has to be some sort of metric driven right so again open telemetry

is here to help uh and and uh finally you know having AI in production you know requires a a you know super uh high availability environments and a lot of resilience. So those are our learnings. Um and I think that's all we have >> that that that was the the main overview. Thank thanks Ricard. So yeah I think that the main call outs is you know why

did we build this I guess is one question which we didn't answer here and it's more just to show the art of what's possible. But if you juxtapose having your game restarting to your services or your AI agents which are part of a long live flow, it's those resilience topics and the observability that are critical, right? So this is like a fun demo because I could have

just shown you know statistical data which probably would have sent a lot of you to sleep. So this is sort of trying to make it a little bit fun. So we're open to any questions. Um I've got a tough skin so you know feel free to ask anything nasty. have a question. >> Were you ever tempted to jump in and play yourself against all those uh crazy

AIs? And >> well, I I think maybe yeah. I uh I mean but uh you know do that that was not the the whole point of the exercise. It's just trying to have something fully automated that you know you know it nowadays with like agents running everywhere and then we think about how random things can happen. We thought this was actually some some like kind of like

representation of how applications are starting to be used in in a gent world right. So, uh, but yeah, I think, uh, yeah, but I mean, it's just also fun, you know, to think about it, right? >> Any further questions? >> Yeah, sure. You might have mentioned it, but was what was the size of the team implementing this and what was the time frame to build all of

this until you reached the end result? >> Size of the team. One, two. Yeah, that's a good question. Yeah. So, uh, we use a lot of Genai, right? So, and then then that's what a lot of people are starting to use. And so it takes it takes time. I mean it takes uh you know understanding some of the output and some of these coding tools and and

um and iterating continuously. >> It was a slow and iterative process particularly the training. I started the training on my a Mac smaller than this and it would be like just sitting there humming away for days and days on end to then real realize that the results were garbage. Right. So, um, I'd say it was a lot of long effort, not necessarily a lot of focused effort

when it came to the training and then bringing all the pieces together. Yeah. Certainly had uh some support on the side. So, >> and we also wanted to use all the the CNCF projects or or not all of them but like several of them. So we kind of showcase how all of these ecosystem projects can uh help u develop a solution like this and um additionally not

just CNCF but you know pietorch part of the pietorch foundation. So we're we're trying to bring in things from the AI world also to cloud native and vice versa. >> Exactly. So so we we didn't want to just have an agent demo and that's why we sort of have both. We've got classic training and we've got generative. So essentially we're running generative AI model plus that you

know custom trained model and we're having them interface with one another. And I think from deployment patterns this is certainly something that we're going to see more and more of right having a specialized model which is essentially derived and distilled from the the findings or the learnings from the you know what's potentially coming up from active data. Yeah, I think so. One one thing that we didn't

mention is that our uh supervisor also talks to uh GPT Mini to to actually create that the trash talk. I don't know if you did. >> Exactly. So I I don't have that much trash talk in me. So we went for an outside source. So >> yeah. So >> cool. All right, guys. Thank you. >> Very welcome. >> Any other questions? >> Awesome. >> Maybe also a

fun question. Did you also think about having a doom team against a let's say rescue team? So kind of good against bad. >> So So that's actually an option. So I I became I was never good at Doom when I was a kid. Um but I've learned a lot in the last months. Um and and basically you've got deathmatch, which we're showing here, but you also have

co-op where you basically have players like a team of players that are playing against the bad guys. So that's also an option. But after we wrote up the the submission, I thought, "Oh, can't change it now." I said, "We're going to do deathmatch." So, yes, absolutely. An an >> kind of comparable to hackers against people trying to break something and >> I got to give it a

try. >> Yeah. So, yeah, I mean that this also kind of relates to security and blue teams and red teams, you know, trying to, you know, one up each other, right? So, yeah. So I mean this is just just like a game scenario but then this game scenario can actually be portrayed in a different context or a different different u application. So not just necessarily Doom, right?

But other other types of games >> questions that we're holding you up for. I think there's there's some food out there, but I mean we're just going to hang out here for a little bit. If you have any questions, happy to take any questions. We're available online on LinkedIn. So if you want to connect with any of us, feel free to do so. We're happy to to

chat more or if you have questions about CNCF or projects or how how to do something like this, you know, feel free to reach