Ballroom A Friday Mar. 06 - SCaLE 23x

This talk covers the significance of simplifying technology and collaboration in the context of growing tech complexity. The speaker, Cat Morgan, draws from her extensive experience across various enterprises to identify patterns in human behavior and technology use. She emphasizes the importance of making development processes efficient and convenient, particularly in the realms of CI/CD and code management. By discussing concepts like AI tools and technical debt, she illustrates how scaling human effectiveness is crucial for navigating the complexities of cloud environments. The session aims to foster community engagement and continuous learning, ultimately advocating for the need to create user-friendly systems that prioritize simplicity.

Oh, it's hot. anyway. So, Test test. Hello everyone. Welcome to Cloud Native Days LA. I am Justin Garrison. I am one of the organizers of this event. Thanks for coming. Thanks for uh you know, it's it's Friday morning, so it's a it's a work day, so it's fine to come. Um but bra, if anyone's anyone local, driving in, braving the traffic, I mean, it's easy on Friday

mornings. which is great. But um yeah, so thanks everyone. I don't remember how many years we've been doing this now. Uh I've been coming for more than a decade. Um and it's had a few name changes, but it's always kind of the same content. So is it anyone's first scale? Awesome. I love seeing first- timers. This is great. Um again, I've been coming for more than a

decade, so I don't remember all the things we've done that's different. But if you have any questions about scale or cloud native days, feel free to find me. Um Josh and Gwenny and Steve, we're all around. We don't have organizer shirts because this is just everyone that runs this thing is a volunteer. Um, so we just are here because we love the community. We want to give

back. We want to make sure that you all have a safe space to learn and we try to make the tracks and topics as interesting as possible. Uh, which is exactly why I invited my friend Cat here to come give a talk about all the amazing stuff she's been working on in a variety of fields. Um, you have so much breadth of knowledge about things. I thought

you would have a great a great time sharing with this group. So, It is does not >> It does not sound like it's on. No. Flip this up. How the heck? There you go. >> Now, yes, that's definitely my voice. >> Um, so yeah, hand over to Cat. >> Thanks, Dustin. Um, also, thank you for all the work that you put into recruiting speakers and all of

the stuff that happens before we even get here. I know that a lot of people put effort into that. Um, where are my shy nerds? if you can raise your hand. Okay. Where am I really exuberant nerds? >> Also, yes. >> Okay, cool. All right. So, um the people who are also Yes. exuberant, uh feel interrupt every now and then. I'll have some more thematic slides when

we're there. If I don't remember, raise your hand or anything like that. We're going to cover a lot. Um, also I am really, really excited about what we're getting into, but there is a bit of a drift between what's on the abstract and what we're going to cover today. Partly because I sat down to write everything that we need to talk about before we get into the

deep hands-on technicals and realized that wasn't an intro, that was the whole talk. So, uh, we will definitely be following this up with a lot more community engagement and hopefully coming back next year. [sighs] My name is Cat Morgan. I have a litany of small experiences across a very large number of enterprises. Most of my um, work history is in support and consulting, which means I was

very fractional and spread across a lot of organizations. Um, one of the things that I'm really excited about because of that is just the patterns that begin emerging from human behavior and the way that we use technology today that largely I I don't think I would have been able to find um and and dig deep enough to get to without all of that exposure. So, this is

a little bit of um a thrown together riddle. So it grows without headcount. It outlives all migrations. Everybody feels it. And then what are we actually talking about here? It is everything that holds back a day a practitioner on their daily routine. Um a support engineer who is trying to get to the root cause of a problem. And where do these things really [snorts] come from? um

whether it's a simple GitHub actions failing for the 15th time when setting up a repo for the first time or if it's jumping into uh collaboration with a peer and finding out that it's working in CI and it's working on one person's machine but not on the other and that's not just a trope that is a legitimate um prognosis in my opinion. So, how many people does

it take to build something from scratch matters today? I don't think that really mattered four years ago. Um, for those engineers who are playing with AI tools, if you aren't today, um, at least have enough exposure to be aware of what they feel like in people's hands because everyone building technology today is prompt engineering. If you commit code that's all handwritten on GitHub, you just prompt engineered

somebody's future. Um, AI cost. When a CI fails and a corporation has invented uh invested in heavy AI analysis and responsiveness, CI failure uh costs explode exponentially very very quickly. Um, a lot of times we deal with the additive uh I can't use that word the add additive failure modes um by adding heads to the team which scales up to a point. [sighs] But how many people

does it actually take once we've invested in pushing that out the door to root cause a problem and then get everyone aligned on identifying the solution from there. And then the educational cost. This isn't a one-time myopic timeline that we're on. If I introduce an architecture that causes a lot of CI failures and we overcome that with AI and now we have a lot of success. If

we haven't actually built the architecture to support that efficiently and that maintainability over time, we are setting up a cliff of technical complexity that new entrance into the field are going to struggle to overcome. Um, so simplifying things is not just important for ourselves, not just important for AI, not just important for the the juniors and the interns that we're bringing into the field. This is a

really big deal because we have a lot of tech debt on the planet and now tech debt is scaling at AI scale. Um, we actually have to scale the human pipeline along with that. And I think the main theme that I want to get to here is every time you hear AI engineering or AI optimization and things like that, what comes back to uh the fundamentals that

we have to jump back to is if we want to make anything easy and reliable for AI, we have to make it easy and reliable for people first. If you completely remove AI from the room, are the people successful and comfortable and efficient and creative? And that's what we're really going to get into today. there are 180 million developer accounts on GitHub. That's actually a really big

number. Um it also grew a lot last year [snorts] if GitHub was a country. Um it would not be a small time country. Yeah. >> You mentioned it had increased a lot compared to last year. Do you know the numbers? Um, so 36 million new accounts last year. Um, obviously I have more than one account on GitHub. So there, uh, you know, there there are insights that

we don't have into what those numbers really mean, especially if you consider um, enterprise accounts and GitLab and all the other ways you can version control. But fundamentally code scale is already a pressure that we're not keeping up with. So uh tackling this the scalability of human influence and human effectiveness and collaboration is uh the first priority to tackling the complexity of cloud. So when you sit

down and you look at a new repository that you haven't approached before, um, at the very least, in my opin experience, I'm looking for a readme that gets me started, does not cause stress, and lets me achieve some semblance of initial success in the first few minutes. If it can't do that, there's a good chance that other people are falling off that train the same way, or

I'm going to go look for a different tool. Um, if it's not convenient and it slows me down during the day, I'm going to look for the path of least resistance. I'm a lot like an electron like that. [snorts] So, we want the readme to just work. Also, the prerequisites. I work on a Mac during the day. My favorite system is always going to be a Linux

system and I have a lot of colleagues and other community members that really matter who are on Windows. So, how do we actually make sure prerequisites are not tedious and are not toil that slows down adoption or that initial success that invites people to be successful with your project or projects that are important to you. CI/CD doesn't just have to work. It has to be consistent with

what humans experience. So when you're not in CI/CD, which is really just double-checking your work, the developer workflow that you're running locally on your system should be as close to, if not completely identical with CI as much as possible. When you're working solo and you need to identify the root cause of an error, if you have wildly different automation in CI than you have locally, the now

you have to introduce an entire understanding and domain of knowledge to be able to interpret between the two. And that does not increase the scalability of one repo, let alone an entire project full of repos. So also when it comes to friction and scaling the community engagement, contributing should actually be the onboarding tool that lets someone who wrote a little patch in their free time see the

fast path into getting credibility with the maintainer groups. So that now you are encouraged to continue contributing and um you feel not just successful with the technology but successful with that community and successful um momentum with the next repo that you might pick up on that project. So, how much toil does it take to switch? If you pull something down from um Kubernetes 6 and you successfully

patch a document there, now you want to move on to a technical change or you want to patch a document in another repository. How much additive work is there between each uh accumulative contribution? And then conventions when it comes to having that success, it has to be intuitive. So to say not even 1 million repos clear that bar is an understatement, but we're giving um credibility to

the benefit of the doubt here. Approximately 1 million repos actually have some semblance of the artifacts that we just talked about. Whether they work or not is a totally different conversation. And there's another 499 million repos to go that don't even meet that minimum bar. Not every repo has to be for public consumption. So, we're going to acknowledge that. But if even a million repos are out

of reach right now for the scale of AI contributions, that is a hockey puck curve at this point. We're going to need conventions. I hate to show up to a room and say, "Hey, let's make a policy." But it's okay. There's there's ways to make that convenient and comfortable. Realistically, when we step further away from the tech and we look at why are we doing Linux? Why

are we working together as a team? Why are we trying to scale open-source planet scale and run most of modern technology somehow or other on Linux? And fundamentally, it's because people like to do things. We have thoughts, we have ideas. We want to go do things with those thoughts and ideas. But especially in a corporate environment and this is representative of open source projects that are not

corporatized and even smallcale group projects where there is a litany of steps that you go through cognitively or not before you go from having an imagination epiphany and an idea on how to do that to actually diving in and then seeing that outcome in the real world. Uh, one thing that I have said to my partner frequently is, um, a lot of what I work on is

cloud, which means nobody actually sees it, touches it, feels it. There's no tangible evidence of what I've dedicated my life to work on. And we see all kinds of things from front-end developers and hardware developers and these people that actually show up and change real life with the technology that they create. I think that's fascinating, but realistically the disciplines required to cover that entire spectrum are so

broad that it's hard to imagine any one person actually being successful across that deep of a technology stack, especially if they want to take robotics and IoT and distributed computing and all of these things and deliver brand new experiences that make humans connect better and look at screens less, which is really my passion. So the efficiency conversation is not just an organizational efficiency conversation. It's can we

actually connect the dots for people at a scale that lets them use cloud technology even though they're a roboticist or use their IoT gadgets and build that cloud experience that they want to deliver to themselves, their families, their communities. Um, and fundamentally that's what I'm here for. So, [sighs] we've covered the problem. I love to be different. Um, it's a bit addictive to the point that if

someone tells me I have to do something a certain way, I'm probably going to do it wildly different from that just because they spoke up. Um, it is not the best plan, but here I'm going to go ahead and suggest some possible conventions and you can completely fork off into a totally different direction and throw tomatoes. But ultimately, I was struggling to keep up with literally dozens

of projects across half a dozen git servers. And each of those progress or projects had at least one repo, but most of them had three to six repos that I was patching before actually achieving the outcome that I was looking for. [sighs and gasps] And I tried a bunch of things. For the longest time, I've had a uh git directory in my home directory. And I have

attempted to align to some kind of habit that makes it predictable where to find things and how to get what I need. But five months later, I usually cannot find the repo that I need or I didn't find it the last time and I cloned it down a third time. So what I actually learned a lot from dev containers was the workspace convention is really really useful.

We actually stuff some things into that level of the uh workspace path and we move it up a little bit, but the workspace path breaks the entire git uh tree outside of your home directory. So now it's no longer subject to just your uh home directory uh sovereignty. So there's there's a number of opportunities we actually get to break into there. um for regulated industries. If you

have worked in um HIPPA or defense or other public sectors that are se uh sensitive um baking things like that, you'll find a number of STIG requirements where you actually have to harden your partitions, separate different uh partition types across different volumes, and there might be different um read write and group ownership and other kinds of conditions that you have to conform to. By breaking our git

server out or our git paths out into their own workspace, we actually significantly improve our uh adaptability for shipping and operating code in those um environments without having to document 15 times where we put code. Our individual user isolation is actually more important today than I've ever seen before. Um, I started using a Bastion uh VM or things like of that nature a lot when working in

air gap networks and it was actually an unexpected catalyst for team cohesion. Basically what happened is we all shared one user on a core OS box and it was not exactly condoned by Red Hat engineering how we were using it but we got to deploy a Red Hat core OS box up in an air gap space with two networks. one on the low side that it was

connected to the clear net and then one on the high side that is um attached to the simulated air gap space that we would develop air gap capabilities on and then you'd sit right there in between and if you needed to pull docker container images things like that you could just do it of course that was fraught with tedium so we automated lots of anible and terraform

but fundamentally we found that by sharing a bastion and sharing a common um code repository layout. We could interact and co-develop and experiment and validate each other's work live and it was very multiplayer. That's an even bigger deal today given the um aggressiveness that developers bring to the table when they're using LLM tools and things like that. So ah now we have the next layer which is

the actual server itself. If you're working on an internal corporate server or a GitHub server or if you're jumping to GitLab and things like that, it can get really confusing especially if you need to clone the exact same repo from two different servers. So that actually adds to the depth of collision avoidance. And we really actually adopted that from the uh Golang um library cloning convention where

and tacked that onto the per user and workspace path. So now we have this larger um workspace environment that actually gives us coherence and reduces how much we have to think about I need to clone a repo where where do I need to put it or I already did where did it go? And when you're working on very large complex projects, that adds up big time. That

also offers the opportunity to build a new convention. Last year, I was um fortunate enough to be at Defcon for my first time and I got to speak with a um developer uh at Microsoft and then a director at GitHub. And it is painfully uh clear to the industry in general that we don't have a common way to make sure people are successful in an ergonomic fashion

without having to go and ask someone who knows. And fundamentally um have centered on this convention of workspace as a canonicalized uh convention that centralizes all the tooling that is common across many repos. in a in a project so that the the one clone success story doesn't actually clone coupe cuddle if you want to fork coupe cuddle and make a PR instead you take the workspace repo

and that workspace repo has all the tooling necessary to clone down the other repos in a multi-root layout and offer the task automation. So if you're using make or mice or mis I'm not sure how to say that um or task cli all of your automation exists and all of the downstream uh repos that consume that workspace can just assume consistency there. Ah oh no my diagram.

Hang on. Yay. Okay. So, once we actually introduce this concept of uh active multiplayer and in a collaborative layout, um now suddenly we're answering a number of other problems and consolidating confusion into a shared workspace. If you want to um understand the risk vectors and the activities of a AI agent, for example, we're looking to find or I'm sorry, it's actually been a minute since I took

a gave a speech, so take a second. Okay. So when it comes to open code, cloud code, um, codec cla and other tools like that, currently there is a lot of investment trying to treat them like special problems. They're not actually that special. They're just another agent out in the wild that is capable of great and terrible things. So when it comes to people, we wrap them

in user accounts. We constrain them with permissions. we give them um pseudo or don't give them pseudo uh based on risk and pro uh posture. We can do the exact same things with codecs and those CLI tools where they actually get their their governance from the organization the same way any human does and they are isolated in their own user accounts and then we allow humans to

execute those across the user boundary. The other thing here is I was frustrated with having a [snorts] [sighs] limited convenience with understanding what's happening in CI. So by taking uh the the CI runner itself and for this it was an experiment with for Dejo server um gave for Dejo its own runner account like you would expect in GitHub actions and it is also another peer on the

exact same directory layout. So we have one route to rule to all but what's that actually enable? When I sit down to clone a whole entire project, I GitHub clone workspace and then I run my task automation to clone the rest of the obviously necessary repositories and then peace meal after that can grab any others if I'm not shipping the whole entire project all at once. Which

comes to the conversation of habitats and habits. If we really want to scale effectiveness and comfortability in cloud computing and other complex uh systems, we have to create a habitat that leads to um intuitive and implied conformity. Um if you work on Linux, you you at some point you may be putting end users at risk or other organizations with sensitive information. Habits are your best defense against

risk. We can implement any technological barrier to um exposing secrets or uh vulnerable deployment. But the most the best guarantee comes from making sure the humans just intuitively and obviously do the natural and and safe thing. There's a lot of nicks on this deck and I'm going to acknowledge that I do not think that everybody has to do nicks. I have actually only taken Nick seriously in

the last probably six months. Um and before that I attempted to do a lot more with it. Um probably four or five times and I gave up. It was about the same number of times that it took for me to pick up Kubernetes and finally be successful with it for the first time. I think Nyx is a capable system. Um, but if we're going to scale it,

it also leaves a lot of user experience on the table. So, I'm going to acknowledge that the should not versus cannot. Um, it sounds very harsh, but fundamentally we're talking about policy versus um So if I get commit, do I have git commit hooks that make sure that all my llinters are passing and that um any static analysis is done before my code ships to the CI

runner. Remember we talked about CI failures significantly um impeding um development velocity. So [clears throat] on on that uh concept of empathy and understanding where we actually want to shape the user experience, it's are we telling people to do the right thing or are we giving them tools that automatically uh asserts that the right thing happens so that humans can focus on the creative problems and not

on the tedium of did I use the right number of spaces in this particular YAML document. So this is where Nyx actually offers some really interesting capabilities. All of these things are possible with Ubuntu or any other generic Linux. I have achieved every single one of these capabilities on a different version of Linux. Nyx just makes it really really reproducible. So that's why we have it uh

for the presentation ma subject matter. And really what happens here um I'm not connected to the internet anymore. No. Why are you doing this to me? Oh no, hang on. No, my laptop being crazy. Okay, [snorts] it was just stealing the focus of the screen. So what ends up happening is we have a flake that wraps normal bashrc files, normal linting config files and things like that.

So you can use the CLI as you would expect as you're developing those uh filtering tools or static analysis tools. And then when you build the flake, either using it on Mac OS or using it on something else, uh you have those configs stored in a static location in the next store path. And then every time you invoke the CLI, it's already preconfigured to use those um

deterministic config patterns. So there's no more chasing down did I set up my git hooks correctly? uh globally for them to all work in each of my uh multi-reo repos or um am I leaking some other config from my um hosts bash rc that's changing the um confidence and the durability of the automation. A lot of that was learned the hard way, not going to lie. So

once you've actually commodified the the quality uh outcome gates now when you're looking at a team of people with different preferences across a lot of different tools their story and experience becomes much less noisy and they can focus on what they're actually doing that's shared across the um codebase that they're working on and avoid overinvesting. in conversations about which tool we should use. By the way, Vim

is the best. Zed is pretty close. [gasps] Um, so I mentioned just briefly Mac versus Linux and how this all crosses all of those. The next flake pattern actually does really really well on Mac OS with meeting um all of these same constraints equivalently, all from a common codebase. And we'll get into a little bit of how that code is constructed. Um, but fundamentally once you have

the the flake working on Mac OS and Linux, now you have it easily consumable in a double WSL context, but it gets even better in that scenario. Um, for other reasons. And then the OCI containers piece is actually where all of this started for me. I have been a heavy user of dev containers for a long time and they have significantly reduced the barrier to entry for

people who want to experience even things as complex as Kubvert and now they can just spin up a GitHub code spaces and it has everything necessary to launch Kubver Kubert VMs. Did you know you could do hardware uh HVM virtualization in code spaces? you can do fully accelerated uh VMs on Kubernetes inside of GitHub code spaces. That was when I really um attached to OCI containers or

dev containers. But fundamentally that leaves a lot of people out. Um a lot of people aren't interested in dev containers and they require a lot of extra investment in tooling in networking architectures and other things to maintain that compatibility. So fundamentally they add up to eventually becoming a barrier instead of just um a developer uh quality of life improvement. So that is when I really actually outgrew

the concept of dev containers and sat down and thought am I going to invest fully in just you have to have a virtual machine to really do this and have a consistent experience. um when my brother got me onto Nyx. And so now actually seeing that flake experience cross the boundaries of platforms, I'm honestly going to say the only problem with it is we just need to

make it easy. so that whole outputs and sending it to different targets once you have uh [snorts] the generator already building an ISO and a VM and an AWS AMI adding a um OCI container as an output or adding um Proxmox as a target. All of those things are already built um to be commoditized. So now adding a new target for your environment requirements, excuse me, is

as largely as simple as going and finding how that output syntax is designed and jotting it down, rebuilding, and now you have a new output. So our actual goal is to start building things. Building things does require CI. How do we get this whole convention to flow in a CI workspace? Um, GitHub actions at this time does not have the ability to follow the convention uh of

cloning that workspace.getdown. I did figure out how to do it. Um, it's messy and it takes a lot of GitHub actions code. Um, so then I sat down and said, but what else is like this that we actually have influence over as practitioners? Um, and I think it was raw code. Uh, David Flanigan. I saw his post on Twitter about um, for so I went and checked

it out. It is GitHub compatible runner and I think it's actually built underneath with some of the codebase from like ACT CLI if you've ever used act locally to uh, simulate what's going to happen in your CI runners. And that gave me the opportunity to go ahead and fork it. Not the interesting part of the story. The interesting part of the story is once I actually forked

the source code successfully built it to um clone the workspace down before it clones the clawing repo so that it maintains that uh convention. All I had to do to build it into my CI runner, which of course is this Nyx was add a line pointing to the repo where I had forked the runner. And the Nyx Flake was able to build from that source. So now

I can actually develop my CI runner as fluidly as any other software that I would run in the runner So that meant I actually got to take upstream software, fork it, change it, deploy it, use it significantly in production and know the edges and the limits of this capability before actually showing up with a PR upstream and saying, "Hey, this is a big change in concept, but

in practices, it's just a few lines of change. Can we actually start this conversation?" I haven't done that yet. But um I'm really excited about that workflow and how it scales largely across other projects as well. And we'll see what happens with Foro. The next problem when it comes to building the convenience and trust required to scale large community efforts fundamentally if I don't trust random name

on GitHub just because they're on GitHub how do I actually forge that relationship and maintain trust that if I accept their contribution now and that accelerates over time am I exposing any code bases to a malevolent actor in the future. We've seen what happens with XZ and how that affected uh SSH in the past. Sometimes thread actors can dig in and be very helpful for a significant

amount of time to build that relationship. that threat still exist but largely we can at least influence a lot more confidence in downstream consumers that are actually pulling these updates as well as uh have clarity across um peers in the contribution pipeline. So we have several or three steps. We need to know our intent. We need to guarantee that what we intend to have happen actually is

going to happen and nothing else. And then we need to know at the end of the day, did that meet our expectations? It's um in Nyx land, I'll show you the three artifacts that make that pretty pretty clean. Um so the flake.nix Nyx itself is where we actually show up with an entry point to to Nyx and we say these are the things that we're going to

be shipping um and it'll either pull from pre-built sources or build locally from source in the case of Fred HoRunner. You can see how that fits in. There's lots of ways to consume this. Um the easiest of which is just Nyx develop. And then once you actually do the Nyx uh Flake update, you get a Flake lock that pins all of the versions. This matters significantly in

regulated environments where you need to be able to attest to every dependency that you've shipped. It's actually really cool if you just do it out the gate because now you can ship really customized Vim experiences or really customized VS Code experiences and you can start asserting control over things like um VS Code extensions which um I don't know if you know but VS Code extensions are really

scary in production environments um especially as anyone with a the ability to vibe code is able to p push um extensions now more and they have access to a lot of secrets and have been uh used for successful um attacks already. Um and I'm sure that that risk factor is just growing as the number of people show up to develop software, but that's hearsay. I don't have

numbers for that yet. So um another thing that's difficult is when you have inconsistency with versions or when you have consistency with versions but you have to write the version down in 15 different places across three different repos or more and having that handled globally is actually a big deal. Um it is there are escape hatches. So when you jump into one of your um multi-root repos,

you have the ability to also have a flake in there that's just a small file with a few lines so that you can also meet the very specific needs of that scope. This is not an all or nothing solution. It's just you're not going to fall through the floor um if you're adopting the workspace uh convention. So over time as things grow you can also increase how

uh comfortably uh surface those other versions in the centralized version.nix and then that is what globally defines across the entire rest of the flake what happens next. Um you can also set u other kinds of conditions that you want. So I might have environments that are pre-pro and I'm not going to expect them to be as rigidly conformant for now until they hit their own prod levels

of maturity. So I might actually allow for degraded run states that are not um fully compliant with readyto-release code. But I still want to know very loudly um when that happens. Ah that's what I was going to say a second ago. Um, all of these things start ma to matter even more when you expand your scope of how many humans are we putting in a harness to

how many humans and non-human agents are we putting into a harness. Um, fundamentally Claude cannot escape this. Um now that we have all of this metadata easy to access um for the purposes of building um a commoditized virtual machine uh disk image for example the number of times that I have deployed a dev container or deployed a virtual machine and then after the hour hour and a

half that it takes to do that whole team jumps in and oh no, we actually deployed the previous version of all of this stuff. It's all invalidated. We have to tear it all down and redeploy it in the cloud. That ups to uh adds up to eating a lot of human time. Um it's even worse when you do that and you don't know for the next hour

that you're still on the wrong version of what what you're working on. So um have adopted a fingerprinting technique where it's just collecting all of this metadata that we have readily accessible now and saving it in a fingerprint file um that's read only in the root of the file system. So that now all kinds of other automation that happens in this environment can self-inform what environment it's

in and what it's allowed or not allowed to do. And we can also externally identify whether or not traffic from this VM is fully compliant and following strict production requirements or not or if we're just running it um in a degraded mode and need to limit our um capabilities based on that. So besides the fingerprint file, all of that metadata also shows up in a PKI. during

the build we have a provenence um step that creates initial build certificates. These certificates are short-lived. The next time the VM starts the exact same systemd service runs again and regenerates these certificates. But every time it generates these certificates uh cumulatively over time. [snorts] um it is picking up the provenence information and identifying if the system has changed. Um just because it says that it was built

with this doesn't mean it's actually still conformant to this state. So being aware of that drift before uh shipping PKI is really important. If you're using self-signed certificates though in a secure environment, you're going to want to know before you jump into the VM or trust it what's happening. And the cool part about this is now all of this metadata just shows up in your SSL certificates.

And you can click in your browser, look at the certificate, and see what git repo and what commit the VM that you're in actually shipped from. And that actually um snowballs into a larger uh layer here in a minute. Um oh yeah no we're to that part now. So uh in the scenario of a de dev container deployed to kubernetes or an entire kubern kubert v virtual

machine deployed umert manager is really really helpful for taking the clusters uh global pki and signing the certificate in the VM with the cert relevant to your pre-pro or prod status. So that now you can see that every service started in the VM has access to uh a PKI that's relevant to um its identity and how it's going to be used. Um but just because you issu

you you sign it with pre-pro today doesn't mean you can't sign it with pre-prod or with actual prod tomorrow. So this is an accumulative capability. Let's see. How are we doing on the time? Oh, no. Okay. [snorts] Um, another part of that entire attestation framework that allows me to trust what I'm working in and who I'm working with um is the actual VM dumps all of its

uh build log into a serial console output. That log is captured and the providence file finger uh fingerprint file, the build log and the QCA2 disk image all get wrapped up together in a from scratch uh OCI image and shipped off to DockerHub or GHCR or whatever um Zot in the case of deploying locally um the way this has been going and now you have all of

the artifacts required to pass most um sbomb um and and other analysis, static analysis for uh the contents of a digital asset. that actually leads to some very noisy situations because what happens if you start shipping every single CI log that uh you're used to seeing in a GitHub action runner, it's too much noise and nobody's going to know what to do with that unless they know

what to look for explicitly. And now you're back in the position of I need a policy that educates developers on what to look for instead of making it very simple for them to find what they're looking for at time of need without additional effort. trying to see how much further I have to go. Okay, we're going to zoom. So fundamentally the idea is to increase the signal

to noise ratio so that when you are looking at something you know it's always going to be meaningful. Um noise in general at conventional scale is tech debt not just for the log aggregators it's tech debt for the people that have to read those as well. Um and it leads to fatigue where people are not paying attention to logs. At the same time when we have a

problem um we need to not shut down the whole operation just because we have a small bug. So uh design of software and systems should be tightly constrained to if this fails how can it possibly fail how do we keep everything continuing to run that's possible while mitigating the risks of the outage mode. Carrier grade infrastructure is really important when you're talking about telekcoms. They are heavily

regulated industry and when they drop a call, even if the end user doesn't notice anything unusual, they just have to call their friend back, they actually have to report that to the FCC. Um, and there's a significant uh amount of pressure on industry when that becomes a a regular pattern. Um, but what I'm talking about with carrier grade infrastructure is aircraft carrier grade. In the event that

you are isolated from the entire rest of the world, does your infrastructure continue doing its job? And that's actually really big deal, especially when it comes to peer-reviewed compute. Um, AI today is really neat, but it's all oneoffs. Every single uh frontier model today would be impossible to know you can reproduce. That's terrifying. Um, peer-reviewed um, automation is or peer-reviewed systems is a frontier that we need

to unlock before the 2030s, but we need to do it without it being complicated. So developer environments, VS Code, um TTY like terminals and things like that in the browser reduce the level of difficulty for engaging. Um, I've had a um peers and other contributors able to jump in and experience the same um premerge experience of fully developed features and work on them and brainstorm them. It's

like whiteboarding but live in code. And that multiplayer experience, especially with many humans and many AIs, is the experience that we're really driving towards with all of these conventions. It's not just rules. adding to the accessibility of cloud um largely people develop outside of the platforms that they deploy. um by deploying a VM or or a dev container inside the Kubernetes cluster and using traditional um OIDC

and other um user access models, you end up with the ability to build a Kubernetes operator inside the virtual machine and use it with go build commands inside the Kubernetes cluster without having to run an entire CI operation to deploy the change to a cluster and Okay, now does it work for the 15th time? The numbers of praying hands in um commit logs can go down. Um

so now we've talked about how this is a hub for humans, a hub for CI, a hub and how that turns into a live multiplayer experience. Um access is not limiting. You can actually jump in on your laptop and experience this whole uh workflow um when you're on Mac using Docker Desktop or jump into the full Kubernetes platform experience. Um and all of that is commonly accessible

from your browser potentially um if it depending on how you implement it and what constraints you're looking to meet. Um one of the things that's really really neat um is now that we're starting to increase the amount of reach that individual contributors um have with uh LLMs just helping you learn more faster even if you don't use LLMs to write write code. Now we're seeing problems scaling

much larger and every time we see scale we learn new patterns that we have to conform to for success. um whether it's CRDs enabling a wide uh adoption of cloud platform [snorts] like Kubernetes or um other conventions enabling people to just intuitively every time we scale um we're not talking anymore about individual platforms largely we're talking about platform factories and software factories Um let's see. So the

actual developer experience we develop conductor inside conductor. This is this Nyx experience for now. Um but this can be any virtual machine or developer environment and it's very or because it eats its tail when on the inside loop um you are building your flake internally making sure that it works. you can rebuild it. The source code ships inside the virtual machine itself. So it is self-building, self

attesting and can reproduce itself. Um in the mid orbit you're looking at actually being able to build Docker desk uh the entire platform in Docker in the virtual machine. So now you can deploy the platform deploy the developer environment in the platform all inside the blast radius of the virtual machine And once you actually have that running cleanly in the shortest possible loop, now you're talking about

deploying it on the outer loop where you literally deploy to the platform that you are developing in. This takes some paranoia to get right so that you don't accidentally break your cluster and delete your code. I've done it. But it also introduces a lot of awareness and intuitive responsibility in maintaining um a stable promotion pipeline. yeah, there's the slide on shipping the source. So when this virtual

machine builds, it pushes the source code that it builds from directly into the VM. So that if you need to make a change, you CD to that directory. You make your change and then you rebuild switch and now your VM has drifted from the fingerprint, which is a knowable drift. Um but now you have uh more adaptability and flexibility. Fundamentally it sounds complicated. Computing is computer science

for a reason. Um but science creates really really complicated In this era of cloud we are shifting from building complicated things to taking the science and building very simple things. Um when we achieve simplicity, that's when we achieve the ability to collaborate and communicate effectively and scale beyond the barriers of the um human limitations that we currently have with the tech debt accumulated. So the before and

after is really a story of enabling really crazy outcomes based on obsessive uh elimination of the complexity of systems. So, everyone feels it. And when you feel it for the third or fourth or fifth time, that's a good indicator that maybe it's time to just think about where the real root cause is of any given symptom because sometimes you are at the very top of the application

stack and what you need to fix is uh at the provisioning of the platform level. And if that is an unknowable depth of technology, then we're just going to keep bandating and making the abstractions deeper and deeper and not giving humans a way out. Um, I skipped a few slides there. I didn't practice the time quite perfectly to have this just down, but we are at the

end. I think we have what, two minutes for questions, conversation. I have to escape. [applause] Yeah, we can take a couple questions, but we also have 15 minutes between talks if you want to do it in the hallway as well. So, raise your hand. We do have a live stream, so I want to get on the mic if anyone has questions. >> Um, I unplugged this, but

I will put my laptop on the back table with a QR code open. >> Yeah, if you want the slides. >> Oh, I don't have a QR code for the slides yet. Um, I think I actually do in the top right. I need to push to my user account under presentations. I am starting a trend of being very boring with my naming conventions on git repos. Um,

join me. >> All right. Thank you. >> Thanks. Hold on. There we go. >> It has this really weird switch. So, this actually is the power switch on top. >> it's just really weird. >> So, you have to >> Yeah. Does it go up and down? It it goes >> is locked. >> Okay. What about this guy? Test. Test. >> That guy's not on. That's a normal

should be. Oh, sorry. This switch >> test. >> yeah. >> So, blue is muted. >> Hello. >> Hello and welcome. >> Yep. Cool. Okay. Cool. Push it out. So it's You want me to leave it live for now? >> Uh >> understood. And then So it's on right >> Um yeah, I'll leave it on. Yeah, maybe I'll just put it >> and then >> we still have

a few minutes, right? >> Yeah, we've got like four minutes. So, um right at 11:15 and then um do you want questions time for questions at the end? >> Um yeah, we should have time for >> and so I'll just kind of like run the mic around and at the end of it we'll open up and I can do the mic as well. >> Sounds great. Thank

you. [laughter] testing. Yeah. Cool. All right. Hello everybody and welcome to the cloud native track at scale. We're going to get started in just a minute here. Thank you all for being here for the second talk of the day. Um, so we're really excited to be able to hear from Maya Singh. She is going to be talking about uh conversational troubleshooting for Kubernetes ask a new shell

debug. So everybody please warm round of applause for Maya. >> Thank you for the warm welcome Marielle. Um hi everyone my name is Maya Singh and today's pres presentation is on Kubernetes. So it's called ask and you shall debug. Let's dive right in. Why is this an important topic for us today? Let's just take a step back and think about times when we have been under stress

in a troubleshooting scenario. How many of you all have felt that pressure and stress of having to troubleshoot something in any sort of system? Yeah. So, you know how it feels to be under the clock juggling a bunch of different tools and maybe a little stressed. What we are trying to showcase here today is how we can use tr conversational troubleshooting to streamline the process of troubleshooting

and really expedite that meanantime to resolution. So when we get down to the problem statement and why we're here today, the first piece is around the time sensitivity that comes with troubleshooting. Incidents are timesensitive. There are oftent times customers or users on the other side of an incident that are being impacted by either downtime or latency and being able to address issues quickly is of the utmost

importance. Secondly, troubleshooting often requires many different tools. I think we can all find ourselves or remember a time in which we were just overwhelmed with the problem and had just about a million tools we could have turned to and were almost overwhelmed by the vast number of tools that we needed to juggle and then how to use them whether those are certain cubectl commands or CLI commands

or even UIs that we needed to use. Um and then just the scenario trying to pinpoint the exact scenario in which we'd use these particular tools. And then finally root cause analysis is cognitively expensive. I think we can all think back on a time when we've tried a hypothesis of around why our workload was failing. It didn't work out. So then we moved to the next thing

and the next thing and that can be very draining and mentally taxing. So our goal today is to showcase how we can make real-time troubleshooting via natural language possible to ultimately decrease meantime to resolution and alleviate the mental burden on operators. So a little bit about me, my name is Maya Singh and I am a product manager at Microsoft working on observability for Azure Kubernetes service. I

am a maintainer of the inspector gadget project which is the sort of key open source project that we'll be featuring today and I am a product analytics and data enthusiast. What I have learned and sort of I guess witnessed over my last few years of working in the Kubernetes observability space is that a few years ago we were very much working in this domain where you know

sort of using cubectl commands was the go-to approach to troubleshooting uh Kubernetes issues maybe looking at the cube audit logs and really sifting through all this information. And what I've realized over actually the past like year, year and a half or so is that this introduction of Gen AI and using natural language for troubleshooting has really revolutionized the troubleshooting experience as we know it today. So that's

why I thought like if I have the opportunity to show like one use case or one scenario in which uh this can hopefully save you all some time, I thought this would be a great uh opportunity to do so. So as for our agenda today, we're going to open up uh right away with a demo. Um unsurprisingly, we're going to be addressing DNS. That's it's always DNS.

Hopefully that catches your attention because you've deal dealt with that problem many times before. We're going to then zoom out and take a look at the open- source project inspector gadget and that is uh what is ultimately powering the data under the hood of the demo that you'll see today. We'll talk a little bit about MCP servers as well. This is the avenue through which we are

surfacing up the information. That's sort of our tool to access uh the data from Inspector Gadget. And then we'll do some quick hit demos so that you can see the versatility of the project and hopefully gather some ideas as to how you can use this uh the next time you're under pressure or just for fun at home as well. And then we'll wrap up and I'll share

some resources. So, I want to really set the stage here in terms of putting you all in the shoes of someone who might be on call. So, um you know, you're sitting uh at home, you're on call, it's a Sunday afternoon and you get a Slack message, you get ping that, hey, something of the on your workload uh seems slow. There's some requests that are timing out,

but not all of them. It feels intermittent. and in your mind you're like, "Oh gosh, not again." Um, so you open up your laptop and you get to work. So what I'm going to be showing you today is how we can use natural language to address these uh problems, this DNS issue to see if we can figure out what's going on and root cause the issue. So

let me move to VS Code. Okay, awesome. And so what I'm showcasing here is uh Copilot within VS Code and it's already configured with the inspector gadget MCP server. Um, basically we have a bunch of tools here under inspector gadget and you can see that there are a bunch of different gadgets and we'll get into this later on but I just wanted to show you all that

uh for this first demo we're already uh preconfigured with the IG uh MCP server that is ultimately going to a act as a low-level data collector for the LLM. So, let's prompt it with our sort of problem that we're seeing and see if it can uh help us out a little bit. So, I'm seeing uh DNS timeouts on my cluster. Can you help me figure out what's

going So here what I'm doing is I'm basically prompting the LLM and asking it like hey like literally what do I know today? I know that you know someone pinged me and they were seeing intermittent DNS timeouts on their cluster. Can you help me figure out what's going on? So, what it's doing right now is it's actually looking at a specific skill that I had created. And

we'll get into skills um in a second, but it's going through this protocol. And the first thing it's doing here is it's getting um the pods on the cluster and just kind of doing some general checking of what's going on. And the first thing that it wanted to look at is the core DNS pod. So within the Kubernetes cluster, we want to understand is core DNS working.

It looks like it's healthy. So it's going to keep running through the protocol that I've set out. And now what it's doing is it's going to use the uh tool within Inspector Gadget called trace DNS. So what this is doing is it's tracing the DNS requests within the cluster and trying to identify when they uh DNS queries are resolving and uh accurately getting responses versus when they're

failing and unable to resolve. So it's thinking and it seems to have found a potential issue in our workload. So what it's doing here is let's see I can see some interesting patterns. Let me filter specifically on the default namespace to focus on the DNS request pod behavior and any high latency or problematic queries. So what it's doing is it's uh narrowing the range in which it

wants to investigate to specifically look at a uh particular pod that it thinks is uh problematic. Essentially, so what it's doing here is it's basically said, I now have a clear picture. Let me do one more targeted trace to capture the problematic uh SoCal Linux expo.org queries I saw going earlier uh that were unable to be resolved. Let's see. Okay, so it says now I have a

very clear picture of what's happening. Let me summarize the findings. So right now it's basically what we've seen it do is it collect a bunch of information, aggregate it, do some further investigation, and now summarize the results. So let's see what the summary says. Okay, so DNS troubleshooting results. Let's see. Issue one, hard-coded non-existent DNS server. Root cause of the timeouts. So the DNS request pod in

the default namespace is sending queries for socalinexpo.org directly to one 1234, bypassing the DNS cube DNS, the core DNS entirely. No response was received. This is not a real DNS server. So these queries are timing out every time. Um this is basically this is likely caused from a hard-coded DNS server. um 1 2 3 4 uh fix basically fix this hard-coded uh DNS server uh using the

command and that basically was the main issue that we were uh injecting into it. It says it found some other issues as well. Kubernetes defaults to this and it only has four DOS resolver all search domains before trying the bare name. Okay. So, what the good thing it figured out is that basically their core DNS is working, which is awesome. But the issue, which is the one

that I like deliberately put in there, was basically that we had a sort of bogus uh custom name server upstream. That was the sort of root cause of the issue. So let's go back to our slides now really quickly and we will yeah let's go to the next one. Awesome. So let's just really quickly recap what we saw. So at the source of the issue basically at

the beginning all we saw were like we got this ping we're seeing intermittent DNS requests some are timing timing out some are able to resolve properly. What's going on here? And so what we did is we prompted the LLM and basically we found out that the cluster had a pod that was making two DNS queries in a loop. First was for example.com that was going to core

DNS and resolving perfectly fine but the other uh DNS query that was happening around uh SoCal Linux Expo was going to this uh unreachable name server which was timing out every time. So what the LLM enabled us to do essentially was prompt it with the issue and then underneath the hood it was collecting the data using trace DNS which is a inspector gadget feature that I'll get

into in a second aggregated the info did a deeper dive even several times it ran that gadget and then ultimately summarized the information for us and uh showed us the uh exact root cause of the issue. which at the beginning I really had no idea about. So ultimately this um can be used in a real scenario as well. If at your company or your enterprise you have

your own custom uh name server upstream, it'll collect this traffic as well. And this can be something that you can use in real life if you're ever facing these sort of intermittent DNS issues because maybe um there were some configuration issues with your custom name server. Cool. So, now that we did that first opening demo to sort of showcase what can be done and sort of how

easily this can be resolved versus going through a bunch of cubectl commands, looking maybe at the logs, looking at networking data, it was able to really quickly resolve this issue and pinpoint the uh issue. Um, now we're going to get into inspector gadget. So this is the open- source project that was running under the hood. Anytime you saw like trace DNS or I'm going to use this

gadget called trace DNS under the hood, what it was doing it was is that uh LLM was using a project called inspector gadget. So I wanted to share a little bit about this project with you guys. So what is inspector gadget? Inspector gadget is a tool that enables us to collect lowlevel data data from the kernel and map it to the context of our particular environment. So

whether that's containers or if you're using uh Kubernetes, if that includes like the pods, the nodes, the name spaces, what Inspector Gadget enables us to do is take very granular uh low-level data from the kernel and map it to uh those higher level concepts that I just mentioned. And inspector gadget, as you can imagine, can be used in a variety of use cases. The first one that

we are sort of well the main one that we're showcasing today is this troubleshooting the sort of ondemand troubleshooting when you're in an incident that's a very common use case where the project is used but it can also be used for continuous monitoring and it can also be used for security projects like cubescape if you've heard of that open source project use it for security and Microsoft

dependent for cloud use it for security so there are broad uh range of applications as you can imagine to very low-level kernel data being mapped to things like pods and nodes for example I will also note that this can be run on the Linux host directly as well so don't feel like it has to be um run on a Kubernetes cluster you can use this on VMs

even bare metal um if it runs on Linux you can use inspector gadget I also wanted to call out on the left hand side We are talking about the MCP server as sort of like a mode of operation of inspector gadget today but um in reality you can use it um using the CLI you can export the data as a JSON file if you're looking for met

metrics it can be used with Prometheus open telemetry logs and metrics you can visualize it in graphana so there's a wide range of uh ways to use this and use cases so let's get into the actual like core functionality those are what we call gadgets. So gadgets are essentially the different tools within the inspector gadget framework that enable a very specific use case and functionality. So these

are essentially purpose-built tools um that are built with ebpf. So we have trace gadgets, snapshot gadgets, top profile and like I mentioned earlier, what really brings this data collected by these gadgets to life is the container enrichment to provide that context around Kubernetes and containers. So today we'll see a bunch of different gadgets come to life through the MCP server example and using Gen AI for this.

Um, but the reality is is we have like a huge library of gadgets that can be found on Artifact Hub. And um, if you don't see one that fits your needs, you can even contribute your own gadget as well. I like to also tell folks that it's a great way to get started with EVPF as well since, you know, we're at the Linux conference, so there might

be folks who are interested in eBPF or wanting to learn more about it. The team has created a bunch of different gadgets that um you can definitely use off the shelf, but you can even tweak the existing ones and uh contribute your own gadgets or make your own specific use case uh for the gadgets as well. And then the framework does all the container enrichment and everything

for you which is pretty cool. So yeah, we are at uh scale after all. So I'd be remiss to not mention ebpf. Um so what is ebpf? ebpf is technically it's in kernel bite code runtime but like what does that even mean? Ultimately what I like to highlight about ebpf is that it allows programs to run in the kernel in a safe and performant way. So ebpf

is event driven. So there are basically these triggers called hooks within uh the code that when a certain event occurs whether that's a sys call or a network related event it will basically trigger the ebpf program to run and yeah that's basically this concept of ebpf hooks in regards to it being safe and performant. It essentially runs in this sandbox environment in the kernel and the EVPF

programs also run through a verifier to make sure they're safe to run. Additionally, EVPF programs use a just in time compiler uh to make sure they are uh it's running very performantly. And finally, the last sort of like big concept that I wanted to cover for eBPF really quickly is just that there basically use this thing called ebpf maps that shares information between the user space and

the kernel. So there's a call called BPF sys call that essentially enables uh user space applications to access the kernel level data. And ultimately I kind of think of EVPF as a superpower for seeing exactly what's happening inside of the kernel. The good news is you don't have to be an expert on EVPF or even know what it is to basically reap the benefits of it using

inspector gadgets. All of the ebpf is kind of like abstracted away from the users when you're using inspector gadgets. So if you're interested, it's there for you to go look and dive into. But if you just care about getting the data and the functionality out of the gadgets, you totally don't have to worry about EVPF at all. So that's what you know, one of the things I

like about the >> [gasps] >> So, we talked about how we have like a huge library of gadgets ultimately and that is sort of a blessing and a curse because we have a bunch of different gadgets with different functionality whether that's you know you'll see a bunch today you've already seen the trace DNS one resource utilization around uh CPU for example uh networking related information even security

specific gadgets. So, we have this huge library, but the reality is it's kind of hard to choose which one to use in a specific scenario, especially if you're in an incident scenario and under a little bit of pressure. So, we found that picking the right gadget quickly, especially in an high high pressure situation, was pretty challenging for uh users. teams ended up either memorizing certain commands, defaulting

to other tools at times that were less effective or just reaching for like the gadgets they knew about essentially. And so that was like a problem that we wanted to be able to address headon. We wanted folks to be able to describe the symptom, get the right gadget, collect the data, and then understand their root cause and next steps as quickly as possible. And this is where

the MCP server came into play. So really highle overview, what is MCP model context protocol? It's essentially a standard way for LLMs to call external tools in a safe and structured way. So basically we created this uh IGMCP server that exposed a bunch of different tools. In this case, I'll get into it, but those are our gadgets. And then the MCP server allows the LLM to choose

which particular tools to run. And then the LLM can aggregate the results of the tool and explain it to the user. So in our particular case, the tools are our gadgets and we're using them via the IGMCP server. So this sort of turns what an LLM exists is more like just the chat for example into a chat with sort of prescriptive actions and tools that can be

called. So just a very high-level overview basically um the LLM accesses the IGMCP server or any MCP server using a standardized MCP protocol and then in our case the IGMCP server will call these different gadgets essentially which are the tools in the MCP server tool set. So the gadgets are what are providing those real time signals, the granular signals and then the LLM is what takes that

really granular data, aggregates it, analyzes it and turns it into actionable next steps. Um, just for your awareness, the MCP server transports, it has a couple of different transports. Uh, standard IO transport for local use and streamable HTTP for remote MCP servers. Um, and I just wanted to also call out that one good thing about using LLMs in combination with these real-time tools like Inspector Gadget is

that it enables the LLM basically is able to take a global look at your system, especially in scenarios where the hypothesis uh area is pretty broad. like if you really don't know where to start, what's kind of nice is that the LLM is able to take this broader search and a global approach and even like execute and parallel some of these hypotheses. um versus like previously if

you're kind of doing it the old school way, it may have been a more sequential hypothesisdriven approach to troubleshooting which can be time uh you know draining of your time basically and frustrating at times. So, especially when there's a broad area in of investigation, I feel like that's specifically where the LLMs and these uh real time observability tools like Inspector Gadget can be super helpful. So, I

also wanted to touch on agent skills really quickly. So, agent skills are essentially AI's playbook. These are essentially structured instructions for the LLM to be able to execute in a way that has some legitimate like human input, right? So, we all have our best practices from troubleshooting. We have a way that we want things done oftent times and skills enable us to give the LLM these instructions

by sort of defining defining what symptoms to look for which gadgets in this case to run and how to interpret the results and even how to output the results. Um what's interesting or I guess important about agent skills as it relates to MCP is that it le leverages progressive disclosure. So, I'll show the skill. Actually, I'll just show it right now. Basically, um we have these different

skills essentially. Let's see. Is this it? Yeah. Cool. So, we have these different skills essentially that are these run books or playbooks for the LLM. And at the beginning of the um skill, we have the name and the description. And so what that enables, those are the only things that are preloaded into the MCP server. So the LLM can look see what the name and the description

is and only then if it sees that oh this one my user said they had something around DNS. This skill uh is about DNS. Let me read the rest of this AI skill so that I can get the context only as needed. So in this world where we're using tokens and have limited context, being efficient with using our LLMs is super important. And this AI skill framework

allows us to give a little like enough of a hint, I guess, to the LLM that like, hey, you need to look into this particular playbook, this particular skill to get more information on this problem domain and how to address it. So that is basically I just wanted to call this out as like an avenue through which we can essentially um get through a more efficient way.

>> Oh yeah, sure. Thanks. I did think it okay maybe it'll cut um in and out a little less then. Let's see. Is it a little better? >> Okay, perfect. Thank you. Um great. Yeah. So, I did want to show you guys the agent skills because if you all decide to sort of write your own either MCP server or uh even add your own skills to maybe

this existing IGMCP server, um it's important to make sure you have that name and header uh correct basically so that it gives enough of a clue to the LLM to like read what's there. Uh what else did I want to say? Oh, yeah. I also want to shout out that like every company or user has their own unique environment and system that they're running. So the these

agent skills enable you to sort of document the intricacies of your particular system so that the AI is aware of them and has a sort of predictable path that it can pursue based on your specific uh system. So teams can author custom skills for your own infrastructure or your own failure patterns that keep coming up and coming up so that uh the agent knows exactly what path

to pursue. Um oftent times we're troubleshooting and we want to save what we've learned and then reuse it for the future. Skills are a great way to do that as well. Alrighty. Okay. So, now we're going to get into some quick hits. I wanted to really make sure that you all were able to like see this in action and see the value that is derived from taking

an LLM and giving it this very low-level context that is derived from Inspector Gadget. So, our first quick hit is going to be around disk pressure. So I think maybe we've all faced this scenario where we're seeing latency when our workloads either dealing with a database, writing or reading from a file and we're trying to figure out what exactly is going on. So in this uh video

I'm going to showcase how the LLM plus inspector gadget can help bring our basically root cause the issue and bring the scenario to life. So Okay, cool. So here I'm basically prompting the LLM that I'm facing latency when conducting DI disk rel related operations. Can you uh help me identify the problem? And so what the LLM is doing is similarly to the last time. This time it

saw a skill related to Kubernetes disk troubleshooter. So within that uh particular skill, I had the name called Kubernetes disk troubleshooter and a little description about what it does that gives the LLM enough context to then deep dive into that particular skill because we want to provide it just enough context to basically know that hey they need to pursue this path. The first step that it does

is it basically it profiles using the profile block IO gadget. It profiles the uh block IO operation latency. So what's do it's doing here is it's recording the block IO operations and it's uh plotting it on a histogram essentially and it sees basically that we have significant counts in this high latency range. After that, it's using a gadget called top block IO to identify which pod is

actually the sort of culprit for this high block IO activity. Where can I basically take this broad approach a first figure out there's a block IO issue and then narrow down to the noisiest uh pod and then finally it's looking for the exact file using the top file gadget. So here it creates the summary for us and it did indeed find a problem. So what we see

here is it first profiled and then it identified the noisiest pod and then it found that there was basically a a bunch of processes. It shows the exact process IDs that were running that were ultimately contributing to this stress on the system. So here it basically even shows that the like pod is running stress a deliberate disk stress uh tool which obviously that's what I deployed to

the cluster that's saturating the block device and causing latency for all other workloads. So what I'm asking it here is also like can you plot the histogram for me here because I want to see those like initial uh latency ranges essentially. So let's see it takes a second. Yeah. And there you can see that we have a ton of operations that are happening in a very high

latency range which is not great. You'll see a bunch are happening also a bunch of operations are happening in a lower latency ranges as well. Those are sort of just the normal operations on the cluster, but we are seeing a ton in that high range, which is really something we uh want to avoid. It's not showing up super well on the screen, but this is actually a

chart as well that shows uh the different the bar graph essentially. So it says there's a primary peak um at the normal operations uh range basically and then a dangerous second peak kind of at the high high latency range which I think is uh pretty insightful and using the tools we were able to see exactly which uh processes even were the culprit. Cool. So for the next

scenario, I want to show you a little bit about CPU utilization. I think this is another one that we see all the time in terms of like either there's a noisy neighbor situation, there's just a ton of CPU saturation on the node. Um, and we want to understand what exactly is going on in the background. So in this case, it's a little shorter, but you'll see a

gadget called top process that brings this resource utilization specifically in the context of CPU uh to life. Okay, cool. So here I'm asking it basically, can you help me understand the resource utilization on my cluster? basically. And so for this one, we don't have a skill for it. This one, the LLM is looking at the tool set and it actually just selfidentify that, hey, we need to

use this gadget called top process to root cause this issue and figure out what's going on. So for this one, no skill is kind of doing its own thing and it looks like it said I have a comprehensive view of the uh CPU utilization of the resource consumption on the cluster and here's a summary of what it found. So uh it shows actually memory as well. This

gadget top process shows CPU and memory and it says the cluster is healthy overall. Um, however, there is this one process, this sort of stressengy CPU process that is really taking up a ton of CPU. And again, like I put that there, so no surprise. Um, and it's using up a ton of CPU. And so from there, we can figure out which pod is the culprit, which

node, I mean, which node of course, then the pod, and then the exact process that is putting strain on our system. And this is all thanks to basically this low-level data collected from the kernel by inspector gadget that I mentioned earlier that's then enriched with this Kubernetes data that allows us to figure out the pod namespace and node that uh is essentially driving this and driving this

CPU usage. Okay, cool. So, the next one that I'm going to show is going to be in not in VS Code, but rather using the co-pilot CLI. So, I know everyone like works in different environments. This one is going to be a more CLI driven approach, but it's literally the same exact thing. Um, in this case, we're going to be addressing a pretty common issue around network

configuration issues. So, we will basically have an app that can't connect. We're going to see some uh TCP issues as well. And in this particular demo, you're going to see us use a couple different gadgets. one called TCP dump which is if you're familiar with that that's sort of like a typical TCP dump with that container's enrichment that's sort of the value that our team has added

to that in this particular gadget and then a snapshot socket gadget to enable us to see the port uh details around where the different uh containers are listening. So uh let's dive right into the demo. This is the CLI uh UI for co-pilot. Just this is our you know mode of operation of choice. And we're kind of asking it, hey, I'm having connectivity issues in this specific

demo TCP dump name space. uh can you please help me um figure out what's going on or investigate this? So it's it's the same thing that we were doing in VS Code essentially but uh just within the CLI if you prefer that UI. Um, and so right now what it's doing is it's basically checking the network policies first, which I actually think that's like a good first

step to take to make sure, hey, are there any like network policies, firewalls, anything like that that is acting problematic because oftentimes that is indeed the issue. So I do think that's a good first approach. Again, no runbook here. It's kind of doing its own thing. Um, and then it says, okay, so the network policies are all good, like no big deal. And then it looks for

DNS. DNS resolves correctly, but the connection is refused on port 80. Let me check the web server pod, what the web server pod is actually listening on and capture the network traffic. So here we have uh an enginex pod that's basically listening. Let's see. We'll see. It's listening on port 80, but we'll see the issue. It'll basically it says it found the issue. Let me use TCP

dump to confirm and capture the traffic flow. So here we see the essential diagnost the diagnosis. It first basically used TCP dump to trace the traffic and it identified that there was sort of this TCP reset that was happening and then it used snapshot socket to sort of confirm that oh there was a port in which it was misconfigured. Essentially the service is basically forwarding the traffic

to port 880 but the enginex 8080 excuse me but the enginex container inside the web server uh pod is listening on port 80. So this sort of mismatch of the ports is a pretty common issue that we see and ultimately it used the TCP dump uh gadget to confirm this. The PL the client is sending a packet to port 880 and the server is responding with this

TCP reset meaning nothing is listening on 8080. So the snapshot socket also confirmed that EngineX is listening specifically on port 80. So we have this situation where there's this port mismatch happening which is uh driving this sort of failure in the network space and like just to emphasize I think this would have taken a really long time to dig into the network logs and just like figure

out what was going on. But with these tools, TCP dump and the tools that we showcased, basically you can really see that it was able to sort of take these tools. You didn't even have to really think about which tools to use, which gadgets in this case to use. The AI figured it out and came up with the hypothesis. It did try to look at the network

policies first which I think is totally valid and then dug in further to see oh there is this port mismatch issue that's going on. So yeah, just to really quickly recap because some of the networking stuff, it sometimes helps if there's more of a visual there, but basically we had this uh client sending traffic to the uh Kubernetes service that to port 80 and then ultimately the

service was forwarding the traffic to port 8080, but the EngineX pod was actually listening on port 80 and basically the port mismatch configuration was resulting in a TCP reset that was identified by TCP dump and the snapshot socket filter confirmed that by saying oh yeah the engine X is only listening on port 80. So that's why we're getting this failed connection. So that's sort of how we

can take the two data sources bring them together to create the whole picture. Awesome. So that's it for like the quick hit demos. Basically, I did want to shout out another open source project there. I just want to like call out that like this AIdriven troubleshooting is evolving every single day and the ecosystem is growing. HomePT is another open- source project that our team is starting to

learn about and look into. And this is basically an open it's all open source CNCF sandbox project and it is basically this AI powered on call assistant for Kubernetes. So it's even more integrated than what you saw today. So inspector gadget is shipped with it by default. So that's awesome as well. But it in it basically is able to like take alerts from your alert manager, whatever

your alerting tool of choice is and investigate it automatically and then can even integrate with Slack to provide you updates and even resolution paths as well. Um, so it kind of is like an even more integrated system, which I think is pretty cool. And it's cool to see how the ecosystem around uh AIdriven troubleshooting is evolving pretty much every day. Uh, and yeah, you guys could definitely

check it out if you'd like. I would say it complements the MCP server approach in the context of Inspector Gadget. It uses these same kind of underlying tools or gadgets, just a different interface basically. Okay, cool. So, I did like before wrapping up want to really quickly overview the sort of best practices that the team recommends. So, as always like treat AI recommendations as hypotheses until validated.

Like it's great that we have all of these sort of uh ideas from the AI and the assumptions that it made and even some recommendations to how as to how to fix the problems we showed today. But it's up to you to sort of be the owners of your system and make sure what it's recommending is valid and the data that it is uh looking at and

analyzing is also accurate. So that is one key piece of the puzzle. Another one is keep sustainability in mind. So every query we do to the AI is of course having an impact on our environment. In that sense, the MCP server is actually a pretty cool interface to limit the context of your model because if you remember in the VS code when I showed all the different

tools, you can select the tools that you want to use. So if you know you have a networking related issue, you don't need to provide all the context of the other tools. Just select the networking ones so that you can be sort of a responsible user of these tools. And in that way the MCP server and the tools underneath it allow for that customization and for allow

us to be sort of responsible users of AI. Um the security piece of the puzzle is of course of the utmost importance uh to us. So in the IGMCP server repo, you'll see there's a security guide with best practices on there as well. So definitely something I'd recommend you look at. You all know your systems best and what security guidelines you need to work within. So definitely

keep that in mind. And then similarly to my second point, reduce your scope using prompts and skills to avoid overloading the LLM and our environment basically. So if you know which name space is problematic, like make sure you give the LLM that information. If you are able to like have skills that direct it more specifically, like we kind of showed uh earlier, like please, please do that.

um it helps the LLM be more efficient and also saves you your tokens and context as well. So it's win-win situation. So those are some oper uh like sort of best practices and everything that we wanted to share with you all. Um and then just to yeah just to quickly wrap up. So what what did we cover today? We covered inspector gadget. It's essentially real-time observability powered

by EVPF with Kubernetes enrichment. We showed how you can use it via the MCP server to use a Genai expedite your troubleshooting uh path and meantime to resolution. We shared a few scenarios that hopefully inspired you all to try this out in at home whether that was around DNS uh DNS disc pressure, CPU saturation, networking and um the last thing I wanted to shout out before what's

next is that all of these things are great. like you saw this AIdriven approach to troubleshooting, but I do want to call out the slide where I kind of talk through the different modes of operation, whether that's the CLI or JSON or using this within like with Prometheus and Graphfana, that sort of like old school troubleshooting that's hands-on is not going away. So the LLM can expedite

your meantime to resolution, but I can almost guarantee you that there you're going to find a problem that the LLM cannot solve. And in that way, like please use the tools you have available to you. And yeah, cubectl I don't think is going away anytime soon. Some people might argue with me otherwise, but um what's nice is that you can our project inspector gadget, we have cubectl

gadget. You can use it in that context. And for the networking one, I did want to share we have uh TCP dump integrated with Wireshark. So for any like hardcore networking folks who are really into Wireshark, we have uh integration uh using a XCAP plugin basically that enables really easy use with Wireshark which people have like really loved. So yeah, old school troubleshooting it's not going away.

I think this Gen AI thing that I demoed today expedites things. Super helpful. Can resolve a lot of use cases, but we're still going to need to rely on our like subject matter expertise or domain knowledge. You guys know your system better than anyone else and the historical context. I know many of you have some battle stories uh that you can draw upon in terms of historical

context. So, what's next for us? User feedback. We want to hear from you in terms of if you use the IGMCP server to do this sort of real-time troubleshooting or if you use it in the more traditional way maybe using cubectl via the uh CLI. We want to hear from you. We want to build out more skills with that context for best practices and we want to

continue testing on the different models. Here I was using Claude. Um we've been testing with other models as well. Of course, every day there's a new model to test. So, uh, lots of variability as it relates to that. But, um, yeah, that's that's all for me. The QR code will take you to the repo with all of the materials from today in the manifest folder. You'll be

able to like deploy those to your Kubernetes cluster and like break it like please do it on a test cluster, not like please don't break your own clusters at home. But then you can try out uh what we saw today on VS Code. It's super easy. like I was shocked as to how easy this was to set up. So, thank you for your time and yeah, I'm

happy to take any questions now. Thank you so much, Maya. So, we all have about 10 minutes for questions. I've got one here from Josh. So, >> Yep. So you you kind of touched on this um but I wanted to see if there are other ways to implement this, right? Because one of the problems you run into when troubleshooting is a problem I call the light is

better over here where troubleshooting teams just look in specific areas where they expect there to be a problem. Yeah. >> Even though the actual source of the problem might be somewhere else. >> Yeah, absolutely. >> And going through co-pilot is going to make this somewhat worse because they're not looking at the actual data. But it feels to me like you could add something up front to get

co-pilot to warn people about anomalies felt in other places than what they're asking about. >> Have Have you done anything like that? Have you seen anything like that? >> No, we haven't. I do think yeah, when it comes to the AI driven and troubleshooting approach, the good thing is you can take a more global view. Like if I'm a networking specialist and I see latency on my

cluster, I'm just only going to look at like DNS uh request latency for example, like maybe that's where I'm scoped. But what is good a good thing about the AIdriven approach is that we do have a broad range of tools. So maybe it will go to like it will take a look at the disk and the CPU etc. and have a more global approach. And I think

to your point, we can even instruct it like using maybe the skills framework to like look beyond what you know is right in front of you. And here are some I mean we all have like the best practices we use in our like battle stories or whatever like making sure that the AI is aware that like hey my system's unique in XY way. Make sure you look

in these places that aren't often thought of in all the data that you've already been trained on. So, I think um the problem isn't going away necessarily. There's no like uh silver bullet or whatever for what you're saying, but um maybe some approaches that could help. Thank you. Any other questions? >> Okay, thank you everyone. >> Thank you everybody for being here. Another round of applause for

Maya. Great talk. >> And and the next talk will be coming up at 12:30. As a reminder, ballroom A and B will be the cloud native track all weekend. So, um, thank you all for joining. >> Thank you, Check. Check. One, two. Test one, two, test. >> Okay, sounds good. Hello everybody. Um welcome to our third session of the day um in uh a borrow the cloud

native track. Uh before our speaker gets started, we have a brief announcement from our sponsor for today uh Minimus. Um uh we are very happy for our sponsors because they allow us to have additional stuff here at the conference that we wouldn't get to have. Um so if you work for a company that might be interested in sponsoring the cloud native track next year, please talk to

me. So welcome Minimus. >> Thank you. Uh my name is Adam Clark. I from Minimus. Um just wanted to share what what Minimus does in case any of you are interested. And so Minimus um is a company that ships and builds OCI compliant purpose-built um images that are compliant by default and generally um significantly reduced um from a vulnerability standpoint. And so we achieved that. We we

basically set out to um you know recognize that remediation of vulnerabilities in general is remains a a painful thing especially because of operating system bloat and whatnot. So we take um images, we build them from source, we build them from the bits themselves by monitoring open source software projects and do continuous builds of those um you know every day. And so what you're left with are um

purpose-built images that function the same way, are built to you know deal with scale, but then at the end of the day are compliant by default and have significant reductions from a vulnerability standpoint. So if anyone's interested in hearing more, we're outside in the booth and we'll be in the scale exhibition all weekend. So thank Thank you. Yes. And please do go and talk to them at

the table outside after the session is over. So um on the Maximus end of things, uh Joshua Lee um of Altinity is going to explain to you where to store all of your gigabytes and pabytes of observability data. Um so welcome Josh. >> Thanks Josh. All right, thank you so much for coming. I don't think I will quite take up the full hour, so we should have

some time for questions at the end. And now is your warning. If you're like, I absolutely must listen to a speaker for all of the scheduled time, then you can go ahead and leave. I won't be offended. All right, let's talk about observability databases. This is something that I am a huge nerd about. So, first of all, maybe you're a huge nerd about this, too, because there's

this quote I like to use from Patrick McVaten. Uh, he works on Cassandra, which is one of the databases that we're going to be talking about. Developer will never ask you, "Hey, what file system is that?" But, uh, you're special. You're in this room. So, you care about these kind of nitty-gritty gitty gritty things. And we're going to get down into not quite the bit level, but

but pretty close. So, thanks for coming. So, quickly about me. My name is Josh. I am an open source advocate at Alinity. Uh we do hosting and support for ClickHose and my background is in observability. So that's one of the reasons why I'm a huge nerd about this. Um our lawyers wanted me to point out that um we're not affiliated with Click House Incorporated. They're separate from

us. They maintain the project. We're just humble open- source contributors. All right. And let's start with the definition. This is a simple definition, right? Observability to me is just visibility and understanding. Do we have enough feedback from the system that we can see what's happening? and do we have enough context about what we're seeing to understand it? So, that's simple, but it's eliding a lot of work,

right? We need a lot a lot to do this. And one of the things we need a lot of, as uh Josh alluded to, is data. So, just as a general rule of thumb, your observability system will contain 50 times more data than your system of record for your actual application. Um, it can be even more than that, right? But it's going to be the question is

going to be how many orders of magnitude not how many multiples. Um it's a lot of data at scale. We're talking about pabytes. We have customers who are ingesting two pabytes a day. And that's actually pretty small in the grand scheme of things. What is actually in that data? Well, the bulk of it is the actual telemetry signals, right? So the telemetry signals is this uh how's

that? That's better, right? Okay. So the actual telemetry signals that we're storing, right, are of course metrics. This is like the very first uh signal, right? Um traces, which are sort of the fancy new one. Uh logs, which uh we're all familiar with, right? Like application developers best friend. Uh profiles, which is also a new fancy one in open telemetry. And events, which telemetry. Although also if

we kind of step back from like the the um definitions of the tools that we're using, I would argue that you can kind of think of all of these things as events and they're just events with different types of metadata that we might care about. Then we have getting into that metadata, right? Labels and tags. Um this is like in the Prometheus way of doing things where

we have just like these labels and tags or we can have uh the open telemetry way of doing things which calls it resource metadata where we get a whole I'm so bad with these things. I'm sorry. It's like falling off of me. All right, that's better. Um, okay. So, in open telemetry, we have resource metadata. This is the metadata that describes the actual entities that our signals

are talking about, right? And, uh, I don't think we talk about resource metadata enough because I think it's actually really, really interesting part of observability because I don't actually as a developer, right? I don't care about logs or metrics or traces. I care about a service or a process that I'm trying to debug that's that has an error or is causing latency or something, right? Like I

care about user flows and I care about the code artifacts that I own or control that contribute to those user flows. And those things are represented as entities or as resources. Once I have those things represented as resources or as entities, I can start to graph them, right? I can graph the dependencies from between them using tracing. Um, I can graph stack dependencies by talking to things

like hypervisors or systemd or docker. Um, and I can build topologies of all of these things. Once I have graphs and topologies, it becomes really really useful to store point in time snapshots of those graphs and topologies because they can be expensive to calculate, right? And then it can also be really nice to store deltas because that's a nice cheap way to store changes from those snapshots.

And then finally, right, we're going to have configuration. This is stuff like your users, your dashboards, and your alerts. And this stuff doesn't really need to be in a fast database, right? Like this can just be in SQL light or something. This is this is important, but it's it's very light data relative to the the the telemetry signals that we're actually So what do we need our

database to do? Well, for those other things besides the configuration stuff, we need really, really fast ingestion, right? If if we can't send our observability data somewhere, that's bad. Best case scenario, we're just missing observability data. But the worst case scenario is actually that because the observability backend is down, it causes the sort the service that's sending telemetry to crash because that call is failing, right? And

it can causes a buffer to fill or memory to explode or something like that. So you ideally have architected your system so you don't find yourself in that situation. But I have seen it happen and we don't want that to happen. So we need our database to be really really fast to keep up with whatever we throw at it, right? we might have some uh change or

deployment that dramatically changes the volume of uh telemetry that we're emitting. Often there's a cascade, right? When one error starts, it can cause more errors which is going to dramatically spike the amount of signal coming into your database. And that's the worst time for your database to break. All right, now we have all of this data. It's a lot of data, right? So that's expensive. We need

efficient compression. We need efficient storage. That makes it cheaper to store and faster to query. Um, observability data almost universally is oriented around time, right? So, I'm going to care about a specific slice of time and I'm also going to have retention periods where I say after a certain amount of time, I don't care about this data anymore. We can go ahead and get rid of it.

Um, or condense it or do a roll up or something like that, right? And then lastly, I need some measure of like real time, right? This is in quotes because it means different things to different people, but we really want to be able to get answers in the order of like seconds, not minutes. And milliseconds would be really really cool, but uh that's tough with the volume

of data that we're talking about. I got to stop moving my head. Okay, what is real-time analytics? Well, I like Peter Marshall's definition here. Anything you can do with a group by, that's what analytics is. Oh my god. >> Yep. Perfect. I'm going to tape myself up real quick. Sorry about this, folks. Thanks for bearing with me. All right, that should help. Okay, so Peter Marshall said

this. It's great. It's hilarious. Anything you can do with What he means by this right is like if we need to do any kind of aggregate calculation over thousands, millions, billions of rows, right? We need to know the sum, the mean, the P99, the P95, whatever of a metric value. That's basically what real time analytics is. And if we can do this slicing and dicing across different

dimensions, even All right, so we need really if we translate real-time analytics, it means we need fast multirow analytics. And then on the retrieval side, right, if we're being really lazy, we need full text search so that we can find our log messages that we care about. If we're being a little bit more consistent and prescribed, then we need really good tag and label search so that

we can find the things that we care about. Um, if we're doing alerting, we need to be able to get the last sample across a huge uh number of different dimensions really, really quickly so that we can alert on that. and your um your query volume right your read volume is going to be your alert frequency times your alert configurations so that can that can create a

lot of load on a database and then updating data I would argue most of the time we don't need to worry about there are rare situations where you'll want to update observability data but that's pretty niche and for most of us we can just treat this as an appendon immutable situation databases come in a couple of flavors and I'm going to talk about the different flavors of

databases and we're going to talk about different inst uh examples in each one of these categories. Um but at the end of the day, right, um taxonomization is hard. We'll get into that. So, OLTP, uh this is the type of database that we're all most familiar with. This is your MySQL, your PostgrSQL, right? It's a transactional database, which means that it has very strong acid guarantees, right?

Atomic, consistent. Um so, that's what we're all used to. OLAP is another type of database that we're going to talk about a little bit today. This means online analytics processing database and so they're a little bit more suited for these types of like really really fast ingestion that we're seeing um we need for observability and also with that real-time analytics portion. Um we're going to talk about

time series database which is sort of an architecture uh Prometheus would be the the canonical archetype of and then finally we'll talk about search and analytics which is uh sort of a category that I've made up but I think uh you'll see how it applies. So in terms of examples right we have postgres is the is the OLTP I'm going to be looking at. MySQL is a

perfectly valid alternative. Um there are some nuances in the architecture of the two. So I'm specifically going to be talking about the Postgres architecture, but it's not that different. Um, I'm going to talk about Cassandra a little bit. Um, I'm not that qualified to talk at length about Cassandra. I'm going to talk about Elastic or Open Search, um, Prometheus and Friends and Click House. So, right, I

said taximization is challenging and really what's been happening over the last 5 10 years is that all of these databases have been borrowing tricks from each other. So, they can all kind of do anything that you need them to do with a few exceptions. Um, they're not going to be the best at it, but if you need to, you can kind of make one database do everything.

Um, and here's kind of my map of like where the databases that we're talking about kind of fit in, right? So, let's talk about storage on disk for a minute. This is the most critical part of database speed, right? it's always going to be the bottleneck in your database. So, all of these tricks that I'm talking about really come down to like how can we minimize the

amount of writing and reading that we need to do from the disk without introducing um unreliability in case of a system crash. So, it's important and there are a couple of different styles that these databases adopt in terms of how they actually store the data on the disk. So, the first is a heat pages with like some sort of commit log or wall. If you were on

the post track, you probably heard a lot about this sort of stuff. Um, you can have time series blocks, which we will talk about what that looks like when we talk about Prometheus. And then you can have parts or segments, which we'll talk about um when we talk about uh the lucine family. So, first let's talk about heat pages. It's kind of like the Jbod, right? Just

a box of disk, just a bag of discs of storage tiles. And basically the way that it works is um we have we're trying to com compact all of the database data into as small of a space as possible. So when we need to write a new row or column or document depending on the orientation of the database that we're talking about but usually it's a row

for postgress for example and a row is stored just as a tupil of values right because we know the schema and so we just need the tupil of the values for the row and then that gets all those rows get added to pages and then when you're doing inserts or updates right um new pages get created old pages get marked for deletion there's a vacuum process that

happens either in the background automatically or because it was triggered and that actually removes the page from the disk and then the next time a page needs to be inserted, it just gets inserted at the next available slot. So, we're just keeping everything nice and compact. That's sort of the goal here. And if we combine that with a write ahead log, which is how we achieve um

durability in case of a crash when uh that database is is being updated and written, we also have heap pages, right, which we just talked about what that is. Multiv- view concurrency control is a trick that um Postgres use is to uh um do the consistent part of the asset guarantees right so each process basically is looking at a single snapshot in time of the entire database

um and then it uses bree indexes as the way to actually store the data which is great for really quickly finding data but not great if you're doing really really frequent inserts because you're constantly shuffling around your indices. So what ends up happening right when we use this is we're optimized for updates and upserts rowle reads because we're storing data as rows. We have really really strong

acid guarantees but we basically have achieved concurrency by keeping everything on one system and not introducing network latency and so scaling horizontally becomes really really challenging. Let's talk about analytics and um right so let's talk about vector engines and search and specifically here we're talking about elastic search and open search they add a couple of tricks to log structured merge tree to make your data even easier

to find so that we know exactly which part of the disk we need to read and we don't need to read the whole thing to find the data right this is where indices come in so to start we have inverted indexes those are pretty straightforward then we have bloom filters those are a little more complicated I will try to explain them um and then we have approximate

nearest neighbor which is for vector search. So, inverted indices, this is it's we're just flipping the script, right? Instead of our keys pointing to our data, we're taking the values that are in contained in our data, usually words or tokens, right? And we create an index where those point to the pages or the documents that contain that token. So, this way, if I want to search for

a literal, like I want to search for dog, I can have a an index that tells me exactly which of my rows contain the word dog. somewhere in a field, right? This is cool, but it's also kind of expensive because you have to keep these very very large indices, right? They're quite dense. Um, so they usually have to be stored on disk. Bloom filters um are another

way of doing this, right? So it's a way of doing an inverted index where you don't have any false positives or sorry no false negatives, but your positives might be false. So it says this value might appear in this row and it will tell you for sure which rows it doesn't appear in. The way that it does this, right, is we have kind of I'm going to

call it a hash, but it's not a cryptographic hash, right? We're just taking the strings through a function. We chunk it and then for each chunk we either do or don't flip a bit in the in the resulting value, right? So we're basically condensing the space of all of the words that we might store in our index down into a much smaller numeric space. And in this

case, right, if depending on how you tweak the settings on your Bloom filter, you can get it so that OTEL collector prods to the value 31 and OTEL collector prod 10 also maps to the value 31. So a bloom filter that has 31 in the index, we know that it's one of those two values or something else that maps to 31, but we don't know for sure

which one it is. So, we're still going to have to go read that row and do some final checks. Um, so we do have to scan a little bit more data than we need, but the result is right, we have this much much sparer index that maybe we can fit the whole thing in memory and then it's way faster for us to know which parts of the

disk we need to go look at. So, that's pretty Then we get to approximate nearest neighbor. I have a somewhat fuzzy understanding of this but um as I understand it right dog the the like literal string do og that's one thing but if we want to search for the concept of dog then we're talking about vectors right and vectors are just multi-dimensional points in a multi-dimensional space

and in that multi-dimensional space we can calculate nearness so we can say this vector is nearest to this vector and then once we've done that we can kind of create these neighborhoods and then we can create layers on top of that that allow a query engine to sort traverse through the neighbor through the layers and then through the neighborhoods to find the vectors closest to a specific

point and that way we can search for something like the concept of a dog right all right so then let's talk about Prometheus and friends or time series so time series blocks we're back to an append only immutable structure right except it's much simpler instead of storing tupils we're just storing values individual sample values um you have one time series database file per time series, right? And

then you just add the samples to the end. And this is really, really convenient. It's really simple architecture. And it's great like we can think about if I wanted to show this on a graphana dashboard. Well, I just look at that file and and like read the samples or I can read from the end of the file if I want the most recent samples, right? And that's

very very easy query and it's very very easy to just then graph that. But it does have one problem, one major problem. Cardality explosions. So if you've ever actually run Prometheus in production, I'm guessing you've run into this, but the number of time series that you're going to have is roughly going to be your cardality to the power, right? Your maximum cardality that you have in one

field to the power of the dimensionality, the number of fields that you have. So that can explode really, really fast. And if we think about having a single file on the file system for each one of those time series, right, that is not really a way that computers like you to interact with the disk. if we're talking about millions of these or hundreds of millions of these.

So, we have to really really carefully guard our cardality and our dimensions when we're dealing with these types of databases. But they do have one really really cool thing that they're doing, right? I mentioned that that data is all laid out ready for us to read, right? You can kind of think of that as a column oriented database. So, in a row oriented database, we're storing tupils

with the values for each column next to each other for that row. um we can't really compress the data because it's going to be of varying types within each row, right? And we when we read the data, we have to read the entire rows even if we only care about some of the columns. So if we're trying to do math on say one or two or three

columns, we have to read the whole table pretty much. In a column oriented database, we only need to read the columns that we care about because the data is actually stored in by as the values in the column, not as tupils for the entire And so, click house takes is a columner database that uses an engine called merge tree, which is sort of like an evolution of

log structured merge tree. And with this architecture crap, we can take let's say we have a table that has 109 columns. It's 60 gigabytes roughly, right? And we want to read only three of those columns. Well, if we're using a row oriented database, we have to read all 59 gigabytes. That's going to be really, really slow. We have to pull that data off of the disk and

process it, right? If we go into a column oriented database, immediately we see this benefit, a 97% reduction, right? We only need to read three out of those 109 columns. So that's huge. But then because those columns are all of the same type, they can be compressed much more efficiently. So then we're down to only 21 megabytes of read, right? And then we can actually parallelize this

across multiple threads or even across multiple nodes. So each thread, if we have eight of them, only needs to read 0.0044%, right? fraction of a hundth of a percent. And this is how we um an OLAP database can take a query that would take thousands of seconds in a transactional database and run it in under a second. Um so here's how merge tree works. It's very similar,

right, to log structured merge tree. The only difference now is that we are storing columns instead of key value pairs. And one more trick click house has up its sleeve, it makes uh heavy use of sparse indexes. It also uses bloom filters and inverted indexes. But sparse index is basically when you have an ordered key like say time which we're always going to have for observability data.

We can create a sparse index on that ordered key and we can have a really really cheap in-memory structure that just tells us roughly where to go on the disk for a specific point in Okay. So let's talk about how you can actually apply this and choose make some choices for your own situation. So this graph I made uh when I first wrote this talk. This is

what was running in my home lab. These are all great choices. Um sort of the canonical choices if you will, right? Like Prometheus is the canonical time series database. Um Elastic Search or Open Search is the canonical full text search database. And uh Click House is pretty awesome for tracing. Actually, if you name an observability vendor, odds are that they're using Click House at somewhere in their

stack if not as the entire engine. So these are all good choices. This is uh more up-to-date vision of what I'm working towards in my home lab. So, we have a bunch of open telemetry collectors working as agents. We have Click House as the main observability engine. Um I'm finding that getting all of my data into one database and then interacting with it using an MCP is

like my favorite way to deal with data today. So, that's fun. But then I still want to make sure that I have like monitoring for my monitoring. And I want that to be way simpler because ClickHouse is awesome, but it's also kind of a bear to operate. Um, so the simple meta monitoring stack that I'm moving towards is probably going to be Victoria metrics and Victoria logs

and a simple graphana dashboard. And there I'm not so worried about some of these things because I'm going to very very carefully guard the cardality of exactly what's going into that meta monitoring database. Uh, in our own cloud at Altinity, we use Victoria Metrics. Um, it's great. It takes a time series database architecture like um Prometheus and combines it with some of those background compaction tricks from

log structured merge tree and merge tree. So it's kind of like the best of both worlds. Um if you're dealing only with metrics, uh we also use Loki in our cloud at Altinity. Um it's a really uncomplicated logging uh service from Graphana. It's actually architecturally very similar to Prometheus. So you have an appendon data structure that the logs are going into. The indexes are by label. So

if you have good label indexes, Loki is awesome. If you need full text search, Loki will do it. But it will have to scan the entire database and it's going to be really slow. And then some honorable mentions. Um Cortex, Thanos, uh these are CNCF projects. Um Mamir, which is like the graphana evolution of Cortex. Um there's time scale DB, which is a plugin for Postgres. Um,

there's Apache Solar and Apache Druid, which are both good at some of these things that we've been talking about. So, these are all great things to look into. Uh, I just didn't have time to talk about all of them So, more advice on terms of what to choose, right? Because this is a complicated question. I can't just give you like two examples or three examples, but at

a very small scale, I'm a huge fan of just using what you already have. So if you already have Postgres and you just need a little bit of some observability data or something, you can maybe get away with this. If you really really love full text search, there's really not a better choice than Elastic Search or Open Search. If you want to use one database for everything,

uh like the example I showed, ClickHouse is pretty cool. It's pretty good at that. Getting better every month. If you want to do wide event analytics, Click House is just awesome. Really shines. If you really really care about alerting and that's your primary use case, then you want a time series database like Prometheus or Victoria Metrics because you're going to have much more efficient time doing those

last sample reads all the time. And finally, here's a table uh kind of summarizing what we've talked about. Um yeah so right we have our row oriented postgres uh with SQL. I'm a huge fan of SQL. Um it's using heat pages and B tree. We talked about this. It's good for update and upsert with guarantees. It's good for your actual transactions. Cassandra can be a transactional database

that if you need like a little bit of wide event analytics but you mostly need a transactional database, Cassandra can be a great choice. Um, we talked about why um, Prometheus and Friends are great if you care about alerting. Um, open search and friends are great if you care about full text search and OLAPS like click house, uh, Druid or Doris are great if you care about

real time analytics. This is my last slide with the QR code if you want to scan it. And I see a question. So yeah, this is I go ahead and open up the questions. Sorry, what was that? >> Oh, Seth. Yes. Yeah. So, if you if you if you have a lot of stuff, uh Seth is the backing engine, that's a whole other can of worms, right?

Because some of these databases um are distributed applications themselves and then like combining distributed application on distributed storage. There are ways to do that really well and there are ways to do that really poorly. Uh yeah and if there if there are other uh thank you and happy quering. Um, so what would be good for people that have like a small to medium infrastructure in a cloud

service? In my case, it's AWS where you want to get observability monitoring, you know, metrics, don't have anything yet, but also don't want to have an explosion of 50 instances or a whole bunch of stuff. you know what's a good in your opinion basic simple infrastructure to get >> um so a really popular one would be if you want to kind of like glue the pieces together

yourself and have complete ownership of all of it the graphana stack is a great good way to do that so whichever bits and pieces you want to take from the graphana stack that's really really popular um there are also some like there's been a Cambrian explosion thanks to open telemetry of just off-the-shelf open source solutions so kroot and signals are two of my favorites um click Click

click stack from click house incorporated is also another good one that's basically turnkey. So it'll have its own databases. Um all of those use click house. Kroot also has a Prometheus instance. Um and it'll it'll sort of spin those things up for you and then also provide an observability platform on top of that. But I also like if you're using hotel you could then also fork that

data into click house and use an MCP just like I am. And yeah that's the beauty of open telemetry is we can kind of send our data to multiple destinations. Uh yeah, >> question back. On your last slide, you had a uh little star next to TSTB. Ah, yes. It's not really columner. I'm calling it columner, but it's like I don't know that other people would call

it that, but it is it's just the values all next to each other. So, is that not a column? But yeah, since I came up with that, I just was a little I was hedging. Uh so the question is am I still using graphana for time series visualization? Yes. Yes I am. >> Can you talk about your uh use cases for the home lab? >> Yeah. Um

I did a talk on this as it's recorded um for the hotel end user sig a couple of weeks ago. Um but basically I have two use cases. One is like it's actually a lab, right? like I'm using it to do things for work before I do them in an environment that costs money. Um so running like VMs and Kubernetes clusters and things like that running uh

like just workstation VMs if I want like a clean environment to try out a tutorial. Um, and then the other like tenant that I consider who's also me is like I'm running some services um actually in Docker, not even Kubernetes because I keep it simple stupid, right? And uh that's just for like things like replacing Google Photos with self-hosted apps. Any more questions? Oh yeah. Okay. Um

in your presentation I think you use click house as consuming the data the events do you know if the click house itself emitting hotel data so that um >> yes great question I give a talk about this if you search my name and tracing click house I'll you can get the more like hourlong answer but the short answer is that click house stores a bunch of uh

metrics and logs internally um you can get those out using hotel and it also if you turn it on will store internal traces is in an hotel compatible format, but it all just the way that ClickUp does its monitoring is it's all just in the system tables and you have to pull it out of the system tables. >> There's an hotel exporter to send any hotel data

uh logs, metrics or traces to ClickHose. Yes. And I think that the ClickHouse schema forel should it's it's looking like it will become the deacto schema for any SQL compatible tool that ingests open telemetry. It it looked like you had an MCP server hooked up to your um Click House. Yes. I was curious what how you're using that. >> Uh for me that I go there before

Graphana. Now I can have Claude I ask Claude, "Hey, what service is using? Why is the memory high on this VM?" And Claude will figure it out or I can ask, you know, what are the top pods? Uh yeah, it's amazing. And then because it's also sitting running in a directory with my anible inventory, if it has to SSH into something, it can. Guess >> Ah, well,

yeah. I guess anything that comes back from the MCP, yes, is getting scooped up into their into their cloud. I'm okay with that. It's not that sensitive. I'm pretty public person, but yeah, there's there are situations where you wouldn't want to do it that way, for >> Uh, speaking of moving data around, in one of your examples, you have logs, metrics, and traces going to three different

data stores. Yeah, >> right? Which then causes some challenges for correlating that information. >> It absolutely does. Now, what's nice is the res if it's all open telemetry, it's all going to have the same resource metadata hopefully. Um although yeah, this can be a problem if like if you don't have the right metadata on your container logs to necessarily correlate it to like the hotel SDK that

was running in that process. um that can be a challenge. Yeah. Um that's one of the reasons why I like putting all the stuff in one database, but there are definitely arguments for more of like a a Unix each thing does it one thing well philosophy. >> You mentioned topology. So if you're gonna start doing everything in one one place only, do you have to worry so

much >> do you have do you have to worry so much about topology? >> If you're just an individual developer um working on your own, you don't care so much. Topology where where I find topology is useful is when like you start to have questions of who owns what because you have so many people working on the thing, right? Right. So at enterprise scale when you have

you know 70 teams working on a thing topology becomes really really important because it helps you drill down to ownership. >> I definitely uh felt what you were saying when you said that uh time series databases like Loki are not so good for full text search. Mhm. >> Because I will sometimes try to search for logs and then Graphfana is like, "Oh, your query is too big.

You got to refine it some more." So, like, >> do you have any sort of tips for like how to um I guess do better full text search with Loki? Is it just add more labels or do you have >> Yeah, if you're sticking with Loki, it's add better labels and don't do full text search. Do label search. >> Um because those are the only indexes it

has. You could switch to something like Cassandra Solar or Click House or Open Search and all of those have um better inverted index support. >> Uh question over here. >> I can agree. I can agree. We don't do any uh searching uh in Loki. Uh it's all eventually in open search and all searching through logs goes is an open search search. Uh our value that from Loki

is that uh it is matched up with Prometheus so that you have the same um cardality so you have the same structure of logs. uh and uh metrics uh from the beginning and then uh presenting it to someone who's working uh the production is uh more along the lines of open search will search things uh much better. So we have a lot of links out of graphana

to uh kibbana uh so that you know you see the odd thing you want and then you go find out exactly what it means. So, >> Yeah. That's pretty that's a pretty common way to do it, too. >> Anyone else? Don't be shy. I have a bunch of stickers up here if you'd like some. And I also have more of this t-shirt if you talk to me.

Oh, one more >> This talk doesn't have the link yet. You gonna add that or is that just a >> Oh, it's not up there. Yeah, you have the DevOps and magical mystery tour, but not this one. >> This is magical mystery tour. >> Oh, this is okay. [laughter] >> My bad. My bad. >> Sorry. You Yeah, the first slide was uh >> Yeah, it wasn't there.

It was like kind of angled and Yeah, I needed I didn't need a theme song. [laughter] >> Okay. Well, thank you very much. and and the cloud native track will resume at 2:30 p.m. in this room and ballroom B Yep. Hello. Hello. Yep. Welcome back everyone. Hope you had a good lunch and you're ready. You had plenty of brain food. So now you're ready to learn all

about advanced Kubernetes scheduling with the API. Yep. For priority and fairness. Um so uh Naresh Amratham is going to teach us all about APF um and how you can actually use it to schedule your workloads. So take it away Nesh. Thank you Josh. Hello everyone. My name is Naresh Amritam and I'm a software engineering at um Cruso and today we're going to talk about demystifying Kubernetes API

priority and fairness. Before we begin, let me ask a quick question. How many of you actually uh deal with Kubernetes on day-to-day basis? That's good to know. And um do you like how many of you are Kubernetes cluster admins? Got it. that set some context on like how I can talk about the talk. So what is APF? So API priority and fairness in Kubernetes is actually a

mechanism to classify requests and isolate them into a more fine grain way by introducing prioritization, queuing and isolation. So critical requests always get through and no single client can start the rest. And this feature has been uh released as part of uh version 1.29 as a stable and then it also like gets enabled by default uh for your cluster unless you um specifically disable that feature. So

why do we need APF? So uh before APF in Kubernetes the only way you could control the number of requests is through two flags. One is maximum requests in flight and max mutating requests in flight. So one of these the first one controls the read request and the second one controls the write request and uh by default it is set to 400 read and 200 for write

requests. So this kind of creates a problem when you have a busy cluster where you have a lot of clients interacting with the Kubernetes API server and specifically if you have a noisy clients they can essentially take away all the requests which are available and then impact the regular operations within the cluster and probably even bring down the cluster and sometimes during the situation even Kate's admins

won't be able to access the cluster which is in a a problem to actually stabilize the So as you can see here the noisy clients requests from the pool from the API server and then it can kind of halt the cluster. So the the main concerns are like there is no awareness of what client is actually taking up the requests and one noisy client can consume all

of your request capacity and uh critical requests have zero protection and can be instantly dropped and there is no protection for the cube API server. So with this uh it's pretty much required that you need to have some mechanism to handle the request in a more fine grain way where you have some protection for critical requests and at the same time you kind of low prioritize the

requests which are not essential. So these are the goals around designing the APF. The first one is fair sharing, prioritization, queuing and flow So phase sharing when you have a limited set of requests that can be allowed, you basically need to uh equitably share the request pool among different clients in in a way where you can actually assign um a good amount of requests for critical uh

clients and resources and uh probably a small portion of requests probably a small portion of requests for like some non-essential and non-critical clients. And prioritization is again um how do you prioritize critical requests while serving the requests and at the same time give low priority to the requests which are not critical and queuing helps in actually handling the sudden burst of traffic where um instead of like

dropping the request you kind of get them queued and then serve the request. it does come with a problem of like high latency but at the same time it's better than actually filling the request. So uh when the requests are full um then the request get cued so that once there is still connections available it can serve Uh flow isolation is again uh to limit the excessive

and noisy requests. Even though you kind of have a mechanism to kind of handle uh different wide variety of clients, you still need to understand if there is a noisy client in this whole list of clients and then you want to make sure that that noisy client doesn't impact rest of the requests. So at a high level uh APF works in this way. You have incoming requests

and then you have uh a mechanism to match the request where the request gets classified based on um with the user um the request being made for certain name space a resource like pods and also verbs such as list get I think there's some problem sorry excuse me and also verbs such as get list put patch and so on so assigning Priority is the next state where

once you classify the request in one of these categories, you assign the priority based on um uh things like request. >> Try try it next. Okay. So the next step is to assign priority to the request and for that you basically need to assign the request to either a critical request or a leader elections or it's a controller request or it can be a catch all default

uh priority class and then you have the next step of queueing where either requests get skewed or request can be like you know dropped if it goes beyond this assigned assigned capacity and the next is to actually um classify the request to be exempt from these cases. For example, in case of uh um like requests coming from Kubernetes admins or some like leases or probes, you probably

want them those requests to actually always go through irrespective of in what state the cluster is. Um so that's the exempt and then after this the request is handled by the API So let's look at some of the uh resources which are part of the API priority and fairness. The first one is called flow schema. The flow schema basically classifies the request based on the user verb

resource and namespace and routes them to the appropriate priority level configuration. For example, in this case we can see um these are the default priority uh flow schemas which are gets which gets created by the Kubernetes itself. And um as you can see the first one is exempt which means that these requests are exempt from um being handled by the APF. So these requests get through no

matter what the cluster state is. And um the second column is basically the priority level. So the first is flow schema to classify the request whether the request has to be uh exempt or is it a request of type probe system elections and so on. So some of these are actually um work in a way where you can define the resource that being called. For example um

uh let's look at uh service accounts which is in the bottom. So any request which is coming from any service account user gets assigned a priority of workload low and uh similarly any request coming from nodes in this case system nodes that gets a priority level of system and so on and you have something called matching precedence where you have a list of numbers along with it.

So the the lower the number that gets the highest priority so that um you make sure that requests are being handled in the right way. And the distinguisher method is basically is telling that the request is of type by user where you kind of classifying the request based on the user. And uh by namespace is basically saying that this request is being made for a specific name

space or you can also have none where you don't really want to classify based on the user or namespace. We'll go we'll dive deep into this further flow schemas and how the spec looks like and how you can configure them further. So here is uh on a high level um how the matching precedence works. Let's say we have a user who is making a list call. At

the same time, we have a service account user also making a list call for a non-cube system namespace. And as you can see, we have uh three different flow schemas. The first one is uh a list call for cube system. The second one is list for service accounts and the third one is list for any user with the matching precedence. So in this case since the regular

user is not a service account user and also not making a request for a cube system namespace um the request gets uh uh I mean the APF picks the flow schema as the third one because that what it matchs for and the second case like in cube the service account user who is making a request to a cube system resources um basically sorry this is a non-cube

system so basically gets picked by um the second one which is match match list for a service account. If a request is being made for in this case let's say for cube system the first one would be picked. So here is the flow schema resource spec as you can see at number one that's the name of the flow schema in this case the flow schema name is

service accounts and the second one is uh at the distinguisher method by user. So basically you're trying to classify this particular request by a user. It also takes the values of by name space or none as we have discussed earlier. And then we have something called uh at number three matching precedence. That's the ordering where the APF tries to match a request and picks the one which

is having the lowest number. And then you have something called priority level configuration uh at number four which is basically how do you take this request and assign priority. Uh that's what this defines. So in this case I was not able to capture the the subjects but if you can see in the side it's basically saying that any system service account user will be part of this

flow schema. Um next is priority level configuration. So basically once a flow schema is defined for a request the next thing is it forwards the request to be handled by the priority level configuration as the next step. So once um the priority level configuration basically defines the priority tier um how many concurrency shares and uh whether the request will be going going through a queue or not.

So this particular uh request is handled where uh when uh when there is a over capacity where you know the you basically are consumed all of your concurrency or shares you're going to either admit the request or Q or reject based on um where the Q currently stands So uh you can display all the priority level configurations by running cubectal get priority level configurations and then um

as you can see catch all is basically like a uh a default uh priority level for most of the requests and the nominal concurrency shares assigned are like five and similarly this doesn't have any cues or any such thing. So if the concurrency shares are completely being utilized, the request gets rejected right away. And then we have exempt where uh requests coming from critical resources or K8

admins or few other critical requests which you think are very important. They will go through the exempt um priority level configuration and in this case um APF is not enforced meaning the request will be always be allowed. And then you have other uh nominal concurrency shares for global default leader election node system workload high and workload low. So by default Kubernetes doesn't give you a mechanism to

actually u distinguish uh a mutating request and a non-mutating request but you can definitely add u a further flow schemas to make sure that you kind of prioritizing the right requests and then allowing a excuse me and then allowing a separate um queue for the read requests. So I think in this case I think one thing to note is the type uh you have of limited or

exempt. um exempties uh where you know APF is not enforced and the request is handled and um allowed no matter what is the state of the overall concurrency shares and the type limited basically means that um whether you want the request to get cued whether you want to sorry whether you want to get the request cued or whether you want to get the request to be rejected

and uh number of cues dep defines like how many cues that particular flow control has and hand size is something which is used for uh shuffleing. We'll discuss more about it in the next slides. And q length is basically the length of the each of these cues. So at any given point of time when you say like my que is full you basically mean that number of

cues times the q length would be the total requests. I'm sorry. So total requests which are currently in the queue. Um so let's look in detail what exactly uh priority level configuration means. So as you can see this is of kind priority level configuration and the one we are looking at right now is workload low. This is generally used for all the read requests within the cluster.

And then um this is if we go to number two it says it's of type limited. The other type you have is exempt where requests are allowed no matter what the cluster state is or the cues are. And within limited you have uh two types. One is Q and reject. So Q is used when um the request doesn't have like there is no availability of any concurrency

or um uh shares or you basically cue the request and if you have a type reject when the cues are full you basically reject the request right away. Um, oh I'm sorry. So when you have a type Q, you basically cue the request. When you have reject, uh, you basically reject the request right away when there are no more concurrency shares. Um, additionally, you have some other

parameters to fine-tune how you want to handle the request to limit the um, collisions or have better isolation. For that um, you define the number of cues, what is the queue length and hand size. And then the last one is lendable percentage. uh this basically allows you to lend the concurrency shares assigned for this particular priority level to other priority levels when um these concurrency shares are

not being utilized. So it kind of when you when the cluster is free you basically have rest of the request use the concurrency shares but when the cluster is busy you basically you do not lend it. So um I've talked about a lot about concurrency and seats what exactly that means. So the analogy of seat is basically how a particular um like how seats are generally assigned

in an uh train or aircraft where you have limited set of seats and you're trying to accommodate passengers. In a similar way you have limited set of requests and you want to assign um um how many uh shares a particular request can take. So in this case uh for in APF the concurrency of number of seats uh per request is basically or per priority level is being

calculated using the following formula which is basically take the total number of requests API server can take times um nominal concurrency shares for a given priority level divided by sum of all nominal concurrency shares. So what does that mean is basically in the column three you can see that's the nominal for example I want to know um the nominal concurrency shares for catch all so it has

configured to be five so you basically take five divided by sum of all the concurrency shares to get the the number of concurrency or seeds allowed for a particular priority level. So here's an example where we are bas where we are basically calculating the the concurrency for catch all and uh as you can see the nominal concurrency shares is set to five and uh assume that maximum

allowed requests is 600 and uh sum of all concurrency shares is 245. So you basically divide 600 * 5 245. So that is 12 which is basically uh at any given point of time your API server can take 12 requests of type catch all and uh that's what it means. So whenever the requests exceed that uh 12 concurrency in this case the type is set to reject

means the request gets rejected with um HTTP 429 too many requests uh if at all if it is a queue the request gets request gets ceued and eventually being served once the uh concurrency is um available or when it's free. So how does the the APF uh calculates number of concurrency or seats to be used for a request? For simple for simple requests such as get, patch,

put and so on. These are like um a simple request you kind of request an object or you make a a right to patch an object and so on. So the number of seats allocated is one. However, there are certain requests which are list um which kind of returns uh a set of objects. APF kind of um estimates how much objects are being returned and then it

kind of takes in take into account like number of seats based on that. For example, when someone runs a command cubec get pods across all the name spaces, you probably um let's say get like thousand parts in that and um Kubernetes kind of um estimates for thousand parts request it kind of estimates probably like you know 15 or 20 or 30 um seats being utilized for the

particular request. So this is a way where you are kind of making sure expensive calls are being um uh throttled in a way. So you don't necessarily get the API server to get spiked with CPU and memory. So let's look at Q's. So we did discuss about like uh uh in priority levels you have cues and cues are utilized when the utilized and you kind of queue

uh cue the request. So how does that work? So since if you have more cues meaning like you have fear collisions and when you have only one queue you effectively saying that you're disabling the fair queuing mechanism because if you have just one queue every request for that priority level will get into the same queue and you essentially can have if you have a noisy client uh

pretty much that one client request will be filled up in the queue and you're essentially like impacting other requests. Um so therefore uh more cues means uh fewer collisions and then we have Q length limit. So in each queue how many requests do you want you want to save. So in this case um when you have low limit uh in this case like when you have sudden

spikes of requests this causes the request to get dropped. When you have a higher length of Q length um even sudden when you get the sudden burst of request the request gets cued. However, this still causes a problem of like you know heavy utilization of memory and also it also impacts the request uh latency uh for the cube API server. So let's look at what exactly is

hand size. Um one of the parameters for priority level configuration is um hand size which is used to fine-tune the overall queuing mechanism. And um the way it works is in generally like when you have a set of cues and you want to distribute a particular resource or a request to the cues you basically um modulo the request based on certain attributes. For example in this case

we can take uh user name space um we can take resource and also the verb and then we can do a model on that and based on number of cues and then then you send the request to that queue. But um since multiple requests can still modulate to the same queue, you're essentially not really isolating the request from any of the bad actors. So here that's where

the shuffle sharding algorithm comes where you define a hand size and uh when you have requests coming in you basically create uh calculate the hash and modulo that particular hash with number of Q's but with hand size uh and shuffle sharding you basically instead of getting one Q you get three Q's because in this case it's hand size equal to three. So uh the resulting set is

three cues and uh within the three cues API server will pick the one which is u least used and then assign the request to that. So in this case although you have two requests flow A and flow B coming in even if the flow B is kind of noisy and is like is trying to um bombard the API server with too many requests the chances of u

that flow fully impacting the other flow is still less because of the hand size. Uh if you increase the number of u hand size it can further reduce and gives better better isolation. uh but overall going really high on these numbers will eventually like you know create a problem with like you know overall uh CPU and memory consumption. So let's look at how the request actually flows

through the whole process. So when the request uh arrives at a API server cubernetes API server the first it has to go through the authentication followed by authorization and then that's APF kicks in if it's enabled the first step is to classify the request based on u the flow schema the flow distinguisher and the priority levels and um once that has been classified based on the uh

the priority level assigned to that particular flow the request gets to the uh state where you know it gets um it looks at the priority level for that request and then if the request has the priority level with exempt the request gets executed and in the second case let's say the request is of type Q and um uh in this case what happens is the request first

checks I mean APF checks whether do we have any seats available to serve this request if there are then and it executes a request and if there are no seats available at this point of time the request gets into the queue and then again it checks back if the seats are available the request gets executed so that is so that that is that's the place where you

are kind of adding more latency for the request because the request is kind of staying in the queue and being um verified for the seats to be available before it can gets executed. So in this case let's say there are no seats available and uh the queue is full in such cases the request gets rejected with HTTP 429 too many requests and in case of uh priority

level of type reject um if seats are available the request gets executed and if seats are not available meaning there is no cues to be used in this case. So the request gets uh rejected. So that's how the overall flow works like for every request which comes into the API server uh with APF enabled. So uh here are some of the metrics which are useful in order

to understand the overall performance of APF and whether you want to tweak any of the existing uh priority levels, Q sizes or other things. Um um so apart from the regular Kubernetes request latencies, error rates and overall request total count uh Kubernetes also Kubernetes API server also provides you with u certain metrics which are very specific to API priority and fairness. And as you can see um

all of these requests are prefixed with API server flow control because APF is about flow control. Um the first one is API server flow control dispatched request total. Um this particular metric counts the number of requests which are currently being dispatched by the API server. And the second one is current requests which are in Q. Basically when requests are being served and if the concurrency shares are

busy the requests are in Q. So that this metric kind of gives you that some metric always in the up high and uh if you think that um the requests which are in Q are basically some things which are like critical you probably need to um increase the number of concurrency shares so that you're not necessarily actually making the critical request to uh experience high latency and

stay in the queues for a longer time. And uh one thing I want to mention is that all of these requ uh metrics have uh two dimensions. Uh one is called uh flow schema which kind of tells you like you know which flow schema I'm looking at. At the same time they also have other dimension or label called priority levels. So you kind of know which flow

schema and which priority levels are uh performing well where you need to tweak and so on. The third metric is API server flow control rejected request total. So this metric kind of tracks the total number of requests that are rejected when either cues are full um or when you don't have any cues being utilized but you're getting too many requests. This is not necessarily a bad thing

because the goal is to actually ensure that you are not taking in noisy requests or you're trying to limit expensive calls to the QBI server. So in a way seeing this number is not a really uh a bad thing but um [clears throat] this kind of gives you an overall performance of how many requests I'm getting which I think um like you know API server is kind

of helping and protecting the API server from um getting like you know destabilized. Um the fourth request is API server uh so fourth metric is API server flow control current executing request. This metric kind of gives the current max current uh requests that are being executed by the API server uh under flow control and then uh API server flow control current executing seats. This kind of gives

like how many um concurrency pools or concurrency seats are being utilized at a given point of time. And uh the last one is API server flow control request weight duration seconds. This kind of gives you how much time a request is spending in the cues while APF is trying to accommodate the request to go through. So kind of gives an overall like how much additional overhead you

are experiencing through the process of APF. There are a few more metrics but I think these are the basic metrics which kind of give you the overall performance of the APF and the health of uh Kubernetes Um also it comes with uh some debug uh endpoints where you can um dump the priority levels using your cubectl. At the same time you can also dump the overall cues.

There are few other command uh endpoints such as like dump the request and so on. Uh so what this kind of gives you is at a given point of time how many cues then the first one is like how many um cues are being utilized in a given priority level. It also gives number of dispatch requests so far while the API server is running till now. And

the second one is basically giving you um information about all the cues. Uh in this case as you can see the system uh with index 0 1 2 3 and so on. Those are basically individual cues of um system um cues and it kind of telling how many requests are currently in pending, how many are executing and seats in use and so on. Um next dispatch R

is basically uh a performance metric which kind of tells you like number of seats per second being processed and so on. Also u one thing to note is API uh kind of logs on the client side. If you're using native Kubernetes client for example when you run your cubectl command with uh verbos equal to like let's say level 8 you kind of see this messages in the

response it sets some uh response headers um as you can see here there is something called x- kubernetes pf- flow schema- uid that basically is telling you that for this request the request has gone gone through this flow schema and this priority level. Um the UID is basically the actual UID of the resource flow schema and priority level configuration and sometimes you might also end up seeing

uh a log message something like this on when you turn on a verbose where um the request has waited for certain time due to client side throttling not not because of priority and fairness. So this message is kind of giving you clarity that the request has um failed due to client side throttling and APF is not involved in this. Um if you are aware of it, Kubernetes

native clients actually provide you something called max QPS which kind of controls this client side throttling where um essentially like the client from utilizing or making too many calls. But in Kubernetes world obviously you can't really control how a client makes a call because anyone can write your own client and then still make a lot of requests to your control plane. Um and then uh you do

get some logs on the server side in the cube API server. Um when you turn on the verbos level three or more uh for every HTTP log uh you kind of basically get the APF uh priority level. In this case, PL is priority level flow schema name, number of seats at the time of u request being processed and then number of seats u after the request has

been finalized. Um usually that particular number doesn't go up. it only goes up for list requests and then it kind of captures additional latency um because of APF if the request has been sitting in the queue for a longer time and then the overall execution time by APF itself and the response time is the regular response time of Kubernetes um API server. Um additionally one thing to

note is um uh lot of uh users do make watch request and usually watch is comprised of uh initial list and then followed by stream of uh events. So in such cases the list call is handled as a list call where Kubernetes API APF kind of estimates number of seats to be utilized and then followed by the each of the events are also estimated based on how

many uh create update or delete events are being streamed to the client and based on that APF estimates are being um made. In terms of exe commands like uh exec and cubectl logs those are u do not fall into the APF because in a way those are being served by cublet directly when you connect to the node. um some of the best practices I have learned through

um by running large clusters and how APF kind of help helped our teams is um basically first of all you separate your uh mutating and non-mutating requests because Kubernetes APF by default doesn't do that isolation. So having a separate flow schema for mutating and nonmutating will be helpful. uh you basically create a flow schema resource with uh the verbs create, update, put, patch, delete. So that will

be your mutating request and make sure you kind of assign the the precedence number low so that that gets picked up for the requests which are actually of mutating type. And then um second one is throttle or limit expensive list calls. So list calls has been um a pain when you have a really large clusters um because you you can't really control someone from making uh a

big exhaustive list of uh resources and the only way you can protect the API server in this case is by uh limiting number of requests at a given point of time because essentially when you do a list um API server actually makes a request to HCD get the resources um massage the data in the memory and then serve the data back to the client. So that involves

a lot of memory and CPU um utilization by the cube API server. So when you have like let's say hundreds of clients trying to make the list call at the same time that would definitely bring down the API server. uh however following I believe 1.34 Kubernetes introduced something called streaming list which kind of which kind of helps with the whole memory utilization in the API server because

instead of returning the the data at once it basically um streams the the data back to the client so you're not essentially spiking up the memory um ensuring non-critical clients have lower priority so you if you have like regular users trying to interact with the cluster that's considered as non-critical. You can always make sure that those requests are given lower priority so that regular cluster reconciliations and

other uh interactions happen u without any um uh impact from any noisy client or user. Um assign high priority levels to essential uh system components. In this case, all the requests coming from cublet um all the requests for node leases leader elections all these ones are kind of priority and we want to make sure they have their own concurrency or set of requests that can be allowed

all the time. So that would make sure the cluster is in a a stable state all the time. And the next one is um set appropriate flow schema for critical requests for example Kubernetes administrators later elections leases we just covered that um ensure requests from non-critical demon sets are within the limits. So uh a lot of us run uh several demon sets uh and they are good

in terms of like what they kind of facilitate for the cluster. But let's say you kind of have large clusters, you go above 1 th00and or 3,000 nodes. That's when having demon sets making a call to the control plane API server is a problem because imagine you are like restarting all the demon sets. U they make uh calls at the same like multiple nodes make the call

to the at the same time to the API server that causes a lot of spike in the request. So if there are like non-critical demon sets which you think uh are okay to retry the requests you can essentially apply APF for that and u the way you do it is basically through the demon set service account user as the one of the distinguisher. So the request you

know that request is coming from a particular demon set and you want to uh throttle and then um there is a case where uh you might end up in recursive request scenario um which kind of doesn't work well with APF. Uh let's say you have a Kubernetes web hook. You make a call to the uh API server and the API server makes a call to the web

hook and the web hook again makes a call to API server. So all of these requests are actually going through the um APF and uh not having the right level of prioritization between the subsequent call and the real call uh is going to create a scenario when uh the cluster is busy and might lead to deadlock in some cases. Um I think that's all I have uh

for Kubernetes APF. Um so uh we are hiring if anybody is interested to join Cruso please feel free to reach out to me and uh if you have any questions I can take that now. >> Yep. Yes. >> Yeah. I guess I was wondering um I don't think my platform has done much modifying of the defaults of this. What are some what are some uh indications that

you might want to start modifying uh the APF away from the defaults like particular metrics or particular behaviors like when would you consider it worth it to to start stepping away from the defaults? >> Yeah. So the question is basically like you know Kubernetes comes with default uh APF configurations in what cases we need to start modifying them. Um so uh if the cluster size is small

let's say you have a hundreds of nodes of cluster size and then you have uh not too many in such cases the default configuration will still work so the way we had to implement APF is by because we kind of run clusters with thousands of nodes that's where we started to see the problems there were cases where um we have some essential ial controllers they were unable

to write back or patch resources back to the API server. So the first thing we did is to actually separate out mutating and nonmutating requests so that uh you always have a dedicated concurrency shares as allocated for write request versus read request. So um that's one of the first thing you can attempt to do and the second one is addressing the list calls. Um if you have

a lot of clients who make uh or interact with cluster with too many calls like you know when you have too many list calls being made you kind of spiking up the memory of API server. So if you kind of have that problem then making sure a separate uh flow schema and priority level for list is also And uh the third one is demon sets. So to

give an example, let's say you have an uh metrics agent uh which kind of let's say measures like uh persistent volumes or whatn not and kind of provide some new like further metrics for you and the metric agent makes a call to the API server to get a list of persistent volumes and pers persistent volume claims. Um these list calls are kind of expensive and imagine if

the demon set is crashed for some reason and restarted at the same time across multiple nodes. you're kind of bombarding a lot of requests from the demon set to the API server. And if you don't want that to happen, you can limit the calls for that demon set by at any point of time, I would like to only allow 10 requests. So I can create a separate

flow schema for that by um scoping that particular flow schema to a request uh uh service record user called metrics agent for example. Um um that's the most thing you probably need. you don't necessarily need like further classification into priority levels but um those are the things at least I had to deal with. >> I I [clears throat] have one. What's same thing a client or a

CRD that was created before APF? Right? So it doesn't it's not looking at any of the APF messages. What's going to be the experience of that client if the queue is full? >> Um can you repeat the first part? Sorry I didn't get it. >> So for a client tool, right? Uh so a client of the API server, right? um or a CRD, something else that interacts

with the API server. If it was written before APF, so it's not paying any attention to APF messages and stuff, right? But the API server is hammered and the queue is full. What is that going to look like to that old client? >> Um so to if I understand the question correctly, so you have um a client which is interacting with uh the cluster at that time

APF is disabled. Uh >> no I mean the client was written the code for the client was written before APF existed. >> Oh I see. >> So um in that case basically like depends upon what custom resource that client is interacting with. If the custom resource is part of the one of the flow schemas then the request will go through the flow schema. But however if the

particular custom resource is not part of any of the flow schema there is a a catch all which is where the request goes through and I believe catch all doesn't have any cues. So if the concurrency of that particular flow schema is fully utilized the request gets rejected. Okay. Well, thank you very much for that and and thanks everyone. Uh we have two more sessions in this

day um coming up Let's see what we got. We have deterministically built containers. Um, which is involves a bunch of Nyx's container stuff. Uh, and that starts 3:45. Yeah, that starts at 3:45. So, see you then. >> Thank you everyone. Apologies to my call talk. Bye. >> It's a check check friends countrymen. Is that a y in display? check. All right. >> All right. Good afternoon everybody

and um welcome to or thank you for attending cloud native day at scale. Um, it's been a great day of great talks so far and I am very pleased to introduce Morgan who's going to get us all excited about containers and ny unless you're already excited in which case enjoy >> which case we can be excited together. >> Please give him a hand. [applause] >> Thanks. So,

uh, the title of my talk is very long, but it's, uh, a reference to a 1964 movie, uh, Doctor Strange Love, uh, which, uh, I watched on the flight here to, cuz I had seen it like 10 years to remind myself. And I'm going to I'm going to like do a bit in the middle of this that ties to a specific scene from that movie that I

regularly drop in Slack, uh, like a GIF of generally when anybody at my old job, I get fired. But you'll see in a second why that is. Um, so what this talk is about is about I'm going to I'm going to kind of introduce Nyx a little bit. Um, and then I'm going to talk about running containers based on Nyx packages but skipping the generation of an

OCI image. So directly running Nyx packages in in containers specifically on Kubernetes. So me uh Morgan Held I'm a software engineer at a company called Flux. We actually put on Planet Nix next door. Um, Flux makes one of the three methods I'm going to talk about here and I worked on it. So that's that's pretty cool. But I'm talking about three different ways in the Nixie ecosystem

that do this. Actually use the second one in my home lab. But um, a little about me. So I I uh I've been doing tech stuff like I'm I'm an infrastructure engineer at Flux. Um, but before that for 15 years I worked as a like a automation engineer at an oil refinery. Um, so like very different when I say automation like control valves and sensors and safety

systems and for some reason my microphone just No, I'm still here. And uh gaskets. Uh there was a while when I got really good at inspecting gaskets. But I think one of the big things that I I took with me from manufacturing and kind of brought into tech stuff is uh is like like I'll say reproducibility, but I didn't use that word then. But like if I've

got a valve in a line and like it uh it works and it's been fine for 20 years but it fails because whatever it's old and we we want to buy exactly the same valve. So I go outside and I scrape some rust off the name plate and I call the vendor and I'm like hey like okay the CV is 42. Uh it's a 3-in valve. Uh

here's the make. I need the exact valve. And he calls me and he's like well I can give you almost that valve. He's like we haven't actually made that valve since 1987 but I can give you this year's model if you'd like. And I'm like, "No, I need I don't I don't I need it to be the exact same thing. I need you to give me I

need you to go find me one in the boneyard and give it to me." Um because it in manufacturing it was very much about like I know what the unit does and I like it and so I want it to keep doing the thing that I like. Um and that that's that's similar in software like I want to I want to build something today and I want

to build it and if I build it tomorrow, I want it to act the same way. Um, so like those are like lessons I kind of took with me and then when I discovered Nicks about four years ago, I was like, "Oh yeah, this is that." Um, okay. The bit. So this is a scene from the movie, uh, Major King Kong played by Slim Pickins riding a

bomb into the ground. Um, and so the metaphor I'm going to use or try to carry as the through line in this talk is the rocket and the guy. So the guy is an application. In this case, it's going to be a Go application. And the rocket is the delivery mechanism. Um, and so we can say it could be an OCI image. It could be a container.

But I'm going to posit that maybe I can get him to the ground uh on something that does the same work, but is like a jetpack. It's lighter. And I did not have to build it in a factory somewhere else and then ship it to the launchpad or put it on the plane. I could just build it. I I could just put it on his back and

push him out the deal. Um, so that's the metaphor. We'll see if my through line works or not, but that's what I'm going to try. Okay, so the guy on top, the application, it's not really important, but it's a very simple Go application. The only thing about this that's somewhat uh notable is the fact that I have one external dependency like I'm importing one package that'll come

up later. Otherwise, this is just what I'm this small little little web server is what I'm going to carry with me throughout the talk. So before we talk about Nick stuff, I'm just going to spend like a couple minutes talking about OCI images and I'm gonna I'm gonna I'm going to zoom in on the part that is relevant for this discussion. Uh so how do you build

a how do you build a container image? Typically you reach for a docker file or a container file if you're in today's parliament I guess. Uh and this is a very naive example of a docker file like like I'm sure like everybody in this room is like oh you didn't you didn't do multi-stage like why didn't you like this is extremely naive on purpose. Uh, and this

talk is also not about the this talk is not about OCI images are bad. This is to highlight that there are times where like maybe there's another option, but I picked a naive example anyway because it lets me kind of highlight like exacerbate some of the flaws. So you start with a base image that's like an operating system. You start with an Ubuntu base intrinsically that has

a package manager in it, but I'm running a Go binary. I don't need a package manager necessarily to be shipped. And then I run then I use app to install the Go uh the Go tool chain. Um, and that's fine. But now I've suddenly ballooned the size of my image up significantly because I have the tool chain in it. And then I copy the source code into

the image. Now the source code's in it. And I build my image. Uh and then I set the I set it when you when you run this image, it's going to run my application. So again, a very naive example, but at the same time, like I have seen images like this in production before because it works because you're trying to do something fast and it works. Um

but there are some pitfalls. So every statement there produces a layer. Um and the layers are ordered. So like if I change the fourth one, I got to rebuild the first three. Um and the layers are additive. So like if I don't engineer it right, if I don't like think about stuff like this, if I don't think about what I'm leaving behind, then like I end up

blowing up I end up with I I end up with stuff I don't expect to be in the image. Uh there was one time where I I had a I found like it was like a 14 GB Docker image. It was like they were using TensorFlow and what I found is they they lost the Docker file and so they they like they did from the original image

and then they uninstalled one version of TensorFlow in one layer which didn't do it right because it's still there. It's just masked and then they installed a new version. So we're shipping around two versions. I I about I about lost it. I was like all right. But that was really the moment where I really internalized this. I'm like oh that's what that's why you always do app

update and and app install and and delete the delete the app cache. Got it. Um, so yeah. So, so if you do it like that, what comes along for the ride again? It's you end up with a gigabyte image and you you have a you have an issue with reproducibility like you know you could potentially make a Docker image reproducible or you could get darn close to

it if it were minimal enough. That example is not I I have been bitten personally by the apt pattern, right, of like I'm installing something and then I come back to rebuild the image 6 months later and the upstream has been deleted. Uh, usually my fault, right? Like last time this happened to me was Hash Corp deleted their AL2 repository because AL2 is going to be EOL

but like I don't know man I was still building the building AL2 like I don't know. Um so so you you can build it today and it's definitely going to work but 5 minutes later it may not work. Um and then Docker files can be difficult to compose like obviously you can use copy or you can uh you can import like you can use from right but

in general it it can be difficult to to say like I want a base and then I'm going to build on top of it. that you either end up with copy paste or importing images. And if you're importing like if you're using from then you end up rebuilding like a chain you end up rebuilding a ton of things um to be able to to be able to

compose like your final product which again for small applications like this is totally fine. It's only when you when you scale like when you when you end up with a ton of like a bunch of language ecosystem dependencies or a really big project or uh or extra stuff like like models or libraries that you're shipping that you that this really start you start to feel it like

you don't necessarily feel this for a simple application but I picked my contrived example and again I kind of highlighted multi-stage can solve some of these problems but not all of them. it can't really solve the reproducibility problem like you still have the same you still have the same gap there but you can get a smaller image like I picked a naive example um and so then

once the image is created a container can be created from it uh so so just the the entire life cycle so you build you build it you run you run docker build or or podman build or whatever and you end up with those layers and then you push the layers to a container registry um and then when you're ready to start then the layers get pulled and

the snapshot assembles them into the the root fs the file system of the container and it starts up. So I basically so so what I did going back to my we'll see if it works the metaphor from the movie like I I basically like built the rocket in Alaska uh and then I you took it took it apart into 10 pieces and then I like shipped it

to Florida and then I built it again. Uh because I basically built the file system once and then I built it a second time when I was going to run Okay, so that's like enough context about stuff that most people probably already knew about OCI images to to to kind of set up the the stage. Um, now I'm going to talk about Nyx a little bit. Uh,

I guess who in here has ever heard of Nyx before? Okay. Uh, and and like used it. Okay, cool. Um, so what is Nyx? Nyx is an overloaded term. So I'm going to try to disambiguate the word Nyx a little bit here. So Nyx in general is a crossplatform package manager, but it's a functional package manager. What that means is every package is a function of its

inputs. So I I put something in and then I get something out like like any function. Um I'll talk a little bit about sort of the implications of that. But you can think of it like that that that being a function means it kind of gives it intrinsically some reproducibility. Like it is fully defined by what I put in. So as long as I have strong guardrails

about what I put in like I I can build if I can build it today I can build it tomorrow. Um, Nyx packages are defined in the Nyx language. Uh, so the Nyx language is what you is what you write the the uh definition of a package in. Um, and it's like akin to a separate functional programming language like it the Nyx DSL. It looks kind of

like Haskell I guess is because it was made by Haskell people. I don't know. Um, and uh, and yeah, it uh, it kind of messed with my I never written a functional programming language before I started working with Nick. I'm like, where's the for loop? There isn't one. Uh, Nyx packages is a monor repo on GitHub with with Nyx language definitions of 120,000 or so different packages

like anything like uh like you know language tool chains like Linux desktop environments like Kubernetes itself um every you know Nyx packages is what makes Nyx so powerful because all the foundation is already there for you. Um, and it is kind of scary like it's a it's if you look at the number of open PRs, you're like, how can this be sustainable? Um, but there's a lot

of automation and there's a big contributor base. Like I maintain like 30 packages. I maintain AWS CLI too. I don't know. It feels like my little my little my little rock that I'm pushing up like Sisphus like whatever. Like I I can keep one package that I use up to date. And then Nix OS which is not a thing we're going to talk about today other than

the fact that it's on my laptop I I hope it doesn't mess up because that would be awkward. Uh Nyx OS is an operating system composed from Nyx packages. So you say, well, I packaged systemd and I packaged the kernel and I packaged KDE. Why can't I just install them all and then make an operating system out of these packages that are built in a functional way?

Um and turns out that's actually pretty cool. And then I can extend it and say, what if configuration were also a package? What if every comp file in Etsy were its own package? And that's kind of what Nixxos ends up being. Um, and it ends up being this really neat reproducible. If I have a problem today, I just roll it back kind of thing where like it

is very predictable. But again, Nyx OS is an operating system and it's not the subject of this talk. We're talking what what I'm talking about is packaging software with Nyx and then using the moving those packages around. Um, actually I will probably say Nixos one more time in this talk, but that's another here. So what does the Nyx language and Nyx packaging look like? So this is

a Nyx package expression for that sample application we talked about earlier. Okay. So I I'll just walk through it. So the first line so it's a remember I said it's every package is a function. Okay. So this function takes one argument. The argument is build go module. What is build go module? So in the Nyx ecosystem there's a lot of like build functions for different language ecosystem.

So build go module there's a build cargo I can't remember what it's called build cargo something. um you know yeah build python package like there's all kinds of builders for different ecosystems or different language ecosystems in n in in nyx and so in this case I'm using build go module which knows how to run go build it knows how to run check tests it knows how to

skip tests it knows how to pass lb flags like it knows how to do it knows how to build a go application so I call and build a module itself is a function so my package function takes another function and I call that function with what's needed to build it first things I pass is just the name of the package echoserver whatever I pass a version string

that's neat uh it takes a source argument and so what this source argument means is build the source code is in the same folder as this file um but the source can be uh can be fetched from a git repo fetched from GitHub fetch from gitlab you can fetch it a zip file like like there there are a bunch of fetchers to just get source code for

you um it's uh it's pretty flexible on where you get stuff from. But for the purpose of this, we're just saying we're building with the source code in the same folder. And then the last argument to build go module in this example is vendor hash. And so what vendor hash is is you think I want this to be reproducible, right? Um but part of building that go

package is pulling pulling that one outside dependency that I had, that one thing that I added. Um but I want it to be reproducible. Like I need to I I need I need to be able to be sure that what I'm building today is the same as what I built tomorrow. But nyx builds happen in a sandbox like when go build is run there is no network

access um which is part of the reproducibility guarantee. So what build go module knows how to do is it knows how to use go mod and go sum to go pull those ahead of time and then put them somewhere on the file system where the build can get them. And what it does is it calculates a hash of those of what it pulled based on based on

go mod and go sum and then it and then that goes in the next expression. And so even so if I were to change the source code and add a new external package like it would fail because it would be like it got some other stuff. I don't know what it is. That's probably not what you want to do. Um, and so that that's part of how

reproducibility can be guaranteed is that, you know, you don't necessarily want to handle go dependencies with Nix. You want to use the native language ecosystem fetcher. Um, but you still want to have a way to ensure nix based reproducibility. And so handles like this are kind of how you can have best of both worlds. So that's that's a Nyx expression. So what do you get? What do

you get when you build something with Nyx? So you get that you get that path up there. What does that mean? So every package built with Nyx goes to the same place on the file system. No matter what system you're on, it goes to the Nyx store directory. And what that mean and and there's a there's a few implications to this. One of those is that like

things within Nyx are statically linked to their dependencies. So that means that the dep that all of its dependencies have to be at a predictable location on the file system. And if I build it or you build it, they both got to be at the same place in both of our computers. And so the way Nick solves that is it puts everything in the same place. uh

it puts it in the nick store directory and then at the beginning of that there's a there's a hash and that hash is a function of all of the inputs to the package expression. So the source code and like the actual what I wrote. So for example if I were to change like add an LD flag like that hash would change because even though the source is

the same I change something about it. But think about the consequence of that. That means that I can have like for example let's say I compile with a debug flag. Well, now I have both my original package and the one with the debug flag both on my file system in the next store directory at the same time and there's no collision. So I could do that with

gibbc. Like I could I could rebuild gibbc with some other flag and I link my package to that but then the rest of my system is still using the the gibbc without that flag and they can both coexist because they're all going in into the this this place that is input addressed which is super powerful. Like I I'm not gonna I'm not gonna go down the rabbit

hole of like the override pattern, but there's like there's so there is so much you can do with this to to be able to just like test stuff or does this patch fix my problem? Um I mean heck for me like like I I'm having like a weird AMD graphics driver issue and I was like well I wonder if I can revert this one commit from uh

revert revert commit commit from the kernel. I mean like think about how much of a pain it would be if you wanted to patch your kernel to revert one commit like all the all the hoops you would have to jump through. It was three lines of Nyx and that was it and and a kernel rebuild, but whatever. I can get some coffee. Um, so that's so that's

the output of a Nyx package. But what about its dependencies? So then that's extended to everything that needs to run it. So so as part of writing writing the Nyx package and building it, what what also comes out is what's called the closure. And I'm trying to not use too many NYX words, but the NYX word for this is the closure. But it's basically a dependency graph.

So at the top is the package I built and the other seven blue boxes are all the other packages that need to be on my system to be able to to to be able to to run this. And so and so like if I give you this expression, all these same packages are going to appear on your computer too because it's just part of it's part of

that. It's all implicit in that Nyx expression. when Nyx takes that and then turns it and turns it into basically a build script that runs go build part of it is the input address hash of every input and so what I basically get is I get a full dependency graph of everything needed to run my application like just implicitly those with those five or six lines of

nicks and let's go back to the gibbc example so let's say that I patched gibbc to do some like I I patched the gibbc to turn on some flag for this particular package so the gibbc hash would change which means it would also change the hash of my echoserver package and so you can see that the hash of my package is a function of the hashes of

every other package. So anything I change about any of these will affect everything downstream of it in the tree. There are downsides to this. Uh in the Nyx ecosystem for example when the XZ thing came out XZ was very far up the dependency graph of a ton of packages and so people were grumpy because it it took a few days for the rebuilds to happen. like like

Nyx provides a binary cache where they have built all these from source for you but like it took a while to rebuild the world because XD was very far up the tree. So there are upsides and downsides to this but at the small scale of building your software you can think you control everything and it's all very easy to get access to very easy to know the

provenence of all the inputs. So it's very powerful. Um I covered some of this already but I'm going to circle back and try to kind of summarize. So what this means you you get a sandbox build with explicit inputs meaning like the build does not have network access. There's one step at the beginning to fetch everything needed and once that happens and it's there like it will

fail if you do not have if you need something if it you know tries to do network stuff it's not going to work. Um but that kind of gives you confidence in the output, right? Like the thing the thing that it is doing is the thing it will always do because that's all it had from this that means it's reproducible. So you as long as you have

access to the sources you could come back in one year, 5 years or 10 years and you could build the same binary again and you could have confidence that you got the same binary again. Um, and uh, and there's actually like a little project in the X ecosystem where somebody went and tried to re build like Gnome and and Firefox from like 2009 just to see if

they could on a modern system and there's there was about a blog post amount of work about it. So like it took a little bit of effort but not like it was like a blog post. It was not it wasn't a big deal because that's because it's so powerful. Long as you have the sources, you're good. And you get a complete esbomb. You get you get a

complete bill of materials by doing this. you know first everything that went into it and you know everything that's needed to carry that carries along to run it and you know some of you have probably used like a scanner on a on a docker image and you you basically try to reverse engineer what's in the image you say okay what did I actually ship um when you're

using ny to build stuff like it's only the only only what you put in it there's a star wars bit in here somewhere about about the cave and only o only only with only what you take with you or whatever but like but basically it can only be what it is I mean It's portable. All you need is the the expression and the source code and then

you can materialize And you and then there's even more in the Nyx ecosystem. You can you the Nyx ecosystem offers the ability to take that those that close to those packages and put them into familiar familiar uh transport mechanisms. So I can build an OCI image from Nyx packages. So like I I I pick the echoserver thing and in one more line I use a native Docker

image builder to produce an OCI image out that just has those eight packages in it and that's it. Uh it's it's it's one line. It's a built-in and it doesn't require the Docker Damon to do it either. Like it just it just happens. I can even go further. This is my other mention of Nyx OS. I can even go further and I can I can build a

VM VM image in about 10 lines 10 more lines of Nix. I can I can render a Qcal 2 or a VHD VM image that runs this as a systemd unit. Get 10 more lines of Nix. I don't need Packer. I don't need I don't need any of that. I can just I I can just print it. The build takes about 90 seconds and and I'm done.

If I change my package, the build takes another 90 seconds and that's it. I can even build an AMI like which there's another VM image, but I can even build an AMI. Three more lines of Nyx. I print an AMI. Done deal. I can ship it directly to AWS. I run my deal on the system to unit. And it and it's because once I have the Nyx

package expression and I know everything to to to run it, I have I have all all the all this uh these tools in the ecosystem to be able to turn it into whatever I want. And we could stop here. I don't know how far I am, probably like 20 minutes, but like but like we could stop here. You know, we talked about Nyx. We talked about the

fact that that it gives you the reproducibility. It gives you the bill of materials. It's pretty lightweight. I can print anything with it. And we could say that's pretty cool. But what if we didn't have to stop here because I think I have more time. so if I know everything I need to run it and I know we we talked about Kubernetes at the beginning, right? The

and I and if I go back to talking about uh containers and the root fs and OCI images, the container file system is just files. Well, what is the collection of nick store package paths and packages needed to run the echoserver package? it's also a collection of files. So what if I could just enable uh something around Kubernetes to just pull those pull the packages I need

to run my application and just run it. So I don't wrap it in anything. I just I I enable it to run Nyx packages semi-natively. Basically the uh la last time I'll probably do the bit thing but basically I put a jetack on the guy. I send him I send him right right there. I don't build anything ahead of time. So that's what we're going to talk

about. We're going to talk about three ways in the Nyx ecosystem to to to directly run things you package with Nyx on Kubernetes without building an OCI image. And they have three different entry points into the ecosystem. So it's kind of a choose your adventure. You know, which one solves the problem that's more attractive to you and fits fits in better with what you do. The first

one is by Flux, which again is my employer, but this is not intended to be a vendor pitch. Like I worked on this this I worked on this, which is my introduction to the other two. Um, and they're all super cool and now I run the middle one in my home lab. So um but what the flock one does is it it we provide a containerd runtime

shim. So we basically sit in the middle of container creation and inject nyx store pads into the container spec as it's being created. We'll go into more detail. Nick CSI says what if I could just create an ephemeral volume with with the nick store attached and mount that to the container. Uh totally valid another another way to get them in there. And a third way is at

the snapshot level of what if as I assemble the root fs I could just al assemble it out of nickto paths. So again three different three different ways in the ecosystem to approach this but all all to the same destination. Uh being able to run stuff on kubernetes without building an OCI image ahead of time. So we're going to talk about the flux one first. I keep

I keep like bowing and scraping and saying I swear it's not a vendor pitch. The flux one I think is the most logical like mental model from a Docker file to doing this. Um the other ones are a little more Nyx native. So I'm trying to kind of ease you into the NYX the Nyx hot tub uh as we go down this rabbit rabbit hole into infinity.

Uh all right. What is Flock? Um aside from the company that pays me. I'll stop with that now. Uh so Flux is at its core a tool uh for creating reproducible development environments. So most people would use flocks to to create in in a similar they might use dev containers. Um it it it allows you to create a virtual environment composed of packages but packages from any

ecosystem. So you can have python packages and you can have um you can have like postgress and you can have terraform and you can you can create an environment containing all of them and it's portable because it's composed of nyx packages. So every single one of them is like a input address goes in the next store. You know I I can have 10 versions of them on

my computer at the same time. Um and it's a reproducible environment in the sense that it comes with a lock file that defines exactly what I have what's in it. So I build the environment on my Linux box and I send it to you and you you pull the same the definition on your Mac and you and you do flock activate, you're going to end up with

the exact same version of packages as me and you're going to end up with native binaries. There's there's no containerization here uh unless you want it. It's it's it's pulling, you know, binaries for your system, compiled for your system on onto it and running them. It runs on Mac OS and Linux, x86 and ARM. Um it's it's crossplatform and and the thing with Flux is that it

we've we've made it such that you don't have to engage with the next language if you don't want to. Um it it's got familiar package manager ergonomics. So we we can take a look real quick at again the flock development environment workflow. So um I want to use Go. I don't have Go. I I do flocks and netit. And what flocks and does is it creates blank

metadata for an environment that I can then populate with my packages. And then I do flocks install. So again, familiar ergonomics like like you do in a or brew or whatever. I do flock install and I install some Go ecosystem stuff. And what that does is it is it starts to populate this manifest file on the right. And this is what I would commit along with the

lock file. That's the environment definition. And so this plus the lock file both allow you to install the same things as me and make it reproducible. Um, and now I do flock activate which puts me into a subshell. Um, so it's basically sets environment variables to tell me where to find stuff. And now if I do witchgo, I have go on path but it's from the nick

store. And what that also means is I could have 10 repos with 10 different flocks environments with 10 different versions of Go all on my computer at the same time because they're they're they're in their own place in the file system. They're they're isolated. They're not they're not going to they're not going to collide. Um and then I have go and that's it. So that's the dev

environment What the flock kubernetes integration does is it is it is it extends it and says well what if I had a flux environment that was just my application. So I didn't install the I didn't it wasn't for tooling. It was just the echoserver package I talked about. Then I have something that's declarable or declarative and composable and reproducible. Um because again it's it's just metadata that

tells me what NickTor pads are needed to run it. And then what we do with that runtime shim is we we pass it we pass it to the we pass it to the containerd runtime shim. It reads the metadata and then it pulls those nicks store paths, all eight of them onto the onto the node and bind mounts them into the container and then what I end

up with is the container file system now has those eight packages in it but like only that. So it's super lightweight. I didn't build an OCI image ahead of time. Um, and if I change one thing, it's only going to pull the pads that I changed. It's it's not going to repool everything, right? as long as I as long as I have a cache of the previous

time, it's just going to repool the differences. So just kind of a high level overview of how it works. So containerd says, hey, I want to make a container and and so we present as a like using the runtime argument. So so it it calls our shim binary and our our shim binary uses an annotation we'll see in a second to pull the environment metadata. So that

manifest file and then pull the nick store paths onto the node and then adds mounting them into the container spec and then it passes that that that updated spec to run and run starts the container up but when it starts it's got your package in it now. So it it it I I got it there based on the fact that I I was able to know what

what Nyx packages were needed to run my application. And so what it looks like practically when you when rubber hits the proverbial cluster uh on the left side again there's that manifest file um with the one thing installed in it and on the right side is a podspec to you to consume it and the two key here keys here really are runtime class name. So you create

a runtime class resource in your cluster and a runtime class pairs with uh a runtime that's available to containerd on the node and so you set runtime class name flocks and that's how the cubleta tells containerd like use the flock runtime shim binary and then you pass the name of the environment as an annotation. So that annotation corresponds to the manifest on the left. And so this

is how I tell it do this one, do this thing. Um, and I a container spec has to have an image of some kind. Um, we provide like a like a 48 byt nothing image like it contains nothing. Um, but in general you could do anything because ultimately this is just adding mounts to the spec, but you don't need anything like this is enough for the container

to start and run your binary. like it it will you you you the command is echoserver like it's your thing and it just runs. Um and that's it. That's that's pretty much all there is to the flock solution. Um now we'll go and talk about nick CSI. So what is CSI? I won't do any jokes about TV shows because that's played. But uh CSI stands for container

storage interface. Um it's basically like the way you get persistent volumes or volumes into a Kubernetes pod. So if you you've done EBS on EKS or you've done GCP persistent discs or Longhorn like like the mechanism that it uses to attach those is CSI and so you basically just reference CSI what CSI driver you want a podspec and Kubernetes handles the rest of calling the CSI driver

getting the volume provisioned and then attaching it. So Nyx CSI is a is a community project that creates ephemeral volumes containing Nyx packages. So you you you write a Nix expression or you give it a flake reference and we didn't talk about flakes. So basically you tell it you tell you use one of three ways to tell it what packages you want and then it pulls those

it so nick CSI runs as a Damon set. So it pulls those packages onto the node and then and then it composes an ephemeral volume out of those nick store paths out of those packages and then it um and mounts that as nyix inside the container. So what you get out of that is it's just a volume like it's a standard Kubernetes primitive. Um so the consequence

of that is it runs on anything. Um the downside of the flux implementation is that you have to be able to install it on the node and like for EKS for example that's that's not a big deal like um you know the 10 lines of Terraform you create you create a node group you use user data to install it it's no big deal. Um, but some things

like GCP can be a little bit harder. And so it really just depends on what you have available to you, like what your what handles you have. But Nick CSI will run on any cluster because it it it just it's just a few manifests you apply. It's a Damon set that runs. Um, and it it plays at at a at a standard part of the ecosystem. NXT

CSI also includes a cluster level cache. So think like like when I run a like EKS cluster, right? And I and I I basically want had to it to have the minimum amount of compute provisioned possible to accomplish my objective. I do not want to pay AWS for compute I am not using. So I'm running spot instances and I'm using carpenter and I have it tuned pretty

aggressively. Um but the downside of that is I I lose my like warm caching. I lo I lose I lose the ability to c if I'm using those images. I lose the ability to cache layers. If I'm using Nixto pads and I'm storing them on the node, I I lose the abil, you know, I lose the the quick runtime of pod 2, three, four, um because I

am I'm turning nodes over real fast. Nixsi includes a cluster level cache. Um so basically it's like a staple set with a persistent disc that runs on the cluster and um anytime a a path is pulled onto the node or it's it's also put in that cache and so successive node starts the dam set will look in that cache first which means like at the cluster level

you're you're sharing things you downloaded between nodes. So, so after the first time it's both faster and there's less risk of like upstream disruption, right? If your upstream goes away, it's going to check the cache first. So, it's chill. Uh high level uh explanation of Nick CSI. Um so, uh you know, Cublet says, "Hey, I'm going to create a container, but hang on, I'm also going to

call the CSI driver, and I need a volume." The CSI driver does the nick stuff. it it it either builds or it pulls those packages and then it composes that volume and it um pushes that forward and then it gets attached to the container and and you know the container is notified that it's ready to start and it rocks and rolls. Um example podsp spec for next

CSI. Um the upstream recommends distic but really you could use a scratch image for this too because it again doesn't really need anything to be in the container file system. Um, and the way you define it is with a volume and a volume. Um, use volume attributes to pass something telling it what Nyx packages. Um, in this case I'm using a flake reference, but but basically you

just need some way to tell it what packages to install. Um, and then you just run your application and you're off to the races. Uh, the trade-offs with Nick CSI compared to Flux. So Nick CSI is like pure Nix like so you're you're packaging your stuff with Nyx like like regular. Um, for me that's like cool. I'm a Nyx enthusiast. Uh, I think I think sometimes the

the the trade-off with Nyx is it's so powerful, but there's there's a there's mental burden to engage with the ecosystem. Like you're basically learning a completely different way to think and you're learning effectively another DSL. Um, it it it's and then that's usually fine for like the next guy at your company. Um, but like my problem in like business land where I want where we want to

do the business and make the money is that like it's a lot of risk to put all of your ecosystem on one guy or one role. If that guy ever like gets fed up or takes vacation and you have a problem, you're kind of in trouble. Um, so like so like there's a lot of companies that are very NY forward and there's tons of people that know

Nyx, but it really just again depends on your application and your like what what what your tolerance but uh but nixi is pretty cool though. Uh the packages themselves live in / nix instead of user bin because all I can do is mount that volume. So I do have to point path at where to look. Um but that's that's not a huge deal. But it's a but

it's a different mechanism. Like it's a volume mount versus composing the container. So like there's it's it's a little bit less clean. You think the the flock one I literally just added two things to the manifest. Otherwise it looked normal like it looked like a conventional pod and this I'm having to intentionally add a volume. um which like in some applications can be And then the third

one I'm going to talk about a little bit um is Nick Snapshot. Um and caveat I haven't used Nick Snapshot much so like I did a little testing before this so I could talk about it but it's the the third big thing in the Nyx ecosystem around this. Um so Nick Nick snapshot is a containerd snapshot plugin. So what that means is it it replaces the the

the thing that unpacks layers into a file system in containerd and what it does is it fetches nyx packages instead of OCI layers and then it composes the file system out of nyx packages. So yeah every package becomes a bind mount and you don't end up with the same layer limit. Um the cool thing about snapshot is that you can mix OCI layers and ny store paths

in the same container. So you're able to basically say like I want to start with this base image but I also want to put in these five store paths and you pass all that to snapshot and it composes the file system out of like a combination of of both of them. So so if you have some artifacts in one type and some artifacts in another it lets

you like mix and match which depending on your application can be super cool. Um or you can just use Nick store paths just like all the other ones. It's it's up to you. Um example podspec for snapshot. Snapshot runs off of like a stub image. So you ultimately render that that that image is basically just metadata. It's what Nick store pads to do. Um so that has

to be available, but then you pass that to snapshot and it composes the the file system out of the store pads. So that one and then you can run your command. Um some considerations about snapshot. You again you have to install it on the node. So you have to be able to change stuff about the node. Um and you need more nyx knowledge. So it it it's

kind of uh compared to the other two requires more NYX knowledge than flocks. Um it's similar to Nick CSI in that way. So again like it just depends on what you're trying to accomplish. So that's three ways. Um and that's that's Nyx. When would you pick each? Flocks I would pick if you you you excited about this but you don't necessarily want to want to have to

deal with the next language. I mean you control your nodes. I think once you get once you get the flock shim installed it's super low friction. like um I did a I did a deal where I was able to run like GitLab CI jobs um where the GI where I wasn't building CI images. I was just I just used the same flux development environment I was using

to work and so the CI would just run using the exact same packages every time because I was using the exact same definition and it was just pulling those packages, right? So it was super fast. I didn't have to rebuild my CI image if I changed something about my dependencies in the development environment. Um, nick CSI if you want that managed Kubernetes portability. Nix CSI is very

easy to try and it's very flexible. Um, it's it's pretty cool. And then snapshot I think if you want to mix OCI images image layers and NYX packages in the same container like that uh nick snapshot is an interesting choice for that as well. Um, in summary, you can potentially ship lighter. You can wrap the guy in a jetpack instead of sending him on an overengineered rocket

or bomb. Um, Nyx gives you complete knowledge of what your app needs. Like those eight packages in our example, like that is it. If I ship those around, that's all I need to run it. Um, every [clears throat] dependency is input addressed. So, as long as I provide the same inputs, I get the same outputs. Um, it can it's a lot more predictable in that way. Effectively,

that's a complete bill of materials for free. I'm not trying to reverse engineer my SBOM. I just get it based on the build instructions. they are the sbomb. Um, and there's a ton of tools in the Nyx ecosystem to turn that into like the SPDX or like there's a ton of ecosystems to take the Nyx expressions or the NYX derivations and turn them into sbomb conventional sbomb

formats that can then be passed to other tools. So like like this is something in the NYX ecosystem that is like a there's a lot of people that are really excited about the fact that you can actually take that stuff and pipe it directly into conventional So for heavy workloads like if you run into a 13 15 GB image that's a pain to rebuild maybe consider skipping

the image and uh just use the packages directly. Um it's pretty cool. So that that that that's about my talk. Um some links up here to the three the the four project three projects. So the flock kubernetes runtime shim nick csi and nick snapshot. Um, the fourth is a uh something by far Zakaria about just an example of what Nyx is. Um, I cribed from his deal

because I just trying to it's it's a weird thing to explain and I I feel like I feel like sometimes I do it good and sometimes I don't do it so good. So, um, his his example is really good. It is he he kind of walked through a really regimented like way of explaining NYX that I think works really well for a lot of people. So uh

and I think that is my last slide. That is what I had. I think one guy. >> Awesome. Thank you so much. Um we have just a couple minutes for questions. Um I will come to you with the microphone so that the question gets onto the recording. Uh I'm I'm trying to understand the uh you mentioned NYX snapshot you can mix layers and the NYX packages. >>

Mhm. so with the shim that's not possible. >> No it they they could all do that. >> Um if you were to use the runtime shim and you pass an image basically now your file system is whatever was in the image and what the shim wants to do. Um, I will say that's not something we that's not like that's not the happy path for us. So I

know it works because I've done it, but potentially you could end up with file system collisions maybe. Um, and so like like your mileage may vary. Uh um, if if you end if you have an image that has something at the same place the shim wants to put something, it may block it may it may it may squash >> and that's the same case with the nick

>> Yeah. Well, CSI is just going to it's only going to work in /nix. So, so the the thing that the flax shim does that I didn't really talk about is it actually takes all the binaries and puts them in user it sim links them to user bin. So, it actually tries to make the file system look like an OCI image based container would. It put it

puts stuff in normal places in the container file system to try to make it seem more normal like an FH like an FHS structure >> and so and so that you might have the possibility to like exec into the pod like and know and know where bash is. >> Yeah. Yeah. Exactly. It it it makes normal stuff easy like the GitLab CI example like that like it

made that possible. Yeah. >> Hi. Well, I I saw the other version of your talk the other day. >> Oh yeah. Hey, >> but [laughter] I came back anyways because I want I was interested but Nick snapshot part. Um I and that gave me a lot to think about. I was wondering first for the next snapshotter do you have experience running that on for instance EKS or

Talos that's >> I do not have experience running on EKS I I think that it would be easy enough to install using user data like like EKS um even using AL2023 it's really easy to install to to do stuff because that's how the so I did write the documentation for the flux deal and like and like it's like 10 you know the user data is like five

lines okay >> um so I would say like really I say look at my look at my documentation and then look at the snapshot or installation guide and you can probably do that like what I did will probably work for them >> and and for something like Talos >> um Talos I would what's it called extensions their extension pattern >> um like I I an extension would

be able to do it no problem okay I I have like a I have a pocket branch of doing flocks as a Talos extension that I haven't pushed anywhere yet and it was pretty chill like it's the same kind of deal as user data you basically write what you want to do but yeah an extension would be pretty easy to do if somebody hasn't done that already

>> okay >> I run CSI and Alo like at home but >> and if I understand correctly like the three will give me functionally pretty much the same thing. Uh now in terms of between big big quotes like performance and by that I mean like how fast it is to between quotes pull the image and also in terms of disk usage is any of these like uh

I don't know maybe one introduces some duplication or whatn not all the same. I again I'm not as familiar with snapshot so I can't I can't say for that but I think that Nick CSI's cluster level cache is a big is a big help because you you you know once if you run for a week you're going to have most of what you needed until you change

your inputs like you're everything's going to be like localized to your cluster so your network's going to be super fast you're not going to be going to the internet anymore um I I I I think I think that like like if I were to make change to the f like we do more work on the flock thing that's one of the first the other first things I'm

going to work on is the cluster level cache so >> ah very cool One more question. >> Oh, there's one more behind. There's one behind you, too. >> Sorry, I didn't see >> I'll also be hanging around if anybody wants to talk about Nick stuff. >> Um, I just wanted to ask, um, at what point is this is is using something like Flux's K8 integration or Nick

CSI like what benefits am I getting versus just calling, you know, uh, Docker tools.build build image >> um and and pushing that to an OCI registry somewhere like I >> instead of like building these at runtime. >> So one of the things that I I I will speak anecdotally based on my own experience. One of the things I bumped into using build layered image is that it

will do 126 small layers and one big one um for a complicated pack like for something for for for a you know for a complicated image. And so what I end up doing is rebuilding that one a lot and it can take you know I mean I mean admittedly for a small thing it's a minute or two like it's not a huge deal but if you were

in the 13 gigabyte territory you know there there are limits in the Nyx ecosystem around those especially build layered image. Um so I mean that's one thing. The other thing is you're basically taking something and packaging it like just just from a a philosophical standpoint. You're taking something that's packaged and packaging it inside of another package and then you're shipping that other package and taking it out

of the other package. Um, but again, I use Docker. We like like for Flux's SAS, we use Docker tools and we run on ECS. So, totally valid like like it just depends on if you bump into the limits or not. >> All right. Um, thank you so much for your talk. I really I I learned stuff that was great. >> Me too. [laughter] >> Um, and yeah,

thanks everyone. The next talk in here will be in uh will be from 5 to 6. And um I know that you've all been reminded to check the online schedule, but I believe uh let me pull this up real quick here. check the schedule outside. Thanks so Try it. >> Hello. Hello. >> Okay, that sounds good. Welcome everybody to the last talk of the day for cloud

native day Los Angeles. Um the um which where we are going to talk all about failure. okay Miguel Aliel is going to explain how you can fail with Kubernetes. So let's welcome him to the stage. Hello. Hi. So my name is Mikuel and as said thank you. We will learn how to fail with Kubernetes, but also how to mitigate everything. So most of you are probably using

Kubernetes if you're here in the room. And your Kubernetes cluster will fail. Not might fail, it will fail. The question isn't if, it's basically when, how badly it will fail, and whether you'll know why your users uh before your users do. So over the past few years at Commodore, we've been obsessing over that question and we then analyze over 1 million actual real life incidents uh running

on Kubernetes. And what we found was very humbling and really surprising I guess. Um and some of what we learned really confirmed what we thought all along and a lot of it did not. So I'm here to share that with you today. uh basically practical patterns of failures uh actual cases examples and also obviously where AI fits in all of this. So first who am I and

why you should listen to me. So my name is Mikuel. Uh I'm the R&D tech lead at Commodore. Um we build a autonomous AI SR platform for cloudnative Kubernetes infrastructure. I've been doing on call work for over a decade. uh which means I've been paged at 3:00 am like some of you more times than I really care to count and I don't want it to happen again.

So I've seen cluster really fail and catch fires in way that seemed impossible. Uh and I spend way too much time staring at cubectl commands and outputs uh for it to be interesting anymore. I really really like tech and open source. And yes, I can really count to 12 with my eyes closed. Uh you'll understand why So let me tell you we're talking about today. Uh we

have the talk split in about five seconds. Uh and I want to be up front that is not a classic like best practices I'll teach you how to do it talk. This is really more of a story about what do we set out to build at Commodore what we learn and how reality punched us in the face and how we adapted to it. So we'll start with

the reactive chaos. Basically why Kubernetes troubleshooting is broken by default. Um I'll share a bit what we found when we looked at a million incidents. We'll go into the six failures categories that we well six patterns that we've discovered. Uh and then we'll talk about the predictability products. That's a very uh crazy name but uh the moment basically our clever plan all fell apart and we had

to adapt. So finally we'll talk about where AIS kind of fits into all this story uh and what it takes to do it right for your production cluster. So let's start at the beginning uh picture of pain. So if you take a look at this, this is uh troubleshooting flowchart for Kubernetes, right? So I want you to take a look at it and this is the the

basic version the vanilla version of how you would go about troubleshooting any issues on Kubernetes right uh that's what you would put in a runbook for your developers to fix uh or investigate more so it starts with cubectl get pods it branches into a dozen different paths routes you through describe getting logs uh image checks the looking at probes issues resour quotas, ingresses, network policies, you know,

and finally, of course, it goes to consult stack overflow or Google or just escalate. So, I think it sucks and it's also very overwhelming. So, imagine now that you're the engineer. Uh, it's 3:00 a.m. Your phone won't stop buzzing. You haven't fully woken up. You have no idea which branch of this tree you actually need to look at. U and your status page is on fire. your

customers are calling and you know they're complaining. So this was the problem that we wanted to solve, right? The chaos of reactive manual troubleshooting wherever something catches fire and issue of being the human router between a sea of signals and problems and an actual diagnosis, right? So two years ago, about two years ago, we decided to do something about it. And here's what we built. So we

asked ourselves, can AI fix this, right? Can we use machine learnings and LLMs to eliminate and basically come through the chaos that is Kubernetes? The hypothesis was pretty straightforward. Yes, if we could look at enough like real incidents, we could build a deterministic runbook and find the patterns that emerges. a system that given any Kubernetes error basically could tell you exactly what is the next step to

follow to get a solution. Um, and we expected to find patterns obviously. So we did but not the ones we expected to find. And now look at this. I call it cloud of incidents. Uh, this is what real incidents looks like. not neat organized categories just like a swarm of ambiguous signals uh kills crash loops you know helm drifts some security issues u and in production I

feel like it never comes just one at a time when something happens in your cluster whether it be related to infra or to applicative issues uh it usually happens together right there's always something that snowballs into something else um and it really is chaos, right? [snorts] So before I get into what we found, I think that one million incidents really deserve that moment, right? So now that

we've acknowledged that and my very good sense of humor, um so after all that analysis of data, uh we found that pattern patterns emerges and we Hello. Yeah. So we stepped back and looked at all these incidents. We found that the vast majority of them could really be uh grouped into six different categories and it was generally exciting like we felt like we found the solution, right?

Like we've cracked the code and we we discovered the holy grail of Kubernetes troubleshooting. Um so I'll walk you through each one in a show just so we understand kind of the six different categories because we're going to go through a few actual examples in a minute. Uh so first one resource exhaustions you know your regular kills memory leaks uh just CPU throttling uh disk issues GPUs

things like that. Uh for example if your pod's memory limit is too low then you know it gets killed and your application crashes. Uh if your CPU limit is too low your pod will run but then not really at his peak effectiveness. uh and so on. And node resource starvation is basically the most scary one. It's basically when the entire node goes down and then everybody is

escalated. Um second category, image and deployment. This one sounds easy and naive and you know good enough to fix on our own but then you know the image pull back off kind of hides a multitude of other issues inside it. Uh it's deceptively complex. Um, is it like a wrong image tag that just isn't pushed yet? Is it a registry that's rate limiting you, Docker? Is it

an image that just suddenly doesn't exist anymore, right? And you try to pull it thanks to Bitnami. Um, some layer got corrupted in transit, anything can really happen. And so when you have an image pullback of error in your cluster, you need to start peeling these layers of issues and data and signals to understand what actually is Second category, uh config and secret management. So this is

really a a silent killer one, right? A secret might be rotated, uh config might be changed, and do you even know if your like workload actually rolled out that new configuration secret? Has the pod restarted? Did it get that new value? Um, it might, it might not. And then you're kind of in a whole world of drift and chaos where you don't know exactly what application is

running what version of configuration. So this can then manifest in infrastructure error where kind of you know you change something and then a pod cannot restart and it just crashes or the moment it does and your application loads it expect a different configuration and now you don't know because the secret was changed like two hours ago and now it restarted. How do you correlate that even? It's

it's a whole mess. Next one will be cascading failures and this is where things start to get interesting uh and also terrifying right so a single pod fails and because other pods are dependent on it then they start failing too right and how do you figure out that this was the actual cause right something fails might be a log might be not might be memory issues uh

but in reality it's just related to something higher level than uh storage and persistence. Um this is I think everybody that's working with storage in Kubernetes knows that PVC issues are just fun. Um and lastly app versus infrastructure take an example crash loop back off right is your application crashing is the infrastructure rejecting the pod and then that's the reason it's crashing uh do you have like

GPU errors if you're working with like AI workloads is it like hardware issue uh is your probes failing for whatever reason right it can be so many things So it's six categories. We had a million examples and we found like these six patterns that emerges uh in Kubernetes and we were very confident that you know this was it. We we we thought we were done and we

found like what works right. So we were not and very quickly we tried to build deterministic runbooks uh on top of these six categories right. So we run into a problem and the problem was that production environments is much different and much messier right than the names we gave these six categories. a startup running 10 microservices on a single cluster right behaves very differently for example than

a bank running 500 services uh across multitude of clusters uh with custom admission controllers and network policies. It's the same Kubernetes version. They're running basically the same application on different scale. U but it's radically different profile of failures and the same errors mean completely different things. Right? This is the very big one. A crash loop back off in one cluster means that uh the app just cannot

start up and another it means that the p timeout is just too aggressive, right? it needs more time to get a response and in a different cluster it means there's a dependency that just this pod cannot reach right this cascading failures we talked about so same exact error right but a different different diagnosis depending on where it's running and what it's running uh and a runbook treats

that identically so it will be wrong twothirds of the time in that case right and that's not what we tried to solve so at a certain scale uh deterministic approaches fail, right? We started encountering environments with thousands of nodes and tens or hundreds of teams uh managing these these uh clusters and a simple runbook couldn't really keep up, right? The edge cases weren't really edge cases anymore.

They were just the norm. So, it was hard to differentiate between what to do. And lastly, context is the key insights here that we that we found out. If you look at this diagram, um we start from noise to signals to grouping data and at the end we try to give it context, right? Because for example, we found these six categories, these six groups or patterns, but

with no context, these six group might mean different things or or what goes into these groups might mean and understanding this cluster that I'm looking at or this team convention and how they work uh this application behavior right and this organization how it handles incident what needs to be without that layer uh if we use AI to try and find the next steps on what to solve

it will just give you a generic answer to a very specific problem right kind of hallucination. So, context, imagine if you're now sleeping, might be 3:00 a.m. or 3 p.m. I don't really judge. Uh, everything is on fire, pods are crashing, and you know, your on call alarms are going off. Uh, and the only thing that can save you is knowing enough about the environment to understand

what's actually broken and what it means, right? So let's look at some real example. I'm going to show you three real incidents that happened to our customers and in each case it was the same symptom, same kind of errors, just a completely different story and completely different solution or Um let's dive into it. The first one is killed, right? So seems simple, you know, not enough memory,

right? But why? So it's the exact same error right killed two pods in the same name space and they happen around the same times and they required completely different solutions. So if we look at the cache loader um it gets killed immediately on startup right when the when the container starts and it happens so fast you don't even see like a time stamp of when it was

done like it just completely crashes uh and then it enters crash loop back off right because it failed too many times. So what does that tell you? Right, that the memory limit is most likely just too low for the application to even bootstrap and start up its runtime. So it's or at least what the first step to fix is uh to increase the memory limit, The second

uh part that failed was the event processor. It also is killed, but it runs for about 13 seconds in that case. Um, and that shows that there's some kind of memory leak or like growing memory usage uh with that application use. And the fix here or like the real fix might not always be just increasing memory limits. It might help, right? But not always. And the real

thing to do is to look at the application code and see what's taking so much memory, right? when actually being triggered and used. So, same exact error code, same Kubernetes reason, uh two completely opposite fixes. And if you would be looking at a runbook in the middle of the night, it would just tell you, oh, perfect, whom killed, just increase memory limits. So, it might work for

the first example, it might also work for the second one for some time, right? And then it doesn't. And what do you do? I don't know. The runbook just says increased memory limit. basically that's why we started seeing that deterministic approach of like grouping something uh into logical steps to start to fail right it couldn't distinguish between um the configuration like the limit is wrong or like

doesn't fit to between the application was wrong and is not behaving properly right so next example is cascading failures right This is an early one. And in this example, we can see we have six pods and they are all in crash loop. Not all, sorry, there are three in crash loop back off. Uh, so some are running fine, some are crashing. And there's even one that's running,

but it has restarted six times, just not fast enough to enter crash loop. And knowing how the application works in this environment, we can see two dependency chains. um and they look similar but they're pretty different problems, right? So the first one there's the order processing pipeline. We have a notification service that uh triggers our order processor service whenever somebody makes an order and this order then

gets written to a radius everything was failing up to the radius cache. But uh really the actual underlying issue was just that the Kubernetes service for radius was um wrongly labeled right. So it has a a selector that was labeled app radius instead of app cache. So radius was working properly. The service was existing. It just didn't map to any endpoints to any actual pods uh of

that instance. So when the other processor in that case tried to uh write anything it just failed right it tried to connect to the database to the cache and it just cannot so it crashes and same thing for the notification service it's it's trying to um push notifications but they cannot be accepted it cannot be written so that fix can fit in a single like cubectl command

and just fix that label selector the second one uh different application case. So it's a data analytics pipeline and we have a dashboard that basically try to query from uh some kind of walker and this is getting data from a database right so here the data store is actually running uh running fine but it's returning a bad response bad data right uh and in this case there's

like a I think a comma in the JSON payload and when we try to read that JSON it fails right So we don't have enough error handling and then that service crashes uh and it cascades further up and then the dashboard also just doesn't render properly. So the fix here is not really something about infrastructure. It's something about the application itself to fix the response and to

fix the data that's being served. And these two cascading failures might look identically the same based on like the symptoms and the signals that Kubernetes gives us on what's wrong but without really understanding without having the context of how this application works and what they require it's very hard to find um what is the what am I supposed to do right to troubleshoot this problem and uh

last example will be the app So again, same error, different root both of these deployments are accumulating pod restarts. Uh because a livveness probe has failed, right? Exact same error message, exact same symptom. If you do cube cattle describe, you will see livveness probe failed. Um but in reality, the causes are very different. So the first deployment is just a misconfiguration, right? The application itself works properly.

We have that health endpoint we are looking someone mistakenly configured that livveness probe to check just port 9090 right when the server really is listening on port 8080. So Kubernetes cannot really reach that server and it thinks it's unhealthy. So it will try to restart it again and again and again but just it never works. So fix is obvious here. And the second again same livess problem

that's not working but what happens there is that there's a blocking operation uh on the applicative level that basically somebody's generating a report in that case uh and it's just blocking the main thread right because somebody decided that that's how it's going to be. Um, and when the livveness probe tries to pull basically for if the server is alive, it just doesn't get a response. So, it

times out, right? And then it fails. Uh, and in that case, changing the port isn't really the solution. It's not even a problem. And we need to uh most likely just move that logic to something that is not blocking the main >> [snorts] >> um same Kubernetes CL one would be the fault of like operations and and infrastructure and one would be more of the development team

and applic uh and again like uh a system or runbook has to know the difference right and it can't really because if you look at a runbook with this information it will tell you oh crash loop back off try this you don't have more information to give the runbook for it to give it uh something more specific. Right? So this is what's wrong about predictability. You can

categorize the failures, you can put them into groups, you can detect a signal. Uh you cannot really prescribe uh an actual fix without understanding the full pictures of the problem. So what happened when we tried to ship that runbook that we made, right? uh spoiler, it did not work. And I want to be honest about this because I think it's really the most valuable part of this

talk. Um that failure is okay, I think the way we learn things. So the problems we ran into is that we kept finding for example uh very subtle differences. For example, incident A and B were just very similar, right? We required fundamentally different path to diagnose the issues like we just saw and every new customer environment we on boarded with this uh runbook solution introduced new failures

and new things that we just haven't seen before even looking at one million completely different real incidents and these runbooks really handled context um poy right so when a customer had a custom admission controller or just you know some networking or storage thing that was a bit different than how Kubernetes works by default uh even like a very customized Helm chart that works a certain way then

the runbook assumptions just were not correct right it just broke down the whole system and we had no [snorts] way saying I don't know enough about this environment or what's happening with this problem uh to be confident in what the next step it is that I'm going to Because a runbook has like a limited number of path, right? that we made when we updated the runbooks, right?

We try to improve it and um it would break a previously walking path in the runbook, right? Kind of a regression. So we are basically playing whack-a-ole, right? We're fixing something else and something else pops up. Um and in the AB test results you see here it's basically uh just the difference between two different uh root cause analysis that we made for a specific issue and how

it improves or uh fails in different cases. And these represent like weeks of really tweaking runbooks and just playing with the logic and testing it and all that. And every time we see some metric uh or like success uh percentage growing, you would see a different thing just going down. And that's not exactly what we So we understand that, you know, organization handles incidents and investigation a

bit differently, right? One company's s team wanted a step-by-step guide, right? How to go and fix the issue. Just tell me what to do and I'll go through those motions. Another one wanted root cause analysis and hypothesis with supporting evidence and kind of explaining what is the issue at a high level so they can take it from there. uh a different one wanted just immediate remediation, you

know, just give me a command, you know, to fix it, let me run it. And so having a standout uh between that level of diversity was just on the verge of impossible or it was very very hard. And when 80% of cases involve some unique aspect, it's no longer really an edge case, right? It's it becomes a norm. So your runbook becomes the edge case. And we

had to rethink the approach of what we of the solution to the problem we're trying to fix, right? Which raises a very important question. Did we fail? Some think yes but no because otherwise I wouldn't be here to tell you about it. Um so failure actually taught us exactly what needed to be built, right? We learned from that uh experience and the reality was that the having

a runbook was really the wrong tool for the problem, right? And what we needed was something that could think like a good S u not a flowchart, right? [snorts] So what does a good SR or actually do? They got a context. They investigate. They reason. They adapt. And that's what So the solution is what we call agentic AI sur. It's a very powerful name. And I'm going

to explain to you what it actually means in practice because this phrase gets thrown around a lot and often means very little. the breakthrough was that LLMs handle large context windows pretty well, right? And just gets better with time. Uh but not infinite context. So you can't just feed it your whole organization and all code base and expect it to just walk and be smart, right? You

need to know uh that there are limits. But a well ststructured context window with the right information at the right time is much much more powerful than any runbook you can encounter. And when an AI knows that the how the cluster topology works and the recent deployment that your application underwent um the previous incidents that your team has gone through and the specific error that's happening right

now hello yes okay uh so when that AI got all these contexts and information uh and the specific error, the diagnosis quality really dramatically increase. And so we also thought that we needed some kind of operational knowledge base, right? Because we collect ton of data from customers and we ingest like how things run in Kubernetes, everything that happens, all the resources that change, events and metrics, all

that. But also we needed some architectural blueprints right in in some way. How how things are connected together, what is dependent on what um how they deploy, how they configure, what third party uh tools and cloud providers they're actually using, right? And that way the AI isn't flying blindly to just a generic Kubernetes environment. It has the context to know exactly their environment, right? uh it understands

this cluster that it's trying to find a problem a solution for. one important things that during our testing we found out is that the way this works best is with a multi- aent approach that has a very narrow focus. So rather than just one large agent uh that has all the context and you know ask all the questions we use specialized agents. So we have smaller pieces

of AIS that basically knows how to do one specific things very very very well and that works wonderfully into uh preventing basically it lying to us. Um and so the way this works is you have the main agent that is basically the S sur right the brain of the operations that will call different employees or sub aents uh that knows how to do one specific things and

once you get a concrete answer or like specific piece of knowledge it will take just that part instead of the whole context that you provide that sub agent. And so this really really helps us keep the um the context window as narrow and focused as possible so that we have better So we have resources agent that knows some capacity and scheduling. we have networking uh you know

things like that evaluating these kind of agents and framework that we build is the thing that separate uh something that works to something that doesn't right it takes a long time and a lot of effort to get to a point where you can actually trust this thing you built um and I can tell you that at first like the success was okay, but like I wouldn't trust

it to just blindly accept whatever it tells me. Um, and we got basically to a point the the quality of what we shipped here increased to a point where we just could, you know, [snorts] be confident enough to tell that to our customers. This is a recommendation. This is what you should check. This is how you should fix your problem. And the way we do that is

we use LLM as a judge. So we have an other AI model that basically it is whole purpose is to validate and test the output of what this AI SR is telling us to do, right? It will take that uh output and problem actually implement the recommendation that it gets and sees if it actually fixes the problem, right? So we have a sandbox where we inject like

all these kinds of scenarios and and outputs and we and we uh validate these things. And we also run like every new version of uh models or code changes as like kind of canary testing. So like side by side without really uh having it in production. And so whenever we do some changes, we also have the ability to feel confident that any change like that would be

beneficial and not cause a regression. then we thought what would a good SR actually do, right? This is the question that you must ask yourself uh when designing a solution like that. Not what would a flowchart do, what a runbook would look like, but what would a great experienced engineer like yourself would do with all this information it has, right? That's the target. That's the goal. And

this AI sur went under Claudia. this is our solution and basically it's a S that does not sleep and does not complain. And there's three things I want to highlight about how she operates. So she works continuously, right? It's not something that's on demand. It's not a chatbot you query when you have an issue. It's something that's always watching, investigating, and surfacing problems uh proactively when they

happen. And reactive uh when something actually crashes, it automatically like uh starts a root cause analysis to that way you you have a an analysis and a remediation. as fast as possible. Second thing, uh the capabilities of this agent expand daily, right? And this is very very crucial. The AI space is something that moves very fast and there's new techniques, new models, new data points and so

we build that system to be as modular as possible. So we can upgrade components and change behavior without rewriting the whole thing. and um to have the ability to still feel confident that we're not breaking 20 other things when changing one small piece of it. So that architectural flexibility of building a system in such a way, it's not always the best solution depending on what you build,

right? But for something like this, this was the best approach we found. And let me show you what it looks like at scale. So out of 4,700 uh flows in that past day, this is actual production data. Uh there are 12 different sub aents and like specialized uh AI models that run. And now you know that I can count to 12. Um and so we have a

99.2 success rate of actually having root cause analysis. that really mean something and help troubleshooting that problem. And we do that in about 84 seconds on average. So think about what I showed you at the beginning, right? that weird huge flowchart that runbook uh that a human engineer an S sur you have to go through and spend basically like 30 minutes to an hour just loading logging

into your systems and finding the data and checking things and reading and finding all the pieces and and correlating them together. So an AI like Claudia can do that in under 90 seconds with the context that it needs um at 3:00 a.m. without complaining. And so this 99% is our evaluation score, right? It's how often the investigation output really is accurate enough uh and something that's actionable,

right? Because it's fine to say you have a problem. If you don't say there's a way to fix it, it doesn't really help that And this is a real number like it's not a marketing number. It's our internal quality gate kind of right. This is the number we look at before releasing new features before feeling confident enough to uh uh ship a new version and and and

go ahead with such a thing. So that's when we knew basically that uh this solution moving to something AI based was working right and let's talk about what I think you should actually uh what you actually should think about when considering building or buying an AI SRE. Okay. So first thing and I know I've talked a lot about context but context is very important right um this

is the most important questions and does the system that you're uh buying or building understand your specific cluster your specific application your organizational context your team how your systems and works and their convention or does it just give uh generic Kubernetes advice that you could get from any other chatb based LLM for if you're thinking of buying such a tool, it's like data and kind of points

that you need to ask for. Uh second thing is what I just showed you that evaluation framework, right? Everyone can write whatever they want. It works 100% of the time. How do you know that's true? Right? It's something that you need to challenge. It's something that you need proof. Um, and you also need to know that when you're putting your trust into such a system, especially with

AI, uh, that system should have a rigorous like evaluation pipeline and quality standards. Uh, so the first question is how do you measure accuracy, right? If you're building, if you're buying, it doesn't matter. How do you measure that something actually works the way it should be? And that should be the first question for anything you're building really. Um, so you need something repeatable and that's deterministic. And

lastly, does it know what it doesn't know? What that means is the product you're building um or buying knows its boundaries, right? So with AI there's been again less and less lately but hallucinations right because an LLM cannot say I don't know it will just give you some kind of answer right so you need to know to put these boundaries and you need to know that they

exist so that you don't blindly uh trust or guess get guess get guess get guess get guess get guess get guess get guess just like false advice from such a tool again The the last thing to consider is basically price, right? If you're a small company and your scale is not that big, you might be better off just buying a tool and it's just easier. If you're

a platform team in a large bank with 500 employees, maybe you can build it, right? Maybe it makes sense. Maybe it's something more customizable for your needs. Uh these are the kind of things that I think are important uh when thinking about that and yeah the bottom line is AIS is a real thing. uh I think personally that it's going to really really change how platform team

operates and how monitoring as a whole uh comes into play into companies but only if it's built the right way with the right architecture and that the data I talk with is actually data driven like with the the correct context and not just generic problems and issues. Uh that's it. Thank you so much for Thanks for the talk. I'm curious about the many agents with narrow focus.

How good are they at recognizing this is not my department versus proposing some wild solution? Sorry, I'm I can't hear that. >> um, so with the many agents with narrow focus, how good are they at recognizing that it's not their department instead of proposing some wild solution? >> How does a I sorry, [laughter] >> how do the agents recognize when they're the right agent for the analysis?

>> Right? So it's not that the AI recognize uh that he's the right one for the uh to do the analysis. It's the main agent basically has a a set of criterias that it based uh which sub agent it needs to ask for depending on what problem. So it think of it like a runbook but smarter, right? So here also there's categories of basically which uh types

of data um are related to which sub aents. So like I said, networking for example, there's one agent that this is its whole purpose, right? Does this answer the question or Yeah. >> Uh other than the company you work for, are you planning on releasing any of this so that we can actually try any of the techniques that you mentioned? I don't know that there's anything open

source we're going to be releasing soon uh in regards to that. Um so no we do have a free trial so you can look at how it works and what we do. Um but yeah we try to give out the knowledge as best we can at least on how it works and what we built. So like you helped sort of categorize like six broad patterns of symptoms

of issues. I'm kind of curious if there's been efforts to try to like categorize like the cause of like those changes in terms of like for me when I think about incidents is like sometimes due to code change, config change. It could be due to like scaling load or like another example being like upstream issues you know third party. >> Has there been any thoughts of like

or work on categorizing? >> Yeah. So this is a really good question actually and given the amount of data we have and that we come through we have the answers to this. Uh the problem is that it's hard to put categories into play when you have so many of them, right? The upstream uh reasons, the the actual like root causes that falls into these categories are just

there's so many of them uh that we found it hard to put into labels on it, right? Um it's it's data that you can definitely like do. You can say you know top three contenders you have uh configuration management systems uh networking and uh resource management for example. >> Any other questions? >> Well I have one. What was the skills development process like for the agents? >>

What was the skills development process? it started by just like a thought process of again what would an SR do if I were to diagnose an incident and uh troubleshoot it. What would be the first thing I think like right and then going uh going back on previous incidents that we personally fixed in the company or helped customers fix. We basically wrote down all the different skills

or different capabilities that an autonomous thinker would need to do and we try building them one by one. And then using these skills, we put them through that evaluation pipeline I talked about and basically try to understand how reliable they are, right? And then we iterated over that uh scale by scale. Okay, thanks. >> Any more questions? >> All right, so in any case, if you do

have any more questions, you can feel free to add me on LinkedIn or catch me here later or tomorrow. Uh, and if you do want to try Claudia, we do have a two weeks free trial. Uh, if you want. Thank you so much for Okay, thanks everybody and don't forget we have upscale plus you know some snacks and drinks um in the main keynote room uh starting

in 45 minutes I think.