Ballroom B Saturday Mar. 07 - SCaLE 23x

This talk discusses the use of BootC, a tool designed for provisioning and managing hardware with container technology. The speaker, Josh Burkus, explains his journey with BootC through a personal project of creating a music server using a Beelink x86 machine and outlines the historical context and evolution of Linux deployment technologies. He delves into various deployment models and the benefits of using bootable containers, which combine the functionalities of traditional containers with system-level operations. The presentation highlights the challenges of scaling in edge computing environments and emphasizes the importance of rollback capabilities and efficient resource management. The speaker also provides insights into the future integration of BootC with container registries for streamlined deployment processes.

We'll get started in one minute if you want to get seated. I'll give a few safety announcements now though. Uh given that there's we're in earthquake territory and let's hope there's no fire. There's an exit there, an exit there. If you get really excited about the talk and somebody needs the defiill, it's out the door just about 15 feet that direction um mounted on the wall. Um,

if anybody feels unsafe here or at the conference throughout the venue, there are signs with contact information to report uh your concerns with safety. There's one taped on the back door. Okay, it's our official start time. Uh, welcome to the cloud native track at scale. Uh, our next talk, uh, featuring Josh Burkus is on what you see here, how to provision your hardware with a container. Now

Josh is one of my personal favorites at scale and I think I see many of you who are repeat visitors who probably know this that he's been giving talks here for over a decade. But the nice thing is that it's not the same thing over and over and I think he's particularly good at spotting new and useful technologies early. you know my the first talks I went

to from Josh were on Postgress but later moving to containers Kubernetes and cloud native track uh talks and uh today's topic I think is pretty unique that it isn't typically covered at tech conferences if you don't mind save any questions to the end and I'll turn it over to Josh can we give him a round of applause if you do want ask questions Please raise your hand

so I can give you the mic and your question gets recorded. >> Thank you, Steve. Uh, am I loud and clear in the back? Awesome. Thank you. So, let's talk about a little use case here. So, both my wife and I work from home. We have separate offices on different floors of the house. Um but uh one of the things that's come up recently is um we

don't want to listen to Spotify anymore because issues and we actually have this huge CD collection um in including a whole lot of uh folk music that's no longer available. And so we'd really honestly like to have some kind of a music server that we could both share in the house, right? because uh you know I don't know something like you know 60,000 tracks um you know

we might as well listen to our own music. Uh but that's actually going to require some kind of a music server device. Um and since I've been experimenting a little bit with Bootsie um which you're about to hear about um I decided to actually make this a test case for uh you know building this. So, you know, I bought myself a little BeeLink, which you can see

right here. Um, you can come up and look at the hardware later if you're in the back. So, just a little Beink x86 machine. And then, you know, I did what anybody would do, which is I wrote some code and config stuff and I packaged it up as a container. And then I use that producing ISO and I went ahead and installed it image the system. And

a lot of you who are CIS admins, how many people's a CIS admin? Yeah. are going to be like, "Hey, hey, wait a minute. Wait a minute. What was this part? What are you doing there? Why? How was the container involved? Let me explain." Except to explain, I'm going to have to do a little bit of a tangent. So, I hope you're ready for this tangent. Um,

where I'm going to talk about the history of Linux deployment technology. Um I all of which I was personally present for. Um so this is this is my memory lane here. So um how many people here have been doing Linux since the 90s. Do you remember having to regularly compile your own kernel in order to install it on the hardware you actually had? Yeah. Yeah. In the

mid90s that was how I started with Linux is you literally had to compile everything. So installing a new system was like a three-day weekend, right? because the compile often didn't work the first time you did it, right? And then you had to start over again. It was going to be another 13 hours, right? So that wasn't a lot of fun. And so what happened was in the,

you know, mid late 90s, Red Hat introduced packages followed closely by Debian and Susie and that sort of thing. So the idea of having actual binary packages for a lot of components of the system that was a huge improvement, right? Installing a new Linux system went from a 13- 14 hour affair to being a 3 to four hour affair with a lot more reliability. However, going through

those interactive menus and selecting every single package that you needed was really, really tedious. Right? So, a bunch of folks, including some folks I know, came up with this thing called configuration management, right? We're like, hey, we're going to have some code and the code is going to select all of the packages that we're going to install, right? And that way it's three to four hours, but

it's not like four hours of interactive work, right? It's four hours where the computer is running and we're just paying attention to it to see if it crashes, right? So again, another big improvement. However, still some problems, right, with the configuration management package thing, right? Number one is it's fine if you're installing one machine or two machines or four machines, but what if I have to install

a thousand right? You have a certain limited amount of parallelism and it's just too slow. And the other problem is you think of packages as binary things that are just being copied out of the file system, but unfortunately that's not real OS packages, right? In the RPMs for example, they have this thing called the postinstall script which often does a lot of interactive stuff the output of

which is not determinative which means you can get unique failures on each system which it is not able the individual system is not able to recover from. Right? Again one system not terrible but if you're doing a thousand systems and 105 of them each have their own individual unique failures you're in for a bad couple of weeks. um parion is challenging um you know in terms of

imaging a whole bunch of systems and we'll get this there's no concept of rollbacks right if you start imaging installing a system and something goes wrong you can't just snap back to the systems previous good state so then we got the cloud right and with the cloud brought with it some actual improvements in installing which is that you could actually use whatever your cloud provider ider was

you could use their native configuration system in order to define a VM image that had all of the software that you wanted, build that VM image and then just deploy that to virtual cloud instances. Right? This was a huge improvement in reliability and scalability. However, it still had some problems. Number one, each cloud, you know, each cloud provider has their own set of proprietary tools for building

this, right? Later on we got Terraform and stuff to actually make it less proprietary. Those images are really slow to build relatively speaking. Um these large images are hard to distribute. They only work on that particular cloud. They don't work on other clouds let alone on bare metal. And we still get no roll backs um if we want to reimage an existing system, right? because they're assuming

you're going to deploy a new virtual machine, right? Which is fine if that's what you're doing. But if you're going to mix bare metal with cloud bare metal, you want to think about rollbacks. So in the meantime, in the area I was more working on because I wasn't a system admin anymore, I was a database engineer etc. Um we got this thing called application containers, right? Docker,

right? So we've now got these binary images, these tarballs to distribute an installed and configured application. And this had a lot of advantages, right? We had very simple, easy tolearn build tools based roughly on bash. Um the images build a heck of a lot faster, right? Easily two orders of magnitude faster. Um the smaller images make it easier to distribute. You could use this thing called the

container registry that was just like, you know, a package mirror or whatever, but it's for a whole, you know, system. And containers were useful both in the cloud and on bare metal, right? You could move them anywhere around, bare metal, cloud, edge device, whatever, cross-co compile them to a different architecture, you know, that sort of thing. And importantly, because it was a binary image being installed sort

of bitwise, you could actually roll it back, right? you distribute version seven of your application image, it turns out to be broken, you can roll back to version six pretty quickly. So this is terrific. So wouldn't it be nice if not just for an application but if we could do that for the whole system whether it's a cloud system or a bare metal system and that is

where bootc comes in. bootc is the concept of bootable containers right the idea is that you can build a container images and then you can boot it on a virtual system or a real system as an OS um it deploys the containers the system images it manages system status um updates roll backs everything else and importantly supports roll back so that I can image even a bare

experimental system and then roll back to the last good state if it turns out that I made a mistake. The way that it does this is so your standard app docker application container right has a base image which consists of basically Linux user space and then all of your app code on top of that base image right um a bootable container contains those things but we are

also copying kernel space into it because we have to be able to boot it right so you have the bootloadader and basically the whole Linux kernel so this image is going to be significantly larger than you used to with application um in order to make it bootable. But from the outside, aside from the larger size, as far as container registries are concerned, as far as container tools

are concerned, it looks just like an application container. So now that we have Bootsie, actually building that system. Right? So this is again our initial provisioning right. So I have some code I use the code to build the a container. Um I used the container to create an ISO because I'm deploying it to an unprovisioned system on my desk and then provision that to my system. Um

now need to talk about this step container. ISO right because it's not actually that simple unfortunately. um we have this thing in the middle called Bootcy image builder um because we're taking a container right and from that container we can actually produce um gods I don't know like 11 or 12 different kinds of system images u if we actually look here let me show you a fun

little tool called podman desktop which has plugins for a lot of these things and if I take these are my bootable container images that I have on my system right here. Um, so like I'm going to take CentOS here and if I want to actually build a new bootable OS image, right, I can do all of the common system image formats including QCAL for, you know, basic

VM, VDMK for, um, Virtual Box, um, or, you know, Anaconda ISO installer for installing to, um, hard drive, plus some some other formats for for doing more sophisticated installs, But so the problem we discovered with that for a command line tool is um that being able to build all of these different formats requires a lot of different build tools. Um uh some of which are actually really

annoying to configure correctly. And so what the Bootsie team what did was they created a container image called Bootsie image builder that contains all of those VM and ISO building tools. So I'm actually taking a container image, feeding that container image into another container, having it process that and output an ISO, which is a fun little bit of bootstrap levitation. the um however I'm not actually going

to demo that because those two steps are pretty slow. Um I clocked it to see if I actually could demo building the ISO in here. It's like 15 minutes, that sort of thing, which is not terrible if you're doing it like yourself in the office, but for the middle of a conference presentation is not great. Um so you're not actually going to see that part live. And

also the other reason you're not going to see that part live is in the future that's not what you're going to be doing. What I wanted to do the original path was Oh, by the way, here's a here's a little shot from the middle of me installing everything. So, this is it launching the Anaconda installer. Um, I have this little micro Raspberry Pi screen, uh, which works

well for this. what you should do in the future. Um and uh I ran into a couple of blockers on this and uh Colin, you know, who's one of the maintainers of Boots Cy is like working on stuff. Um, which is what you really want to have happen is you build your container and then you push it to a container registry. And then what happens is you

pixie boot the individual device and it pixie boots into a very minimal Bootstrap operating system that knows how to talk to the container registry. And so that communicates with the container registry and all the Bootstrap operating system has is what bootable container it's supposed to be running. It pulls down the bootable container, uses that to overwrite the file system, and then we are booted into the application

server for the device. So expect that to be working in a couple of weeks. Um I'll I'll update my repo um with instructions for that um once I've actually got that working. Um and that is the path for the future because while this looks more complicated as a path, the important thing is all of the all of the slow steps happen without human oversight, right? The human

just initiates, hey, we're booting a new system on this image. and then they don't pay any attention to it until it tells it's until it says it's up and ready or that it's failed, right? And one of the other reasons why you want to do this is what this is what you already do when you want to update the system, right? Because once you've got your bootc

system up and running, anytime you want to push an update to what your device is running, right? You update your code and config. You build a new container. You push that to a container registry. The device communicates with a container registry. Um I think default is once a day. Um I don't I don't remember how often it checks by default, but you can configure it once every

>> twice a day. Okay. But you can configure the interval at which it checks with the container registry, right? That's completely configurable. Um and then it updates its code um and runs the new version of the application. Um, I'm going to show you this interactively because obviously we don't want to wait hours for the update. Um, so, uh, this is the demo code. Um, I'll go back

this in a minute, but in the meantime, let me show you the demo. So, first of all, like I said, I'm not going to actually interactively go through the setup because it takes too long. But I will show you all the code for this. So, this is pushed to that repository. Um, I have an explanation of what everything is. So, one of the ways if you're building

an Anaconda ISO as your installation option and the the simpler way to handle it is this thing called the config tommo file and that contains the configuration for what your ISO image is going to look like aside from what's in the container, right? Um, and so I've got in here of what kind of keyboard am I using, what time zone is it, what's the initial network setup

because you obviously need that or it can't communicate with anything else. um uh partitioning and um the um um you know and the modules for Anaconda um the um most of this you'll recognize is I've embedded the kickstart file in the config config toml right because it's just easier to have everything in one file and so for this demo what this is building um is not my

music server but um a very simple web server thing. So here is our base um bootable image. So the easiest way to get a bootable image is that you start from copying somebody else's bootable image, right? Um so this is Fedora version 43 bootsy image which is available from the Fedora project. You can get CentOS. You can get a bunch of other flavors which I'll go over

a little bit later in the presentation. And all we're doing here is we're installing Apache and auto starting it. We're installing Apache, auto starting it with an index.html that we've actually copied in. Um, so here's that initial index.html, just a very basic web page. So, copy in, push it over, um, uh, boot up. Um, and, um, I've got a couple other things going on here. You'll see

in here um because I wanted to do this without relying on the internet, I have a container registry running on my laptop. Um the um and you can see actually here I have a lot of the commands I used to build it. >> Yeah, this is the next generation. The question was is it similar to how you provision things with core OS? This is effectively the next

generation from that. >> Yeah. um this is actually the command that I used with Bootc image builder in order to build the Anaconda image. Um the um where it it pulls it out and you have to give it a lot of configuration. Uh this is for running the registration. Um, so, uh, let's go ahead and, so I've got this on a cute little local network. And as

you can see, my my little web server is up and running, right? So, here we go. Um, uh, that's all it does. but we could change it. So, let's go ahead and update it. So, I'm going to call this version two. And somebody want to volunteer. What? How should we change the text here? What should we put in here? >> Yeah. Okay. Um, and I'm not going

to go find the upside down exclamation point. Sorry. The um um Yep. Um, okay. So, go ahead and save that. And let's see. Is this still in my history? Yes. So, let's build the new container file, which just has one new layer. Um, push it to my local registry. Now, let's go ahead and look at this. We're going to go ahead and So I can actually show

this to you rather than using the console on the very insecure login but it makes it easier for demos. as I mentioned boots is not just a way of building bootable containers. It is also a management utility on the booted So I can look at what the current bootc status is here, right? And it tells me which container image I'm running. That version number there is pulled

from the Fedora base image because I haven't overridden it in my own image. So it's just going to inherit that version number from Fedora. Although you also check the SHAs because of course I'm changing one layer on top and so it's going to change over time. And you can actually see you know how there is um so then we can go ahead and we can update this

and when I put in bootc update what it's going to do is it's going to go to its original container registry which is information it has saved. You can change that later on if you need to move where the container is kept. um you can go ahead and update it and it is going to fetch the new layer um and then it is going to apply that

new layer virtually so that it is ready on reboot. So we've added this and if you actually if you look at bootc status again right you can see hey we've boot this is the image we're currently running but that's the image that's going to start up when we restart and so yes you do have to restart so that'll take just a second actually look at rebooting here.

Okay, we're not ready yet. Are we up yet? >> Right. Yeah. No, it's not. No, this is running um I don't know what it is. Lightweight AMD server, that sort of thing. Yeah. No, but I mean the thing is you can cross compile, right? But cross compiling involves its own troubleshooting, but you could run it on an ARM system if you're just ready to deal with the

cross compiling or if your development machine is also an ARM system, then you don't have to deal with cross compiling, right? >> Yeah. Yeah. The question was about doing rolling updates over a whole fleet of devices. And the answer is yes. And there are a couple of different projects to handle that. One of them is uh Cube Stellar um and also another one called Fleet Manager. um

that are specifically to handle that situation, right? Because you're going to want on your you're going to want a control plane that actually has an idea of managing your whole fleet. >> Orchestrator. >> Yes. Exactly. But it's an orchestrator for edge devices instead of an orchestrator for just containers. Right. so if we look at bootc status and the rebooted system, we've got the new image booted into

and the rollback image. And if we go ahead and reload, we now have Yep. Now, um, those of you who know your system control, know that there was a slightly faster way to do what I just did. So, let's go ahead. We're going to do version three. We're going to go ahead and push it. which is that BootC now supports something called soft boot. Um, which will

do a lighter weight restart that does not involve a full p power cycle if the changes that you've made don't require it. Right? If you're making changes to the kernel and stuff, it still has to power cycle. But if all you're doing is changing application code, which is what you're doing most of the time, it doesn't have to. Um, oh, whoops. Yeah. And so that'll happen in

one command and it'll be significantly faster. Give it just a second. Yep. So, that boots up a lot faster we got version three here, right? uh one more thing I mentioned roll back um because that was part of the whole point of doing this, right? is to have roll back. Um I So how should we break this? How should we break this s simple HTML file suggestions?

>> H >> like what? >> I want to mess it up. Yeah, the um Okay. So, you know, say say I screwed it up we're going to go ahead and apply that Okay. And give it just a second. There we go. Let's see how much that messed things Not a lot. The the problem is the Apache web server is really good at work at handling bad HTML.

So here let me let me do something actually saved which which really does mess things up. >> Is it? >> No, it's six. there we go. And now let's check that out. Tada. Now I could have avoided that first of all I could have avoided breaking into the first place because one of the things I mentioned is testing right so we've been showing all this booting of

the system but one of the things I actually want to remind people of is well this is a bootable container that is booting to the system it is also a I could have tested that and avoided breaking the running system right Oh, wait. I need supports here, don't I? Oh, whoops. What happened there? So if I actually do that, I could have checked it in advance and

you can see I could have actually checked that and caught the problem before it deployed to the running system because even though this is a bootable container, I can also run it as a regular container and so run it through all of my CI stuff as a regular container on Argo or Tecton or whatever. um in order to do testing before it gets deployed to a system.

Um or if I have to do special hardware things then I can create a different kind of VM image out of it and do my testing on VM. >> Yeah, the question is does it require podman to understand that? Um I haven't tried to run one of these on docker but I don't know why it wouldn't work. you know, it just ignores the kernel stuff basically. Um,

so if the system you're testing on is a radically different kernel version than the system you're deploying to, then that could be a problem in your test validity. Um, so you want them to be at least fairly similar, right? But other than that, but I didn't test it. So that means I need to use here we go a very important feature here which is And now you

can see with the roll back um well it actually doesn't tell me that it's going to boot into the roll back image but we've got this roll back image and we're going to go ahead and reboot and that'll take just a minute. updating the thread. >> Yeah. Yeah. I'm not versioning the tags. >> Um uh the important thing to keep in mind is by default bootc is

looking for latest. So, if you're going to use actual version tags, which is an excellent practice, you need to also remember to update your latest version as latest because it's looking for that latest tag. Yes. >> Yeah. So the question was, can you label your images so that you actually know what the changes were in each version? Um, and the answer again is yes. This is very

similar to his question, right? Which is using more informative tags. But like I said, you have to keep in mind that Bootsy is also looking for that latest tag. So you want to make sure to apply the latest tag to the latest version as well. Um and the other thing is by default for space reasons the default for boots is it only holds on to one to

two old versions but that is a thing you can configure on the system. You can tell it to hold on to more versions which would allow you to actually roll back further. And then what you would add to that roll back command is an SHA to tell it to which version to roll back to. So, that should be rebooted and fixed. It is not rebooted and fixed.

>> Oh, I'm in the wrong one. You're right. Thank you. There we go. Um, and that is that is rolled back and fixed. So, with that, let's go ahead and finish up our slides and then I'll take the rest of the questions. Um, so again, let me hold this up for a minute. So, the code that I just used is on this repo. um as I get

my my little music server up and running, I will add extra stuff to that for um the further things uh for doing that. and so the update advantages are you're not going through choosing which things are getting updated and which things are not getting updated. Um, one of the other things that I can't really demo in a simple example like this, but um, I Fedora got an

intern who worked on actually doing intelligent algorithmic image layering so that um, at least for Fedora and CentOS systems, it actually arranges the layers in a way that for most users will result in doing updates that are the smallest um I that they can reasonably be you know considering that a lot of underlying libraries and stuff are probably getting updated etc. um there's no possibility of a

partial update, right? There's no possibility that like half my libraries are going to get updated and the other half are not, right? Either the whole thing gets updated or it doesn't get updated. Um um I one of the things is if the update actually fails, bootc will automatically roll back to the last good version. So if it if it refuses to boot, it'll automatically roll back. Um

one of the other great features here that I'm not prepared to demo, but I'll talk about is bootc switch. See the idea of bootsy switch is I told you I can configure where that container image is. Now uh the idea I can configure where that container image is. Now obviously one of the things I might be doing is just moving around where my image server is right

but I'm not limited to that. I can tell it to use a different image like a completely different image. Like say I've got the operating system image one here. Like say right now I've got the simple web server, right? But when I want to put the music server on there, I'm going to have a completely different application and I'm not going to be running Apache. So obviously

I could reinstall the system, but I don't have to. I just tell it to use a different image. Um, and it pulls down the new image and it does a bitwise change from the old image and I am now running a different service. I can even do a wholesale change, right? Because right now this is running Fedora 43, but what if I decided, hey, this is a

headless box. Maybe I want to run CentOS instead. I can literally do that. That's going to be a big change. It's going to change a lot of bits and it's going to pull down a lot of layers to do that, but it works. Um, as a matter of fact, the um for for some of the other bootc systems that didn't have installers, what they would tell you

is install Fedora Silverblue and then have it pull down a new image. Yeah. Well, no, you have to reboot after the new image is installed. the um the other thing is obviously I've been showing you all this for basically an edge device. Um this is not just for edge devices. Um here at the conference you will see talks about um doing it on desktop devices um including

things like the Fedora atomic desktops and bluefin um also gaming consoles. The Basite project um uses gaming consoles and this is all bootc based. So and let me actually Whoops. What just happened? Okay. Um, let me actually show you that. So, this is not in the device. This is So, my laptop. Oh, right. Hold on. This is my laptop which is running blue fin. And you can

see right here um that oh, I've got a pending systemd update. Okay. Um, aside from the spending system do update, actually here is my here's my example of automated roll back. Shortly before the conference, I I have I'm I'm running on basically a twoe old version of of Blue Finins skin of And shortly before that, we actually got an update that broke systemd. And so my laptop

just automatically rolled back. And so you see we've got the staged one that with an error message and it's running on the last good version. And so I didn't bork my laptop and be unable to use it right before the conference and let's get rid of that. So what's next from here? Well, for me, I need to finish provisioning my music server, at which point you won't

see me bringing this little box in the road to get a new box for on the road demos. Um, because it'll be at home and my sweetie will not let me take the music server away. Uh, you know, so I need to actually, you know, fix the, you know, there. Um, I'm probably actually going to use Quadlet and run the Jukebox, uh, program in containers because it

actually has several different components, but I haven't decided on that. Part of part of what I got down here was I got bogged down into the different open source music players there are and I wasn't sure which one to use and how they actually get configured. And I also need to figure out a way for the storage to be separate from the rest of everything because I

don't want to include all um 50 gigabytes of soundtracks in my image. But that's coming. What's next for you is um go ahead and try and play around with Bootc um for a simple pointandclick version of doing a lot of this. You can use podman desktop um and then install the bootc module for it to play around with bootable images etc. Um and that avoids you having

to actually come up with a long command line for using bootc image builder etc. um Boots Cy uh open source project CNCF project. Um so please if you get into this join in ask questions become part of the community and with that we have time for you to ask me questions. >> So I I think the description said this is also good for VMs and physical boxes.

So I think a lot of people are kind of frustrated with the state of some of the auto deployment tools especially in the Red Hat family. They're >> they're kind of not great. >> Um and junior admins are just completely unable and AI is unable to deal with it either. Is this a viable solution if you have a lot of junior mid-level admins to do full like

item potent deployments of physical and virtual Linux boxes using a you know everything from Rocky to Yubuntu and more uh and and to do this in a way that that doesn't get management to say enough stop the line events because your thing broke and stop doing that. Yeah, I mean you still need tools to manage your whole fleet, you know, environment of machines, etc. Right. And those

tools are still going to be complicated. >> Yeah, that that would be anible or something. But I'm just talking about for the deployment part. >> For the deployment, yes, >> because the Anible part works really well. It's just the deployment part is really poor. >> Yeah. And this can make that a lot simpler, right? I mean, I can't necessarily say what they're doing with RE um because

that's a supported product pipeline and they tend to be very conservative about changes to deployment there. But Rocky is impacted by >> but right but if you're using one of the upstream versions then yeah you can use something completely different and use anible and like I said again um once we resolve a couple of the users having the shim OS which will not vary which will only

vary by machine as far as the hardware is concerned right is going to make things much simpler for a junior cis admin. Um, hi Josh. You you said uh bootsy containers are also containers. Yeah. So you can just run the runtime and use whatever the host kernel is. >> Does that mean that there are limitations to what kernel you can put in boots containers? Are they limited

to Linux kernels or they can you can I run another operating system as a booty container? >> Um I would say theoretically yes. The trick is that you need somebody in one of those other operating systems to create a master boots image for it. Um, there's nothing that would prevent you from running FreeBSD or Nyx. Uh, well, I don't know that much about Nyx's architecture, you know,

but theoretically there's nothing that would prevent you from from creating a boot image for those. I just don't know that anybody's done it yet. >> Are there issues with uh doing this on a physical system with UF UEFI secure boot enabled like signing of the kernel and things? Yeah. Um it's more complicated. The config is more complicated because you need to do all the UEF and all

of the signing and everything else, but that is supported, right? Because keep So um part of this is like the reason why my employer puts has put a bunch of engineers on this um in addition to the community folks is that this is all supported through RE 10. Um and as a result, they were not going to push RE without having secure boot. So that's all supported.

Um it's just that the upfront configuration if you're doing the upstream if you're doing an upstream distribution the upfront configuration for you because you need to actually check the signature authority and everything else it's going to be a much longer config to file. uh when you rolled back I was wondering if on reboot it's just going to go to the latest or is there a mechanism for

no stay at what I tell you to and is there a mechanism for say you have a set of VMs or hardware that you says like no this should always be on the latest major release or something is there any mechanisms for that >> yeah um so if you're so we're talking two different things for roll back right normally roll back just goes to the last good

version that was booted Right? Um you see that roll back version tag. You can manipulate that particular if you've changed the system already to keep more than one version in reserve. Right? By default out of the box, Bootsie only keeps one version in reserve. So you don't have a choice. Um but you can change that configuration to have more than one version in reserve and then actually

pick um for updates, right? Um, again, by default, it's going to look for that latest tag, but you could tell it to pull a specific image instead. Um, and then that would be up to your automation to handle that instead of just using the defaults. >> All right. My question is this. So, I I see you have it on your laptop and start for a server someday,

but what happens if I need persistent data? um does this format the disk to make a space for the images or does it run in memory and if so like how do I keep data persistent through this? >> Um so um it only formats the disk when I first did the install. Um uh after that it's only changing the places the file systems that Bootc covers. Um

one of the things that it does change is if you want persistent data you can't just stick it anywhere you want. Right? there's specific portions of the Linux file tree. So the ver um uh direct ver temp um any additional roots that you add right so don't stick any persistent data in user or bin um you know because those will get overwritten but for other base directories

you can stick persistent data in those because those will not be overwritten by the image and we've got actually this guy at the end has been really patient >> okay it was the same question got it >> uh So, can you do this with uh um without a hard drive? Can I run this just in memory? >> So, the question was, can you do this without a

hard drive? Can you run this in memory? I don't see why not. I mean, you'd create a virtual file system. Um and then install to the virtual file system, right? So, I don't see why that wouldn't work. Um haven't tried it. Uh so I work in a regulated industry that mandates a particular file system uh layout like separate partitions for bar and bar log and audit and

all that. Uh how do you configure that with bitc? >> Okay. Um that's not something I've contended with. um that might actually end up being a problem for you as in if they've stuck things in directories that Bootsie considers to be a part of the Bootsie image. Um then you might honestly need your own fork of Bootsie. Um I don't know that there's any way Laura do

you know any way to configure that like at runtime. for reference. >> You can actually do some sim linking from the var directory to those directories to be able to write there and then that var directory won't change. The actual underlying project that you would want to look at to understand how it works is called osree. Um there's also compose fs depending on which system you're actually

running. I think the universal blue family is all on compose fs right now. Um but OS tree itself is underlying the fedora based ones. So just for a clarification there and there's a lot of information on those two as well. Current Fedora does have composs but I think still running on OS tree until Fedora 44. Thank you. Uh, one of the things that got brought up was

that you're using your target device is x86 and uh, so you didn't have to cross compile. Um, I understand how cross compiling could be added, but for the part where you run you're able to run it uh, outside of bootc just as a normal container. Do you know what's involved in in doing that? So like if you had cross-co compiled and you have a Raspberry Pi application,

you want to try that from your computer. >> Yeah. Yeah. I don't because I don't do a lot of multi-arch stuff. Um Dave, if somebody wants to run an ARM container for testing >> on their laptop, how would they do that? Uh, Quimu can do it because you can always just run like a virtualized arch environment to test your Arch image. It's slower, but it works. >>

Yeah, absolutely. There's a huge performance if you're doing that, but you can do it. Otherwise, just have multiple bits of hardware for your tests. >> I did have one question about uh bootc direction. >> Yeah. Um, you demonstrated the soft reboot feature for just restarting user space and leaving the running kernel up. >> Are there any plans to get that hooked up into different desktop environments so

they automatically do it when possible? >> I don't know. Um, the um I don't actually know what it's doing now in terms of reboot. I haven't really paid attention. The um it might like like Bluffin might already be doing a soft reboot. I haven't checked. >> I can't I can confirm they are, but it would be really nice if they did add it. >> Um, >> yeah.

>> Um, I would say I would say the person to ask that question to is George Castro who is somewhere here at the conference. Um, so you could still use bind mounts and volumes like any Docker container. That's correct. Yep. >> So that's the answer to what was asked >> about your system. You just want to take some care into what file tree you put those bind

mounts into, >> right? >> Um or use sim links. >> Any other questions? We have time for one or two more, I think. Okay, then please join me in a round of applause for to I'll sit in the back. Check check >> Yep. Think of it like Yu SSI. I hate Google Slides. >> yeah, I might not be able to take the full hour, but let's see.

It really depends quite a bit on the demo also how it goes and how well everything works and we're still one minute early before we get started, but I'll just uh give a few safety announcements uh since we are in potential earthquake territory and hopefully we don't have fires, There are exits both here and at the back where you came in. If somebody in the room gets

really excited over this great presentation and needs the defiill, it turns out it's right on the opposite wall as you go out the door and to my left about 15 ft. Um hopefully we won't that. And if anybody should feel unsafe at any time during the scale conference, uh we take that seriously. Uh there's contact information on signs throughout the venue, but there is one definitely taped

to that back door with uh the contact should you feel unsafe at any time during this conference. >> Could you grab a couple of photos? >> Sure. >> And then share later on. >> Okay. I think it's time. Uh I'd like to welcome you to the cloud native track at uh scale. Um our speaker today is Jussi Numalin who's going to talk about pushing Kubernetes to the

far edge. Uh he is an authority on that subject. He's a senior principal engineer with Morantis uh working on the K0 project. Uh take it away. >> Thanks. Oh, and can you withhold questions to the end preferably and raise your hand and I'll bring the microphone. >> All right. Uh I'm not really sure if I'm the authority of on on these topics, but I know a thing

or two, but you know, Uh yeah so I'm Yi and today I'm going to share some information knowledge tidbits experiences learnings the hard way about how to how to push Kubernetes to the for edge and and and of course that relates to things like IoT AI inferencing and and those sort of things and and I think today I'll I'll actually going to tried to spend quite a

bit of time on on kind of architectural patterns and and how how we can architect our Kubernetes environment so that they fit hopefully better into these far edge and like like setups and of course my clicker died too. Everything is computers are against me today or electronics are against me today. All right. So of course we will first define what is far edge in at least in

the context of this talk. Then couple of words about why why Kubernetes in the first place in this sort of environments architectural patterns and then how how our K0 open source dro fits into the picture. Uh I do want to introduce also to another CNCF project about which is about res resource discovery at the edge. How do we discover what's kind of beyond the edge computing like

IoT sensors and and things like that. I do have an actual demo sample architecture. So, please all pray for the demo gods and hopefully in the end we'll we'll have some Q&A time and and let's see how it goes. Yes, there will be live demo. I do have my Raspberry Pi here with me and let's see how it whether it wants to work. I have a fancy

AI hat on it and the PCI PCI cable actually broke down while travels. So, we have to suffice with the CPU only models for now. yeah, let's let's have a quick look what is what is edge and and what I mean with far edge in the context of this talk. Edge is one of those those kind of hard terms in a way that uh if you talk

to 10 different people about edge, you might actually get 10 different definitions. what is edge? So there's like like it's not like mathematics in a way that that you have one single answer for things. You have different views. Uh but I think if we think about like your traditional centralized data centers, cloud cloud provider environments and then you go to further further away from the internet so

to say uh and towards the actual data source real life data sources like IoT sensors and and and those sort of things. So gradually we move towards the edge of the network in a way and while we while we move from away from the big data centers cloud environments towards the edge our resource con constraints go a lot smaller right you don't have on on edge you

don't have the luxury of spend uh spinning up 100 AWS instances up just for fun, right? Because your your hardware at the edge will be super super constrained. Also, what I've seen in real life is that the software stacks running on those devices will become more and more um I'm looking for the polite word here. Um let's call them weird. So when you go towards like like

IoT environments in in factories, the hardware and software stacks you're going to see are very custom made for that purpose. So you you you cannot take your standard Ubuntu packages and just shove them in in in a PLC controller. It's just not going to work. They are very custommade things both in hardware and software. On the networking side of things, our constraints also go go harder in

a way when we move towards the We're going to have lot lot worse latencies in our network and there's going to be a lot more disruptions in the network. So especially in the networking side of things, nothing in the edge is stable Like for example in my demo setup, I'm connecting the Pi to internet through my OSX and it's very very flake. If you ever done it

with OSX, it's actually very very flaky. But it's kind of a good in the in the context of this talk because you know that's edge and then also uh from the security point of view the the our security posture goes a lot weaker in a way because not only you have to think about your software stack and the security of that security of your communications on all

different levels TLS connection ctions and those sort of things. But we also have to now start start to think about things like actual physical security of of things. So you cannot have your your Raspberry Pi or whatever compute device you use, you cannot have it really physically accessible or if you have it physically accessible, you have to still secure it somehow, And this is where I personally

draw the far edge. So it's really the smart device edge connecting to the to the actual kind of physical world in a way to IoT different like IoT sensors, security cameras and and those sort of So that's my definition of forage. It might differ from other people's definition but that's how I see it personally. So like I said, we're talking about environment where where the infrastructure and

the device Yeah, but I'll >> Okay, I think that actually summarizes my whole day today. So, we're definitely living on the edge. So yeah, we're talking about infrastructure and devices located at the at the furthest edges of the of the network and and more importantly they are positioned as close as possible either to the users of your system or to the data sources where you actually have

have to process the data. So when when we think about those sort of environments, there's there's few different problems that we need to figure out how to solve. Fleet management. How do we manage all those hundreds, thousands, tens of thousands of of different devices? How do we manage our applications? Whatever your application might be, how do you manage those across various different network sites? various different physical

locations and and so How do we observe things both the system at large, individual applications, networking and all that stuff, let's call it stuff. one of one critical aspect is also how do you how do you operate your system? >> sorry about that. >> Check. Check. All right. That's a lot easier this way. Sorry So, how do we operate the system at large? Can you or should

you be able to really operate it remotely or do you actually have to rely on some human person to go on site to fix stuff? So, and of course being good engineers, we need to think of of solutions. How do how do what kind of technologies to use to to solve these problems and why Kubernetes? Well, I guess the obvious answer is that we we're in a

in a cloud native track. So, we all love it so much that we want to push it to any device we can we can get our hands on, But Kubernetes actually gives quite a lot of answers. It doesn't solve all those problems, but it gives quite a lot of answers to many of the problems. We get workload portability. So you can actually use Kubernetes to orchestrate your

workloads across those different hundreds thousands edge devices, right? You have the best thing in Kubernetes which is declarative state management. So you as an operator you declare your state where you want to be and then eventually the edge will implement that state for you. Right? Remember Kubernetes is an eventually consistent system. It might eventuality might be an infinity at points in time but but it is an

uh eventually consistent system and I think one of the biggest drivers today is actually the CNCF landscape. Think of how big and wide the the CNCF landscape is today. So you'll actually find a lot of tools, a lot of best practices already around Kubernetes. If I'd set up edge devices in a very custom way made way, but if I want to do something like service mesh, I'd

have very very tough time to without Kubernetes and containers and all the kind of uh power that the platform gives me. I'm not a fan of service mesh, but that's just one example. And there's like I said there's a lot of practices out there for CI/CD and and those sort of things around Kubernetes. All right, let's dive into some of the architectural patterns that we can use

when when thinking of pushing Kubernetes to the edge. Uh, of course, as as architecture in general, there's no one solution. There's no right or wrong answer on things. So, I'm I'm going to present few different alternatives. There's others. Uh, unfortunately, you have to be the judge what is best for your use case and requirements and operability models. And it really boils down to kind of architectural requirements

too. Whether you want to have like like every single edge site is it going to be the same as as all other sites or are there variance between your edge edge sites? Do you want to run less clusters or more clusters? because managing one or handful of clusters that's easy right that's Sunday afternoon but then thousand plus clusters that's an that's a systems engineering problem already right

and then as as always it really boils down also quite a lot on on the on the fact that what resources do you have available at the edge do you have the luxury of actual servers at the edge sites or are you running like Raspberry Pies? Okay, it's working. It did some beep. Uh or are you running Raspberry Pi or Pi size devices at your edge? Because

that's going to give you a lot of constraints. One very common pattern that I've seen and heard is is to have independent clusters at each edge site, each location. What I mean by that is that that on a site A you have one or more independent clusters. So independent here really meaning that the the cluster is fully located at the edge site. Both the control plane and

the worker plane everything is located at the edge. And I when we're talking about edge I mean physically located at the edge. Of course different sites have different clusters. And now if you scale, you have 100 sites, you end up having 100 and you have 100 control planes to and 100cd clusters to manage at an remote edge So again, I guess you can pick up that that

this would definitely not be my favorite pattern to use. I'm not a fan of HCD. uh the pros of this approach is that uh the clusters are very resilient from networking point of view now because there's no no uh connectivity between the say internet and the edge site I I mean of course there's probably connectivity but from the cluster operations point of view we don't need any

connectivity Our failure domain is very natural now because it's everything is physically located at the site right data is local both the the the Kubernetes state the declarative state is within the site and also the actual real world say IoT data is also local at the site so the cluster is sort of autonomous OS the bad side of things is then the fleet management problem because you'll

end up having hundreds of clusters to manage. You have to figure out things that how do you back up a site? How do you restore a site? How do you build your observability in a way that the site might be without network for days? And if that's the case, how do you operate it? Well, probably the last resort is always to send someone in. But whether that's

possible, obvious concern is also that the resource needs because we're going to run control plane nodes there too which I'm sure you all know they are huge resource hogs. Cube API server and CD together will will eat all your RAM that you have available more or less. Another pattern that that I've seen around is is uh what I call single node clusters. Meaning that a single node

a single Raspberry Pi device is actually both the controller and a worker at the same then each each of your edge site might actually run x number of these setups meaning that each of the devices are really really autonomous in a way. I heard one example of this use case uh where they actually run this sort of setup on cable uh this what is it the what's

the word here settop boxes on on cable so when you have your in your living room living room you have the setup box it's actually might be running a single node Kubernetes cluster for don't have any any actual actual details on that case but I just heard that someone is actually doing it. footprint is of course sort of minimal because everything is in a in a single

single device a single node. uh operationally it's it's super simple because it you you don't have HCD HA problems you don't have to figure out any availability things because it you've designed it to be a single node so it is a single node both in good and bad data is still local operationally it's it's kind of autonomous of course it is a single point of failure because

we designed it to be a We still have to figure out backups per Data storage might be a problem now. And again the observability and operability are bit of a challenge again because if your network is is is broken between you and the edge site, you have zero means to observe anything or operate anything. Third pattern is uh what I call centralized control planes. turns out that

with Kubernetes you don't actually need to run the control plane and a worker plane next to each other. So what we actually can do is we can sort of externalize the control plane fully out of the cluster. What I mean by fully out of the cluster I mean of course worker plane and control plane they are always connected one way or another right but but uh the

control plane node they don't actually have to run any workloads. You don't have to run cubelet on a controller node. You don't have to run containerd on a controller node, which means that you're not going to have to run CNI on a controller node because it's basically it's CNI that forces you to run everything in a kind of flatish What this means is that uh basically my

edge sites will run pure slim worker nodes. now which are as we know a lot easier to manage because there's no CD and other nasty bits involved and you can scale your sites depending on how many workers you need on site A or site B or D or whatever. So some of the pros and cons for this approach is is uh on the positive side the control

plane operability is a lot easier. Now we you can actually operate with the control plane even if the edge site are without a network. I can still talk to the control plane with cubectdl. I can see what was the last known state. I can see some obser observations. What were the last known observations and I can even do cub cube cdl apply I mean it probably won't

have any effect yet but it will take effect next time the site connect connects to the control plane. uh data is still always local to the to the edge sites because the workers are the ones processing the data. So we still have the data locality at Your multicluster user experiences is a lot better in a way because you manage the you can manage the control planes in

in the same exact way regardless of of to which site that control plane belongs to. So let's take a maybe a bit stupid but practical example. You could have an an Amazon EC2 instance running control plane for site A. Now you can basically clone that to run control plane for site B. So me meaning your operational model for the control plane is is lot easier that way.

bad side of things is that the the wide area network is now actually part of our control loop. if my site loses a connection completely, I will not be able to push any changes to the site because cublet is not able to pick up the new data from the from the control might be here, but data storage is always an issue in in Edge, unfortunately. The blast

radius is is kind of a kind of a 50/50 because it's also pro and it's also a con. What I mean by that is that that on a on look let's look at it from the positive side first the blast radius is still kind of natural in a way because if we think about cluster as a logical entity we I have a dedicated cluster for site A

but now I'm separating the control plane and the worker plane so the blast radius is actually kind of mixed in a Okay. And then depending on how you set up the control planes, you might end up having more blast radius. So say I run all my control planes on on AWS US East1. I know that's a bad idea, but if I run it on one region only

and that region goes down as we know it tends to do, I lose all my control planes more or or at least you lose the operability of those control planes. You probably don't really lose lose them as as such, but you get the point. But like I said, architecture is is set of opinions. There's other possibilities or combinations of these. So feel free to mix and match

and and figure out the best way how do you operate your environment. How do we how do we make these or how do we support these sort of patterns in the in the K0 project which is a CNCF CNCF pro sandbox project. we've designed Kzer in a way that uh it has a minimal possible footprint when running on the nodes. I'm not saying that it has a

small footprint. It has a minimal possible because with Kzeros what you get is So there's like our hands are kind of tied a bit there. Uh we try to make the oper operability as easy as possible. We package everything into one single binary file which is pretty simple to operate, right? I drop the binary on my server. I run it and now my server is a Kubernetes

node. I didn't have any dependencies to install. I didn't have to figure out how do I install cubelet package on my DP old super old DPN box or whatever. It's statically compiled. So there's actually zero dependencies. And why that makes oper operability easier? Well, when I have to upgrade my Kubernetes I put new version of the binary in, restart the unit, and boom, my upgrade is done.

course, we all know that upgrade with Kubernetes is actually well, you can do it that way, but you probably would want to drain your nodes and all that fancy stuff, but you can actually do an K0 node upgrade that easily. One of the kind of cool bits especially in these edgelike environments is that uh we've actually built in a a component that what we call autopilot. The

autopilot component can actually upgrade the cluster itself. So it's a component running within the cluster and the it can orchestrate a full cluster upgrade by itself. It does all the things that you would typically do in an upgrade. Upgrades all the controllers first one by one to ensure availability. Then moves on to the worker nodes. Starts to update one node. Drains workload out of it. Replaces the

case binary. Restarts itself and then moves on to the next worker node. And we also have few optimization flags for these single node setups. So you can actually run k0ero's controller d- single which will run a simplified control plane. By simplified I mean we don't runcd in that special case because we don't need for a single node case. So it actually automatically replaces CD with the local

SQL light database in K0 when we started the project we did this uh architectural decision what we call control plane isolation. the fact that I mentioned earlier that Kubernetes doesn't act actually force you to run workloads and cubelets and all that stuff in in control planes. that gives us couple of things. There's no way for me to schedule workloads on controller nodes either on purpose or by

accident. And then it really allows a lot more flexibility on our placement strategies for the controller nodes. In Kubernetes, we do actually need connectivity between control plane and a worker node. So basically when you do things like cubectdlx, cubectdl logs, API server actually calls cublet API to get the data. So we need to still cater for that communication. For that we actually utilize sort of reverse tunnels.

in an example where I have edge site somewhere far away from my control plane, it's actually the edge node calling home calling home to the controller node and opening a a tunnel. Think of it like an reverse SSH tunnel in a way. It's not implemented on SSH. It's conceptually exactly the same. And then when API server needs to connect to cublet, it goes through this So especially

handy for these edge like setups where you probably have layers and layers of firewalls and nets and whatnot in the network. As long as the worker can call home, your Kubernetes cluster works as expected. All right. Let's move on to an practical my sample application is is as sample applications always are simple and stupid. So my goal was is to to to recognize objects from video feed.

Think of like like home security system or or something along those lines. Uh I have my my edge node that I want to connect to a video feed. All the data has to be processed locally and locally only and have an AI model doing the actual object detection also running locally at the edge or AI to make things fun and and to introduce another CNCF project. I'm

I'm in in the demo application or or demo architecture. I'm using a component called or project called ARIR. So ARI really is a is a project for IoT and IoT like device discoveries. So if you think about edge, your your typical use cases are IoT and IoT like use cases. and and what we want to want to do and what we need to do is is kind

of uh connect our Kubernetes cluster to the world around it in a way. So what a actually can do is it can can discover the surrounding devices from your node. Devices like IoT sensors video cameras whether they are USB like like in my setup or whether you have uh a network video feed uh on this uh what's the protocol called ONVIF or or something like that. It

exposes those found devices as Kubernetes resources which means that I can actually use those devices now as scheduling constraints on my pods. So I in my application I can I I I I can tell that that wait a minute this application needs to run on a node which has this sort of device available like you would do for memory or AI or So with ARI you basically

config give it a configuration object. You configure a discovery handler. Say that you want to discover like all the onif video feeds on your network or UDEV devices like in my setup. It'll for every device it finds it actually creates this instance object telling that okay I found this device on this node through this node and it has these properties available. One of the super cool things

with Docker is that it can also automatically deploy brokers. And a broker is basically, as the name implies, something that brokers between your application and the actual device. So in my example, because I'm using USB camera, I'm using the Udev deis discovery. So what a actually does it automatically deploys a broker B pod on the same node. Well I have a single node but on the same

node I have the USB camera attached. And what the broker actually does it gives me a gRPC interface to the frames. So my application doesn't actually have to even know that that how do I access on a USB. I don't know how to actually do it. There's probably some cool library to do it. But now I can just call a standardized gRPC service to get a frame

or it could be some IoT device and then AR. I think for many IoT devices, it actually deploys an MQTT pod to broker the data, which is super cool. So this is basically my sample I have my mother ship cluster. Remember I said that the control plane is externalized and now control plane is not running containers or cubelet containerd or anything. So my control plane is actually

running as a pod in a mother ship cluster. That's actually a cool pattern. and and my team also has a project called Cosmotron to to facilitate that stuff which is also a CNCF project. So control plane running on on cloud. My single site is my Raspberry Pi here connecting to world around it. Well the camera only unfortunately. So, Raspberry Pi, I do have an AI hat on

it. I use the Halo AI hat. Um, unfortunately, I got to mention that uh the all the Halo stuff is unfortunately not open source, which makes it pretty, excuse my French, but pain in the ass to actually set up. Uh, I'm I was running YOLO V8 from the Halo Model Zoo, but the PCI cable decided to go bust on me while my travels. So, I'm actually not

going to be able to run anything on this Halo Hat Halo hat. Now, I do have a sample application running on plain CPU, but you can imagine that it's it's not that fast, right? But you'll get the point of the the architecture and and stuff. Anyways, basically my application running in pod is talking to the ACR broker to ch uh getting the frames from the USB camera

and then it feeds it to the YOLO model for object detection and then shows us the output. Is the font big enough? Gypsy delegate node. So, single Raspberry Pi as expected. And don't worry, I'm not recording anything. So, get configuration. So, this is the ARI So what I'm basically telling here is that uh we use the UDEV discovery handler and then I'm basically telling the rules that

okay look for video devices on the kernel and that's about in the configuration I already tell that if you find devices on a node deploy this broker pod on the same nodes so that I get the fancy gRPC automated interface for me automatically. And now for the actual let me let me put it up there. So AR found an instance of a device using the UDEV configuration

on a node Raspberry Pi 5 local 3 hours ago. Wait a minute. I I booted it up like less than an hour ago. Maybe it has maybe it had the object already in the API probably. Okay. And now for the actual demo application, I have a deployment detector demo scale deploy detector demo. Oh, replicas one. I'm not running it all the time because, you know, I don't

have a even a fan on my Pi. So, it would probably turn down And to be able to access it, I did already expose the service as a node port service. cubecdl get service and the port is what 31. This is act uh when I I mean I'm I'm by all means I'm no expert on any AI stuff or models or whatnot but when I started to

put this demo application together I got actually surprised in a way that uh that just with Raspberry Pi and a I can actually get a decent frame rate. I mean, it's not like you would do any gaming with that frame rate or anything, but but there's probably a lot of use cases where that's actually enough. And my application is probably doing a lot of stupid things, too.

So, you could probably optimize the hell out of it and get lot more frames. Uh when I did the Halo AI hat version of the application, I was able to get like 20 frames per So it's actually a quite a big increase just by adding an a a dedicated processor for the for the AI model But of course, your mileage varies depending on what you're actually doing.

What kind of models are you using and how do you do the inferencing uh mine has been working with a a nonprofit organization where they've taken this ar same exact same architectural path pattern to the extreme. So they've actually developed this underwater buoyu of sorts which is analyzing the um the the the natural habitat of of coral reefs. So basically the the environmental conditions on on coral

reefs. It's actually running Raspberry Pi and it the the Pi is a single node single worker node connected to control plane running somewhere else. There's a QR code to the much longer blog post going into the details what they actually do and and and and so on. But I think that as an example of edge deployment, that's kind of like like you can't really go much further

than that. I mean, if you go further than that, then we're out of the planet already probably, right? But that's super cool use case. So as a summary, there's a lot of use cases on edge to AI workloads. Think of like smart cities, industrial automation, retail stores, and even underwater for things. Kubernetes can work in these sort of setups and architectures. But you really have to think

what's your operational model. Do you want to separate the control plane worker plane? Do you want to run single node clusters at your edge? Do you want to do some even more fancy stretch setup or what fits your use case? with K0 as an ARY. You saw neat nifty simpleish architecture. I think we've got time for some questions. Please raise your hand if you have one. If

you if you have the centralized control plane, does Cublet uh control uh call back? So if you were trying to uh uh save resources, could you have Cublet decide, I only want to touch base once every hour? >> No. uh cubullet and and and pretty much all of the Kubernetes components that are that that need to be in sync with the state, they're all implemented using using

the watch concept. So basically, it's like a sort of a call back. So whatever when whenever something changes on on the API server, it's immediate call back to cublet at least. Yeah, I'm I'm pretty sure that you cannot configure Cublet to do polling >> Yeah. Hi, I was wondering um you said that you couldn't run uh a control plane in AWS for Kubernetes uh in multiple regions,

but can you make it redundant? So, of course, you you you can run your on uh multiple regions. You can do it. Yes. But the problem is that the region to region traffic gets a bit slow. And in Kubernetes when we when when we talk about highly available control plane we mean we need to have ancd cluster and CD is super picky about the latencies between the

nodes. So multi-reion I'd say no I haven't actually tested but I'd say that that it probably won't work. Multi-AZ definitely works that I If you don't want in multiple regions of the control plane, can you at least use like uh different nodes within one region in AD us >> to make it a little more redundant? >> Yeah, you can. You definitely can. You and can do and

and should do multi-AZ for your control planes. Absolutely. One thing I actually forgot to mention mention or or actually add to the slides uh because a lot many times I get the question that that what about latencies between the even if the network is is kind of up between the controller and a worker. What about latencies? I've been actually testing and I'm I'm working on writing a

blog post uh testing that that how how bad I or or my kind of idea or premise for the blog is that how bad can I make the network before cubelet starts to throw up. Basically, so far I've been able to make one and a half second latency and 50% packet loss. and Cublet is actually still happy. I mean, yeah, of course it does mean delays from

your cubecdl applies to when actually things boot up, but does it matter? In most cases, no. But I was hoping to get the blog up to this event, but but you know, time and time and life happens. uh you said the latency uh for SCD between regions, how much of a latency can you have with CD? So CDC typically needs uh I don't remember they they have

they have official documentation about the the the latencies and stuff but but I I I always say that your typical node to node latency bitcd is max like like 20 milliseconds or something >> 20 milliseconds. >> Okay. Thank you. I think we have time for one more if anybody's got a question. Okay. Could you join me in a round of applause for this presentation from >> Thank

you. Thank you. >> It's pretty good timing. >> If you want a picture of yourself on the podium, I'll take it for you if you want. my friend. >> Thanks again. Good. >> Sorry about your >> go. It's weird because if I if I put the PCI cable in, >> it won't boot up at all. >> It's It might be shorting something in that connector, but >>

my guess is it may be fixable, but not in a hotel room with no tools. >> Yeah. Exactly. And I think I mean how much would these cost over like >> is it a replaceable cable on both ends? >> It's really like if I get this from demo or something probably like 20 cents. >> How much should those halos cost? >> It's about I paid like 20

which I'm running frig with image but classification at home but using the coral. >> I've heard these halos are better I mean it's it's pretty okay. I mean, with a setup like this, if I when I can get like 20 40 frames per second, it's pretty okay. And I could probably go a lot better than that by fine tuning some of the application and and and testing

the various different models and their >> parameters. Most of those most of those image recognition use cases though, you don't even need the 20 frames a second, >> Exactly. >> Really four or five work. So you don't probably even need the AI, >> Which actually means that uh I can then I can power this with the power bank. >> You know, I think the image use cases

that kind of need the accelerators. It's one thing to recognize the person, but then if you want to compare it against an inventory of people or if you wanted to do OCR like >> you're capturing um car license plates, but you want to actually OCR those to get those into >> um something because because you don't really want to send the raw picture to be actionable, right?

You you really from edge you if it's a car, you want to deduce it to the license If it's a person, you'd like to have that mapping to an a name >> or a recognized individual occur out at >> And then I think that's where you probably need the accelerator because that you're doing it's not yolo anymore. It's yolo to get the fact that it's a person,

but then it's a second pass to get the >> Exactly. Exactly. But then then comes the the architecture again in the picture that that that you know you probably want to do that sort of lightweight detection at the edge. >> And then when you recognize that okay this is a car >> that was speeding or something then send that frame out to cloud for actual processing. Maybe

there's a lot where maybe you don't, but >> I mean, if it's a privacy issue, you might actually still be procluded from sending it out to cloud. >> That's true. That's true. Yeah, >> that's true. But you can do a lot with these small devices nowadays. >> Yeah. And it's getting better. I mean, the interesting thing I'm in the CNCF, I'm co-chair of the uh edge working

group >> and the interesting thing that >> okay it's sort of dying off because a lot of that has gone into AI centric uh because AI has justcliped the interest level of >> eating anything and everything now >> yeah but all of the people making embedded processors now and I do mean all of them they're bundling it with inference engines they're not like general purpose AI accelerators,

but the fact is >> you don't you don't and shouldn't be doing training on the P. You should only be doing, >> you know, >> it's it's inferencing always. And >> I've never heard a use case where people >> Yeah. I mean, >> you only do that once. >> Well, in fact, >> you take the penalty of moving all the data, but you don't even have to

do it once really as long as you have the bandwidth to move some of these things up to the cloud to run it there where >> because the fact is it's going to be way more efficient to do it with rented GPUs because the training is typically, >> you know, something. Well, it's a temporary step or maybe you repeat it once a year and you don't need

to sink all that capital into enough to do training when you're only going to do it incidentally >> and but the fact of doing it in a cloud does make sense for training because if you've got a distributed edge scenario like say I've got traffic cameras out at 10,000 locations, you obviously can't you can't train it effectively at any one of them because they don't even have

the data to mask. Uh you need to move that up somewhere, but you're in no great hurry either. So you can you can kind of opportunistically u manage to collect that data, do the training, get improved inference models that you then push back down to the edge. I think that's the way it should. That's the way people are doing it. >> Did you see the I know

about >> It's a cool >> It's a cool thing. It's It's kind of kind of one of those things on my to-do list to get my hands on, but that the problem in my life is that that list only gets longer. >> Yeah. As a general principle, I think is one way to take advantage edge really for practical implementations, you know, for redundancy and resiliency, you probably

already need to keep out towards the edge too because you know if you if you can't do that >> use other presentations at scale. There was one a couple years ago with the actual container containers on top of the target. There's a whole concept on the Xeon processor. So theoretically you have a cloud provider that's even short of doing things to get signals off of after it's

been >> well actually consumes your decryption point that becomes a lot more secure anyway I I wasn't advocating that as a So far, nobody has these unclaimed chips. It would be great on the edge given that it's an insecure. >> Well, there's probably a market for it given that one of the unsolved problems of edge is there's no physical security. So, you put these things out there.

>> People can either steal it or worse. A colleague of mine is working on a on a demo where run basically same setup that I had, but the node is actually running in the fully disc. So everything out of memory, >> which means that that if someone goes in and takes my device, they have to unplug it, which means that everything is lost. >> there is a

solution to that. >> Contractor A lot of these things with finance involved steal completely stealing device isn't actually your worst case. It's actually tampering with it and leaving it in place. >> Well, thank you very much. I look forward to having the smartest system. >> Really? My my my goal for the past year and a half has been to build kind of on top of this architecture

build a a kind of a sort of a security on the other hand nature watching system on my on my summer cabin. >> Or or general animals that come to the the property and sort of things. Basically, the idea has been to use the and other models locally to pick up whether there's any like activity in general. >> And then it >> Yeah. Yeah. Then then then

do do like do do it in stages in a way that that first detection is like whether there activity of sorts and then second one would be to to actually figure out which animal was. Then if it's a bird then third model specialized in in kinds of birds that what kind of bird >> so I got I went down the path I've had the frig running at

home for security it's a different use case than animals it worked. I'm happy with it. It's still there but then you know decided I need cameras anyway. There was some sale on those TVos and they have a line that's all solar charged. One of the with it at these places is, >> you know, running wire and not just the signal wiring because you could arguably use Wi-Fi,

>> but you still have to power it, but those actually have a little solar panel and they actually go into a deep sleep. So, they actually use kind of a conventional motion detection. >> They don't say what it is, but I suspect it's kind of like infrared or something. In fact, it must be because it works even at night >> and then it it wakes up only

when something is there. >> And it actually seems quite reliable. I mean, it it's detecting smaller animals like it actually detected once a spider that was crawling across the lens arguably, but >> um it's clearly detects dogs and cats. It was a big one because it's uh Southern California. So yeah, like they they literally have tarantulas here. >> Uh but anyway, they're called Topos and I got

it on some sale, but they were it was less than $100 a camera with the battery and the solar panel and >> cheap. >> And I I thought when I was at that price point it would be a piece of junk, but it wasn't. Then I went and bought two more of them. >> I mean, I'm even I was even thinking now this is in Southern California,

too, where I have the luxury of lots of sun. So, I'm going, you know, these connect back and that's actually a big power right there. and just getting enough periwive but it's it's now been up for over six months and through the winter they work. >> So when you plug your Raspberry. >> Yes, >> it's work or not. If I now >> I now look at that.

>> Oh, I get it. It might. >> Yeah. If I if I now go QC at nodes, I would see reply not ready. >> But once it wakes up, i.e. plug it back in, it connects. >> it pulls up the last state that it needs to run. And >> it's how Kubernetes is designed. And then if you think about what what happens when when it's if it

would be now plugged in, I mean in in power and if I just yank the network cable, what happens? keeps on running the last known state until it knows anything else. >> Yeah, some of the people brought up the issue of you know like these stretch clusters over large domains with big network latencies but actually I think once you start going there try to use Kubernetes to

manage edge at scale I think the bigger problem the scale up that you know getting up there to literally thousands of nodes which is not unusual in edge especially 10,000 that's that's kind of beyond the initial early days of Kubernetes just didn't imagine you would take a Kubernetes cluster that big >> and I can't say that I've kept tabs but it is quite a challenge to where

one of the aspects of those gigantic counts is that you need a bigger and bigger SC installation and it gets really prohibitive. But I've heard of places that tried to go there and ended up with AWS bills just for hosting that were in the hundreds of thousands of dollars a year I want to say Google but not anyway. a blog post where some group of people took

it to the extreme and they had like they have like 100,000 >> but they had to do very very nasty tricks to the >> Yeah. And like I say, the cost of keeping that cluster is so enormous. You know, your fallback then is you don't manage the whole world and break it into regions and that might work for some >> Uh but you know some of these

edge applications like whole automotive fleets like BMW wants every car in the that's going to be millions millions of those if you went kind of this It doesn't. It doesn't. >> That's why I have rubber bands. >> And that's why probably the cable broke >> 3D printer. >> Yeah, I think the people I don't think it's getting traction but it's an interesting concept where it's a hierarchy

of Kubernetes clusters and then it's one of the reasons that it's so costly at scale is you're sending massive amounts of stuff they are the way Kubernetes works is you have a declarative state of how you want to be at the upper tier cluster And then at the lower ends they can be as small as single clusters but they're not really standard things at that topmost cluster.

It's just keeping a blob that it doesn't know about that's been compressed. Like imagine it's not literally PK zip. >> But if you took if you took the YAML specification or something and just squeezed it down to where it's as dense as carbon diamonds. That's what it's like. >> And it just knows that this is actually owned by my child cluster. It's going to >> be the

authoritative uh thing in charge of action that's on this and you just push it down and you you have the illusion of centralized control because you can define all these things up >> but the things are tagged as this isn't really mine. So just compress this down and kick it down to the next level. And it's interesting concept and it's at this point that project's been out

there a couple years. Um, and I I I actually like the architecture of it. It makes sense for a solution to that, but it's >> yeah, I I I do like the architecture in a way, but then again, it's kind of a blunt way of doing things also in a sense. >> I've seen also people use virtual cubelets in this space. >> Yeah. So your virtual one

one cublet is actually controlling a fleet of devices behind it as >> where which kubernetes is as one node in a way that but there's like I mentioned it's architecture so there's possibilities opinions pros and cons and >> yeah there I think it's clear there's no centrally recognized right way to do this but It is something that someday will get solved because there's enough demand for it

>> and it's only going to get worse as AI moves into the physical world. >> Well, the whole AI thing is going to actually create demand for comput and not just the compute like the images and whatever the software delivery but also the feeds of actual data back. It's just like you captured that with the camera, but like you and I were discussing, you're not going to

send frame by frame images for analysis up in the cloud. You actually need to compress that data down to small amounts and basically turn it into events and maybe events with supplemental things like if the event was somebody broke a window and I have a picture, you know, the event is window broken and then you attach the picture to it and Thank you very much. >> Yeah,

thank you. Yeah, they don't. Okay, I didn't see that. Maybe I'll walk you There's something Test test test. Hello Hello everyone. Our next speaker John uh will be discussing Apple's recently uh open-source containerization framework uh which allows you to use containers in interesting way on your Mac. Uh John is a software architect and a developer at Apple. um a versatile software um architect with a broad background

and spanning web, mobile and embedded systems and enterprise storage and uh with a passion for elegant software. So John Logan silence my phone. Hi everyone. Um see before we get started for real um I have the GitHub site up here. Um, if you want to uh play around with container, uh, I'll shortcut that and you just go to the release page and we just released uh 0.10

last week. And if you scroll down below all the text, you'll see a signed installer package there. It's like an Apple package installer package. You can download that, doubleclick it. Uh, you'll be prompted for your admin password because it has to put something under user local bin. Uh, but other than that, you should be good to go. and and there's uh there's a couple of pages that

you can um try out if my computer will let me. So there's here we've got a docs uh directory and there is a little bit of markdown. The ones that are probably worth doing first are the tutorial and the how-to. And then of course the read me on the front page has a lot of the getting yourself up and running stuff. So, with those preliminaries out of

the way, um, we ready to go. Cool. All right, let's do this. See if I know how to run my Mac. Do that. Do that. Good afternoon. Uh, I'm John Logan. I'm a developer on Apple's containerization team and I'm happy to be here today to talk to you about our open-source containerization projects. Um, last June we opensourced uh the Apple containerization framework and it allows you to

build, run, and publish Linux container images on your Mac. And we did this with an approach that focuses on security and privacy. There are two key projects. Um there's container which is the tool I just kind of talked to you about installing that right. It's a tool written in Swift and it's for managing containers and if you've ever run any other tools like that it's going to

function very similarly to those sorts of tools. It'll feel pretty familiar. Um and then there's also containerization and that's the Swift library underneath that the container tool uses. Um the slightly awkward title to this talk alludes to the role of this containerization library. Um it's the basis for container, but you can also use it yourself to embed Linux containers in any Swift application that you write. Um

so here's what we'll cover today. Uh first I'm going to set the stage with a brief overview of containers. Um, and after that we'll cover the design principles that we used as we built the containerization library. Um, I'll demonstrate how you can add a subcomand to the container tool, how to extend it, as well as look at an application that, like I said, was built straight on

top of containerization. So, you're not running Podman or Docker or our tool. You've got it embedded in your application. Um, I'll talk a little bit about why and how we did it in Swift, and finally wrap up with a word about the open-source community, how you can get involved, and a little bit about our goals and plans. Oops, there we go. Okay. And I didn't do the

buildup. Okay. Let's start with what are containers? Um, we'll start by framing what we mean by a Linux container. And there are a lot of ways to come at this. Uh, some of them are more pedantic or technical than others. I'm going to look at it from two perspectives. Um, a runtime perspective where we're looking at containers as a way of virtualizing applications so that they run

securely and run privately. Uh, and then a second sort of view of it is as a packaging construct. It's a development workflow. containers as a development workflow for publishing and deploying your apps anywhere. Now, a key property of a container runtime is it virtualizes the application. It makes the application think like it's running on its own private hardware. And we say that this isolates the application from

the underlying host. And it's done in a way so that it allows the application generally to run at full speed. But anything that could escape that container has to go through a checkpoint. Sort of a fuzzy definition. Um but this virtualization also implies that the um that the applications are isolated from each other and together this means that applications run securely and that containers can't interfere with

each other or with the host in any way. and also that they run privately and that means the containers can't see what they shouldn't see in another container or on the host. Um, both of these properties are really crucial when you're deploying at scale and especially crucial when you're deploying in a multi-tenant environment where you have customers uh different customers that might be running on the same

hardware. It's also important for local development though because containers allow we developers to rapidly create, run and delete all kinds of dev environments uh without impact to your base system and without conflicting with Okay. And the last thing that I want to think about when it comes to the runtime is performance. It's a final consideration for a developer a container runtime. As a developer, we don't want

to be waiting all day for commands to complete. So, our container tool needed to start up and shut down containers efficiently and responsibly. And I'll talk a little bit about that in just a sec. So, as far as packaging is concerned, um let's talk about how we package and deploy applications. Uh let's imagine a serverside app that we could build directly on our host. Um say a

rest service that talks to a Postgress database. what's needed to run that app? Um, well, there's the application itself, like the code that you wrote, and that's often compiled in some way. And then there's everything that your app depends on. Uh, there's libraries. Uh, they might be compiled from source dependencies that have been pulled in, or they could be linked in statically or dynamically. You might need

a language runtime depending on your language. And you're probably going to need some data, content, assets, those sorts of things. If we're talking to a database, we might need an init script, for example. Now, if you're developing all this right on your Mac, you might have to install some thirdparty apps or on an Apple system, you might have to install some homebrew packages. Um, and you can

put some of that setup and make files or scripts or uh eventually inconsistent readme files. Uh but but the problem underneath all that is how do you make it so that anyone can build your app and even if they're not running the same type of OS or computer system as you? Well, containers give you a way to package all of that as a standard container image uh

created with a set of host independent instructions. So what you do is you make a Docker file and that Docker file pulls in a base image, installs some dependencies, builds and installs your app and your your data and assets and you might give it a command for it to run and an environment for it to run in and that defines your container image. From there you can

build that image and test it and publish it on a registry once it's working so that it can run as a container anywhere. In a nutshell, that's what containers are all about. It's running your applications in secure private isolation. And it's also an easy way to package and deploy applications to to any kind of platform. Okay. So before we move on to the containerization library, I thought

I'd uh just show the tool in action just a little bit. It's going to be a really simple demo. Um see if I can get this pivot done right. How's that? Is that Is that Do I need to font up or is it It's all right. >> Boas. Yeah. Okay, we can do that. There's going to be some wrapping at this point. Oh, wait. That's No, hang

on. okay. I work for Apple and I don't know how to make a font bigger. What's that? >> Sorry, I missed that. Oh, there it is. Okay. Oh, what's command plus? I thought I was doing We're pretty big there. Okay. How's that? Good and back. Awesome. Okay. Yeah, I thought I made it large enough, but maybe not quite. Okay. So I kind of have all the commands

we're going to run here, but um so started today I'm not running container right now. So I can run container if I can type it container system start. And what this is going to do is it's going to bring up a bunch of XPC services that get containers running. You see it starts pretty quick. And from there you've got pretty much you know you're managing resources, right?

So container ls. Oh, I have a thing already running. So we'll Stop web had one uh left over. There we go. Takes a second to shut down. Okay. So, you can see I can list a container. I brought my image down already so that we don't have to worry about the network. Um I have a Okay, I have a volume. We're not going to use it. Um

and then you can create one or more networks. You get a default network here. You can see it's uh on this subnet. And then there are a couple other things. We keep track of registries for you. Oops. Let me just use search Okay. And one interesting one is we allow you to create local DNS domain. So we run like a little local DNS server. And you can

see I've created kind of the standard uh local domain test. And this is you'll be able to see this in use here in just a sec. So those are all our resources resources. And what I've got here if if I look in my HTML directory is just a simple um there we go. So you can see just simple HTML file, nothing fancy. Okay. And then we run

a container. So let's go through this command. It's a lot like a docker command or podman command or or kale lima. Uh container run dashd runs it in demon mode. So it's going to run in the background. rm is going to get rid of it after I'm done. I'm naming it web and that's going to be important because it's going to apply my default domain test to

it. So it'll have a address record of web.est. Um my working directory is user shareh. you know, we're mounting in my HTML, my local HTML directory into user share HTML in the container, running Python Slim, just a lightweight Python image and starting up a de dev web server. And one other thing is I'm publishing it on port 80. We'll talk about why I can do that in

just a little bit. Okay. And there you go. And if I go open, should automatically bring over. And there you go. Okay. Yeah. So, oops, that's not what I wanted to do. Let's do that. Okay. So, let's go back Yeah. So, you can see it's it's pretty familiar, right? If you've worked with these sort of tools before, um let's move on and talk about the highle design

principles that informed the containerization library or informed our effort to develop it. Um we're going to first show you the traditional approach to running Linux containers on Mac OS. It's it's a little different than running Linux containers on Linux and then we'll contrast that with Now to start with I'm gonna we're going to go back to running bare metal running applications directly on Mac OS. Um say

these three applications are an engineext web server some flask middleware and a Postgress uh Postgress database. Um now those of you are really familiar with Mac OS. I'm talking about unsandbox Mac OS apps. uh Mac OS sandboxing gives you isolation, but it's pretty much another talk entirely. We're talking about what sort of isolation exists between regular old Mac OS processes like these. Okay. Um you get virtual

addressing and you have a user kernel split like any modern operating system and that helps keep you from blowing up your system and blowing up the processes next to you. Um, but amongst everything else you run, uh, with your own user ID, it's sort of a free-for-all. Okay? I mean, your web server can enumerate every other process on the system. It has access to any file that

you own. And if you started it up with say like the Mac OS terminal app that ships with Mac OS, it can access everything on your local network. Okay. So moving from there, we'll look at just a single container and examine the isolation between the container and the host for now. Okay. Now to run a container on Mac OS, a Linux container, you need a virtual machine

and a virtualization framework uh that Apple offers is is happy to dispense you one. Um once you have that Linux virtual machine running, there's a hypervisor that sits in between the VM and Mac OS and it's there to basically let the app run at full speed except when privileged operations are are um involved and then it acts kind of as a gatekeeper. It keeps things safe with

respect to the host. Okay. And what your container can potentially access well you control that. Okay. You control that with your VM config. Okay. So like storage, network, CPU, memory, you get to choose all of those things. Uh and those are the only things that your VM sees. So that's how we isolate from the host. Let's now consider containers next to their neighbors and containers and that

kernel underneath and the the basically the system underneath the containers. How do we isolate there? Okay, we use a combination of Linux technologies. I say we, this is any modern container runtime. And they work together to restrict what a container can see and use. Um, there's several of them and I'm not going to get into all of them. Most notably though, we're using namespaces to determine what

a container has, what it can see, and we use croups to implement quotas over the resources that it can use like CPU and memory. And that's how traditional container management works on Mac OS. It runs multiple containers in a single large VM. Okay. With this approach, what you have to do is at the start of day, you have to choose the resources. You have to configure that

VM and and pick the amount of most importantly memory uh that all of your possible containers will possibly use overall time. Now, your containers get to share this common resource pool. And the same thing goes for your host data. Okay? So you need to configure the VM to share everything. You pretty much have to share the world from your host into this VM because you don't know

what you're going to share with your container. And then when you create a container at that point like that mount instruction I had on mine, the volume instruction that goes from the VM into the container. Okay. But you basically you've got the all of your data, all your host data shared in that Okay. And that brings us to containerization. Uh, like I said, its design emphasizes security

and privacy. And the main thing we do for security that's different from what you saw is that each container runs in its own VM. And that VM, we've we've really crafted it to have a minimal attack surface. And for privacy, it's basically that you don't need to mount the world into that VM. Every container that you have sees only that every VM only gets the data that

it absolutely needs and no more. Okay. And then remember we were talking about performance and and you might be thinking well starting and stopping VMs that's sort of an expensive operation, right? And it's something we had to consider. We worked really hard to make the performance more container-like than VM like okay so let's start with that traditional approach and this is what we do. Okay. We're just

VM for every container that we run. Now each of these VMs is really lightweight. uh you get sub-second container launch times that you saw it started up pretty quick. Okay, the other thing that you saw was that each container gets its own IP address on the virtual bridge. You might have seen that the when I listed the containers like 192 168 643 or something like that. Okay.

And that means that you don't have to publish ports. You don't have to use that d-publish that you would use with other things to send things over to your local host. you're not running a proxy or setting up IP tables, rules, or anything like that. Um, you're getting direct access to it. And that's why you can mount a privilege port like port 80 on the container because

you're not mounting it on your local system. That would be prohibited unless you're running as root. Okay? And then when you're sharing directories and files, u you share them only with the container that needs them. Now, since each container is a VM, you have to specify for each container the amount of CPU and memory that you're going to use when you run it or or you let

it pick a default which may or may not suit you. Okay, so you have to do that. But the other thing that it means is that when no containers are running, you're not using any CPU or memory. You're not committing all that to some big VM. Okay, so I talked a little bit about or I referred to a lightweight VM. So what does that containerization library that

we have, what does it do to make a VM lightweight? Um, and we'll start by looking at devices. Uh, containerization configures its VMs to use a bare minimum of the devices the container needs. Really only the devices the container needs. In in something like this, we don't need a chipset or physical peripherals, okay? We just need a couple pair of virtualized devices. One thing we need is

a a virtual socket, a VSOC, and that's going to facilitate communication between the host and the guest. Um, we're going to need a block device to mount the container image into the container. Uh, and we would need block devices also if we have named volumes. Aside from that, uh, if we are attaching it to a network, we need network interfaces, a virtual network interface. And similarly if

we've got host shares there's devices for that uh like verios shares. Um so and as I mentioned before containerization of the library is going to allow you to pass down the parameters to configure CPUs So we've got this lightweight configuration. What do we do with it? What happens when we launch it? Um containerization sets up the VM to direct boot the kernel. Okay. what we're doing. We

don't need firmware. We don't need a boot loader. And that saves on boot time and it also reduces the number of moving parts that we have to maintain and keep track of. Okay. The kernel itself has a minimal minimal kernel config. Um this both shrinks the system footprint footprint. I'm going to stick with that. Uh the footprint and uh reduces the boot time. So going from reset

to user space being ready is is really fast. Okay. And then for security, um, that minimal config also means that you've got a very small attack surface or a smaller attack surface. So let's say we get to user space, let's look at the initial file system that we're working with. And and here I'm not talking about the container image. I'm talking about our image that's running the

the the supervisor, right? That's supervising all this. Okay. Um, you know, with a typical VM, it's going to be pretty fully loaded. Uh, it's going to have a standard C library. It's going to have dynamic libraries to make sure all the wonderful apps you have run properly. And you're going to have at the very least of those apps, you're going to have some core utilities like CPLs

PS. Okay. So, as with the kernel uh containerization, we're trying to strip all that back, minimize, make faster, right? Make the attack surface smaller. Okay? We don't need core utilities to manage a container workload. So, those can go overboard. Okay. And we'll just say no to dynamic libraries. Okay. And even the standalone standalone lib C, we really don't need it for what we're doing. So what are

we left with? Well, container containerization manages everything through a guest agent process and it's purpose-built for just this one job. Okay. The name of that exe of that executable is VMinit D and it's written in Swift and we cross-co compile it on Mac OS over to Linux and and Swift gives us this Linux static muscle SDK for doing this so we can produce static Linux binary that

we'll then package into an initi image let's look at what VMD does before during and after we run the container process the the primary workload I'll Okay. Okay. The first thing it needs to do is sort the network. Okay. It needs to configure the network interfaces. What do you have to do? Uh you have to bring up links. You have to assign IPv4 addresses. And you need

to configure routes. And then there are a couple other things you might want to do like set a host name, configure an Etsy host file, and probably a resolver comp if you want to resolve names. Okay. The other thing we have to do before we start that primary workload is get the container's image mounted. Okay, mount the root file system for the container itself. So we do

that and also we do the same for any named volumes and any host file system shares. Okay, now at this point we can actually launch that primary workload and it we pretty much go into process supervision mode. Okay, so we're just kind of watching things and the other thing that the user could be doing is running execs and so BMI minuted will launch any exec processes and

then it just track tracks everything. When a process exits, it reports it back to the host. Okay. And when the primary workload exits, it shuts down the container. Now the client of VMinad running on your host gets to orchestrate all this. And so BMI minuted presents a gRPC API for doing that. And the final note, it's not in the slides, but all of this stuff that BMID

doing, it's is doing, it's doing it like it would with a traditional container runtime. So it's using namespaces, it's using croups, it's using pivot route, it's using all those things. And that's containerization in a nutshell. Um so we'll wrap up this section with a review of those three focus areas of security, privacy and Okay so um our minimal kernel gives us security and boot performance. Okay, direct

booting that kernel helps with speed up, gets it up and running quicker. Um, VMinutd uh minimizes the runtime attack surface and it also is really lightweight so it makes it can get that workload up and running really quickly. And then the main thing of running these in individual VMs helps across the board. You get stronger isolation, stricter access to host data and more res uh efficient resource

utilization. Okay, so um let's move to the code. But I'd like to walk you through some ways to play around with container and the containerization code. Um the first thing I'll do is describe how you can easily extend the container tool by adding subcomands. Um we give you a simple plug-in mechanism. It's pretty rudimentary and it allows you to add these features, add subcomands without modifying or

building container. And that's the the CLI plugins that you see there. And then there's a second type of plugin uh that we call demon plugins and that allows you to run background services. They could be uh XPC, you could provide network access uh network API, whatever. Uh but you can use those to work with the other services that we run to add functionality. Okay, we use the

service plugins and container for almost all of our key system components. The one that you would see that isn't that way is the main API server. uh we're not doing it as a plugin yet. And creating a plugin, well, it's it's mainly a packaging exercise. I mean, the programming of it is just create a binary, right? I mean, you don't even need to use Swift. You just

need to do something that handles commands in a conventional way. Okay, so for us, I'm talking about a CLI plugin. So, for that CLI plugin, you need that and you just need a little bit of metadata JSON. Okay. And those two files, you would install them in a conventional location relative to the install route of container. And then when you start container, it's going to pick it

up or when you run the container command, it's going to pick up the CLI plugins. Okay. So, let's move on and look at a Mac OS application that embeds containers directly inside of it instead of going through the container tool. So the containerization library, it implements all of the core logic you need to work with containers and container images. Um some of the libraries you see here,

they include uh OS abstractions and IO abstractions. um uh being able to work with Netlink uh to configure network interfaces and we even have some code for creating X4 file systems on Mac OS and we use that to to snapshot our images effectively. Okay. And as you might expect there's a sort of layering that happens here of those APIs. At the bottom, we've got our kind of

our low-level system programming stuff and and like our primitives for OCI image manipulation. Above that, we've got our core types for virtual machine process control um IO relay. And then on top of that are our really high level abstractions for uh container and uh image management. Okay, remember how I mentioned that on that last VM minuted slide that we use all the same Linux uh containerization techniques

like namespaces and croups and whatnot. Um you can see that at work in that Linux pod construct that we have at the top. Uh basically we're using that we can use that or you can use that to create containers that share resources uh like a coupe pod. Okay. So, um I've got one more demo and this one we can just walk through. There's a container example in

the containerization repo that you can look at. Uh and I was just going to scroll through that here. So, let me get to that. I hope this doesn't wrap too much. Um let's see. Well, I can project this. There we go. So, I'm just going to go down. So, if you can see, I'm in the containerization repo. So, I'm going to go down to examples, and then

there's like a CTR example here. There's a readme file and all that. Um, and some basic instructions. The one thing I've done already is run a make target to pull a kernel down here. And we should see that there's, uh, yeah, you see there's a VM Linux here, right? So, uh, let me see if I can just search and get that. Oops. Okay. So, we go into

sources and there's just one file. So, we're talking tons of lines of code here. Um, our fancy header. Okay. We do a few imports. We start by setting up setting up our terminal like the terminal that we're running in. We want to put it in raw mode. I should probably tell you why is that we're just going to run an Alpine a shell in Alpine for this.

It's a very basic example. So we're going to put our terminal in raw mode so we can attach that shell and it won't behave like a child. Um after that we need to pick an init file system. So we do publish on GHCR these init file systems that has the VM innit D in it and it has one other thing in it called VM exec and it

kind of works like crun. So Vminet D actually delegates the starting of your containers to VM exec. Okay. And vmanit is this OCI image that that has just this. Okay. And we're going to pick our kernel which you saw. At this point, we're going to start that super high level at top of the stack container management container manager instance. And we're just giving it the things that

that we talked about and we're telling it to create a a simple And then it's just some other administrative stuff. We're going to set a container ID. We have to pick our Alpine image and we tell the manager to create uh using the container ID and some basic stuff and then we get a config back in this in this block and we do the rest of our

config. As you can see CPUs, memory, we're passing it in the terminal to attach to and we're telling it what to run. We're going to run binsh in in root. we'll clean up at the end. And then uh this bit the container create is what bootstraps the VM. So at this point we'll have the VM running and then we do the await container start and that's going

to fire up our workload. Okay. And we resize our terminal and just wait for the container to finish and at the end we'll stop the container and the deferral will hit us there. Okay. I can show you. I mean, um, yeah, let's do bat make file just so you can see there's not too much in the make file. Um, but the build target, you can see we're

just doing a swip build. Uh, we're doing a code sign with some basic uh that it's basically we need to attach a a security entitlement for virtualization. It's one that anybody can put on and that allows us to create VMs. And then we're just copying the binary up and that's it. So, we'll give it a shot. Hopefully it won't take too long to >> What's that? >>

Um, actually, I'm we're gonna I hope I finish early. I'm not going to do any on mic questions, but I'm happy to talk to y'all and get you set up or whatever. >> I'm sorry, I can't hear you. >> Oh, there you go. Thanks. No, we got Yeah. So, here we are. We got our alpine Uh let's try that again. All right. Um a word on Swift.

Okay. Um, let's see. Oh, there we go. Let's do Oh, I know what I needed to do. Sorry, I did it so that my presenter notes weren't showing. There we go. Okay, a word about Swift. Um, I mentioned a couple times how we wrote it all in Swift. Uh, so I'd like to share now about why I mean, aside from Apple being Apple, that we did Swift.

um and what our experience with the language was like. Um your typical languages for this sort of work would be Go or Rust, uh C, C++, those would be the usual suspects. And our team has a lot of prior container runtime expertise, but we didn't have a ton of Swift expertise uh going into all of our runtime work at Apple. Um one of the things that we

liked about Swift is it's got a really good uh interoperability with C. uh Swift gives you a really loweffort bridging to uh system calls and C libraries. And an example of this that that I dealt with recently was uh we needed to integrate Zstandard decompression into if you're familiar with that into the project. Um we needed to do it pretty quick. And if you look at the

Zstandard project, it's C maybe a little bit of C++. But the other thing if you poke into it a little more is there's a package swift file in the top of it. you go into like lib or whatever, there's a module map and that's that's all we really needed to be able to import Zstandard into containerization as like a Swift library. Is that easy and and really

adding support we needed for uh Zstandard kernel images uh because the Kata project moved over to those and but also it gives us Zstandard um image content layer support too. We did it practically overnight. So like the CI interoperability uh it's good. Um other things um Swift has enums with associated data. So if you need a tagged union it's pretty easy and the optional syntax is really

pretty ergonomic. Um it's a safe language. All the all the major language features are memory safe and swift 6 has this uh structured concurrency and uh strict concurrency checking that gives you compile time validation against data races. um talked about the static Linux SDK before. This was essential for us um because we needed to build that statically linked VMITD and VM exec um and we needed to

use library code that we're were sharing for both Mac OS uh and Linux. And finally, uh, being able to directly access all the Mac OS frameworks and features, it it just really simplified the implementation of the whole thing of both containerization and container. Um, so I'll close with a little bit about our open source community and what we're trying to foster and in broad terms where we

think the project is going. I mean, where we all think the project is going uh the community. Um, now since we launched the projects last June, we've been uh really excited by the positive response and a top priority for us is to encourage contributions and grow that community. Uh, we welcome all feedback uh whether it's issues or pull requests. Um, we have discussion forums for questions or

ideas. Um, a couple recommendations if you want to get started. Um, and these are I mean they're pretty pretty much common sense for any open source project, right? Is you know search the issues and PRs if before you start coding and if issue doesn't exist and please file one, get some feedback up front on things that you do too um before you just like push 20 files

up, right? Uh if it's really trivial, then you know maybe an issue isn't needed. Um we use AI a lot now and and we love it. It's great. But if you use it, uh, please review what your AI wrote, uh, before sending that PR up because then you're just making other people review what your AI wrote. That's not always fun. Um, and then if you got any

issues, uh, just like any big software thing, uh, clear reproduction steps, try and make a minimal reproducer and show all your commands, output, environment details, and that'll help everybody in the community triage and figure out your issue quicker, and you can get a quicker uh, a quicker remediation for it. Okay. Um, some other things that we're shooting for, uh, we're looking forward to seeing new projects built

on top of One of the things that's personally interesting to me is is using containers as as jails for AI agents, right, as as basically way to control their access. And this ability to configure precisely what a VM gets, right? So that even if it escaped the container, it's not escaping the VM, right? And the ability to create multiple virtual networks and some of them can have

internet access and some can't. It gives you a lot of possibilities to let your agent just go yolo, right? Uh but only within its secure world. Um for the container tool itself, we want to foster the development of upstream and thirdparty plugins, external plugins. Uh looking further forward, there are some features and integrations uh which we're looking at uh right now like in the next couple months

we're really trying to get the APIs stable. Like right now if you've if you've coded at all against it, we're just blowing things up all the time and I we need to stop doing that and we need to reduce a little accumulated tech debt. But moving on from that, um that that stuff will make it easier for ourselves and for outside contributions and thirdparty Um one thing

we're looking at as far as command plugins is is what we are calling a DRO plugin. And the idea is that gives you longunning Linux environments with tight host integration. I mean it's it's nothing new, right? I mean it's a lot like what a Windows subsystem for Linux would do on Windows. And um we have a discussion page for that too. So if you're interested in contributing

or or you know uh telling us how you think it should be designed uh go participate on that. That'd be great. Um we're also looking at getting Kubernetes development environments going uh for cloudnative development. This isn't like full stack Kubernetes integration. We're talking about mapping Kate's nodes onto containers. If you've ever played around with KIN, for example, it's doing kind is Kubernetes in Docker, right? Um, and

we've used Kind with a container tool um on our own fork, right? But there's challenges with upstream integration. Um, KIN is really built for something else, not really for everybody to to map onto their container things. They call it a Kind experimental provider for that reason. So we've talked to him and and one of the things that we kind of both came to see is that you

know Kite has to do a lot of things because they're not running in containers like we are like in separate VMs. So you know kind of what we all came away with is maybe it'd just be easier to do our own solution for that. So we're right now we're looking at what a great UX would be for that. And again we'd love input from anybody that's that's

interested that likes to toy around with that stuff. Um we're always looking to improve performance and one of the areas that we got started on uh was um to to make the builder better. Uh right now the builder is we have a swift wrapper around MOI build kit running in a container and that's so that we bridge our swift APIs to build kit in that container to

the Golang APIs. Um which all works. It works pretty well and Moby B kit is great. I mean, it's the de facto solution, but that bridging has an IO performance cost to it. And so, one area we'd like to do is is come up with something where we're doing more stuff natively and that would should help build and per uh build performance, but uh that sort of

thing is a large and complicated task. So, just expect more discussion in the community over that in the next year. And then um a lot of container ecosystem integrations. We're we're looking at one of the big ones is compose. A lot of people want compose and compose is great. Um we're also looking at plugging in at the Kubernetes CRI level. And dev containers are another big important

thing that I mean personally I love dev containers. I I want dev containers. So the trick is this. Um, Kubernetes has a nice standard API CRI that we can integrate. It's built for integration like it's built for everybody to integrate against whereas you're working with a de facto standard for things like dev containers and uh compose. And so it it gets trickier because there's just such a

big surface area there. Um, so we want to do it and there are community solutions actually for compose today that you can check out. We've had some really intrepid developers work hard at it and so I'd recommend trying those out but at the same time um we're looking at how in the container project itself we can actually move towards getting those things working and that's it. So

to finish up, if you're on a Mac, uh go ahead, try out the tool. You know, you have the you have the link in the package now. Um and we love back. Um take a look at the APIs if you want to code and yeah, build your own apps and container features. And yeah, I thanks thanks every for coming. Yeah, I will be here. I'll do your

questions, but down there off mic and uh yeah, enjoy the rest of the show. Thank you, John. So, you can do a little sound test. It >> Some of them are Some of them have a little weird, you know, swivel and then this one, I think, is just the button on and off. >> Um, but we can do >> my voice is really terrible because of the

long But I think that's not going to be a problem. I think once you're on the mic. >> Okay. One, two. >> Perfect. >> Yeah. I'm going to switch it off. Hello, afternoon everybody and welcome to everybody that might be joining us again for cloud native days or here for the first time. Um, we've got another great talk coming up on green observability, what needs to shuffle

in open source. So everybody please give a round of applause to Diana Toa. >> Yeah, thank you so much and I really appreciate it. This is my first time with scale and my first time in California and as you can hear my voice is really husky which gives like a cool you know groove but uh you know I didn't aim it like that on purpose is just

a long flight. Uh I'm coming from Spain today. Um and um yeah, let's kick it off. So what I'm not doing uh when I'm not working as a dev at the Victoria metrics, I'm an open telemetry member and contributor. Uh I'm a co-organizer of cloud native days Romania. So if you're ever around in Europe uh in Romania, you can uh give it a go. Uh and I'm

a co-lead of CNCF uh neurodeiversity group. uh this is part of merge forward uh initiative and I'll have a call out slide at the end and talk bit about that as well. So before I get into like the whole talk I want to maybe also give a bit bit of a background uh bit about myself and also the talk. Um so I've been working in tech for

16 years um and in observability for the last seven years. uh I was before joining the let's say a more official community role um I was an SR and a DevOps engineer in observability so yeah I kind of like handled a lot of situations um which brings me to to this talk to uh today um the slides obviously uh you can download them uh from my GitHub

repo and they're also uploaded to this um to the repo from scale um and above this talk I printed like a smaller version like a shorter version of this talk at Fosam conference in Belgium this year. I don't know if you know about Fosdom anyways this open source uh conference in Belgium that is really uh popular because it has like a lot a lot of dev rooms

and a lots of talks that you know they come from different different areas of open source. Uh so this makes it really interesting and this talk I had only 50 minutes actually to present it this year was part of the energy uh dev room. Um and it was really interesting because it made me also uh research a bit more besides the observability u and software uh ecosystem

and also was a very good opportunity for me to learn more about the carbon footprint, energy consumption etc. H so um for me like I've been working in observability like I said for a few years a long time let's say seven years um and we all know that the cost of observability is pretty high um obviously it really depends on the company you are part of but

generally speaking if you're not careful with how you are driving your observability cost they tend to get away right um and it's not a small issue is not a superficial issue actually an observability bill can be around 15 25% of an infra bill. Um also if you're adding AI LLMs to it right now everything is just exploding. Um so we need to be careful around not only

the cost but al also the resource and that bears the in mind the question why sustainability matters in um and why we should be let's say consider it like like like a very important topic to uh to bear in mind. Well, obviously besides the cost, we also bear the the cost of more telemetry, the resource consumption, uh the energy uh that is being spent on all this

uh telemetry and the carbon footprint. Um and while we are all thriving to to take care of our planet, I think it's also important to think outside of the let's say software ecosystem and in particular to observability as well. So I'm going to walk you a bit in in this area of um where it all started uh like where the green software uh let's say definition or

concepts began uh and how slowly it evolved into the nowadays speech about green observability. So uh I'm not sure how many of you are familiar or more familiar or not with the green software foundation. No. Perfect. Awesome. So uh that that was for me actually a novelty as well. Last year uh I didn't know about this foundation. Uh last year I participated to a conference a platform

engineering conference actually. Um Andy was a person that was part of the green software foundation at that conference that gave a really um nice talk one-hour talk about the mission the principles uh and everything that they want to do. So basically the green software foundation is uh was founded in 2021 by various uh companies and with the support of Linux foundation and its mission is exactly to

uh reduce the carbon footprint in software. Um and from all this statement or from this mission uh they also started to begin to focus on very particular actions to to drive the reduction of carbon emissions of software. So uh for example three main important actions of uh of this um foundation is to use less physical resources uh less energy and use that energy more intelligently. Right? So

um they're really thriving for it. It's an open source obviously um foundation. Um that is really nice. It drives like for example more companies to join and help them in the uh in this mission. Um and they're also right now with AI they're also focusing on you know how we can drive uh AI let's say initiatives to be more uh and you know to be focused more

on sustainability. Well um also researching the idea of green observability I came across a few protocols or let's say a few standards because I wanted to understand you know was the link between the green software and actual observability um and I wanted to know if the green software foundation actually made a very particular accent or added a very particular focus on observability. So far it hasn't. So

that's kind of something great because we can focus on something new right now. Uh and also I some protocols about um the greenhouse gas protocol. So that's the first one which is a globally recognized foundational framework for measuring and reporting emissions. Then the the other two ISOs. So the first one is an international standard uh that provides principles and requirements for the quantification and report of greenhouse

gas emissions at an organizational level. I know it's super long. Um and the last one is actually focusing on carbon footprint of products across their life uh life cycle. And this one actually the last one can be applied to uh specific software product or service. So um yeah, you might ask why is this important? So uh it's important because it drives um you know it gives us

a bit more context you know you know educates us a bit more like things started than what we have currently at the moment on in the industry and in the market. Uh and actually um something really interesting the green software foundation came up with this standard um well it's not yet a standard it's more like a specification which is called the software carbon uh intensity um and

what measures measures how much carbon and uh your software our software emits per unit of work. Well I'm not going to explain like the entire um um you know equation there if you want. I mean I can explain it uh but what it uh is basically focused on the idea of energy consumed by software uh and it's not a um it's not a total is a rate

right so it makes it comparable it makes it trendable and observable observable so what's when I stumble upon this specification I was like wondering okay but can we apply the SEI actually to observability would that be a very interesting use case for it and how we can apply it. Um so that's food for thought. Uh I will be very interested to actually continue this in a let's

say not only a theoretical realm but also in a practical way. Uh I would even go a bit further than that uh to explain to want to actually add this specification um as a SLI as a service level uh indicator for observability. that would be something cool to have in the future. So um in economy for example we have a something called sustainability paradox um and is

actually when you have um when trying to improve the sustainability you know and uh efficiency uh and actually the when we have systems that have lower costs we tend actually to buy more and consume more uh and that makes actually you know it's driving more the the the environmental impact facts and instead of reducing it which makes it a paradox uh and actually yes it has a

name it's called the Jevance paradox or the rebound effect um and it also obviously hits the software industry and if we drive a parallel with observability we can talk about the same about an observability sustainability paradox why because uh in our rush or let's say in our ambition to have more observability data to produce produce to have more insights basically into into our systems. We are logging

too much observability data and we are raising actually the system overall carbon footprint. uh which is not something like we want to achieve and definitely um this bears in mind the question how well are we designing our observability systems right with the idea of sustainability in our heads um so for that I wanted to extrapolate a bit and take uh the example of a few uh open-source

CNCF projects um that will lead us to the idea how much basically we are not focused on sustainability or green observability at all. And for example, if we would just take a random example from Prometheus, it comes with default scrape configurations that collect everything. Uh also the scrape intervals they're not very adjusted in the sense that the default ones they might not re reflect the reality we

might not even um need to you know have such high let's say scrape um scrape jobs uh intervals we also don't need to have high cardality labels like p request ID user ID session ID um and we necessarily don't need to have the same retention for all metrics. So what we want to do instead, we want to collect less. So it's not necessarily a game for ambition

of collecting everything. It's actually uh a game to be a bit more stingy. Uh and actually collect for the basics. Um and yeah, it's true right now because we are a bit spoiled that we have a lots of very cool uh tools, projects, you know, we can do everything right now. Uh but unfortunately we are not very um we are not very analytical right we are not

thinking a lot uh that with few metrics actually we can have more than enough information that could give us insights into our systems. So for example, I was talking about collecting less, right? So we could keep only the metrics that we actually need uh or actually use actually sorry we actually use that in effect if you look at reality we only use 50% of the actual metrics

that we are um uh we are designing for then we can actually drop the high cardality labels early uh and as an effect we can reduce the time series count the memory usage uh and the query CPU So right if you're going to think why is this green observability uh why these actions represent sustainability well yeah because uh in the effect we're going to have fewer uh

time series um that will produce less RAM uh then we have smaller time series um uh database with less disk input outputs faster queries uh less compute time u and obviously just maintain the same SLO visibility and but will have lower Um the same thing for Loki. So if um we take a couple of say logging open source um um yeah projects basically the defaults are necess

not necessarily the best ones for green observability. Why? Because uh we are logging everything at the level of info. Then we have this like very large unstructured messages. Usually we have long default retentions. Uh and yeah blogs are actually the largest storage consumer. We might double think double think exactly how how we want to keep those those logs. So what we want to do instead maybe uh

we want to drop the debug logs before indexing uh keep only actionable log levels uh reduce ingestion and index size. And yeah, the same it goes for green observability because less log volume obviously will occupy less disk, smaller index will drive the faster queries, lower injection rates, we will have less CPU uh and obviously we'll still be able to find errors quite quite fast. So uh for

fluent bit um normally in fluent bit the same uh as default we have logs that are being shipped raw uh that uh include noisy or low value logs. Uh then the filtering happens to late uh and we have wasteful network or storage uh usage. Um so there are a bunch of things that happen you know with the default uh parameters that we want to double think about

that. So uh in this case we want to drop um filter drop normalize logs before shipping uh drop health check noise control label cardality early and reduce the traffic network traffic. So um it will be uh great because from a sustainability point of view we'll have early filtering uh that will have you know the best impacts smaller payloads with less energy per request and cleaner logs with

better signal to noise ratio. If we think about telemet so let's say to more like um in terms of metrics logs and traces if we think about open telemetry um yeah people nowadays it's very hard to to capture the the you know the right telemetry data that we want uh normally yes we do collect lots of telemetry data so if you are like sticking with the defaults

or we're not you know designing very well the around it we're going to sample 100% then every request will create a span the CPU will be used for the span processing then we have memory overhead storage explosion more story you know and the traces basically will get we will never uh never be viewed or analyzed. So uh maybe we want to do something So for example uh

we can also use headbased sampling in the hotel collector configuration. Um that will give us you know more um let's say green um methods will have a more green approach to uh to telemetry. Uh we'll have like 90% let's say reduction in trace volume uh lower backend storage uh processing costs. um you know we can have the same visibility but we'll still see the latency distribution and

errors. We can also think about another solution uh that will be the same in the auto collector configuration to use tailbased sampling. Uh and in that case we can keep the um errors high latency traces and rare endpoints. So in this particular case uh we will actually uh be able to use only 5 to 20% of normal traffic and store 100% of the actual important traces. There's

also a third solution with this. So if the traffic spikes we can reduce the sampling percentage automatically uh maintain constant uh resource envelope um and when the system load increases we can reduce the telemetry loads um and yeah in the end we can have the green observability that scales uh So um anyway these are a few examples that I wanted to give because definitely uh in open

source and beyond open source uh in observability we are not um let's say designing for sustainability right so we're not having this as in mind uh as the first thing you know when we wake up we're not thinking about sustainability for sure uh so the idea would be like okay but what what if we build green first observability so what why what if we actually you know

in the design principles in the actual planning we we come with this um um you know objective and we want to focus on so we can actually think about building it kind of does it drop on and off right uh default short attention policies um right so we can have for example cardality warnings with energy impact Anyway, these will be like, you know, ideal world nice to

have. Uh probably some of them they exist at the moment. So anyway, that's that's nice to to know. Uh sampling presets based on energy budgets, uh storage footprint, trend alerts, obviously not collect everything and collects intentionally. So there is a cool project that I'm pretty sure you heard about it, Kepler. um that yeah it's a CNCF project that uses EPF and system counters to estimate energy um

use of container spots virtual machines processes in Kubernetes environments um it's really cool because sometimes um I mean I I was talking u at some point with a Kepler maintainer or contri contributor and is really cool um that what they're trying to achieve and you know the use cases around around So most definitely it's exposing the energy consumption via Prometheus right can also integrated observability stacks that

includes open telemetry however um if we are backtracking to um this SEI so the software carbon intensity specification uh is not natively implemented with it. uh it would be nice to think a bit proactively or think you know how we can actually make this kind of uh projects work together or at least make them integrate natively with one another. Um and that's something that I will also

want to explore in the future and keep coming back if I will ever give this talk again. So um one last let's say um use case that I want to focus on is parts of the Victoria metrics open source projects um because at the moment the way we are focusing not not only on the observability pipelines but alternally uh in the let's say in the coding practices

etc of our develop uh development teams is something that maybe we we can share with the community and we can um uh explain explain for example what worked for us. So it's our own use case. So um yeah it's nice to share with the community. So I will start with the language choice because it's something that that always comes uh comes up in in uh discussions. So

as programming language for example we use go uh many companies or anyway many uh companies usually prefer Python but uh we kind of focused on go because it's very simple it's more efficient um it's quite sustainable to manage um yeah and we do make some parts of Python codes uh like the majority of all code is written in go um so that's pretty cool also another point

I wanted to make is about the careful use of external libraries. Uh because I don't know but um external libraries they often come with with a lots of features that we don't use we don't necessarily need and they just basically add more carbon footprint there. You know they waste resources. So yeah if you're a software engineer developer you need to be careful when you want want to

use this uh external libraries. uh then actually obviously we can practice the green coding um that that you know I heard about um obviously other companies they they do it um it's really nice because basically what what happens here is that engineers come and they refer and keep refactoring codes um they for example they drop like unnecessary code uh that they try to come up with different

algorithms to improve the code make it work faster Um, so yeah, that's nice when you see that the actual engineers come and then stay and think a bit longer before they ship, I don't know, 10,000 PRs that probably don't solve anything. Uh, so it's definitely uh, how can I say an intentional uh, this is what I mean an intentional uh, action that's uh, definitely interesting to have.

Uh, the part with observability and profiling and um, I noticed that other observability vendors are doing it and this is actually pretty cool. uh when the engineers are actually dog um when they actually use their own internal tools, observability tools to understand what they're producing with their codes. Uh basically with profiling, you can understand you know uh if your code is actually useful or not. You can

do synthetic uh monitoring etc. And uh that's always good to use observability not only for the external world but also for your internal projects. Right? So going a bit even further into the use cases here uh all these let's say like different actions that I was explaining were actually successful in driving the reducing the energy costs in uh for once Victoria metrics so that's a metric uh

open source projects uh you know with achieving like less infrastructure um costs um know the cost for network were were reduced and also the energy cost. Uh so the benchmarks obviously they're published, they're online, they're free to use. Um and for us it workware and energy reduction. Um and I wanted to present you know short demo here the cardality explorer u that's one of the unique features

in variometrics. Um and you can obviously scan the QR code to go directly to the playground. Um, and here, so this is the playgrounds. Not sure if it's super clear in the back So going to the explore cardality here, we have something called metrics names with the highest number of series. And we could actually see when they were last requested. So for example, if you see there

zero and you have a metric that's expensive and it kind of like, you know, explodes your time series database, then maybe you want to delete that from your code and reabel it. Um and also you can check the different labels that drive that cardality up. In the top queries we also have queries most summary time to execute and different other um for heavy queries um or you

know queries that take more frequent time to execute and more memory to execute. That's pretty cool because uh as as an S sur I always like to look at how um observ observability vendors and beyond they design their own dashboards. Um not only you know thinking about what cool uh visualizations you can have but also thinking how you can actually debug uh an incident and start um

thinking about you know the queries that drive the pain or might be um you know the the cause of a of an incident in production. So for the logs basically it's the same u principle. Um and here well this will take you to the documentation but what I wanted to show you are a couple of features here that uh hint to um how logs in Victoria logs

are actually running faster is uh there are actually two things here I mean just one the other one is just an explanation that takes you to the playground the QR code so this is streams right so it's log streams uh that basically what they do they they group um stable identifiers like namespace, pod, service. Um so all the log are basically uh grouped together uh for that

specific instance. So it makes it actually faster uh to to take that um to group that stream fields and give you the results. Um and it's interesting because so for example the message um I don't know let's say for example Kubernetes P IP and they're separate because they will always change uh so they're not parts of the stream um field also in the overview. So once you

check the playground in the overview section there you have a couple of different tabs uh that can link you to the cardality of uh of the stream log. Um and it's interesting from let's say an engineering perspective how you can debug this um and understand a bit more okay uh how is my log doing or uh if I can um understand if how it's driving the cardality

up uh etc. So for Victoria traces uh which is currently not in production uh but is still available and you could test it if you want to there definitely we are adding the same features for um and yeah uh it's it's good that um at least from from our perspective we are trying to drive this from the beginning right so we have um we are trying to

design this this uh projects we are trying to design with uh very specific focus on sustainability. So um this part is um I wanted to make an extrapolation with um few so I'm not know but anyway it's um in economy there are a few um principle that are called the nine Rs which were reduced to six um that are refuse reduce repair resize reschedu and repeat um

and what I did like for talking to the community around uh was the fact that also these kind of six Rs they we want to apply them in Kubernetes. Uh and I really liked the community that were uh explaining this how they can be applied to Kubernetes and I like the idea and I actually um you know put it in usage for observability. So if we take

it from these six Rs and actually apply them to observability what we can have. So we can definitely refuse uh and reduce uh the useless metrics. So we can collect a lot less obviously uh and also the logs and high cardality labels. Then we can repair definitely the noisy alerts broken inefficient queries. Uh we can resize with rollups, downsampling and stream aggregation for rescheduling. We can change

when how we observe. We can lower the scrape frequency. We can use adaptive sampling on demand deep diagnostics. And for repetition obviously we can use u the same proven signals, queries, dashboards, alert patterns across services and teams and yeah stop wasting time reinventing everything. So I'm almost at the end of the talk. Um so for me the takeaways and obviously I think it's a very important topic

green observability and in general Um it's nice to have and definitely we should starting uh the green software foundation actions for from day one. Uh would be you know let's say a safer world from that perspective. uh also green observability you know it doesn't depend on specific tools I it's not you know suddenly become greener if we use I don't know Prometheus or LO or telemetry or

it's all about intentional uh design choices and are all responsible to maintain it right so it's not necessarily that we need to wait for a manager or company to say we are becoming green right now but it's also individual decision right so in the end as engineers we all decide how we're going to refactor our codes and how we like to work on a daily basis. Um,

and this one is the part that I was mentioning at the at the beginning with the merge forwards uh initiative. That's a new initiative from CNCF. Um, which is really cool because I'm also parts of of one of these groups. I'm a neurode divergent person. Um so right now in the merch forward initiative we have seven different uh groups. You can check the QR page will take

you to the explanation. We're going to be at CubeCon Amsterdam and always at future CubeCons also here in America and hopefully at other CubeCons across the world. Um yeah so it's it's really nice that we are finally giving a voice to underrepresented groups in tech uh in open source and CNCF. uh we had very nice let's say collaboration last months uh exchanging ideas meeting for the very

first time like face to face discussing a lot more um and starting to to you know grow as a community. So if you have obviously and if you want to join don't necessarily have to identify with any of the underrepresented groups you just you know you want to join or be an ally please do so we are always looking for more friends to chat with um I

left a couple of resources so everything I was talking about right now you can definitely check it out. There's more about this on the CNCF Slack um more about this on the community. This was obviously just a um how can I say a taste, right? So definitely there's a lot more going on and uh there's a lot more to do. Hopefully this talk can resonate in the

future and we can do something more about it little by little. Um also if you want to chat with me or ask me any questions u about mer CNCF whatever you want uh you can chat with me and that's about it. Thank you so much for indulging me. >> So, we do have some time for questions. Are there any um questions from the audience? >> I I

I really like the idea of being able to log just based on an event, right? Um, do you know if anybody's kind of built any proof of concept on that as of yet or is that something that's still >> So do you mean the the the part where I was showing the stream >> here? >> So you you you mentioned where you know if there's certain events

maybe increasing the logging fidelity or and kind of dynamically increasing or decreasing the the logging based on that event. And I'm wondering if there's any anything that you know of right now that that works like that. >> Okay. Um okay so you want I wanted to understand like if you are referring to this part because yeah sorry >> ah okay right yeah um right so if I

know yeah the question was if I know any tools or any projects that are at the moment um sorry if you can repeat again my voice is like sorry. Give me two seconds. >> No problem. >> Right. So there are any tools like or projects that at the moment they can dynamically you know um group logs based >> or or adjust volume right like that was that

I think that was the one of the things you mentioned was is changing the volume of of the logging based on like an event or some some kind of signal >> right u not sure I mean there there are something around that idea so that's why I was mentioning it So part of me mentioning a few open source projects was uh to actually focus like obviously on

on theory but beyond that um with Victoria logs that I was showing is like maybe focusing on the cardality in logs um and like you know grouping um I'm going to say specific identifiers in a stream log that will make it easier to um you know query will make it faster but not necessarily about the log the volume of the log right? Dynamically um using that I

mean in theory somebody that probably designs that could give you a more um I don't know like um like like a more direct response but at the moment I'm I don't have any particular use case in mind um or I don't know myself of any use case in that case but it's a very good question so I I will also be interested and I going to do

some research about that >> So I I I thought it was interesting the idea that the things that are good for sustainability are also good for budgets, right? And so the things that are also good for budgets also sometimes can be good for compute costs and and and just, you know, good good practices also can result in good sustainability. So thought that was just an interesting analogy.

But um uh I I had not heard of Victoria Metrics, so I I'll definitely check you guys out. But as a Graphana user, for example, um and a Prometheus user, a Loki user, um what what are some good ways to identify what really is important and what is not because I definitely know that we are g we're tracking a lot of metrics that we don't use. uh

and I can I can associate logs I can associate dashboards back to which metrics are there but I know that there's a lot of people who are just hitting the explore button and looking at things. So is there any any good tricks or techniques to identify what's important what's not especially if you're say you know if you're doing Kubernetes everybody should pay attention to X all these

other things are rarely used right so any rules in in those regards >> yeah definitely thank you for the question yeah those are very good questions so basically the idea um you know there's a very you know the um how can you focus actually on on the best practices in logging you know metrics whatever traces what actually is important to to to log um you know because

obviously we all of us have a curious mind and we necessarily you know definitely um go a bit crazy there and then start you know looking at a lot of metrics so basically that's I think especially in open source and especially in CNCF cloud native but that's why we have uh the projects and we have the maintainers and we have the best practices for for example in

kubernetes we have actually instrumentation special interest group that focuses on the best practices um like specific um how can I say um specific levels of um of metrics >> so you're saying that there's already some published >> best practices for what to focus on there at that level >> correct and it depends obviously it depends on the project so not all of them have them exactly for

example with open telemetry because is a let's say the new kid on the block it would be nice to but this is like actual an ongoing e effort at the moment from the community part um that they are focusing right now on instrumentation right so they keep uh communicating with one another for example open telemetry with prometheus try to figure out okay what are the best practices

where we can reach each other uh based on the level of experience of other graduated uh projects so for example kubernetes has that it's called instrumentation special interest group and it works at the login and the metrics uh you can give it a go also it's a repo called mixing that is based on alerts um so definitely there's a lots of best practices around that uh yes

it needs more love and actually I give a different talk about that focused on instrumentation in kubernetes it needs more love around the on boarding for beginners for developers uh because there's a lot of ask just like you you ask right now like I don't know I'm a developer I'm somebody new just the will really pop on that page. So this is something that we brought up

to the community quite recently. Um and right now we are building a whole case around it. Uh because if you go to the Kubernetes instrumentation page, you have a huge list of of things and you're like okay but what should I do? Which one should I pick? Which one is the basic? Which one is the events of this actually open a issue last year. So we're are

working on that. Um um what I would suggest besides that so besides the official way of looking at things depending on each projects on each project there's always going to be a best practice there's always going to be but that's more like not necessarily documented it's more like an experience um how can I say companies or engineers that are more experienced that know exactly what to login

uh if you search like online for example in Prometheus world there's a repo that will give you a bunch of alerts um that you can definitely implement. It goes a bit more than the basic level. It goes to intermediate, a bit more advanced, but you have like the whole world of alerts there around Prometheus. And I think it's a very very good repo and definitely worth checking

out. Um but around the best practices is something always that the community should definitely strive to better uh increate. um what I would suggest in effect and it's something when I used to work as an SR is always um focus on the very basic of those alerts or very basic of those dashboards. Uh never do something that is not needed. So always think twice and basically cut

be very very stingy at that level. Why? Because you're going to complicate your life. You're going to complicate basically your infra. going to spend time actually maintaining those alerts over and over again and going beyond alerts dashboards and you know that's the very first level that you're going to have when you uh are dealing with infra and obviously in as an effect you're going to drive your

observability costs your your budget you're going to eat all that budget just on a few alerts and dashboards so I I totally get it Okay, any other questions. Uh, thank you for the talk. Um, about logging, some people were shocked to discover that pod logs get written to files in vlog pods initially and then from there they get tailed by something like fluent bit and then shipped

to the logging stack. Is that generally true? And if so, is that something worth optimizing for? Like less disc.io. >> Okay. Can you repeat it again, but like a bit further from the microphone? Sorry, I'm also a bit death because my >> So, is it generally true that pod logs get written to files in varlog pods first and then they get tailed into the logging stack? And

if that's the case, is that worth optimizing for less disio for example? >> If that's the case, sorry, >> is is that something worth optimizing? >> Well, yeah, definitely. I'm not sure if it hasn't already been done. Yeah. Uh but definitely I mean you have a very use case is I mean I do want to say it in a way that that's a question but uh anyway

uh definitely obviously this is always something that we can we can explore. Yeah. All right. Any final questions? and espanol. I'm just kidding >> in Romanian. >> So will not basically my curiosity is more for your personal experience. So we share together a lot of technologies like Prometheus, open telemetry. In this last year, open telemetry has become more standard than what it was. But in your career,

16 years you mentioned at the beginning. How is your feeling about how we adopt logs? Where are we failing on monitoring? In example we are getting out from solutions like New Relic uh Splunk also elastic search we have a lot of experience on failures because we were just ingesting data and not using that data. In your experience, what would be something that you would do if you

enter a company that is piling up a lot of data and is not using it like you do in your demos or aggregations, logs, tracing, metrics, loss, the the pillars of the uh observability. How is your suggestion or suggestions for people that are entering the observability world seriously now? >> Yeah, perfect. Thank you, Fabio. So the question was like what would be your advice to um let's

say people that start right now with observability and start you know looking at the observability signals and what they they need to ingest, how to do it, how to design it um based on our previous experiences with so many logging um tools. Uh yeah definitely I mean uh right now we have a plethora of of tools and projects and whatnot. um the same like what he said

earlier is like always look at the basic try to think uh very how can I say must to have type of level uh and is also like pick is very hard but you have to do it at test trial right because unless you are going to test that specific uh technology then you're not going to know what it produces or how you can play with it right

so people will definitely going to I know let's say elastic search or whatever uh and give it a spin, right? And see if it does what it should do or or not. Um but the idea is I think it's more about education here. So uh how do you educate yourself in designing observability pipelines and also how do you think uh about green observability? You don't need so

many things to begin with, right? depending on the infra um yeah the infra that you have uh in your company you don't necessarily need all of that you know um so you can definitely start with very few things few metrics logs tracing is the most expensive one but if you really need to have it then you know okay you can do it uh I know other companies

that for example have entire stacks observability stacks made of open source and they keep like for example open telemetry zipin jagger and god knows what because not one is s sufficient to realize you know to complete the like the entire uh visibility on on it. Um so it's it's hard I think uh experimenting is the uh tool here. Uh there's definitely not one solution because you know

I could go the vendor pitch way and say okay try Victoria metrics or whatever. That's not the idea. The idea is that what you want to achieve is something simple or not. If you want to stick with the design principles then stick with those uh for and that's why I think the the whole appeal of green observability is to focus on very simple very important how can

I say strategies when you define observability. So I I've been part of of S sur um teams that were saying okay we need to redefine our legacy observability we need to cut we need to adopt more open source we came with like PC's over and over again how to do that with open source and in the end they refused them all. Why? Because it's way easier to

use a commercial vendor than it is to use an open source one. It's very hard to actually design something, customize it yourself, manage it yourself than it is to just use a plugandplay tool um from an observability vendor. So um yeah, sticking to the basic to the simplicity to you know all design around green sustainability it's very have and but yeah it takes a village in that

case. >> All right. Any more questions? Still have a bit of time. >> All right. Well, great talk, Diana. Thank you very much. Really great engagement from the audience. Applause. >> Um and the next talk here in this room is going to be on uh data and Kubernetes and that is going to be at um 5:00 pm. So in another about 25 minutes we'll have uh the

second talk in this the next talk in this room. Test test. That's and then One, two. One, two. One, two. Yes. One, two, test. Yes. >> One, two. That works. Yes. Hey everybody, welcome back to the cloud native days track. Um, in our next session, uh, Matias Kra is going to explain to us how we should be storing our data on Kubernetes, particularly for the most challenging

use case, which is databases. Mhm. >> Um so with that please everybody welcome Matias to the stage. >> Thank you Josh. Welcome uh hello cloud native days uh LA. I hope you've been enjoying the conference so far. At least I know I have been. Um, and I'm going to talk here about uh databases, uh, which is maybe a bit of an odd topic in a cloud native

room, but I'm going to try to tell you what we do. So, my name is Matias Krales. Um, I live in Belgium in Gen, so I'm a long way from home. And I've been an open source user for a very long time. Um, I started in my teens as a Linux user and then later administrator. Um I early on in my career I was a PHP developer

on your typical LAMP stack that you that you know in the early 2000s. Uh and no one in our team back then had any idea what this M in the lampstack meant the MySQL thing. Our SIS admins did appget get install MySQL server and that was about it. Uh but there's a lot more to it. So I I got involved in in learning MySQL, becoming a MySQL

DBA and I'm currently employed at the Planet Scale where I uh take care of our enterprise customers in the post sales cycle of our of our process. typically I speak to DBAs so um they don't always understand what Kubernetes is. I think cloud native people should understand what Kubernetes is. But I I looked up on on uh Wikipedia and the the definition that Wikipedia has it's an

open- source framework for managing, deploying and scaler scaling containerized applications. So it's it's typically for for applications and a containerized application is is something that is typically very lightweight um very portable and self-contained which a database typically is not. So running a database on Kubernetes is is maybe a bit of a of a strange thing to do, but I I know several people tried and and failed.

We currently we we tried and we are quite successful at doing so. So I want to tell you a bit more how we do things. Um the title of my presentation was stateless storage. Uh which is a contradiction I think uh because storage is typically stateful and not stateless because you want to keep what you have. Um so the difference between stateless and stateful is is typically

um uh on a stateless application uh you run your requests independent from each other. Uh and the the requests don't know about one another. So if you typically think of your uh your web server, your application server or a DNS server, you just ask it something, it gives you a response and it doesn't uh store anything in in the middle. While stateful is typically where where you

do store things uh and typically in database you want to store the state of of of the things that you do. Uh so first a little bit about Planet Scale. Um we were founded in 2018 by the uh team that was originally developing uh Vitas and Vites is something that grew out of YouTube. Um it was founded uh in 2010 at YouTube uh to solve the problem

that they were growing just a tiny little bit um and they had to scale their databases and they reached the part where they couldn't buy servers big enough uh anymore to to host their and at some point um they they developed fit uh to help shard their data on MySQL. Um and then Google decided um in 2018, end of 2017 that they wanted all the large data

sets within Google to migrate to Spanner. Uh which was a new product that they were launching in their own cloud. Um and they they were uh thinking like we have to eat our own dog food. So we have to have our own data sets go to Spanner. And what they did is they released Vitas as a project because it was still interesting. they thought in the CNCF

cloud native computation. Um and then the core developers of Vitas left Google and they founded Planet Scale uh initially to sell uh support for Vitas and to develop it further in in the open source community. But in 2021 uh we kind of decided Vitas is a very complex beast to run and we know how it works. We're pretty good at running it. So why don't we run

it for our customers and that's how what plan scale is doing now. Um originally we were only a MySQL vitas based solution but last year we also started doing Postgress and we run all of our databases on Kubernetes. So I've talked about fitz already. So this is um something more about that. Um, and so a big thing about running a database is that you really want to

save state. Um, most people kind of like their database to save the data that they put in there. Not everyone wants to run it on a black hole engine. Um, but that databases can become quite large and that makes them very slow to operate uh slow for any any operations like backups like move moving data over. Um and we typically think about running databases in the cloud

uh as running them on uh the cloud block storage like EBS in in Amazon like persistent disk in in Google and that is network attached storage um and that is slow compared to local storage um because there's a round trip to the storage and if you have hundreds of thousands of QPS per second those few milliseconds that you have to try to transfer back back and forth

to storage makes a big difference. So at some point we decided why don't we run our databases on the local NVMe storage that the cloud providers have. Um but typically that's uh that that storage is called ephemeral storage because um whenever you stop your virtual machine um the storage is gone and you don't you cannot get it back. Uh it's not really meant to be stateful. So

that is why we've been talking about stateless storage uh because it's running on an ephemeral disk. Um but it is really fast. So you can you can get a lot of good uh IO performance uh out of that uh those discs. But obviously there are some problems with this approach. Like you run a database, your client talks to a database. If you would run it like this,

I know you can see quite from a distance that this is not a good idea. If you run on ephemeral storage with a single server, um as soon as your server crashes or you take it away, um your data is gone and no way to get it back. So, first thing you do is you start adding replicas. So, um MySQL is particularly good at replication. Uh we

learned this year that Postgress can do it too, but there's some caveats there. So we had to patch it just a little bit. Um but MySQL is really good at replication. So what do you do? You put uh replicas in multiple um availability zones um and you start running it like this. So you have copies of your data, you have a replica ready to take over if

your primary goes away. Um but still you're now running um at least one possibly n number of servers that you're not using. The client is only talking to the to the primary. So you're going to add load balancers. Um a load balancer to a primary is always having one uh back end on on the on the on the back end. Yeah. because uh you only write to

one location because if you start writing to multiple locations you start getting split brains and you replicate that data to your your replicas. Uh and for the replicas you can just load balance your um your your traffic across all the replicas you have for reading. um which gives you a good scale out if you have a uh like many people have a read heavy application like think

of your typical uh web shop that you that you would browse you typically browse different products before you decide to buy one. Um so most of the workloads uh on the internet are read heavy. So really easy to scale out um uh for reads uh by just adding replicas. Those replicas provide also the um necessary uh copies of the data. Uh so if something would happen to

your primary, you can safely flip over to a replica and uh and still have everything you you need. Um it's not completely safe yet uh by just running with replications because accidents do happen. DBAs are only human, developers are only human. So sometimes you uh you do something wrong and you need a backup. So what you do is you start backing up your your servers to object

storage that also gives you a persistent copy of your data. So if if all hell breaks loose and all of your database servers would go away, you still have your data in uh in the object storage and the the object storage in the cloud like for uh AWS the S3 storage uh as well as the Google uh cloud storage in in GCP. They both give you several

minds of of durability. So we are quite certain that if you get a successful backup in the object storage um that you will not lose your data. That graph looks familiar like like the Vitz architecture like you what what Vitz does is you have the the load balancers here. you have uh several VT gates which you can load balancer. Vitigate is the proxy and then on the

back end you have several uh MySQL clusters that you can shard into multiple uh shards to distribute and scale your data. Um this Vitas runs MySQL. Uh there's a VT tablet next to each of the MySQL instances which manages the the MySQL protocol uh the the manage management uh MySQL demon. Um and it also communicates with uh the topology server which is up here. Uh Victas supports

um CD just like Kubernetes uh runs at CD. Uh it also supports zookeeper and there's still some support for console but we when in general when you would run vitas we would recommend you to run uh with withd that's a service we also run at planet scale for our topology servers and and it's been proven to be very stable and very good. There's a bunch of of

side processes uh to test like the control demon to make changes to your cluster uh VT orchestrator VT or uh to do high availability if your primary tablet would crash or would would go away then VT or will promote one of the replicas to become the new uh primary and then there's a VT admin uh UI which gives you a graphical overview of of what's running in

your cluster. And so um Vitest serves millions of QPS in production every day. All of these companies are running vitas in one form or another. Um we are planet scale. We are on this slide. Uh we run everything that runs in planet scale on MySQL uh runs on Vitas. We don't have any plain MySQL offering. We only have a Vitas offering uh for MySQL. That's different from

our Postgres offering. While post while MySQL always runs on Vitz, Postgress still runs plain Postgress. We are currently in the process of developing uh a tool that we've um named Niki and that's going to be uh the Vitess for Postgress. So we are building that out and we hope to release that later this year. Um, and we run some of our databases, not all of them. Like

for some of the some of our use case, some of our users, EBS storage or persistent disk storage is still good enough. If you don't have a very high QPS or don't have a lot of data, running on EBS is still good enough. But for the databases that are more demanding uh we run planet scale metal and we run your database on ephemeral storage. So we run

on the ephemeral discs and we make sure that we don't lose your data. this makes your database really fast with virtually unlimited IOPS. Well, there is an IOPS limit to to what the NVME drives can do, but despite our best efforts, we have not been able to find it. So, so far, we haven't found the limitation. We run our clusters in multi-AZ deployment. By default, we don't

run multi-reion. We can run multi-region if that's some a requirement for your uh but by default we just run multi-is um I think if uh an availability zone like us east1 would suddenly entirely disappear there's bigger problems than um so right now multiaz is the default uh multi-reion is possible and for the durability guarantee we run our mysqls with semi-sync replication uh and and we require a

cross acknowledgement. So every commit that goes to MySQL will be replicated. The binary log will be replicated to another availability zone and at least one of those replicas in a different availability zone will have to acknowledge that it receives a transaction before the client gets the acknowledgement like this commit is good. So that's important. For Postgress, we do the same or the similar thing. Um we have

by default two replicas, two standbys uh in another availability zone and we require at least one of them to synchronously acknowledge um that the commit is is there. Um and this is the place where we had to patch Postgress. Um because in postgress there's an option for the client to cancel a transaction but cancel doesn't really uh roll back the transaction. The transaction is already committed. But

cancel just cancels a replication to one of your replicas. And at that point in time, you would end up with a committed transaction on your primary that gets an acknowledgement back to the to the client, but it's never replicated to one of your replicas. So you don't have a guarantee that this is um uh safe. Um so that's why we did a small patch to Postgress to

make sure that if uh there's no acknowledgement from any of the of the um uh replicas that we if you do a cancel that we just refuse that. That was a requirement for our system u to to become available online because otherwise we couldn't guarantee um durability of your data. To make sure uh that we don't lose any data, we also do daily uh full backups of

all our um MySQL and Postgress instances and we also do a full verification of those backups every day. So a default backup process at points scale um looks like this. We spin up a new machine, a new pod in Kubernetes and we restore the latest successful backup. We hook it up in replication. We let it replicate for a while and then we say okay this is working

well. We stop the instance and we copy the data directory uh to the cloud to the object storage in the cloud to uh uh Google uh uh cloud storage in GCP or to S3 in uh in AWS. Another thing that we do um for uh any upgrades that we do like uh our system is a fully managed service in the cloud. So we also make sure that

we keep up to date with any uh uh any patches that need to happen like recently there was a um uh a CV found in um in Postgress which was quite bad um and we had to patch all of our things uh to make sure that we were uh not vulnerable. Uh so what we do in the metal uh offering uh what we do in in the

non-metal offering like running on on um u the EBS or persistent disk um is we take a replica out we upgrade it we put it back let it replicate go to the next one do repeat the process at the end of of of where all our replicas have been upgraded or patched we will um promote a new primary and then we do the same process to the

old primary uh we cannot do that in metal because um we need our durability guarantee. So we want to make sure that we always have three copies of the data. So what we do is we search for upgrades. So we add a new we add new replicas, new instances to the cluster and we run those in the new uh new version. We replicate them and then one

by one we take the old replicas out of the equation and then finally we promote a new replica a new primary uh from one of the new uh installed replicas on the newer version to become the new primary and then we uh finally decommission the old primary. any questions? >> So, I was sure what the uh the splitting process looks like. >> The what process? >> The

splitting process like you know a primary gets too large. >> Okay. Yes. So that is part of our Vitess offering. Uh our Postgress offering currently doesn't do that yet. So Vitess has a feature called uh VR replication. And VR replication is what we use uh to to do multiple things like initially it's it's a process that we use to do imports like import data from an existing

database. Uh we replicate it in but it also is the uh the process that we use for um uh shard splitting. So if you have a primary that grows too big too too big for uh the um the server that you're on, we um uh start a new keyspace is what we call that in infest terms uh in the new um sharded uh way. So three two

shards, four shards, eight shards, however many shards you need. And then we do a vreplication process to uh copy the data to the new sharded environment. And then uh finally we switch over the traffic to the new uh uh keyspace and we decommission the old one. Uh what vreplication looks like is it will first run a select star from table start streaming all the rows to the

new um uh key space and then uh it catches up on the binary logs of MySQL to bring the the the target table up to speed. Um view replication is really powerful. It can do uh uh filters on on where this where you want the the data to go sharding wise but it can all it also works for uh our um online DDL uh solution that we

use at plan scale um to do schema changes. So it can also convert the the data on the uh source table to a to the new uh uh format of of any uh alter more questions. >> I have a couple. >> One is so it sounds like the Postgres version is going to be a separate >> It is >> from Vitas. >> Yes. We're not uh trying

to adapt fits to Postgress. We're basically rebuilding it from the ground up with all the learnings that we did from uh from Vites uh and and building it for specifically for Postgress. So they they are separate things um but uh we we still use the expertise that we build up over the You mentioned that data is all affirmal. uh how do you hydrate it for the very

first time when uh when somebody's like booting up a new instance? How do you get it from the object store or the backup or the replica? >> Yes. So um if you start up a new instance, we restore a backup, we hook it up in replication. Um like for every upgrade that we do, we do the surging. uh so basically we start up a new instance. We

restore the latest backup. We hook it up in replication which is the process that we do every day for our for our backup method. So we know that this is will that this will work because we did it at least 24 hours ago it it still works. So we are quite confident that our that our backups are always going to going to restore. Um, if you do

an initial new database, you start with an empty database and you do an import from wherever you're coming from. Um, so that that's how how that gets on. >> Yeah, a follow on to that is after a while if you have a right heavy database, the backup and restore and creating a new instance always gets uh the provisioning time gets bigger and >> Yes. Yes. So how

do you like manage to load the minimum working set of the database based on the workloads? >> So that depends on on the customer like um if you really really start growing very very quickly we will uh shard your data into smaller segments. So that at that point in time instead of of one giant 100 terabyte database you might have 100 shards of one terabyte and we

can all back up and restore them in parallel. So those backups are really really fast. Um while uh in postcress currently there is no uh no such solution yet. So there it it really is um the time is growing as the data keeps on growing. Uh so we tell uh potential customers that are uh coming in with hundreds of terabytes of uh of data in Postgress either

to migrate to Vitess uh to get them sharded or to wait for migration uh until the new Postgress product that we named Niki is coming out year depending on on how the development and testing is going. one from me. Um I'm curious as to what would be different about the storage setup if somebody was running on their own metal. Um if you're running on your own metal

like your own bare metal server or data center um that would just work fine. The biggest issue um is we run everything in Kubernetes. Like the biggest issue then is um what's the lead time for you to provision a new server if you need to uh upgrade like all of our machines uh are upgraded every 30 days for security patches. So we don't have any databases that

are hundreds of days running. Uh we we reboot everything and patch everything at least every 30 days. So we re we do this this upgrade process like continuously to continuously test the process and to continuously also uh replace the the machines uh for security updates and all of those things. So if you would technically it would work running it in your own data center with your own

Kubernetes. It's just much more difficult uh to provision new machines. >> Okay. Well, thank you very much. >> My pleasure. >> Um we will have one more session today in the cloud native track. just in ballroom A. There will not be a session in this room. Um and then of course don't forget about game night which is tonight. Thanks everybody.