Declarative Edge Kubernetes: Immutable Clusters with Talos + Zarf - Brandt Keller & Merijn Keppel

This talk introduces declarative edge Kubernetes, focusing on deploying applications in air-gapped and edge environments. The speakers, Brian Keller and Marek Kubel, share their experiences using Talos Linux as a declarative operating system, emphasizing its immutability and deterministic configuration. They demonstrate how to set up a Kubernetes cluster using Talos and manage applications with Zarf, a declarative package management tool. The presentation highlights the challenges of maintaining and deploying applications in environments with limited connectivity, while advocating for sustainable practices in DevOps. Through live demos, they show how to efficiently handle updates and package management in such scenarios, ultimately leading to reliable, secure, and sustainable Kubernetes operations.

Welcome, welcome. Everyone, uh if you want to hit uh to this discussion on declarative edge Kubernetes. Uh I myself am Brian Keller. I'm a staff software engineer at Defense Unicorns, a CNCF ambassador, and a maintainer in the OpenSSF. My name's Marek Kubel. I'm a principal consultant at TrueFullStack. Um this is my first time speaking at KubeCon, so stoked. Thank you. Thank you. Um big crowd at 5:00

p.m., so um you're all welcome. Um because we're doing a live demo, I brought some hardware. It's a home lab that we get at TrueFullStack. Uh well, you can get it if you want to uh to tinker around with stuff like Kubernetes or other CNCF stuff. Um and I pre-provisioned these nodes with Talos Linux. Uh it's a maintenance mode. We will talk about Talos Linux a little

bit more later on, but um for the sake of demos, I will not boot them via USB stick. It takes too much time. So, take it away. All right. So, today you're going to walk you through a bit of a journey from kind of the things we prepare for all the way out to orbit. Um you're going to see our spaceship, if you will, taking off through

each of these stages, and we'll walk you through each of them. Um this is a bit of a disclaimer that uh the slides are not going to be very useful. They're going to be just generic high-level stuff. Um we really want to show off that we can do all of this live. We can do all of it here with this hardware. And, you know, kind of for

KubeCon North America, I really want to create shirts that say, you know, friends don't let friends do recorded presentations. So, uh with further ado, um I think we can move over to Oh. Uh each of these stages, right? And I think we I mean, we kind of talked about them. Um prep we'll go through real quickly. Take off um why Talos Linux is really a great fit

for us. Um for particularly thinking about like declarative, deterministic, immutable. Um that's important. Uh ascent is like how do we get our workloads deployed? Um Zarf as a declarative package management tool will kind of help us do this. And really once we get to operation and orbit, we're going to see that like we want to treat these things in a sustainable fashion. We want to make it

so that when you're doing an application update, it feels the same as if you're doing an operating system update. And those kind of at times can feel very conflicting. Uh they can be very different. They can be managed in different ways. And so how can we how can we bring this experience back in such a way that it feels more sustainable? Right. So, whether you're running Kubernetes

on a windmill on the North Sea, Kubernetes on spaceships, or perhaps something more challenging at a conference. I know everyone knows Wi-Fi here. It's kind of a thing, right? Um the challenge is similar. Bad or no connectivity, and we're trying to run a piece of cloud-native software that was designed to run on a connected system like the internet. Exactly. And you know, for air-gapped Kubernetes, Kubernetes now,

you know, well over a decade, running it in the air-gap, running it at the edge isn't necessarily new, right? We're not going to come up here and stand up here and be the we're the first to do this. We're not. Um but I think that we've seen many examples of this. Many of you have seen, possibly, air-gapped deployments of Kubernetes, how you deploy applications to air-gapped Kubernetes,

and kind of this anonymous uh relation that has to edge at times, right? Maybe edge isn't a single node. Maybe an edge is multiple nodes. Maybe the edge is high-powered computing. Um and that definition varies. And you know, I think what we really want to focus on is that we want to we want to create like highly available systems that are reliant on the the they have

available. If you have cloud infrastructure, by all means use cloud infrastructure. If you don't, if you have a single node sitting somewhere in a far off location that has a very small amount of bandwidth available, we have to treat that very carefully. Is there ways to do this where, you know, maybe we manage applications the same for either scenario and kind of buy down on this like,

you know, true devops integration that we really have been pushing for a long time. Um so with launch prep uh we want to if we're going to take anything we really treat air gap I'm going to say air gap a lot. you know, really enjoy working in air gap environments because it's kind of the most extreme. If you've worked in those environments and you operate with them

and you can deploy to an air gap environment, you can deploy to a connected cloud way way much easier. Um but also at the same time uh the fundamental still apply. If we're going to go there, we need to know what we want to take with us. Uh we need to know what their dependencies are. Uh and we see we're okay, wonderful. Get to grab those things,

put them somewhere, do something with them. Um but we want to put them somewhere, do something with them in declarative and deterministic ways. Uh so that ultimately we're creating this feedback loop, right? How do we develop applications so that we can operate them in air gap environments where we don't put necessarily the burden of one or the other on a sole persona. Uh but we can make

it easier for everyone to do with confidence and that's really where the determinism comes in and predictable and scoped and I think this comes back to the immutability angle, right? What is immutability really trying to solve? It's really the ability to make predictable and scoped changes. You know, how can we do something that is in a safe and reliable and ultimately replacement model as opposed to a

modification model. after we prepped, um we should take off. Our spaceship needs to fly. Uh, apparently the windmill will also move during your session, right? Um, before I'm going to talk about Talos, uh, we're doing a live demo. Demos take time. So, let's head into the terminal, install this cluster. Once it's installing, let's talk a little bit about Talos. So, let's drop in there. That was a

seamless translation. So, um, I'm going to to apply a configuration to by running the Talos CTL commands. Uh, I have these nodes configured with static IP addresses. Uh, for me it's easy to under- to remind myself what they are. So, it's the 254, 253, and 252. There we go. And, um, let me switch over to a mode where we can actually observe the output of these nodes.

Um, they are now in a booting stage. That means for Talos, "Hey, I got a configuration. Um, I'm waiting to join a cluster or to form a cluster, depending on what what scenario we're in now." Currently, um, I just booted the thing. I just said it, "This is your config." It's not a cluster yet. So, let's say bootstrap the thing. So, I can run this command against

any of these nodes. I just happen to take one of them. Once I do that, um, we will see uh, one of these nodes becoming running, and the other nodes one of them. Uh, one of these nodes, um, forming the primary etcd member, the others will join in, eventually becoming a Kubernetes cluster. So, while that's running, let's switch back. There. Like the take off. I was thrown

off by the color here. Talos Linux, what is it? It's a declarative operating system. It's immutable. It's run by supplying it with configuration. So, the fancy animal you're already familiar with, we can just supply it to the cluster. Tell it what what it needs to do. It starts OCI images. It takes in these these things and eventually become the thing you tell it to do. So, Brandon

already said something about determinism deterministic systems. What we do is we we have a deterministic way of configuring an operating system to make sure that what we want to do what we want it to do actually happens. This is specially if you're going to run your your workload in an isolated environment. Because if it's an air gap or if it's an edge situation where you don't have

connection to your cluster, well, how are you going to manually mutate all these these configuration things? So, there's no manual intervention by me. This cluster is now actually spinning up. It's booting. It's becoming ready. And before we are going to talk about Zarf, this cluster is also doing a Zarf in it. So, I'm going to head back into the terminal and to see what's actually happening with

the trusty tool of K9s. Absolutely. So, Zarf for those who want to look it up, zarf.dev. I'm going to browse over some of the quick highlights of kind of what's happening in the background. Maren walked us through, you know, configuring and bootstrap for this Talos cluster. Part of that bootstrap process was a Zarf initialization. And so, every time that you want to take Zarf to a cluster

and use it to manage how you deploy applications. The first step is just to initialize that cluster once. Uh what is initializing that cluster look like? Uh really goes back to the you know, fundamentals of how do we deploy applications and the chicken and eggs that we came across along the way. Again, I believe that there are many of you who are familiar with air-gapped Kubernetes. If

not, welcome. Um but ultimately, we get to this point where, you know, the first thing we want to do, we want to deploy an application, uh well, we need a registry to pull that application image from. Uh if you're on a single node and you just configured it with Talos, uh we need a registry to get the registry image to deploy a registry. Uh and very quickly,

there's a problem. There's a blocker for us. With Zarf, uh just to kind of give you the quick spiel, we bootstrap the cluster with a registry so that we have a registry to pull those images from. Every subsequent package deploy thereafter can now put those images into that internal registry and then we can have the container runtime pull those images from that internal registry. That's great. Um

the second part of this problem, again, coming back to how I've said sustainability a couple times now, is just to highlight that we really want to shift down the economics of how do we make a package of an application, it could be a package of one application or many, how do we make this highly portable? Um we don't want to build applications that have environment-specific information in

them, right? We don't want to have to say I'm going to build my Argo CD package for environments ABC all the way through Z and have to manage those all independently. And so when we think about what do we have to change when we go to environment that's air-gapped? We have to change that in that registry image references in all of our Helm charts, right? Um so

we want to start to figure out like, okay, well, how can we shift this down to have Zarf be our baseline? So bar so Zarf has a um um an agent that has a mutating webhook for it. And that agent is going to provide us with the ability to mutate image references. Uh so, I can deploy an application CubeCTL run podinfo um pass it docker.io or GCHR

for podinfo in particular. And it when it hits the mutating webhook, it's just going to mutate that for us and point it to the internal registry. So, we're never touching image references. This applies to our GitOps for using Argo or Flux. Same things happen. We can bootstrap a Git server with the Zarf init process. We can push repos into that Git server and then we can mutate

those Git server references in those Flux and Argo resources. And you know, kind of create this cohesive, hey, I have a Flux uh application deployment. I I rely on Flux and I want to take that same Zarf package to many environments. Um so, that kind of provides us with what's required to get started. Once you have your Zarf init cluster, then we just do package applications. Yeah,

exactly. Um well, this cluster has been initialized now. Um Cilium is running, some other stuff is running, Longhorn um because I wanted to complicate stuff for myself. true. However, um once once the cluster is running, we have a a working Talos Linux. Um I did not mention, by the way, this cluster is actually not connected to the internet. If you want to know how I did it,

you can ask questions later. I can definitely show you that it does not have any internet connection. This is only for me to connect to the cluster because uh it's air-gapped, so no Wi-Fi to it as well. Um we will actually start going into the uh Zarf part. Mhm? Where we will um Well, you already talked talked about it a little bit. We're actually going to deploy

a package. And uh I will do that via the terminal again. Yeah, can you bring up the Zarf YAML for me? Certainly. So, let's go. Um let's go over DOS games, right? There's the Ooh. Let's just delete that. Yes. So, uh while we're talking about applications and the declarative piece of this, I just wanted to bring up really quickly what this might look like. This is only

one example. The combo components can consist of different types of artifacts. What you're going to see here is a application deployment for something called Doom. Um and you'll see how we're going to deployment how we're going to deploy it with manifests. You'll see the images that we need in order to deploy it in particular. And this is declarative, right? We can you know, reproduce this process for

this application. You can use the same declarative manifest to create all of your application packages. And in particular, this is great because you may want to have something highly portable that you pass around from a version controlled repository, for instance. I may have this application. I could run Zarf package create just that on this manifest, and it will perform the all of the packaging Uh I could

pass this manifest over to Marine, and he could run that same process and get these same package as the output. And that makes it so that we don't have to pass around in air-gapped environments. We have to pass around these packages, and we've seen very large Zarf packages before. Tens of gigabytes. It happens. Uh those things aren't inherently portable in and of themselves um when they get

to that state, but the metadata is highly portable. Uh especially when you're in the connected side. And so, there's our manifest. What we're going to demonstrate really quick is deploying the Doom game. Oh, why not? Let's go ahead. So, that will be Zarf package deploy uh DOS games and there is the 1.3.0 version of the Zarf package that has been referenced by the metadata. Uh to speed

things up, I actually created the package from that metadata uh beforehand. Obviously, otherwise, I would have to download the images. But, let's go ahead and deploy that. Um you want to talk us through what we'll see on screen here before actually hitting yes or Yes. So, one one kind of like broad overview for Zarf, again, is that this is a when it's packaged to put into basically

a single archive, a tarball, so that you don't have to copy around many different files. Um you run Zarf package deploy against the tarball. Uh you can push this tarball up to an OCI registry and pull from there, deploy from there, etc. But, for sake of the baseline concepts, we want to basically reshow what are you doing? Um because we want this to be highly transparent ultimately

at the end of the day. Uh you should be able to pass somebody an archive containing your application and they shouldn't just inherently trust it, right? We don't want We don't want that. Um but, we also want to augment this with the supply chain. And so, you can take a created Zarf package and you can sign it, for instance, and then you can verify its signature. When

we are actually doing the packaging process, we're grabbing that image from wherever it is upstream and we're going to create an S-bomb for it. Uh we're going to put that S-bomb into the package itself. We're going to put the signature for the package when it was signed by something like Sigstore and Cosign or even just local keys, really. Um we're going to put that signature into the

package so that wherever you take that package archive or that package as it's published to an OCI registry, um you get all those artifacts inherently. You don't have to copy around and manage each of them independently. Great. Well, let's continue. I don't care about supply chain in this case. It's Doom. It runs everywhere. So, we get a warning message from Talos, which is fine. I don't worry

about it. Trust me. And we can actually uh use ZarF to proxy or create a proxy command for us to connect to the deployed application. If go back to uh K9s real quick, we can actually observe that uh the game it's already highlighted. That's a surprise. Um that the game is actually uh on the cluster somewhere. And if we, well, go ahead and go back here, ask

it to connect to Doom. And this is the the thing it creates a URL for us that we can actually open. I'm not sure where it's going to open, either there or here. Well, lo and behold, we got something that works. Now, got to figure out how to to close it again, right? It says control C to end, but it does Oh, no, we cannot quit Doom

anymore. Oh, well. That's annoying. Yeah, I saw the tab up there. Yeah, it's I'll just close the browser and start over. I mean, why not? we got Doom back. Yeah, you can laugh all you all you all you want. Doing demos on stage is really hard. I I noticed. we should be able to just go to the other tab. We are in the presenter mode. We were

in presenter mode. So, let's move that Nope. Nope. Nope. Nope. >> And while we're getting that set up, is this kind of like the things that you saw really quickly? We said, "Yes, we want to deploy this package." And there are a couple things that are happening there that you saw that kind of talked to you already, right? We took the images that are inside that archive

and pushed them to the internal registry and then we used, you know, tools we're all familiar with in order to install the what was needed for the Doom game to be installed into the cluster. Uh and so in that at that point in time, you know, we're we have Helm releases that we can go and inspect. You're not locked into this, you know, Zarf tool. Uh it's

doing things with other tools that you're all familiar with uh and that you can kind of cooperate with through other tools in the ecosystem. Cool. Well, um that's the thing about applications. We deployed Doom. Um so, I've already tried to explain Talos is composed of containers and manifests. um I I not sure if you if you're familiar with this thing called Kubernetes. Um we're at KubeCon and

apparently Kubernetes can handle these manifests very well. Talos has an API. Um so, combine an API with another API and I think you can do magic. Um so, one of the things that I've looked into as an experiment is to actually update Talos Linux from Kubernetes. I mean, Kubernetes is running on the same node. So, how are we going to do that? Well, we're going to leverage

the Talos API to send the upgrade command to the nodes. And as an experiment, um like I said, I wanted to challenge myself. I wrote a little controller um around the Talos API, created a CRD for it. Um basically, what it does is it it picks up some uh some information. Uh so, what version are you going to actually deploy on your um nodes? Uh what image

are you going to use to uh uh to deploy it? And well, that's the information we can feed into another Zarf package. So, let's have a look at that Zarf package. Uh we should be able to close that one. Um so, let's inspect the other Zarf package here. That will be the update. And basically, what I say, "Hey, I want to update to Talos 1.12.5." Um I

put in the manifest for my controller here. It's not very interesting at the moment. Um otherwise, uh I also put in some images. Uh so, the images you are probably familiar with around Kubernetes, CoreDNS, etc. And uh at the very bottom, the actual operating system image, which is just um an OCI artifact. And that specific OCI artifact is the the thing that Talos can use to actually

update a node. Um but because we're also running this on a control plane, the control plane will, you know, reboot itself. Um the process takes about 5 minutes, so uh I could ask you to grab a cup of coffee, but let's not do that. So, let's initiate the the upgrade process. Watch what happens for one of the nodes. I will let it run on the screen to

see completes to the end of the session. Um but we're not sure, right? It's demo. let's go out and do a Zarf package deploy of the update Talos thing. So, here's the package I pre-created. Again, we have the information. Again, I have the SBOMs available. So, let's actually deploy the package. And I'll have a look at K9s. So, um let's inspect the nodes. And they're all ready.

And I can actually get the I'm not sure if it's readable eventually. But, we should be able to observe one of the nodes going into an upgrading stage. So, um I see on the left side that there is a node actually upgrading. And it will, in theory, become 1.12.5. Uh because we're running 1.12.4 at the moment. I'm not sure if Again, if it's readable from the back,

but again, you have to trust me on this. Sorry for that. So, basically, we're we're shipping applications. Talos Linux is an application. It just happens to run on bare metal here or your VM. And while that is running, um let's talk about sustainability. Exactly. Um so, kind of the final stage of our journey here is to try to figure out how to make processes more reliable, uh

more secure, and ultimately like we want to reach for sustainability in such a way that it it feels uh like we can we can manage these things in a way that provides a confidence around the feedback loop, right? And so, some of the things that we've done with we've set up uh for like those who work in like operations, we set up air-gapped environments before, set up

air-gapped clusters. We typically have to, you know, kind of be the middle person between the person app like developing applications and then the person who uh ultimately has to deploy or maybe that's the same the same person for operations. Uh and you know, when we're thinking about trying to shift left as much of like the confidence that we can inspire when we know we have to do

these things, uh we want to be able to do it in the area or the the place where we can have the fastest feedback loop. If we're going to go to an air-gapped environment, deploy something, and find out we're missing something at that point in time, that's a very bad place to be cuz it's just a long, you know, runway in order to get back to the

point where you need to actually make the change again. Uh especially if you work in secure regulated environments, there could be a lot of implications along the way of what that means and how to get a new update back to that environment. And so, how can we shift things left in a Zarf particular, that's where we've been looking at workflows that are like, "Hey, during Zarf package

create, what things can we check in order to figure out exactly how to fail faster?" Uh if we we need to pull images and those images don't exist, we want to make it very obvious to you that that didn't work. And really, that's kind of the workflow that we instilled back into the Zarf package create process. If you're missing something that you said should be here, this

is a declarative manifest, we couldn't find it, fail, let you know, and you need to go and remediate that so you're not missing something at that point in time. When you do a Zarf package deploy even in a connected environment, sometimes things can, you know, silently work. They get pulled from some external source because of, you know, your three abstractions deep into your GitOps environment through customize

and helm and other things. Uh that sometimes it works. Uh but we want to, you know, try to provide a way, a structure that can help you like test along the way. And for the testing that I even do in connected environments, I've got a node sitting on my desk that is disconnected from the internet. I can connect to it from my machine, it can't connect out

to the internet. And so, as as a last, you know, last fall back every time I want to test something, I go ahead and I do Um and so, ultimately, when we're in orbit, when we get to sustainability, these are the problems we're trying to solve with immutable clusters, with immutable like state and packages. We want We don't want to modify things. We don't want to use

imperative workflows. We want to be declarative and and ultimately immutable. Surely. Well, I've seen one of the nodes rebooted already. Um the other one's rebooting right now. We can wait around for the the last one, of course. Can we bring up cube CTL and and look at those numbers? Sure. Sure. apparently, one of them is not ready, which is expected. So, let's grab the trusty cube CTL.

Just get the nodes and if I output it as a wide thing, sure, let me grab up the terminal there. And see if I can get that on a readable manner, the output. Not that much. Perhaps. I don't know if that's It is readable at the back? >> It's cutting off right at Yeah, it's on the leftish. Yep. >> So, it's on the left there. It says

12.5. The other one's rebooting. Eventually, if we give it enough time, everything will come up to 12.5. So, note that I only did a minor version upgrade. That might be something that you need to do for security fixes. Um I tested it out with a major upgrade, as well. Um it's a bit more complicated in terms of Talos because you also need to ship new version of

at CD, etc. Um and we can do that, but for the sake of demo, it takes too much time, of course. so, if we run this again, I believe the other one's booted. So, we see two nodes on 12.5. And eventually, once we're all done, we have a happy customer again. We have our operation done. We upgrade the cluster. We fix the problem. Um perhaps we even

ship new applications. You could do get ups with this. You could do anything you want. Um just use your imagination. All the things you need to do is already possible in Kubernetes. Um and Talos is just an extension of that. Also, um a great operating system if you want to go into deterministic systems. little shout out to Sidero Labs for providing us with this awesome operating system

and the fans unicorn for this awesome tool to make this all possible. Thank you very much. And if you have time, our session feedback uh is available as well as if you want to find out more about Zarf, you can go to zarf.dev. If anybody has questions, we'll be hanging around. Um I think we're the last session anyway, so We still have 2 minutes for questions, so

you can run up to the mic or just come on stage and ask in person. It's