Tutorial: Your Application, Batteries In... Brandt K, Austin A, William C, Merijn K & Jessica K-D

This tutorial session delves into the Zarf airgap package manager, which simplifies application deployment in isolated Kubernetes environments. Brent Keller and his team, comprising experts from Defense Unicorns and other organizations, guide participants through the nuances of packaging applications, highlighting the significance of deterministic, declarative configurations. They discuss the importance of all dependencies being included to ensure seamless operation regardless of connectivity. The interactive workshop showcases practical steps using the Instruct platform, allowing attendees to create, deploy, and inspect applications. Key topics include using the Helm SDK for deployment and integrating supply chain security measures effectively.

Welcome everyone. Uh today we got a tutorial session for you. Uh for anybody who is is there anybody here who joined us on Tuesday for the Talos and Zarf presentation? Awesome. Welcome back. Uh so today we kind of wanted to shine additional light on that one, right? Like we typically will use Zarf in airgapped environments. Uh, but there's a really neat part of this that is just

kind of like how do you package for an extreme of the air gap? That could be something you could apply back towards the determinism you want even in connected environments. Um, so before we hit off anything in particular, uh, my name is Brent Keller. Um, I'm an OSS maintainer at Defense Unicorns, a CNCF ambassador. And with that, I'll pass it over to Meen. >> Hello, Marine Keapple,

a principal consult stack. Um well you might have seen me on Slate Tuesday um working on open source products for Truthful Sack and well I really enjoy working with Zarf so that's why we're here. >> Awesome. There you are. >> Yep. I'm Austin Abro. I'm an OSS maintainer specifically maintain Zarf also defense unicorns with brand. And yeah thank you all for coming. Pass it off to Jessica.

>> Yeah. Hi I'm Jessica. Um I'm working as an engineer at Isala. Uh it's a hospital in the Netherlands. Um it's the first time for me. Uh I'm excited to help you today with a tutorial. So yeah, William, >> nice to meet everyone. I'm William Crum, a six-year Marine Corps veteran and a solutions architect over at Defense Unicorns. Additionally, I am the co-chair of the public sector

user group for Lissnf. >> Awesome. So if you are joining us today, um there probably is a variety of reasons. Maybe something on Tuesday piqu your interest or maybe something from the abstract as you were going through the schedule uh specifically applied to you. Um some of the things we focus on with Zarf uh well I guess I should mention logistics really quick. We'll cover that. Um

we'll go through the packaging intro and we'll go through the workshop. Uh the workshop itself will be multiple modules. So, you'll be able to do package create, deploy, inspect, and then we have a a playground setup. Uh, basically, once we get to the playground session, though, it's kind of a free-for-all. Uh, if you want to package your own application, uh, we can help you with that. Raise

your hand. Um, we'll talk about the Instruct platform here in a second. Uh, but also, if you want to see us package an application, uh, throw out your favorite cloudnative computing foundation landscape application and maybe we can run through that as well. kind of see what time allows. Uh, so logistics. Um, today we're going to be using the instruct platform. Uh, I don't know how many of

you have used it before, but if you go to zarf.dev/tutorial, it really allows us to try to provide some guard rails, if you will, around this tutorial. Zarf's really meant to work in airgapped environments. Um, you build everything on your connected side and then you take it to air gapped environments. That's all great and all, but we still have to rely on the Wi-Fi. If you want

to run this locally, we do have a separate copy of this at Zarf dev tutorial on GitHub. If you have your local Kubernetes cluster is really a requirement, you're going to need Zarf and the NIT package that's in the readme. By all means, feel free to do that. If you need help with doing any of that, we can help you get that set up as well. Uh,

but for sake of um instruct, we can kind of help you help provide you with focusing on the application today. Uh and we can gloss over some of the things we talked about on Tuesday which is like how do we initialize a cluster? What does that mean to Zarf? Uh we could spend a lot of time just on that section but we wanted to focus today on

the application. How can we make your applications portable um deterministic and declarative. Uh so just the last piece of logistics if you go and hopefully please uh head to the tutorial link. We'll keep that up this whole time. Hopefully as new people come in that's apparent or else we'll call it out throughout the session. Um, but two things. Uh, one, if you're having trouble connecting, if this

is just blinking in here and there's nothing going on in Instruct, hopefully this doesn't happen, but we just want to guard against it. Go ahead and hit refresh in Instruct the platform, not your browser. And hopefully this can reestablish connection. Should be good. Everything's looked really great and solid so far. Um, and if you have a question, uh, we do have our instructor console set up. You

can ask a question and do it through the platform or even just raise your hand. Hey, I have a question. I need some help. Um, we can have somebody come across as we got multiple presenters doing the talking And this is where I was going originally. Uh, why might you be here? Um and so there's a lot of the the fundamentals for ZARF application portability, right? If

we solve for taking an application to the air gap, you need everything all of it. You can't leave anything behind or else something might not work. You need it, you need the application itself and you need all of its dependencies. Uh and so when we solve for that air gap problem, we believe that you know it kind of solves all the way back in the stack for

connectivity. Maybe you don't need all the components and we allow you to kind of separate that UX in different ways. Um, but this provides us with an opportunity to provide this declarative experience which is how do I package up an application not only to take it to an specific environment but also I want it to be highly portable. I wanted this one package to be able to

be deployed to any environment. I don't want one specific package for this environment and for a different one for this one. We want these things to be portable. Um deployment determinism is the other part of this, right? You're declaring declaratively what you want to take this environment and how to deploy it. And this is really important, right? Um sometimes these Zarf packages as you'll see change hands.

They will go from the person who created the application or the person who works with the team who creates that application and it will it will change hands. We'll publish the ZARF package to OCI. somebody else is operations and they don't have maybe all the same context available that you might from the person who built the application and so there are ways with which we can build

various different applications in Zarf packages that allow us to cater to specific things uh maybe want a fully secure by default deployment of Argo CD that changes the scenario we might configure Argo CD in different ways um and so there's some benefits there we'll talked to the security vaults I just covered. Supply chain security is actually really neat and I would say I'm very biased in this

because it's something I heavily believe in. Um, but the opportunities here to wrap up at package create time and perform as much provenence as we can, right? We want to make sure that we can not only grab the things but that we can check them for provenence hopefully before they ever leave the collection phase. we should fail early as much as possible. And so we can package

up and grab things. We can check their providence. We can create sbombs for them that will get included in the package. We'll cover more on that. As well as we can sign the package. And I really like to pause there for we can sign basically everything that has gone into this package and put one signature on it as opposed to having to sign every single artifact along

the way and then have to go and verify every single artifact along the way. And so that's something Zarf is heavily working on. What if we verified at package creation time signed it and now we have a single signature attesting to the providence of all of those artifacts that were collected. So enterarf um this is a open ssf sandbox project uh so is donated to the open

ssf. It is an air gap package manager for kubernetes in particular. It can be used for other things. you can just bare metal transfer file, use file, transfer binary, etc. Uh those things are possible and very much a part of what we do, but we do focus on this Kubernetes full endtoend experience um to really lower the barrier to entry with deploying applications. We'll be using Zarf

today. You can abstract a lot of what Zarf does back to the tooling you may already be familiar with. Zarf deployments are Helm charts under the hood. You can look at the Helm releases when Zarf does a package deploy. you were never locked into Zarf and that's something we really like to communicate. You could deploy an application with Zarf today and decide never to use it again

and you could still use the Helm releases that Zarf has deployed for you through those processes. Um so one I think we've covered most of these topics in a variety of different ways already but I like to one think about this declarative package creation for those who weren't here on Tuesday. something we really highlighted I think um was that if we have this declarative manifest it says

what's going into it we can version control that in a variety of different capacities u but what's really nice is it's metadata right I can take this metadata and I can create a zarf package from it and it could be again a crazy zarf package could be tens of gigabytes that's not inherently as portable as the metadata itself and so we can put this zarf manifest into

version control and then at that point, you know, I can share it. Somebody else can take that same manifest and run Zarf package deploy and they will have the same output for that tarball that is sitting on your file system that you can then take somewhere or publish to OCI. And really the separation here is kind of just to help communicate that on the left side of

things we can shift as much of this responsibility left to the person who has the context about this application. How do you deploy something in is step one? How do you deploy something securely is step two. And oftentimes there can be a gap or a void in there. Like a a user base of an application may understand very well how to deploy the application. It says Helm

install in the readme. Wonderful. How do you configure that thing to maybe have security defaults? How do you configure it on deploy time such that you can actually prevent specific security gates from being violated? Those are all part of the experience. And so that's why the creator really has this uh packaging experience on the connected side to really try and get things as right as fast as

possible. Feedback loops that simulate the air gap uh and you know try to fail early and fast before you publish that and make it available to the deployer who really only needs to run Zarf package deploy. And at that point in time there's no changes to your Helm config. you don't need to change your Helm uh you know values for every single other environment. Zarf's mutating things

under the hood and trying to make that user experience as easy as possible. And we see a lot of resonance across the community for that kind of experience for let's say your users of your open source application or your customers of your application that you sell. And with that I'll pass it over to Austin. >> Thank you Brent. Well said. So we talk about getting into the

specifics of how do we actually do this? How do we take our application, put it into a briefcase and move it wherever we want? Next slide. We got to think about what dependencies does our application have. Thinking about a Kubernetes application in particular. Thank you. We know we're very likely to have Kates manifests and home charts. We know we're very likely to have OCI images. If we're

deploying something like Flux or Argo CD, which is going to be the example you guys go through later, we have GitOps and we have to get our git repos to be packaged alongside our application. And a lot of cloudnative applications don't need shell commands or other files, but some of them do. Some of them need to be configured through an API. And so to make it truly

all one experience, Zarf will also give the option to do that. And I think it is the big green button, not the laser button. And so how you're actually going to do this is Zarf is going to take the manifest or metadata that Brandt mentioned earlier. In that manifest will be all of these items specified. Zarf will take all of that, generate an sbomb for the images

and the files and then package them into a single tar binary which can be deployed by you or your deployer or an OCI so that we can have a community around it and anyone can deploy or you know make a PR to it. And just to give a quick example of this looks like um with saying deploying a flux application this might assume that you already have

flux in your cluster or you have a different zarf package that establishes flux for everyone in your cluster. You would have a package. It has metadata like name and version. Within that are components which you know you can specify where you make the boundary. But you know really the meat of it is going to be I'll use this fun pointer. This charts which is going to say

where do you pull this chart from? In this case it's a OCI home repo. the values files that you want to specify along with that chart and any images that you need or oh I was actually wasn't doing a flux deploy earlier um this is the flux deploy and this one is going to need the git repositories so that the flux release or flux git repository can

pull from that and it also will have you know its images and the files in this case it's manifests these manifests are the you Flux getit repository and then flux getit release. Uh and with that we're going to start the instruct demo. So once again the tutorial is at httpsarf.dev/tutorial. Click the start button and yeah excited for everyone. >> So I think from now what we're going

to do is if everyone wants to go to again zarf.dev/tutorial dev/tutorial. It'll kick off that instruct section with a local K3s cluster in your environment. Um, we're going to switch over. So, the terminal is going to be up here and then we're going to kind of moderate creating that Zarf package, creating the configuration, deploying it, inspecting it, and kind of doing everything from zero to deployment. Um,

so give us a few seconds as we switch. I'm going to play along with everyone too, so we can give some commentary. And again, if anyone has trouble connecting, just raise your hand. Uh, we'll walk over. There we go. Like two more. So, the first thing that Austin is going to go do is create that zarf.yaml. That zarf.yaml is the most important part of the configuration of

Zarf. It's kind of the the custom resource that is going to be containing all the documentation, components, git repositories, OCI artifacts, and then additional metadata, which allows us to kind of version everything from zero to infrastructure. Um, so what he's going to have here again is just the kind, which is going to be the Zarf package config, the metadata, which in this application will just be an

Argo CD deployment, and then the version 944. In this instance, the metadata here is specifically for the entire Zarf package, not necessarily the individual components themselves. So, as you kind of might have, you know, a couple Helm charts, you can version those Helm charts differently than you do the entire Zarf package. Um, the next thing he's going to do is simply add that component uh value. So,

what you he's going to have here again is the components associated to the Helm chart repository. So, um, looks like we got a couple hands up. Um, here you're just going to see Argo CD, the description, and then the primary value charts. Um, within charts, uh, you're you'll reference a Helm index. Um, and then associated to again the version of Argo CD here. >> And if I

can cut in real real quick, uh, if anybody is hitting a play limit exceeded error, we're going to work on that real quick. >> Do we need to increase that >> size? You changed it to >> So, we had a few people running into uh limits for instruct. They're in the back. They fixed it. So, you should be all in now. If anyone still has troubles, just

raise your hand. It's good. Great. can ask. So the next thing that Austin is going to run is a Zarf command that specifically is going to templatize all the different components and then search through those components to grab all of the OCI images associated to that. Um so he can go over into the terminal and he's going to run zarfdev find hyphen images. Again, what this does

is essentially templatizes all of the content and it searches either through regular expressions or kind of compiling those traditional Kubernetes custom resources or resources in general and grabbing those images. um he's going to get an output of another components uh array which will just solely contain all the images found within your zarf highle uh values and then we'll essentially want to take those and throw it back

into the zarf yaml. So in this case he has the Docker Reddus Alpine image. He has the Argo CD and then some additional artifacts associated to Argo CD cosign artifacts. He'll throw that back into the Zarf YAML and then just ensure that he only takes you know those images array as it's you know under the components and images. And then the next step will be um creating

the values for the helmchart itself. Along with the zarf package and zarf package components we'll ensure that we want to package the values for the helmchart. And so in this case that is what the baseline hyphen values.yaml will be. This will be like kind of the direct representation of if you kind of do a helm install for argo cd and referencing of values. So if he goes

over to the baseline hyphen values, what he's going to see is kind of again the base Helm values and he'll want to add a couple things. Um, specifically, uh, Zarf tries to package a bunch of different additional resources and commands to kind of make it the one-stop shop for deploying things within an airgapped environment. So this also includes additional helper tools such as uh, Zarf Connect. Uh

what Zarf Connect does is once you have and add additional annotations and labels to your deployment and application, it allows you to easily run a Zarf Connect application. So it's essentially you don't have to worry about which name space or you know what deployment and what service you're using. You can just run Zarf connect to your application and then to a local um and host port. After

he set that, the the create package is just zarf package create within the same directory that the zarf package exists in. So he'll run zarf package create period and then it'll kind of download all of the components. So the helmchart, the images, and then kind of bundle that into one tar.z. Let's see. It's going to take a little bit and hopefully we don't get rate limited, but

after this gets completed, um you'll just have again one tar.zst we'll be able to take into initialized zarf cluster and then also deploy forward. So after a few seconds, you should see again the Zarf package uh AMD 64944 tar.z and that will be everything that you'll need to take to deploy Argo from here on out um for the next few steps. Anyone have any questions thus far?

Well, we're running into Docker rate limits. So, you could do a Docker login, but I mean, >> yeah. No, >> if you fancy, but it should reset in a few minutes. If anyone else has a question, then just raise your hand. I'll scoot over. There Do we want to continue to module two? Maybe we wait a little bit for How's everybody doing? Anyone already finished by a

show of hands? Wonderful. Anyone not finished? But, you know, seeing who's paying attention, >> I like it. >> Yes. Yeah. >> Oh, yeah. You might you might want to put it in the values. You'll need it in the next step. >> And Austin, if you can Am I Yeah, I'm on. Austin, if you can go back to the just the code editor really quick on the up

there >> and just as a general call out specifically probably on the Zarf.yaml. Uh, but if you copy things from the right side, you might want to check your indentation on some of those. I think we probably like talked to it in some effect, but look at the indentation just to make sure it everything has copied over properly. Um, and specifically the if you copied the images

piece, uh, make sure that it's not doesn't have two images blocked for that single component. Any other questions? Anybody need a hand? >> Yeah. So, do you mind if I repeat the question? So, could you uh replace the Docker registry that we provide with the Zarfanit? It's already in the cluster, correct? The one you're talking about uh with something else? And the answer is ultimately yes, you

could. Um, and so we use distribution as a just a baseline default kind of uh largely accepted as the reference architecture if you will. Um you the Zarfa init process and package itself just to kind of step back a couple steps you need the Zarf binary you need the Zaranit package to init a cluster a single time and then that's putting the registry in there so that

every application deployment thereafter has somewhere to put images um if you have an external registry you operate in an environment that is hosting one for you could use that one you don't have to use the one internal to the cluster or you could take your init package and replace the components. Some of that components might need some customization with how it like maybe you do off or

some other things but very much Zarf has a uh for package deploys we need the ability to write to that registry and then for u the actual application deployment when it's being pulled by the cluster we need a pull credential as well and so we do separate that by default we try to maintain like a security stance so that we don't put a credential in the registry

that can write to your registry credial on the cluster that can write to your registry. But that's a really good question. The ultimately long answer uh the short answer is yes. And while we pause, I'm willing to take any other questions about Zarf generically or if you need a hand with something, just again feel free to raise I have a question here. So, >> so this might

be a lazy question because it might be right in your docs, but since there were no other questions. So in this example we're using um Helm charts and I have kind of two related questions. First one is um when and how does the Helm templating actually run? Is that something internal to Zarf or does it leverage a Helm executable somehow? And then I'm also wondering does it

happen on creation of the package or deploying of the package and can you choose that? Yeah, it's a really good question. Austin, I'll let you take this one because I feel like you're the expert >> Yeah, so Zarf uses the Helm SDK very closely. It the binary is actually built into Zarf for convenience. You could do Zarf Tools Helm and use as long as you bring the

Zarf binary along, you also have the Helm binary as of the latest The Helm chart is not um I believe it's not templated on create. We can go through the we'll actually see for sure in a later module. Um but we wait until deploy bec actually 100% we wait until deploy as on deploy you'll be able to kind of give variables that'll change the potentially change the

value so we don't template the manifests on create because you know we want some level of uh dynamic deploy configuration and so I think that's your first question um and I guess I answered the second one too with the helm SDK so yeah great question >> um okay thank Thank you. And I was wondering a followup. Well, okay, I have another followup >> second. Okay, so we

have Helm. Uh, does Zarf also um support uh customize in any way where you could or or maybe some other mechanism that fulfills similar uh mechanics uh where you could uh do customizations. >> Yeah. So customize is supported. You can do just generic Kates manifests. We do end up wrapping that in a Helm chart. So we can get kind of roll back and some of the other

stuff that Helm gives us for free. But yeah, there's another key in the Zarf package. Um, instead of charts, you could do manifest. And I don't remember the exact schema off the top of my head, but I believe it's, you know, something like name files. Within manifest, there's also an option for customize. >> Uh, yeah. >> Very good. Okay, last question and then I think there's already

another question. Um, regarding Helm, so we established it happens on deploy, which I think is logical because then you can do some additional values or if you need to. Uh, does it then always do a Helm install or can you also opt to say, okay, I do want to leverage Helm, but I want to do a simple QCL cubectl apply uh, so to say, and not use

the Helm mechanics uh, for deployments. >> No, that that's something we're considering and we might do in the future, but right now everything's managed uh, through Helm. Thanks. >> Great. There was another question somewhere. >> There we are. >> No question. You need help? Okay. >> Okay. I'll come over there. >> Is anyone else still on the create module? Anyone else want some more time for that?

>> I have a microphone for you if you want. >> Right. But now the question would be um can we in general package any OCI um artifacts? >> Yep. So one use case we actually have sorry I think it might interrupt you. One use case we have is for flux OCI repositories and I think we have that example in our repo and so you can package up

a flux OCI object and deploy it you know through fluxes >> in general any other OCI >> any OCI object. >> Thank you. Yeah, with a caveat there need something in the deployment paradigm to be able to do something with it and you have to kind of orchestrate that that yourself, right? So for instance, you can include cosign signatures as OCI artifacts in your images selection. What

are we going to do with them? It's kind of up to you. And so we'll deploy into the registry on deploy and then if you want to, you know, have a policy engine enforcing on those things already existing then yes. >> Yeah. Exactly. Say for instance our custom operator will don't just pull uh pull this OCI reg uh this artifacts from this temporary registry right >> exactly

>> great question another one >> good about >> you need help or you have a question let's go >> you want to hold the mic >> uh yeah sure >> um is there a specific reason why you explicitly need to do the zarf define images command and why it's not automatically. >> Yeah. So, I'll take this one, Austin. Um, in particular, you don't have to do find

images, right? If you know what they are or you're the maintainer of a Helm chart or what whatnot, you can put them in yourself. Uh, the concept for find images is that we see a large part of our user base who is integrators of software. They maybe don't own the Helm chart and every time that there's a bump in the Helm chart, they're like, "Okay, here's what

I need to do. Find images is merely a helper function that'll be like, okay, well, what can I discover about this Helm chart in the manifests by templating them out in order to find the images that are in the values or somewhere embedded throughout the system. So, you could even use Helm annotations if you wanted to. In the case of operators, we see this a lot. Operators

don't often define all of the images that they might use or need. Um, and so find images is really a hey bump the helm chart. I want to find images real quick and get the output so that I can repackage and go. One one other thing I I'll mention there is we do have I don't think it works with instruct because just some weirdness with the terminal

and code editor but we do have the new option for zarfind images- update where if you don't have those operators with kind of maybe hard-coded images another use case where we can't just assume what the images are and we have to be keep it in the manifest would be something like a git flux git repository where your flux git repository points to git repo. Zarf doesn't have

access to that git repo. So we don't know what the images are ahead of time. And so that's why we kind of don't have that as the default option because we don't want to override images that the user is putting there. One thing to note um as an end user of Zarf uh if you use a manifest that points to you know uh multi-archch the zarf does

not resolve the multiarchch uh hash for you if you use a hashbased approach which I highly recommend. So you have to figure out um using other tools perhaps maybe we can build that into Zarf should should do that to figure out what the actual um hash or content hash for the image you're you're trying to get is so that it's able to pull. So that's a caveat

that I want to share. >> We we do have an issue for that and yeah it is it is on our road map. It it gets kind complicated in the air gap. >> I know >> but yeah that's that is definitely something uh I would like to see. Somebody else not finished to completion. >> You need help or you have a question? Is anybody still being rate

limited on >> still? >> Yeah, a little bit of a rabbit hole in order to change that image out and then put it in your values file and then recreate the package. Yes, appreciative of that. Uh the scale of the surprised us a little bit. Uh and it was a last minute concern that we had uh that we were hoping wouldn't raise its head, but it is

what it is and we're And this could be a cool also opportunity to point out that like because we used the internal registry at the point with which you've done a deploy with these images that if you ever were getting rate limited by something in the past and then maybe this was a common concern that we are essentially populating in your cluster a registry with those images

so that they never have to be pulled externally again. And so for a variety of different reasons, maybe you know AWS DNS is down or GitHub is down or DockerHub is down. Uh we've all seen this in the past. Hey, I need to pull an image but there's an outage somewhere and I can't pull my image. I need it. Uh this kind of provides for some redundancy

and better availability. I would argue >> question or help needed >> question. >> Question I'll repeat it back because we don't have a microphone. Ah let's use this one. Let's get it on the recording. >> Hi thanks for the presentation. Um so you support Helm charts natively. Uh you said you support uh customizations. Uh do you also support all the sub tools available in customize like helm

inflation uh tools or uh patching uh whatever >> if in uh I would say in different capacities are they plugins in particular? I don't claim to be a >> No no just native tools uh native to customization. >> If it's native to customization it should be supported in some capacity. It would be interesting to probably go through all permutations to see what does and doesn't work with

regards to when is it being template like when is it affecting the templating uh was we want to I think with customize we have a limitation where on create is largely when we do a lot of the templating and because of like how we have to run the so we can't do as much on deploy time um that's more of a us limitation not a customize limitation

But uh we don't honestly see a lot of issues come through for customize. >> I'll just have to uh test it then. >> Please. >> Yeah. Uh I know I know we do have an issue for shifting customize to deploy time. Uh there's uh I think a few caveats there, but it's something I think would be reasonable. It would also make more sense with how we do

our variables and uh be more similar to Helm, which is our usual mechanism. R limiting solution >> change the tag. Okay. Okay. So changing to use Reddus from Quay. Now we now we can go to deploy the surf >> Let's do it. >> Yeah, everybody is done with creating >> And if you're not, that's okay. We can start we can try and filter around and give you

maybe some of the workarounds. Uh but for sake of time, we probably need to work through deploy and inspect and then we can do our package playground as well. >> Yes. So um before I'm starting with deploy the package, I was thinking about my hobby dive. Uh before I'm going under there, I check my equipment. So the first step is to check the environment. You can do

that uh with the first two commandos um comments sorry uh ls and um you can also do the sarf tools cubectl cluster info. Uh the first command uh you can check if your package is still there and the second one you can check if your um kubernetus environment is still available. If you're yeah when you're doing that then you see that there's one control plane node um

then you can uh do deploy the So when you're um using that command then uh you see the the yeah the package config that's the config that you going to um if you're using the yes deploy this zarf package then you see that they pushing the images and they deploying the package on the cluster it takes some time. Everybody so far so good or are there still

troubles with pushing the images? Yeah. At this point, we just have to wait for everything to reconcile. So, Argo CD is deploying. Oftent times I'll spin up another terminal and do a wash of cubectl or Zarf tools K9 which is a built-in binary to Zarf um does K9 to let you monitor. But looks like we just wrapped up. >> Yay. Woo. So if you want to check

if your pots are up and running, you can use the uh the next uh command. Um Zorf tools also uh you can use with them the cubectl and when you're doing that you see the pods, the services, the deployments, everything is up and running. Um there are some points above uh the page you see a tab applications. So when you're um doing a get uh secret you

get the secret to get access to your admin dashboard. Um in the tab application you can see above then you can um get access to the application. >> So I'm going to click application right in the terminal right after I run my Zarf Connect Argo CD. >> The username is always >> admin. >> Admin. And woohoo, we have Argo City CD right >> That everybody has access

to Argo City or are there any any questions? >> Yeah, we can open up back up for >> there. Um, oh no. uh something I think we have been like kind of glossing over is there are things built into Zarf uh that maybe we haven't called out in whole there are quite a few things built into Zarf actually um but we talked about canines and cubectl right

we talked about helm these are all built in now integrated into the zarf binary the idea here being very much that if you are going to an environment do you want to bring zarf and the zarfit and helm and cubectl and K9's and X Y and Z to that environment or you just want to take one binary and so we try to make that experience a little

bit better. It does make our binary a little bit larger but it provides people with a little bit of redundancy of not having to manage tools and versions in this environment. Question. >> Yes, I've got a question. Um when soft copies all these images contain images to the nodes I suppose it copies it to all the nodes. Can you specify whether or not it should um through

labels or annotations or whatever which particular nodes you'd like these images on because some admin stuff should be on these nodes and worker loads should be on these kinds of nodes. >> Yes. Uh great point clarification because we we glossed over in it uh a little bit here. Uh but for sake of a package deployment when those images are being pushed into the cluster they are being

pushed into a registry running in the cluster. So it's a single registry maybe it's an HA deployment of that registry. Uh but it's a reg registry such that it can host those images and make them available to maybe other services you want to run in your cluster um via the internal service name or in the case of Zarf um we have a couple different deployment mechanisms but

we make it so that the cublet and the container runtime can pull the image from that registry and then whatever node needs it at the point of scheduling time is the one that will receive it via native Kubernetes processes. >> Okay, perfect. Makes sense. Yeah. >> Are there other questions? Uh over here question. >> Let's do your mic. Thank you. >> I got a question about the

package itself. Is there a way to verify the integrity of the package like is it signed do we check some or any other way to make sure that uh the content of this zip file or whatever right u it's not >> yes this is my favorite question actually thank you um so this happens in a variety of different ways actually um so when we are creating the

package every file that goes into the package is checksummed and so now you have a checksums file that's in the package. We'll actually show this in a little bit in the inspect process, but every file gets a checksum. And so there's a checksum file that contains all of those. So when you do a Zarf package deploy, we actually run do run through that checksums file and ensure

that everything has a checksum appropriately. The Zarf manifest that you create with your configuration is put in the package as well. And there's an aggregate checksum which is the checksum of the checksums file. And so now we're doing and checking for the checksums file, the checksums and the checksums of all the files. After that or even during the Zarf package creation process, you can sign your package

and at that point in time you can establish a signature and a signature what is really happening is you're signing the Zarf manifest. The Zarf manifest then has the aggregate checksum. The aggreate checksum has the checksum for all the checksum file like everything there. So we try to create this full structure here where ultimately yes the integrity is something that we very much emphasize from the start.

>> Exactly. Yes. So to repeat the question, can you use the signature when you're loading? Um yes, we will automatically try to verify the package signature uh whenever you do something with that package. Even if you're just looking at it, right? I want to inspect for the intransparency. Um, we will try to there can be a lot of variables involved with verifying a signature. Um, and so

it may be something that you need to pass in >> Exactly. You need to pass in some additional metadata in order to verify based on how it was signed. Um, but yes, that is a great question. >> And if you'd like to see an example on screen, um, in our metadata, we have an aggregate checkum. Um, and if you are to decompress the package, which we'll show

later, but since you asked, we'll show now. Um, that checksum is a check some of this file that has all the subcheck sums in it to build a little bit of a tree. And there is an enforce flag as well on deploy that you can use to fail. I want to fail in the event that I cannot do the signature. I cannot verify the signature. Uh we

will always fail on integrity checks. Uh so if there's a file where the check sub doesn't match, boom, automatic fail, do not proceed. Uh and we do that at the very beginning. Uh but with your signatures, there can be variables involved with what availability you have in the air gap to get this the signing material or the verification material, if you will. Um, so we do allow

a very obvious warning in the event that you can't. Um, but you can put enforce and it will again it'll fail. You can't proceed any further without verifying the signature as well. Is everyone past the registry override rate limiting issue? If you're not, happy to help. So I get two people asking the same if you created your package without actually supplying the images, you're not bringing all

the dependencies back into your cluster. Um you cannot go back here. So I would then just say restart the session and rebuild it with the images in there because you for some reason we delete the the tarfl for for your package. Um in any case um you will get stuck on deploying the helmchart because the images cannot be pushed to your repository and as such it will

fail. So, anyone running into that issue other than these people and don't be shy about it. I make mistakes. It's good feedback for the session as well. I think we tried to tried to build in some guard rails, but there might be there might be areas we can improve. >> And if you want to know how to restart the session, just go back to Zartodev tutorial, click

stop, click next. Um it there is a hot start still available for everyone. Um if you run into rate limit issues, please give us more IP addresses. While we have more time, I guess, uh, Austin with K9's earlier, you were showing what we have deployed to the cluster. Um, we've talked about it in a couple different capacities now, but at the bottom there, you see like gy,

we talked about the registry. We talked about Zarf and GitOps being, you know, very compatible. It's kind of fun to point out like how portable you could make GitOps in itself. It's not a very like difficult problem to solve with bringing things along, but there are some nuances that I think really make Zarf shine, and it's the mutation that we do, right? Again, earlier mentioned we have

the helm charts. We don't want every single person deploying a Helmchart to their environment to have to override the values to say, "Hey, here is my registry for my environment." So, we mutate that when you then do any Helm installs thereafter or Zarf package deploys in particular. Um, those will all those images the images references in your pods and uh for your deployments and staple sets etc.

they will be mutated just to point to the internal registry. No configuration change necessary. We do the same thing for git repos. Um, so you can include git repos in your declarative manifest. We will clone those repos and put them in the package. We will then on package deploy push them into a git server that is running in the cluster. We use gy. Um, and this is

you know a phenomenal time to say okay well I I use flux or I use argo. In the case of Argo, um we will be you know looking and monitoring with the m mutating web hook uh for various other resources besides just the one the you know resources using container images and pod like pods. So we will look for you know flux resources. is we will look

for Argo resources that we are aware of that we can automatically change and mutate so that it points to the internal registry in internal git repo in this example for git and just make that experience hopefully a lot uh easier a lot more seamless >> so question here um Maybe you can answer that over here to left or right. Yeah, left. Got that messed up. Anyway, um

Ilko here, he he was wondering if you can actually uh use a registry not provided by Zarf. I mean, it's called YOLO mode, I believe. >> Yeah. Uh great question. So, you definitely can use a registry not provided by Zarf on Zarfanit. we have several flags. Um, this will allow you to specify your registry. Um, so you can give a registry address or registry URL. Um, as

well as the credentials that you need for your registry. Zarf will still in this case do the mutation, but instead of doing it to the internal registry, which is no longer spun up when you provide the registry URL, it will be to your and uh you uh want them to adhere to stain defaults like uh uh immutable container images and immutable tags, stuff like that. >> Yeah,

we we do want that >> forced I suppose, but >> it's it's not forced. We do have some guardrails in uh in place to kind of avoid taking different images um from that you're not supposed to, but I'm sure you could figure out a way to to mess it up that we won't stop. Yeah. Yeah. And that can that registry that you deploy into the the Kubernetes

system uh that can also be used as a regular registry y >> from within the Kubernetes cluster. >> Yep. And so there's >> from the outside. >> Yeah. There's a service we point uh we expose that >> as long as you have access to that um service, you'll be able to call the distribution registry we stand up in the cluster like normal. And and I think there'll

honestly be more use cases for it as time goes on. Um, usually with Darf it's just used to pull the images. Um, I think Flux is a great use case with OCI repositories, but as OCI images become more generic, maybe with cosign, maybe with ASBOM addestations, um, I know OCI 1.1 it introduced artifact types, right, that you can maybe attach to your OCI images. And, you know,

we'll we usually stay on top of that and see what we can do, but yeah. Great. Thanks. >> Yeah, that one's really fun. I love kind of finding different services that we can integrate internal to a cluster. Like Flux is always a great example because instead of round trip like pointed at a service that's external to the cluster just so it goes back in, we can just

use Kubernetes native internal like networking and just route straight to it. Uh I had a presentation at OpenSSF community days in Denver last year u that kind of represented the same thing. I want to be able to push my sbombs into the registry as I would any other registry so that I can use other services and integrate them with Zarf without them being strictly even you know

knowing of each other. And so I used Guac in this instance deploy Guac in the cluster. Guac was watching the registry. Zarf was pushing images and sbombs to the registry. And at that point, they just seamlessly mesh together. And there's a lot of other interesting opportunities that we would love to explore. >> And if we really get into the nerd stuff, I think you might have been

alluding to something like this, but actually it's quite pertinent for some rate limiting we're seeing. But imagine you had a package or two packages that don't know each other pull in the radis image from Quay, you know, from GHCR and from DockerHub. If we push all of those without a checksum, you would think there would be a collision, right? And so we saw that problem and what

we've done really really for a while in Zarf is when we push that image, we push the image normally, but we also push it with a checksum of the initial registry name or of the initial image name. And that way when we mutate it, we mutate it with the initial image name. We add this you know dash checksum to it so that people who are using tags

and multiple images with the same name from different registries don't run into that conflict. Very astute question. I'm impressed. >> And there's also a labeler annotation that contains the original image. Which helps with mutation web hook problems which happen sometimes because hard to track everything going on. I think if I Yep. So now we know I use Docker here. If you used Quay that would show up.

>> Should we move on to inspect? >> Any questions? I think we still have folks roaming around. Multiarchch. >> How about >> multiarchch? Yes. If you want to ask your question for the recording. >> Inspecting looking at the inspection of the the manifests. It says architecture. It says AMD 64. What about multi architecture packages because we love Raspberry Pies on ARM or >> Yes. Something that has

come up in a number of our issues. Uh Austin, I think you have some experience with kind of navigating some of this terrain. >> Yeah, so multiarchch isn't really natively supported. I believe some people have made it work. There are some kind of interesting problems that I think we're close to solving. We have introduced we've started using Oras Go, which which I think gets us a lot

closer. We actually stop. We don't allow you to build a zarf package with an index shaw multi-archch essentially shaw that points to the generic um points to essentially all versions of the OCI image. Getting a bit into the OCI spec here. Uh the reason we do that is because oftentimes images like EngineX images like uh EngineX you know they might have 10 12 images under their index

jaw and so if we were to natively allow you to kind of bring that in you would have to publish that you your package size would explode right and we didn't want to give users that foot gun especially because not everyone you know understands the OCI spec at such a um kind of minute level and it's not a case people run into often right usually you're in

connected environments I think there are there's an issue to fix that right now. Um I think there are good options. One, we could just kind of have users specifically opt into that maybe with an arch multi. Um and so okay, I accept this risk. Maybe I mirror just the images I need to a specific um to my GHCR or my you know registry on the connected side

and then pull it. um some registries I think distribution 3 has the option to send an incomplete index shaw and that you don't need every manifest and it won't error when you do that. I don't know if the spec kind of allows that but I think registries probably will start doing that for this use case. Um so it's definitely something we want to do. You could bring

in the Shaw separately and you might have to figure out how to deploy your manifests to different nodes maybe with a node selector. I assume you could do but yeah it's something we want to do more natively. Not yet. All right, next question. >> Hi, thanks for the tutorial. I just have a question related to if I have more than one zap file in the same directory

>> like you have similar one for docker where you just write the different names in dot add docker file and you can just switch the file names. So I was trying to create like to check if I could create a new file and in the create package just give the file name and uh I just checked around that it doesn't exist. It's like a feature request I

guess. >> Oo could you maybe elaborate for us? Um in what regard? Like you want to create a newarf manifest file. >> Yeah. So if I have more than one file uh how do I differentiate to say like okay I want to work with this file but I don't want to work with this file because I already created a package for >> Yes. So if it's um

you should be able to kind of name it arbitrarily. Uh so if you have a zarf.yml and that was for our argo earlier or maybe we accidentally blew that away. Um you could create a zarf-1.yaml YAML um and whatnot. And you should be able to pass that into Zarf package create. Uh by default, if you don't pass it anything, you just do that here, the dot uh

for your current directory. It's going to assume the zarf.yamel. Uh but you could give it the path explicitly. So path to my arbitrary.yaml file, and that should work for create. If it doesn't, please let us know. >> Okay. Thank you. I think we'll we'll move on to the next Make sure my check works. Looks like my check work. Keep things moving along. also if anyone has come

on into the room later on uh and you want to get to the tutorial, it is zarf.dev/tutorial. And because I've tried this before and I've seen it hang, if I just delete the page and click continue, it'll put me in the right spot. It saves progress. Sweet. We're at inspecting a Zarf package. So, we want Zarf to be easy to inspect. We want it to be transparent.

I think I showed this a little bit earlier when um the question was around how do I is there like built-in checksum verification to Zarf. There is. Um, we could show that with something like Zarf tools archive decompress. Lets us decompress this Zarf package. This is a pretty advanced use case. Probably you're not going to do this, but it is nice to see what's going on under

the hood. Um if I was to let's install tree why not show a tree of the zar package. We can see we have a OCI layout that stores all our images. We have uh checksums.txt which builds a tree of all of our files that are in the package. So we can have a check some of that file and therefore have a Merkel tree that where if anything

changes we kill the deploy. Um we have our sbombs which we'll talk about more in a sec and we have our zarf.l and so you probably don't want to sort through a tar file. So we have some helpers with zarf package inspect. If I look at the help for that we'll see the options we have here. definition. You might have saw that earlier. I did that to

show the checksum. But it essentially shows the zarf.l that you If I run that from the correct directory, then I will get autocomplete on the file and we see the definition. Uh the nice thing about this is if you're in your aircraft environment, you can see totally what went into your package. You can extract it, you can pull it out, you can make your own package, take

it with you. We want everything to be transparent. We want everything to be easy to make using the tools you already use. Helm, Git, Docker, etc. another command I quite like is Zarf package inspect manifests. And so you can see what's actually going to go into your cluster. This is especially useful if you have ZAR variables, which you know we didn't get into. We wanted to give

an overview, but if you have a dynamic configuration for your Helm charts, you can do somewhat the equivalent of a Helm template and know how those configurations are being applied. >> This takes a while, I think, because the text takes a while to get from, you know, your local directory to instruct. Usually that's instant. ends tomorrow at 9 p.m. >> Um, similarly, you could do values Um,

if you have dynamic configuration of your values files, it'll show the final values file that you give to Helm. >> Kindon, correct me if I'm I'm wrong. Maybe I just missed it if you mentioned it. This can help us diff between two packages that are built to see what is different here on the outputs of what's being deployed. >> Yeah, that's one of my favorite use cases.

Let's say you change from one of your package versions to the next something about your manifests and you want to make sure that that's configured correctly. Zar package inspect your manifests from OCI. So you can actually you know do deploys and even inspects directly from OCI. Um okay I will have my local version. I'll cat one of these to let's say GCR- my package um original.txt and

then diff.txt. txt. Do a diff between those, see what's different, and hopefully that's, you know, useful for your workflow. Another thing that I think a lot of the security-minded folks uh will care about is inspect sbomb. And so this will add to your directory, copy into your directory from the zarf package a directory of your package name. And this has sift sbombs uh into it. So I

could cut one of these sbombs, pipe it to jq and we can see the um sift asbomb because zarf packages sift as well. Uh we can convert this sbomb into a format we want. And yeah, Zarf also comes with HTML files to view the sbombs in a more human readable way if that's something you're interested in. Uh and we can also inspect at least the definition of

a package that is deployed in the cluster. So if I do Zarf package list, I can see every Zarf package that's deployed to the cluster. The init package which we had by default and Argo CD which we deployed in step two and I can see the definition of that just from >> I can also remove that package. So again, if I was to do Zarf package list,

I could run Zarf package >> remove Argo CD. It'll prompt me. I'm going to say yes. >> And successfully remove it. >> SVC basically all. >> If I run Zarf package list again, it's all gone. We just have our lonely in it >> baseline server there. And if you go in there, I think it's >> going to continue on from here and and kind of get to

the fun part. >> Here's the last module. This one's free to play. You have a Zarf binary. You have your existing directory. Um, you have some change me here. Feel free to make your own package. >> how how long do we have? >> When does this session end? >> Five minutes. >> Okay, so you have five minutes to to build your own package. I I've seen packages

built faster. Um, trust me. So, I think you can do it. Maybe if it's your first time with Zarf, it'll take a little longer. >> But, >> yeah, as we as we walk around, any any questions, anything you want to know about Zarf? >> Um, obviously it's free and open source. should always be free because it's in a foundation. >> Docs.zarf.dev. >> Hopefully it'll answer your questions.

If you think it can do anything better, >> make an issue, make a PR, get some stickers over there. Some of my buddies at Defense Unicorns brought them along. question was had can you add additional scanners to SARF tool scan and can you add additional I guess plugins to the scanning functionality >> yeah so there's a there's a question out there about can we add additional can

we add additional scanners no at this time we always use skiff sift I think I believe Zarf tools scan is an alias to sift if I was to do the the -h. Um or no. Okay, it's just Zarf tools sbomb is the alias to sift. But it is something we've thought about is like do we always want this to be the esbon format? Brent knows a lot

more about that, but sift at least does have the ability to convert it into different formats. So if you have opinions there, make an issue. We definitely really love community feedback. Anything you want to add there, Brent? Uh, no. I think I think that covers it pretty well. We've got some ideas about what we want to do and scanning is kind of something you could do very

broadly, right? You might want to integrate something that looks at those sbombs. We have a variety of kind of prototypes for that. Uh, prototypes for changing the format comes with implications then on how are those ingested, how are they stored, etc. Uh, how do we even toggle between different formats when you want to select one for something and one for the other. A lot of that's a

work in progress and a lot of that is you know kind of just why we believe Zarf is a great alignment with the open SSF the open source security foundation for those un unfamiliar uh and yes if you have opinions if you have thoughts uh we are working on a lot of various like supply chain integrations and problems that uh we would love to hear more from

the community. I think we have time for one more question if somebody has a question for All right. Well, besides that, we didn't have time for the playground. Uh we can hang around up front here for a little while unless they kick us out. Uh and if you have questions, thoughts, ideas, uh or you want to contribute, by all means, let us know. we can help you

find good first issues. Uh and we are always looking for, you know, return contributors who want to come and be a part of the community and one day, uh possibly be a maintainer. So with that, thank you for attending. Uh hope to see you all next time.