Django Beek - Building minimal images with Wolfi OS, declaratively with melange and apko

This talk focuses on the need for reproducible software images and the trustworthy creation of those images using open source tools. The speaker, Jon, a sales engineer at Chainguard, discusses the challenges related to supply chain security, specifically the complexity of managing software vulnerabilities through various vulnerability scanners. He demonstrates the shortcomings of existing scanners, highlighting issues such as metadata manipulation and the creation of deceptive 'Frankenstein' images. Jon introduces tools like Melange and Apko, which facilitate building reproducible APK packages for the Wolfi OS Linux distribution. He emphasizes the importance of verification over blind trust in software and encourages attendees to explore how to implement secure and reproducible builds in their own applications using these tools.

Test, test, test. Ah, no. I see someone. All right, good. Um Yeah, honestly, it's a bit weird because I can only hear myself. >> [laughter] >> But cool. Thanks for you couple to join. If you want to come more to the front, you are free to do so because there's nobody else there. Although you can probably see it from afar. So Um I'm Jon I am a

sales engineer at Chainguard. Um but we're not going to talk that much about Chainguard today. We're only going to talk about the open source tooling and the Linux distribution Wolfi OS that we have open sourced at Chainguard and how you can get started building these images yourself with the tools available and also the why of this all and that's actually what we're going to start off with

today. So just I know everybody knows this already, but just to double down on the supply chain issue and then we'll start looking for who we actually trust, who we actually cannot trust. Can we trust anyone? If it's not about verification rather than trust and how you can produce these kind of images yourself that are fully reproducible in order to overcome blind trust, so to say. So

first things first, something you all will know, right? The the software development life cycle. Um we're not going to go through this at all. The only thing I want you to to see here and notice is that actually in every part of this cycle you think about what kind of open source foundation am I going to use to build or publish this this this piece of software

whether it's images library dependencies or language dependencies or VMs. And if we flatten that software development life cycle, we'll see that actually and everybody knows this of course here, we pull this well, the status quo is to pull this from the internet with all of the issues that come with, right? We've seen it the last couple of months. First Trivy, then downstream we've got light LLM, max

and so forth. Um and that's a major issue because often it's a a trade-off, right? You have a critical CVE perhaps that you want to remediate as quick as possible, so you want to get the the latest and and and greatest to remediate that CVE, but at the same time you might have a cool-off period for your libraries for 2 weeks, for example, in order to well,

refrain from pulling in malware. So, either you remediate your CVE, um but perhaps pull in North Korea with malware, or you have your CVE there and wait that that sort of the cool-down period for perhaps malware to be spotted already. So, there's no best case. The question Well, and if we make this problem practical, uh won't spend too long on this because we want to get technical.

This is sort of the the reality of the operation, right? The full-blown red dashboard with CVEs that every day need to be um remediated unfortunately by engineering teams that we like to see just develop features, right? And enjoy their developer experience. So, this is what we tend to see, that doom cycle where you build a certain feature in application and and upon scanning you're back to the

drawing table to patch and remediate the issues that you need you that you don't want in the first place. >> [snorts] >> And from a risk perspective, this is obvious again, I don't need to dive into this. I don't want to spend any more time on this, but it is growing, right? And interestingly, we still see well, millions and millions. This is just a plain example of

still vulnerable Log4j downloads, the one that everybody knows about, right? Um and then with AI, this sort of the time from vulnerability to exploitation, uh well, it's just exponentially um lowered. In a couple of days, it will be perhaps hours up to minutes that we exploit them. So, the question is who do we trust when pulling in these kind of software artifacts? >> do we actually trust

anyone? But first, I would suppose that nobody actually trusts, I guess the internet, where they pull from. Um especially the with the experience that we've seen, right? So, what actually do we trust then? Usually, you have some kind of trust in the scanners. At least if you have to trust something, then you're likely trust your scanner to say this is wrong. There's a CV error or there

might be something of of of malware in this. But scanners are fortunately no magic touch, right? So, let's talk a bit about scanners and introduce those those those I guess the problems, the dark slides for a dark reality. Um ah, there's apparently a click-through. I didn't know there was a click-through here. Uh but scanners what do do CV scanners? They rely on metadata. Um only that such

to say binary names or metadata. And they cross-reference the the the software versions and what they find. They cross-reference that with databases they are following. Um vendor databases of those images or public database database like the NVD. Um what it is is quick and it is very scalable and that's exactly what we need. It is almost impossible, although perhaps in some years sort of an LLM might

be able to do this, but at this point it's really impossible to search um an image for specific binary patterns in order to to to identify a specific vulnerability. >> Um that just also uses in massive resources. Again, it's not That's why we have CV scanners. So, uh it is entirely metadata dependent, right? So, that quality needs to be in order. Something that's not in the metadata

will not be flagged as such. Um and because they don't have context, they could bring false positives, right? So, either you miss one or you might have one too many. Um this is just the the sort of the the the reality of Um and I just want to just as a an example, let's let's run one example here. So, actually what I want to do Oh, interesting.

Um yeah, I'm going to run a I'm going to remove this here and just have a small compare script that I want to run in the background. Come. Typical when typing live, right? There we go. Um so, very simple. Hope you can all see it. I can make larger if need be. Um they're just three base images uh that we're pulling here. I should have run this

a second ago but because it might be downloading some databases right now. Well, we'll see in a minute uh and I will skip a bit to the results already. We'll see discrepancies, right? Because we're uh running these with Trivy and with Grype. Um they have a different approach for scanning. And a couple of things that you note, well, first of all, none of them have the same

output, unfortunately. Um and for Alpine specifically, you'll see none for Trivy. And that's because Alpine specifically only outputs advisories for those vulnerabilities that actually have a fix, an explicit fix. Um well, in this case, so there the Trivy only uses the advisories of, in this case, Alpine. So, Trivy doesn't use, for example, the NVD. And and Grype does, so they are brought up here. So, just just

to show the discrepancy, right? And I guess already already went to in a couple of these con- uh conclusions. So, the thing is and I just wanted to show this to show what we can do with this and how Trust can be manipulated in that regard because you can very easily uh create a a wrong image and and act like it is perfect. So, first the the

the metadata manipulation. We can just delete or change certain metadata that the package manager would read such that the the whole piece of software is not not findable, right? In that image. So, the the scanner won't output anything of it. Um second, and we've seen this actually, I'm going to show some examples as well, what we call Frankenstein images. So, essentially, instead of using a package manager

or a compiling or a a a composing tool, which we're going to talk about in a second, um they just drop in binaries that the package manager is not aware of. So, if you scan it, you might have the binary but there is no no manifest or no stanza, depending on what kind of metadata you have in there or what kind of image you have to actually

report on that image or that software and that version. Uh Um it might not be malicious. The first one can definitely be malicious. You can hide a CV and then later use it to exploit. Uh the second one you could also say it's just a bad practice because the moment in production you want to pull in additional software, your package manager might break because it doesn't know

something is is uh actually installed or not installed. And the last [snorts] one is is suppression of of CVs by Vex. So um this really relates to uh um if you build an image and you derive your base image from upstream such as a Debian and Debian says, "Hey, actually this CV we're not going to solve for this version." So three, four, five, six months later, um

then your derivative could say, "Well, actually Debian is not going to fix this, so we don't have to fix this. Um we'll just suppress it and therefore it won't show up in your in your scanner." False sense of security, of course, because that CV is actually there and exploitable. >> Um it's just suppressed. So um let's look at a demo. Where is my mouse? I'm wondering if

this finished. Ah, yeah, it did, of course. Um I want to show you some As As you As you've noticed, I've decided to just script it all instead of doing it live. Otherwise, it would be a uh an interesting hassle. Going to make this a bit bigger again. Um So the first one is we're going to Oops, scroll a bit further. We're going to uh delete that

stanza, right? So that that metadata. It's a scan of this Python 3.12, just a a plain Docker. Uh all Well, it's an old one. There's a lot of CVs. There's a subset of CVs criticals here that we see. And here's an example of what the stanza is. In this case, DPKG, so for Debian, uh this is what it looks like and this is what the scanner reads

in order to to infer, well, the version um and then cross-correlate that with a database. So if we would delete this, so we have a couple, so we have a couple of stanzas here. And actually we're going one step further because some scanners also read binary names, we're also going to rename some binaries such that if you would search for Python, you wouldn't get it. Uh but

that the actual binary is still there, but in a different name. So, as an attacker, you could still exploit uh the CV or whatever brings that that binary brings, you could still use it to exploit. >> Um so, essentially what we've done is with these very two, that's also an interesting element, very simple commands, we've manipulated the image to in the end well, it's an arbitrary arbitrary

number. We could have done this for all CVs, such to say. But essentially, if you will scan this, um you'll get the the after output, right? And no scanner will recognize that you've done so. So, the second part is um what I talked about breaking package managers. So, essentially, if you drop in uh a certain binaries, your it it can mess up production. So, here we see

an example of a Docker hardened image, in this case Node, uh a development variant, and we see that the the the the OS name is actually Debian, right? So, this is a derivative of Debian that build on top of Debian. And as such, the the packet the the package resources are all of course Trixie, right? That also means that if you have hardened [snorts] a specific binary,

they have hardened a specific binary, put it in that image, and you override that with the package manager, it will just it will just override with that non-hardened uh publicly available uh package. So, another example is if you want to install something, um the package manager will say, "Hey, we do have the package. We have the software, but we don't see the files." Um and that's exactly

the kind of breakage that you will see if you drop uh these binaries without the package manager. Um there's a number of other stuff, but it's not as interesting right now. Uh symlinks, [snorts] similarly. So, again, Node. Of course, then Node is important in this image. And we can see that if we would scan this, actually, you can see they're actually in the optional. So, it's I

guess it's how usually you would drop in a specific binary, but not such a binary. If you build one yourself, you might uh a piece of software, you might put it in there, but not something like node or yarn. You need to to your package manager to to be aware of that. So, [snorts] we scan these with gripe or trivy. They won't actually So, they they might

notice in this case they notice um an npm module, but the actual runtime is just completely missed. Uh again, they'll just miss and won't show you any of the CVs that are actually there. And the last one that I want to show is the suppression, right? And this is also very interesting because this is again a legitimate way to suppress to say, "Hey, it's not uh affected.

Your image is not affected." And uh in this case we take a Docker hardened image, a Postgres version. We see there's a whole list actually if we pull for this specific fix, whole list of statements to well, reason why or justify why a certain CV doesn't affect the image. And [snorts] we can see that sort of all the major of the 82 of code cannot be controlled

um are justified by no DSA. And that simply means that Debian says actually, "It's not a priority. We won't fix it until 4 months from now." But the the actual uh CV is exploitable and in there, right? Um [snorts] and also there is a ton weird stuff here. Um not to go too deep, but for example, there's two same for the same CV there is one that

is not affected and one that is still under investigation. And also that one that it does say not affected talks about an Alpine package and we we are talking about a Debian image here. So, there is a ton of things that go go go go wrong. And interestingly, I've used Docker hardened images as example here they are trying to do like a a hardened image program and

with as least zero CVs as possible. Um what they say is, "Hey, you get a a minimal CV free image for us." Um but this is actually what you get. And that's why sort of is it still trust that we can build on or do you need to be able to verify? Because again, okay. how how do we solve that trust issue? So, we've looked at scanners,

you want to at least have two or three scanners that have a different say and look on on that image because you want that discrepancy. Um but more importantly, you need trust. Or do we actually need to verify what's in that image? Um [snorts] and therefore there is first two tools to introduce. And the one is Cosign. And Cosign is actually also developed by founders of of

Chainguard. Uh it's an open-source project. Um it it rests on three elements. So, Cosign is basically the CLI where you sign and verify your images or your attestations. Uh Fulcio uh gives out ephemeral certifications. Essentially, the identity that signs the image gets an ephemeral ephemeral ephemeral certification to sign that image so that you don't have long-lived credentials as such as well. So, that identity can be yourself,

of course, or the software factory that outputs that that artifact. And lastly, Rekor, which is an append-only uh database that it is immutable and and public in that regard. And the goal is to essentially make you make the the crypto so that the hash or the digest of an image make that verifiable with the signature. I'm not going to run this demo because I think it speaks

for itself. The idea is if you would tamper, if you would do any any of these manipulations, such to say, then of course the signature wouldn't wouldn't match the the the the digest anymore and it would caught the the tampering. This only says something about what happened between I guess delivery and and or sort of the source and your consumption of that image, of course. It doesn't

say anything about the actual build just yet. >> Um that's where the SLSA framework come in comes in. So, SLSA framework also founded by Chainguard founders um is getting more attention right now these days because it speaks about >> [clears throat] >> uh well, open-source but also for any proprietary software um how is your your build process uh um I guess documented or set up as such

that you can trust what comes out of that factory, right? So, there's a framework and there's a couple of levels. You begin with anything could happen and you go all the way up to that you actually know when, where, what was built. However, this is a framework. In all fairness, and that is for example, Docker images, they sign uh they sign their digests and they have the

salsa framework. So, you can have both of those, but still suppress CVEs with facts. So, still that false sense of security and could also, just like we've seen, create Frankenstein images that either break in production or otherwise create that false sense of security. Um So, in both ways, there are still a couple of elements that we need to overcome. So, for VEX, sure, you can use multiple

scanners because if you use gripe, for example, it's not VEX aware, so it will output those CVEs. For all other elements, the only real solution is reproducibility. You need to be able to verify that what the the provider of that that software said it did that it actually did what it said. Right? So, um that's the remainder of of today is how do we get to um

how do we as Chainguard, but also how do you yourself can get to these reproducible and declarative builds? And I'm going to introduce three tools, and they were in the title already. So, that is Melange, apko, and our Linux distribution Wolfi OS. So, um what we see here is Melange and apko. The idea is is that anything in Wolfi, and Wolfi is the open-source version of Chainguard

OS, so to say, which is a fully bootstrapped Linux distribution built from source. Um and Melange is the tool to package a piece of code into a package, an APK package, because we use APK, Alpine APK for for Wolfi. And you get that that ecosystem of of packages, and then the second step is to compose an image with apko and I'm naming composing because it's not a

Docker file. It literally puts them directly or puts these these packages directly into your root file system without any ambiguous run or whatever statements that you usually have in these Docker files which make them again reproducible. just shortly I guess. So what Melange does is it uses apko under the hood as well to spin a build spin up a build environment that's fully reproducible as well. And

essentially you will see those examples because we're going to build one ourselves with a Melange and an apko yaml. It defines what packages are needed for the build for building the package such you say. It spins up the environment. It builds the package. It tracks what software is included so you get a per package S-bomb and then it admits that APK which is signed RSA signed in

this case because it's dependent on APK. And then the next step is apko and that composes that image from those APKs. And again only those packages that are that are declared to be defined are put in there. And then all of the S-bombs per package S-bombs are aggregated into a full S-bomb which is built at at during runtime or during build time such to say that you

know exactly that the software that is in your image is also shown in your S-bomb which doesn't have to be the case if you scan it after build time especially not if you disguise certain binaries or put just drop them in there. So how can you get started with this yourself? First Wolfi. So a bit of Wolfi. You could go to the repo right now. It's This

is an old photo. It's a pretty huge massive mono repo. And what you will see in that repo is all of these yamls. And again these yamls are Melange yamls essentially all of the open source software that we can find or requested, we build them from source and we make them publicly available as well within Wolfi OS. If we dive in such this might be a bit

small, but we'll dive into an example. This is really just exemplary. This is what a YAML or a Melange YAML looks like. For anyone who has ever built Debian packages, Reddit packages, there's a lot going on to actually build a package. This is one single file to declare this newly built this package. And if you run this yourself, you'll get bit for bit the exact same uh

package. Um I don't want to spend too much time here. There's a lot in there. Uh just at top you can see some package information. This is Vim. Um and you see below you see the actual source that we that that that that that the the code is pulled from and the expect expected commit there. Um interestingly actually for Vim you can see it on the version

number. They tag a new release for every commit. So, actually we rebuild Vim on a daily basis probably because every new commit is a new release and then Chainguard will need to rebuild. Um actually you could see there's a sort of split stream where anything rebuilt there is a subset that goes to Wolfi OS and a subset that goes to Chainguard OS, which is sort of the

enterprise version of uh Wolfi OS. And again, for those who have been building packages, interestingly these are the only two lines that you would need to bump in order to bump the uh the version and therefore build the the sort of the newest probably CV free version from upstream. Just the two lines to get a a new version on your package, which is very nice. This is

also internally of course, which we're going to see in a minute, if you package yourself with Melange uh and build those images with with apko. So, essentially what happens is you've got all of these source repos that we build into packages and that creates an ecosystem. Um and on the Chainguard end we then cherry-pick from those packages which are needed to run that primary application. So, suppose

you're running a Python application, then that will will need some essential OS elements, some Linux OS elements, and we need Python, of course, and not much more. So, these are cherry-picked, and that's how we get these minimal declarative images that you can fully reproduce. So, that's a bit of Chainguard and a bit of Wolfi. The question is, how can you get started without paying a dime? Open

source. if we look on the left, we've got the source like that those are of course that open source software, right? That's what you want to make That's what you need and make use of to run your own applications. You've got your own code, of course. Suppose a Python app, and we're going to see that in a second. The idea is that you package that as well

with Melange, and then you compose depending on the goal of that that image, of course. Again, if it's a Python, well, then we'll need a base Wolfi, and we'll need Python, and not much more depending on what we want to do, of course. If we might need if we need a curl, we need to add a curl, and so forth. That's also a thing in terms of

sort of a paradigm change. If you have minimal images if you're used to a blown-up Debian or or Ubuntu distribution, you'll have those tools in there already. If you start minimal, you'll have to explicitly add them. But, the end result is very minimal with the smallest attack surface that you can And the added benefit here is that you build from source essentially, not only your own application,

but also all of the packages that you include in your image. Because we've built them from source, and Wolfi OS is single version stream. So, anytime you pull anything out of Wolfi OS, it is the most up-to-date version of that software with hopefully no CVEs, right? That's the idea. So, let's actually build that. Let's go Again, what I've mentioned, I've scripted it all for the sake of

of a So, first, I have a very simple Python that can just run. I believe I have it over here. Let's see. There you go. Not more than this. Really, it's really about the idea. Just to show you that it works, and then we'll move on and start packaging this. So, the first thing that we'll do is Let's actually have a look at the Where Let's have

a look at the melange uh over here. So, at the top, as we've seen already, a bit of package information, it's a dev world demo version zero, the epoch, and so forth. And this is the environment that is, again, spun up by APKO under the hood in order for the the We can go through. So, the essential elements to build this APK. So, first of all, what

repositories and and and keyring are we using? Well, we're only using Wolfi elements, so we need that keyring and the repository. Then, [snorts] of course, we'll need a base layout, an image layout. So, that's the Wolfi base layout. Then it's a Python app, so we'll use Python. And we need we need pip to install that one dependency, Flask. We need busybox, a couple of shell tools we

need to actually build the the APK and move out the APK and do some folder arrangement. And we need certificates in order to make the connection to pip. So, those are the only essential elements. And then below here, this is basically essentially the declarative way because it runs in a container. So, that's why you have this sort of the target that's there. That will in a second

will show here when I've actually run the script and oops. Never mind. I'll just do it like this. I'm going to run it first and then going to get that away. >> And then we'll see doop doop doop doop doop. I've now run the script. I'm just going minimize turn hiding on. There we go. So, we see it running in the background. It has now built that

APK, and we can see We've got the packages folder. There's two elements here. First APK index such that APK can actually understand this as a package, right? The metadata for the package itself. And then the APK itself, right? well, we've seen this over here. Um not much more to say about that except for the fact I suppose that the build scheme, so the Melange scheme plus the

SBOM is embedded automatically in the artifact. So you'll see that as well for anything that is built with either Melange or APK or similar to APK, you can extract the embedded build config and rebuild it yourself and inspect that bit-for-bit comparison. Um so the next step is we're going to do an eye on the time. Oh, not Okay, let's just do it like this. Um so essentially,

this is of course on the left APK, but this is the APK YAML, a bit differently formatted. So again, which repositories? We have our local repo, right? For our own package. We have Wolfi again, the key rings or own RSA key as well. And then or what are we going to use in the end image? It's just Wolfi, uh the base, Python, and our own app, of

course. Some simple stuff, we'll set a path, we'll set an entry point, we'll make those user accounts, the architecture we want, um some annotations if we need to, and importantly some layering. So the strategy in this case, origin, basically means that we'll try based on the origin of the package that it will optimize the layers in such that you will group uh a package that are likely

need to be rebuilt uh at the same time um just for efficiency, of course. So let's actually also do do do do do do do do do do do do Let's build. So importantly, it already says it, OCI image. So it's a fully OCI compliant image and you can just load it into any runtime. >> If I scroll up a bit through these logs, you'll see those

layers. It's I think pretty interesting to see see. So, we've got Python here, some crypto here, Glibc, GCC, and the rest matched together. Yeah. So, the output of course we can inspect that over here. That image would be over here. We've got the actual tar, so that's the actual image you could Um there's the the complete SBOM. Um again, aggregated from the individual per package SBOMs. And

then I guess to the metadata. So, this is the manifest for in your registry if you want to pull a specific architecture version and I could run it, but I believe that you believe that this will just run as an image. I want to skip a bit to the conclusion end um because here a couple of elements come nicely together. First of all, perhaps a couple of

years ago sort of diving into a new build tool and a new build chain that would have been pretty gruesome. These days with I'm going to be honest with Cloud Code, you could start your testing your testing pipeline tomorrow. And the result that you will see in your own testing pipeline is pretty significant. If we look at this Python 3.12 and a number of of of vulnerabilities

in there versus Wolfi. And again, this is just open source. We rebuild this daily for everyone to use And added benefit so is that it is declarative as well. It is super quick because it is so efficient with all these packages. And the APK itself is efficient by because it just composes directly onto your root file system. So, the performance of your pipeline would increase massively. It

is fully declarative and reproducible, which is internally also a very good nice thing to have if not even needed for certain compliance elements. It's hugely smaller, well attack surface lower. And another, you will inherit a hardening elements that we perform with Wolfi and Chainguard. For example, that we always strip the SUID permissions from all of these packages. Which actually is related to the kernel exploit that we

had 2 weeks ago because that made use of an SUID. So not having these would not make you exposed to that Linux kernel CV that we've seen 2 weeks ago. And then the last thing, if we rebuilt this again, you can see we have exactly that same hash. And that's the whole idea, of course, to be reproducible. Because otherwise, you cannot inspect and verify that what that

vendor tells you is indeed true. So, in the end, it is a verification or at least the ability to verify. You need to be able to to smash them and say this is not right. Not trust blindly. And be critical to the image provider. And also your internal policies. And start experimenting, I would say, because this is all publicly available. You can have a pipeline in a

couple of hours with the clock code. And I hope fully I've just shown a couple of downstream huge positive elements of of building. And again, it is open source. It is right there for you already. And that in a very hasty feels way already concludes my talk. If you're interested, we're out there in the back with Chainguard if you want to have a talk. Thank you.