Real-World Supply-Chain Security - Alex Leong, Buoyant

In this talk, Alex Leong discusses supply chain security, specifically through the lens of his work with Linkerd, an open-source service mesh project. He explains the importance of understanding the software supply chain, which encompasses everything from executable code to dependencies and build tools. The speaker outlines how to achieve effective supply chain security by creating and managing various documents, including signatures, Software Bill of Materials (S-bombs), build attestations, and VEX documents. Each of these components helps software producers communicate with consumers about potential vulnerabilities, thereby enhancing security and transparency. Leong emphasizes the necessity for software producers to provide detailed information to enable consumers to understand and mitigate security risks in their software while acknowledging the evolving nature of this ecosystem.

Hi everyone, welcome. This is supply chain security in the real world. Uh so my name is Alex Leong. I am a software engineer at a company called Buoyant. Uh Buoyant is the creator of the Linkerd service mesh and I have been working on Linkerd for just over 10 years now. So it's been a big part of my life for for a really long time uh and uh

service meshes uh obviously as well. Um and so why am I here talking about supply chain security? Well, let me give you a a little bit of context to understand that. Um so Linkerd itself is a service mesh. It's an open source project as part of the CNCF. It's one of the graduated projects. It's actually the first uh very very proud of that. Um and kind of

in addition to that, there is also something called the Buoyant Enterprise for Linkerd or BEL B E L, uh which is an enterprise distribution of Linkerd created by Buoyant. So I have the uh the privilege of working on both of these things. I spend some of my time as an open source uh project Linkerd maintainer and and part of my time working on this enterprise distribution. And

one of the things that really differentiates the enterprise uh from the open source is the fact that we do a lot of supply chain security. Uh this is something that, you know, is is very important for enterprises. And so that's a big the big piece of the value there. So over the past maybe year or so, I've learned a lot about the supply chain security ecosystem and

what tools are out there, how people are interacting with them, how to to use them. And so I wanted to share what what I've learned uh about that ecosystem, which is I think it's really rapidly evolving right now. Um and so all of this is kind of going to be mostly from the perspective of buoyant as a software producer, but in order to understand that, in order

to do that effectively, you also really have to think about software consumers, the people who are who are using that software. So, kind of by show of hands, who here considers yourself a software producer? I would say most people. Who considers yourself a software consumer? Should be everybody, right? Everybody is a software consumer. So, you're always going to be running software, and you want to know what

software you're running and want to know that it's secure. So, that's that's uh uh what I'm going to be mostly talking about today. Uh so, we're going to talk about what is supply chain security, why does it matter, why should you care, and how uh how does it work, and how do you use it? Okay, so what is supply chain security? Well, in order to answer that,

we have to answer the question, what is the software supply chain? So, the software supply chain is everything that influences the way that software behaves. So, that's the executable code itself. That's all of the libraries that you depend on. That's transitively, and in modern software, that uh dependency tree that you depend on is is usually very, very large. Um it includes all kinds of stuff, and we

we've seen this uh time and again when some dependency way down in your in your dependency tree has a problem, that can affect you. Um it depends on your build system, it depends on your build tools, it depends on what version of build tools you're using, uh it depends on the container that you're running inside, the operating system that you're running on, the hardware that you're running

on, the networking stack that you're running on, CNI plugins, uh etc., etc. Depends on Kubernetes, depends on any APIs that you interact with or utilities that you use. It's this huge uh set of things that can influence how your software behaves. Um but as software producers who are, you know, producing software and shipping it out to to software consumers, uh the ones that we are kind of

in control of are the library dependencies, the build system, and the container. And so that's what we're mostly going to be focused on because that represents the artifact that we're we're shipping out to our users. So, ideally, supply chain security would mean that we can say, "Here is our software. It has no vulnerabilities in anywhere in the supply chain." Done. Easy. Um unfortunately, we can't do that.

That would be nice. But that's very difficult, or that's impossible rather, because security vulnerabilities are often discovered in real time, right? We may ship out our software, and it may have no known vulnerabilities anywhere in the supply chain, but after we ship it, vulnerabilities can be discovered. And this this happens all the time. CVEs are discovered and disclosed, and you know, that software is already out there

in the world running. So, the best that we can do is if we can't guarantee that there's no vulnerabilities, we can at least describe the supply chain that we're providing. So, we can describe what packages we're using, what dependencies we're we're using, what versions they are, how it was built, etc., etc., etc. And if we give all of that information to the people who are running the

software, we empower them to make decisions about it, to know when they're affected by vulnerabilities, and to know what next steps they need to take. So, as a software provider, supply chain security is really about describing the software that you're shipping in as much detail as possible so that it is accurate, it's verifiable, and consumers are empowered to to do stuff with that information. So, what kinds

of things can we describe about the supply chain? Um so, I'm going to talk mostly about these four kind of categories of things that we can describe. I think there's probably more facets of the the supply chain that we can talk about, but but these are the ones that are kind of top of mind for me, and so those are the ones that we're going to talk

about. And each of these has a different purpose. They're all uh describing different facets of of the software supply chain. So, these are signatures, S-bombs, build attestations, and VEX documents. So, signatures are kind of the most simple ones. This is just a cryptographic signature of the software you provide so that you can guarantee that the software came from you, right? If a consumer receives a a software

and they want to run it, uh they want to look at the signature and say, "Okay, yes, this did actually come from Booyant or whoever." And we can verify that cryptographically. Uh this also lets them know that it hasn't been tampered with in in transit or whatever. It hasn't been, you know, swapped out for some piece of malicious uh code from someone else. It verifies that it's

intact. This is kind of the the minimum bar. If you're doing nothing else, you should at least, you know, sign sign your packages. Um there's lots of different ways to do this. Uh the easiest is probably using Sigstore with the Cosign command. This lets you do OIDC keyless signing, which is very very convenient because you don't have to manage keys or or, you know, set up anything

like that. Uh this is what we use for for uh Booyant Enterprise. Um it's very easy and convenient to do. There are some downsides to it. I think some of the information about the way uh it's set up kind of leaks into the the the signature. So we're probably going to migrate to using something like notary in the future, which gives us a little more control over

how it's signed and is a little bit more like traditional key management. Um that's probably in the future for us. Um or you can do this any number of other ways by kind of rolling your own or providing your own signatures however you like. I would really recommend if you're not doing this already using cosign to just do keyless signing is the easiest way to get started.

Uh the second piece of information that you can provide about your supply chain is something called an S-bomb, which stands for software bill of materials. And this is just a list of all the different dependencies and their versions that are in your software. This is hugely valuable. Um there's lots of different tools that can generate this. Sift and sneak, for example, are what we use. And there's

a lot of different formats that this data can be stored in. SPDX and CycloneDX are the ones that that I'm most familiar with. And the reason why it's so useful to list of all your dependencies and all the versions that they're running is that consumers of that can cross-reference it with vulnerability databases and they can know if anything in the supply chain is affected by a vulnerability.

They can say, "Hey, here's all the things that are included here. Are any of them affected by anything known?" And they can do that continuously over time so that as new vulnerabilities are disclosed, you know whether it affects your software or not. And the S-bomb itself can actually be signed as well so that you can guarantee that yes, this S-bomb document did come from us. It hasn't

been tampered with. It hasn't been modified. Authoritative and you can trust it. Um so if an S-bomb kind describes what is in your software, the build attestation describes how the software was built. So, these are typically published in something called a supply chain levels for software artifacts, pronounced salsa. Uh, these documents give you a step-by-step description of how the software was And this is also really useful

if you want to audit the way that the software was built and make sure that it's following best practices, make sure that it's kind of doing everything that it needs to from a compliance standpoint. For example, if you're using FIPS or if you need FIPS compliant builds, this lets you audit and guarantee that. And again, just like with other kinds of documents, this can be signed as

well. And then finally, VEX documents are things that let consumers know additional information about vulnerabilities and how they affect your software in particular. So, for example, you can say that this such and such vulnerability, CVE whatever, has come out. It affects one of our dependencies, but not in a way that is exploitable in our software. And so, this lets you give consumers that additional context and say,

"Hey, don't worry about this. We can guarantee that it's not a problem. it's okay." So, these VEX documents are really useful for providing that extra context about how vulnerabilities affect this software in particular. And these also can be signed and should be signed. Um, so you may hear me refer to kind of all any and all of those as attestations because that's kind of what they are.

They are us as the software provider attesting various things about the supply chain. We're attesting that here's the list of libraries that are packaged up with this. We're attesting that here's how they built. We're certain vulnerabilities don't are not exploitable in this software. And so the end result is you get this kind of collection of of attestations that describe the supply chain. Okay, so easy, right? That's

that's it. We've got we've got some software. We've got a bunch of attestations about it. Done done done. Um but it is a little bit more complicated. you know, generating these documents is not typically that hard, but once you have them, what do you do with them? How do you deliver them to software consumers, and how do the consumers interact with them? Right? That's where where things

get a little bit slightly more complex. So if you're using something like Docker Buildx or the build push GitHub action, this has support for for SBOMs and Salsa documents kind of built and baked right into it. You can if you're using GitHub actions for example, you will just set provenance to true, and that will automatically generate a Salsa document which describes exactly step-by-step how that artifact was

built. You can set SBOM to true, and that will SPDX SBOM which describes all of the packages that are included in that. and so that's you know, if you're using the tools it's very very easy. You can also generate these things directly. You can use Sneaker Sift to generate SBOMs. Or you know, there's a a wide wide variety of tools that are aware of these formats and

can produce these these SBOMs. And I think kind of support for for SBOMs is is is growing by the day. So generating them is easy, but what do you do with them and how do you deliver them to to consumers? And in order to answer that, we have to talk about the OCI format. So OCI for open container initiative. This is the format that containers are are

stored as. So typically you'll have like an OCI image registry and in that registry there'll be a bunch of tags and those tags point to something called image indexes. And an image index is a bundle of images usually one per platform. So for example, you could take this tag that we have here. This is gcr.io/point.io/proxy:enterprise-2195. This is the latest version of the point enterprise for Linkerd proxy.

So this is a docker image tag. But it actually points to an image index that contains two images. One for arm64 and one for amd64. So it's a multi-platform multi-arch image. And so it it just looks like something like this, right? It's pretty simple. Image index contains two images, one per So if you build with attestations, if you set S bomb true or provenance true, what's going

to happen is it's going to put those attestations directly into the image index. So now that image index will not only contain those two images, but it will also contain those attestations, the S bomb and the salsa document. So this is very simple, very easy. Any consumer who wants them, you know, just gets them kind of bundled up in there. but what happens if we want to

add more attestations? What if we want to for example add a vex document in there that says hey this vulnerability just came out and we want to tell you that it's it's not going to affect us. How do you do that? So this is what it would look like now kind of in that world where you have those documents bundled in directly inside the image index. You've

got those two container images and you also have attestations. The problem here is that because those are bundled into the image index, now every time the attestation is changed, so for example, if you want to add a new attestation to add a vex document, something like that, you have to resign the entire package, right? It's one uh one unit. And this is problematic because it makes it

look like the images are changing, which is not what's happening. The images are not changing. The images are staying the same, but because the attestations are changing, the signature changes, the whole thing as a unit changes, uh and it and it looks like the images have changed. And what we really want here is immutable images, right? We want to say, "Hey, we're delivering these images to you.

They're immutable. We're not changing them out from under you. We're not doing anything weird. We're just giving you more information. We're adding more attestations, more documents. We're providing you with more information about the supply chain, but the images themselves, they're immutable." And so in the OCI version 1.1 specification, they added something to do exactly this. Uh so they added this concept called refers, which let you add

uh documents to the registry that refer to other things, rather than needing to be bundled in So there are these uh objects that live uh in the registry but outside of the index and have these refer pointers to refer to other things. So for example here on the right, I have the uh OCI uh entry for a salsa So there you see the artifact type there is

application/salsa. It's in JSON format. And it has this subject field which says that this is a referrer and it refers to some other object in the registry. So in this case, it's referring to a um an AMD 64 container image. So, this is a salsa document that refers to this other image over here in the in the So, if we use refers rather than building these attestations

directly into the image index, the picture looks like this, right? We have these attestations which are now in the registry, but they're outside of the image index, and the image index itself can now be immutable. It's never going to change if we add more attestations. They live alongside it. They refer to it, but they don't change the index So, this is really great. This gives us kind

of the properties that we want, this immutability of the images. When the images don't change, we don't have to change them. Their signature doesn't change. Um, and it also gives us this really nice API for discovery where we can query the registry, and we can say, "Hey, tell me all of the documents you have that refer to this image. Tell me all the different attestations you have.

Tell me all the different signatures you have. Give me all that information so that I know everything I need to know about this this image." And and so, you don't have to worry about like hunting down kind of where are all the SPDX documents? Where are all the the vex documents? Where are all the salsa documents? You can just get those back from the API. And so,

this is what we do in in in VMware Enterprise for LinkedIn and Bell is we use these this these refers to provide this additional information about our supply chain. Um, and in fact, we wrote our own tool that will take these attestations that are initially built into the image index, and we kind of repack the OCI layout so that we take them out and turn them into

refers. Um, and that way we get this immutability of images that that we want. So, there's this really nice tool called ORAS discover, um which uh uses that API that I talked about. So, this is how we can get all of the referrers to a particular image. So, we can say ORAS discover, we can give it a tag. In this case, we give it the controller image

tag from from uh Enterprise Linker D, and uh it comes back and it gives us all of the referrers to that image. So, in this case, we say that image has a certain SHA uh C72, whatever whatever whatever, and it comes back and it shows us the two documents in the registry that refer to it. One is Nest PDX document, so that's an S-bomb, and the other

is a Salsa document, so that's a build attestation. Okay, great. So, we've taken all of this supply chain information, we've stuck it in the registry, we've used OCI referrers to link it all together. What do consumers do with this? So, the whole point of this of doing all of this is to communicate information to the software consumers to give them the information that they need. But, if

they don't act on that information in any way, the whole thing is pointless, right? They need to do something with that information for for this to have a And so, this is the part of the ecosystem that I would say is most in flux right now. There's a lot of active development going on, but there's a lot that people can do right now with that information. So,

like I showed a few minutes ago, you can use the uh referrer API or the ORAS command to discover all the referrers to a particular image. So, someone ships you an image, Boyant ships you an image, you can look at that registry and say, "Okay, show me all the referrers that so I can get everything I need." if there are signatures in there with their which there

should be, you can verify those using Cosign. Say, "Okay, yep, these signatures are valid, they're legit, this software has been signed and and not modified since then. Uh you can take the S-bombs that you discovered and you can feed them into a tool like gripe, which is going to cross-reference it against the vulnerability database and tell you if there are any vulnerabilities that are known that are

affecting anything in the So, uh and you can I do that on a continuous basis, right? You can do nightly scans or whatever so that as new vulnerabilities are discovered, you're you're you're checking do these affect me? Do these affect me? Do these affect Um and then here's the the coolest one is that you can use policy engines like Kyverno to define and enforce runtime policies. This

is really neat. So, uh if you haven't heard of Kyverno, it's a policy engine that lets you define different uh policies about images and it will block the creation of pods if the images in those pods don't uh fit the criteria of your policy. So, for example, you could write a policy that says, "If I'm getting any image from Buoyant, it needs to be signed by Buoyant

and it needs to have an SPDX attestation, which gives you all of the packages, uh the S-bomb, and the S-bomb itself has to be signed." And if any of those criteria are not met, we're going to reject that. We're not going to run it. Too suspicious, too fishy, something weird's going on. Um so, this is really really cool. there is We have some really nice docs about

how to set that up. It's very straightforward. I couldn't fit the uh the Kyverno policy CRD on one slide, uh but uh it's not that bad. And and and that's kind of that's kind of it. It's it's pretty straightforward once you kind of sift through No pun intended. Sift through all of the uh different pieces that you need to and figure out how they all kind of

connect the really the point of all this is just communication, right? As supply as uh software producers, we want to be supplying as much information about the supply chain as possible so that the consumers can can know if they're affected by vulnerabilities and and and can act on that information. And and even though this ecosystem, I think, is really rapidly evolving, there's a lot you can do

with it both as a software consumer and as a software producer today. If you are a software producer, I really highly recommend that you start providing these things uh because it's it's pretty easy to do so and it provides a lot of value to your consumers. And if you're a consumer, I think you should really think about asking for or even demanding these things from whoever is

supplying software to you so that you can get these kind of guarantees that you know what software you're running, you know if you're going to be vulnerable to any kind of vulnerabilities. And and I kind of suspect that this uh will get more streamlined as time goes on and these these tools continue to evolve. So, that's all I had to say about supply chain security. I also

wanted to talk really quickly about uh EKS. Uh we've been putting a lot of effort on the Linkerd side into making sure that Linkerd D runs really well on EKS and that's kind of the best experience possible. So, if you are running on EKS and you use a service mesh or you're thinking you might need a service mesh at some point, I highly recommend this guide. It's

really really good. And I also wanted to really quickly plug that we are hosting a party tomorrow with Merge Forward uh which is an escape room party. It's going to be very very cool. Fippy has been kidnapped and is is being held in a vault, I believe. Uh it's going to be a a lot of fun. So, uh spots are limited. I recommend you register as early

as possible, but it's going to be really cool. This is tomorrow night. And uh I'll just leave this slide up as uh I take questions. >> So, thank you for your talk. I think I gave a really similar talk two weeks ago to my company. And so, I totally subscribe to what you have said. What I've seen is kind of users are kind of not interested. And

And I think it's just because there is too much friction on the verification side. And they Yes, we provide cosign, we provide these things, but as soon as they have to write a CLI that it's longer than three lines, >> Yeah. they get upset and they don't do So, I just want to kind of ask you what is your take to make this a bit more uh

straightforward. Uh I mean, from uh really your perspective, uh buy and cosign and all these things already do it. Do you think we can go further? Yeah, I mean, I think it remains to be seen how things will kind of be streamlined in the ecosystem going forward. I think as software producers, we can just kind of do our best to make it easy for people by providing

uh good documentation, by saying, you know, here we'll we'll even provide you with Kyverno policies if you want to just, you know, plug those in and use them. Just kind of documenting the verification process so that people can can do that and make it as easy for them as possible. Hi. Uh thank you for the talk. I want to ask a bit more about the tooling for

verifying that we've got all the documents on the the build process and such. Um but maybe not just by the developers that might use our images, but maybe also by some non-technical people which which you know about some some some tooling for example with a graphical user interface which would summarize, yeah, we've got these images with these attestations, etc. Thank you. Yeah, that's a great question. I

don't know of any tools with graphical user interfaces for this, but I think a lot of this comes down to like policy, right? So, if if you set policies that kind of describe your your security posture and and what you require, you can kind of set those and and then leave them in place, right? And then as new software comes in, you know, it's going to have

to meet those standards or or else it won't won't be accepted. Hi, same to you for this talk also. Usually we use image in a chain, so we are using a public image and doing stuff on them and adding our own software on it, but how so the link of verification? Uh yeah, so this is you know, when you're looking at something like the S-bomb for example,

that's including it's picking up the dependencies from all the layers, right? Including the layers that you've built upon. And it's the same thing with a build attestation. It's going to describe the process that that image was built including, you know, what layers did it build on top of. Okay, thank you. Hello. Um as a consumer, how would you go about images that are not signed in that

way? Um because you have to look at it from a from a wholesome perspective, right? And I have hundreds of different images from different sources. How would I go to protect my whole my whole supply chain? Yeah, I mean, you you're getting images from lots of different places. Some may be signed, some may be not. You can kind of only do the best you can do. If

images are coming in without signatures, you can't verify them. But you can specify policy that says from these sources, uh we require signatures, from these sources we don't. For example. Thanks. Yeah, thank you very much for the presentation. Uh I have a I have another comment because I think like this whole supply chain transparency thing is getting more and more crazy, right? I mean, it's not only

knowing like which sources went in there, but also in the future like okay, which AI model did you use to generate your code? What was the training data that was used to train that model? And so forth to be sure, you know, like because people want to know what that tool may be put into your code that nobody knows any longer. >> Mhm. Uh we also see

the requirements on who developed the code. Mhm. Where is he living? Where is he from? Like all these kind of information being needed. And I think it will be like really interesting to see where this whole story goes. Um I think thank you very much for your presentation. Yeah, I know, that's a great comment. And I think like a lot of these formats like the salsa format

are very flexible for that reason, right? They let you specify a lot of information about how the software is uh produced even as the way we produce software kind of changes. Thank you very much. Hello and thanks a lot for the insights. Uh two questions. So, how are you creating or which tooling are you using to create your VEX documents and video for the risk management? And

the second one is um how do you retain visibility of all your S-bombs, especially regarding of the EU Cyber Resilience Act? Uh so, first question, how are we producing vex documents? We aren't at the moment. Um so this is kind of an area for for future uh exploration, but I know there are there are tools out there and I have looked a little bit at them. Um

sorry, can you repeat the second question? The second question is how do you manage all your S-bombs because I think with the EU Cyber >> Mhm. which is in rolling out right now, you need to basically know where are you using which dependency in case of an incident. Yeah, I mean so when we whenever we do a build uh an S-bomb is produced as part of that

build. And it like I described, it kind of lives alongside it in the registry and and is connected by a referrer. Um but that's like an artifact that's generated every time the software is built. So you have like 10,000 of S-bombs and then if you need to figure out which libraries use in all your artifacts. Yeah, I mean so those artifacts are produced when they're built. You

know, we only publish when we publish in a version, we publish it. It comes along with, you know, whatever S-bomb came with it. Okay, thanks a lot. Yeah. All right, thank you very much.