Why Isn't the Fix in My Container? Tracking CVE Propagation Across... Mor Weinberger & Lior Kaplan

This talk covers the complexities of addressing security vulnerabilities in containerized applications, specifically discussing the journey of Common Vulnerabilities and Exposures (CVEs) from discovery to resolution. The speaker analyzes data derived from tracking CVE publication and remediation across numerous open-source container projects over a year, identifying that while 90% of vulnerabilities have fixes, about 40% remain unaddressed in active containers for more than six months. He highlights the challenges that arise in the path from discovering a vulnerability to users implementing the fix, emphasizing the necessity for better collaboration between maintainers and security researchers. The session concludes with practical recommendations on managing dependencies and optimizing security in container images, emphasizing the importance of using minimal versions of base images to reduce unnecessary vulnerabilities.

So, welcome everyone. Actually, it's my first time speaking in a KubeCon, so I'm very excited to be here. And today we are going to talk about the question that every security team has been asked before, why isn't the fix in my container? Uh we tracked CVE publication across thousands of open source project to basically find out where the patch gets stuck in the way to your running

container. And just quick intro, uh Leo Kaplan, my partner for this session, was supposed to be here, but he couldn't make it really in the last minute. So, I think I will just uh skip his slide. No, kidding. Uh I'm on it, no problem. And about myself, I'm Alon Weinberger, I'm a software architect at Aqua, where we're building vulnerability-free uh container images. Uh I have been around

vulnerability management and container security for the last 10 years, work at Aqua Security Innovation Lab, and also um taking part part at Microsoft around Defender for Cloud. And today we want to share with you what we actually learned when scanning hundreds of open source container projects every day for the entire year. So, basically how it works is that we choose um the most popular containers images process

uh project across the world, like different workloads, uh different communities. We scan those uh every day in the span of a year, um which means we process around 10 million findings. We build what we called a CVE journey, which basically it's the timeline, it's the story of the CVE from the point it got published, uh discovered on the image, and until it got fixed and pulled by

the uh by the users. We craft around 40K CVE journeys and run some analysis, basically to help us understand better what is the mean time to resolution, the average one, what is the percentage of fixed availability and more. And the first thing we saw was quite shocking. We discovered that around 90% of those CVE has a fix. Which basically mean they are like remediable, right? They have

a fixed version, you just need to to to to bump to it, and you're good to go. And the second thing that we saw was that around 40% of those CVE still live in your container for more than 6 months. Which is crazy, right? So where is basically the gap here? Um So in order to start and digging about the different gap, we need to understand what

is the the path or let's start with the happy path of the of the timeline of the CVE. So it all start by reporting security issue. It could be either by the maintainer themselves, some contribution, or like security research that basically say hey there is a security issue here and the CVE got reported. Then the maintainer need to act quickly, fix the issue and new release is

being cut with the with the fix inside. The container need to be rebuilt with the fixed version inside, and the user also need to pull the the fixed tag, right? This is an ideal world, world, and we can see a great example like happened with the curl project, cURL project, that it took really few days from the point that the CVE got reported until you got clean

curl image. But this is the ideal world. In the in in the actual world, we can see many challenges around this path. Okay, so one thing is is could be that there is no fix available. Basically, you don't have nothing to do with that with that, right? If you're using this vulnerable package. Another thing is that the fix is available, but the project didn't release a a

new version with the fix inside. Another thing is that even if the fix is released, the image didn't pull the the fixed version and rebuild um and release new tag. And the last one is that we saw that we as a user not always aware that some tags fix those issues. So, at that point we came into conclusion that we actually need to split the the CV

journeys into two pieces into two parts. Part one is talking about the discovery to fix from the point that the vulnerability was discovered until it until the fix was developed and accepted. And the second part is around the there is a a version with the fix inside and until the user can actually um use it and and pull it. So, let's start with journey number one. We

start by defining the issue, right? We report it. CV might create it. Maintainers works on the fix, right? Or maybe we can just use the eye for it, And instead, let's go skiing or something like that. So, let's play a little game. Raise your hand if you heard about the last Entropic announcement where they said they found more than 500 security issues in open source project. Did

you hear about it? Okay, nice. Quite nice. What about the latest really latest announcement by Opus that they found 22 severe CVEs inside Firefox. Did you hear about Okay, a little bit less than that. And the last one, did you hear about OpenAI Codex Codex, sorry, that scanned millions of of commits and found tremendous number of high severity issues. Did you hear about Not that. Okay, this

is quite a recent recent one. So, it sounds like this problem is solved, right? The journey one from the CVE that got discovered until the is proposed and is accepted. Quite done, right? The AI companies is totally on it. What do you think? Or maybe we should ask the maintainer what they think about it. Um so, we can see here an example uh the core maintainers of

curl that actually 2 months ago decided to close the bug bounty program. Bug bounty program, if you are not aware of it, is the way of the project to attract highly skilled security researcher into actually researching and finding high quality vulnerability within the project. Basically, stopping this project means reducing the monetary motivation for those highly skilled security research means um less research that taking participant in this

in the program of curl Um so, why why why the commentator wanted to do that? So, the reason for that is what he called as the death of thousand slopes. Which basically mean that he and his team on curl actually needed handle with thousands of low quality AI slope reports. Uh and you can think about it. It takes like tremendous amount of time to filter out, to

triage them. Um and actually closing this project is bad for the community and bad for all of us. Another example is where the core maintainers of libxml, which libxml is think one of the most popular XML parsing library inside your operation system, you might not aware of it. Um he actually decided to step down from uh maintaining his project because he mentioned that big companies actually investing

in automation, fuzzing, AI in order to find security issues in his And eventually he need to deal with the tremendous amount of issues working on the fix. And he doesn't really find those issue as a critical one. He even said that those issue are not important more than non-security issues. And this is why he uh decided to step down. And you might think, "Okay, like this challenge

is only relevant for the security issues, the security side of the open source, right?" So you might thinking about it again. This is like just published really few days ago by the CEO of Hugging Face where he shared that the Hugging Face open source project got new PR every 3 minutes. Let me repeat, new PR every 3 minutes. This wasn't actually was the situation before we have

those AI coding agent, right? And actually GitHub, which is the most platform for open source, actually taking this matter and trying from their platform side. They started a product discussion that trying to help maintainers handle the increase of low-quality volume of contribution. So, as you can see, the struggle is really here and like nowadays, but we also saw some good outcome from using AI to to fix

security issues. And we really need to be more focused on the quality less than the quantity, as you can see. And we really need the AI to focus more on the fixed side and the mitigation side other than the the discovery side. Um and we also want to see the AI work closely with the maintainer side in order to reduce the fatigue. So, when we're talking about

security report steps, there are some goods, does, do and don'ts, right? So, first thing about when you're reporting a security report, you should do it responsibly, which basically mean do it in a private channel, don't disclose anything publicly, um follow the format, respect the SLA. You should help with the CV assignment about the affected affected products, severity, stuff like It is super important to try and from

the very beginning of the report trying to help with where is the affected code, where is the actually weakness, what is the suggested fix, and if the maintainer propose fix, please try to help him actually verify that it will close in a in a way. And the wrong way is actually keep asking and barging to get a CVE. As I mentioned, disclose maybe any information publicly. And

as a user basically we encounter when we encounter a CVE and we check like publicly and see there is some discussion, please try to avoid adding more pressure on the maintainers without really helping or providing any new context. And as a reporter of a security issue, if you report an issue to a different party other than the maintainer, please make sure that the that the maintainer is

part of the discussion or is aware of it. Try to CC him. Uh stuff like that because we saw some example um this is for a libxml maintainer that got a new issue that basically said, "Please see you have a CVE on this project." And he admit that he never heard about it before someone create an issue about it. And also Leo uh saw that there is

a CVE for few weeks uh in in in tar project and he also heads up that this CVE there is no like live discussion, there is no any activity. And again, the maintainer said to him, "This is the first time I heard about this CVE CVE that affecting his own project that he maintains." So, we can see that there are some open source project that trying to

take the the lead and to to improve the situation here by just announcing they are now a CNA. A CNA is basically means that you are part of the advisory tree. Uh and that's mean that if someone will wants to um report a CVE on your project, you are the guy you're the the party that need to to contact with. we are right now uh on the

stage that the maintainer actually need to fix the issue, right? So, few things that it's to understand here. First of all, the maintainers, there are cases there are projects that runs by individual or by one to three um maintainers. So, please be respectful to that to that. Try to offer to help if you have the time, you have the the context. There is no uh SLA. Those

projects, some of those was developed as a hobby. They didn't develop uh this as a project that will be used by by uh by most of us or by big companies. They didn't intended uh this at the first place. So, you need to be aware of it. So, right now I think we are good to go to the second part of the journey, journey number two, from

fixed to the user. Let me just take a sip. Okay, so now the patch the Sorry, the patch was merged. Now what, right? So, first of all, we need to cut like a new release, right? If it's like critical, mostly most chances that the release will will cut quite quite fast. If it's less than that, it could take weeks, even more than that. But still, when there

is like uh a new fixed version, the scanners need to detect it, need to be aware of it. And the way that the security and the the vulnerability scanner works is that they are consuming for multiple advisories uh the those information which package and which version has those CVs, so they need to consume it, build their own database, and this can also take take some times. And

on your end, if it's direct dependency, so it might be uh quick, but still it require like code change, pull request, stuff like that. If it's a indirect dependency, it could be a way longer. And still, you need to rebuild your container with the fix, which uh again takes time. So, you might think, why I just can't use the the new version, right? So, the thing is

that um this version might be too new, which means it might contain API changes, like some uh a some code changes that you will need to introduce, some dependency conflicts. Um and in this case, um you might want to wait with the upgrade, and wait maybe for backport, which basically mean backport is taking the fix from the from the fixed version, from a newer version, and adjust

it in a way that it will fit and resolve the weakness in an older version. This uh this process uh is been um it's been around uh with the with the distros, like Debian, Ubuntu, Red Hat. they are doing it in at some cases, and this process is longer, as you can uh can you understand, and there is less interest from the upstream maintainers to to do

that. Um so, let's get back to the research uh itself. So, around So, let me repeat it a little bit. So, we build CVE journeys, 40K of those CVE journeys uh we scan 250 container images discovered almost 8,000 unique CVE. And the thing we saw is that more than half of those CVE are within uh are within a package that you might not need it. So basically

it means those are packages that maybe are like optional dependencies like unused US utilities, some dev tools, stuff that you probably doesn't need in your way of production. And we can tell that because we compared with this uh those CVEs with other flavors and with and against Echo produced containers. So we can say which package is essential and which one probably not. So we saw that around

90% of the remaining CVE has a fix available, right? The the around 10% without the CVE so we tried to dig a little bit deeper than that and we saw that eight of those are basically disputed. Disputed means that there is a disagreement between the maintainers and and the ones that report the CVE which there there will no be any any fix here. We can also see

that the higher the severity, the most chances that the fix will introduce. And around in the the the patching gap. as I mentioned around 40% of those CVEs still living the container for more than 6 months. If we taking out the bloated one, we it's reduce the number, but not that much. So, what we can actually do about it, right? Let's get into the practical one. Um

so, when we try to compare the latest version again the older tags, uh we saw that around 80% of the the latest version has fewer CVs in the latest tag. How many less? Around 40% reduction in the CV volume if you will use the the latest tag. What happened with the the other 22% as you might you might ask? So, the answer is that they added new

functionality, new feature, new packages, new CVs. This is why there are cases where the latest version has more CVs than the oldest one. Yeah. Um my suggestion is you should start with just choose your flavor distro. Because if you will go with the default one, you will start full tag, right? With the full version of it. Which basically means you get everything in. It's around 1 GB

in average. You will end up with 1,000 to 2,000 CVs. And most of this image, like most of those packages, you probably not needed, right? There are some utilities like Git, uh you might uh SSH, Wget, you might not need it your production application, uh build tools, stuff like that. You might want to use them as a multi-stage. So, in your CI when you compile stuff, when

you build stuff, it might be good idea, but not in your production, that's for sure. Um you should go with one of the minimal flavor uh out there. There are like uh few options. Um each one of those uh choose his own direction. Basically, they all try to give you the the the essential binaries, the essential uh capabilities uh that will work without any um behavioral changes

in your Um but you still need to be to be aware that it will not contains uh the development uh stack that you are used to it. regarding um actually, what what you can do when you choose your your preferred distro and still you have few dozens of CVs. So, there are still tools that can help you with that. So, the first thing is about focusing on

your application dependencies. There are tools like Dependabot and Renovate, which uh basically uh scan your uh your packages manifest list, see which dependencies you are using, uh compare it against the the advisory stuff like that, and they actually uh if there is a fixed version, they create a pull request with the with the fixed version and help you to to make the uh the bump. Few things

that you need to care about is that it's still creating a PR, it need to run in your CI. most chances, you need a human review here. Um if it's like major change, you might want to to add some test unit test to make sure nothing is break. and if we are talking about transitive dependencies, so this is really get complex. Another direction that is really cool

is Koopa. Koopacetic, which is basically a CNCF sandbox project, sorry. It allows you to patch directly the It is based on a buildkit, which is the Docker default builder tool. And basically what it helps you is to add a lightweight layer with the patch inside inside your existing containers. Means you don't need to rebuild your once again. This is super relevant when the CV is on base

images that you are not have the control on. Um and in case it where there is like, you know, some one-day CV that drops on Friday and you don't want to wait for the upstream base image to uh to fix it. You want to act it uh right now. So, you have the ability to actually uh add a layer to the existing container with the with the

bump. Um okay, so let's summarize with the really takeaway here. So, we saw that more than a half of the CV actually originate in bloated packages that probably you don't need it at your production. And around 90% of those CV as fixed version, but still 40% leaves in your containers for more than 6 months. And this really emphasize that the real problem here, the real propagation lag,

is not on them on the maintainer side that need to deliver the fix, but the propagation lag is on the process that requires uh in order to run the the the process to to clean the image. and that's it, basically. Thanks for joining, and let's make our container clean again.