SB💣💣M: Making SBOMs Play Together - Jacopo Bufalino, CNAM & Agathe Blaise, Thales SIX GTS France

This talk focuses on the challenges of achieving interoperability among software bill of materials (SBOMs) across different tools. The speakers, Yakopo Buffalino and Agel Ble, discuss how various SBOM generation tools report inconsistent results, particularly in the context of vulnerabilities associated with container images. They highlight a study conducted on a representative set of containerized applications, detailing discrepancies in the number of packages and vulnerabilities reported by different tools. The speakers also investigate the complexities of parsing package URLs and the implications of these discrepancies for vulnerability detection. Through their research, they aim to establish best practices and guidelines for improving the consistency of SBOM outputs, ultimately advocating for compliance with existing standards.

Good start. Yeah. >> Okay. Hello everyone. So, welcome to this talk. Uh, today we will talk about sbonds and especially how to make them play together, meaning how to make them interoperable. So today with me there is Yakopo. >> Hello. Um, my name is Yakopo Buffalino. Uh, I'm a doctoral candidate at AL University and recently I'm also a researcher at the Kam Institute in Paris. And my

name is Agel Ble. I'm a researcher at Talis 6 in France and I'm studying the virtualization of network systems and the security. So just a quick note before we dive into the technical content. So this work has been funded by a European project uh which is SE for sec. So I put the link and the QR code here in case you're interested in our activities. So a

little bit of context uh during the last CubeCon Europe in London last year with Yakopo we introduced our research on enhancing software composition analyzes resilience against the container image offiscation and during this presentation we studied the impact of container opiscation on the Sbomb generation and especially we showed the limitation of Sbomb generation tools uh in handling images in which important metadata are deleted. This can happen for

different reason in particular if the size of the image has been minimized or if packages have been installed in a non-standard way. So during this presentation we show you this large table where you can see the impact of opuscation. Uh so on the road you have the different uh list of obuscation techniques we apply into a based image and you can see for several tools the number

of vulnerabilities and package reported as ground truth. So this is the first row. We can see we use TV, sift and scout as open open source tools but also Microsoft, Google cloud and Amazon tools. And during this presentation we noticed that no one asked us why or noticed uh why different tools has very different numbers of packages and vulnerabilities reported as ground truth. So you can see

the number of vulnerabilities ranged from 123 for scouts to 1,164 for TV. So nearly 10 times more vulnerabilities. And if you give a closer look, you will notice that this difference arise mainly on OS packages and it is also quite woring because base layers are reared a lot. So during this presentation, we will focus on those OS packages and analyze them more in depth. So today we

will study the particular topic of sbomb interoperability across different tools. Uh for that we first derived a set of different objectives we would like to cover during our presentation. Uh so first we want to analyze why very different number of packages are reported for the same application. So we will often uh look at the reasons why uh we have very uh inconsistent results and packages across different

tools and this happen especially when scanning complex container image. Then our second objective will be to analyze the impact of this inconsistencies on vulnerability detection because as you know his bombs are mainly used for vulnerability detection and then we want to see how it affects um the the vulnerabilities reported for given products. Finally we will investigate whether there is a way and how to do to automatically

mitigate those inconsistent SBAM outputs. We will also discuss best practices and guidelines as a community to improve the transfer freess of those sbomb outputs. Before we dive into our contribution and main work, I will reintroduce some of the key concepts related to the sbombs package identification and vulnerability detection. So as you know a sbomb here is a structure inventory that lists all of the components dependencies and

metadata of a digital product and they are usually shared between people like the maintainer of the application the developers and the final users and it is typically a JSON files composed of the information about different packages that can be found uh in the software products. It includes uh for each package a different information. We have the name of the package. We also have the file where it

has been found, the source package, the vendor, the version and some metadata. For example, the architecture, the maintainer, uh the Debian version, the package URL which is the unique ID of the package. uh during this presentation we focused on per but we also have the common platform enumeration or CP and information about the license and among these different elements we don't really know which ones are important

or not and which one to consider for our ESBO uh you have to know there is some particular guidelines from the cyber security and infrastructure security agency on how to uh on the minimal set of elements we need to provide providing the fbomb for compliance. Uh we note also that we found similar challenges regarding the AI bomb. So AI build of materials or extend this concept to

AI based systems and it relates to the AI models uh frameworks also the data sets and different tools for the training pipeline for example. Uh so now in terms of pearl and CP uh we focus mainly on Pearl. So those are standardized strings that uniquely identify software packages across different package manager and repository. So it mean if you install a Python package uh via Debian it will

result in a different P that if you install it from source. Uh so the format is the following. You first have the package header and different information. In particular we have the type of package. uh optionally the name space, the name and the version and some qualifier that represent the metadata and among these different elements some are mandatory like the type the name and the version and

some of them are optional. So it's hard to know exactly the information to to include to to be complied with the standards. So if you look at the different examples below, you will see multiple parts that describe the same package but with different levels of details uh and content but all of them remain compliant with the standard. Uh finally what do we do with sbombs? We primarily

used them for vulnerability detection. So first we need to read the package identifiers. So the pearl uh from the sbomb the boss sbomb can be produced by ourselves but we can also use external sbomb produced by other people and we need to compare each package against the database of vulnerabilities with the exact version matching. uh so it can be a public database or our own again and

uh this is quite important to shared sbomb for compliance in relation with the cyber resilience acts and the digital network acts uh the AI bomb are mostly concerned with the AI acts so now let's plan our research uh we will introduce our methodologies uh we follow to answer the three objectives I presented earlier To understand the main differences across the tools, first we need to select a

representative and diverse set of tools uh to handle both the SBAM generation uh and the vulnerability detection. And we also consider both opensource software and commercial tools. Second step is to construct our data sets. For that we selected real world containerized application from docker hub. For each application, we first need to compute the Sbombs using each of the tools. And next step will be to do a

cross tool vulnerability analysis. Meaning we will use the Sbomb generated by each tool and process them with each other tool for vulnerability detection. Finally, we will study and analyze the difference and trying to understand the reason of these discrepancies. So this is the list of tools we use for Sbomb generation and vulnerability detection. In total we have four free software and including free open source software and

four cloud-based tools. So we have encore uh 3V uh tools and we also have Amazon, Google and Microsoft commercial solutions. This is the data set we selected. So we have the 20 most downloaded container images from Docker Hub supporting both Debian and Alpine images. Here you can see the details on the number of package and number of vulnerabilities. Now once this data set is collected, we perform

our analysis into two steps. First the SBA generation, then the And now Yakobo will present our main findings. >> Sure. Thank you Aat. So uh as said uh our our research is kind of split into two parts. We have the first part where we're going to do the sbomb and package analysis and the second part on the vulnerability CV type of thing. So let's start with the

asbomb analysis. So to give a little bit of context what we did here, we took all of these images and we started uh computing the sbombs and from them we collected the number of packages. Simple as that. as you can see here uh well some of them produce some asbombs provide uh more packages than others. And uh this was a little bit surprising not that much uh

but a little bit. And so we started digging a little bit uh deeper and we started this asbombs. We compared them one by one and we came up with some reasons to understand like there are some reasons why they're why they're different and we're going to show them now. First uh this this is more more like a Debian type of problem. It's the binary versus source package.

So whenever we install like in Debian there's this concept of bin of source package which is kind of the git repository where all the code is there and and then uh when we install uh the packages they come with a set of binaries to make it more clear we have the shadow uh source package which installs the login uh binary the password binary uh lib sub id

uh uid map. So basically uh some of those tools actually the ones that show more packages what they do is that they they list in this asbomb both source package and binary package effectively duplicating the the entries. Uh and this is one of the main differences in the in the package output. And then we saw another interesting pattern. We saw miscellaneous packages. those type of miscellaneous packages

are packages that are there just to give more context information uh in the asbomb uh but that they are not really useful. For instance, uh they record information about the uh the fact that the asbomb is for a container and not for something else. Or they add information uh regarding the um package manager or as they call them operating system uh handling the the operating system So

we're kind of happy. We know uh the more or less the the differences on on those packages. So it's it's not that bad as it seems. But uh let's go to the next step. The next step is comparing the package URLs. So first we took the the uh the package URLs and from that we extracted the name and version. Simple as that. And u so yeah we

use the name and version and then we compare them one versus one. This sort of matrix testing that's going to be like the north star of of this part of the of the presentation. And as you can see, we have many cases where we have perfect matching in terms of number of packages and uh version number and name. Uh and in some cases, as expected, we see

a bit less compatibility because some tools they actually show more packages. So that's that's expected. But what was not expected was what happened afterwards. Afterwards, we compared the package URLs with each other. and and and uh so compatibility goes to 0%. like only the diagonal which means like the same tool the other zero complete um uh first question is why why I don't know so we we

had to find a way and we said okay let's go back to the standard because package is now an ACMA standard so there should be something written there so we selected a very complex package that is Python 3 magic++ coming from Debian and that is interesting because it nice combination of letters, numbers, it has a plus+ which is a plus. You will see why. And um and

basically we started by looking the standard uh we took we took the package that is standard and then we we created we generated the package else using the other tools and first tool had complete differences when it comes to the encoding of the plus+ that's why it was interesting then we have problems on the version number and all the this optional qualifiers were completely mismatching the standard

because believe it or not even if it's optional the standard mandates if if it's optional. So, but if it's there, it should be like this. Like if if it's optional, you need to have the architecture like key and also the DRO key, not something else. Then we continued another one also in this case optional qualifiers. Another one here we have double problem uh qualifier and encoding of

the version. Then we come to to to this one here. Basically, we're missing a bunch of important information here. We cannot basically say which Debian version was used to install this package. We don't know the architecture. It's kind of okay. Then again next one always problems with the optional qualifier. And here again uh as you can see like there are lot of different issues mainly related in

in the Debian case with this epoch on on the version but also on the encoding part. But so kind of now we know why they're different. Uh let's put that that aside. We're going to look at that later. And uh let's move to the second part, the CV analysis. Uh I want to move to the CV analysis because now we have kind of a interesting uh enough

idea of of the packages of the package URL and u now I want to keep it there a little bit. So let's go to the CV analysis. So with that uh what we did is that we started by just listing the the different CDs and again we have spike somewhere there and uh again that we had no way of understanding what what that was about. So we

checked all the CV reports of the different tools. Uh we checked the number of packages that were affected by each CV. We looked at the duplicated ones and uh we actually found a very interesting uh initial uh issue here. So most of these vulnerabilities are actually kernel vulnerabilities and since containers kind of share the kernel with the host those shouldn't be there. Maybe it's an opinionated stuff,

but I think the community agrees with that because also we looked at the all the different tools uh the ones that were open source and there were issues concerning that fact. So we went from a spike here to kind of normalized situation where we have some tools that are actually detecting very similar CDs, but still we have some out some some something odd happening on on on

the right there. So we did some more digging which is not complete but it should be helpful to to give you an idea of what can go wrong. So first thing that we found uh was irrelevant CVS. So some tools that were reporting CVS that did not affect the analyze environment. Let's say uh we have vulnerability but it's for Apple products and we find it in a

Linux container. Okay. Then uh another issue was the package URL parsing. So some like in some cases the one that I showed you earlier this package URL doesn't actually have all the information to understand the type of of packages and therefore also the CVS that that map to that and that was causing some mismatches between what we thought was vulnerable and what the tools showed as vulnerable

and finally precisions. Believe it or not, some cases when vulnerabilities were too old or deprecated and the tools they treated uh the package as nonvulnerable only because the database that they were using was up to date too much up to date. And another interesting uh thing to tell here is that for instance um in um in Alpine it looks like they only give you information of the

CVS that they have fixed not the packages that may be affected by vulnerabilities. For that you have to go somewhere else and not all the tools uh knew that. Um so before continuing here I want to look a little bit at the things I was talking about. So first case as I was telling you this is this is actually coming from the Dian security tracker. We have

some cases where we have a note saying this is not for us it's for Apple. So people shouldn't like tools shouldn't for Linux shouldn't decode that as a vulnerability for for Linux. In another case that's actually interesting. uh maybe this opinionated but maybe then we will have time to we will have time to discuss that. The second case we have a crash in a CLI tool and

for them it's no security impact. Now it is interesting for me because while this is true for a bare metal or for a virtual machine if we run this binary in a container as as a main process and we have a crashing CLI tool actually it has a different impact with respect to the the virtual machine case. And finally, we have this other case where basically uh

this ties a little bit on how Debian works, but the idea is that uh the community here said it's it's something that could go wrong, but it's not that important that would require us to create a hot fix for it, which is okay. But maybe security conscious uh people uh like me we're we would like to have everything fixed even if it it requires a huge set

of events to to come by at the same time. I would really like to fix everything as soon as but yeah so this is more or less what happens with the difference on vulnerabilities. So now we have kind of an idea of why the asbombs are different and why the vulnerabilities are different. Of course, there may be more stuff, but that's roughly 80% of what is there.

So, now let's do an experiment. Uh, I think this ties back to what Agat was saying. Let's simulate like a scenario. So, a vendor gives us an NS bomb. What do we do with it? Boom. We scan for CVS. That's first thing first. Uh, maybe we put in a CI/CD pipeline and every day or every couple of days we check for vulnerabilities. And we did the same.

Uh this is what we got. So in gray uh it's like it's grayed out because this cannot happen because the tools actually do not support scanning for third party asbombs. But in the other cases you can see that every time most of the times that we try to search for vulnerabilities in an asbomb that was generated by a different tool we actually see zero vulnerabilities. We have

couple of cases where there is some sort of compatibility but most of the cases it's zero. So basically um this is kind of a problem maybe because uh I don't know if you give me an asbomb I use my own tooling then I find zero CVES maybe maybe there is something wrong and think also about the case of this hard images because for those we actually expect

them to be zero CVS so this wouldn't actually surprise us while instead they could have vulnerabilities So we said okay can we do something for that and we built a really really simple script to kind of map those fields in the in the asbomb and uh the situation dramatically changed. So in most of the times we are actually able to uh kind of find the same vulnerabilities

across all the tools. So it is actually possible to translate the asbombs but tools they're actually not not compliant with the with the standard. Oh I need to hurry here. Okay let's let's have a summary of what of what we said here. So coming back to the original questions why those packages are different. Well tools have different ways of looking at the container file system and mapping

the container file system to to packages and they use this duplicated packages. Then what is the impact? The impact is is tremendous. We tools are not interoperable at all and it's big problem and can of course can cause missing important vulnerabilities. And there's actually a way we could somehow u map those packages automatically, but the asbomb generator should actually agree on the standard. There is a standard.

Use it. Come on, it's there. okay, this is over. No. Um so how do we make this research visible? Like sometimes me and Nagat we talk about how can we share this knowledge with with with all of you community and uh we did two things. We open sourced whatever we did and uh we open source this tool to translate this this asbombs but we also created VIP

code UI to show the differences between these asbombs and TVs. So allow me to demonstrate a little bit what what what we did. Um so here pretty good UI anyway. Uh let's make it okay looks better. Uh so let's uh search for a bookworm container. Let's try with this. This actually this is a container coming from the docker library. And here we can look at the at

the summary of the uh content of the asbombs. So first we can see that we have a total of 459 packages. Half of them they have conflicts so they differ and some of them are actually the same. Uh so here we can see that some tools they have unique packages that other package didn't find. And similarly here here this is a nice view of what I said

before. So tools seems to be very compatible when it comes to this package ID if we only take name and version. But then if we go to the package URL, boom, totally different. And quickly quickly quickly, we can actually look at uh let's take a look at some common packages. Let's say this one. In this case, we can see that we have discrepancies in terms of supplier.

For some people, this person is an organization. For other is a person. And here they are like matched with this brackets. And here with parenthesis also licenses seems to be kind of problematic in this case people here there's an or here there is something that I don't understand and then here even like some ids that I don't know and uh okay quickly quickly quickly so here we

can also see the difference in this package we divide each bit of it and we can see here that we have like OS name and here we have some upstream and dist draw and So this is like a way to practically go there and and look at what what we've done. Um now we can quickly do the same for um um for this for the CVS. And

here we can see that we have a tool that has a bunch of vulnerabilities discovered. And if we go there and look what what this is, this is the famous Linux lib dev which if you don't know it's Linux headers. It's bunch of header files. So yeah, this is um what the tool looks like and uh we'll give you links later. Now quickly quick quickly take takeaways.

So first more vulnerabilities detected doesn't mean better tooling. uh we see a fragmented asboom ecosystem. So what can what can we do shortterm inform your customers on the tools to use or ask your vendors to give you uh the tool that they used to to build the asbombs and longterm bomb vendors come together and define common interfaces. when it comes to to us, we will be there

watching them and trying to ensure that there's going to be compatibility. And uh thank you. And here you find a link to the UI that you saw before. Now I think we have a couple of minutes for questions. Okay, questions. >> Hi. Uh, you mentioned most of the differences were because of irrelevant CVS or incorrect CVS. Did you find cases where the tools missed important CVs so

valid and maybe even high or critical vulnerability? Um yes, we found we found uh some cases of those and those were especially on uh on on the Alpine because tools were not looking for packages that are vulnerable but unpatched and also sometimes tool had issues in Deian when the version was particularly complex. So you have like Dabian version 1 something minus something else plus U2. So yeah

that's what we found. >> All right. Thanks. >> You're welcome. Thank you. >> Yes. Hello. Uh I have a question about the format of the Sbombs you guys used cuz I saw in the UI used SPDX. Why was that chosen over Cyclone DX for example? >> Uh okay. Can can I answer? >> Yeah. So there is a lot of dispute in the in the the community which

format to use but uh we ended up using SPDX just because the tools that we use they all supported it and um honestly we didn't we we wanted to focus on the problem itself that I think it's beyond the format that you use because package URL is usually used in both SPDX and CYPDX. >> Thank you. thank you again.