About this talk
This talk addresses the evolution of software vulnerability management in the context of today's package-centric development environment. The speaker discusses the challenges of tracking vulnerabilities due to the vast number of packages available, highlighting the importance of effective identification mechanisms. They introduce the concept of Package URL (PURL) as a standard that allows developers to easily associate packages with their respective vulnerabilities across different ecosystems. Additionally, the speaker explains the role of standards like SPDX and CycloneDX in making Software Bill of Materials (SBOMs) actionable. The talk emphasizes collaboration in the community to improve the identification and management of vulnerabilities, ultimately aiming to enhance open-source software security.
Full transcript
[music] So you guys are can hear me? All right. Right. Good. Um, so quick words about me. I I maintain a bunch of open source tools and and data and standards to help figure out what's in the code, where does it come from, what's the license, are the nonsecurity issues, and and eventually are they health issues. Um if we look back a bit in the the 90s
and early 20s early 2000s sorry the the way the approach to volity management the things were really centered around is there a problem in Microsoft Word or Adobe Acrobat there were very few vendors and that's where you you had the emergence of uh vulnerability management The way we develop software today is around packages, meaning we we literally take things a bit like Lego. And the skill is
selecting which package you put together and what's the glue you put around that to build a real working applications. Um, and there's been really an explosion of the number of packages. We're talking about millions. I mean 20 30 millions npms, three four millions of Maven 200 million repos on GitHub. So there there's a truckload of these everywhere. Um and we we can build apps very quickly. You
put a package of JSON, build a JavaScript app, you containerize it and you have 10,000 package in really a snap. so back in 2017 we we were building already this tool called scan code and all of our users were asking oh you you can scan package for origin license but what do you do about security and said we do don't do nothing and they ask what do
you do about security and says we don't do anything and third times a charm we said well let's look and there was this thing called the the ational voluntary database in the US. There was another stuff called OSVDB which mean opensourced vulnerability database which was absolutely not open source very proprietary but at least some of the data were available and we have a scan and we're trying
to search to find if there are nonsecurity shoes in in this database and it was majorly painful I mean very manual intensive process and lot of weird things like For instance, uh, Zilib, which is a well-known compression library, was reported at the time as being from the GNU project. And if you know about you're aware open source, Zilib is produced by uh Mark Adler mostly is the
lead maintener today and and Jean Lugali, they have nothing to do with the the GU project. Um, so that was difficult to find a way around. Um and so we we came with a way to very simply extract an identifier from the scans. And since we could not find a way to easily search in existing voluntary database, we we built our own um not really building but
more aggregating existing data source to make sure we could key it by this identifier. um which would help us to go smoothly from a scan say an npm or a Pippi package to a lookup for vulnerabilities and that became what's called package URL today. Uh we introduced it first in at FDM and there was a bit of a social experiment there in a way which is it
it was resonating with quite a few other people which had the same problem we had um because it was painful for everyone to look up in vulner database. So we took the time to extract a spec and invite everybody to chime in. Um and eventually uh it became a standard which was clergy or not really a standard but a spec which was cludgy but useful adopted a
bit by everyone and and eventually we we became a formal standard since uh last December at ECMA. The the the important thing here is that it's not something that was made top down, but really grew from a need that we had just to be able to connect two tools with a simple identifier and and taking the time and effort to make sure we could share and invite
others to to participate. Very often people have no idea that we are behind the package spec which is fine because it's been adopted by a larger community and that's how it should be. So if we go back a bit what's what are or where the alternatives to package rail there's this thing called CP or common platform or product enumeration at the national volunte database um which is
anifier which works very well when you have a package that comes from Microsoft so it has a vendor and it has a name and it's there's not too many Um, another way is to to compute check sums. [snorts] You you have uh standards like omnibore or the software ID which are essentially checksums. They're they're ga one blobs to to make it short. They're great. The the thing
I like to think of is if I go to your house for a barbecue, you don't give me the GPS coordinate of your barbecue in the backyard, right? You give me your street address. That's where the pearl is. Check sums. You have an infinity of them. Um, and one bit is enough to have a different check sums. and in a way so they're useful but very often
a bit too precise for for being really something that can work easily at all scales. Um the there's there's a lot of lookalike system that been adopted um recently in the CV schema the letter CV schema for about a year now something which resemble pearl with an ecosystem a name and a version um also in the Google OSV schema and and OSV database they've both also are
using Pearl and you can use also plain download URLs Um so the point is that there's no perfect solution. The there there's a few contenders in many case we have to maintain mappings and ways to to resolve one to the other. It's hard if you've looked at the specification for CPE um several tens of pages and it's it's a whole system of configuration which is surprisingly complex
and super hard to implement in code if you ever tried. I've tried and I gave up because it was too complex. So standardization we we worked over the almost a year now uh with ECMA and specifically a a group called TC54. You may have heard about ECMA for the standardization of JavaScript which is TC39 I guess TC stands for technical committee or I think that's what it
is. Yes. and together with other projects including cyclone DX uh oies t transparency change API and a few other related specs. Uh again coming from the ground the ground up the ground up bottom up to build a set of minimal standards that make sense when you use uh uh and convey data across software supply chain. So sbomb is one, package rail is one and and t is
another one. So just to give some examples there's a a spec also called CL called common life cycle enumeration and the intent is to know that well this package is no longer maintained or this Python version is going out of maintenance at the end of 20 26 uh in and and be able to capture this information structurally. So one thing to import to to to recall there
sbombs are they're an inventory literally if you if you abstract everything about this bomb is just a list of the third party packages you use in a product. The thing is Pearl is making these actionable and it's not missing that there's actually some researcher that was saying yes Pearl make sbomb actionable. The other thing also is that uh it makes Sbomb less relevant because if you have
an SPDX SBOM or and a Cyclone DXS bomb and they both have package RL you can parse and extract that and and use the data for further processing which is a big win and I I I can talk specifically there. I'm one of the co-founders of SPDX and I'm also a core contributor to Cyclone DX. And at Fosen, by the way, we we we were I was
on stage making a presentation with Steve Springet, the leader creator of Cyclone DX. And and so we're trying to to make things work together um to remove some of the burden you have on processing all these documents. Unfortunately, you will have both to support. you have to you will have to support both SPDX and ECODX because your users will demand one or the others and you will
receive one or the other. That's that's a guaranted fact. And semantically beyond packages they don't always mesh correctly at least on the package side if you have pearls in one pearl in the other things should work and and pearls are kind of central to to What is Pearl? So it's super simple. It's that stuff here on on the right. Pkg as a prefix like if it were
HTTP for an HTTP URL, a package type which is really a short code for the ecosystem which is very obvious. NPM for npm, Pippi for Pippi, Maven for Maven, um dev for Debian, RPM for RPM. So things which are as simple and obvious as possible. and then a name space and a name depending on the ecosystem. For instance, in Python, it's a flat list of package name.
There's no name space. In npm, you have an optional scope with an at sign, which is sometimes print, sometime not. In Maven, you have the group ID and artifact ID. So the group ID would map to the name space, the artifact ID to the name and a version and that's it. There's no more to it than that. That's super dumb and simple which makes it easy to
use and and and easy to implement in many places. So if we fast forward today, um pearls are really central to Cyclone DX. I mean they've been really central since version one of Cyclone DX. Um they're also integrated in SPDX and also other related interesting specs in this field. Uh CSF the common security advisory framework or format um is supporting Pearl to identify Um, OpenVEX, which is
another format for vulnerability exploitability exchange at the Linux Foundation, um, is also centered around package rail and pearl. most SCA tools, so most tools in the space will talk Pearl. And if they don't, you should ask your vendor if you're using proprietary rule or ensure that opensource tools use pearl. If they use sbomb, they will usually do that. So at some point of time, you'll have to
deal with pearl which is great. Um it's now as I said a standard at ECMA. Um thanks to the support. How many of you pay taxes in Europe here? You raise your hand quick. Thank you. So you've actually paid for the standization of package at TechCMA because we received a a a bit of funding from the European Union on the one end and if you're German specifically
from the the German sovereign tech agency. So thank you for that. It's yours basically you paid for it. Um we are next going with ECMA to present that to ISO which is yet another level of uh u I'd say complexity or formality. It's very intimidating. Um but that's great. They have a special process to streamline standardization from ECMA to ISO which is called GTC1. There's a lot
of acronyms there in standardization. They love it. related and and standard we're working on that's already adopted in Cyclone DX and CESAF and a few other places. It's verse which stands for version range and the idea is simple is when when we wanted to store vulnerable version ranges we need to have a simple syntax and every single advisory format every single ecosystem is using a different notation
for version range. Maven is using mathematical notation with square bracket and parenthesis. uh npm is using something people believe is sver but has nothing to do with server. It's their own convention adopted by node and npm and so on and so on and they're all very similar and slightly and every vulner yet another format sometimes a bit more verbose like like CV orv. So the idea is
to say let's come with a way to specify a range whether it's vulnerable range or a function constraints saying I depend on jungo five and above but not 55 which I don't want because it's buggy uh or I don't want to use any of jungo version four uh and and four five because they're vulnerable kind of thing. It doesn't prescribe anything on how you compare to version.
This is very specific to an ecosystem. We we've built a library first called universe and there's now implementation in Ruby, Java and Go. I think there's one in Rust coming up to to actually compare two version and it's surprisingly difficult because each and every ecosystem evolved a different way to say this version comes before and after this version and a lot of ecosystem say oh we're using
server and that's a lie most of the time um I was check yesterday actually checking very briefly a a thing that was a jive And uh technically this is sverver but when you look at some of the versions for instance this is not sver that's a CV record by the way and the problem is you cannot pass this with a sem compliant passer. It's a semish. Um,
Nougat for instance says, "Oh, we use a server like schema with not three segments but four." And every single ecosystem is own weird stuff. Um, down in some case to bugs in the programming language or specific frozen accident in the Node is a very good example for that. We'll be moving that to ECMA because it's already part of uh ISO and ISO standard CSF. It's already part
of uh ECMA Cyclone DX, but it's not standardized yet. So, we we need to to hurry up. Um the goal is to have this ready by the end of the year. We've been able to build a few funny tools for For instance, a a proof of concept in which is doing non vulnerable dependency resolution where on the one end I have the volunty database which is kit
by pearl and I can have a set of vulnerable and fixed version for a given CV and on the other end I have my functional constraints says I don't want this package version or this is the minimal version that I support and you can combine the two feed that to a dependency resolver and says, "Hey, here's a set of dependencies that are both functionally what I expect
to work with my set of packages and my code and which hopefully are not vulnerable. There's a small problem there is that most of the time you cannot resolve a dependency tree that's not vulnerable. Uh that's which makes the the effort a bit a bit futile in a way. So we were talking a bit about option. It's it's adopted a bit everywhere. Who who has never heard
about Pearl here before? Who has heard about Pearl then? Good. Great. Are you using it somehow or are you forced to use it? Yes. No. You've seen it at least. So it's all these tools. Uh what I like very much is also used in proprietary tools which uh sometimes reluctantly um for instance ex Jeffrog and their X-ray tool today uses Sbombs in Cyclone DX and produce pearls.
Initially the idea and the concept of pearl came from a project from Jeffrog where they had something which resembled pearl and I I credit them for that. I don't know why they didn't wanted to contribute further afterwards. uh but initially came from an idea they had on a joint project they were doing with Google called graphas and it came back to them afterwards anyway because they have
to use it in in in a way because of the the presence in all this spec one thing I'm very proud even though I can take credit uh but I didn't do much for it is the inclusion in the CV schema since October which means that it's possible there's very few at the moment but it's possible to have upstream CV to contribute to miter with a pearl
and we'll come back to it in a in a second I hope it's clear to everyone that it's we we're in a world that's package centric you may have noticed there was a conference I think that was last week or the week before last in the US called Voncom around vulnerability management and the the folks maintaining the national V announced that they were relaxing a lot what
they were doing and they they would give up uh doing curation of all the vulnerabilities which were they were not able to do it's important because they were the ones granting what's called a CP to a vulnerability they they cannot cope with that anymore. Um, and it's it's interesting. It reinforced what we're doing. It reinforced also several European initiatives like uh all GVIP is one. There's the
volunteer lookup is a project in Luxembourg which is the back end for the NISA EU and and eventually there's a lot of work to make sure we have better data, but it's it's um I think it's useful to see that we we have an an identifier to support that. So the the point here I want to say is that vulner database are not helpful. In fact, in
in the tools we build vulnerable code which is aggregating all these vulnerability database we were naive and the original schema was to say hey here's a vulnerability here are packages and they're related together through a version and the idea that there's a single vulnerability for a package was completely uh flawed and naive. There's instead advisories where there's an advisory which says this Debian package range is vulnerable
to this security problem which is not the same as the same related package maybe Zib upstream uh which has this set of vulnerabilities but different patch level and the the the thing is not so much a central vulner database but a database of package with vulnerabilities probably much more important. So if we if we look at some of the the challenges we have here is it's it's
common to use a lot of packages as we've more and more eventually with AI I don't know if any of you have heard about this stuff called malus.sh SH which was presented at FOSDM. The idea is to say the heck with dependencies. We're gonna vibe code a refactored version of every dependencies going to have a new identifier and there's no volume. It's a new package. Nobody knows
about it, right? Um so that's that's eventually something that's coming on. That's a strategy. uh the the problem is uh we we use more and more package there's a lot of tools it's confusing there's a lot of requirements you we had a presentation on SR just before that so you you've seen some of that it's it's really a mess the to understand the the complexity here is
that even when you're trying to do something looking at this exact vulnerability we're looking at a bit earlier this is a page that's very similar to what we do on vulnerable code. Uh I don't have vulnerable code up by the way because I I went offline. It's a Dutch vulnerity database published by the the Dutch National Security Center. It's something on Jetty. It doesn't matter really which
one. The point is uh when you aggregate all this data around a vulnerability, you get this mess of uh information about different origins of the data from Red Hat different RHSA and uh SAP, Oracle, NetApp, Oracle, NVD. Everybody comes with their own bits and none of that is really actionable. What you care is really I use this package in my product in my system in my organization.
I don't care about the the the whole gamut of stuff across uh uh every single packaging system across the board. That just gives you a bit of an idea of the the mess we're in when we go to a mode we are vulnerability centric. the other thing which is interesting, it's something that dawn on me yesterday uh with the emergence of agent TKI as being viable way
to uh analyze code and find vulnerabilities. I mean there's been a lot of hype and marketing hype around the entropic clone mythos which is essentially a a souped-up version of opus 47 without guard rails and and and longer thinking capabilities. But the point is that becomes eventually super cheap to find voluomities and exploit it. And if you say it's taking a hundred euros, unless you are naive,
fixing this and securing a package going to be much more than hundred euros. I mean detecting the vulnerability is going to cost you €100, but fixing it is going to be much more expensive. And if you use 10,000 packages in your codebase, are you ready to go to your management says, "Hey, here's my budget. It's going to be€ 10 million euros just to secure my opensource software.
Are you ready to pay for that? Nobody's going to be ready to pay for that. I think it's it's reinforcing the value of open source because we can probably pull all of our efforts there when every single uh bug and security bug can be mutualized in in terms of securing auditing attesting to the security of open source project. So let's go back to the main topic now.
So some of the tools we can do with helping with license compliance. Why? Because if you if you abstract a package to just its pearl and spdx license expressions, you can query a database kit by pearl and get license information back. Um this is really helping a lot on on automation because you again have a single identifier that you can carry across different tools. this is enabled
as I said both by cyclone DX and SPDX it's I I'm painting a rosy situation. Well there there's a lot of issues also I mean things are not perfect far from it but it helps a lot. The second thing is for vulnerability responses. Uh lock forj of course is always the the poster child uh of the the the thing in the space where was in December 2021
if I recall correctly where a lot of vulnerability uh researchers and defenders were on the hook to fix that. Being able to look up and know that you use a vulnerable version of LO 4J across your product was something a lot people scrambled and are still scrambling today. Uh but the point is it's enabling that making it easier to look up on the side. We we have
at about code started a project with the the maintainers of lock um funded also with your taxes. It's a small project, so no worries. Um, to look up for known hidden Maven jars that have a what's called a shaded version of Lo forj, which is still lingering there five years after and nobody knows about them or maybe some people knows about know about them, but they're still
out there and they're exploitable, but they're not referenced as being vulnerable. Um so the the thing is to do a binary analysis starting from the binary jar and check if we have in the jar the corresponding source and if we have specific telltails about lo forj and then we'll be able to extend that beyond lock forj of course. So CV in any case is going to have
and has in its schema support for Pearl, Cyclone DX, open vexafit together with OSV and OSV is also the format that is used by the the GitHub advisory database by consistent workflow. If you cannot figure out when you have a package that goes through your supply chain that it's the same, there's no automation Uh, and I wish that Pearl was perfect there. It's not, but at least
it helps. if if you have this automation possible, you can literally take the best tool for the job to enrich, look up, uh, and collect more information and eventually making sure you have something which is actionable. So if you receive an sbomb if you're not already rece who's receiving sbomb from third parties already raise your hand. Okay so sbomb don't exist in fact right [laughter] but if
you're receiving an sbomb the difficulty is going to be to do anything about it. Maybe you're going to receive an SBOM from one of your software vendors and you want to know is there any package that's vulnerable. If you have that then you can look that up easily because you have free and open volunte database like ours but not only could use OSV uh to the same
effect or volume lookup to do this kind of check. Same thing you can check vol database and and there's much more you can do. You can have specific policies. Maybe you don't want to use any of the open source projects of a competitor. Uh or for say patent reasons, you may not want to use a specific codec from Jstamer or FFmpeg because uh uh it it's it
has specific patent requirements. So some of the [applause] these are some of the some of the workflows. So key point is really moving to a world where you can mix and match tools in in a virtual neutral vendor neutral way and mix and match also database some of the examples of database. I'll put the slides by the way. I hope we can upload it but I'll put
all the slides. So you have all these pointers there. You have a bunch of projects that exist there in the space. I'm lying because clearly define is a project that I maintain and lead at the OSI. It's not kit by pearl yet, but it's in our road map and it's very close to pearl. It predates predates packagl but pearl db devs.dev use pearl ov ecosystems sonatoss index
which is also commercial volunte database are all kit by pearl. So clearly defined we're soon to be kid by pearl. We have 55 millions and 70 terabytes of scan code scans which want to unlock and make make easier to to uh to reuse locally. we still have a big problem. You don't use sbomb so you're not impacted right but sbombs as they come out from tools are
problematic. You take two tools and they won't come out with the same input with the same output. It's a mess. Literally uh tools don't agree on basic set of system package installed in an RPM container image. They don't you'll get different opinions. It's supposed to be a fact. So we still have a lot of work to be doing there. And some tools also uh invent packags. So
they're they're good boys. They're doing the package rail thingy, but they they make it up, which is as bad as not having one because it cannot be automated. So we've been working a lot on making it a bit less messy. Standardization is one thing. uh we've putting out their tools including validators to make sure that you can integrate this in your software locally without any dependency on
anything. Um to avoid creating pearls that are not usable afterwards. Um so we've talked about some of what's next. ISO standard for pearl ECMA for verse. Um we're working on clay define. We're also working with software heritage by the way um to scan the whole archive both for license and package metadata and expose also in the archive package URLs and and uh want also to work on
a benchmark. So rather than say your tool is bad and not say not knowing why, we can say pass this benchmark and at least you can have some guarantee that if your tool is able to process and spit and detect packages correctly, we we we have uh we are in a better place. There's just to finish this very big problem here which is this small set of
packages in C or C++ like say curl or the Linux kernel they don't have a pearl because the whole premise of pearl is to say an ecosystem is the one that gives name it's a the package types pipi npm maven they they are the invisible hands which guarantees names are uniques and there's a convention for naming and all that. There's no such thing for a lot of
package of package registries and we're working with something specifically first with the kernel to make sure they have a way to cent pearls for for the kernel that are easy to relate to and they want to essentially back field all the CVs published by the next kernel and forward uh enter pearl going there knowing that if you follow a bit the news the the kernel project is
the second largest producer of vulnerabilities among open source projects actually globally uh open source and open source they they they're very prolific um it's an important thing so um that's pretty much it so we talked about the ECMA standard uh uh and the different projects here including OT which is right there. It's a community. We need your help. Uh we're supported by a public benefit nonprofit organization.
Some money from the European Union and and some other government including the STA. Um but if you can help and come and join us u for instance right now we're being submitted to a deluge of vibe coded and the AI generated pull request. It's a truckload of work to actually review them and decide is it ham or is it spam and and that's something like when I
say deluge across all of our repos there's probably about 200 to 250 of these and among these there are a few genuine contributions from aspiring or new contributors or existing contributors I don't know which one and I we need help to be ble to just going through triage triaging this uh that's that's really a lot lot of work and that's it. So you have questions briefly. No
questions. [applause] Good. Go ahead. So you mentioned that everything is to be considered as a package but and we as software engineers we put like the when we create software today we take a bunch of packages and then we create some clue around it and that's basically our application. So when we then have our application can we give a pearl for that as well or how should
we treat this also as a package. >> So ideally yes. So for instance um if you think about an aggregate of package that would be a container you can have a pearl for container and part of what we're doing for CC++ packages here is also to have a way to christen pearls for private usage and commercial applications. Uh it's going to be based on the domain name.
So literally the pearl uh for the kernel is going to be kernel.org. I mean SID as a type and kernel.org or and then we we have a simple way to to ensure they can be registered if you want to and that means yes you can crist on these and because your your application whatever that is I don't know what you're doing but whatever that is is something
that if you're distributing maybe subject of its own to security issues maybe including bombs and something that especially in the context of theory you'll have to convey attributes about. So the answer is not just now but yes it's literally a couple weeks away. We've been discussing it for months in in the the the bi-weekly Pearl community calls. >> Okay, perfect. Thank you very much.
