Paolo Mainardi: Building a Trusted and Resilient Software Supply Chain

In this talk, Paulo Menardi discusses the importance of building a trusted and resilient software supply chain. He explains the current state of the ecosystem, highlighting the unique threats posed by open source software and the complex interdependencies in modern applications. Menardi utilizes the SALSA framework to categorize potential threats within the software supply chain and emphasizes the need for better understanding of project dependencies. He introduces tools and strategies for enhancing security, such as digital signatures and software bill of materials (SBOM), which help establish trust in dependencies. The talk concludes with a call for developers to proactively manage vulnerabilities and automate the dependency lifecycle to improve overall security in software development.

[Music] ladies and Gentlemen please welcome our next speaker Paulo manard presenting the topic building a trusted and resilient software supply chain thank you for joining this talk building us trusted and resilient software supply chain but before we begin let me introduce myself very briefly my name is Paulo menardi I am the co-founder and C of Spar fabric we are an Italian company based in Milan but we

have always operated as a fully remote company we do a lot of cloud native custom development and Cloud infrastructures mainly based on kubernetes and I am also part of The Advisory Board of the Linux Foundation Europe which is a newborn Foundation the European branch of the main Linux foundation with the mission to accelerate the growth of Open Source and Open Standards in Europe feel free to reach

me out if you want to know more about the foundation and how to be involved you can also find me on my personal blog palom mini.com where I randomly post about things I find interesting where you can actually download the current slides using that QR code if you want to follow me along okay if you want to take a picture just some seconds okay let's get get

started in this session we are going to see what is the software supply chain why it's so important to know today then we are going to see the current state of the ecosystem the threats and the mitigations we can Implement with things like digital signatures and software build of materials and finally how to improve the knowledge about our project dependencies and the strategies to choose them more

wisely and safely so to understand what is the software supply chain I want to start with this very generic definition which says that a supply chain is a network of individuals and companies who are involved in creating a product and delivering it to the consumers it's generic enough that can use to describe any kind of product including all types of software products however open source software and

the way we build applications today has unique characteristics and threads that make it different very different from all the other Industries and products this diagram comes from the salsa framework SSA dodev uh which is an open- source project created by the open source security Foundation the op ssf and this project is not a software framework per se instead it's a checklist of best practices that we we

can gradually adopt to improve the security of our digital artifacts as we can see in this map the threats are grouped into three main categories Source build and dependencies and attackers usually exploit dependencies as the main attack vectors which Remain the weakest link in the supply chain today it's estimated that a modern project contains an average number or more than 500 dependencies and that all over 90%

of our code is not actually coded by us now we can imagine with these numbers how much we rely on external unknown actors and how large the tax are phes just on the side of uh this the chain dependency side but taxs as shown in this map can take place at any Link in the software supply chain and these kinds of attack are increasingly public and disruptive

like the cases solarin Cod log for shell and the very recent XZ attack that we are going to see soon but before jumping into threats let's try now to imagine the salsa diagram in action when building a modern application we of course start by very simple by installing our framework such as nextjs Lal jungo angular or whatever we like or we have to use then the next

step is usually to install other dependency from our ecosystem and depending on how we build our application we may need other libraries from other ecosystems such as I don't know JavaScript dependencies to build our from 10 stack or python for machine learning and AI stuff and so on and so forth and on top of that we have our operating system which is strictly bound to our application

we may need system libraries for for compiling additional framework extensions for example it's the case of pH PHP we may need a specific kernel for things like ebpf or other security stuff this is quite common in Cloud native applications or other custom binaries such as I don't know tools for converting images video Etc then we usually have a cicd system with a lot of jobs to do

linting and QA to run tests to collect metrics and so on and so forth and pipe pipelines can even use external Services as well like codco covers code climate Etc which means having an external access to our code base and More in many cases like configuration variables and secrets and this can be quite dangerous when our application is ready we have to ship it on the cloud

where we have several options like AWS gcp Azure versal platform Sage you name it and when you using Cloud platforms we have the flexibility to choose from a wide range of services to deploy our applications and the services may include kubernetes virtual machines object storage MySQL postgress radius Lambda functions and so on and so forth and lastly we have our local or Cloud development environment which is

today another very weird a glomerate of dependencies such as I don't know the code extension jet brains plugins VA modules and bundles or whatever we use to add new features to our de it simply means adding several other external dependencies to our project it's also very common to have access from our Dev environment to the cloud platforms maybe because we have to The Bu we have to

see the logs to we have to check some metrics and such and this can be quite risky as we have seen in the s diagram no point in this chain is free of threats and attackers will try for sure to use all of them especially when the map of services is so complex and interconnected like this where we barely know our direct dependencies what we see here

is maybe the biggest case in the history of cyber security attacks related to supply chain this case happened in 2020 to this company called solar winds they are a big at security company and one of their Flagship products called oron a software to perform network monitoring at large scale for data centers was compromised by a malware that run and detected for months and then was shipped to

all solar winds customer as a regular security update it was later discovered that the compromise was in the solar winds build systems where a vector was designed to inject some malicious Cod another Vector basically during the compilation phase of oron and then hide itself again this is exactly the point C described in the salsa framework as you can see in the image the biggest Tech giants like

Microsoft Intel Cisco were affected by this Vector but even the most important national security agencies like the Pentagon the homeland security and other important and public entities and worldwide national States this case was so big a concerning that the the US Congress invited solar windo and Microsoft president to testify on the case to understand why companies that are supposed to protect us from these kind of cber

attacks where instead their s victims of a long running Global and large scale attack like this and just some months after the solar winds case in 2021 a critical security vulnerability was discovered in the log forj Library log forj is the standard factor for logging in the Java ecosystem and it is basically always present in all Java U code bases either as a direct or a transitive

dependency some have defined it as the most serious software vulnerability ever and according to the Washington Post it borders on the apocalyptic let's see why log for Shell is a vulnerability with a severity score of 10 the highest possible because it allows remote code execution just by injecting a specially crafted string in the HTTP headers so it's quite easy to replicate and to exploit super easy actually

and to make the scenario worse almost immediately after the disclosure was made public multiple proof of concept were published on giab and from that moment on organization had basically zero days to respond and mitigate the threats that started but the good news is that from 2021 to now more than three years later everyone has updated and the issue is gone everyone has updated right the fixes were

released 3 years ago well it's not that easy this is a fresh Snapshot from blog for Shell download dashboard made by sonat type and this data clearly shows that 35% of the total downloads just in the last week were a vulnerable version and we are talking about more than 4 million downloads weekly this is a clear indicator of the fragility of the open source supply chain and

how it's hard to eradicate critical vulnerabilities even when they are public and the fixes are made available before and now fast forward to April 2024 just one month ago for another important well organized and quite scary supply chain attack defined as the best executed supply chain attack we have seen described in the open and this case was even discovered totally by accident how the story begins on

March 29th when Anders FR a postgress developer working for Microsoft posted this message on his Mass while he was doing some micro benchmarking operations he saw that some sshd processes were consuming many many more CPU Cycles than usual so he got very supicious and started to provide them and just some minutes later he sent this message on the OSS Security mating list announcing his big Discovery basically

he found that someone planted an highly sophisticated Vector in the XZ library with the intent of compromising SSH if you want you can scan the QR code to access this message this is a very interesting detailed explanation of what this Vector does and how but let's take a step back and see what is XZ and how SSH has been impacted by this Library XZ utils are open

source projects that Implement lzma compression algorithms that are widely used in the Linux ecosystem even the Linux kernel is compressed with with this format open SSH does not require XZ but some Linux distributions patch it to support systemd notifications and the result at the end is that open S depends on lib and zma 2 has an effect of a transitive dependency this is a textbook example of

software supply chain risks related to dependencies but the scariest side of this attack is that required over 3 years of work to complete by a group of attackers in 2021 someone named gatan joined GitHub and made his first comit to an open source project out of the blue some months later he become a comer of the XZ project thanks to his apparently legit work but even thanks

to a pressure campaign start by some new users who joined the development mailing list of XG just to accuse and blame the project maintainer lassin to not update the project enough in the next two years a series of new patches got merged to XZ which now in retrospect they just look like as groundwork for the future back door but the time they were quite good and when

everything was was considered safe and well hidden gatan dropped two new releases with the back door inside in the following weeks Jatan and his group pushed developers of buun red hot and Dean to merge the new XZ release and eventually they were actually merged in the development branches ready to be deployed I don't know why it's missing a light um and for sheer luck the nightmare story

ends here Andress unve the back door and X Z gets reverted from Linux dros but only thanks to its nature and the fact that open source meets open access to any Cod base basically the thund FR was capable to discover by accident this Vector just a few days before it was widely deployed with extremely dangerous consequences but the incredible part is that he found it just because

his machine was suffering a very very little slowness but the other side of the coin is that open source can be unsustainable for maintainers who sometimes are required to work under very stressful conditions even they are working for unpaid OB projects and maybe those projects are critical like XZ this is exactly what happened here where the maintainers was suffering from burnout you can see this message posted

by the maintainer ltic call into the mating list at the time in 2021 when decided to give gatan a Bigg role in the project because it looked like just someone to trust he was helping the project he was pushing patches in this case is a big security wake up call for everyone involved in open source both as consumers and contributors and maintainers since 2019 attacks like this

have increased by more than 700 % per year this is a graph from the sonatype report and the first reason is a great Global demand for software okay I skipped that slide and uh well we know that there are a lot of global threats and they are increasing a lot and all these reasons together LED government to work on new cyber security laws US Government promulgated the

national cyber security strategy almost one year ago in March 2023 which is a long-term strategy to improve the security of organization and explicitly talking for the first time in the history of computer science about making software vendors liable for insecure software this is a giant step forward for the cyber security liabilities the U commission also worked on a law that is called cyber resiliency act the CRA

you will hear a lot in the next years about this and the purpose of so this law is basically to regulate all the products that incorporates digital Elements which includes any product that can run software it is the equivalent of the C Mark that we see in electronic devices since years but specifically designed for software it will have a huge impact on the software industry if you

are interested in this subject and you want to know more you can just follow the link you see on the slides or scan the QR codee and you will find all the Linux Foundation Europe and other foundations initiatives about the CRA but the good news is that even the response from the community has been strong and well coordinated and one of the responses has been the creation

of the open ssf which is a not for-profit and cross industry organization formed in 2020 and part to the leux foundation the goal is to set basic security standards for open all projects and the way to achieve that is through a combination of tooling education and extensive Automation and one of the biggest issue you know with security especially in open source is where maintainers with their limited

personal time have to choose between between new features back fixes a enement we just see the X Z case this is why Automation and low integration friction are crucial for a sustainable and widespread adoption of security we see some of the mentioned projects in this slide and how they work in a moment well now that we have a general overview of the software supply chain and the

main threats and how D dangerous they can be let's see what we can do to mitigate them or even anticipate or prevent them starting with improving the integrity and Trust in our dependencies so the first question is what is the trusting model between us and the digital artifacts that we are going to use how can be sure I be sure that what I'm running is really what

it claims to be and coming from a trusted well it is essentially the same question Ken Thompson posed in 1984 in his famous paper Reflections on trust and basically the answer is that we cannot really trust the code we don't totally write ourselves instead we should put more trust in the people who wrote it maybe it was possible when Ken Thompson wrote this paper but today considering

the big amount of use dependencies it is a pretty impossible task so to enforce the integrity and Trust we need to know who build when and how the artifacts we are going to use using things like digital signatures and attestations and we also need to know the list of things who made the Arta which is basically like the list of ingredients of food using digital signatures with

key pairs we can solve the first set of problems totally because they ensure Integrity authenticity and non repudiation which are the key elements in building trust between known and actors but the problem is that managing Keys is uh very hard and boring because we have to deal with Distributing the keys we have to keep our private keys in a very secure place place and if or better

when they are compromised we have to revoke the old Keys we have to inform our Network users and start the process again and again this is why digital signatures are not yet widely used by end users even though they have existed decades but there is a new project called sixer from the open ssf which aims to fix all these issues related to managing Keys their goal is

to be what let's encrypt has been for TLS certificates one of the ways Sor allows to sign artifacts with keys it is implementing a keyless signing approach it works by using a public identity like GitHub Google or whatever open ID connect provider to generate a short lead certificate and use it to sign on your behalf so we can get rid to the of managing Keys having a

public private Keys we need just our uh public profile and another interesting feature is that signatures can also be stored in the oci Registries alongside container images removing even the distribution issues the other key element in the software supply chain are the software build of materials known as as bomb they represent the list of ingredients who made the artifacts we can use it for things like vulnerability

scanning license policy find abandoned dependencies so on so forth but generating software below materials is a very very complex task because dependencies live in different layers and they are managed by different package managers and they differ a lot in how they package stuff but there are some tools available that can help us with this very tedious task of generating as bombs the most advanced one are sift

gripe and trivy that we can use on code basis and containers to generate the full software below materials and vulnerability scanning and recently even Docker has a buil-in bomb generator which eliminates the need to install an external tool when you deal containers now we have seen the importance of digital signatures and software below materials and how to to generate them to help us to improve the trust

the integrity and transparency of our artifacts but how can we wisely and safely choose our next project dependencies while still navigating the vast oceans of open-source libraries yeah vulnerability management is still one of the major issues and this issue is caused by a big fragmentation of the ecosystem and not actual easy ways to automate the tracking and the issuing process of vulnerabilities so SV was created to

address all these challenges and the OSB is a project born in Google and now is part of the open SS ssf it consists of a new data format the osv schema which is designed to contain information like versions package names and and ecosystems with the intent to make this data more easy to consume both for machine and for humans and more open specific the other key element

is the osv database and it's free API which is a database of non vulnerabilities populated from various sources with goal to provide precise and actionable information that we can directly integrate in our development or security workflows it is very well adopted it supports the biggest ecosystem such as go Ras python Java nodejs PHP and major distributions another important aspect of picking up new dependencies is how to

understand the overall quality of projects not just in terms of code quality but even about their security practices the contributions the active maintenance if they have continuous integration in place and how is the state and in general any nonfunctional aspect that is part of the quality equation of a project doing all those checks manually and for each dependency we use it is almost impossible for a human

being and this is where op ssf scorecard project comes in help it consists of a set of automated checks that anyone can run on any project by using the CLI or by integrating it in a cicd pipeline and each X returns a score out to 10 and a risk level when they are used then they are used to calculate an aggregated score to represent the global project

score and alongside the score these tools also provides remediation prompts to fix the discovered issue and the this is quite and super handy especially for contributors and mainers you know and still talking about tools to help us better understand our dependencies open source Insight or depths. has some very unique interesting features it is a project developed and hosted by Google with the goal to provide dependency analysis

for open source projects for each project we have the complete dependency graphs sec advisories all use licenses and the op ssf scorec card data the services the service at the moment supports different ecosystem such as rust with cargo go Maven npm new get ppie and as a bonus the data are accessible for free via an API so you can easy easily integrate those data in your development

or security workflow another very interesting project is guac from Kumari they recently joined the open ssf as an incubating project and guac stand for graph for understanding artifact composition and it's a sort of database that collects data and inside of our supply chain and it works by ingesting security metadata like attestations and then the data are extended using external sources like osv and depths. tab and finally

everything is interconnected as a graph the result is a database that we can query using the guac CLI or just using the graph ql query API Expos exposed by the system and another way to see the data from guac is the graph visualizer which is a utility for visualizing and inter acting with the supply chain graph from a browser where we can filter by package type such

as golang python JavaScript or by using name space package name its version or highlighting the graph the nod graph using some attribute like as bomb vulnerabilities Etc and it is a very nice companion for graph Explorations and may be very helpful in different situations I don't know like demos trainings and quick and dirty data analysis what if we could have something that automatically manages dependencies for us

maybe even opening reviewing merging PO requests with security updates minor upgrades is It Something Magic no because both renovat and dependa Bol can do that and they can do much more renovate is an open source project that we can use basically everywhere while dependabot is a GitHub native project but they share the end a similar set of features and the same goal of automating the entire life

cycle of our project dependencies I suggest you to try them as they are very straightforward to integrate it's just a click dependabot basically and they can cover from simple to more advanced scenarios so just some takeaways as we saw a lot of stuff a lot of name a lot of Foundations a lot of project software supply chain risks are a real threat and they will increase in

the future a lot especially now that the global demand of software is uh growing a lot and open source is the standard factor for developing we can improve the integrity and the knowledge of our dependencies using more Cloud native tools supported by open source foundations as op ssf using tools like seor sift gri to sign and generate our bomb and check for vulnerabilities and thanks to project

like osv scorecard and depths. Dev we can choose our next project dependency dependencies way more safely and informed than than before and finally we can automate Project dependencies Life Cycle using tools like renovate or dependabot so we can get rid of the most boring tasks like appr prograding dependencies and also to be quicker to integrate security fixes as they are or critical fixes as they are publicly

released without doing nothing basically and there's a bonus I have recorded the demo that you can find scanning the QR code or following the link where I will show you all the tools in action how to use sixer to sign a container how to generate the software Billow materials and scan for vulnerabilities and the strategies we have to reduce the attack surface and the number of dependencies

and vulnerabilities when dealing with containers by changing the base image from the standard ones I don't know like thean Alpine from smaller ones like this right on Shard chuard images so I am done thank you very much for your [Applause] attention so are there any questions yes in your opinion what could what could be the solution to supply chain attack I think all the things we have

seen so far they are uh things that we can do to improve our security Supply Chain management uh there is there is a just one solution we have to deal with many many uh threats starting from dependencies to all the other things seen can free open source Solutions substitute services like sneak or Sona type Solutions or do paid Services perform better in any way but I don't

think that they uh are um one can substitute the other depending on your use case and how budget you have to build your own solution using open source software because they are quite easy to use but you know when you have to integrate your um open source new open source software in your workflows it costs a lot and maybe depends on the use case they are very

good Services the commercial wantes truly helpful presentation can you share it yeah I will share it you can also find me on my personal blog but I will share on the Dropbox folder that organizers have shared is private artifactory something like jrog worth the effort I think so because um as far as I know they uh already have something in place to do vulnerability scanning generating software

and Below materials so yeah they can work the effort which ecosystem has the best protected dependencies Linux or Mac OS npm or cargo Dean or Windows we is the leader in Supply Shake chain attack mitigation okay which ecosystem best protected dependen is I don't know I think that any open source ecosystem has the same threats uh maybe not is the worst one just because they have um

as is designed um no and npm they have very small libraries so when you install a package like react basically you get like five 600 dependencies but it's just a number not because the code is worse and Linux or meos uh the best I think that is Linux because it's open source um npm or cargo as I said maybe cargo it's better just because uh the dependencies

are created in the cargo ecosystem instead of node which which is uh famous for having a lot a lot of dependencies and Dean or Windows deban because is open source and we know everything behind the supply chain of Debian we don't know basically nothing behind the supply chain of Windows neither me OS and we is the leader in supply chain attack litigation I don't know you are

the next leader I I don't know I don't think that there is one today are there any other questions here room so thank you very much