Getting Real with the Supply Chain: From SBOM Data to Action #FOSSBack

In this talk, the speakers discuss the complexities of managing the software supply chain at Doan, particularly in relation to open source software. They outline their journey in collecting data to develop actionable insights that improve software operations, emphasizing the critical role of software bills of materials (SBOMs). The framework they adopt is based on a holistic SBOM strategy that entails generative processes and centralized inventory management. They explore the balance between risk, value, and people involved in their open source initiatives and the importance of fostering a healthy relationship with the developers and communities behind these projects. The session highlights the organization's efforts to lead with transparency, create practical governance around open source usage, and ensure compliance while remaining responsive to both security and operational needs.

Yeah. Um, we have talked a lot about the supply chain, the software supply chain over the last couple of years. And yeah, we want to talk a little bit about the practical aspects of that. How how we really get real with that at doan. So Max and me we are taking care of open source at doaban and we will go through a little bit of a journey

how we used data to gather insights to in the end create action when we are talking about doaban um we are talking about big numbers um I won't read all of them uh but maybe just one number there are 5 million people traveling every day by train um alone here at Berlin main station 350,000 people are there So this is mass massive a lot of people and

these people don't care about software that much. They don't care at all about open source and if we get it right they they want to arrive safely. They want to arrive on time. So in this area of course the question is what's what's the role of software and clearly I mean you know that we know that um it's not possible to do do um an operation on

the scale without massive IT infrastructure um and we have thousands of systems um running um software um a lot of that is open source and we will get into some more detailed numbers u a little bit later uh but for now for for me the message is I mean we have a huge operation software is a part of that but it's not our business purpose our purpose

is moving people and goods by train. Uh but of course we have to make sense of the software so that everything works. So we need to know what software we use. We need to we we need to know what components what is open source um what what is happening with that. And this is quite complicated situation because we don't have one product where we can actually analyze

that. But we have a variety of things. Um we have software we build, we ship, we have software we run as a service for customers internally. Um for competitors even um we have things we build ourselves, we have things we buy uh we buy services um and we also have things which where software runs on devices in trains in um ticket machines and so on. So it's

a quite diverse thing and and one big question is yeah and I hope Max you can answer that. How do we gather data about that? Yeah, that's a really good question, Cornelius. Yeah, this brings us to the first step of a framework that we will present to you in three steps. Fortunately, in order to gather data about our software supply chain, we have meanwhile found a common

language to describe the composition of software. And you might have guessed already from the title, this is sbombs, software bills of materials. Asbombs are often perceived more or less as a tool or as a service that you buy in, but it's rather a common methodology to tackle not only one use case, in our case the open source um license compliance, but multiple use cases from IT security

over business continuity management and so on. And this is thanks to their flexible and openly standardized nature but also thanks to the vast ecosystem and the healthy ecosystem around it with all the tools and other best practices and other uh standards. So for us it's clear they must become a shared infrastructure within our organization and a shared a common language. But how do you introduce such a

new methodology into an organization that is that big and already has been doing things before which didn't involve sbombs strategically. So what we did is to come up with a holistic asbomb strategy and an asbomb architecture more or less. And uh before that or up to this point when we started the program roughly two years ago it has been rather local activities. So doing sbombs here, another

proof of concept there. But we knew we need something that encompasses all of that. And as you can imagine, this isn't easy in such a context like with this huge organization that we have. So we defined for ourselves and a small team some procedural but also technical and architectural principles and most importantly it was for us to get stakeholders in on board from the very start on

ideally as part of our small interdisciplinary group but also as but we thought big from the start. So we knew we have to take all the different sourcing streams that we have at Deutschean into consideration and therefore all the different sbomb types that we could have. And another big win or another big moment was when or a good idea was when we started to think in capabilities

and not tools. And this is an a problem that I see many times not not only in other organizations but our also in our own that we rather think in tools but it's more about capabilities. So I think where are the gaps that we need to fill in order to have this huge picture this huge architecture that we have and this common language implemented in our existing

systems and workflows. And for us the mental model when it comes to sbombs is that we split the generation of sbombs from the storage of sbombs and the analysis of sbombs. So the generation of sbombs can happen decentrally from the very different sourcing streams the workflows that we have in our subsidiaries and the use cases that the teams have. But we have only one place where we

store all the ES bombs in our organization, the so-called ESBOM inventory and enrich it or put it in context with other information from our organization. But then the analysis of ESBOM can still happen decentrally or um federatedly depending on the different use cases that we have. But it's clear like if you have this huge picture and and and this holistic approach that this the implementation cannot happen

overnight. So what we did is to find um prioritized increments for um the implementation of a nbomb strategy. So we looked for priorities, our priorities, but also regulatory priorities and of course also lowhanging fruits that we found. And our strategy was that we focus on asbomb adoption first. So have was data on our software supply chain at first and a huge adoption rather than directly and from

the start looking into the best possible sbomb quality that we could get. And with this we can then tweak some some things in order to improve the sbomb quality uh gradually. And we made quite good experiences with that. And for us as well, it was important that we look for quick wins both for teams, so the people that have to adopt, but also for governance owners like

ourselves and the others. And our current state is that we now have sbombs for all source code repositories at Deutschean and also for a lot of build pipelines that we have throughout the organization. And during this year we focus on getting the sbombs and good baseline quality sbombs from our runtime. So containers, cloud servers and so on. And on the road map is also having asbombs and

other builds of materials for instance from operational technology and from it very close to hardware. On the developer side looking back or looking apart from the just the strategy the top level strategy we designed the typical sbomb workflow how we call it. So here the idea is rather that we give developers feedback and input already on their sbomb generation and the analysis that they can already draw

from it. So we come from a rough generation of an sbomb to an enrichment especially regarding licenses. then also are ready to an analysis for uh IT security issues or for license uh policy issues and that allows them to get direct feedback and not wait for the analysis based on the ESPOM inventory here and this is flexible and modular. So we have a lot of use cases.

We have a lot of edge cases where the standard tool chain with the recommended tools that are good enough for most things if they are not suitable things can be exchanged. So this is a huge part of our yeah we need to please our users in this. And now you might say well show don't tell. Okay I can show some rough data that we already gathered. Um,

so this here is from a snapshot I took a few uh days ago from our Sbomb inventory. So from 85,000 Sbombs that we have, mostly source and built sbombs. And these 80,000 sbombs will rather become during this year 150,000 sbombs as we connect to the runtime instances. And we expect that we get around uh 500,000 sbombs per day flowing into our systems. And I won't present all

the different numbers here. Um they just show you the the the sheer size of our software supply chain. But very importantly, and this is not not a surprise to us, we depend a lot on open source. So more than 100,000 distinct open source project that we depend on. And this again is just looking at a subset of the sbombs and the infrastructure that we have. So Cornelius,

how do we make sense of that? >> Very good question. Let's try to find an answer. So when we are looking at um the data we have how how do we get insight what what that actually means to us and how can we come to to an action. So one glimpse into the data would be we use a lot of web frameworks for example we found the

diversity there not really a surprise there. Uh we also use a lot of programming languages which start with the J. Um also not a big surprise. Um what actually did surprise me a little bit that there are for projects you using Julia. I didn't know about that. So um that's interesting. Um and we could of course dig into a lot of these details um and find out

what it is. But in the end I mean we are talking about 100,000 components. We are talking about um many many thousands of projects. So how how do we get um more yeah insight in terms of patterns and u dimensions we we look at and one very important thing is of course in behind these open source projects there are people it's it's not just a mechanical machine

there are people behind that so that's one dimension we look at um and we see there are many many peoples there are individuals there are companies there are people who are employed for um company by companies who are people who work self-employed um it's it's a lot of people and uh we recognize and we see that that we we have some responsibility there to not just consume

the software but also to see what is what is behind that and we consider ourselves as part of this as part of the community as part of the supply chain. Of course the the the overall community contribution is huge. Our contribution can only be a small part of that but that's the that's the way how we see it. Uh but what we also see is um it's

sometimes difficult to interact with these people uh because we are a big corporation. So our tools are uh not humanto human communication but we have processes we have structures and so on and then sometimes really hard to interact with the community. Um and for us it's it's um actually the the most easy way is if there are business entities traditional business entities where we can just do

business as with everybody else. And this is reflected in the community. So a strong business ecosystem behind the community is actually a big advantage for corporations like us because we have ways to interact in a way our organization understands. So that's a people dimension. Um why do we use so so much open source software? Of course because there's a lot of value in that. Um the question

is how how much value is that and and can we maybe quantify that? Turns out it's quite difficult. Um I mean we all know the supply chain looks like this. Um we also have some numbers. Um what what what the value of the supply chain is. Um but if you you all know the or most of you probably know the Harvard Business School study which actually puts

a number on the supply chain. But once you look a little bit deeper you realize okay there are a lot of assumptions and there the the the overall number is something which is quite hard for us to make sense of. So we are trying to look a little bit deeper. Um but that's also where we see there is a lot of value in the supply chain. We

know that. Um but we also see that a lot of this value is not material in the terms that you could put a clear money tag. There's independence, there's transparency, there's sovereignity, there is community, there are a lot of things which are very valuable and and we want to appreciate uh but it's hard to put a number on that. So actually a lot of the um results

of our our insight is not really quantitative on the data but more qualitative that that we try to um yeah steer open source in a qualitative way that that we look at the things we we are seeing and um then see how we can deal with that and one part of that is contributions. Um, of course there are ways how you could try to quantify the value

of contribution, but for us it's a little bit more a matter of principle. We do that, we make it easy for people to do that so that it comes from from the normal work and uh we can contribute to what is important for us um which is identified by the people who take care of that. And that brings us to the uh to the last dimension we

want to look at and that's risk. And that's maybe the most important um uh dimension for us as a corporation because that's the way how big corporations work. They manage risk. That that's that's a way where um we we do. Um with this size of the supply chain, it's quite difficult. 100,000 components um there's a lot to analyze to to see what what kind of risks are

there. Um and also many of these dependencies are not even chosen by us. They are dependencies of dependencies. Um so they come in and but we of course have to um yeah see see what what we do with that. Um and what is also becoming more and more a problem I mean these dependencies these whole supply chain is under attack. Um there are malicious actors out there

who are trying to consciously undermine the supply chain. Um and there are also these people who mean well and still create problems like changing licenses or stuff like that. So there's a lot of risk in the supply chain which is which which we have to manage. But on the other hand um managing risk is something we actually can do quite well that that's what our organization is

is built for to to some degree and uh doing security and doing license compliance. We have built over the years quite robust mechanisms how to do that. So it's kind of daily business as usual. And also what we see is not all dependencies are really equal. I mean there are many good dependencies which which are very solid but there's also a lot of smaller things which might

be not that relevant in a overall risk perspective. So what we are trying to do is actually to um yeah get the people who deal with the software to take over responsibility. We need to educate them for that. They need to be aware of how how to do that. But that's something where we can kind of scale this um handling of risk um and then in the

end take these three dimensions together. So to show that in a picture what we are trying to do is we have these dimensions of people, risk and value and we see our responsibility as the people managing open source in bringing this together and balancing that and then deducing actions from that. It's not very easy. Sometimes if you look into the details um it gets a little bit

um tricky because you have to actually look at the details and maybe one example um so one dependency which shows up quite a lot and um I would imagine some of you know that is balanced match that's that's a node module which finds uh brackets uh quite important for some use cases probably not not used in production for issuing u train tickets but it's a mildly popular

component on GitHub, a bit more than 100 stars. But then when you look at who actually uses that, it's amazing to more than 40 million repositories on on GitHub directly or indirectly depend on that. And then if you look a little bit deeper, what what is this component actually? It's 70 lines of code. So the question is who are the people behind that? What's the value of

such a component? It's used a lot, but it's small. What's the risk? I don't know. I will leave the question open for now. Um yeah, but then Max, we have gained some insight. How do we take action? Now, you're asking really good questions today. We cannot delve into all of that obviously what what we do and what can be derived. But I think the main part is

here that we need to find a balance between these three dimensions or the these triangle. So on the people side I think it's very important to understand that there are actually people behind it both internally and externally. So we need to establish a fair healthy relationship between us and our people and maintainers of projects and ecosystems. So we also have to empower and train them to do

this properly and that means also establishing an open source culture within an organization because this is how the world works today. On the value side, we need of course to understand the value both of open source which is very hard of open source packages which is also very hard but I think we we can do this qualitatively and we can identify thanks to data what we actually

depend on and how valuable or critical certain projects are. And so for the parts where we uh depend on which are really critical for our organization, we want to contribute and we need to contribute in all the different aspects that we have like from uh financial but also with human power in those projects and and uh indirectly through associations organizations on the risk side. I mean there's

a lot to do but here the balancing is really hard. We tend to overregulate sometimes. So rather we have to look for a lean governance for our organization to balance these things and I think an important thing is that we uh need to enable our teams to make the right choice. So to make the right action the easiest option for our teams and this can be supported

by tools of course but here we really need to think about people. Tools don't integrate themselves. It's actually people. So the users that are bound to use these tools and processes, they vote with their feet. So what we need to do in order to help them make the right actions is to take them with us and to make them happy actually and then they will actually adopt

processes and tools and whatever you thought of and also provide very helpful feedback for you. And the most in effective leverages that we found is to make adoption of these processes and tools very low threshold for them to please them with a lot of automation and also human support because things don't work from the start and there will always be problems. So we made quite good experiences

with offering open office hours for all the users of these compliant tools uh regarding sbombs for instance. And another important factor is to really think about the severity of issues, the effective risk that things might have on your organization. For us in the licensing field, one part of that is that we um bound license findings or license issues like licenses that might that are unclear or whatever

um that we bound this to a concrete use case or usage scenario how we call it internally. So and this defines the criticality or may define the criticality of such a finding and also make this transparent to users so they know what are the really most important issues that I need to work on. And let me quickly show you how this looks like in practice in uh

one of the tools that we have internally um the compliance portal or internally it's called the compliance suite. This is an internal web service that helps teams to um understanding their assets that they have and also the the attached issues and findings to this. It directly accesses the ESBO inventory. So those um more than 80,000 sbombs and also derives some insights by yeah using certain scanners and

analysis tools. And this here is displayed then nicely also aggregated to GitLab groups for instance or to enterprise IT applications. Technically it's um a custom backstage plug in but again tools don't matter here. It's really people. So here you can see there are different uh like also with the colors like um really trying to categorize the severity of issues not only regarding licensing but regarding security regarding

secrets and other compliance issues that we have to bring that all together but again people are more important than tools and we must not only talk about teams or developers but actually also about ourselves and I would call ourselves like yeah governance owners. open source compliance, CESOS, business continuity management and other compliance people. We need to work together actually. We need to make governance in all these

areas in all our yeah working spaces consistent with each other integrated with each other and that's challenging of course. Yeah, as you might know from your own organization, but it's possible if we have a common understanding of our software supply chain, if we have a common language how to describe software composition and if we share a view on yeah this a balanced view on this triangle of

people, risk and value. So, Cornelius, it's time for a conclusion. What do you think? What what do we want you to take away from this? So we showed these dimensions, people, risk, value. These are important dimensions and balancing them is one one of the key responsibilities of what what we are doing. Also, we showed a lot of data or some insight into data. Um that's also something

which is quite helpful, but it's not enough. I mean, you need to deduce actions from that. you can easily get lost in data. So um try to focus on making sense of the data and deducing action from that. And what we also saw is that actually regulation um can help with this this effort. So the CRA for example I mean when when sbombs become more mandatory that

helps us because we have more means to um get this uh in established in the So when we have this um what is important for us is doing this in a pragmatic way. What we have seen is you can easily get lost in details about license compliance or security or whatever and you have to somehow take a pragmatic approach to make sense of this massive um amount

of software we are using and we do want to do that in a productive way in a risk oriented way so that we do that what actually matters. And another very important thing is what what we realized of course internally that bringing people together from different areas that's one of the main uh tasks we have as people managing open source. So internally the different governance owner especially

for example license compliance and security that's always something which could could be closer together because we are doing a lot of similar things but not the same things. So we need to work together and the last thing and that's why we are here and why you are here working together in the community. I think we have very very similar challenges across all the the industries across different

industries and we are using the same open source modules. These 100,000 modules would show up in our sbombs way they most of them will show up in your sbombs as well. So one thing which we can do to make this efficient and to make this in a way that we can take responsib responsibility is to work together on that. So that's it what we wanted to tell

you about the supply chain how we handle it. Um if you have more questions um talk to us any time. Uh there also is a follow-up session a panel um where we will discuss this with some other people at 12:30 which is here. So, you're invited to join that as well. Uh, but for now, we will stop talking and let you ask questions if you have any.

>> Yeah. Thank you again for this interesting and inspiring talk. Uh, Cornelia Schuma and Max, we still have three minutes for questions. >> Thank you for the talk. Nice to see you again. Uh there was one slide one on managing risk where you mentioned something well you didn't mention but I think it wasn't a slide about an automated checks and a red flag or something. Uh do

you can you say a little bit more like what kind of KPIs you you manage or or you monitor automatically and uh what kind of rules trigger some kind of a red flag for dependencies for risk management. >> That one red flag checker. Yeah. Yeah. in terms of KPIs, it's actually not not easy because um, it's hard to get really reliable quantifiable data. So, our red flag

checker is more like like an indication we we use um to to analyze open source projects. We have a list of criteria um for for teams selecting open source software. Um, we have actually published that as well on our GitHub. So uh the approach how how how you as a team would qualify a project and and what kind of dimensions you you look into um that's uh

yeah we have shared most of that publicly um and from that it's uh more like people would write an architecture decision record to to define that and uh our view on that at the moment is more more like an observing view looking into the data um and we are not driven by KPIs to um steer that at the moment. Uh we have one more question from the

online people. Uh I love your approach here. They say >> good to hear. >> I'm curious if you've seen practical changes in development since starting this process. A change in deployment rate, security incidents, or other things that the bosses care about? >> Do we want to answer that? >> Oh, do we have an answer to that? I mean the question if the bosses care is always of

course a challenge uh because there are a lot of bosses in in a large organizations like like ours. So what we have seen one thing and and Max show that what which I I found um not not really surprising but but confirming that developers react well to um an objective view on what they are doing and helping them with practical measures like like the compliance suite we

we showed where uh the people in the organization um get things to do as a checklist and they know they are done when when when they have processed the checklist. So this this automation in in a way which is compatible with how how developers think and work. Uh that's something which very which is very helpful and for me that's actually the even more important thing than what

the bosses think because if our supply chain is secure or compliant that's something the people who work with the software every day have have influence on and they decide on that not the bosses. >> But I would like to add one thing it's really hard if you get more data because then you see the nasty things in your >> That's true. So to wrap that in a

nice paper and actually actually uh explain why you see this data and the alo threats that's also then the communication challenge for governance owners like us and security people. >> All right thanks you uh thank you again for the talk. Uh I'm pretty sure there's more questions for you in the coffee break but now we need to stop here and we have a next talk in five

minutes. >> Thank you. Thank you.