What makes an open source project sustainable? Finding meaningful indicators

This talk focuses on the sustainability of open source projects and the complexities involved in managing their supply chains. The speakers discuss the risks associated with dependencies, such as security vulnerabilities and maintenance issues, which can deeply impact dependent projects. They emphasize the importance of understanding the ecosystem of open source contributions and have proposed methods for collecting and analyzing data from repositories to classify their risk levels. By utilizing clustering techniques on historical data, they identify various archetypes of projects, ranging from flagship projects with strong community support to resource-constrained projects facing significant risks. The speakers also highlight the potential for using this classification to guide support distribution and funding within the open source community while encouraging collaboration for further research and development of effective risk management tools.

So we are Max and Swen representing Bosch in this case and also the University of Lapseek and I think the bold question that we want to go into the next couple of minutes is what makes an open source project sustainable. I mean the question what sustainability means is a bit beyond the scope but we can all agree that when we build projects we often end up in

a situation where we have a supply chain which is not completely in our control. So more the technical side would be even though I have a project I might have a sub sub sub subd dependency project alpha which has an issue might be a security problem might be license change might be that the maintainer stopping the maintenance for whatever reasons maybe because they become underfunded and obviously

this propagates into your project and I think this is quite recent because we've seen quite a few supply chain attacks in the past which actually attack on this vector the more meme side of things is here I think everyone has seen this already if you're a in the industry is this famous XQCD. So the whole digital infrastructure looks like this. But we really looking forward for what

is this particular stone which everything else is built on. So where are the random dudes in Nebraska and where are these random projects that are really in important for the infrastructure? Um this problem has become quite relevant also during the time of our research as you can see by all these different news articles. So some people already said that March 26 and just just last month was

a very bad month for supply chain attacks. So we had a lot of attacks basically somehow originating or attacking the maintainer and then this propagated to the whole supply chain. For instance, one example just as we are Brussels was the European Union. So Europe.eu EU los 350 GB of data because one of their security scanners actually got compromised which then had access to a lot of their

secrets and infrastructure. So you see this problem is not just bound to a particular project or one open source project it's really propagates. Another problem is as we have seen with lock forj which was a couple of years ago that fixing the problem was actually quite easy but upgrading the software that relied on this dependency was quite hard. So many organizations had big problems or were struggling

with upgrading that and was actually incurring a lot more cost compared to the actual fix in that case. So what we then argue for is um if you see the open source ecosystem as a ecosystem as it is and maybe a forest which is also an ecosystem. The idea is okay, how do we find the project or the trees or the members of the community that might

be in struggle that might have might be in need of support to really have a more sustainable ecosystem here and also proactively find those in your supply chain before it becomes too late or it is already shipped in your own projects. Um, and if you look into the communities, obviously we're not the first ones to have this problem or deal with this um idea. So there are

way other communities. So the most famous I would say it's sorry it's Eclipse event. It's a Linux foundation activity. It's the chaos community which has a lot of um properties on evaluating the community health or the community risk for a project. There's also the open SSF project more looking into security. It's more like a security angle for the the supply chain which also has a scorecard project

which you can basically run an automated scanner through your project and see like how it is judged. Um, one problem that we've seen with those kind of ideas is it's more an opinionated or well educated let's say guess by most of the community and I'm not saying this in a bad way because uh it's more okay we say yeah a project is better if it has more

maintainers which might be in general maybe true maybe not um but we we could definitely make examples where more maintainers was even worse let's take xz for for instance where there was a maintainer coming in and actually worked against the project. So what we had as an ambition was to really look into the actual data from the past to yeah kind of mine data from projects or

metadata that we had to see what are the actual leading indicators for problems that occurred later in those projects. So really go the long way with the data and the vision somehow was okay let's uh start with bbombs. So when you take your own project you obviously have the chance to create an sbomb. How this looks like is completely different story but let's assume you have one.

Um then you try to collect a lot of data also historic data about the project and uh basically do magic. This is kind of what the research is doing now. Okay, fill this magic like pre-process the data, come up with a model and then out of this model have some kind of report which explains you yeah the criticality or the confidence that you have in this particular

project and this is exactly what Max has uh been doing in the last couple of months and he will tell you what and how he did it. All right. So before um we do some actual data crunching um we need to think about which projects we actually want to look at and other terms we need to think about how our population will look like and um common

way of doing this in research is to use some form of uh popularity based metrics like um monthly or weekly downloads and stuff like that or GitHub stars which is the one uh we end up using and the reason for that is um that we think that um these projects tend to represent the projects which are in real world as bombs and um for that we looked

at 15,000 uh projects the 15,000 most start projects um on GitHub and now the question how do we get the actual projects for this population and GitHub or the GitHub search is not that that open about um for example 15,000 projects. Um so at this time um I stumbled upon a blog post by the ClickHose database project and what this this guys did is they um provided

a public instance of the database and of the GitHub archive data set and it's actually the um largest data set of um GitHub interactions data um and we can use that to query the data set and get our projects Um but turns out when we look at these projects huge amount hu huge amount amount of projects are non-software repositories. So for example um system design primers or

coding interview questions uh stuff like that and books and uh projects written written in niche languages. Um we are especially interested in real software project not software related projects right. So we have to filter out uh these ones. We did some um systematic literature study to um get signals um which uh open-source project health and came up with a list of 39 indicators um which basically can

grouped into the following dimensions. um repository metadata like licensing and the language the project is written in. Of course, develop development um activity data um organizational data like um GitHub's community health score um which basically represents um the presence of different files like contributing files uh and stuff like this. Um, of course, software is developed by individuals. So, we are interested in community and engagement of this

community. Risk is also um a factor like DDS scores, development, distribution score similar to truck or bus factor. Um, we interested in open CVEes. Um, and of course popularity and adoption. Now that we have some form of population, we can look into how do we actually get the data for these repositories. And besides obviously GitHub which provides us with activity um related data, commits, issues, releases and

stuff like that, we have different I would say platforms like ecosystems and open source insights which we ended up using. And these platforms kind of act like aggregators. Um not only aggregators but mostly um as aggregators for these package uh registries like great io, maven central, npm uh etc. And um using that we can get more package metadata and community signals and especially um for open source

insights we get a dependency graph um list of dependent projects um licensing data etc. Um for security related data there's a project called OSV. It's a database which also acts like an aggregator for multiple vulnerability data sets um like the one GitHub uh is publishing or rust pipi um etc. Now can now we can get into collecting our data right so we took a small subset of

our 15,000 more stars repos more start repositories um we took 1,700 repositories and wrote a collection pipeline in Golang um to collect all the information I was talking about earlier so all the static and uh time series based data for a 24 month period Um we so the period starts in March 2024 and ends in March this year. Of course uh doesn't this doesn't come without its

challengence challenges uh worth talking about and the first one is the construction of pearls. So pearls are these tiny little package URLs representing a package, its ecos ecosystem, it's located in and versioning number um and some other things. But the cool thing about pearls is that once we know um that a repository is building a package and have a corresponding pearl, we can use this as an

identifier for all sorts of platforms like the ones I uh showed earlier. Um the problem here is that of course we're looking at repositories not packages and there's of course no standardized mapping between these two and therefore um the pearl derivation become quite cumbersome. But the way we um overame these this challenge is that we looked at the code. So um we scanned their um manifest files

like the one uh uh package JSON file like the one mpm uses or um for composer and the cargo toml etc. And we look for names versions um the ecosystem of course and therefore can infer the uh pearl and check ecosystems against this pearl. Another challenge is of course licensing information. Um turns out huge chunk of these um repositories we are looking at are not representing uh

their license licenses in a standardized way. Quite common problem. Um but there are some u platforms or projects which are better suited for getting licensing information than GitHub. So the problem is if you uh for example this uh C project if you present your um licenses in a non-standard way or have directories licensing um under different license than others um the GitHub crawler have a hard time

on inferring the right license. Um, a third challenge, um, is that the GitHub stars might be a bad proxy for popularity. Um, and there's actually ongoing research on that. Um, and what these researchers basically say is that GitHub stars um, might be at risk at being artificially inflated. So projects mainly because of malic malicious reasons um use these stars to place their projects um in databases or

searches and um to get basically more attention. It's famous famously are some uh crypto projects but um the the research project actually um provided a data set which we checked against and there's no overlap between our data and the malicious projects they found. Um, another thing is that as you can see in the image on the right, there's actually a big spike in 2024, late 2024. And

most of the projects we have looked into are older than that. All right. So, we have our population and we have the platforms holding all the information we need. Um, we built a collector and now we want to walk more towards the visions when showed in the beginning. And we somehow want to classify these components. But speaking of classification so normally for classification tasks you need some

form of target variable or um label something like that. Um the problem here is that we actually don't know how to construct these labels or we don't even know how these labels can look like or which is a suitable label for representing these risks. Um so we use clustering to identify um groups of similar open source projects and discovered categories based on these characteristics they show. And

then we looked at these clusters um analyzed them and um analyzed what properties define these these clusters. Of course before we can do some uh data analytics we have to do some pre-processing. Um I don't want to go into much detail here but as Sven said in the beginning we're mostly interested in um the evolution of projects over time. So time series based data is of special

interest for us. Um therefore we calculated trends um of our longitudinal data. Um we created month specific features for features like uh commits over time and um releases and how many uh people interacted for inside issues via comments and stuff like that. I did some uh um encoding of our variables um scaled the data and transformed it and um employed or removed highly correlated using that we

can build models right um and as I said we want to use clustering for that um and there's a common pipeline in research which is composed of um dimensional reduction ction via UMAP and clustering with HTTP scan. Um is basically a dimensional reduction algorithm um which you can use to project um highdimensional data which we end up having uh into lower dimensional embeddings and this is practical

for clustering. Um, speaking of clustering, we use HTTP scan to cluster the projects HTTP scan is a neat uh clustering algorithm because you don't have to specify specify an amount of clusters you want to get and that's good for us because we don't know how many target variables uh there are or how these labels look like, right? Um another thing is it builds a noise cluster and

real world data often is very noisy right so um as it is for our open source projects and uh it turned out to be pretty useful what we ended up is having five we call them arch types um for the projects we we tried to generate some small images to make them more uh feasible. So the first cluster um which emerge are we call them flagship projects.

These are the the ones having the lowest risk. Um they have relatively large communities, active communities. Um they have multiple contributors or maintainers. Some of them have corporate involvement and they tend to have very mature project structures. The second one uh we call them resource constraint peripheral projects. These ones are much smaller. Um they have um higher risk because they are maintained by only a few maintainers

or just one. um they have no mature project structures and rather small communities. The third one critical infrastructure with concentrated ownership. Um as the name suggests these are projects with tend to have a um large amount of dependent projects. also they tend to have medium-sized um communities or smallsized communities and face high risk because they only have one maintainer and that's a huge problem especially if you

um think about the number of dependents they have. So they um sort of sit like um yeah critical The fourth one are declining midscale um these ones they are moderate in every sense. So they have medium-sized communities. they um tend to have a medium response times to issues and uh communication. They yeah medium-sized governance and resources and so they are moderate in every dimension. The fifth and

last one are the capacity constraint mature projects and um they are low in risk because they have some maintainers left and have quite mature structures. Um the problem they are facing is that their development activity is actually declining. Um but um they're mostly projects which are are mediumsiz and in a sense that they were popular at the time um but are declining So the a question is

um how do these clusters emerge? And clustering generally is considered hard to interpret and um we use a um technique called chap. Um and we can use this one to basically train a classifier on these clusters and look how um valuable valuable uh valuable each of these um features is that we use and we try to find key differentiation features. Um and this has basically two objectives.

The first one is that it allows for cluster interpretation and the second one is um we end up using couple of variables which um are strong in predicting these uh clusters and we we can use that to build classifications classification models using these variables. Um speaking of primary predictive variables um interesting is that the primary drivers for the clusters are licensing and community related files. Those ones

had the highest impact. So um if the project has contributing uh guidelines um if this if the license is known for a project um if it has external documentation um secondary drivers are um contributor counts and the trend based metrics I was talking about earlier. Um the lowest impact for these um clusters are the um commit accounts and active maintainers and pull request response times and some

specific templates like u PR uh pull request templates and code of contact conducts. All right. So what do we have at hand? Of course we have asbombs, plenty of sbombs and we talked about the data collection process um using OSB, GitHub, ecosystems and all these platforms. Um we briefly talked about pre-processing. So how can we calculate trends um on these time series based data and how we

can prepare our features. We built basically two models. So the first one being the clusters which we identified and the second one um we use trap to um determine primary variables um which have high impact on these clusters and we can use that to build classification models. Right? We can basically use any arbitrary um classification algorithm um for classifying that as long as it has enough capacity.

Um and we used XG boost for that. It's quite common and considered state-of-the-art for classification. Um and by using that we can uh as Ven said generate reports. So we shove in sbombs and at the other side we get um of all these components um classified um by risk. So we tested this in action um on a project when involved. >> Sure. So maybe coming back to

that slide. Um as you've seen we came up with a model now. So we have the tooling that we envisioned after a couple months of working in data crunching and the question is now where do we go from here? Like we have the tool but let's use it. So one example is um we have the cooker project. It's a Eclipse cooker to be more precise. is a

yeah growing project in the Eclipse Foundation from the SCV ecosystem and it's mostly about doing vehicle abstraction. So um decoupling applications from um the vehicles. So for this talk it's not really important what it does but it's a rust project. It has couple of lines of code and we think the most important feature is it has 546 uh packages it depends on. So the ESPOM is large

but maybe not too large and we just put this ESPO through the tool and I think the findings were quite interesting because we had um seven features or project that were critical infrastructure and we had another 24 um which were of these resource constraint peripheral projects. What this means for us is out of the 546 packages we have 31 where we might or should take a look

at um because here we consider being in one of these two clusters as being a high risk while for instance for the flexure projects we don't see too much risk. So at least just looking by the data there it's more fine. So now we have the data but what do we do with that? It's always the second question uh or maybe should be the first. Um one

thing more from a from your personal opinion or standpoint is you select projects maybe based on these findings. Let's say you have three projects you can select from and then you can run some number crunching and some of the model and see okay maybe this project has way more um yeah credibility or less security issues in the past. So let's go for that one. Um, another thing

is risk monitoring over time. Let's say you have the project or might even some of these projects also in the field so that you can constantly look into the data from your ASOM when you ship the the project and re-evaluate whether there have been problems or whether you might expect problems before they occur because again if you do this more proactively you have a way better way

to react. Speaking of reaction, um another thing and that's also I'm really curious to hear your input thoughts from the uh in the room is it also can help to distribute support more effectively because coming back to this analogy with the open source ecosystem being a forest or a whole garden. It's a good way to actually at least identify where you might have problems in the future

and then um distributes the resources. But most importantly, and that's also part of the reason we did this talk is we're really interested in your use cases. So having this number and also these clusters at hand, I would be really curious to hear like what would you do with that or is there a good tool to build put this in um you've already thought about a couple

of tools which deal with bombs and also compliance where this could be added. Um so I'm happy for these conversations about exactly these kind of questions like what do you think about the approach? Is it pure nonsense? Is it too opinionated? Does it make any sense? Would you like to collaborate on? That's also something that we currently consider. How do we continue? Because um it has been

more on a academic research site so far, but I think it's also open for collaboration or maybe you have done similar things. We already talked to some universities which have a similar database in place. So I think there's room to grow this together. Um then a very important question and I think that's also more something we can have on a personal conversation is if you follow the

talk closely you might wonder we haven't said any project by by name yet and uh we did this on a by choice because we wanted to avoid such like a data blame game like someone saying hey they said your project is bad because of that and that and I think it's still quite opinionated in a sense that in your particular situation These numbers might add up to

the situation that you end up in one of these clusters. But it's just a let's say very generalized overview not that to consider every individual case. That's why we are very cautious with uh naming particular projects. Um and I'm wondering whether I'm a bit too cautious here but um yeah that's the thing. Um another question Max already pointed it out. We basically use the clustering to find

out what is a good indicator for a good project because good and bad already is quite opinionated. Is it the number of CVEs? I would say not because if your project is not well maintained, you won't have any CVES but you're a perfect project. Not sure about that. So um that's also where we come back to these communities like chaos for instance. And last but not uh

least and again this could be also a good way of finding projects in need of support. So what does having such cluster or such research mean for um setting up OSS funding mechanisms like the servant tech agency that was set up in Germany. And maybe funny side story to that is we talked to people that did similar research and this was exactly the reason why they got

funded because they actually the funding agencies wanted to know whether the other funding for other open source project was actually successful. So they needed someone to evaluate that and this resulted in a research about metrics or health indicators for open source Um this runs through the data. Um we are very happy that you're still at least physically in the room and are here. Um and looking forward

to your questions, discussions and um input. Um there was one hand raised before. >> Yeah, exactly. But you all they don't want to start playing game. was asking actually for good examples like for examp >> okay sorry then I will just reiterate my question uh so more or less you already answered it um I was um thinking about whether there are good examples for the archetypes um

but you stated that you don't want to start the blame game I was just curious about um yeah examples like where to put kuberneters I guess that's very obvious maybe also Yeah. Yeah. Um that was one of the points. Uh and after you've put your example, maybe one question. Um so in the sbomb case, how deep do you go to assess that just um single level sbomb

or really deep until there's no more dependency because sometimes it will get cyclic or >> Right. So in the case of um the Eclipse cooks data broker, we just um got the ESBOM from GitHub and I think it um they're around 5 to 600 dependencies there. So it should be transitive dependencies. >> So the full dependency tree, not just the first layer. But I mean for the

tool itself or the what what he's we've been building is I would say it's quite agnostic because it basically takes the project and then evaluates each projects on itself not taking the context whether it's like the third fourth or first layer in your tree. So >> and what we also don't evaluate so much is like how the project is actually used. It's more like a binary. Okay

the project is in your sbomb. what you actually do with that it's I think part of another observation or analysis >> I mean there's tooling for that uh in front of this whole uh tool chain for regarding response so >> maybe one further question regarding that um so you more or less have like um the dependency direction to one way right so do you also evaluate uh

in your model or clustering the other direction so for example I have a project how often is this is this project used because That would also be interesting in case I'm a maintainer and I would like for example thinking of making it a business model for example and then I can evaluate if there would be a direction for me. So if I for example am the maintain

of co and I could evaluate how many persons are using co also via this sbomb metric >> or maybe very specific also and complex and getting that >> I mean for the clustering it's here it's I would say not the main goal to figure that out. Um there is in some case data available. So for instance it GitHub if they have CS bomb for a project they

also provide you means to let you know who's using your project in their spawn which is already a good indicator but obviously this doesn't tell you anything about um closed source adoption of the project so it's really hard to get to that data and also way beyond what we aim for. We I think yeah we included this public data which GitHub is providing there but also more

to see I mean one of the reasons we put that in is or more as an indicator how to blunt to say bluntly how many other projects depend on that. So if you depend on a project and the only one using that chances that someone will fix it if there's a problem is quite low compared to a project like cur where you have millions of dependence or

the lock forj case. All right, thanks. I will hand over. >> Yes, we have other two question. Uh, but he raised the hand for you. Sorry. >> Hello. Thank you for the talk. I'm an export employee. So therefore, I want to be a little critical with my former >> Here um we also work together in the same building. So therefore when I was there uh I also

let's say on our project profited massively from open source and you also said okay how you can uh let's say detect risks and so on and it's a little bit weird to see from a company which makes two billion in profit last year to say we need public funding despite their business case really depends on open source. I know it's not your fault in the end. It's

like something where I think we as developers has to stick together and say no we go not for public funding you pay it if you don't pay them for instance and sustainable software and especially sustainable open source software means that the maintainer can make a living can continue maintaining the software in the end and let's say OSS funding is set up like okay Bosch receives some funding

so therefore some employees let's say will dedicate work on work on a specific open source check but a maintainer doesn't see a dime from this but therefore this is a perspective I would like to see changed for instance >> and it's like like a call from my side that we all work together and say talk to the manager and say no this is just absurd making 2

million billion is profit and say okay now right now the public scene has to fund this project because our life depends on it or let's say our product depends on it but we can't pay for it >> I see where this argument is coming from thanks um I maybe question I guess you refer also to this last question here. I don't want to correct there but there's

a particular reason I didn't put public in here just funding mechanisms because I'm also thinking that this funding that's my personal view it's not like for forgets about the logo in the corner now um is that we need a funding mechanism which goes beyond public funding but also private funding and um I think one step towards that goal is to first of all have the transparenc arency

that there is a problem in the supply chain. If you just say it's imagine you go to a manager who has like to lead a whole company with a lot of hardware business so not too much software knowledge. It's another topic. Um and tell them go fund open source. This is so broad and it will be also really hard to distribute these kinds of funds. Obviously, it

feels uh I know it feels a bit weird to start with the distribution problem. If there isn't any fund at all first of all, but I think you kind of have to do it think it together because if you really know where these funds are getting to, it's also easier to get to those. And so that's thanks for for the input and that's that's also part of

why I put this question in here because we kind of currently look into how such funing mechanisms could work or would not work because um we heard a lot of voices but bit also what you just uh said between the lines is um doesn't really help to have let's say a single grant where you just say okay here's I don't know a couple of euros and then

we have to wait uh or find a new one because then it's more like um like someone compared it more to I don't want to use a word but I don't have better words to do drug dealer where you say oh here's some money and now the maintainers get into a mode where they always follow these next fund next fund next fund independent of whether it's a

public or private fund more into a way also that the SDA did where it's a bit more regulatory coming in also having the groundwork uh finance like maintaining the infrastructure who's actually maintaining the DCI servers things like that. So I think starting with the transparency is the first first step here. >> There is the other question before. >> Yes. Um hi thank you. It was very interesting.

I have a question about the archetypes that you defined earlier. Um the archetypes mapped to the clusters that you identify with the correct data that you use to train your model. Okay. So if you use you add more data you will you could be able to find more archetypes, right? or define them. >> Um, it's a good question. Um, >> it depends, I think. So, I mean,

you can it's it's complicated because um you can tune your clusters to basically fit in 200,000 projects into these clusters, make it more broad. Um but I don't think there need to be more clusters representing these uh this data set. So um maybe you can um make it. So it depends on your use case. For example, if you need a really fine granular um model which um

end up using 10 clusters, you can do that. You can tune your um clustering to basically uh sense that. >> Yeah. Yeah, on the contrary I was um I was actually interested in this talk because a spoiler I have a talk on Thursday that actually introduces uh score metrics that we want to use at the foundation to define project health. >> And this is actually very it

overlaps a little bit. So instead of fine-tuning it, I would like to have like a more general overview and see if I can redefine or reuse this to redefine those archetypes based not on packages but on projects themselves. Um so yes maybe we should discuss this after the talk but I I was just curious how >> uh flexible and how adaptable this model you use is to

define these project archetypes. >> So f uh so one last thing um basically make this really elastic. So as I said you can get a arbitrary number of clusters but it depends what you're looking for and the balance is um that if you have a large amount of clusters they get tend to be really hard to interpret. So there's a sweet spot there. um you can uh

go with three clusters but they're hard to interpret because um projects sitting in these clusters are very diverse and yeah it's about interpretation basically so in this case yeah >> thank you >> okay we have one last question okay that >> yeah um about the like dependency or the different types of dependencies I'm not sure if you mentioned it but you mentioned infrastructure on the previous question.

Um so did you consider infrastructure and dependencies of the infrastructure also as a dependency of the project? >> I think Sven answered the question uh in the beginning. Um the short answer is no. um it's just about projects and we haven't looked especially into dependencies of dependencies and the order of that. >> Yeah. So we considered each project on its own but not the dependency graph around

the project but it's a very valid point that um if for instance um I really like the presentation that I can recommend from the airflow community where they really looked into all of their dependencies and see what how they how it and sometimes work with that. So one of the FTO maintainers even took over the ownership or the maintainership of one of their dependencies afterwards had some

yeah discussions there. Um so you're right it's very valid to do that but for for now we had like each is a single box if you will or single Okay. So we close the day now and uh just an observation that if you are customized too much maybe this is also a question to the future if you are customized too much maybe you are making too much

selection and there will be a dock uh archetype then for one single project. So yes, when she was saying it's it's better to have more general so you can compare more project and not so maybe having too much details for closing this day. I'm happy to see you here. Tomorrow we will be here in the same room at the same time. As I mentioned to you, we

have mandatory breaks and we have another one that starts one minute before that is a reception that is uh now going in the main room in the main uh you know exhibition area. So we can mingling together there okay with falter question for our speakers and to networking between us. So thank you all and see you tomorrow. Thank you to the speaker.