The OSS-ential understanding of open source community health and metrics

This talk focuses on understanding open source community health and the metrics that accurately reflect the state of a project. The speaker, Idem Owoh, emphasizes that commonly used surface-level metrics, such as GitHub stars and commit counts, do not always convey the true health of open source projects. Instead, he advocates for a deeper examination of community dynamics, documentation quality, and contributor engagement. Owoh discusses challenges faced by maintainers, such as burnout and lack of compensation, and highlights the importance of sustainability and survivability in project management. He introduces the CHAOSS project, which develops metrics and guides for assessing community health, and stresses the need for projects to foster diverse contributions beyond code. Overall, the presentation calls for a holistic approach to measuring project vitality and maintaining active, healthy open source communities.

Bonjour. Thank you for coming to my session. There was a little misunderstanding with the time. I thought it was 45 minutes originally. Well, it's just 15 minutes, so I'll try to see how much I can cover. So, today we'll be talking about the essential understanding of open source community health and metrics. Moving beyond surface signals to real project health. So, a lot of times we we judge

the health of project based on very surface level metrics we can find out there. Um, a lot of times we go with the stars um, to judge popularity of an open source project. Um, they have lots of stars on GitHub. Um, you want to look at their commits, you want to look at issues to judge whether a project is active or not, but most times these metrics

do not really tell the full story. So, my name is Idem Owoh, and I will be talking about community health metrics that we need to, you know, look at or in to judge the health of a project. So, I'm a community and event coordinator at MLH community. And I'm a DNI badge reviewer at CHAOSS. I have been um, badging for events and projects for the past three

to four years at CHAOSS. And basically what we do at CHAOSS is we develop metrics to judge open source community health and yes, projects as well. So, we'll get to that part. And I'm a kid romance drama lover as well. So, how many of us here are maintainers? Do you maintain any project? Okay, awesome. Okay, um this talk is mostly for maintainers, actually. So, um I pulled

this report from the Linux Foundation TideLift 2021 State of Open Source Maintainer Report. It says that, "Okay, while 80% plus of software in the technology in the tech space is majorly open source, but most times some of these open source projects have maintainers that are unpaid or that are burnt out. They're considering to quit, you know." We see lots of issues with maintainers getting tired of managing

projects because they don't get enough compensation that they need to keep sustaining this project. Most of them are just one or two people managing a particular project, which could be a lot. So, we we we get to see this, um reports all the time. And that's basically what this talk is all about, right? now we want to look at activity is not equals to health. So, most

times what we see as an activity on a project does not necessarily tell the full story about what that project is going through, you know, inside. So, most times we could see high commit velocity, um spikes and issues, but that could mean unclear docs. There are some projects that do not have very great documentation. I know we've come a long way. Way back 2018, 2019, you would

see lots of open source projects that do not have readmes on GitHub. You want to contribute, you don't even know where to start. You don't even know what the project is all about. But we've come a long way. We've seen projects make improvements on that part, but that doesn't mean there is, you know, the job is done. We still have projects that need to improve. So, um

those are also issues that we need to fix to actually help maintain this project. And then most times we could see many PRs, but that could indicate poor review practice. It means maybe people are not actually reviewing this PRs. on the flip side, you could see a project that has very few commits, low issue counts, low releases, very small team, but dedicated. And it could look like,

"Oh, nothing is going on here." I do that sometimes. I check a project. I want to see when was the last time someone made a commit or a PR. And I'm like, "Since 6 months ago, 8 months ago." I'm like, "Nothing is going on here." Well, maybe a lot of things could be going on in the background as well. So, I love to um bring um this

case study of what happened way back in 2024 of a maintainer that was managing this executables. He was burnt out. It was just a a single maintainer. The interesting thing about this case study was how it happened. A maintainer was managing this particular project. He was burnt out, no help. And then someone came in, said, "Okay, I'm going to help you manage the project." Made a lot

of commits. And we didn't know this person had a material motive. And when they got access to the to maintain the project, they planted a backdoor, which would have been, you know, very disastrous because this is an important project um to the Linux distros. And had it been it wasn't discovered way in 2024, it would have been distributed to to all the distros. So, and this with

this is metrics would have showed us if we if people actually paid attention. So, on the surface, the project was showing active commits, a new contributor joined, there was regular releases, issues were being addressed. So, you are going to say, yes, this project is actually very healthy, but then, on the flip side, there was just one maintainer. So, it was at risk, a huge risk, cuz what

happened What if this maintainer decides to say, okay, I cannot do this anymore? What would have been the case? And it also shows there was no governance model, no documentation, because you cannot just come to a project and become a maintainer. There There needs to be processes, you know, for you to get to that level. So, the verdict is this project was at risk. So, how do

we bridge the gap between visible activity and real community health? I love to go with two foundational ideas, and they are sustainability and survivability. Then, on the sustainability part, what are the things you can do to sustain a project? And that comes to contributor experience. How do you welcome newcomers? Do you even get newcomers at all? Because for a for a project to be sustainable, people needs

to There needs to be an influx of new people on that project. Because one day, the old maintainers are going to retire, they're going to go do something else. Who is going to take up the the leadership mantle? And then, documentation quality. We are looking at your onboarding, your communication style. Um, you know, do people get responses on time to their PRs, to their questions? Or is

it like someone is talking to the void? They're asking, um, "How do I contribute to this particular working group?" And nobody is responding. So, these are some of the issues that you need to look at to your project to make sure that it's sustainable. And then, survivability is Can the project survive a crisis or sudden change? Even maintainer decides to quit tomorrow. Is that going to be

the end of that particular project? We also looking at organizational diversity, governance strength, succession planning. Do you a documentation where someone could actually see how they can get from point A to point B or from being a user to a committer or to a maintainer, stuff like that. And that brings us to the CHAOS project. CHAOS stands for community health analytics in open source software. It's a

Linux Foundation project that was established in 2017. And what we do at CHAOS is we develop metrics, metric models, practitioner guides for maintainers or suppose, community managers to use to measure the health of their communities, of their projects. Over the past few years, we've developed up to 90 metrics or more. We have about 11 practitioner guides. Um even on how to access the viability of a project.

If you want to sunset an open source project, we have a guide for that as well. If you want to measure the impact of funding on your project, we have a guide for that as well. And then we have working groups, about 10 plus working groups, D I O S P O. Science, funding, AI in case if you're interested. And these are the links if you want

to check it out as well. And then we have tools, the Grimoire Lab and the Augur. These are two um Python library based tools that you can use to visualize. Um it's very technical, so I didn't want to go very deep into it cuz it's going to take take all the time. But you can check it out. I have added the necessary links as well. And then

these are some metrics I pulled out from the CHAOS software health model that we can look at quickly look at. So, we have the contributor absence factor. How many people make 50% of contributions? We see a lot of projects very active, but it's just few people that are pulling those activity that you see. Just very very few people. The rest are just in the background. That's not

great, but then we have the elephant factor. We have the time to first response. How long does it take for a human to respond to an issue or to a PR or to a question in your community? Then we have the change request closure ratio. We have the organizational diversity. If your project is dependent on organizations contributing to rate or supporting it, how many organizations are contributing

to it? How many organizations are If it's just one, what happens when they decide not to do that Is that project going to survive? And then we always say numbers need narratives because if you look at numbers most times they do not really tell the full story like I said earlier. you could say a project is dying, but then probably they reached their stability. They They are

done building. You could say a project is going growing very fast, but then it could be that they are very confused. They don't know what they are doing. You could say a project you know, someone is very dedicated, but then it could be that they are they are burnt out. They are tired, but then because they are very passionate about this project, so they have to keep

going. And I always love to say that health is a lived experience. So, most times we talk about projects, projects, we really do not look at the communities holding these projects. And then I pulled this from the Tidelift 2014 and the Eclipse Foundation Open Source Congress 2025 report, so we have a succession crisis at hand. We've seen lots of projects that do not have people to take

over the leadership um role. We had this conversation yesterday at diversity lunch where we see projects that have Okay. And we're trying to say we do not want projects to get to this level where you're trying to retire, but you don't have mid-level or experienced contributors to take over leadership roles. So, how do How are we going to do that? How do we make sure that projects

do not get to that level. I'm going to come to that. And then I'm also pointing out that health is beyond code. So, as from research, we we've seen that a lot of open source projects depend on non-code contributions. So, we want to make sure that projects do not rely solely on just code. So, what are other parts of projects um contributions that actually make a project

whole, like the documentation, the community management. We've made a lot of improvement in the aspect, mentorship. So, we've seen projects trying to mentor people now. We've seen uh projects take community management seriously, social media management, because if people want to contribute to your project, your community needs to be active. It needs to be healthy. People need to find you as well. Some projects cannot be found. I

People ask me, "I want to contribute to open source." And I'm saying, "Do your research." But then, your project can't be found. How can someone make contributions to it? we also need to learn better learn to ask better questions when it comes to the health of a And what does healthy actually look like? It looks this. Instead of just one you know, leading the project, we need

to have one or two or three more. Your onboarding needs to be great as well. Have good first issues for new newbies to start with, you know, something they could try their hands around. So, just actually make it easy for people to contribute to your project and you could start with this as well, having a contributor ladder where someone comes to your project, they know how to

move from point A to point B, from being a user to a contributor, to a committer, to a maintainer. Some people take open source seriously like a full-time job. They want to be sure that, okay, someday they could sit at the board or become a make it very clear to them that you could actually be whatever you want to be on this project as long as you

keep showing up, you keep doing the work, and you know, you're very committed. And that's why you become a committer. So, because of time, um I'm rushing a lot. what can you do today? You can evaluate projects differently. You know, ask about sustainability and survivability, and not just about, "Oh, how many stars do they have?" and all that. You can also check out the CHAOS practitioner guides

to access projects. You can use it to understand how to, you know, measure the sustainability of different projects. And then you can also join us We're open to new members. You can also join us to build, to develop this metrics because we keep developing new metrics and with the influx of AIs and maintainers complaining people using AI to, you know, draft PRs and all, we are going

to keep working on new can use to measure that as well. So, um we hope to see you on the project. I think that's the end of my presentation because of time. My time is up. So, if you have a question, you can meet me outside and, you know, I'm going to answer all of that. Thank you.