About this talk
This talk discusses the Eclipse metrics program, which aims to gather and provide insights on the health and activity of Eclipse software projects. Boris Baldassari from the Eclipse Foundation explains that understanding and using these metrics can be complex due to differing interpretations and contexts in which they are applied. He emphasizes the importance of careful analysis when evaluating metrics such as commit counts and project activity, noting potential discrepancies when comparing different projects. The session covers the technology stack used for collecting and analyzing metrics, including Grimor Lab and OpenSearch. Moreover, the speaker reviews the challenges faced in metrics computation, particularly around identity management and project evolution. He concludes by inviting feedback and encouraging the use of the metrics tools for deeper insights into project activity.
Full transcript
[music] >> Hello. Thank you for joining this session about Eclipse software projects in general health and activity. We're going to talk about the Eclipse metrics program. So, it's about gathering metrics on Eclipse projects. The idea here is to let you know about what we've done, what is available, what you can reuse to provide practical examples of how you can actually use it. And some caveats, warnings, and
we would love to get some feedback from you. quick word about myself. I'm Boris Baldassari. I work at the Eclipse Foundation within the EMO team. So, basically what we do at the EMO team is we help projects Eclipse projects succeed so by working on the best practices, governance, onboarding, training, providing education material, and so on. And so, we if we can help them with communicating about the
activity and so on with stuff like metrics, we're happy to do so. My background is in methods and tools for software development, so software engineering basically. I've been involved in open source for 25 plus years. And I've done uh quite a few projects around metrics. I've been involved with metrics for over for yeah, 15 years, I guess, something like that. My intention here, as I said, is
to present what we've done and once again get some feedback. So, please do not hesitate to get to me after the the presentation or whenever I'm available. So, you can find find me and my contact on the Eclipse Foundation website and so on and so on. You can easily reach out to me if needed. And I would love to hear your one word about metrics. Some people
see metrics as oh, I can demonstrate something with metrics and that will be easy and so on. And as soon as I show metrics about my project, things will be just easy. Not at all. So, metrics can help, they can provide very valuable insights, but they can harm. And and metrics and visualizations can harm uh if they're wrongly used. And even if you use them correctly, uh
depends you who you are talking to. So, the thing is it really depends on the context and what people expect from the from the metrics. So, if you have something in mind and you want to prove that with metrics, chances are that someone in the in the audience will have a different understanding and they will argue and that can last for hours. I've tried it many times.
Um it's an issue. So, um we can use metrics, we can use visuali- visualizations, but uh it's no silver bullet. Still, it it can really be useful. A few things to keep in mind when dealing with metrics. So, the first thing is they're really hard to compute. And so, there are different ways to compute them and so, which give different numbers and that doesn't mean that each
method is is wrong or not. They're they might be all right, but then it depends what you count, what's the scope, what's the the meaning. do not expect exact numbers even if the metric system are are good. I'm going more into details shortly after that. Another thing is be cautious with the meaning of metrics. There is no good or bad metrics, so do not use metrics to
have a judgment on anything. That highly depends on the context, what people expect. Well, what people expect is usually not what you expect. So, you've been warned. we've started this this program 6 months ago now. And so, I already got a number of questions. And the question usually is I went to GitHub and I saw this number of commits and so on. And people want to compare
that with what we have. Guess what? The numbers are different. And then you start explaining that it really depends what is the the scope of the of the metrics. So, the classic regarding commits is do you count empty commits, merge commits, do you count bot commits? On what Git branch does that apply? So, the metrics we provide are on on all Git branches. Usually on GitHub, you
only have the number of commits for one branch. And it's not easy to know the total number of commits by looking at the different branches because of the merges So, be ready to get some questions, argumentation, and there is no simple answer like oh, yeah, I do have exactly the same the same number. Um the next thing is about the semantics Let's take a very simple metric,
which is the number of commits. It's supposed to show the activity of the project, but never ever try to use that to compare the activity of two projects because one project so, one team might use development feature branches to do some development and maybe at the end they they squash all the commits, so you only get one or few commits instead of say, 50 of them. That's
fine. That's absolutely valid workflow. And some teams will use very small commits, atomic commits, so they can wait for the CI/CD to get back to them, execute the the the test and so on, then do some correction, they can revert if needed. And that's absolutely fine, too. So, using the number of commits to compare the activity of two projects is wrong. That won't work and you really
need to pay attention to meaning the semantics the semantics of the commits of the metrics. Last but not least, I often get requests about oh, you said you had metrics and I want the download metrics. I want the cyclomatic complexity metrics. No, we only provide activity metrics, so that's basically metrics on commits, issues, and MRs/PRs, and we also have mailing lists, as we'll see. So, there is
no download metrics and no code metrics. These are very interesting metrics, we just don't compute them. you've been warned. Let's get into the the details of what we have. So, we started working on the on the metrics program in 2024. We have some quite strong objectives. So, this should apply this must apply to all Eclipse projects, 450 plus projects. We do not count archive projects. We our
stakeholders are the working groups, projects, developers, and end users. and as you probably know, we use a number of different tools. So, there are we use GitLab, GitHub. We used Gerrit as well. So, we needed to find a way to compute metrics on these different forges and systems and basically be being able to plug them any system that we have. So, and possibly in the future things
like Jenkins and so on. There are quite a few challenges with that. One of them is so, some projects use GitHub, some projects use GitLab, some projects use both. Some projects have repositories in the same organization, but not always. Um and depending on on the people in the team, some projects will use the mailing list for communication, but some other projects will use the issues and discussions
in GitHub. we really need needed to find a way to kind of map some of these features and be able to being able to address everything. So, GitLab, GitHub, Gerrit at the time. The next challenge is projects evolve at fast pace. So, every week or so, we have new projects, some projects are terminated. So, projects can add repositories themselves. So, there is no process to add a
repository, to change or archive a repository for a project. So, they can add 10 repositories in in a week if they want to and we have to keep up with that. So, when someone creates creates a new repository, we need to be aware of that and couple of days after the creation, we need to have the metrics for that repository as well. another big big challenge is
the identity management because obviously people do not use the same accounts and all on GitLab, GitHub. Uh and we compare that the Eclipse Foundation accounts. And trust me, there is absolutely no homogeneity between GitHub, GitLab, and the Eclipse accounts generally speaking. And even more since we also analyze the mailing lists, people can use whatever address email address they want to actually register on the mailing list. So,
we have something like several hundred thousands different identities. the interesting thing is to be able to map this. So, we know that this So, this identity on GitHub, on GitLab, and these three different email addresses actually belong to the same person. What we want also is to have affiliations, so we know who works for what company. And so, because in the end what what is of interest
is to know what are the companies that worked on these projects and try to have a a percentage of associated to each company and so on. Which is great, except that some people work for several companies at the same time and then they change companies. So, the commits that they've done before this date will belong to this company and after this date will belong to this another
or several other companies. And they don't tell us. So, we need a system, an identity management system that's kind of can automatically merge um say most of these different identities. We need to be able to actually fix some of them and that's a nightmare. Several hundred thousand identities, once again. And we want to do that by and still respect the GDPR, the uh privacy policy that we
have at the Eclipse Foundation that we take very seriously. So, that's a number of challenges. So, what we came up with is some something that is I'm entirely based on open source solution. That's nice. We provide two different views. So, one for a quick overview of the on the on the project, on the working groups and so on. And another one for more deep research. for people
who are more advanced and who can build complex queries and dive into some specific um data sources and so on. So, the stack is based on Grimor Lab, which is open source developed by Bitergia. That's for the data collection container orchestration and so on. We also have Sorting Hat for everything the identity management. Uh we use OpenSearch, so that which is all part of the Grimor Lab
stack. Uh all the data is stored in a OpenSearch, so no SQL database. Um which with an interface web UI like. So, it's a fork of Kibana. So, if you know a bit about Kibana, you can use that very easily. And we and we also have a bunch of Python scripts, uh Pandas, Plotly, a number of open source um libraries that are very handy. quick overview website
is metrics.eclipse.org. You can go there and from this one actually you you can go to every everywhere else. So, from there you have the the documentation, you have the metrics for the project, for the working groups and you have a link to the other open or the the OpenSearch dashboard. So, if you have to keep one link in mind, that would be the one. And so, from
there you can go to the OpenSearch database. Um the dashboard, excuse me, which is this thing. So, you can really focus there on project on on on a specific project, a set of specific project, a working group, um individual users and Of course, because of GDPR, privacy and so on, you cannot see as a anony so, as an anonymous user you cannot see uh PII, so you
cannot see names and you cannot see email addresses, but you can see organization names and basically everything else. And for the record the the records, we also propose custom reports for stakeholders. So, say for working groups or for specific cases and so on. So, I do believe these custom reports and I'm not going to do a custom report for everybody, but if you have some specific needs,
you can go to your working group manager and we can start a discussion and so, usually I can we we can accommodate the needs. We can combine the this activity metrics with metrics coming from the PMI and and stuff like that. what do we provide? So, that's the eclipse.metrics.org website. As I said, um so, about yeah, that's about the project. You some documentation, that's really important because
that's where you will find definition of metrics, data retrieval process and so on. The list of Eclipse projects or all active No, no, not all active, all non-archived um Eclipse projects have their own page. So, you you go there and you can search for the project and so on. Same thing for all working groups. And the Grimor Lab dashboard is the one at the OpenSearch one. still
on metrics.eclipse.org Let's say that you clicked on the bottom left projects button, you go to the list of project and that's basically what you get. Um here you only have basic activity metrics. So, it's on a one-year time range. You have the number of commits, number of committers, issue um submitters, the PRs, MRs, depending on if it's on GitHub or GitLab. Number of people submitting issues. and
basically that's it. But the good thing is that we could combine this information with information coming from um basically Eclipse API. So, you have PMI information, the EMO reviews, the releases made by the the project, the IP Lab requests submitted by the project. We also have information about the security, which comes from Auto Doc. for some specific project, depending on the working group they belong to, we
can um specific additional tabs. So, Eclipse on Chaos is belongs to the um Eclipse SDV working group and for projects, there is this maturity assessment badge um that you that you get. So, Eclipse on Chaos is one of them. So, in we for that project specifically, we have this um um this tab. For every information we we we provide, we try to provide reusable elements. Um CSV
files are very simple to use. So, we have CSV files on one year's time, on once again, for commits, issues, reviews. So, what I call reviews is PRs or MRs, depending on the fork. You have the list of IP Lab requests. You can also get for free PNG plots of issues and commits. So, that's static files. And you can also export the SDV badges. Everything So, all
the how to do I reuse this content is in the documentation page. Um so, if you talk with your working group manager and we can also easily add some new tabs. So, as we did for the SDV working group, but for other projects belonging to other working we can absolutely add um tab for specific concerns for each working group. Um just one word for the CSV file.
So, the intent here is chances are that the plots provided the information provided is not exactly what you were searching for. Um so, you probably have some um BI tool or not Excel, but LibreOffice Calc stuff. So, it's really easily it's really easy for for you to just download the CSV file. This is the CSV file always has the same URL. So, you if you download it
every morning, you will have the updated um data. So, you can easily download it and do your plotting and computation, whatever. So, that's really easy to actually get updated data um uh continuously. Same thing for working groups. Um so, you can download so, you have access to the list of projects, the list of repositories, number of commits, reviews, issues. Um you can download the CSV files. So,
for commits, issues, reviews and the list of repositories. Once again, it's it's dynamic. So, if you download these files from this same URL every morning, you will you will have a different set of um of information. And so, for every working group we have the about repositories activity with the plots over one year. And once again, depending on the working group, you can have specific tabs. So,
for the SDV working group, we have the maturity tab that explains what the maturity assessment project is and so Um the documentation part of the of this website is very important because usually that's answers 2/3 of the question I get. Um so, and if you want to be a bit picky, you absolutely need to know about how we retrieve data, how metrics are computed, what to expect
from the from the content of the website, from the content of the We provide we provide guidelines to reuse contents. So, all the URLs to get the CSV files, images and so on are described there. And we also list known limitations and and caveats. So, I will get back to that later on. the So, now for the Grimor Lab dashboard, so you have full access to this
to this, except that uh you do not get the names and email addresses, but that's very powerful. So, you have the list of projects there. Uh you have a number of uh predefined dashboards uh on the top of the uh of the page. There is this um the time range here, so you can easily select uh 6 months, 1 month, uh 3 days ago 3 days ago
that won't work, but you can go back to the beginning of time Git-wise, I mean. So, that usually starts uh when we the when the project was created or or when So, for Git history, when the the first commits were were made. So, we can go up to 15 years uh It's really easy to um uh focus on the single project or to uh focus on a
single organization, stuff You have a huge number of visualizations. You can easily build reports and modify the visualizations If you know a bit about Kibana, but it's well documented um on the internet. You can build complex queries. Um that's not that hard because you have this uh filtering mechanism um where you see empty commits, not bots and so on, so you can easily play with that, add
uh um filter with through the UI, and you can also write your own uh queries if you feel like it. So, talking about these quick filters, empty commits and no not bots, um that's uh the default. Um so, we don't count merge commits in the number of commits as an example, and everything that is marked as being made by a bot is not counted, but you can
still count them. You just you you you can simply remove the uh these filters, and the the numbers will change, and you can add some filters. So, no SQL database, so you have number of indices with some fields, and you have the usual fields like the creation time The projects are registered by their full name, but you can also um research the specific project with with its
um ID. So, automotive.eclipse.org as an example. Um we also provided uh the So, some metadata. As I said, we have the the project ID. We also have the technology as defined in the PMI, uh the working group or working groups the project belongs Once again, uh Kibana is well documented, so you can just go on the internet, find some documentation, and you will be able to apply
that to the to to the this dashboard. We also have predefined dashboards for pure Git stuff uh with uh things like um uh the areas uh the code areas impacted by the commits, and uh so, everything that is pure Git, whatever the the forge. Then, we have specific uh dashboards for GitHub, for the issues, for the PRs, for the repositories. Uh repositories means forks, stars, uh subscribers,
stuff like that. Um same for GitLab. We only have issues and emails. We do not have the stars and forks for for GitLab. We also have the mailing list for our projects, and there are a few fancy features of um Grimor Lab in the mostly in community, where you can get information about uh I don't know if you've heard about that, but every community can be divided
into core contributor, casual contributor, stuff like that. So, it computes that for every project. Um these are predefined uh views because they take a lot of um time to compute, so that's a background task. So, they have a bit less flexibility, but you get a lot of information still. Um Kibana is great for that. You can also very easily share. So, if you define your own dashboard
for your own project, working group, whatever, you can easily share it. You can easily embed it in um another at HTML page. Um so, it's top right. You won't see it here, but uh it's all self-documented. And there is some more documentation on the website the metrics.eclipse.org website for the how to share and how how to reuse and share the the data. Now, some specific use cases.
Uh let's say that we want to evaluate the activity of our project, not for comparison, because that doesn't work, but at least evaluating the activity of our project. For the very basic information, you can go to metrics.eclipse.org, and actually the activity tab on the project or working group uh pages uh are um JavaScript-based, so uh it's Plotly. I don't know if you know it. It's great. Uh
when you hover the graph, you can get the the numbers. You can zoom in, zoom out. You can export a PNG file on the the zoom that you specifically um asked for. And that's true for commits, issues, uh reviews. Uh we also have plots about the all over 1 year. So, it's not 1 year, it's 1 year and the beginning of the month. Uh so, we have
um coherent consistent bar plots so that you can actually compare. And if that's not enough, you can still uh to the Grimor Lab dashboard for more in-depth analysis. Um you can select the project. So, there are two ways to actually select a project. When you have the full list of projects on the dashboard, if you hover the say Eclipse Open Vehicle, >> [clears throat] >> Excuse me.
Uh you have a little plus that uh that shows up, so you can just click that, and that will focus the entire dashboard on this specific project. You can do that on almost everything. So, organization, same. And one thing you probably want to do if you focus on the single project, organization, working group, whatever, um once you have um uh on the little plus, you should click
pin across all apps. So, you click on the new filter, and you pin across all apps. So, when you change uh switch to another dashboard, you you keep the same filter, and then you can just play around with the the dashboard and with only the data that you that you need. Um there are some very interesting views, so I'm not going to describe them all. The best
thing is to just go there and have a look, but you have um the number of active organizations over time, so you you can really easily find the bus factor of the project over time. Um you have things like uh uh recently lost developers, so um people stop uh who stopped contributing, and newcomers, um and you have the trend over time. Uh same for stars, subscribers, forks,
so that we that really shows interesting information about the activity of the project and the behavior of the um of the community. uh more examples of interesting plots. Uh so, I talked about the code areas. So, basically, with each commit, you have the the the files modified, and if you take that at a higher level, actually you you can easily get the files that are the most
modified, so the hot spot within your code base. Same for the most active directories. You can also filter by technology. So, by technology, actually, it's the the file extensions, but you can focus on these files and see who is um uh modifying them and so on or what organization is modifying them. You also have stuff like uh the backlog, so for um MRs, PRs, and issues as
well. So, you where you would have the the number and the list of issues that have been opened since 600 days, um stuff like that. Um and also information about the community time and so on, which maybe gives an idea of the technical depth or also some information. And once again, you will can export the full dashboard um you can share the the full dashboard. You can
share specific visualizations. You can create yourself um new visualization um and so on. Um so, once again, you you might find useful to just dive into the dashboard and spend some time there looking at your projects. Um usually that works. I mean, that was that becomes interesting quickly. Some known limitation and caveats. Um So, what one question I I get often is, "Well, I made this commit
yesterday. I do not see it uh in the in the metrics." Um so, the thing is there is a um quite some time to retrieve the data, to recompute the indices, and so on. So, it takes 24 hours to from the get into the system and and visible vi- um be visible. Uh so, the thing is really not um intended for real-time monitoring. It's really for historic
insight and trying to find trends and stuff like that, but it's not real-time. So, you might have 24-hour delay after um some new commit shows shows up. Um when you try to commit to compare numbers so especially especially with the GitHub insights tab uh, remember to have a look at the uh, the time range because GitHub on only shows up to until 1 month I guess something
Uh, do they do they take into account merge commits, empty commits? But usually it's only on one branch the number of commits you get. So caveats that usually explain the difference between between numbers. Also, as I said affiliations are very complex topic. So we are working on it like every day to try to map this person with this employer from this day to this date, but it's
a never-ending process we will get better over time, but sometimes there are still some hiccups or the unknown organization um um, maybe you might see that from time to time. Um, another caveat is as I said some dashboards are pre-computed for the community metrics. So things that usually take time to compute or you take a long time range to actually do the computation, these things are are
pre-computed so dashboard is uh is feels dynamic and you're not stuck. So which means that um you cannot easily change the you cannot change the time range on pre-computed dashboards. It's always written somewhere on the page. So if you look at the page and read through the running text you should know if it's pre-computed or not. And if in if in doubt feel free to submit a
help desk issue. I'm watching and we usually answer um quite fast. And we also building up a frequently asked questions section so we will add more um, answers as we get more Um, you can take the URLs to the different dashboards and once again it's a process it's it's not an event. We'll keep improving that and especially with your feedback. So if you have anything to say
see something that you do not understand and so on please submit a help desk issues reach out to me, reach out to your local Eclipse contact or to your working group and we will take care of that. Thank you. >> [applause] >> And I think we have something like 5 to 8 minutes. Yes, please. Hi, I'm Manoj I'm lead for the GDD project. So pretty interesting stuff.
I had a quick question. I have not used it I should admit that. Do you give repository level metrics also or you are looking only at a project level? Both. Okay, so I can get the repository level. >> So you would go you would need I think on the metrics.eclipse.org website we have both, but actually for more detailed information you would go to the open source dashboard
and what I said about the project and the plus sign you can do that on the project you can do that on the repository. So you have all the information you need at project level or repository level. Okay, one more question just do you get derived metric like you have number of commits you have some 10 people. So something like per capita income right like this many
commits per people do we get is it possible to put uh, request for such a uh, metric? Once again, please excuse >> I suppose one project has 1,000 commits and there are 100 people in the project. While another project has just 10 commits and there are only two people in that. So uh it's like you know per uh, how many commits per person is it possible to
get a derived metrics? Um, so I think you can get it in the interface because actually you can create new visualizations so you cannot save them, but you can play with the interface itself and I think you can do such computations. So you should be able to get it yourself on whatever project repository and so on, but we could also add it to the to the metrics.eclipse.org
project page why not? Please submit an issue so I can keep track of the requirements and so on and yes why not? Well, I'll talk to you after. Thank you. Thank you very much. thank you for your very interesting talk. Um, I have a question which is so if I understand correctly you're not exposing names. Right? I was wondering why this is because when I go on
the GitHub I can on the activities I see the contributors and on the Eclipse project pages I can also see the names of the people of the commits. Um, I'm asking because we were thinking about some kind of new leaderboard for our project because reviews are coming more important um, in today's world and we would like to have like a review leaderboard. And if you have the
data already would be nice if you for example could expose something like this. Um, that's open question. We want to be to be on the conservative side and take absolutely no risk for our users. We still don't know if it's okay to really show names. Um, I know that some similar system just put the names and email addresses That's a choice. Um the the thing is no
one knows if it's okay or not because the information is already public, but then by doing that you you just make it massively available. So if you want to export a full CSV of all committers you can. So there is a difference between going to GitHub and seeing this name whatever and having a full list of names and email addresses. So we are thinking about it with
the legal team and so on, we don't know and as long as we are not so sure we are going to be conservative. that's an open point. Yeah. Okay, thank Hi, I'm wondering uh have you seen patterns that emerge for healthy projects versus unhealthy projects? Are there things that people should be on the lookout for? Um, there have been people studying this question for over 20 years
and they still got no answer. So it's still a debate so well, there are a few things um the the bus factor and if you look at the the different committers committers so there is a way to have the timeline and make it so 100% which shows the percentage of commits made by one person. So this one I think is interesting. Um, and there are a few
others. I mean the number of of commits, issues and so on also show something but honestly these are just opinions. I don't think there is definitive answer, but I'm happy to work. So my take on this because that's what my PhD was on this kind of question and unfortunately I didn't find out at that time. But one of the lessons I learned is that it really depends
on the common agreement that we can get. So if we together a team some stakeholders and so on we work together on that that we might find something that won't uh, silver bullet for anyone out there, but that could be a good answer for us. So we could work on that and I would be there is no silver bullets on that. But nice try. Good. Thanks again.
