To Upstream or Not? Why Becoming the Maintainer of Your Dependencies Matters - Christos Markou

This talk emphasizes the importance of becoming maintainers of open source dependencies, particularly within the context of tools like OpenTelemetry. The speaker, Christos Marcou, discusses the dynamics and stakeholders involved in open source, including users, maintainers, and vendors. He shares insights into a real-world scenario regarding the OpenTelemetry Collector and the challenges faced when managing Kubernetes events. The talk highlights the value of maintainers in ensuring the sustainability of open source projects and encourages developers and vendors to engage with the community to strengthen these projects. Ultimately, it presents an argument for investing in open source contributions as essential for the long-term health of technologies we rely on.

Good morning everyone. Thank you for coming today. It's the last day, so thank you for being here. So, the talk is about why becoming maintainers of open source dependencies matters for us. And who am I? My name is Christos Marcou. I'm a software engineer at Elastic. And I'm also maintainer for the OpenTelemetry Collector Contrib project and approver in the semantic conventions of uh some of the semantic

conventions in uh OpenTelemetry and also a CNCF ambassador. So, yeah. Why this talk today? We are here at KubeCon, which is a conference which is about um open source technologies that run a significant part of our digital world. But coming here, we realized that behind these technologies, we have all these people that develop and maintain these technologies. So, there is a question here. Should we consider about

becoming these people and how we can make this happen? First, let's try to understand or remind why open source is important. And let's take a minute to um appreciate this meme actually, which in a funny way tries to highlight a crucial reality. So, most of the times, our systems, our digital infrastructure, might rely on technologies, on open source projects that are out there. They are developed and

maintained by a random person on on the other side of the planet. And this is quite important because we don't always know if these projects are uh well maintained or well uh staffed, right? So, this becomes critical for us. And we also have uh situations where things go south. We have this recent example of the Ingress NGINX um Ingress NGINX project of the Kubernetes ecosystem, which recently

was announced for deprecation. Uh that project was not well staffed and the maintainers called for help. Unfortunately, this never happened. So, the retirement of the project happened recently. And apart from this being sad, it's also uh a great migration pain for the teams that are using this project because they need to move away from this to something else. So, we're at the central question of this presentation.

Should we engage in open source upstream work or not? And we'll try to approach this not from a philosophical perspective, I hope so, but from sharing real-world insights. So, let's try to understand the stakeholders and the dynamics of open source. First, we have the companies and the vendors that use this open source or they build on top of open source projects. And they usually hire main- core

maintainers of the open source project so as to ensure that these projects will be well maintained and that they can also have direct access to maybe maps or the priorities of these projects in a nice way. And this is more or less an investment that companies and vendors want to do. And this is not bad. Uh that's fine. Um as soon as we ensure that the project

remains open. And uh we need to to ensure that we will have proper go- governance, uh we will have uh transparency and we will maintain the balance in these Then we also have the maintainers. So, the maintainers are key stakeholders. They carry long-term technical ownership of the projects. They are responsible for approving, reviewing, rejecting changes uh on the And they are in general there to ensure that

um there is um the the pro- the the the standards and the quality of the project uh remains there. So, these responsibilities usually are formed uh with uh technical committees, governance committees, special interest groups Uh uh in the Kubernetes ecosystem and the CNCF in general, and maintainers are not supposed to be gatekeepers of the projects, but they are mostly there to to ensure that the roles are

are clear, there are fair processes, and there is accountability for uh these processes. And last, we also have the community users, uh which actually use the open source projects, they put these into the test, they use them in production environment, and they come with honest feedback back to the project. Um they surface bugs, limitations, edge cases, and It's important to understand here that these users do not

necessarily control directly the road map, but their needs and their trust to the project define how the project remains relevant. So, I tend to believe that whenever these three different perspectives converge, we have actually projects that are successful, open source projects that are successful, and remain sustainable. So, we have the maintainers, the open source community, the users, and the vendors that also build on on top of

these projects, and these signals need to align. And let's try to keep this in mind because I will try to explain how this is actually I will try to prove this during the presentation. So, let's now uh take a break and listen to this story. Um this story is about the open source garden. This is a community-driven effort. Um this garden belongs to no single person, to

no single company, and everyone is free to come and join this effort to enjoy the garden, do picnics, uh take pictures, uh walk inside the garden, and also people are encouraged help with the garden, to come and fix what's broken, to plant new flowers, to collect dead leaves, and all this um useful stuff. Also, inside this garden we have uh people with specific roles. We call them

maintainers, and they are responsible to the garden remains healthy. They decide what needs to change, what what needs to be fixed, and they also can decide what new uh can happen in the garden when something new can be planted, for example. Also, inside this garden we have um part of this garden is also vendors uh that they also contribute to this garden. They invest a lot. They

put uh they invest money and effort to this garden. And some of those also offer paid tours. Also, they can plant their own flowers, and people can watch them and buy them if they want. And they also contribute a lot to this effort. But here's the question. What if a vendor um needs something urgently for their business needs? Can they bypass the processes, bypass um the rules

of this garden, and apply the change uh without asking nobody, for example? The answer is no. They still need to go through the same maintainers, through the same processes. And the second question is is this bad, and what the vendors can do in that case? I don't want to spoiler this now, so let's keep this question aside, and let's see if we can provide an answer to

this by the end of this presentation. So, let's get back to boring stuff like open source observability and open telemetry. And let me ask you here how many of you are familiar with open telemetry. Let me see your hands. Okay, that's cool. Uh was expected more or less. So, open telemetry is the second largest CNCF project right now, and it actually is an open source standard for

telemetry, provides support for collecting metrics, uh traces, logs, and profiles recent profiles recently. And in general, it is a vendor neutral telemetry collection provide support for vendor neutral telemetry collection implementations and in general it is a vast ecosystem that enables the users to instrument their applications and their systems to have powerful telemetry for them. So, I will now focus more on the open telemetry collector which is

a project the collector is a vendor neutral telemetry agent. It can it can collect ingest telemetry using the collector the receiver components. It can process and transform telemetry using the processor components and eventually you can export to observability backends using the exporter components. the collector in principle is vendor neutral but usually vendors decide to distribute their own distributions to ensure that the adoption for their users smooth

and they provide the best user experience for them. So, we have examples for from from multiple vendors. We have the AWS distribution, the distribution from elastic and distribution from data dog and many others that I don't include here in that list. There are many out And now I want to focus more on the collector which is a distribution of my employer and a project that I'm also

involved in. And collector is an a downstream distribution of the upstream collector. This means that it has the upstream collector components as direct dependencies of what we ship. So, the upstream receivers, processors, exporters are directly imported in our distribution and if we need to to fix something or to change something or to provide a new feature, we contribute directly to the upstream components. That is the idea,

that is the goal. In that collector follows the release schedule of Elastic, not the upstream release schedule, and in general comes with sensible defaults and some specific components that allow our adapt the product easier and move from legacy solutions of Elastic to pure OpenTelemetry based solutions. And these extra components are also open source. So, now we will talk about a real story that has to do with

Ingest Collector and the collection of Kubernetes events. That's the interesting part. with the collector in general, the OpenTelemetry collector, you can collect Kubernetes events using a specific component, which is called Kubernetes events receiver. So, this component connects to the Kubernetes API can collect Kubernetes events directly, and it extracts the message of the event as an OpenTelemetry log, along with useful information like Kubernetes metadata, then then the

node name or the pod name that is involved with to this with this event and um similar things. But, you can also use another more generic component, which is called K8s objects receiver. And this component again connects to the API of Kubernetes and can collect raw Kubernetes objects and export them as OpenTelemetry logs. But, in that case you have the full object um along with noise, probably.

So, you can So, because Kubernetes events are also Kubernetes objects, you can use this component and tune it to target events specifically. So, with this component, this generic component, you can collect events. So, you have two options here. So, the question is which one you should use. And we also had this question back then we were design when we were designing in the collector and we were

deciding which components to include and back then we found out that the K eventually receiver was marked for deprecation and the plan there was to ensure that there is feature parity between the two components and ensure that the generic component can provide the same functionality that the specific one was providing and eventually deprecate the K eventually receiver in honor of the K object receiver. That was the

idea. And based on this we decided to be future proof so we decided to use So indeed we had this part of the distribution and we were enabling this automatically in our helm charts targeting events. So what we were collecting would look like this raw events as Kubernetes objects as a JSON string along with a lot of information. Luckily in elastic search which is our back end

you can define this field as a flattened type field which means that then you can search inside the nested keys of this raw JSON thing which is really useful because it means that you don't need to apply any additional transformations you store what you collect and you should be fine. So we had a solution that was great. But after a while we started noticing that there is

no actual progress at the deprecation issue and users community users would come and ask questions about this and ask what is the progress there are what is the status and if the deprecation is eventually happening. But there is no clear way forward because there were some other issues that were potentially blocking this deprecation but there is nothing specific yet so this brings confusion to the community right?

We even had complaints from users um, mentioning that the usage of the cage object receiver is not ideal what you collect comes with extra noise, and in order to get rid of this noise, you probably need to apply some transformations. Of course, you can need OTTL of OpenTelemetry, which is extremely powerful and useful, and use the transform processor to extract only what is useful for you, but

this brings an extra maintenance burden to the users, which makes the configuration complex, and you might not, uh, you might, uh, don't want this for your users. at this point, by this time, I was already code owner of these two components, um, at the upstream project, and I started feeling this, uh, confusion, and I started feeling this pressure that this is more or less becoming a technical

debt for us, and we needed to do something about this. I even had questions internally from, uh, my colleagues, even my manager manager asked me, and yeah. there keep coming There are questions that are keep coming from, uh, community users. There are even requests for additional features, but we cannot accept them because, uh, the progress is is blocked there. We don't know yet the future of these

two different And I can keep posting screenshots here, but trust me, uh, there was confusion. I took the chance to, think that I need to do something about this. I felt there is this responsibility as a code owner. So, I asked my manager to allocate some time specifically to investigate this and see if we can do something there to, uh, improve the situation or hopefully resolve the

issue. So, I asked my manager, I got the green light, so I started, um, looking through the issues, uh, go through the history, um, collect all the information, and see, uh, if I can come with, uh, some specific proposals. I came with some alternatives and I tried to revive the conversations with the rest of the code owners there, but I also realized that this is, um, an

issue that is wider. It's, uh, it's not only specific to the collector. We need specific guidance on how to handle and manage Kubernetes events. So, maybe this is relevant to the semantic conventions and indeed there were some linked issues to the deprecation issue, um, that were related to the Kubernetes events in general as part of the semantic conventions project of Open Telemetry. I found also that issue.

I thought that it's a nice idea to join the meeting, the SIG meeting of this, uh, project and raise this issue with the maintainers there. So, that's what I did. I discussed that with the maintainers and I got some, um, early valuable feedback, I would say. But then I also realized that, um, well, this is Kubernetes specific issue. So, after that meeting, I created, uh, a specific

issue for Kubernetes events specifically and the goal was to, uh, ensure that we have a semantic, semantic conventions compliant way to deal with Kubernetes events across the Open Telemetry projects. I also joined the meeting of this group. I was already participating there, but I joined with, I joined with this specific, uh, topic, uh, for the agenda and I discussed that with the rest of the, uh, people

there, the Kubernetes experts and we actually realized that this is a valid issue. We need to do something about this, uh, soon because it is becoming a pain for us. And indeed, after a while, uh, a couple of weeks later, I think, uh, another code owner revived the conversations back in the original deprecation issue and he suggested actually that we should consider about canceling the deprecation and

keeping both components because the generic component was useful for generic use cases, but also the specific uh useful as well. that was a valid uh suggestion. Uh we agreed on this, and we also decided that we need to improve the maintainability of the code between the two different components, and we also need to ensure that we will write this everything we will write the all the guidance

that is required uh within the semantic conventions project to ensure um an OpenTelemetry compliant way to deal with the Kubernetes events, and this is specific guidance that users can find. So, based on this, we actually were able to close this uh issue after long time to avoid any further confusion, and we created some follow-up issues for this, that was great. We managed to solve this eventually, and

this was a technical debt that we managed to um fix, and as code owners as maintainers, we were happy. And also, the community users were also happy because all the issues that they had raised now either have an access or have an answer already, or they can have an access. So, this is really nice when that happens. I was super happy about this, but the most interesting

part uh comes after a while. So, a few months later, I started seeing customers of my employer, customers of uh data collector, let's say, complaining about not being able to properly visualize So, the issue there was that despite having this flattened type field that I mentioned before, Kibana was not able to visualize the nested keys directly um if there is no specific mapping defined for the schema

of the database, let's say. And usually, we rely on uh dynamic mappings that are generated automatically, but in that case, we had to ask the users, the customers to manually create this mapping and this is not the best experience. So, that was actually an issue for us and our customers. And indeed, that was an issue for multiple customers, not just one. So, we need to do something

about this again. Um downstream this time. So, I started the discussions with my teammates, uh my teammate Andreas and we evaluate we were evaluating the alternatives. We could either ask from Elasticsearch and Kibana, which is the UI for uh Elasticsearch, to fix this issue, but this would take time. So, that's that was not ideal for us. Or we could use OTTL and additional transformation transformations during the

ingestion time to get rid of the noise and only store what is necessary. Uh that would be a valid workaround, but uh in general, we aim for minimal configurations for our users. We don't want uh complex, really huge configuration files. So, that was not also ideal for us. But then, I realized that we can have this uh specific use case solved. We can have uh this extraction

to happen by using directly the other component, the Cage Event Receiver. Uh we remember that we were not able to use it initially because it was marked for deprecation, but now it's not going to get deprecated and we know that because we were there when that decision was taken and we were actually some of those that were driving these conversations and we managed to resolve this a

while ago. So, that was a great moment. Uh we were really confident about this change. So, we started evaluating this approach. We uh checked if we can replace uh the components. Uh the tests were really positive and we really fast made this component part of the distribution after uh few releases and after actually for we we we were ready for the next release of the elastic stack.

And we also tuned the helm charts that we ship to our customers to our users uh to use uh the component the new component and we also tuned some of the dashboards uh to achieve feature parity and eventually we could get rid of the uh cage object receiver because we don't need it anymore. So we removed that from not from the distribution but uh from the configurations

that we ship. And the outcome was really positive I we can um imagine here that having happy customers is always good thing. And also the support teams were the support team was also happy because they managed to resolve all these cases uh in a really confident and fast way. This took us like two two two weeks I think uh to ship the change and then the next

release uh of the product was uh included this change. And they also created some knowledge articles to share this knowledge with the customers that would be potentially affected by this. the community users and the maintainers of course are already happy because we saw that before it was fixed at at first place uh before this happened downstream. And everyone is happy eventually right? Which is super good. And

we can now close this loop of this diagram here. Uh we saw that before we have maintainers happy community users happy. Now we also have the vendor which is happy my employer um eventually. I think this diagram and this example highlights uh this idea that whenever the different perspectives converge we actually have indeed open source projects that are well maintained are successful and are sustainable. And to

get back to the story and try to provide a happy end for this um I think the ideal solution is that the vendor representative decides to put the effort in order to become maintainer of this open open source garden. And with this the vendor will be able to move faster, will be able to fix issues at first place with before even having them affecting uh the customers

Not because the vendor can now bypass the rules or can do whatever they want, but because the trust is already there and the maintainer is already uh familiar with the code base and have uh all this technical um experience uh for this project, has already established the uh relationships with the rest of the maintainers and the trust is already there. And this is very important. And I

think this is the magic of open source. So, being present uh pays off most of the times. And my message for you today is that if you are a vendor or uh you use open source and you are heavily um you heavily rely on open source uh you should consider about doing this. It's an investment that uh totally makes sense and uh worth considering about it. And

especially if you are using technologies like uh OpenTelemetry or Kubernetes or other CNCF projects, uh it's something that I would highly recommend doing. to wrap this up, some uh closing thoughts and lessons that I have learned so far. if you eventually decide to do it, I would suggest start small and show up and try to engage with the community, the maintainers, the project. come in alignment with

to to know the guidelines and the processes that the project follows and try to build the trust with the rest of the maintainers. This takes time, but eventually you will get there. So, don't get disappointed if things are not super fast. Um it will happen eventually. So, keep trying. try to also actually help the project. Don't try to go there and suggest a crazy feature that you

want to implement. This will put extra load to the maintainers, especially nowadays with AI coding tools. Uh it can be an extra load for the maintainers. So, that won't be helpful. Try to fix something that's actually that is actually useful for them, and they will highly appreciate this, I would say. And whenever you feel ready, step up intentionally, uh volunteer to get extra responsibilities for the project,

become triager, approver, uh maintainer, whatever the project defines. And while doing all these, always remember to be kind, be open, and um build for the future. What we are doing today in open source will be maintained by someone else in uh the in in the future, in like few years from now, for example. And this is how open source works. And I like to think of open

source not as a competitive race, as a marathon that we run uh all together, and I suggest that we see it like this. It's really useful for me, at least. And with these thoughts, I would like to give a huge shout out to the maintainers that are here today in the conference. Uh if you meet them, say thank you uh to them, show your appreciation. They do

uh very important work for us. And also with the image of the open source garden, thank you very much. That's all I had for you today. If there are questions, I would be happy to take them. Thank you.