Jan Lehnardt – 2.5 Years of STA Bug Resilience: how we helped a lot of FOSS #FOSSBack

This talk focuses on the speaker's work with the Sovereign Tech Agency (STA) to enhance the resilience and security of open source software projects. The speaker shares insights gained from supporting various foundational projects, including systemd, PHP, and GNOME, while discussing the challenges and methodologies of reducing technical debt. Key strategies include analyzing organizational structures, improving testing and release processes, and engaging in documentation standardization. The speaker emphasizes the importance of meeting projects where they are and providing tailored assistance to improve their capacity for quickly addressing security vulnerabilities. Additionally, the talk highlights the need for continued funding and support for open source initiatives to foster a safer and more sustainable software ecosystem.

My name is Jan. Um, this is a wonderful and important stage. Um, you're a wonderful and important audience and I'm want to thank the organizers for inviting me here to bring us all together in a very important time in like free and open source software. So, I'm very honored to be here. Thanks for having me. I've been doing open source and free software and community work since

the late 1990s. I started out in a community that taught each other and other people how to do websites in a project called selfh that maybe like one other person knows about. Um but shortly after that I joined the PHP project for a while doing translation of the documentation then doing the actual documentation then contributing to the project itself like having a very normal like progression to

go get into projects and I've done a lot of stuff since um but the main stuff that I do these days is working on Apache Catch DB and Apache Patch DB which are really nice databases for building offline capable applications which is not what I'm talking about today but if you want to talk to me about that find me after. Um I do this mainly in the

context of a company I helped found called Neighborhoody. Um that's my day job with um a bunch of lovely people. Um we do support for couch and pouch. We do um build offline capable apps for others. Um we're a general purpose development shop as well. And this is where it gets interesting and why I'm here. Um we are an implementation partner for the sovereign tech agency. And

today's talk is about the work that we've done in the past. Um it's a little bit of a softer talk just before lunch. You can zonk out a little bit. There's going to be a bunch of fun anecdotes and some lessons learned from um what we've done for people who want to do this in the future. Um uh and maybe they can learn something from or you

can learn something from the work that we did and make sure in the future it goes better as well. Um this is the team at Neighborhood that worked on all these things. Um, I helped very little with some of these projects, but these are the people who deserve all the credit for the work that has happened. Um, Julia and Nette are actually here in the audience and

if you want to ask them any questions, um, they're happy to talk to you as well. Uh, I'm just the pretty face. The sovereign tech agency does not need an introduction here at this phase, but briefly maybe for people who are watching uh, the talk later online, it's a German government funded initiative to improve the free and open source software system. um and um does so in

different funding programs including um direct funding for maintainers. One of the more mo most useful things just give open source projects money they know what to do with it. There's also a one-year stipen for individuals some of wh are here who do something like hypothetically uh be the release manager for a mainline Python release. Um so they get to focus on the like critical infrastructure for a

year. Um the standalone security audits done by another third party. Um and finally the bug resilience program which there is not a term that rolls off the tongue or is an industry term that anybody would like understand. Oh that's what they mean. And this talk tries to demystify this a little bit. Um but how did the big bug resilience program came to be? um in the face

of heartbleleed and lock for shell, the German government wanted to like take some money and say, "Hey, please uh give this to open source or make sure this money is spent so open source projects are more secure in the future." And one of the most common ways to do this is a buck bounty program where people get paid if they um find security issues in open source

projects or in commercial projects and then those get fixed, the people get paid. SDA said that's great. talked to a bunch of open source projects and they all said, "No, no, no, no, no. What you're doing here is giving other people money to take away some of our time because we have to evaluate each of these security issues that they report and that takes a lot of

time to do this thoroughly, especially if you incentivize them with money. You give them money to take away time from open source projects. That makes no sense because we don't have time in the beginning, right? That's the main problem of any open source project. It's underfunded. Nobody has enough time." Um so the STA said okay we'll turn this around instead of um actively supporting people who find

security issues. We help the projects themselves to be more resilient in the face of submissions of security issues so that they can um fix it quickly respond to it quickly make a release quickly so that we know the turnaround time from an open source or security vulnerability being reported to a new release is as short as possible. Hence the term bug resilience. Um through the last two

and a half years of work that we've done here in various projects uh it turned out that the term there is an industry term for what we're doing here already. It is called reducing technical debt and the new version of the STA that's like it's a new like gets funded every four years then the next iteration starts next month. This program is actually called lowering technical debt.

So we kind of had to have this little dance about how do we call this when we had a term all along but that's how learning works. Um, these are the projects that we've been working with. I'll run through them real quickly. Um, system D, almost any Linux that runs anywhere runs systemd as the first thing it runs and then systemd does everything else on your machine

and it provides a bunch of services to run an operating system. Um, when we're talking about supporting fundamental software, foundational software system, it doesn't get much lower than systemd. There's only the Linux kernel, right? is very very important here. Uh Pippi, the Python package index is how the Python community and by extension a big scientific community um is exchanging source code with each other. Again, it's an

extremely fundamental project. YTO, you may not have heard of YTO, but you have definitely used a product that used YTO to to come into existence. It is a a meta project that allows you to build a bespoke um Linux distribution specifically for embedded systems. um is used in all sorts of things. Um for example, smart home devices. Say you have a want to make a fridge, a

smart fridge. You have have a little computer in there. That little computer needs to run operating system. YTO lets you make a bespoke operating system for just your needs with just the drivers you need and all the tooling that you need for that particular project. I guarantee you, you have used a thing that has been built with YTO. It's so widely used and like it's it's not

a common household name. At least it wasn't for me before we we met the project. Um, Genome also needs no introduction. It's a common desktop environment for Linux machines. There's a strong chance you're using Genome right now um to take notes or whatever. Um, servo is the only independent independently funded um web browser engine. Um, it's extremely important for the openness and sustainability of the web and

its future. It's one of the big most important projects going on at the moment. I'm very proud to be um able to contribute to that. Um, Popppler is another is another one of those. You probably haven't heard of it, but it's a graphics library. If you're doing anything with P uh with PDFs on on Unix machines or Linux, you are using Popppler. Um, PHP runs a good

chunk of the internet. Doesn't also don't have to tell you how important this is. Log forj every Java project that needs to like put some logging into a file to explain about the the state of its of the software itself is using log forj. Reproducible builds is another meta project that isn't a project in itself but helps to define standards and make other projects um give them

the opportunity to make sure that a binary that has been built from from a from this open source project source code is has actually been derived from a specific version of the soft of the software. A binary form is how any project runs on any machine. Um but if you make a binary you can theoretically tamper with it and put some nefarious code in there um that

then does bad things on the target machines but with reproducible builds you can ensure that that hasn't happened. It's extremely important for supply chain security. Typo3 is a very established content management system um and secoia PGP and open PGPJS are PGP implementations in Rust and TypeScript and JavaScript. So again very important for um safe and secure communication. This is a wide variety of projects, but they all

have in common that they are very fundamental to the field that they're in and that's the kind of project that the SDA tends to focus on. So what does bug resilience mean in practice? Um we have developed with the STA and with the initial projects and thanks for the early projects for like bearing with us while we develop this program on the fly figure out what that

actually means. It starts with analysis as everything does. We look at the organization of an open source project. Who are the people in here? Is this a group? How geographically distributed are they? What are their communication patterns? What tools are they using? Um but also um do they have processes for like reporting security vulnerabilities that adhere to common standards? That kind of stuff. Or like is the

contributor is this a single person opensource project and how is their workflow and their funding situation and all that. And then we look at a bunch of technical stuff. Of course, the dependency is a huge vector for for security issues. The fewer dependencies you have, the better. Of course, in a way, um we review the core project itself as well. We look at the testing specifically. Um

is are there tests that can be run? How much of the actual project are they covering? Are tests running continuously amongst multiple contributors or is it a manual thing? Um and then finally, we look at release engineering. um how long does it actually take from a decision like this the how the source code looks right now to a release how long how much time and how much

effort is it to to get to that and then we'll just go improve all that stuff it's basically the same stuff we look at the people in the processes we establish things that they may not have established yet um we help them with their testing infrastructure we help raise their code coverage um automate releases all that kind of stuff and some of these projects are very old

and established and they have all this stuff already because They're they've been doing this for a while. Um, but every project has technical debt. Um, technical debt in this context translates to it takes a little longer to get a fix in or to evaluate whether a fix is safe to release. It doesn't break anything else or um keeps the maintainer's attention away from actually doing anything other

important. So, if none of the the the typical things here apply, we'll go to lowering technical debt. Um, this all starts um with your typical video conference call. So, everybody gets to know each other. Um, and in that call we're kind of trying to establish like what is the difference between this hypothetical model of what an what any open source project could be doing to be most

resilient like it just takes five minutes from report to a new release would be really nice. Um, and what we can do then to to bridge that gap and if there's again if there's nothing there we'll just do the lowering technical debt bit. Another important thing that we're doing here is we meet the projects where they are. Some projects are on GitHub, some are on GitLab, some

are on mailing lists. Um, some have visual instance, the chat software somewhere. And we want to make sure to not take away from open source projects more than we need, more than we need to. Um, so that yeah, we're not wasting any of that time. Um, and then make sure we contribute as much as we can. Um, technically we all technically we also meet projects where they

are. Um, for example, for PHP, we did a an infrastructure project um for an infrastructure that has been organically grown in the last 25 years. And there was a lot of opportunity for helping um change the way they run their servers and the services that like make up what the PHP project needs to do. Release candidates, releases, downloads, security, attestation, like huge things that are incredibly important

for the trust chain that depends on PHP. Um, and we could go in and say, okay, the best practice in this area is you get a team of like six DevOps, developers, SR type people, and you go all in on various things of automation and resilience and kind of stuff. But that's not how that project work. It's just a handful of people who are very diligent, very

good at their job. Um, they just needed a little bit more help consolidating what they have. So, we didn't go in and say, "Okay, this is the the future. You have to go there." which meant for them jumping through several hoops to get there. We just leveled them up one level from where they were to make their lives a little bit easier instead of coming in and

be all smartass about hey other projects do this you should do that as well rather than come in okay how can we make your life actually easier bunch of challenges we found those um some projects didn't even take the initial call with us they looked at our website it's like well you're doing web development and offline database stuff surely you cannot contribute to our project in any

meaningful Okay, it's a waste of my time. I'm not actually going to talk to you. And fair enough. If you don't have the right impression that we would or if you have the impression that we waste your time, don't take the call. I would do the same. But the learning for us is we need to for us and the STA, we need to better communicate the services

that we can actually do. and we'll get to that later that there's very like for all the projects that we did that did take the call and were similarly skeptical in the beginning. We always found at least a little thing that we can do that made their life a little easier. Um but we learned from this to be more upfront about um what we can actually do.

Now there's a dedicated landing page on our website that explains the kind of stuff that we do and we started publishing retrospectives from the projects that we've done so that people can see oh that's the kind of stuff they can do. Okay, that's fine. we we have a bunch of those things as well. So that was a good learning. Um the next thing is each engagement for

each of the projects has a time scope of about 40 to 80 person days. That's a substantial amount of time for any open source project and we get a lot of work done in that time. But some open source projects need a lot more time and it comes in two flavors. The one is the up the ramping up like being able to actually learn all the things

that you need to know to meaningfully contribute to the project independently so that you're not coached by the existing contributors all the time because that's taking away time from them. We had one project that said it takes two years full-time to on board here. Like okay, we can't do that. Sorry. There's other projects I will name it auto tools which is essentially unmaintained at this point which

should make you all very scared. Let's please fix this. Um where we could say no the the ramp up to start fixing it is actually not that hard for the the skills that the team already has. But if we spend like even even if we stretch it to 120 um person days, we would like get like 3% into what's actually needed with these systems with these projects

to going forward. And it's not worth getting that started and then nobody takes it on. So this these projects need longer sustained funding situations and that's what the buck resilience program or now the um the lowering technical debt program just don't provide. So we need to find other models there as well. Um, some projects are really busy. So, we're we've designed this process with the projects together

to make sure that there's as few connection points as possible. So, we don't have to, hey, do you have enough time for another call? Can you review this email? So, we don't have to do a lot of this. There's a few coordination points that we have to have, but we minimize them as much as we can. And still, some projects get through that within a couple of

days, and then some projects need months to do all this. Um, that's not a complaint. I'm an open source maintainer myself. I have done for a long time. Sometimes you just don't get to a certain thing even though you know it's going to benefit benefit you in the long run. Um, it's just the reality of working in open source. Um, and then one thing that I was

sure wasn't going to be a problem, but the team really wasn't sure wasn't going to be a problem is that if you jump into like these 12 or however many they are, different projects, they're all different programming languages. They all have that their own choices of build system and testing environment and CI environment and how they contribute patches. there's a whole slew of new technologies that you

have to learn very quickly um to be able to do a useful contribution and again the team wasn't sure if they could do it but after a few of those um project that went very well they gained more confidence and I I always knew they could do it and but it's very nice to see that they now are very like they take the calls now and say

no no we can do this is fine we've seen this kind of stuff before it doesn't matter if it's PHP or Python we can like whatever we'll just do the thing that's needed Um, a bunch of highlights I want to mention and I've pulled these from the team. These aren't just mine. So, this is stuff that the team said, "Please talk about this. We're very proud of

that particular work." Most of the thing, okay, all of them are proud of as well, but they did the work, so they get to be even more proud. Um, so YTO again is this system that allows you to build your own Linux system and that includes your own drive, like the custom drivers you need and all the software you need to run. I to do the networking

and and and all sorts of stuff. Um and when you put together such systems, your users will um deploy them and they they will also want to make sure well is this actually secure. So there is automated security scanners that say hey is this version of this software say the Apache web server that you're using here for some reason is this actually secure with a version there

and then they compare these automated tools compare the the project against a list of commonly known vulnerabilities or called the CVE. Um there's a centralized list that shows here's every CVE like every vulnerability ever found in every piece of software gets a number and says this version sorry this software at this version is vulnerable against this certain thing and they fix it in this other version. Um

that seems like a very useful thing. Um the problem is this list is not very well maintained. We have closed the CVE that was open in 1998. We have found we've closed the CVE and we found the proof that a CVE was worth closing in a PDF from 1993. We did a lot of archival work in like Source Forge and Internet Archive and all sorts of things

that some of the people here have never heard of because they've long come and gone, but it's all still there that you can like look at and see where a particular piece of software was vulnerable. So what we did um for Y so in the the fun problem is now YTO inherits all those CVEEs because the the scope of YTO is every false project everything right so

the what these people usually do when they say okay I have this piece of software here and then they see oh there's Apache in there and that's vulnerable to this thing because of the CVE from 1998 any human looking at this says well that's clearly bollocks this has surely been fixed in the 90s and we don't have to worry about it but nobody updated the report that

these automated tools make the comparison against. So everybody has to maintain an ignore list say okay yes this will flag but ignore it we have vetted it to be not applicable to our software um or um and the way we're using it or we already know this is fixed but ignore this for now. So what we've done for YTO is go through over 300 CVE reports and

tried to find can we actually close them so you don't have to maintain the ignore like YTO doesn't have to maintain the ignore list anymore and we've submitted them all to NIST the national institutes of standards of technology in the United States that maintains the central registry and u we have over 60 already applied and then they send us an email by the way thanks for all

this this is great by the way we're hiring a bunch of new people so we can deal with all this. So, we basically did a good spam there. And they we keep getting emails like every couple of months like, "Okay, now here's another batch that we got through. Thanks for those." Um, and the nice thing now is that not only YTO has to maintain an ignore list.

YTO just has the longest one because it inherits everything from every Linux distribution, from every open source project, from every version of every open source project and so on. But um Yubuntu, Debian, FreeBSD, Gen 2, they all have their own agnists. And by fixing the CVE at the central place where they originate, we're making all of those acknowledo. So we did this for YTO, but that helped

all everyone who bundles open source software. So that's very high impact work that I'm very very happy that we could do. directly similarly so this wasn't a project that like we didn't work on the core of YTO because that's not like is now a specialtity but we did something in the periphery of the project that was nevertheless useful um likewise the next thing we did is um

Yakto has a lot of web based tooling that allows them to manage what they do and part of it is um like performance graphs So, so they can see performance regressions over time and fix them. But the usability of those graphs were rather poor because they built it themselves and they're not web developers. Now, we spent comparatively very little time to make it a little bit more

user friendly. It's the kind of work that we're really good at. Um, and basically right after putting this live, they said, "Oh my god, there's a performance regression that we missed because of the poor usability and you fix that for us right away." And then they could work on that. So that's like web development for a like systems integration project is very far away from the core

but has nevertheless um huge impact of the project. Uh next one is systemd um a subcomponent of systemd called systemd resolved d does resolution returns host names into IP addresses. If you do a browser lookup um you can extrapolate from here. It's an extremely essential soft uh service of any Unix system or Linux system in this case. Um and we raised the test coverage from about 3%

to about 65%. Um ignore for a second that system decomponent has this little coverage. It's very good code in general. Um as we found we f we did find a few edge cases and bugs that they fixed real quickly. We learned a lot about how DNS works on on lots of levels. Um but it was a pleasure to work on. um made a real difference for everybody

who's running systemd which is almost everybody um if they like it or not and um yeah helped helped out make a piece of foundational software a lot more reliable by raising the test coverage. Um, another big thing that we did for systemd, which again has nothing to do with the core component, is that of for historical reasons, systemd had two places of documentation. Um, and they have

both grown for a period of over 15 years, and it's very tedious and annoying to find anything out about systemd. You don't know where you need to look first. If somebody's documented there, you think that's it. But maybe it's also documented in the other side, but a lot more thoroughly, and that's what you actually need. But you never look there because you found something on the other

side. But now you don't know what like it's very annoying. I don't have to explain why this is terrible. But it what was terrible is that it's it's a lot of documentation. It's a big like system is long has long evolved and it's a large project. So there's a lots of documentation and both of them has grown mechanically. So we unified this into a single documentation system

by finding um a an information architecture that could encompass both styles of documentation into a single format. There we bu we use pandoc for the translations of these um different like physical formats but we had to write a bunch of plugins to like work around the special cases that they have been using. Um, it's a tremendous amount of work that none of the core contributors ever should

attempt because that's not their core competency. Um, they should keep writing very good C code to make systemd better or review our patches um for the for the test suite, but they shouldn't spend that like like many many weeks of work on like converging to documentation system. So, this is a perfect um type of project that the STA can fund to move the make the whole ecosystem

better. Um, servo, like I said, the only independently funded um, open- source ren web rendering engine. They have a sub project called servo shell for desktop and mobile, specifically Android. Um, which is where you can actually use the web engine, like a little browser with a browser bar and back and forward button, like the usual stuff, but it looks like somebody just threw that together for a

demo. Like doesn't the buttons are way too small. There's no no tool tips. um like the usual afford instance that you have in a properly polished UI are just not there because that was never the goal for this project. But now they are at the point where um they want more people to use servo. It is like while servo isn't finished it is getting better and better

every day and they want more people to use them but people not using it because the shell the browser of it isn't really usable. So we did two things for them. A design brief for here's UX changes that you should be making to make this more approachable. Um and then because the desktop version is undergoing more like a technical architecture change that will make it a lot

easier to contribute to in the future within the time that we had we've had to focus on the mobile version. So we're actually implementing the things that we said you had you should be using. So um I don't know by the end of next week or so um there should be patches to servo on Android that make it look like a normal browser. still doesn't do a

lot but looks a lot more than a technical demo that people who are casually interested will turn off right away. Um, again, yeah, for PHP, I'll bring this back. Um, I already mentioned we did some infrastructure work for them, but the nice thing about that is the same people that got me started in open source 25 years ago was still working on PHP. So, I got to

hang out with my very old friends and do some cool work. It's so nice how this all connects. Um, genome again very cool. It has a feature called blueprint. It's a way to describe user interfaces in a kind of in a like easier text format rather than writing complicated code to make it all work. Um, but blueprint had a lot of important things missing. We added a

good bunch of them. I don't know if we got to all of them eventually, but we did a a good chunk of important things there. Um, so that is easier to build good genome maps without doing tedious programming work. And then we also built the llinter for blueprint and automatic checker that could look at things like oh this button doesn't have a title so somebody with a

screen reader doesn't know what the button does or up in the hierarchy you define a black background and further down in the hierarchy you're using a black text somewhere and you're not seeing it because it's far away from each other. We can automatically check okay no you can't there's not enough contrast black on black text you can't read the text and these kind of affordances allow people

to build better genome maps and more accessible genome maps. So that's again very nice. Um and then again something has nothing to do with the core project. We've done other things for these projects as well that are more c more to the core of them. But the highlights that the team asked me to ref to to refer here is the um revamp of the website and make

like the projects more approachable for new people. These are relatively small projects. So one of the things I want to do is more attractive for new contributors. So we added a contributing guide, a code of conduct, made the websites look nice, did a nice logo, um updated a nice log so people look at these and say, "Oh yeah, this is a cool thing I want to contribute

to rather than, oh, this looks like somebody that did it in the '90s because they did." Um, so and then a personal just a personal thing that came up time and time again in these initial video calls that we had with people like some of them are very skeptical where we had to talk a lot about, okay, here's the kind of stuff that we can do and

then maybe they thought of something. And sometimes they came with a list of things. Hey, can you work on these and these things? But there was always this um when we thought about okay brainstorming what else can we do? Um we told them basically give us any technical debt you have any of these projects that are weighing you down that's in the back of your head that

you always wanted to do and never had any time for that are relatively easy to delegate. And at that point there's a glimmer in the in the contributor's eyes in the maintainer's eyes. They say you can do all that. And they were like oh my god. And then it opened like a the floodgates. Everybody who is an open source maintainer knows the feeling of all the things

that you should could and should be doing in your project. Um and then you suddenly see this lift. They they sit up. They have glitter and glimmer in their eyes. It's just it's an unescribable like feeling of personal relationship. It's really nice to be doing. Tiny humble brag. We did all but one project under or on time. Um if you want to learn how that works, talk

to us later. Um um I'll be rushing through the last bit. so you can get to lunch. Um, we severely lo reduce technical debt in all projects in some form or another. Um, sometimes freeing the time of a contributor is just as important as doing a direct contribution to the core of the project as well, so they can focus on the stuff that they're actually good at.

Um, and making projects more approachable has like compounding effects for future contributors joining the project. In conclusion, we're honored to have done this work. It's extremely good to have been part of the resilience process here in Germany and the EU and strengthening open source projects worldwide. Um, by the way, these are all kind of sort of quotes from the team. Um, we'd love to continue this work

and see it expanded. We already know the sovereign tech agency has been funded for another four years with um a slightly expanded budget. But also we um more people doing this. We need more agencies doing it. We need more um like um yeah globally more governments need to put money that way. Um we need more budgets available. Um there's a talk tomorrow afternoon that you shouldn't miss

by Felix Radar about an initiative to get the STA equivalent for the for the EU going. Um please go and watch that. Um and like you see we we could only do like I'm very proud we did these projects and that it's not not it's not just a few projects. It's a sizable number of projects but also it's just a bunch of projects. We're a small team.

The STA has a small budget. Like there's a lot more open source that needs our help. So I really hope this becomes a model that we can develop forward. Um and yeah, if you have any questions about any of these things, um come talk to me. Um thank you. All right. Thank you, Yan. Uh we'll start off with a question on on from online. Is there any

program to pursue entrepreneurial mentorship for false like tech mentorship? I guess it's more a question for >> I didn't get three of the last words. Is there any program to pursue and to pursue mentorship for foss just like tech women's mentorship? >> Oh um probably but um that this is outside the scope of this talk. So uh like yeah finding finding technical mentorship. I'm sure there's programs

like this but >> we'll ask from the SDA folks. >> Any other questions in the room? >> Thank you for the talk. Um, so if a project wanted to uh get stuff fixed, how could they get in touch with you? >> Right. Um, the projects apply with the STA and say, "Hey, um, we're a project. You have to write up like it's a it's a single form

that takes you about 15 minutes to fill out, say, this is the project that we are. Here's why you we think you're important. Sorry. This is why we think you should fund us, why we're important, and then the STA will take some time to evaluate. if you're an obvious candidate like PHP or Python is basically you just like get passed through if it's if you're a smaller

project there's a little more thorough investigation there um but like for example Popular is basically one guy in Spain that has a bunch of friends that help occasionally it's a lovely project to work with and again if you do anything with PDF and open source including all the open deskes kind of software is using that technology so even if you have a tiny project that you know

has a lot of downstream users you can make your case relatively easily and then the SDA um if you have if you have your own money that happened as well if a project has its own money through some other way but they don't have any people to do the work come to us as well we can help you with that as well >> then thank you Yan

>> thank you