How configuration-as-code eliminates invisible work for open source maintainers

In this talk, Lucas Puringa, a security engineer at the Eclipse Foundation, discusses the hidden challenges faced by open source maintainers, particularly related to security. He explains the responsibilities of the Eclipse Foundation's security team, which include incident response and providing training to maintainers. The speaker highlights several recent security incidents in the software supply chain that underscore the risks faced by open source projects. He introduces Autodoc, a tool developed to streamline security configurations and help maintainers enforce best practices consistently across multiple repositories. This talk emphasizes the importance of securing the software supply chain and the need for efficient management tools to aid open source maintainers in their oversight duties.

[music] Hello, good morning. Um, thanks for coming to my talk, how configuration code eliminates invisible work for open source maintainers. My name is Lucas Puringa. I'm a security engineer at the Eclipse Foundation security team. I would have liked to give this talk together with my friend and colleague Cairo today but unfortunately he was not able to make it to Brussels. So I'll try my best to cover

for him. let's start off with some stretching. Uh quick show of hands who here is maintainer of an open source software project. Okay, a good amount. So uh this talk is about um some of the lesser visible challenges that open source software maintainers uh face. Um so some of you will have firsthand experience with that. Um but I'm sure that everybody will be able to relate in

one way or another. Um this is what we're going to talk about today. Uh first I'll talk about what we do at the security team of the Eclipse Foundation. um set the stage a little bit. Um then I'm going to talk about what maintainers typically do. Um and especially that lesser visible part. Uh spoiler alert, it will be related to security. And um last but not least

uh the solutions that we have come up with at the Eclipse Foundation which we ourselves use um but are available outside of the foundation as well in order to um make some of the that work uh lighter for us and for our committers or for software maintainers in general. Um let's start with the Eclipse Uh we're a small group of um happy people. Um that's us at

our latest team meeting in Milan. Um what we do in a nutshell is uh we try to strengthen the security posture of the Eclipse Foundation open source projects. Um some of our tasks include incident response. So if there's a vulnerability in one of our projects that we steward um we coordinate the responsible disclosure process and um hoke the projects that they fix the things in a timely

manner. We provide trainings to our committers but um they're public to the general. So if you're interested, you can browse the Eclipse homepage and look up the trainings in the security section. And we develop tools to make our own work easier and um all under this mission of strengthening the security posture. Um a focus uh of ours, a particular focus of ours is the software supply chain

and its security. Um I guess everybody has some understanding of what the software supply chain is. Here is one definition. Um it's the set of components, dependencies, tools and processes involved in developing, building and distributing a piece of software. So everything from the inception of a software project to writing its code until shipping it. Um much of the modern software supply chain is powered by GitHub these

days. And to give you a bit of an idea of the scale at which we operate, the Eclipse Foundation has um more than 300 GitHub organizations um under its umbrella um and more than 3,000 GitHub repositories. So the software supply chain is not only interesting to us but also to threat actors or actually that's why it's interesting to us uh because uh attackers have found the software

supply chain in recent years as a means of um really having impactful attacks. Um here's a fine selection of incidents that you may have heard of from the recent past. Um, I'm only skim them uh to give you an idea of what can go wrong. Early last year, there was the TJ actions incident. Um, TJ actions is a very popular GitHub action which um can detect change

uh file changes in your GitHub repository. I think it's used by more than 20,000 GitHub repositories and the action was compromised um for the attackers to harvest secrets. Um in the end only about 200 reposito repositories were affected. It means they ran that action in the time frame while the action was compromised. Uh but yeah still um 200 repositories leaked their secrets because of this compromise. Second

uh incident I want to mention here is the Shy Hulut um campaign which came in two waves late last year. Uh it was a self-replicating MPM worm um that started off with a couple of also to harvest secrets. started off with a couple of compromised npm packages and then self-replicated by harvesting more secrets and whenever it found um on a developer machine or in a CI system

new secrets, new mpm tokens to publish packages, it added new compromised packages. In the end, there were about 800 compromised mpm packages and again more than 20,000 repositories within hours compromised by the attackers. real very um effective attack. And then the most recent uh quite famous compromise was the Trivia in incident uh earlier this year. Um Trivia is a very popular good security scanner and it had

multiple of its uh products compromised. The GitHub action docker images um and uh the binary um compromised again to harvest secrets. Um I didn't found uh definite or confirmed numbers of like the impact. Uh I read somewhere that more than 500,000 secrets were exfiltrated by the threat actors. Um so yeah, that number is not so important. Um what's more important is that these attacks exist and are

very effective, very impactful, have a huge blast radius, they have a lot of cuh cascading effects. So for instance, Trivy led to other um attack campaigns. There was canister worm which was again an MPM worm similar to the shy hula thing. There was the light LLM incident on Python which like was triggered by secrets um become exfiltrated from the trivia campaign. Um so they happen they are

very effective and they share similar patterns. Um they all started by um pawn requests for initial access. Um uh for those of maybe show of hands who has heard of a pawn request. Okay. Okay. So I'll briefly describe what it is. So pawn request it's a a word play of being pawned like being compromised and pull requests on GitHub. The way how you can add code to

a project. uh pawn requests uh exploits an unsafe event trigger in in the GitHub CI where um basically an attacker can run the attacking code untrusted code in the GitHub repository with elevated priv privileges to access GitHub uh to access secrets from that repository and to write to that repository. Um there are reasons why this feature is available. Uh but it's generally uh like you really need

to know if you want to enable the feature and in all these cases that I showed this was the the root cause the entry how a token was stolen. Um that token was then used for lateral movement. It was in all cases uh or yeah I think in all cases the tokens were had fraud permissions. So they were the attackers were able to use the token to

jump from one repo to another one and then steal more tokens and so on and so on. And then finally um mutable tags uh very important for attacker to attackers to distribute their software because in in GitHub um a tag under which an action or software is um released can be mutated mutated. So if something is version version XYZ on one day, it can still be version

XYZ on the next day but have completely different code. So that really helps for distributing malware. so yeah, those are important takeaways for all of these attacks, similar patterns and actually things that can be solved. There's med mitigation for all of them. You can configure GitHub to restrict the event triggers for your CI events. You can configure base permissions. Um, you can protect your branches so that

not everyone can or like that it's harder for someone to change your source code uh without you noticing. Um, and as a user, you should pin your dependency like really pin them not to a branch version uh to a a version tag, but to the commit shar so that you don't pull in something uh that you don't want to pull in. So solutions exist or settings exist

to um to to mitigate those problems. The real challenge for us or in general I would say is to enforce them consistently and at scale. So for instance you administer thousands of repos you want to make sure all of them have safe um safe settings set. let's talk briefly about what a maintainer does. Um, or what you do or some of you do. Uh, at the Eclipse

IP Foundation, a maintainer might work with one of these projects. We have plenty more projects. Um, these are just it's just a selection of popular ones. Um what I'm trying to say a digip foundation or in general open source um maintainer contributor um works on a particular problem domain be it um internet of things uh software defined vehicles um fancy code editors um you want to fix

a certain thing write code for it review code um maintain code foster a community um So that's what I guess most maintainers care about. But then there is this invisible work or lesser visible work which becomes very visible when something goes arai and that is um keeping up with constantly changing uh security best practices um implementing them maintaining them and so on. This is re very time

consuming um cumbersome um even if you only have one repository or one GitHub um and it diverts focus from the project. Um GitHub what adds to this is GitHub uh doesn't have so great security defaults. So you really need to make an effort to turn on the right things. Um but it has some great recommendations. Uh some of them I already already mentioned earlier but so general

GitHub tells you to to make sure to protect your accounts, your code, your builds. Um and you can can do different things um when it comes to your accounts. Make sure to apply principle of least privilege. So your member permissions should only be as permissive as needed. Uh you should enforce two factor uh authentication to make fishing campaigns less effective. Uh when it comes to code um

turn on dependabot that's built into GitHub and make sure that your code stay stays up to date that you get alerted when uh like your dependencies stay up to date and you get alerted when a dependency um becomes vulnerable. You can turn on branch protection which I already talked about. Um you have different tools for code scanning, secret scanning. Some of the c the hack campaigns uh

I talked about earlier dumped the secrets into the GitHub log. Um so GitHub has a feature to detect those um those secrets that get dumped somewhere maybe or if something gets added by a committer uh by accident um yeah turn them on. attackers can work around them, but still try to make it as hard for them as possible. Um, when it comes to builds, you can uh

set um the permissions of your workflows. Same as member permissions, make sure the workflows don't have more permissions than they actually need. And um PRI permissions talks about this pawn request thing, like make sure that you don't uh use features that are very risky. enforcing those settings or setting those settings. Um, uh, it's possible. GitHub provides very good documentation for it. Um, the it usually starts with,

uh, in the upper right corner of GitHub, click on your profile picture, go to settings, yada yada yada. Um, yeah, that that does not scale. So um you can do it maybe in one repository, a couple of repositories, but if you want to like have a plethora of repositories that you want to really make sure to protect or like have a consistent security posture, you need something

else. Enter Autodoc. Uh that's the solution we came up with at the security team and which we use for our projects to administer um those settings as code. Um and yeah, I'll walk you through it a little bit. Um at a bird's eyee view, um Autodoc gives you secure defaults. So all the configurations are as code um and they pull in defaults that uh we maintain. So

everyone using autod dog in that autod dog organization uh gets the security faults provided by like maintained by a security team. Um in our case observability um we have a nice dashboard that shows us all the configuration state at any time. Um we can act upon reconfiguration and manage drift if like sometimes it happens that configurations are not what they are supposed to be. Um so we

see that that gets flagged and uh there is a paper trail of the configuration. So um every change is attested for with a g commit. The configuration files themselves live in GitHub repositories. Um, and yeah, you have g commits, pull requests, and enhanced context on the pull requests for a change. Uh, let me Oh, yeah. So, this is just a mapping of all the best practices that

I talked about earlier. Um, on the left side, you see what you would need to do like the the breadcrumbs where you go to uh change them in the UI. on the right side. Um that's the JSON format what it would look like in code. Um let me walk you through um uh the how how autodook managed to do this. So first of all all the configuration

files are in a JSON like format. We use JSON net which is more powerful than JSON. You can also write templates and so on. Um you can define everything or settings, repo settings, workflow settings and much more in there. It's um lives in the same GitHub uh organization as your project does. Um when you want to make a change to it, um you submit a and go

through a normal pull request review process. Um using pull requests has another uh important or very helpful feature. You have author identity and the whole um author team permission model attached to it. So not anyone can make it but you use the same um permission model or can use a similar permission model as you do for your code. Um you also have timestamps and so pull request

is submitted. um autodoc and autodoc GitHub app bot um will then go and look at the pull requests. It will do some preliminary verification if um if the the the code the configuration changes actually make sense. Um it will add a human readable diff enhance the whole um output a little bit and check if uh the configuration change is eligible for automerging. Um, we block a couple

of things for automerging, things that conflict, but also things that would, uh, request security. Um, uh, so that's what we do in our organization. That's what an Otterd dog administrator would generally do. for other dog administrator. We then provide the CLI where they can go um and look at um pull requests that are not cannot be automerged um and use the CLI to um do the same

thing the GitHub app would do on automerge. And then um this other pillar is the um which gives us a good overview um of the entire autodog organization which spans over multiple organizations um and showing all the managed orgs uh notifying about uh drift and so on. So you can really configure most of the things you can configure in the GitHub Do this on organization level, repository

level, action workflow level. Um these are some important ones, the full references on on the autodog documentation page. Um some of those I already talked about. um especially the security critical ones. But um I want to point out that Autodoc is not only about security settings. You also um we also try to um make it possible to to generally configure the repository like profile description and so

on as code uh something that you would otherwise have to do individually in the UI. So um autod dog is really not only about security but a a broader topic which is governance. So you can tweak the individual descriptions, topics, emails uh and so on. Um you can describe policies about the content like um if discussions are uh enabled and so on private uh vulnerability reporting. So

security relevant themes but which fit into the broader concept of co governance of how your organization is structured how you serve your organization. Um we have capabilities of defining secrets and variables in the config file not in clear text. There are there's integration with secrets providers. Um we have the possibility to create uh blueprints. um like say every new repository that gets created needs a security file

or um certain workflows or a code owners f file and so on and we can also use autodoc to actually create new repositories in the organization or delete old repositories or archive them and the whole uh thing will be uh tracked by this paper trail of changes in your git history in your um full request history on GitHub. Where do you get started? Um for those who

are already um Eclipse Foundation open source projects, uh there's good news. Uh Autodoc should be enabled by default. Um at least for the newer ones. If it isn't, please reach out to us. We're happy to help you set it up. Um for those who are not uh so that's a service that we as the security team provide to our projects um but for those who are not

affiliated with the foundation um it's possible to self-host it um and designate your own security team or whatever administration team. So if you want to learn more um the autodog itself is a open-source project under the git uh under the eclipse foundation under the eclipse foundation. So um we have a GitHub repository. Um we dog food autod dog. We use autod dog for autod dog ourselves. Um

come visit the the GitHub organization. Uh the GitHub repository. Uh open a discussion item. There's another link at the end. Um I will share those slides. Um yeah or stay here after the talk. We still we will have time for um for Q&A as well. reach me reach out to me at the conference. Um there is plenty of things we can do. Um before I conclude, I

want to make a quick shout out to Alpha Omega. Um they have been really supportive uh to the security team at the Eclipse Foundation and Autodog in particular. Um so a lot of this work um was able uh was enabled by them. Um and I want to thank you for coming and um if you have any questions uh I can try to answer them now. Um Mikuel

head of our team is also here. He has more experience with Autodoc than I do. Um, >> peace. [applause] >> GitHub tells us that they are more enterprise friendly these days and that uh some of the things that enterprises are asking for are are now enabled for uh you know common uh governance across repositories. uh what's your experience been with that? >> So I think there are

certain features sets available to with GitHub advanced security for instance um that that helps uh but it's far from where we want to be. So yeah, >> thanks Um, first of all, thank you for Autodoc. This is really helpful for project maintainers to be yeah to configure the project on your own and to be self-empowered and not always have to reach out to to the IT team

and bother them to fine-tune some settings in your repository. Um but um while you can configure your repositories in an organization via autodoc and the code it's still possible at least to some parts to still configure them in the UI and then it happens that the UI or the live settings and the configured settings get out of sync and if you later want to apply a change

uh through the through autodoc you get errors and can't merge them because these live and configured settings are out of sync. Is there a possibility to generate a change that syncs the configured settings according to the live settings back so that you can review them and yeah sync them without finding it out manually or figuring it out manually that for example a repository description that is configured

through the UI now matches what is uh written in the in the Yeah, that's so generally that's how uh autodoc works that it applies the settings as it would do as as you would do via the UI. Uh but there is always the problem of configuration drift that you um change something and autodoc does not immediately uh recognize it when you change it over the UI. But

that's actually a good point. So for everyone who's using Autodog um it's um um recommended to not do those changes over the UI but uh try to really stick to the configuration files to um avoid those kind of conflicts. >> We we tried but sometimes may I mean we have a large group of committers not maybe not every committer is aware of of this recommendation recommendation. So

>> yeah, >> it happens sometimes and it would be nice to have some simpler way to sync the the configured settings again with live setting or had have at least a proposal a PR generated for it. It that would be sometimes helpful. >> Um you want to add something to that MA? >> So thank you for the nice comment. Um it was an issue that when the

the configuration was drift the the configuration has code drift from the the the live settings uh that you were not able to merge immediately. So this has been resolved now because the the merge actually only applies uh the changes that come from the PR. So even if you're out of sync we don't overwrite everything. We just apply the changes that your PR actually creates. So that's one

less uh issue. Um but for sure that we actually have a feature request about the this specific feature to have a PR automatically created from the drift so that we can reconcile both the live settings and the the PR. But the the priority was to unlock and to prevent errors when the the settings were out of >> Thanks Hi. Uh just a questions about the uh overall

direction where the Eclipse Foundation is going. I see an increased focus on GitHub and GitHub actions. What about Jenkins? Uh part of our infrastructure is yeah, we have half of our automation on GitHub GitHub actions and half of that on Jenkins. There's I I I can observe the fact that you didn't build something that anog for Jenkins. So yeah. >> Yeah, that's a that's a great question.

Um Autodoc [clears throat] currently only works for GitHub. I think there have been plans to um have something for the other CI platforms that we have in the Eclipse Foundation, but um I I don't know of any actual work being done at the moment about that. Yeah, there was a a reply to that comment there. >> I think there is something there. [laughter] Sorry, I think there's

something similar. It's called gyro or g J girro uh where the but it's only the TTM has right access to the repository. It's in Eclipse CBI organization/zero and there are all the Jenkins instances managed but you can create PRs to it as a project but you cannot uh submit them on your own. So somebody from the IT team has to review it and apply it and deploy

it live to production. But this already exists. So I mean it would be great uh if uh some at least some parts of this would be manageable directly for project teams as well. So maybe this could also be a follow-up request. But uh I think in general these uh things exist um you mentioned uh that uh you meant uh to use commit hashes um instead of tags.

Um, as far as I know, you can also create tags with the same um characters like a commit hash. And Git would prefer to use the tag instead of the commit. I I haven't heard of that. Um, that's new to me. >> Okay. Ju just to add something to the question the earlier question about the genkins kit configuration and and GitHub so yeah we've been focusing on

GitHub because most of the project actually use GitHub and GitHub actions and so on the so genkins is not going away but one thing that we are um aware of and we understand that most projects stays with Jenkins because they use the code signing infrastructure at the foundation that is only available from Genkins. Um, and we are uh in the process of offering the same features that

for GitHub from GitHub actions. So projects will be able to cuts sign their Windows.exe.msi and uh to cuts sign and um notoriize their Mac OS application also from GitHub actions. So that that will be one way. Now we are focusing on that as a new offering because that's what we hear more the most. Um and uh as it was mentioned for for genkins configuration that there is

already a solution for ascod uh one [clears throat] comment to that last comment. So um when pinning to a commit sha you um are actually supposed to look at at that sha like to to make sure that you're pinning to the right thing and at that time you should be able to distinguish a tag from a commit. Um there is tooling for it. Uh we recommend a

tool um I forgot the name now. Is it pin pin act? Yeah it's pinct. I I don't know if it has a check uh to see if a tag is actually like I would suppose that it doesn't pin to a tag that just looks like a commit hash. So that would be a a way to um mitigate that problem. But thanks for the for the input. >>

All right. Thank you. >> There's one more question over there. Sorry. And this is a question about AI actually. Do do you have in your blueprints or templates some uh things to frame the use of AI like agents.m MD or no agents.mmd or things like that? >> Um you mean like um suggestions for that for AI? I don't think we have anything like that at the moment,

but that's something that could be could be definitely added um to other dog documentation for instance. >> Yeah, some templates for you know the project not to have to write them by themselves if there are good best practices to be shared maybe through autodog that could be a way to >> uh we have those for other things but not for AI specifically at the moment. >> Thank

you. Thanks. All right. Thank you everyone. >> [music]