The power of dedicated open source security engineers vs. volunteers

This talk addresses the critical issue of funding open-source security efforts, illustrated through the narrative of an umbrella, a plumber, and two security engineers. The speaker emphasizes that security often receives budget cuts when it is effective and goes unnoticed, leading to a reactive approach where funding only becomes available after a crisis, such as the Log4j vulnerability. The session highlights the formation of Alpha Omega, a project aimed at securing essential open-source libraries, and discusses the measurable improvements in security that result from dedicated funding and personnel. The speaker shares case studies from the Python ecosystem, showing how transitioning from volunteer to full-time security staff significantly enhances response times and overall security measures. This shift not only enables quicker vulnerability remediation but also fosters a more proactive security culture within the community.

[music] >> And thank you for being here. I'm here actually as a Renegade to talk about a story, tell you a story about an umbrella, a plumber, two security engineers who named as Mike and Seth. And by the end, I hope that we can walk away rethinking about what does it cost to make open-source software we like we rely on to to be safe, and who is

paying the cost? think about perfect security like a invisible umbrella. When it works, you don't notice the storm. And if you don't think about the storm, you probably don't think about why you should have an umbrella. And that's the strategic problem with open-source security. The better it works, the less it feels like it matters. And so, when the budget get tight, when the road map become busy,

they're the first thing to be cut. And hold this in your head, the transparent umbrella, because in the next 35 minutes, we are going to see through it. Well, a very quick show of hands. How many of you has hired a plumber for your regular maintenance? No one? Doesn't anyone have a house? >> [laughter] [gasps] >> But then, how many of you have called a plumber out

of crisis because something burst? Also no one. Someone. Okay. [gasps] Otherwise, I'd like you're so lucky. >> So, that's the pattern you see. Like, a lot of time we don't pay for prevention, but we pay for emergencies. And even though emergencies sometimes cost us 10 times, 20 times, and even 50 times of the cost, but we still lose the floor at the end of the crisis. And

open-source security follows exactly the same pattern. We don't fund it when it works, when the pipes are quiet. We only fund it after the flood. And in this case, the flood has a name, Log4j. So, Log4j was disclosed in December 2022. It's 2026 now. It's almost 4 and 1/2 year later. However, we still see about 13% of download are for download of Log4j are exploited version. And

that's astonishing. That means the blood is still seeping somewhere that we don't see. Every download is a project. Is a sysadmin job. Is a Docker image that's choosing the wrong version. And the total cost of our companies and governments to remediate Log4j is in the billions if you count into the instant response, the emergency added, the engineering hour, and the regulatory work. And that's only the work

that we can measure. And but what would we have paid if we hired someone at the beginning to maintain the library, to make it safe? Here is the actual thought process behind most organizations. That security is a cost until it's a crisis. And only then security becomes the most important thing in this world. The irony is that prevention usually costs a lot less than paying for fix

the crisis. And like the plumber, but prevention is invisible. And the invisible work is hard to get funding. What I want to show you today is when that cycle is broken, when some organization are interested to fund the boring and invisible work to prevent crisis from happening, to protect the ecosystem. So, Alpha Omega was formed in in 2022. It's a dedicated funding to ensure critical open-source projects

that we silently depend on are secure. And in the same year, AWS pledged 10 million to open-source security. And a lot of them funded through Alpha Omega. And so, by the end of 2022, we actually have real money and real mission to work on. Alpha Omega's slogan is simple, turning money into security. And it is genuinely not that obvious to see the to see the conversion works.

For years, it was a only open question that if we put more money into an open-source project, does it make it safer? So, the research that has come out of recent years study, including academia research about funded and unfunded project, shown that the funding is correlated with a measurable improvement in security. It can means faster vulnerability response, more consistent maintenance, and better handling of crisis issues. That's

exactly what we are going to talk about today. So, Alpha Omega, we have expect we have experimented various work across four categories. The first one is to staff humans to work on securities for critical open-source projects. The second is secure open-source repositories like PyPI and NPM. The third is security auditing. And security audits usually are the bread and butter that we often engage. Start from security auditing

and the following remediation. The last part is Alpha Omega actually start with the spirit of experiment, because we know a lot of part of open-source security is a book unwritten. And there is a lot of hard and questions that need to be solved. All those four category matters, and all those four they actually feed each other. However, today I want to zoom into a small category. Let's

take a very close look at security staffing funding, because it lets us to see to answer very tangible, a very measurable question. What change when a security volunteer becomes a security employee? Well, the case study I'm going to bring about Python Software Foundation. Why Python? Because it's everywhere. It's in every ML stack. It's in every data pipeline. It's in every developer script. And that's almost certain, if

Python has a problem, everyone has a problem. And because it's so widely used, we can see how m- much a change it can lead when the security is funded. Meet our two main characters first. Uh Mike Fielder. He is currently PyPI CPN security engineer. He was a volunteer in uh ecosystem for years. And before after that, in 2023, AWS initiated the funding to hire him uh through

Python Software Foundation. And later, the funding was moved to Alpha Omega. The same as uh Seth Larson. He is the security developer in residence for Python. And he was funded by Alpha Omega in 2023. Seth has also worked in Python for many years as a volunteer for its security work. Both of them has done security work as a volunteer for Python. And both of them are now

doing it as a full-time job. So, the nature before and after is the experiment that we are going to look at. Here is the security in Python ecosystem used to look alike. So, a vulnerability report arrives in the maintainer's inbox. But the maintainer is is a volunteer. They have a daytime. They have the daytime job. They have pull requests to review. They have a They have a

family to take care of and kids to raise. So, security work only happen when they fit in. And it's usually around me time, before between other priorities. And the response is reactive. Like, some reports Some more reports and issue, the project scrambles, the fix goes out, and the life resumes. Well, here definitely is not a criticism on volunteer, but like their dedication incredible. However, the problem is

structural. Security is competing with every other thing it takes to run a project. And when security competes with everything else. Security loses licenses. Let's talk about the work that people So, PyPI gets around 300 malware per month. That's 300 every month. Before Mike report can come in see there for days before someone have a time to take a look at it, and a few more days to

get a fix. And once Mike was full-time, the reporting handling dropped from days to under a minute. And the solution delivering time dropped from days to hours. That's not a incremental improvement. This a completely different operational model. Because Mike actually built a quarantine feature in PyPI that allow admin to just quarantine a project when they identify some fishy activities, and they can mark the project as potentially

harmful. That doesn't allow people to easily install that, and later they can investigate further and bring it back home if that's uh found out as a false alarm. So, among all major uh package managers, PyPI is the only one that has the capacity now. Well, let's also look at some simple number here. As a volunteer, each of these engineers, they usually land about one major delivery every

year. And as a full-time, they can ship three to four major deliverables per year. And the core here is not just about more output, but the output that shape the position. Because security work compounds when when it has continuity. So, the same person, the same skill, the same ecosystem, but different funding I think two-factors actually is a uh great example for the volunteer gap. PyPI introduced 2FA

support in 2019. Build it, ship it, but like in the following few years, the adoption barely moved. And because the problem here is not about the feature. The core problem here is really to persuade the entire global ecosystem, tens of thousands developers to turn it on. The adoption work uh requires code ordination and advisory uh advocacy. It's the unglamorous repetitive work of nudging, explaining, and nudging again.

So, when Mike take it full-time, he built a phase enforcement, and he wrote a blog post to explain uh the change. He went to He went to podcast to make the change obvious and easy. He also ran a email campaign to users who haven't adopted it. So, as a result, today every package published on PyPI comes from a an account with 2FA enabled. The ecosystem structurally more

safe comparing to 2022. Well, another example is SBOM. SBOM helps us to understand what's inside of the software where that's running, and the whole supply chain uh behind the package. And enterprises, governments are going to ask for them as the regulation becomes effective. However, making soft bomb uh SBOM works in PyPI in Python actually is not that easy, because it takes someone who really understands the package,

understand the security data flows, and understand how a solution can possibly, realistically fit into everyone's culture. And that also requires uh to work with developers to persuade them to adopt it. That's what Seth has done. Through deep focus, monthly long work, the PEP 770 was published. And honestly, that kind of work, you just not can You just can't expect it from a volunteer. Not because they don't

have the skill, but because the bottleneck here is the time and attention that a volunteer cannot afford to put in. Well, here is my favorite example. PyPI doesn't allowed obfuscated code according to its uh acceptable use policy. However, it never enforced it because it didn't have the tool to detect to detect it. And Mike had built a detector to catch that kind of code following certain pattern,

and along the way he discovered that someone has been posting uh code to PyPI for years as a free resource. They took PyPI as a sales funnel to for their own paid package. And they kind of used PyPI to subsidize their own business. Mike found there and gave them a few min a few months to migrate from there. And the side business actually very fast fade away,

because now they have to start to pay their own infrastructure that they used to get free load from PyPI. And that kind of incident, we only find it when we have someone work on it full-time. That we need a pair of eye that look at PyPI full-time week to week, and after years we started to identify those slow drifts. A very recent example, the instant report for

the light LLM and Telnet supply chain attacks published on PyPI blog in this April. If you read a blog, you will find that you will see the shape of what dedicated to respond to a crisis. They investigated. They support the maintainers, and they write public guidance to mitigate it. At the same time, they collaborate across the ecosystem. They're like the telephone pole that connect every stakeholder. And

this kind of cross ecosystem requires someone who is available, credential, and trusted in the room. So, in the moment when the crisis is live, we have someone to call to. You know that you can get help. And that's not something you can ask At the same time, I want to push back a little bit about something I hear a lot, that people sometimes worry that paid engineers

will crowd out volunteers. However, the data from Python Software Foundation shows the opposite. Dedicated security engineer actually empower the the volunteer communities. Before the Python Security Response Team, they don't have a very clear onboarding path. And put it in Seth's word, they're like a trusted cable on the Python mailing list. So, the same few people, they work on triage, coordination, patching, and disclosure. And Seth drove PEP

811, which finalizes how to join Python Soft uh Python Security Response Team, and since the PEP landed in August last year, they have already had four new members join them. You see, that's the compounding effect that you get from a dedicated security engineer. The dedicated security engineer, they write the playbook, and then more volunteers can join the community contribute meaningfully, because the ladder now is visible. This

is a quote from a blog published by Apache yesterday. It's about log for shell and what it takes to build a CRA ready project. I love this quote because it names something that um we tiptoe around. So, it says, "Shifting to a model where maintainers review all contributions and CI builds everything is the right move for But, it is genuinely less fun, and communities resist it." Security

work is like plumbing. careful reviews. It's signing. It's incident response at 2:00 a.m. It's turning every contribution into a PR, a built artifact that people trust. That kind of work is not fun. That's different from the joyful work that attract most people come to security comes to open source. And dedicated security engineer, they absorb the unfun. They protect the joy that keeps volunteers showing up. And that's

another reason this investment actually let me close. Open source is everywhere. Python is in every ML stack. Log4j is in every Java app. And NPM is behind every front end. The umbrella is huge. And all of us stand under it. And very few people are paying to keep it repaired. Alpha Omega and AWS showed us what happens when you actually do. There are three things that I

hope that we can walk away think about more. First is security is a strategy, it's not a cost. Second, the continuity in security investment really matters. An engineer in a seat every Monday will always work better when you only have a volunteer scramble at night. The last, finding the plumber before the ceiling falls, not after. Thank you. I know that we have plenty of time, so like

uh but we can open for questions. Any questions? We can close this session. This room early today then. Perfect. >> [music]