Miaolai Zhou – The Power of Dedicated Security Engineers vs. Volunteers #FOSSBack

In this talk, Mila discusses the critical but often overlooked aspect of open source: security. She highlights the invisibility of security in open source projects and the challenges faced by volunteer maintainers who are often overwhelmed. Mila presents the idea of funding dedicated security engineers to improve the situation, using examples from AWS's commitment to invest over $10 million in open source security through initiatives like Alpha Omega. She emphasizes that the Log4j incident served as a wake-up call, exposing the vulnerabilities in critical infrastructure heavily reliant on volunteer effort. As a solution, she explains how hiring dedicated security engineers significantly boosts productivity and enhances the overall security landscape of open source projects. The talk concludes with a call to action for companies to invest in open source security as a vital part of their infrastructure.

Hello everyone. Really appreciate your time. It's early in the morning. We're all waking up. So, no pressure. Like, I probably am saying things I don't know. So, like, bear with me also. Uh I am Mila. I work at AWS as a open source program manager. And today what I want to talk about actually is something that's essential. However, it has been invisible for a while in open

source. That's security. Well, we need to get excited about security because we don't have enough excitement. That's why we're not investing on that. Because, like, perfect security actually works like a transparent umbrella. It showed you perfectly under storm. However, like, you don't realize it. You don't see it. And the invisibility created an issue because then you don't invest in it. uh that's how open source security has

been worked for a long time. Maybe still in some projects that are same. Like, maintainers, on top of every work they're doing, they still do open source security. And that cannot be done in like immediately or with focus. They can only like compete against other priorities. And today we're talking about uh what will change when we have an open source ecosystem that has dedicated security engineer instead

of relying on volunteers. I have been leading a uh AWS open source security funding in the past 3 years. In 2022, AWS committed that we are going to invest over 10 million in open source And a large chunk of the money actually goes to Alpha Omega. Alpha Omega is a security fund is a funding dedicated to open source security and it's also formed in 2022. So, why

2022? What's so special about 2022? Log4j? Oh, yeah. Of course, Log4j. So, in November 2021 when everyone think about holiday Log4j maintainer actually got a bombshell in their inbox. Uh, within few days limited exploitation just spotted everywhere and in the middle of December when everyone opening their holiday gifts the maintainers actually open their laptop continuously to work a patch after patch after patch. And what makes it

more complex and difficult is that Log4j is not just within software but it's also within like layers of dependencies of other software and the cleanup takes months or That adds on the complexity of fix and that also adds on the time of cleanup. That's why in late 2025 about 13% of Log4j download still use the version that have vulnerable. Well, Log4j is not just a technical failure.

It's actually a human one. Because we are looking at volunteers to drop everything stay away from their family during holiday and to just stop a global security wildfire. That will benefit everyone like everyone using their software, every companies that monetize their software. However, the maintainers are the one bear all the cost and criticism. That's the reality of a volunteer story. It often starts with incredible passion. You

just want to do it. You think it's incredible. However, it ends with burnout. And XZ backdoor incident is another example of the security issue is not about the vulnerability, but it's about the maintainer. People who support open source critical infrastructure, they are often the they are often those who are not paid, who are under-sourced, and who are exhausted. So, burnout is not a theory, but a reality.

And however, when critical infrastructure depends on volunteers working nights and weekends, that's not sustainable. That's a fragile. And Log4j actually is a wake-up call to the industry and government. Like industry after that start to invest more on open source. And one of the efforts is to fund dedicated security engineer for projects. For example, in 2023, Alpha Omega funded the Python Software Foundation to hire Seth as the

security engineer in residence. And at the same time, in AWS initiated the funding to hire Mike Fiedler for PyPI security and safety. And later, the funding to uh Mike Fiedler was moved to Alpha Omega, too. before that, security in Python ecosystem used to look like something that a vulnerability report arrived to the maintainers inbox. However, the main the maintainer usually is a volunteer. They have their daytime

job. They have their pull requests to review. They have a family to feed, a community to to support. So, security work will only be done when they have time. It's usually nights. It's usually weekends. It's usually among priorities. So, the response is also reactive. Someone report a vulnerability and then team will work very diligently to fix the issue. The dedication there is incredible. However, the challenge is

also very obvious and simple. Security is competing with everything else that requires the project to run. That obviously limits us for what we can do within security. we need to acknowledge security often time actually is a long-term coordination work among different volunteers usually doesn't have the time to stay on top of that for months. And secondly, more than software is not just a project. It has layers

of dependencies. It's so interconnected. to really understand the security risk there, it requires deep analysis. And that demands consistent and continuous time investment. Third, there are a lot of invisible work, but they are essential, like documentation, like advocacy. Those work usually are the hardest to be to be prioritized. So, when we shift a security staff model, a very encouraging observation is that we have way more output.

Like, obviously, volunteers are incredible, but their constraint is also obvious. They don't have that much time. So, in my conversation with Seth, he mentioned that usually a volunteer can complete one major uh deliver per year. However, a dedicated security engineer can do three to four. Well, let's just look at how much Seth achieved within the first three months. Uh in the first three months of Seth's time

at PyPI, he the Py- PyPI became a CVE number numbering authority. That means less delay in author- in vulnerability reporting. And also, he helped integrating the PyPI vulnerability advisory into open source vulnerability database. All those improvements are fundamental. That requires focus, persistence, continuous engagement with the community. Mike Finneran also is very effective in improving the security within PyPI. One of the most impactful change actually is the

two-factor authentication for maintainers. That greatly reduce the risk of account takeover. And at the same time, he helped to build better infrastructure for detecting and reporting malicious packages. That's incredibly important for a growing ecosystem like PyPI. Actually, I want to talk about 2FA more because it's a great example to understand the volunteer gap. PyPI actually introduced support for two-factor authentication in 2019. But for years, the adoption

barely moved. That's because security is not just about building the feature. It's more about persuade the whole community to adopt into that, to take action, to make a change. So, adoption requires continuous advocacy and engagement with the community and collaborating with thousands of developers. So, when Mike take this as his full-time job, he built phased enforcement. He wrote posts about the plan. He participated in podcast to

make the whole plan obvious and clear to the community. And he also built the email campaign to notify people who have not adopted. As a result, by August 2024, 80% of users who have been logged in since January 2024, they have enabled 2FA. Another example for us to think deeper about what a dedicated secure security engineer can do is SBOM. SBOM help us to understand what's within

a software. That's the supply chain. It's complex. Well, for it to really work, it requires someone to dig deep into the question. You need to really understand how packaging works, how security data flows, and how we can build a solution that's realistically possible to Python ecosystem. And that's exactly what Zach did. He dived deep into the topic. He did his research, engaged with the community, and participate

in the work group. Which often help which the work group conversation often happens during work hour. So, Seth mentioned that if he's not hired, he will not participate in work group at all because you cannot afford the time. and all those allow him to transfer the idea into a concrete plan that adopted by the ecosystem. that kind of deep and consistent investment is difficult for volunteer just

because of the time constraint they have. But when you make it as someone's job, the progress is possible. but like I don't think a dedicated security engineer will replace volunteers. Like open source thrive because of volunteers help. And the security engineer are there to enable They create process, templates, and guidance that make make it easy for volunteers to fix the issue. And previously, the Python security response

team was a trusted cable in the private mailing list with no clear path to onboard. Well, because of no structure there, the same group of people they work on everything. They triage, patch, coordinate, and disclose. To improve that situation, Seth drove the PEP 811. That formed how to join the team and providing a very clear path for new volunteers to become a board to become a part

of it to contribute their expertise. And we're already seeing results of The Python security response team is growing. Ever since August last year, they have onboarded four more uh security engineer volunteers to the team. And at the same time, the whole process becomes more transparent and the ecosystem becomes more resilient. So, we are at the end of this talk. I want to leave you with two simple

ideas. First of all, funding dedicated security engineer, it works. They fill the real gap in open source security and ecosystem and they drive measurable results. And second, we If your company depends on open source, which is almost everyone, um investing on open source security is not charity. It's investing on in your infrastructure on your infrastructure. So, fund security engineer, support the ecosystem, and be prepared before the

next crisis happen. That's the end of my talk. Thank you for being here. Hopefully, this talk doesn't make you feel more sleepy, but wake you And I'm ready for questions. Yeah, so what are some qualifications that you would look for if you want to hire a security engineer? Like at Alpha Omega, we don't we don't directly funding security engineer. We fund for the result of the work.

So, for example, uh when we talk with Python Software Foundation, we're not saying that, "Oh, we just need a a security engineer." But we talk about what the work they're going to do. What kind of result they are going to drive. Yeah. I'm looking at the GitHub monthly progress and you funded Open Refactory about 180,000 US dollars last year. Their sole report is a read me. I'm

sorry. Their only report is a read me. There's no report at all for 1 year. So, if you are funding by results, does that mean you didn't pay them? We paid them. that's some lingering process issue that we will want to fix. Like, yeah, we are not we definitely ask people to update their progress. We think it's important. Well, like we also don't force people to do

that if they don't do it like a very proper job. But, our logic there is make me want to fund you continuously. If you are not like meeting the expectation, then maybe we're not going to consider to fund you again. Uh thank you for your talk. That was nice. Um I wanted to ask uh within your role at Amazon, when you've evaluated the results of what you've

done here, uh how do you want to take that forward? Do you have plans to expand to fund security in other ecosystems and so on? We definitely look at um like the business coordination. Like, the way I look at it um we are external facing, but at the same time we are internal driven. That means like we collaborate closely with our service team to understand Okay, where

the dependency is. Like, how the investment can improve their security position or relate to their work. Um the more support we can get from internal service team to champion the work, the easier for us to get budget. Like, ultimately, it's a business justification justification question. Like, can I persuade my finance manager and the business owner saying the investment makes sense. You um when you make security your

full-time job, um job security becomes very important. How do you deal with that? So, is it usually 1 year the the planning stage and is it difficult to find people who are willing to apply for a position if they know it's only going to be for 1 year or could it turn into something permanent? I guess I don't think I'm the best person to answer this question

honestly because uh like if for example with the Python Software Foundation, we've we are funding them, but they are the one getting the money and hiring the people. So, I guess they have a better answer there. That'd be easier. We've got more questions. Hey, um tying to his question or in your answer and considering that Log4j was the jump-start for this investment, um does Amazon know like

how much Amazon lost in or lose in a problem or lost on on the Log4j issue? I don't know, but I guess I guess CISO knows, but definitely it's not a public report or things like that. well, I think I actually really want to see number because that tells us how much it cost us to fix the issue and comparing to the investment to prevent it from

happening. Like definitely it gives people it's a great business justification. They're like the number makes sense. Yeah. Big round of applause from you all, I please. Thank you, everyone. >> Thank you, everyone.