Amir Montazery – Success Stories in Open Source: Security Audits with OSTIF #FOSSBack

This talk explores success stories in open source, with a particular focus on the importance of security audits for open source projects. The speaker, Amir Montazeri, managing director of the Open Source Technology Improvement Fund (OSTIF), discusses how security audits serve as a best practice that not only identifies vulnerabilities but also improves the project's overall security posture. The session highlights the collaborative nature of these audits, involving maintainers and various stakeholders, and emphasizes the wealth of benefits derived from thorough security reviews. Specific success stories from recent audits of critical open-source projects such as KIA and STOR are shared, showcasing the positive impacts of collaboration and the implementation of effective security measures. The talk concludes with recommendations for dedicating resources and working with knowledgeable security teams to enhance the security landscape in open-source development.

Hello, good morning everybody. Uh, thank you so much for being here. I'm very excited to talk with you all today about success stories in open source. And this is worded intentionally because we're surrounded with a world full of negativity and headline, you know, headlines and FUD as it's been called as well. And um in our spaces too that that tends to be pretty common. um we're always

talking about XZ and you know all of the uh the the large scale uh issues that have come up and rarely talk about the success stories and some of the good things happening. So I wanted to talk about that today and especially in the spirit of this conference being about not necessarily specifically focused on code contributions um the kind of work that we've been doing with the

facilitation and management and execution of security audits for open-source projects is a nice um overlap and looking forward to talking with you all about that today and yes as Ben mentioned I'm Amir Montazeri, managing director of Ostiff. That is the open-source technology improvement fund. We're a small nonprofit that again that is our sole mission which is improving critical open-source software projects. So to kind of go over

some of the main points um that we'll be talking about today, uh first we'll talk about the security audits themselves and and talk about it as a best practice and why it's so important and then we'll take that and apply it to open source projects and um the nuances that come with working with open source projects and their communities. And then uh in the spirit of open

source definitely want to talk about collaboration and all of the work that goes into doing uh the these types of engagements. And again as the title suggests share some success stories. And lastly because we have been doing this for quite some time and have a large body of work of security auditing that we've done for open source projects. I would love to talk about some of the

benefits we've seen uh recommendations for projects and communities as well as some insight into this kind of work. So to uh I guess no pun intended dive right into it. Um security audits and just the general concept of independent third-party expert review is a very common best practice and something that has is both common in software development and in technology spaces but really many spaces as well.

you think of things like home inspections and um due diligence, things like that. Having that third party independent expert come in to apply their expertise to a particular environment and uh be able to to make help make changes. And while finding vulnerabilities isn't really isn't the only um kind of um isn't the only mission of of doing these security audits. Uh one huge benefit is that uh

the security audits do tend to find uh vulnerabilities and classes of bugs that can really be uh cause issues. I like to refer to this research uh paper titled 0 days, thousands of nights, the life and times of vulnerabilities and their exploits, which did a lot of really fantastic research on how vulnerabilities are found in software, how they're fixed and remediated, and found that especially zero day

vulnerabilities and deep-seated problems in code live a long time they're they're in there and um to fix them or to address them typically requires more in-depth auditing, logic review, and source code analysis in order to go several layers deep. So, I use this image of the mine as an example where as is um the goal in mining, you know, to find to find valuable gems or materials,

it's typically not going to happen during those first few layers that um that you dig down. And it's usually in those very last few that you will typically have those findings. And that's actually very common in our security audit exercises and engagements we've been doing in that a lot of the um really meets or the the findings um and issues that are found typically happen towards the

end of the audit once the auditors and audit team have had the opportunity to dig deep understand the software and and look for issues. So security audits we've seen have really been an excellent exercise tool for open-source projects to improve their security posture. So now that we've established that security audits are an important practice and tool. Yeah, I know it's it's definitely a wordy slide this one.

I won't be going through all of it uh word for word but the the point is that at open source technology improvement fund we took that idea of doing security audits and through years of trial and error and um really just doing bootstrapping this this concept um have finally refined an approach to apply this best practice to open-source Now, not to go over all of it um

in detail, but the points in particular that apply very much to open source projects in their communities would be steps one and two where because we have typically different parties uh involved. It's not like a software project within a company where everyone works for the same company. Um, we really need to do our diligence in understanding what the project's needs are and connecting with maintainers to again

understand where those gaps are and how an engagement like a security audit could fill those gaps and and um improve the project. And then um steps um eight and nine also uh doing more comprehensive tooling reviews as part of the audit um has been very helpful because it addresses the the idea that security audits aren't always just a point in time exercise where when we develop tooling

review and develop tooling for projects as part of these audits we are able able to um improve and develop testing suites for them. And then number uh step nine as well. So our process is very focused on actually making improvements and fixes not just finding problems. Um a very common I would say um conception of of these types of exercises are oh well we already have enough

bug reports that we need to deal with or um any you know especially nowadays anyone can run some tools to and generate bug reports. uh it's really about getting that highest uh signal that we can possibly achieve and then actually producing and testing fixes on those on those issues. So then we've applied again taken this best practice of security audit engagements applied it to the nuances of

open-source and working with diverse decentralized communities typically with all sorts of different governance structures and and lack thereof. And um so we really hone this best practice to So this requires a significant amount of work and collaboration and I'm very grateful to have collaborated with a lot of different open-source communities uh very fantastic projects and the maintainers who keep those projects going as well as organizations foundations

and security firms and researchers um and really all sorts of collaborators in the case has made this work more impactful. Um, so I just I'm one I'm very grateful for it and two I recognize the importance of collaboration in doing this work. And so all of our audits uh and and the reports that are generated at the as a result of the audit are available for free

on our website on our GitHub. Um they're typically posted in a lot of different um medium uh uh media, but uh we also keep a repository of all of the work that we've done. And um it is again freely available to review. And this is just a snapshot of some recent um audits that have been done just to show that this is a very diverse and deep

body of work um that we have put together. And going back to kind of a what success looks like or success stories, we typically as I mentioned are very improvement focused and holistic improvement focused. So a lot of our audit reports will also have um fixed verification, fixed review and fixed verification. And we make a point to um to point that out in audit reports that we're

not just finding more issues and more work for maintainers to do. We're actually working with maintainers to fix problems and and to fix issues. So this is just an example of of uh of a of what we would consider a successful engagement where we were able to among many other things find and fix these type these problems and vulnerabilities. Oh, I'm going quickly. And so, yeah, and

now to give a couple of more specific examples, uh, because I I thought about it and, uh, realized if we're going to talk about success stories, um, we should actually have some examples. So, um, the KIA and STOR projects um were audited last year and, um, we, uh, just recently put out the results for that. This was a result of a almost 2yearlong um engagement starting with

getting to know the ISC the internet systems consortium which funded this work um getting to know them their points of contact working with them to then engage the Kia and Stor communities and um to work with them to put together a request for proposal that then went out to a wide range of of verified um open source security experts and um as a result um we did

these two audits for the KIA and STOR projects which are very uh considered kind of backbone projects for um in some ways the internet and how it works. So this was a a really fantastic collaboration. Um working with ISC was fantastic and being able to again make improvements to some really important open- source software projects um was a fantastic experience. Uh some other success stories. So uh

the CNCF or cloudnative computing foundation stewards a lot of different open- source projects uh under its umbrella. And one thing that works well for them is they have pretty strong governance over their projects and how they develop and grow and mature over time. And part of that includes a a maturity model or a maturity process and security audits and getting that kind of uh third party verification

or review is baked into the process of moving up uh to reach what they call graduated status. And um so CNCF has been working with us over the last I think 5 years now to really manage that whole program and uh fulfill these security audits for these projects. And so um in 20 and uh in 2025 as we do have been doing over the last few years

put out impact reports that kind of summarize all of this. And uh I thought I would take a few out of th out of those out of that report to again share some success stories. Um so last year we did a number of projects. I think it was something like 12 um audits for CNCF as part of this program. And um again it's very common to have

findings to find and fix vulnerabilities um but also recommend um different uh approaches and things that can be done to further harden the project and make it more robust as well as work on uh documentation for example threat modeling and publishing those threat models as part of these audits. Couple more. Again, these were from 2025. Um, just to to iterate that this isn't um something that just

happens to a few projects. It's something that really can be uh with the right resources uh management and administration scaled to on a much larger level to to do a lot of to do a lot of audits. But we did yes uh last year we did some work on linkerd and the carmatada to to share a couple more success stories. Um uh OSIF has been working with

the sovereign tech agency for the last few years as an implementation partner for its bug resilience program and fulfilling security uh engagements for again critical infrastructure. Um and so we've been doing that for the last few years and so tech agency has been fantastic in funding this work and uh being very open to to funding the work that we've been doing and um last year as a

result of this collaboration we put out a couple of audits as well. GNU lib micro httpd2 which we internally labeled as libby because we got very tired of saying all of those letters every time. Um but had a a good amount of um did a fantastic collaboration there in which in addition to um finding and fixing vulnerabilities, custom fuzzing was developed for the project to um to

again to continually test that code and uh keep it robust. And then with uh logback very similar uh experience um in addition to helping them develop and publish a custom threat model as well as look at the supply chain security as well uh which is something we've been doing as a part of our process um to again to really holistically improve uh improve these Uh these are

just a couple of more that we did uh last year. Um couple of Apache projects. Ruby um the Ruby uh audit in particular found resulted in a very um I believe it was I don't remember off the top of my head but I remember the CVSS score being very high for the for that finding there and we worked very closely with um with that community to to

make those fixes and and and improvements. So, thank you. So, going into a couple of benefits and and and things that we've seen firsthand um from experience working with these projects and doing these security audits, we found that the benefits typically come in two forms. One are the immediate benefits. So the first one being that doing this security audit or doing a security audit in a lot

of ways is one of the first kind of security exercises that maintainers do at least the ones that we've worked with um had never for example really thought about um documenting threats and risks and really again just having that mindset of let's think about our project from a security perspective just that mentality shift or just that thinking. We've been told countless times from uh maintainer survey feedback

and just direct feedback that just doing that exercise really got them as maintainers and contributors to think more about security and to kind of have that mindset moving forward. Um as I mentioned threat modeling and risk assessment is very important and um not not often documented. So the uh threat modeling and risk assessments provide immediate benefits as I mentioned earlier the finding and fixing of vulnerabilities going

back to the security audit as a best practice. These aren't just oneoff vulnerabilities. Typically a lot of the the improvements that are made to projects are classes of bugs that are completely eliminated um as a result of the audit which then of course prevents future uh problems so that that class of bug does not resurface over time. And then the audit report uh documenting the process which

can be a very strong indicator that a project is thinking about security has at least done some some work in in in security and we have an artifact to to show that. Oh, and then so talking about some of the long-term benefits. So I mentioned uh it a second ago, but um but yeah, closing classes of bugs really helps a project's security posture. improving security tooling. So

implementing SAS and fuzzing and all sorts of of tooling improvements um really helps a project over time. Going back to when we talked about reviewing supply chain um risk, security risk, um improving that can really um harden a project's supply chain. And then um also threat models can provide be a reference guide for new adopters for new maintainers uh as well as as as um users of

of these software projects. So a couple recommendations um the first one really comes down to dedicating resources. So for any uh project, organization or foundation looking to do this kind of work, um I would recommend to dedicate resources for the administration, the project management when looking to get a security audit. Um this is the whole reason that OTIFF exists. I mean, we we formed this this nonprofit

organization to do this for for maintainers and and projects and and the funders uh that support those projects. But of course we don't have to to do every single audit and that happens in the open source space. So um for any u communities looking to do this would definitely recommend ahead of time earmarking a good amount of resources and time to uh just again handle all of

the back and forth and administration that goes into sourcing this kind of work. And the second one being um to make sure you're working with security teams who really understand open source. Um because we've found almost night and day differences between working with uh researchers and audit teams who are themselves involved in the open- source space. um tends to to go much better than for example other

examples we've seen of commercial maybe more commercial engagements done by um very large firms who um it's not uncommon for them to essentially use their company brand to to source contracts and then have um uh like uh uh to essentially not have the best expertise. So definitely make sure you're working with security teams and individuals um that that have the expertise in what they're auditing. And to

um some insight and guidance um really based on our experience, security tools can be a fantastic tool for improving posture. Um it's definitely not the only tool or the only thing. It's not a like a one uh silver bullet that will um solve all the problems, but it really can be a fantastic tool. And then as we found um collaborating with the maintainers and contributor communities while

that can um get complicated especially with larger projects and larger communities uh typically does yield better results than just kind of doing it in silos or in isolation. Wonderful. Um and then so yeah with that um I uh that's the end of the talk but um we just celebrated last year our 10 year anniversary so it's been uh a long journey and again just want to thank

everyone that we've worked with over the years and um want to thank all of you for coming to to hear about it and uh I think I have five minutes approximately for questions. >> Yeah, thanks a lot for your insights and your success stories. >> Thank you. Thank you. one question already. >> Well, thank you. So, how I get it is that there's someone having an interest

in having some funding going to you and then you make the connection to the open source project and audit researchers. Is there also a way where like you have collected some funding yourself? So, a project can go to you and say uh do you have we would like have to have an security audit. Can you help us? Is it also possible? >> Yes. Um yeah. So, um

there's a a lot that we do beyond um kind of how it kind of looks on the surface, but yes, we have um unfortunately we don't have really our own discretionary fund yet. It's something that I've dreamed of and I think hopefully one day we will get to a place where we will have our own discretionary fund to be able to apply to projects who come to

us. Um, but what we've done in the meantime is we've we've helped a lot of projects with the fundraising as well. So, we've done a lot of advocacy, for example, for certain projects to get funding if they're, let's say, in the running for getting funding, uh, as well as going to our funding partners to, um, try and source funding for projects. But, um, yeah, there's definitely a

lot that goes into it. It's not just, you know, taking money from a funer and giving it to an audit firm. There's so so much that goes into it. Um, and yes, I would love to get us to a place where we have a discretionary fund, but as of now, we just help with advocacy and fundraising. >> When you when you're saying you also you're you have

an interest like not only to report problems, but also to fix them. Is there a way like in the funding that you can like pay the maintainer or one developer to say, okay, if you fix a problem, we can get you a part of the funding for fixing that problem. Is that has it happened? Is it possible? >> It is something that we have done. Yes, we've

advocated a lot for setting some funding aside as either remunerations or some kind of um payment to to thank maintainers for their time and taking time to make fixes. It's not always the the uh top priority of funders um because they again are very kind of bottomline driven and that's more an afterthought. But we have advocated for um some of our programs to set some funding aside

to give to the to the maintainer or maintainers to actually make fixes. Yes. Um but a lot of our process um includes providing uh very in-depth proofs of concept and um insight onto what the vulnerabilities are to make the fixes much uh much easier. >> Okay, wonderful. We got time for one more question. >> So surprised that surprisingly the topic didn't come up yet, but I guess

that everybody's talking about generative AI. So uh from your experience and also your futurism here. So what is the current state with uh with with AI like are we seeing more vulnerabilities coming up because co people are contributing code generated with AI without enough quality and uh are we already using tools to try to detect and come up with uh with solutions for for security vulnerabilities and

moving forward how you do you see the uh like the balance between like all these contributions generated with AI by maybe exhausting maintainers versus using the tool to try to mitigate all the things. >> Yeah. Yeah. Excellent question. And yes, I'm also surprised it didn't come up sooner, but um um based on my experience, I've seen this really work good. See, I've seen both sides where the

bad or the not so great stuff that I've seen is that yes, in fact, people are using AI tools to generate bug reports and are some or even the the the AIS themselves are just generating bug reports and really flooding um maintain already stretch thin maintainers with from what I've heard direct um firsthand from maintainers, it's almost all just crap. um and um not really anything helpful.

Um so I've seen some of the bad sides of it. I have seen some of the the more promising sides of it in that some of the audit teams that we work with are really involved in developing tools to augment their audit work and because they understand the tools that they're using a little bit maybe better than the maybe let's say average security researcher or just average

person. um they're really actually using those tools to do a lot more and um they're able to actually dig deeper and kind of chase down more leads with the help of these AI tools. Um so I can see it going both ways. Um or I've seen it kind of have good and bad. And I think maybe for the future hopefully it will be um more meritocratic where

you know the the the individuals, organizations, what have you that are actually using AI to do good work and not and produce better outcomes for maintainers and projects will hopefully rise to the top. and the ones that are again just kind of flooding projects with slop will be kind of pushed out and not incentivized to to keep doing that. So, it's definitely going to be interesting to

see how that'll change over the next few years as they get better as the AI the AI and the LMS get better at kind of augmenting some of the work. But I think I'm always going to be um advocate for at least having some human interaction in the loop because without that I mean it's just very important. So uh I will be around all day today and

tomorrow. So if anyone would like to to discuss please let me know. And then also please check out our website ostiff.org um if you'd like to contact us or get in touch. So thank you all again. Have a great day.