Commit to quality: AI-enhanced testing in open source

This talk discusses the Adoptium project, which is an open-source, vendor-neutral distribution of Java JDK. The speaker focuses on AI-enhanced testing within the context of the Adoptium project, emphasizing the process of running over 17 million tests during each release cycle. They introduce the Aquaveet project, aimed at improving quality assurance for OpenJDK distributions by sharing tools and approaches with other Java distributions. The talk covers various projects such as Perf Savvy and Commit Hunter, which utilize AI to enhance performance testing and identify problematic commits, respectively. The speaker also highlights the importance of open-source programs like Google Summer of Code and Semesters of Code in advancing these initiatives and fostering contributions from students. They conclude by discussing ongoing developments and the commitment to maintaining high-quality standards in Java distribution.

[music] Um I'm actually uh working on the Adoptium project. I'm Red Hat and IBM and IBM and Red Hat. Uh Adoptium if you don't know is a opensource vendorneutral free distribution of Java of the JDK. Um, and why we're talking about AI enhanced testing in open source and at the Adoptium project is essentially because every release period that we have and there are four giant releases every

year plus two smaller ones, the feature releases with the larger uh CPU releases, we run over 17 million tests during a short period of about five days. That's the ideal situation. Uh so over 17 million tests means uh if things go wrong or if you want to dig deeper into test results, you're not doing it with your human eye. You're doing it with a bunch of tools.

Uh I also will note that I'm a the commit a committer rep on the Eclipse Foundation board of directors. So yesterday we had our board meeting and uh I'm only telling you this to set the stage for some of the future work that we'd like to do with agentic workflows in terms of supporting committers at the project. So how many of you are committers at Eclipse Foundation

projects? Yeah, I I want to talk to you after. I want feedback. Um, but I also do want to thank Lan and Longu who were co-authors of these slides. We originally did this presentation last year in September 2025 which is now these slides are completely out of date. So sorry about that. Um, the Aquaveet project is under the Adoptium um top level project. Aquavit essentially just means

adoptium quality assurance vitality and the re or if you're French you could go vit as in quickly fast uh the whole point of the project was to be able to raise the quality bar for open JDK distributions so we don't just exist to test the tearin binaries that we produce out of the project we exist so that every Java distribution in the world could pick up some

of our tools and our approaches and our processes and use them as they saw fit. And because of that, because of that notion of wanting to make something larger than what just our one project would use, we created a manifesto. And the manifesto essentially there's there's a lot longer parts to it. You can find it in one of our uh repositories. But open, transparent, robust, diverse test

suites evolving constantly. So as we come into this new era, it's not that new, but a lot of the tools are now more available to open source projects. Uh we are evolving quickly because of that. Um and one last thing is every good project needed needs a motto and Aquaveet we want to say make quality certain to happen and I want to talk about how we're doing

that. Uh first of all we think about testing as a kind of a abstract piece of activities right we plan and design tests we implement them. We figure out how to automate them. We run the tests so they get executed. When they do, some things fall over. We have to triage. We may have to exclude, take action after the triage, and then there's this cycle that happens.

Okay, we've figured out the problem, we fix the problem, we re-execute. Uh, and then finally, we create reports. But if a report is just a report and it doesn't actually give you any next action to take, it's fairly useless. So we want reports that are going to tell you what's the next best action for you to do. And that might be publish your product, but it might

be you're not publishing your product. You're going to go and uh triage this further or fix these following issues. Uh I before I move on, there's a lot of little words on this and the reason I want to call them out is because imagine that with each one of these activities, there are so many questions. I've only put a few under each activity and you can argue

some of them should be under different activities, but as you are testing your products, you're probably thinking about what tests do I need? How are my consumers using this product? Um, do I have other information like what tests cover what pieces of code? How often is the code changing that I'm testing? Um, more interesting for me and some of the areas we're going to look at first

are that triage, execute, and exclude cycle. Um, we in the past have done a lot of different prototypes and experiments. Some of them came together and became the Aquavit project. Some of them sit as quiet prototypes that at the time when we were first looking at them, they were too hard to implement. But now in the era of code assistants and you know our friends like Mr.

Bob and Mr. Code and I I'm sure there's going to be a code assistant that has a female name eventually. But uh when we use these assistants and if you use them in a way that a wise seasoned veteran software developer uses them, you can implement some of these notions that we had 10 years ago. Wouldn't it be nice if very quickly and then trial them and

see is this actually helping us make quality certain to happen? But before we get there, I do want to talk about uh and sorry this is some bits of our journey as we went here. We don't have a lot of extra people in our project. We'd love some more. Please come help blah blah blah. Uh but how we move some of our uh pieces forward are participating

in open source programs. So uh last year we participated in Google summer of code and semester of code uh which are two different programs and I'll talk a little bit about it but one of the thing one of the things we make sure we do when we participate in these open source programs are give the participants real work. It's not go fix a bunch of defects. It's

let's work on a feature together and move it forward because those are the things that we don't often have time for at the project. We're focused on, you know, releasing or the day-to-day grind and we really want to move some of this forward. So, we do that by multiplying our hands through these open source programs. It's great for contributors because they get actual real experience and it's

really good for mentors because you grow by teaching and you gain a lot of leadership skills. So I'll just quickly each slide tells about it. How many know about semesters of code? It's a kind of a new program. It's Carnegie Melon University and we worked with teams based off out of the Pittsburgh campus and the Qatar campus and this year I think we will be working again

with them. Uh although the Qatar team will be working remotely from home because of some uh dangerous situation in that region of the world that you may know about. Anyway, uh the I wanted to point this out because with each of these programs there are incentives. The way the semesters of code program works is the incentive is course credit. Um so what we found with all of

the different programs we work with, it's important that the contributors have incentives. Uh we did two projects with semesters of code student teams. I won't talk about the supply chain security project in this talk. We're going to look at the briefly at the Perf Savvy project uh looking at performance testing and the team that uh we worked with there for the Google Summer of Code. Uh the

incentive is they get paid by that program for the term and uh we worked with four different contributors and three different projects. So we had paired up contributors for commit hunter and glitcher. Uh yeah, we do like to name the projects something very hard to pronounce because it's fun. Uh but uh with each of these they were mid-length projects. They changed the program so that it's not

just uh shorter projects. For us it works better to have them medium size or larger size projects which means you have a bit more time with the people that are working on them. Okay. So this is where I get to say aiio. I'm going to talk a little bit about each of these projects. Uh, it's really a quick scan. I'm not going to go into details unless

you ask me hard questions, at which point I'm going to make you put on a t-shirt to ask it. No. Um, but these projects are really uh how we approached them. every time in the past that we worked with a a student program or something, you want to do some researchy thing and then you want to deliver something. Uh and that structure sometimes works well, but we

sometimes run out of time. So last year we decided to structure it a bit differently. We said let's define a problem or say here's the problem space and you we want to have a concrete solution or a concrete approach to to addressing that problem space not using AI and then follow with research and applying an AI based solution to that problem space and that way we thought

we'll get some concrete stuff out of this team no matter what uh and it did work very well that way so Perf Savvy essentially was us uh extending our uh tooling to allow for additional easy addition of extra performance benchmarks and also looking at how do we better detect performance regressions. Commit hunter. Uh, think about commit hunter as after you've committed code, finding which thing was the

problem commit that caused a test failure or a regression. Glitch Witcher took a different approach and said before you commit code, scan code, analyze code, and analyze pull requests and determine what's the likelihood of you introducing bugs with these changes or what's the likelihood of you introducing some more bugs into a file that you are changing. Um and then Vitei u was essentially we skipped the part

about having concrete uh examples to follow. Vitai was born out of the fact that someone in the commit hunter group was so doing so well we wanted to challenge that person. So we said how about let's look at a rag or apply a rag system to our repos and our uh code and see if um we would get better results when we query and ask questions about

it and it also was going to help us as mentors learn more about some of this stuff. So the perf savvy one uh even though we were already running dicappo perform the dicappo performance suites which many of you might already know about uh we didn't have as much automation around it as we wanted. We want to make it easy to plop in any performance benchmark. We want

to make it easy to view the results um and also to be portable so it doesn't matter where you run it. you can always get results and you can always visualize results. So the very first concrete part of this was for us to just produce something that was portable across different uh tools that we had and then the next one was actually just using or we didn't

get completed in this section but essentially exploring the use of AI to look at historical data which we keep a lot of in our um test result summary service which I'll refer to as TRSS. Um, we keep a lot of test results over time there around performance and we wanted to see if we could actually go back in time and see trends of when uh regressions happened

and then associate those regressions with changes in the codebase. So the kind of first part of this, these are uh the one side is a screenshot of our a test results summary service and essentially being able to run uh our performance pipelines which actually kind of interle a test um run and a baseline run and then calculate a bunch of these uh different variables for us. We

haven't really fully uh explored how far we can go with this because, you know, we get a little bit further with our Aqua Vit manifesto, but then we get stopped and have to do releases. But the idea here is to be able to identify and then um act on uh when changes come in that are that cause problems. Not just for DiCapo suite, we have 80 over

80 different performance benchmarks that we run, but at the moment we're not blocking the release on any regressions that we're seeing. So what we intend to do is to start doing that. And it started with being able to portably move around some of the the output from these runs. Uh yes, I won't I'll just jump forward. I'm going to scan through these quickly. hopefully, but keep me

honest here. Um, commit hunter, that was the one where we said after things are merged into the code base, how do we determine which things cause test failures? So, the first phase of this project was more of a rulesbased approach and it was using git bisect to go, okay, there's a good build and a bad build. And the bad build is where new test failure shows up.

Uh and in between good build and bad build because if you're running millions of tests a day, you're not perhaps looking daily. Uh there might be a bunch of changes in play between good build and bad build. Uh too many that you want to say it's exactly this because there was only one change. So the first was the rules-based approach of bisect bisect bisect and identify most

likely and identify them based on uh keywords in the test output versus commit messages. So kind of clunky, kind of dumb, but still sort of how we do it when we're thinking about it as a human trying to go. actually some humans that triage are very quick to go aha even when there's just one word that might tell them something. Um but phase two is saying let's

apply this uh kind of same approach but let's not be human about it. Um and be able to kind of identify the good ad first of all identify them and then test. So once we identified likely the top three likely commits uh build at that point and retest and see if you still if you see the test failure. In some cases that's not easy or possible but

in other cases it's very easy impossible. So determine what's easy and then do that if you can. That's the notion of it. Okay. Glitch witcher. And does everyone know what witching is? where you go around with a divining rod and you try to find water. Okay, so glitch richer is try to find glitches. Um and Glitchricher years ago we had done the prototype. Uh we had pulled

the code out of uh an old paper that some team from uh I guess teams from people from Microsoft I think and Google originally did um where they said it's very hard to determine where defects live in code or there are algorithms that you can use but they're very expensive. Maybe nowadays that's not true, but they came up with an algorithm that was relatively inexpensive. And that

algorithm works kind of like this based on uh the source code. It looks to say how often was a file changed and how often was it changed because someone fixed a bug in it? And if uh if it changed a lot and there was a lot of bugs that it was uh continuing to be fixed in that same piece of code, the likelihood of it having more

defects uh or having uh you introducing more defects is much higher. So it produced this algorithm produced a score and you could run it against uh a GitHub repo or any code source repo and it would score all the files in that repo and it in our case we implemented this again and we could say what are our 10 worst files or our top 10 files that

have the highest score most chance of having defects. Um and then one of two things happen. If you go back to that activities, you know, planning uh and designing phase, you would go actually we need to write more tests for this. Could I have an agent start to write more tests or at least analyze the tests that I have? So while this doesn't um maybe necessarily find

new bugs, it might give you the notion of how you could predict where bugs are or where you should lean your efforts to add new testing. So that was the concrete approach. A much less concrete approach was uh using an REPD model. Uh this was based off of a research paper that we found that we thought was interesting. And again, we kind of use this as a

way for our mentor team to learn and to challenge the participants of the open source program. Um essentially uh what the paper purported and there are opensource repos uh that go along with the resour re uh search paper would say if you scan uh source code files and in fact we scanned we used open J9 for this example because for a number of reasons partly because we

had done previous research on that uh repository but also because it's a very large uh repository tory and it had examples of C, C++ and Java files. So, uh scanning that uh applying the algorithm that was in that research paper and saying what's the likelihood of uh defects in a new PR that we would be creating to um check into these repos. And this was a fun

project from the perspective of when it was just a regular PR from a you know a rockolid open J9 developer. It was hard to differentiate because again there's a score that gets applied and it's a essentially a likelihood of introducing defect score. It's like very little difference and you can maybe see those defective reconstruction errors with the normal changes. we weren't seeing a big percentage of change.

But then we said, well, how do we test that this actually works? Let's create very buggy PRs. And uh, of course, you don't have to merge them to find the errors. So, we never did merge any horrendously buggy PRs, but we did notice that um, this did work for finding certain types of failures. Again, we're learning, we're playing. That's the was the purpose of the second half

of glitcher um vi so the ragbased assistant um we started the phase one by just pointing it at the readme and wikis and blog posts that we uh created as part of the aquaveet project. Phase two we pointed it additionally at all of the test framework code and test material that were in our repositories. Um, and essentially all of the test suites that we use, we didn't

do all of them. We didn't do some of the uh performance benchmarks for example, but and I guess many of you who are here probably already know um the rag story is essentially because creating your own models is expensive and we didn't have the resources at all at the time. uh but to be able to uh augment uh tools to say uh you have a very domain

we're going to give you some domainspecific information and uh that will allow us to then ask questions about Aquavite and get better answers than just if we were using a large language model. So we kind of uh went through the phases of you know processing that data, indexing, storing it. I forget what was chosen. I think it was Chroma but storing it in a vector uh database

and then seeing were we getting some better answers to our questions. And we tried to ask questions from all of the categories. Um, and with mixed results, we we kind of ran out of time at this point because we had asked for one medium project and we stuck uh commit hunter and vit uh in that 350 hours. But all this to say when we're thinking about how

to position ourselves and using AI in our project uh we want to be extremely flexible because we have teams from many different companies and we have many access to many different tools. Uh we want to be model agnostic and of course nowadays you know back in the summer of last year we didn't have access to too many tools. The world has changed again. So uh but at

this point and then what we're focusing essentially our time and energy on as we kind of move forward in this journey is what are the different [sighs] domain specific pieces of data and information that we have that are unique to our project and how do we access them and how do we organize ourselves uh to get at some of that information. And then I as we were

working through this and as we kind of look back over the years, um I also remembered that before we even started the Aquaveet project, we had this notion of small little services because you know remember years ago um microservices and things of that nature that's the way we were that was the language we were thinking about when we're thinking about all of this stuff. But so I'm

gonna at some point in this slide deck I've added that very old slide and I you can cross out services and put in agents in there as a thing. Uh okay there it is. Uh so this was back in 2017 just before we started the Aquaveet project and um what this really the middle layer was oh so first of all I wanted to say there's so much

data for us to look at because this isn't just our GitHub repos this these are many other GitHub repos that we can look at it isn't just our Jenkins server or someone's um asdo setup it's many different ones that we can be looking at. And then the core files, anytime Java crashes, and [clears throat] we kind of got access to a lot of core files prior to

GDPR, [laughter] there's a different story. They have to be cleaned now. But as people send us uh support tickets and say, you know, something crashed or I have this output, um we consider that raw data that we can use. uh obviously with stripping out any customer or consumer specific stuff but in the no in this notion we had prototyped a core analytics service which was looking to

see okay when this crashed what JVM options were used what platform were you running on um we got only so far in 2017 being able to look at that and now we can come back and visit that again Because we have got a completely different set of tools now and we can say look for patterns when you have all of this rich information in those files. Start

to see patterns and you can start to see patterns about platform use and options and kind of some of the inputs to that machine that were or that JDK under test bug prediction. I uh talked about the one approach that gives a score. So that was that previously the little service we had and now we can think about it with the use of AI. Test generation um

and even test selection. I don't know why I shoved bench engine in the middle there but test generation is essentially applying combinatorial test design to source code. And that can essentially be um analyze this. Let's take a small example. Analyze one method. What inputs it has and what output it gives. And say there are three different types of inputs to that method. How do they interact within

that method? You can figure out by looking at it as a human because it's not very complex when certain variables interact. And so you can say if v if the input A and input B are brought into that method and A is divided by B but B is zero something bad might happen. Uh so you can see that but as code becomes more complex it's harder to

see. Combinatorial test design takes a look at how the different inputs interact and then it also um determines what is the smallest number of tests and smallest number of inputs that I would need into a method to get full functional coverage. And it's shockingly different than what we do today where we go I just want a lot of tests. I I earlier bragged seven over 17 million

tests in a release. That's embarrassing. We should reduce that by a lot. Uh we should be able to know the source code so well that we're testing or not us but our little agents to say I do not need that many tests to be able to determine no known defects in this code. So test generation and test selection is sort of similar in that if you're testing

if you're supporting developer work and you're testing a PR, you already know some code coverage information, you know what set of tests are already in your suite. So of those millions of tests that we run, which ones cover the code that's being changed. So we could include those in the PR testing. And we could say, and because of the nature of the code you've introduced, you're missing

some tests. And we could either generate tests to run them against it or warn to say you got to we're blocking this PR until you add more tests. input options was the little playing around with uh I won't go into it but [laughter] it's essentially playing around with what different inputs you give to your uh JVM and um seeing if you break it. It's not a form

of fuzz testing really, but it's a was a playful experiment we had done. On the topic of fuzz testing and since we're all uh recently, you know, hearing of mythos and other things that are helping generate fuzz tests, we had done a work with a research team uh quite a while ago based out of the University of Edinburgh um fuzzing the JVM. Uh and at that time

the challenge what we were trying to do was create fuzz tests that were hu short enough that you could understand it reading it as a human. Um I don't know why we even bothered to do that. We couldn't predict that we wouldn't need to read it as a human maybe. But the the idea there was could we generate on the cheap fuzz tests that would essentially crash

a JVM and that was a lot of fun. We have not brought that back into play. Uh but we may revisit that. Finally, those last three, the results analytics, the results summary and results compare. Those three pieces essentially became our test results summary service. Um what that does is monitor our CI servers and then banks sorts and filters and banks the uh data from our CI servers

and then um presents it in a nice way that you can quickly get to the root of problems and quickly know what your next best actions are. But it is also the placeholder for additional analytics and additional monging of data uh that we hope that our agents the new ones that we will be able to write will sit in. Um Aquavit um th this was the challenge

slide from September 2025. We had struggled to get any access to any resources. Committer's time was limited. The AI field is so rapidly changing. By the end of a program, we had already uh learned that we could be using some new newly available tools. But I wouldn't say these programs were a wash. We learned a lot. We developed relationships and we did it I think in the

right way where we started with small challenges which allowed the brand new participants to open source to learn about our project and to learn about the problem space and then we were able to move into a more researchy and uh newer areas that was new to everyone. So it compelled us to learn and so back in September our retrospective was oh we better update our program plan

to get some funding so we could get some resources. Uh we also wanted to extrapolate the rag uh approach to broader foundation level. So I can never find the committer handbook but it's out there and I once I do find it it's really long. I want a way to ask questions and get exactly the answer. So, we may come back to this this year with a semester's

code team and use some uh agentic workflows to do better committer support. Um, at the time here, yes, we need to use these open source programs to advance our AI position. That's still true. Um, but I will say now we're in 2026. Suddenly we have access to better tools and they're really a lot better than what we had. Uh, we're utilizing, of course, I'm from IBM, I

have access to IBM Bob. All projects have access to co-pilot enterprise. And if you don't know that, uh, go to your Eclipse profile and turn that on, uh, if you're able to use it, if you're allowed by your company. But we have a lot of tools at our disposal now that we didn't last year. We will be participating uh this summer uh doing a log analysis for

um securing our infrastructure even more than we do now. Uh and we focus a lot on that already. But uh one student team will work on that and the other student team will work on uh committer support workflows. Um but the main part here and the message I wanted to say is not just for our project but for everyone let's stay vital and the Aqua Vit way

is by um inviting participation being playful having some fun swearing a bit but then like coming out with uh some things that have improved our position. Uh and that's kind of the end of it. I I maybe went long, but I do want to thank you for uh being here. And I do want to say um the Aqua Test repo is really where you can find a

lot of stuff, including the manifesto. Um we're horribly bad at writing documentation, but this year there will be an agent to help. So, thank you. And if you have questions, let me know. Yeah. [applause] Oh, and if you have questions, if you have questions, you can win a prize. Yeah, because I'm not flying back to Canada with adopting [laughter] toques. I I don't know if you need

to in Europe, but do do we have any questions that aren't hard? All right. Well, talk to me after if you want. And if you're a committer uh at the Eclipse Foundation and you have questions or you want to give me ideas for improving your life and your workflows, uh please come talk to me. Did you want to ask question? Yeah. Well, we'll get a microphone. Yeah.

>> So, the semester of code, is it like GOC where everybody can participate globally? Is it just for people in Kata and Pittsburgh? >> Well, right now that's how they're running it because it's student teams and um from uh Carnegie Melon University. So, wherever they have campuses, that's how they're running it. Yeah. But if you have a project that you want them to work on, you can

submit a proposal. They're not quite, they're new. This is like this year will be only the third time they've run. Uh so they're still kind of getting their legs under them, but I think they are looking for other Eclipse projects to propose ideas to work with them. Yeah. Yeah. Thanks,