What's new at Adoptium?

This talk focuses on recent updates in the Eclipse Adoptium project, specifically regarding the Eclipse Temurin JDK distribution. The speaker, Andrew Leonard, highlights major improvements in automation, build pipelines, and testing frameworks over the past year. Adoptium has integrated automated workflows for building and publishing packages across multiple architectures, significantly reducing the time taken for installations. Key advancements include the introduction of a notification system on Slack for build statuses, enhanced test coverage using the Aquality project, and improvements in infrastructure monitoring using tools like Nagios and WAZUH for security. The session wraps up with a look towards future developments in AI and agentic workflows aimed at streamlining the release process and enhancing testing protocols.

[music] >> Good afternoon. Um my name's Andrew Leonard and I work for IBM at the uh Eclipse Adoptium project. Um I've worked at IBM for 36 years now, for a while, with a 4-year stint at Red Hat, which is a as you may know is a subsidiary company owned by IBM. Um and I'm now back with IBM, moved back from Red Hat uh middle last year. And

I'm the lead for the Eclipse Temurin build distribution and also an Adoptium PMC member. And I'm going to go through uh today um what's new in Adoptium over the last year or so. Um and this is a rough agenda. A quick Adoptium overview very quick for those that maybe don't know. Um and then go over a numerous things we've changed, improved um build pipelines uh our test

framework, our infrastructure, and some new work that we've been doing on reproducible builds in case secure development, and some future direction. So, at the Adoptium community over the last uh uh year or so, we build Eclipse Temurin as I hope you know, which is a free open source TCK certified enterprise-grade JDK built on open OpenJDK open source. Um and it's tested by uh Adoptium Aquality project, which

is a full uh wide-ranging um test suite um from our community, uh and is also Oracle TCK certified. Um over the last year, for example, there's been over 300 million downloads from the various means of distribution. We build five LTS versions from JDK 8, JDK 25. And we also build the latest um feature versions, so currently 26. And 27 is the current head stream version, which we

build EA beta builds for. So, every quarter, um in fact, this week, as you may know, um we're currently doing the April CPU at the moment. We build roughly 58 builds across all those versions and various platforms. Uh around about 12 architectures per version, so there's a lot of building and testing to be done. Um and we also build regular EA beta builds of the latest upstream

development builds. Um for all those versions as well. So, a lot goes on. So, what have we improved? So, um I'll discuss now the key changes on some of our build improvements. Um and I'll go through each of these in turn. >> [snorts] >> um as you know, with Eclipse Temurin, you get installers for the various from the various package repositories uh as as I means. Um

and our old flow used to be uh we'd have to wait for every architecture for a given version, so all of those Linux um wait for those to be published before we could then uh meticulously by hand update the YAML for the installer definitions, and then publish the packages to the um package repository. So, there was a significant delay potentially there, and it was a manual process

um which is also potentially error-prone. So, what we've done um is basically now that they're independent, and it's all automated. So, as soon as as for example, the X64 GitHub binaries are published, it triggers the X64 deb, RPM, and APK package build um creates that installer and is automatically published. So, within practically 10-15 minutes of a given architecture being published, the installers are available. So, a huge

improvement over where where we were. Um the other thing we've been working on uh build notification status in our Adoptium Slack channel. So, if you're a member member of our Adoptium Slack uh you'll know there's a hash build status um hash build channel and we publish daily status on there of our of our latest EA builds, which basically collects the build results from the latest EA builds,

checks checks how healthy they are, is anything missing, anything wrong. It also statistics on the Aquality test pass rate uh and also new the reproducible build status which is again more um secure dev type tests. Generates notification and posts that to Slack. And here's a an example I took a screenshot of from last week. So, you can see all the JDK um JDK versions, EA builds, their

healthiness uh links to the binaries as you need to go and find them, Aquality test results and the new section at the bottom, which is the reproducible build status. So, as you see, lots of 100%s there. I think in this case the Z1, I think um a node ran out of space, unfortunately. And I'll come on to infrastructure improvements and reliability later. Uh and the other uh

workflow that we've added is uh a GitHub GitHub action workflow such that uh weekly we have this workflow that does automatic EA build triaging. So, it verifies are all the expected builds published that should be? And it also does some smart analysis uh of the build and test results for all those EA builds, and generates a smart analysis report and creates a GitHub issue in the Temurin

build repository which then helps our team you know, examine for build or test failures that we may need to deal with, etc. So, improving automation, notification, and helping our team uh produce good binaries in the end result. So, the Aquality test uh Aquality test improvements. So, uh I hope you know, the Aquality project led by Shelly Lambert uh part of the Adoptium project. So, this is a

uh a significantly large set of uh repositories with test suites ranging from OpenJDK JT Regtest test suites to um some some system and performance test groups. So, it's that extra enhanced set of test suites above and beyond what OpenJDK uh has in terms of its tier one, two, and three type tests. Which but we also run those as part of Aquality. Um and I'll only go through

the set of uh improvements here in the next few slides. I'm going to touch on Temurin compliance here, which is the uh the project that does the TCK certification. Um so, up until middle last year, when I was in Red Hat, I was part of that team. I'm now back in IBM, so I'm not in that team currently. Um but obviously we're associated with that that private

group uh because our binaries have to go via that to be certified. And one of the big really big things that we helped at the beginning of last year was picking up um Amazon Corretto's Arctic GUI automation uh open source tool um which is which was a real game-changer for us at at Temurin because as some of you may know, part of the Temurin compliance test suite

is a load of what we call interactives. Um so, testing AWT Java X Swing type um compliance. Um and that's up to you know, beginning of last year was a manual job um tedious, hard on the head, and um uh obviously delays delivery. So, what we were able to do is pick this up and do what we call create Arctic recordings to replay to automatically run those

test test cases in the Temurin compliance project. and uh this is a sort of schematic to sort of showing you uh uh the flow from the public Jenkins at the top. So, the Temurin JDK and binary plus a SBOM, and I'll come on to that in more detail gets built on the public Jenkins and runs tests, the Aquality test Aquality test jobs. Um but it also launches

via the Jenkins remote um trigger support uh to the private TCK Jenkins. The Temurin compliance project has to be private because that's obviously proprietary um test suite stuff um under Oracle license. So, we trigger those jobs into that project. It's over to them, as it were. Uh and they now use the Arctic test framework to do the GUI automation testing, and also the standard non-GUI um compliance

tests as well, all run there. and we've and they and those results come back with cool sort of blind reaching. So, all we know from the public side is did was it successful or was it unstable, basically. Um Uh and we also have interest in these reproducible verification tests, which I shall come on to shortly as well. So, here's um that JDK trigger just broken down a

bit more. Um So, the public Jenkins triggers via the remote trigger plugin to the Temurin runs on the TC on the in the Temurin compliance Jenkins and reports the overall job result back. What this is also showing you is uh the rerun JCK JCK link capability that we've added. so that we don't always have If if one of those uh test jobs, say, for a a given

platform has some unstable test case, what we know is it's reported back as unstable. But, with the rerun JCK link, we can relaunch it in the hope that it is most likely just some instability or, you know, a single test case potentially. So, we can do that without having to, as it were, get on the phone with the Temurin compliance project and say, "Hey, this has failed.

Can you look at it?" Although, they they're probably doing that anyway. Um but, given that, you know, you I showed you the scale of the platforms, versions, and uh architectures, it's it's a lot to done. So, so if we can just click that relink and it resubmits it, um and it passes, then we've we've we can do something a lot quicker that would that without having to

get involved with the Temurin compliance And publish things a lot quicker. Uh another key area of improvement in that accurate project is improved uh test results uh processing. So, we've we've added some more intelligent smart test result parsing and importantly, a new auto rerun logic. So, some of the OpenJDK, for example, tier two, three test cases are not always as stable as they perhaps should be um

compared to say tier one. But, uh and some are more susceptible to say network or host or specific environmental issues that make them a bit more flaky than they perhaps should do. So, this auto rerun logic, which we can configure by test group, uh automatically reruns just the failed test cases um potentially see if they'll pass. Um Or, if it's obviously a uh a genuine problem, it

will report it, etc. So, this again helps us stabilize the and also in the Temurin compliance project, because this is accurate project uh rerun capability, the JCK the Temurin compliance project also gets that capability in auto running JCK jobs as well, which again helps achieve um more stable test results. So, here here's a a just a quick the more enhanced test result summary. So, it's a lot

clearer. Uh you can see it ran four test targets, one failed, and it failed because of that particular test case. Uh and you've got a breakdown of the test cases that that um that ran and passed or failed. So, it's nice and clear. Now, another major thing that we improved on was in the early early part was in the Temurin compliance project, there was a what we

termed auto manuals for um uh well, most the architectures, actually, but um these were test JCK tests that for whatever that they were more flaky due to typically environment-related issues at the time. So, things like uh they might be have issues with network or host configuration. Um and what we did is we spent some time and effort to get rid of all those, basically. So, fix the

environment so that was satisfactory for what the test cases needed to make them automatically the JDK 17 and above. So, that again removed manual work, making the whole process of TCK certification a lot quicker for us. So, if you combine that work I just mentioned with auto manuals removal, with the auto GUI to automate the uh interactives, this has led to a great improvement, especially on Linux,

for our delivery times. Uh and I think as example for that, where that my team back in the office are currently doing the April CPU, and they published JDK uh 25 x64 Linux yesterday 13 hours after the upstream tag was source code was was tagged. Um And our aim is to continue in this direction. I think off the top of my head, I think about five or

six platforms are published yesterday, so that's all within 24 hours. And our aim is to get everything within 24 hours. That is our our target, really. Um So, uh infrastructure improvements. So, two key major areas in scaling with cloud agents um and the the typical monitoring alerts and housekeeping. uh dynamic cloud agents. So, our adoption Jenkins, um we've now got two configured uh Jenkins clouds, one is

Azure one and the other one is an Oracle one. Um Oracle is a project that dynamically provisions uh Mac stadium Mac VMs, uh and we use that for provisioning. Uh we've added Mac OS 15 support test agents in there. So, uh basically, all all our Mac builds are done via provisioned Mac stadium Oracle uh nodes. uh on the Linux side, we've greatly we're moving in a a

significant direction towards mainly dynamic provisioned Azure uh nodes. So, we've uh introduced uh Linux x64 and arm64 dynamic Linux build and test agents. So, this has uh increased our scalability, as you can imagine. Um I think our current limits, we can we can launch up to 30 VMs uh something of that order. Um and it's cheaper um because rather than having to have loads of static nodes

that cost us more money over the long time, we just provision a node and we're also going to be doing this we are doing this with Windows as well. Um just when we need it and it goes down and it's a lot cheaper for us as a project, which is great. And it's more reliable. Cuz you know, nodes over time, if they just kept running, tend to

gather, you know, uh baggage, shall I say? So, this, you know, they spin up nice and clean, hopefully, and it helps us scale. Infrastructure monitoring, uh alerts, and reliability. So, three major areas here we've we've done improvements on. Nagios monitoring and and alerting. So, Nagios is a is a monitoring framework, um and we use that extensively for monitoring our our Jenkins uh our master Jenkins and all

of the uh nodes for all the environments um to help detect infrastructure problems and give us real-time visibility into the system, so we can tell if, you know, uh disk space is an issue on some node or other, gets alerted or or root volume space, uh or there are uh network issue detection, etc., those sorts of issues. The other thing we've um uh increased is we're now

using uh WAZUH for security monitoring for real-time detection um of suspicious activity and vulnerabilities. So, again, yeah, we want to make sure that our Eclipse Temurin deliverables is are secure, so we need to make sure that our environment is secure as well. So, that helps us keep us compliant. So, compliance monitoring and updates uh and threat intelligence, um but also, um possibly just as important, uh it

helps us identify and mitigate um attacks. And we we have experienced that as well. We had a brute force attack on our Jenkins master server, and WAZUH identified and mitigated that, which was great. And we just kept running, which was awesome. Um unfortunately, you know the world we live in at the moment, so uh that's great. Um and Docker host house housekeeping, we use Docker a lot

for our uh test and uh environments. and we have housekeeping in that we added we've added extra housekeeping in there for container uh cache management um to help clean up stale uh containers, optimize disk um so that our nodes stay healthy and reliable, and also perform better. Okay, now, so this is a new feature that we're um soft launching this month, actually. So, this is part of

our um secure development and trust uh uh direct directive in leveraging reproducible builds um to help verify Eclipse Temurin binaries with willing third parties that will help uh do independent build verifications. Uh and we're incorporating that with uh we use CycloneDX for our SBOMs. We're also adding uh CycloneDX attestation as part of this story it it uses leveraging its claim declaration schema. And I'll explain this in

more detail. So, this is all to do with you know, ensuring a trusted secure supply chain so that people can trust the Temurin binaries that we ship and and can feel safe using them. it in essence what it is, there's sort of sort of three extra stages here, two to four. willing third parties can offer to rebuild in an independent secure environment of their choice um their

own built secure tool chain matching those that are identified by the J Temurin they're rebuilding based on the SBOM information provided by that Temurin build. So, as you probably know, Temurin provides the GD JDK along and an SBOM that go in hand with that. And the SBOM has been extended with build uh and recipe information which give details of the exact tool chain information and exactly how

to rebuild that Temurin via this process. Uh and we give as part of this process, we help give guidance on how, for uh an independent provider can uh rebuild a secure G GCC devkit tool chain um from source independently. and and they won't need to do that every verification because we don't uh change devkits or tool chains that often. So, they can create a secure uh GCC

or or download a a verified Microsoft Visual Studio or Apple Xcode and and store it securely in their environment for doing this Uh and that's based on the SBOM and I I've got a flow in a minute to explain that in more detail. And as I said, we use the CycloneDX attestation's XML uh along with digital signing for the uh claim declaration. So, here's a flow uh

that describes how this works. So, at the top you've got the two pink boxes with the official Temurin binary and the Temurin SBOM that goes with it. Which describes the you've got the binary and you've got the SBOM with the materials with that extra recipe and build information on how it was built. The the third party will download a small script from Temurin build um and run

that script which goes through the process of uh as input the SBOM for the Temurin binary they're going to verify. Um so, the bill of materials. Uh and obviously as part of that script, it will pull the upstream OpenJDK source based on the information in the SBOM that it needs to build. Okay, the exact commit upstream. Uh and that will independently rebuild using the third party's independently

built tool chain. So, this extra You'll see you'll see the um Beca because it's done independently using an independently created tool chain, it's and if via the binary comparison, that's 100% identical, it gives a a a lot higher guarantee in the security in in and safety of that Temurin binary if it's 100% bit for bit identical. So, at the end of running that script, it will do

the compare for them um and hopefully tell them it's 100% identical. Uh and as part of the output, it produces what we call an evidence file. Which I'll show you an example in a second. If it doesn't, then we'll provide guidance on how to report that to us uh to investigate is there a problem here, etc. So, here's a here's an evidence file that I I ran

this last week uh on a uh Temurin JDK 21 EA build. So, you can see this is the the epilogue from the uh console. Uh and it tells you what it's compared how many files in the JDK there were. It was identical, 100%. It's got the version, architecture, and OS. And the SHA-256 of the JDK that it actually compared. So, that SHA-256 is actually computed from the

one it actually built. and that's important because this this evidence file flows into the process I'm going to show you in a second and has to be validated uh as identical um as part of the verification. So, this is the next part of the process. So, having the having the third party rebuild it identically created that um the uh So, from that, they'll run a a new

client application that we've which uses the CycloneDX community core Java CLI to create a attestation CycloneDX attestation document or CDXA for short. CycloneDX attestation. and they sign that using their uh certificate and submit it to a new project called Temurin CDXA. uh and then that So, they'll create a PR for this new uh repository uh and the act the GitHub actions for that PR will validate uh

the format of the XML um that they provided a a a detached signature uh and it will also fully validate the content. And importantly, it will also verify the evidence in that ex CDXA document. Um it will make it will do a comparison It's an awful lot it says it's claiming to 100% reproduce this JDK with this SHA and it will then go and check the Adoptium

API for that same version to ensure that that SHA is identical to the one that they reproduced. So, again, we're tying it all together. And if that PR's um reviewed and accepted, it'll be merged into that At which point uh adoptium.net website where you go can go to download JDKs. Um you might know today that if you go to the downloads page on there, you'll see your

JDKs of choice. And you've currently got two icons on there currently which are the Aquavit test passed and the TCK certified ticks. Um at some point in the future, there should be a new tick um once third parties have done that, which is triggered by this process to say it is um uh reproduce reproducibly verified with a tick of some sort. So, again, it's giving uh an

indication of more trust for our users and consumers that they can trust our Eclipse Temurin binaries. And as I I touched on at the beginning, we're doing a soft launch of that this process um this week with this CPU for JDK 25 Windows. So, some uh colleagues in Red Hat are going to do an independent verification and some other colleagues in IBM are going to do something

similar as well. So, we're going to have two independent from the the Temurin project doing a verification to see how our process hangs up. And then we're hoping to get more willing participants as we go forward um as appropriately. So, if you want to sign up, you're welcome to join us on this effort to make sure that Temurin is uh secure and reliable and trusted. Okay, my

last slide. So, what what work we're looking at in the future? And you won't be surprised to hear AI comes into AI code assistance as I'm sure most of you are already using, yeah, we we being IBM have been using Project Bob internally from the beginning of this year. Um which has greatly improved our uh productivity. Uh uh and I I've used it extensively on the work

for this reproducible verification scripts and and it it's helped me A do things are so much faster and also because I because I can do things faster, I can uh achieve better use case um and design because I I can spend more more time thinking about the design and architecture and the interface as opposed to just coding because Bob does the coding for me. And then um

but we also use GitHub Co Copilot Enterprise. As I hope you know, as part of Eclipse Foundation members, you get access to Which is a great bonus. Uh I recommend if you're a foundation member, using it. So, we've started using that for um helping with pull request reviews. It's great at that. Um uh and also started using it on uh actually saying Hey Copilot, can you look

at this issue and create me a PR to fix it, please? And then it goes away and within minutes comes up with something which is awesome. the other main thing that we're going to be looking at shortly is agentic AI flows and agents. Um one possibility for that is helping streamline our tips to many release processes more so that there are still some components that we have

to do manually um which could easily be done by certain Some of the issue management, some of the reporting stuff, um and things like blog automation and things like that. Um or release banners, etc. and another big thing we're going to look at is smart triage agents. So some of you may know we have what's called TR TRSS which are test results summary service which we use

which provides a test results summary for all the uh Tamarin pipelines. Um it is actually backed by uh a significant database um with test results history. Um but it also picks up live test results uh from running pipelines. So the the plan is probably to create uh enable that via an MCP protocol um perhaps link that with some agentic flows uh possibly smart triage uh that could

react um dynamically, automatically, and do the good things. and the last thing I was going to mention is this is to additionally to help enable us to move forward with uh agentic uh agent flows is do a bit of refactoring to our top-level build pipeline um Jenkins code to enable us to better fit into a model of leveraging agentic flows. So for example at the moment we

have situations say uh um so one part of the build pipeline for example is uh Mac OS >> signing which goes off to the um Apple notarization service. Sometimes yeah, cuz it's going off to some Apple service it get you get a 502 or something because it trips over um because of either network problem or or just that Apple's a bit busy. Um we can have agents

that help monitor and react automatically to these flows in a better way, it can help automatically recover and restart that sign signing process for example, that signing step um and continue. So currently our pipeline is is quite a long, at least part of it quite a long um singular path, but we could break that into parallel flows that have uh agents that can independently uh monitor and

react to issues that occur there. Right, I think that's it. So I think I've got 4 minutes. It's not bad given my stuff today. Um So any questions or anything you'd like to add? Hi. Yeah, thanks a lot for the presentation and your insights. Maybe about um the first part, you talked about like that you introduced a Slack channel. So what was also the the reason behind

it? Because my personal opinion is when setting up such a channel it in the beginning it's pretty nice, but at the end you get a little bit annoyed about all this push notifications or you didn't look into it about like the successful build for example. So what was the reason and are you really using So I I didn't quite catch all that, sorry. Can you just summarize

that again, sorry? >> Okay, yes. So my my question is about this Slack channel. At the beginning with the with the build thing. Uh Maybe you can also go to that slide. >> Yeah, oops. Wrong one. The Slack channel. Oh, this? No, no, no, no, go on. Stop. Yes. >> Oh, the Slack channel. This? Oh, sorry. Sorry. I sorry, I misheard what you >> what was the

reason to setting it up? The what? The the reproducible bit? Is that what you're referring to? So this is So I'll I'll relate this to the last section of about the reproducible this that last row on the 20 JDK 21 reproducibility this is actually running the exact same reproducible verification as part of our Aquavit on our Jenkins as a double check for ourselves that it's they're all

100% identical. So it's it's sort of um so the JDK 21 um say X64 Linux binary is with Nest bomb. We then run the Aquavit reproducible verification test which you've added as something new which runs on all those architectures. Um and it actually does an attestation on a suitable VM um to to help tell us that it is 100% identical. Again, this is giving it extended trust.

Um and the the the Z1 that's got the question mark was because the node fell over. And that's where I'm hoping we can help have an agentic flow or something rather than me manually having to go and fix it and restart it to make it work. Does that answer your question? I'm not sure it does. >> A little bit. So you plan like in the future use

this to maybe build also agent workflows on top of it. Okay, got it. Thank you. Yeah, thank you for your question. Hi, yep. Yeah, thank you first of all for providing Tamarin at all. It's really a great help. >> We really like to use it, but for the Eclipse platform and Eclipse IDE we are desperately waiting for the Windows on ARM build for JDK 25 and Can

you estimate when they will be available? Okay, um good question and I Well, Microsoft are part of our community as you probably know. Um and they are working on some patches for JDK as you correctly point out JDK 25, 26, 27 uh Windows ARM because without them it currently it doesn't quite pass all the compliance test suites and it also has various um Aquavit failures. Uh they

are busily trying to get it merged and I can't give you an exact time frame, but we keep keep asking them. Our colleagues We work very closely with them. Um but I hope so. I'd hope by July. If I I can't Yeah, would be great because we are delivering packages of the Eclipse IDE for all platforms, of course also Windows on ARM. Yeah. But without JREs it

means that basically the entire Eclipse IDE is locked on Java 21 and we also would like to use the great and new features provided by Java 25. >> So who are you? I Do you mind me asking? Pardon? What what projects or team are you? Uh Eclipse platform >> All right. PDE, JDT, Equinox, so all the core. Right, cool. Yeah, I I I'd just keep you know,

I mean are you on our Slack? Um registered with our Slack channel so you can see Not on the Slack channel, but I'm subscribed to the corresponding GitHub issues. So >> Okay. Really? You'll you'll know that. Yes. If you if you keep an eye out for you probably know Maarten who who works at Microsoft. He'll probably or George. Probably you see messages from them. Thank you very

much. >> Thank you. Right.