JDK 25's new CPU-Time Profiler, Johannes Bechberger

This talk by Johannes Bechberger delves into the significance of profiling in software development, particularly in the context of Java applications. He shares his journey from an early project where a quiz application crashed due to inefficient data handling to his current work on improving Java profilers. Bechberger explains different profiling techniques and tools, highlighting flame graphs and the Java Flight Recorder (JFR) as essential for identifying performance bottlenecks. He discusses the challenges faced in profiling, especially regarding safe points in Java virtual machines, and presents a new CPU time profiler that captures performance data without bias. The session emphasizes the importance of profiling as a vital tool for developers looking to optimize their applications efficiently.

Hi, my name is Johannes Bechberger and I'll take you on a journey. Um, who of you has read The Hobbit uh, by Tolkien? Only one? Oh, good, more. Good. Now I know who is attentive. you should read it anyway. Um, when you ever read this book, um, in the beginning, um, the great wizard tells the hobbits what they should do and what the problem is and then

they try to solve it after over a couple of years and try to solve the world and destroy the ring. That's essentially what we're doing here, too. I'll be first explaining you what the problem is and what profiling is and then I'll take you on my journey. Um, so without further ado here, I'm starting with the simple question, what is profiling? Because we're dealing, of course, with

with profiles in the later part, um, especially with the new one we worked on. But what is profiling? Well, I'll use two, like, uh, what I did a couple of years ago during the Corona times, I had a task to build a small quiz application where people could answer for small conference that I had to organize, questions like, "How old is the universe?" on a talk on

the universe and so. In this case, I thought like, "Hey, it's pretty simple. I just have a client and an admin interface and they just pull regularly to the server and I'm I'm lazy. I just throw all the data in a JSON file. What could go wrong?" Uh, because to quote Donald Knuth, up- pretty much optimization is the root of all evil. You might have heard this

quote before. so the event came and any guesses what happened? Yeah, it crashed. The server was overwhelmed. Um, I was sitting there in the basement restarting the server all the time so it dropped connections. Because my problem was I misread the Moose quote. I only read part of it. Um but the whole quote is we should forget about small efficiencies say about 97% of the time. Premature

optimization is root of all evil. Yet we should not pass up our critical our opportunities in the critical 3%. A good programmer will not be led into complacency by such reasoning. He will be wise to look carefully at the critical code but only after that code has been identified. So how do you then how do we identify this code that we can optimize? We use a profiler.

And when I talk to my friends who aren't computer science people one of them, hi. No. Um they they think I'm someone like um Sherlock Holmes. Someone who goes deep into things and is like who did this crime? The only problem is I have get blame I know who who it was. Probably me. Um because I was developing this code. But anyway, what the problem was is

quite easy to save with a flame graph. So what a flame graph essentially does it's one of the main visualizations of profiling. On the you have bars that show you how much of the execution time was spent by specific method. For example, in our example here the main method spends all of the time of course that's the main method. Then we have a server loop that spent

most of the time so its bar is almost as wide as the main method. And then of course what does it do? It handles the question requests. This is the request that um the client does and the server receives like hey is there a question and if so give me it. That's what it does in this request. And that's all fine. The only problem is it spent

most of the time parsing JSON. It's not a good idea I can tell you. That's the reason why I was overwhelmed. To quote Mario Fusco, a lot of flame graphs can do something stupid, it punches you in your face and you cannot unsee it." And then as of this, I used MySQL database. It worked. The only problem is like Corona is over. Probably someone has noticed here.

And now I can't use it, but anyway, uh it's good for presentations. I I think profiling belongs to your toolbox like for example debugging, testing, and other things. So, uh yeah, learn profiling before it's too late, before you need it, not when your boss comes uh early morning and is like, "We have a performance problem. Look at it now." But what is profiling then concisely? I quote

here Jargon File. It's like with the IT jargon from the MIT. It tells you that profiling is essentially the way to produce a report on amounts of time spent in a program in each routine of a program used to find and tune the way to hotspots in it. That's pretty cool because that's essentially what we want to do when we do performance optimizations. There are of course

different profilers Um we're talking of course here of Java profilers, but they are either profilers that you attach externally to your JVM or you have ones that are built indirectly into your JVM. Externally, you have uh ones like VisualVM and NetBeans. Um also 1991, some people might remember for the analyzer was created by Sun and then they want to support uh Java, so they added a call

to the JVM, uh which we all later a little bit more. It was there for 3 months, then it got removed, but it was still kind of there when you were a bit hackish. And then I think profilers are to to use them in 2016. Then there's the building ones. Um the main one and the one that you probably might have used already is the J Cave

Flight Recorder, which is open source since 2018. And in JKFifty-Five recorder, we essentially have one profiler. It's the wall clock profiler, and we see in a bit what wall clock profiling means. So, JFR here is simple to use, and that's the cool thing. Like, you just uh say, "Oh, start a flight recording." Give it a file name. If you want to have a bit more precision precision,

just use these two options, but it's all in, and it's built in, and it works on all platforms. It's It's cool. It It's just got Use it. Cool. The talk is over now. Um It It works on all platforms, as I said. And it has so many events that they even created a website called JFI events, where you can look up all the possible events, because there

are over 100 events. Um many per many per Java version. yeah, you have events on compiler and laying on almost everything uh that the J developer likes. And it's updated regularly, so that's when you use JFR. It's not the worst place. then it also has custom events, which is cool. Um but it's not perfect. the main profiler, like the wall clock has slight issues in how they

are implemented. Because how it works, every and those seconds, typically every 10 to 20 milliseconds, we capture five stack traces. So, what you need to know is we capture them heuristically. So, we don't know actually how how long a method on the CPU, but we are only interested how how about like probably they spend so much time, because that's fine. Otherwise, we would have a too big

of a performance penalty. But here, it works every 10 to 20 milliseconds, We capture five stack traces or so. We take the next non-blocked Java thread from a list. So, JVM keeps a list of course it knows all Java threads, but we keep the next one that's not blocked. Then we suspend this thread. Then we tell it like, "Please capture me the stack trace." And then we

resume the thread and continue. And if you want to learn more, there's a fast and presentation here that Dmytro Ganyushin with Yaroslav Pavlyuk from DataDog. And it's pretty simple. And it changed later, but it has one big problem. And the problem and and the problem is you see it here it's a sliding window approach. So, um every 10 milliseconds we slide this window further. The The main

issue is with the number of threads the number of the effective [snorts] sampling period. So, how long between two samples of the same thread um scales linearly. So, there's no real relationship there um between the CPU time that the like that the that the thread spend on the CPU and the sample count. So, the data still not pointless, but I would say it's it's hard to reason

about this data and it's hard to really make judgment. That's a problem. But well, um let's try let's after you know the problem, let's let's try to solve it. So, how did I end up here on this stage? I know. Yeah, I I took the bus to the airport took the train to the venue and this was in JFocus earlier this year. Then I started baking bagels

and now I'm here. Uh no, um how did I really end up getting involved? Um well long story short, I started to work at a company. Some of you might even work at you might guess which this is is of course uh the the company behind Submachine and it's I I work as a OpenJDK developer full-time there. Um it's at SAP, it's like their Java distribution. It's

very cool. It's open source and I like it there and they mostly do profiling stuff. Um and we're one of the many open source projects there. And they are organized they're helping organize this company at this this conference here, so it's really cool. But um essentially um I dropped out of uni like from the PhD program because I was like working on open source science core. So,

it was my first job out of uni and I was naive. Like, I was starting there my my task was, "Hey, we have a problem in this async profiler with a customer. Your task is in onboarding now help improve it." For example, fix in the power PC. And I found that the API that I found there wasn't as good as I tell you in a bit. So,

it was like I can do better. Yeah, I was naive. 4 years later I'm standing here, so I'm still I I survived. Anyway, um you could also reframe this talk as one man's journey to improve Java profilers. So, essentially like The Hobbit um only with less dragons. in the following I I start with Yeah, in in the following um I showed in from where we came. Here

you saw it before. That's not good. It's case linearly. We want to go there. Ta-da. And this it's it's it's it's far better. We want to have something as CPU time profiler. That's eventually what he got and what this talk will show you later how to use it, but how do you start? In 2022 I was naive start this job. And below here you see the timeline.

Um um I first looked into the async profile, what's my job? That's that's nice. Um, and I wondered how does profiling work for async profile? So, I started giving talks on this. Um, so how it works in the most basic principle is kind of similar to in in many cases to what JFR does. Um, how the basic mode box it picks a few threads this time randomly.

And then what it does, it pre-allocates stack traces somewhere um, on the heap and then we use uh, on on Linux and on Mac we can use signals. when we selected them, we ping one thread and say, "Hey, please stop execute something in your thread context and in the signal and walk the stack." And it's pretty nice. We put it in the pre-allocated database and then we

push it into a database into a trace collection. And there it is, the main call used by many profiles up till recently and that's pretty cool thing because essentially what it allows um, is to take a native stack trace that we have on the CPU. As you see here, we have in the assembler location we have on top some native frames. we have some Java frames and

we have again native frames and we want to walk this and this method essentially allows us to walk this directly. We don't have to care about this, the JVM does it for us. It pre- it pushes these in the pre-allocated area and then tells us like how many frames did it walk and we can read it. What information do we got? Oops. What information do we got?

We get the bytecode index which is weirdly named line number. Don't ask me why, but it's like the bytecode index is like the position in the in the Java class file and it tells us also about the method ID. There are only a few problems and this is taken from these these talks the the slides that you see are mostly also taken from the talks that I

made at this time. But essentially it's unofficial, it's unsupported, it's missing tests and missing information. So, a really good API to base a lot of profiling tools on, isn't it? And even the even the one test that it has was broken for like 10 years because it didn't really check anything. Yeah, and that wasn't good. It was at the time the best that we had. Well, till

we have the JVM TI replacement that was my goal at the time. Nowadays, Roman can get data dog stack traces again. Um, but my idea was I was naive why not replace it with a new call, call it async get stack trace. It even got a check candidate. So, yay. That went far. Like In this talk I show you two jabs, not only one. Yeah, it didn't

it it didn't go further than than a candidate and stuff. But essentially it was an API to asynchronously walk the stack trace. And my idea was I want to have more information. For example, why not have for the frames have the information on the BCI that's known, known as BCI, have a method ID, know where it's in line because that's also relevant information for you but at

the just-in-time compiler the job, how much it was compiled at the time, and also it should hopefully be able to walk native stacks just because I can. Yeah, it would be really nice, wouldn't And of course at the time I also did other stuff because like working on the Tripper's hours, especially if you're naive. So, I started working on a profile front end, which is which I

just released a new version. It should be coming to the JetBrains Marketplace hopefully tomorrow. but it essentially integrates, the Firefox Profiler, a custom version of the Firefox Profiler, into your IDE, so you can have a free and open source profiler plugin for IntelliJ. Um which I think is pretty cool. I use it all the time. Uh you can use it, too. It's free. I have a QR

code at the end of my slides, but also here, so yay. Uh and if you have bugs, please report them. It's just open source. So, yeah. Um and and can also visualize profiling data. But it was 2022. That was nice. One year of working at the company. Um Yeah. Then I start to get slightly desperate because I run into problems. This led me to give a talk

at another called Nightmares of a Profile Developer. Johannes Bechberger's Gruesome Tale of Java Profiling APIs. Yeah, I was probably in a dark mood at the time regarding uh my work. Um I show you now why I was in such a dark mood. The problem is here. The line number is fine. We can access that it's an integer. But what's with this method ID? The problem is whenever

you have any class unloading or some some things more difficult with class loaders. It makes boom. Essentially crashes your It's not good crashing your JVM when you profile it. Like it's really not good. Every time I found a crash, I'm like, "Not again." Then and the problem it it takes some time to trigger the crash, so you run it for like a day and I'm like, "It's

fine. It didn't crash after some of my fixes." And then minutes later, it's like it crashes and I'm like, "Darn." Anyway, uh time to develop a profiler plugin um and start doing conferences. In come the safe points. Um What's really important here and why there is so much boom um, is because of safe points. Um, or because of not using them at the type. What safe point

is is quite easy. Um, you have here our server loop from before and at certain parts of the program, um, the JVM just asked, "Hey, in the execution of Hey, should I stop this program? Like should I stop it here? If so, stop and it can do things like garbage collection. Um, should I stop it here? It's at every method call, every start of a method. Um,

and also at every loop back edge, every time we jump back from a loop. And if it's And that's pretty cool because we can do, as I said, things full GC and other things where the thread should be stopped and the thread should be in a control state. So, the thread knows what's going on. So, it's not just interrupted at any point in time. Um, how it

works is quite simple. Um, it it uses seg faults. So, your JVM seg faults on purpose a couple of thousand times a second. And that's just normal. It's It's fun when you debug a real seg fault because debuggers hate this trick. So, essentially how it works, they ev- um, safe points are checks are regular, but um, really going into a safe point is pretty rare. So, how

it works is this check just um, accesses a page with address is in the current thread's object at polling page. And when the safe point is disabled, it's all fine. Like it's a good page, we can just move some bytes over there, it's fine. But if it's disabled, we just link it to a bad page, a page where we can that we cannot access. And then we

get a seg fault. It's It's safe and the seg fault [snorts] handler the JVM just distinguishes between the case of having really a seg fault and having a stack fault and purpose, and it's fast. But anyway, then safe points are fine. Um and we could think, why not just sample at these safe points? That would be fine. We would have less booms, no booms hopefully. Um it

it will be probably it's not that fine. Because consider this here, this is our JVM, and every time it gets darker, we are at the safe point. And in between we're executing. The main issue is here safe point bias, um consider when we're using safe points, and we want to stop the JVM to uh to profile, to get a stack trace. We just tell like, please stop

when you're ready, and then it stops. That's not good. There's a time delay. It executed code between, so you have a bias towards seeing safe points, and you mostly only see safe points um the location where you have safe points in your profiling data, and that's not really nice. Because when we're fully asynchronous, we just tell the JVM, stop. Like the German cliche, we just tell it,

you stop now. And it stops, and that's fine. But only problem is it can crash, and that's not fine. yeah. Now Now the issue is, to to quote Nitsan Wakart, safe point bias profiles are tricksy, sneaky, Philly, all of the above. So we don't want to use like we don't want to profile at safe points, because it's bad, because it gives us bad profiling data. But if

we do do it do the sampling outside of safe we have problems with crashing the JVM. Kind of. So what do we want? uh what do you want? We want unbiased safe point based stack walking. After what you heard before, it does sound like something that doesn't go together. like we want to have the cake and eat it, too. And that was a goal that I had

because I felt like that's the only way we could solve this problem. And this came about I was working on the whole topic and then it was for some user groups in Milano in Italy and I sat like outside of the Milano train station in like the shades because it was May and was already hot. By the way, if you wonder where Goran Solar comes from, it's

the name of a suburb of Milano. Anyway, we were sitting there and then we were talking with Marcus Grellmann from from Oracle and what he proposed was why not walk the bits at the safe point and the bits asynchronously that we can do without crashing and then transfer all the work all the other work to the And it was pretty cool. So, how it essentially works is

we walk the native stacks because we can do this. Our JVM doesn't care. And we walk till we get to the first Java frame. What we do, we enqueue it into uh in into a local queue. Essentially, we store um the registers for the top frame. We restore like the program counter, stack pointer, and function pointer. And then what we do at the safe point, we take

the we take all these elements out of the queue and because we have a safe point every time at the end of a method, we can reconstruct what we have because and we still have all the method metadata, all the method debug information. So, what we can do, we can reconstruct all the other frames. And that's pretty cool. So, we essentially solve this problem of like, yay,

we we walking the stack at the safe point, that's cool. Um but we're still getting the non-safe point biased information. that's pretty cool. Um the frame reconstruction is a bit of work, I can tell you and it's not that easy as it sounds, but it's possible. And I many more ideas at the time. And but the only problem is this this chip faced I would say slight

opposition. you could say it was like, uh, yeah, um, it's a problem when working on source. You have to convince other stakeholders and many people just didn't like it. And it was a problem because that meant that this chip never got in. It was canceled. That was a bit sad, but that's how work goes. I'm happy that I'm paid for all this, so like if it would

be a free time project I would have been like but at least like it's cool when you're working on open source at SAP. I was like to my boss like, "Yeah, it's not getting in." And he was like "Okay." Good. Like what can he do about it? Nothing, sadly. Um, yeah, so it faced some opposition someone wants to contribute to OpenJDK. That's one of the problems if

you're not employed to do this, like when you face it might be even worse than when you're doing it full time as your job. Anyway, and then 2024 came. I started to speak at more and more Anyone remember recognizes this street? Or the congestion? Okay, it's from Bulgaria. I was here, oh, you recognize it, cool. Yeah, yeah, it's it's it's not far away, I believe. Yeah, it's

here at the roundabout. Um, I made this picture 2 years ago when it was also a trip from also talking about profilers. Um, so some of you might have seen a couple of the talk slides. Um, but yeah, I spoke at conferences, uh, enjoyed the traffic jams in the middle of Sofia. That's the Sofia tourist attraction, I heard. Anyway, um then at FOSDEM, um during this year

at the local FOSDEM, I had a chat with Ron Pressler. Um he is someone working closely with the Java architects and Mark Reinhold. we had a chat and he was asking like, "Hey, um what do you actually want with this JEP? You won't get this JEP in. Like, we don't like it at Oracle. Like, there's too much opposition." And I'm like, "Yeah, what? Like to have a

CPU time profiler because like I really like to use it on on on many platforms." He was like, "Yeah, why not just start implementing this and we look through it afterwards." Well, I'm like, "Fine." He essentially told me like, "Hey, um how about you give the Oracle to your team gives the Oracle's to the team of your engineers, kind of like lends them, and we tell you

together what you could work on and how we could facilitate it." "Well, my boss was okay with this." So, we we worked together on this and that was really cool. And so, in comes the So, the main idea was every 10 milliseconds a thread spends time on the CPU, we get one sample. That's pretty simple. The relationship between the CPU time and between the between the sampling

time and the number of threads is linear. That's pretty cool. Uh it's constant. And so, essentially what we're doing, we're ignoring any weights. Um sometimes you need to recognize the weights, especially when you're doing um stuff that is latency critical, but if you only care about the throughput, you mostly can ignore um the wall clock time, but you you're interested what really waste your CPU. So, yeah,

as I said, with the number of threads, the effective sampling period in CPU time doesn't increase because the thread is still just spending 10 milliseconds between two samples um on the CPU. And that's pretty cool. So, now we have uh pro we we have a second leaf on the tree. It's a CPU time profiler. And I put a wall clock profiler in air quotes because as we're

doing some subsampling, we're only picking like five threads. This is not really a wall clock profiler, but that's a whole different topic. Um Anyway, um let's look at an example. And that's actually the example that I got from the um from the uh JEP um from the JEP enhancement proposal. So, how it works with the OpenJDK enhancement proposal is if you have an idea that is like

slightly bigger, you want to have an improvement, you need to propose a JEP enhancement proposal. And it helps if you before you do this, you have support from the from important stakeholders in the community. So, don't start writing JEPs and hope they get accepted. No, they won't. Because you you need support from from the author folks, you need support from other people. And also, the community needs

to know that you are able to support this over the next following years. And that's far easier when you work as a full-time employee at one of the big OpenJDK contributors like Oracle of course or Red Hat or Amazon. But anyway, um in this JEP you also describe how what the what your tool solves. And here what I did, um I implemented a small sample server. And

it's a pretty standard server, and we won't use it. We only use the client side, essentially, we have a server that has two endpoints, a fast endpoint that returns in 100 and a slow endpoint that returns in 1 And then we have some requests, and we do them like we either do 10 fast requests, or we do one slow request. What one would expect us in a

profile, we clearly see that of course we're doing much more work for 10 fast because with the one slow request we essentially just waiting. So, how can you run it? The code is is of course in the chat, but you can just tell it like, "Hey, please plus one." And then with with the normal JFR, it looks this way. It kind of makes sense because it's a

wall clock it's still kind of pointless because the was just waiting. >> [snorts] >> Here, and so the one slow request and the fast request looks the same. Um but with a CPU time profiler, it looks different. It looks actually this way, and it's like a simplification of the real stack trace, of the real flame graph, but we only see the 10 fast requests because the one

slow request was so fast um in CPU time that we never caught it. Yeah, because it's kind of the it is it uses heuristics, so it's it gives you an approximation of what happened in the CPU, and sometimes it works, sometimes um it doesn't catch it. But, you would if you run with high sampling rate expect something like this. But, one slow request takes around 1/10 of

the time of 10 fast requests. So, yeah, as I said, it's kind of sampling, but it's fine. So, this is really helpful for finding computational bottlenecks, um but that's mostly what I do when I optimize my profiling applications. They used to profile other applications or things like this. I'm interested in where the computational body like slides. So, this is really helpful. Especially it being built in into

the OpenJDK. Um, yeah. Implementing the trap was hard. Yeah, I know. Who could have guessed that? Took me only 4 years and I thought it it's I wasn't even It was my first real project out of university. So, now in 2025 was last year. I had some advantages there because they they they were happening some good There was the trap 518 coming in. Um, that's called JFR

cooperative sampling. So, the idea that Marcus Greiner had with like, "Oh, we only we do um, unbiased safe point based stack walking." He implemented this for JFR, like for the normal workflow like profiler. And that's pretty cool because he did all the heavy lifting. So, I didn't have to do it. That was nice because I was running into crashes. He didn't because he's a much more capable

engineer working in this field for far longer than I was at the time. And so, I could just build upon the foundations. It was well nice. >> It barely got in. You probably see if you if you look closely, you see here that it was updated in 2025 in in in June. It barely got in. For for the 25 release. So, Belly is also mine got in

a little bit later. Um almost too late, but anyway, no, mine is then uh the road was and I finished it almost at time, but still had to wait till the other chip was integrated. Um mine was then the chief RCP U time profiler. You can find it as chip 509 uh on the website and it's pretty cool. Like writing a chip is is cool when you

finish it out you're like finally, especially if you tried it first and it failed. Um so, what's the basic idea here? It's pretty simple. So, we register an item as our sentry um on Linux and it's sadly only possible on Linux. We can register um in C++ or C uh something called an item where essentially the operating system pings us back or calls a signal on the

thread every time the thread has a new has has run a new 10 sec 10 ms or so um time slice. So, essentially every 10 ms of CPU time per thread um we go into a handler uh a signal handler and we capture the tip of the stack as I showed you before and push it into a thread Um Um the only problem is um these queues

need to be statically pre-allocated because that's how it goes with signal handlers and so we can also lose some events just because the queue is full and we check this with a separate um with separate event. Um I'll show you the other events But essentially what we then do is at a safe point we reconstruct all the stack traces and create events. That looks eerily similar to

what JFR does. Like normally just with the register item stuff. And that's and that's pretty Um and it works surprisingly well. The >> Is it enough to only walk at safe Yeah, it's a question where the the answer is usually no, and that's also here the problem. Consider you have a lot you have a a big native method that does something expensive. I don't know, calls into

your database and it takes a second or so. We have one safe point at the beginning of the native part and one at the end. Now, the problem is when we only sample at safe points and we're still sampling while we're in this native method, yeah, it's it's not great. We're we're filling up the queue quite quickly and then it flows over and we lose samples. But

we when we can do here, um every time we sample um when we know the thread is native, we have a separate thread that runs alongside and it's like, "Oh, yeah, here. I need to uh clean up the queue." So, it does the full sampling um in a separate thread, but it works because at the end safe point here, we wait till um our other thread is

finished working. So, we aren't working um while the travel thread is like moving forward. And that's very cool, and it's also works. The whole implementation is slightly more difficult. It's only like 2,000 3,000 lines of nicely crafted C++ code. And that's the whole architecture diagram. Essentially, we have on top we have a Java front end that does all the uh parsing of when you when you pass

arguments and stuff. And then essentially, we have our CPU sampler thread. And our that's like the the thread that also does the the sampling while we're in native. And we have like the queue and these are like the big things that we need to do. It works. It works surprisingly well. Um haven't seen any bug reports yet after like a year or so that people might have

used it. Yeah, but it got barely in there in time. Uh Um uh essentially Marcus Green noted approved this pull request to that pushed for the for the JEP. And he essentially said, "It's it's it's probably good enough, and it's anyway experimental, so nobody will use it." if we don't get it in JDK 25, that's not good because the problem is most of you don't use don't

use non-LTS versions of the JDK. And that's And that's fine because for all the LTS versions, you can get support and everything. But, it also means when I get it in JDK 26, there's no difference between me getting it in 26 and the next LTS versions in essentially how many people would see Cuz that's not great. Um it got so The The thing in the end is

got so short in time in that Nicolai Parlog, one of the devrels of Oracle at was posting on the his Blue Sky or or Twitter account um where I was saying like, "Thanks for mentioning um my JEP in his video on the new JDK version." And he's like, "Thank you for pushing it through last minute and making me work late." Barely got in on like the day

before the final cutoff. Do you know the when you get this email when you get this text message at 3:00 in the morning and someone is like, "Hey, hope you haven't gone to sleep yet." I woke up at 7:00 in the morning. I was awake quite quickly, I can tell you. Essentially, we had a problem that there was a mistake in some There was a test failure,

and there was a bug in my implementation. A configuration bug, but still Yeah, I don't know why we didn't catch it before, but it but they the test But, yeah, it was fine. Um Marcus Green noted proposed a fix. I fixed his fix and applied it and everything went well and he got it. But still, it was that was crazy that was slightly crazy, especially um, like

before on the day before um, was working like in the evening with a friend and then uh, several people texted me on Slack like look in your emails and uh, Ron Pressler told me like you have to click this one button in Jira to make your chat go in and I'm like, okay, running back home, clicking the button, going further with my work. It was quite funny.

So, um, it took 4 years but it then went fast. Um, but of course I wasn't alone. Um, I was working together with a lot of cool people in the OpenJDK. So, it's really nice in the OpenJDK with big community. not really big compared to the rest of the Java community, of course, but probably like 90 to 100 people that work together, that help each other out.

Um, yeah, working there is quite it's it's a good a community. But of course it's also work. It takes time. recently I'm not that active anymore. There I'm I'm working in both of the stuff, but it takes a lot of time, especially when you have reviewers that are on the other side of the world and you have to kind of communicate with them, have to arrange meeting

and everything. Um, if you want to contribute then I can really recommend um, I have here the link. It goes to the dev.to page um, where you can simply find where to contribute and what to do. Essentially, you could probably when you want to contribute, contribute by um, trying to find issues with documentation or when you have a bug and find something in the standard library, you

can also contribute. Mostly what you would do, you would go to the mailing list and um, and ask there. But of course use the bug trackers. But of course when you know someone who is a JDK developer, just ping them. In many companies, like in the big ones, you already have JDK developers, which is pretty nice. Yeah, um of course this work also a lot of compromises,

as you saw with my chip. I want to get like the ultimate good solution, and now I'm here with the minimal solution, but it's still fine. It's compromise, and um I hope our customers enjoy it. Yeah, and that's the CPU time profiler. Now in last bit I show you how to use um from this example in the beginning, uh from this chip example, how we use it,

it's quite simple. We just start the flight recorder, and what we also do, we have to tell that the CPU time sampler, the CPU time profiling is enabled. It's disabled by default. It's experimental, but it doesn't mean it's not usable. It's still production It's already production ready, and might probably not go away anytime soon, but still it's it's there. And then you can configure it. Um you

can of course enable it, or disable it, but you can also tell how many events it should produce. You can either say how long the interval is, for example 10 milliseconds, or you can say how many events at maximum it should produce every second. How this works is that we When we When you know the number of cores of your system, which you know, you know how

much CPU time your program can maximally produce. So, for example, if if you have a 10 core every second you can at maximum have 10 events when you sample every second of CPU time. So, this this way you can give an upper bound. That's nice when you're profiling longer time on systems that you don't know. And then you have the CPU time sampling event. It of course

gives you a stack trace. That's very nice. It gives you a thread, but it also tells you where that Like whether it couldn't walk the second type. And also whether it's biased, like whether it failed in a way that you still get a stack trace, but it's biased towards being at the safe point. Um and you get a sampling period if something changes. The sampling period essentially

comes from the operating system. And you get information of the CPU time samples lost when the queue overflows. The only problem is besides it being experimental is that it only works on Linux. That's a problem because I'm using as you might be seeing here a Mac, so I can show you sadly live demo. I could have gotten I could have brought my my Linux laptop, but I

didn't because I'm only traveling light and two laptops are quite heavy. And they look weird when you go to through airport security, especially when you have two phones with you. it's only available on Linux because only Linux has these capabilities. Yes, you could kind of simulate it on Mac or Windows, but they sadly don't offer that's all what I have to say here. If you want to

find my JFR profiler follow plugin, which is actively developed and gets better every day, um essentially because I'm now fixing bugs that people put in since 2024. Um but hey, uh it has no new front ends and it's usable. And if you want to learn more about the whole CPU time profiler and about implementation, feel free to look at this link where I wrote like three or

four blog post long series on the You find myself on Mastodon and on as part of us um as and me at Nos. as mostly not as dot d e on blue sky I regularly blog and yeah. You find my team at sub machine dot io and that's all for me here. Thank you. >> [applause] >> We have probably time for two more questions for two questions.

Does anyone have one? Yes. What? What is the overhead? It depends on how it depends literally on how small the interval is. When the interval is like with 10 milliseconds it was like two to three percent so it was quite minimal. With this project Panama does it work well with foreign function interfaces? It should work just normal. Yes. If I'm honest I I don't know but it

there's nothing that speaks against it because JFR works with it. Sadly you don't see the native portion of course in these stack traces. There are new approaches to improve this but sadly there was some opposition in adding this in. Any other question? Um with with virtual threads it's the new ones are good because they can work with them but the main issue with virtual threads is that

we only see the carrier threads. We aren't profiling the actual virtual threads. So you still see the carrier threads so you kind of seeing a bit where in which virtual threads you do most of the work, but working with virtual threads and profiling and even debugging is is an interesting issue. I think our tooling support for virtual threads is just not there yet. Especially because many people

aren't using it yet. Um but no, it has no specific things for virtual threads. You I think you only see the frames on the when they're uh on the carrier threads. Any more? We have time for one more question. If we don't have questions, then thank you for being here. Have a nice conference. You have two full days ahead have a good time here in Sofia. Thank