My code is faster than yours... let me prove it to you!, François Martin

This talk discusses the performance of regular expressions, commonly known as regex, and their potential vulnerabilities, specifically focusing on the ReDoS (Regular Expression Denial of Service) vulnerabilities. The speaker shares a personal experience of dealing with user-controlled input that led to performance issues in a customer project. They propose several solutions to mitigate these vulnerabilities, highlighting the benefits of replacing regex with imperative code in certain scenarios to improve performance. Furthermore, the talk includes an introduction to microbenchmarking in Java, showcasing how to set up and conduct benchmarks accurately using the JMH (Java Microbenchmark Harness) tool. The speaker emphasizes the importance of understanding performance distinctions through controlled testing conditions and shares practical advice for conducting effective microbenchmarks.

Okay, welcome everybody to the next non AI talk. Uh, next up is Francois. Uh, who didn't answer to none of my emails, so I was wondering whether this is an AI agent or not. >> [laughter] >> Obviously it's not, so Francois is here and he's going to prove you that his code is better than mine and then yours and then everybody's. The stage is yours. >> Thank

you so much, Ivan. All right, and um, you know, with this great introduction, I also want to give a thanks back because just like Mencat said today, um, this is really a big effort to organize a conference like this and uh, there are a lot of people that are involved, the organizers, the volunteers that are here helping out and um, you know, it takes a lot of

time and effort and I feel like we should definitely appreciate that. So, yeah, big thanks for the for all of the ones that helped out. So, regarding the title, you heard it, it's my code is faster than yours, let me prove it to you. It isn't specifically you that I'm meaning with this. I'm pretty sure, especially after this talk, you will be able to do this. Um,

but this is actually based on the real story. I don't know who of you has read the abstract of my talk. I think maybe five people, six people. Yeah, so the rest of you, um, if you want you can read it, but um, you will basically see this throughout the talk a little bit, um, you know, the story. Um, so, I'm going to talk first about the

performance of regular expressions. So, who of you has used regular expressions or regex as they're often called before? Okay, almost everyone, so that's good. And who of you really likes using regex? Okay, I see like 20 maybe 10 20 people. Okay, that's great. I mean and who of you likes writing regexes? Okay, two, three, four, five, six, seven. Wow, that's more people than at most other conferences

I gave this talk before, so great, yeah. Um, I hope I'm not diminishing any of your joy at the end of this talk, you know, otherwise I'm sorry. Also, by the way, I'm from Switzerland and I brought some Swiss chocolate and some stickers, so you can get some here after the talk, you know, feel free to just get by and take some. So, yeah, um, you know,

this is my gift to you. Not not that many chocolates though. I didn't wasn't sure if they are going to melt, you know, thankfully with this weather, you know, that we have today, it's fine, so we're kind of lucky here, so So, I'm going to show you a piece of code that I stumbled across um at some point in a customer project. So, I'm sure everybody of

you can tell me in an instant what this code does, right? Well, yeah, I'm just kidding, of course. I'm not expecting you all to understand this. So, this is actually, you know, if you put this input into it, this is the output that you would expect. So, as the name says, it removes illegal characters, and illegal characters meaning there are certain characters you want to remove from

the string. And you can see if I have this input where I have tons of different, um, you know, weird characters in it, then it will kind of normalize it in some way and replace it with dashes. This is pretty much what the code does. So, as I mentioned, um, I faced this code in a customer project, and, um, so this was flagged because, um, we had,

uh, Sonar, which is a static code analysis tool, and it said that this regex would be vulnerable to a ReDoS vulnerability. I don't know who of you has heard of ReDoS before. Uh, Uh, one person? Yeah, I think two two people maybe? Yeah, two people. So, the rest of you, you can see the ReDoS here is underlined. So, if you scan the QR code on the top

right, you can get the slides and then you can just click on it and you can see more about it and how to prevent them so on. But, essentially it's kind of something like DDoS, like distributed denial-of-service, if you've heard of it. But, essentially the Re stands for regex. So, certain regexes are vulnerable to this ReDoS vulnerability. Where if you have this ReDoS vulnerability, it will mean

that um when you have a specific attack string, like you can see here in the example, um then this could lead to the server taking maybe 40 seconds, 1 minute, or longer to process your requests. And this way you can pretty easily apply load onto a server. You don't need many parallel requests to really make a server be really busy and not do anything anymore if you

have a vulnerability like this. And this is specifically relevant when you have code that gets input that is user controlled because, of course, you know, if the user can pass in anything, they could pass in this attack And um you know, obviously we needed to fix this vulnerability, right? Because, you know, this would be really bad because in this case this was user controlled input. So, yeah,

this was pretty bad. Um So, um you know, I needed to fix it and what I did first was I I've written a unit test um I didn't really fully understand what this code did. And even if I did, I, you know, didn't dare to just, you know, replace this with something else or think about how to fix it um before I made sure that it really,

you know, I captured how it works and that it keeps working in the same way as before because there were no unit tests, believe it or not. Um make sure if you use a regex, I can tell you this from experience, always uh >> [snorts] >> you know, write some unit tests. It really helps a lot. So, there are a few ways that you can fix ReDoS.

So, one of them is you can or is like the easiest or usually best option is if you replace the code by just writing it with imperative normal code that you would write, you know, like with a for loop or while loop or whatever. Um, there's another way. There are certain regex engines that you can use who, you know, which then recognize certain attack string patterns. But

of course, since they're recognizing patterns, you can never be too sure, you know, it it probably doesn't cover, you know, absolutely 100% of the cases. So, and also in Java there is no, at least to the at the point where, you know, I went and get where was in the situation, there was no regex engine for Java that supported, you know, this kind of ReDoS prevention. So,

what I would have had to use JNI or something like that then that would have been really painful. So, I didn't really want to do that. Um, one other option which sometimes works, but it depends of course on your use case, is that you could try to um, replace the regex with a different one that does the same. Because in some cases, it's just that someone introduces

a ReDoS vulnerability by accident because they think that um, you know, this would be the way to write the regex, but maybe they don't know how to write regexes, you know, properly. And then they might have a regex that does work, but maybe it does more than it should, you know, maybe matches for more characters than necessary and so on. So, yeah, these are the different ways.

Um, I decided to go the route of replacing it with um, regular code. And this is how it looked like. So, I don't expect you all to read this. I'm aware this is really big, but um, yeah, just, you know, understand that this kind of replicated the same functionality that we have seen with the previous two regexes. And so, understandably, you can imagine when I submitted this

as a pull request, my colleague came to me and told me, "Well, are you sure we cannot use some regex engine that maybe interrupts the you know, like finds when there is a reDoS or something like that?" I was like, "That is so much code, you know, like the other code that we had before it was so short. And now it is so long." >> And then

I first tried to argue. I said, "Well, but you know, I don't know. You know, who of you finds this to be really readable or really understandable?" Okay, maybe one person, two people, three people. Yeah, but um I'm pretty sure, you know, if you were able to read this, this is just a set of while loops and for loops um essentially. You know, this to me makes

it much more obvious what happens in this specific code. I see some people disagree, and that's just like my colleague. And my colleague also said, "Well, you know, I feel like this is much more complicated." And I can see it. I think this is kind of like a style decision, But the other argument I had made is we actually also had performance issues in our application. And

um I then said, "Well, but you know, the performance of regular expressions is also worse than the performance of regular code like this." And he didn't agree. You know, he said like, "What? There's no way that this is faster than the regex." You know, he said like, "You would need to prove this to me." And so I did. So, thankfully, I already knew how to do this,

and I could show him. But then I realized maybe, you know, I had situations also when I had someone else be in a similar situation that I was. And then, you know, I asked them to write a benchmark for it, but they, you know, didn't know how. And they said like, "Oh, you know, it's just fine." Or they used what many people do is they just put

like a uh system nano time, and then they do this before the operation and after, and then they, you know, calculate the difference, and that's how they you know test the performance of of something. Um I hope by the end of this talk you will not do this. You will learn to how to do it proper way. I wrote the benchmark just like I mentioned and then

I ran the benchmark and these were the results. So as you can see here the the light gray bars are with the code that I've written where I'm iterating through it you know so without And the dark gray bars they are with And you can see there is a pretty big difference. So the difference is at least four to eight times um the performance. Um which is

pretty substantial. And also by the way you see on on the on the Y axis it has this replacement character ratio and string length. So replacement character ratio means that there is one you know illegal character per this many characters. And this means essentially if you have a lower number there you know then this means there are more illegal characters in a string. So more things to

replace essentially. And this is what is before the pipe. And then after the time pipe you have the length of the string that we are using to um you know performance replacement on. And uh one suggestion I got when I gave this talk at the previous conference was to maybe try to um you know instead of to switching to this full approach uh you know in terms

of the performance improvements to instead just um you know compile a pattern for the um you know for the regex in advance and then just save it which is the best practice in Java which the person that has written this code before didn't do. Um but I tried it and it didn't make a substantial difference unfortunately. It was a little bit faster but not by much. Um

this really uh made a huge And essentially this is how I convinced my colleague to um you know accept my change somehow. Um yeah. So, who of you can write Java code? Okay. I would say maybe like 90% of the audience. Okay, interesting. So, um who of you that didn't raise the hand before can write JavaScript code? Okay, so five to 10 people. So, this talk will

be mostly focused on Java, but I have one kind of example that is specific to JavaScript. It's just that in JavaScript it's a bit easier because you will see why, but in JavaScript um you know, you don't have as many gotchas as when you're doing microbenchmarking with um you know, Java. So, essentially microbenchmarking by definition is um you know, about measuring the time or performance of a

small to very small building block of you know, real programs is the focus here. And also there uh this can be a common data access pattern, a sequence of operations, or even a single instruction. So, um it is actually often a single instruction or just not that much code, you know, just maybe one focused method maybe that you're testing. Specifically you're not testing for example an entire

application or a very big part of the application here. And the state of the art tool here to use um in Java and other JVM languages is called JMH, which stands for Java Microbenchmark Harness. Who of you heard of that before? Okay, I think maybe five to 10 people. Who of you has been at uh Kai's talk yesterday? Okay, that those are almost the same people, so

maybe you know, you heard it there because he mentioned it there that uh you can use it to microbenchmark. So, uh here the promised section on microbenchmarking in JavaScript. So, I have some examples here. For example, jsbenchmark.com is a website that you can use to do benchmarks with um, you know, JavaScript. And here I have linked an example that I've written on sorting an array of integers

where I was curious um, you know, which way is the fastest to do this in JavaScript. And then here we have another one where we have um, a website called perf.link. And here I have a regex benchmark um, that you can try out if you're curious to see how it performs in the browser. And then kind of the gold standard, but you know, maybe not the best

option depending on what you want to do is benchmark.js. It's from 2016 and it's unmaintained, but it's still working surprisingly well. And um, this is still like, you know, if you want the most accurate results, it's the best way that, you know, we have so far, unfortunately. Um, I mean, if any one of you knows of any libraries that are out there that are, you know, like

better maintained or like newer, let me know. I know there are some that are really small and not used that much, so you know, I trust more the established standard here in this case. But that is just something you could use if you want to do it in JavaScript. Um, it's not that hard there. Of course, there are still some some gotchas, but um, you know, uh,

and with Java you will see there will be many more. So, the question is when should you use microbenchmarking? So, microbenchmarking, unlike profiling, is best used when you want to compare the performance of different implementations. So, for example, you can see with the example I showed you with the regex, then we had the case where we had the regex and we had at on the other hand

the you know, the implementation with iteration. And I was curious to see how the performance differs between those two. So, um, and this is usually also under controlled conditions, so you specifically, you know, um, try to reduce the the surface to exactly what you want to test and only that part. And um, if If want to understand and improve the performance of real world applications in real

scenarios, then please use profiling. So, usually the way that you go about this you use profiling to find bottlenecks in your application first. And then whenever you found those bottlenecks, then you can um use those bottlenecks to try out and uh implement some different varieties of implementations. And then you can use microbenchmarking to check, you know, how the performance difference is depending on the implementations that you

um you know, are considering. you know, I cannot make you an expert in JMH of course in 15 minutes. But what I'm my goal is today is to at least, you know, walk through setting up a JMH benchmark with you step by step to show you uh that it isn't as hard as many people believe it to be. And um you know, in just five little simple

steps um you will be already at your first microbenchmark. so first, what I would recommend you to do is um read the readme.md file in the JMH repository closely. Um you know, this makes sure that you get the most accurate results. also then you just run this command. So, um you just have to adjust those two things. So, the group ID and the artifact ID. Um if

you want to have it even simpler than that, um you can actually when you um scan the QR code, um you will also So, you will get an email from me and there you will also get um a link to a repository that I've written where essentially already get um this command executed as a, you know, as a basis that you can use. So, you just need

to It's a template repository in GitHub. You can just do one click and then it will already have a um you know, a your own repository on GitHub where you can start writing microbenchmark. This just makes it even quicker for you to to started. And there I also have uh the code that I used in this uh talk. So, for example, for the first example that you've

seen and the microbenchmark for that and the microbenchmark that we're doing together here as reference, so you can use it if you want to get inspired. All right, and then if you run this command, this creates a folder with the same name as the artifact ID, so make sure you don't already have a folder that has the same name, else you have to go into two separate

folders. And open your folder in your IDE. So, the next question we have to ask ourselves is what should we compare? So, this is one of the other cases where I used microbenchmarking in the past, you know, you can see this string here. Does that seem familiar to anyone? Okay, I don't see not not not that many people, but okay. So, this is called an HTML encoded

string. And essentially, if you decode this, this is what it says. And I had a case where we had to in a lot of different things with a lot of different data, we wanted to actually decode those strings. And that's where performance mattered because there were really a lot of them. And you know, this is usually where performance adds up. you know, I saw there were so

many different libraries available for decoding and I was wondering, well, you know, which one of those is the fastest because you know, you know, there's so many different choices. Like we were already having a Spring Boot application, so the Spring web one would have been my preferred choice, you know, because I wouldn't need to add any dependencies. But of course, if it is like, I don't a

lot faster, you know, might have considered to also add one of the other libraries here. So, And so, the question for me was that I wanted to answer is which one is the fastest when decoding lots of different You can see here the emphasis on decoding lots of different strings because as I mentioned, the microbenchmark has to be highly focused so you need to make sure that

really your microbenchmark covers exactly the question that you want to answer because if you have for example a case where you want to encode text then you know that it would look totally different and also if you want to not you know decode lots of different strings but just one string again this can look very different in terms of performance so keep this in mind think about

what do you essentially want to test here. And of course this is kind of optional but I really recommend you to do this um you know think about what is the expected outcome you have and why. So for example I knew already when running the regex benchmark that regexes were in general a bit slower because regexes are just you know general purpose mechanisms that you can use

and then if you use targeted code that performs something for you um you know in specific instructions then this will be more efficient than a general purpose you know I don't know how you can call it not really algorithm but I think you get what I mean. And this helps you compare when we get the results afterwards to see is our benchmark kind of what we expected

or not and if not we can better figure out why is it not the way that we expected And also you you maybe have seen that I said which one is the fastest but fastest kind of depends on the context that you are in. So you really need to know the context of where the code is used in order to have a really accurate microbenchmark. So you

should try to reproduce the production conditions that you have as closely as possible and this means even for example if you can use real data that you observed in production for example or anonymize it in a way that doesn't change you know it too much because you know we also need to be careful because if you anonymize it then you could change the way that microbenchmark behaves

and ideally it's even better if you can to benchmark on your production hardware with the same JRE because of course with different JREs from different vendors, there will be performance differences as And of course, different versions, you So, and I can tell you from experience depending on what data you use, the performance can vary significantly. And uh one example that I can give is um you may

know quicksort which is a very popular sorting algorithm. And it has generally a complexity of O of n log n uh which is one of the fastest sorting algorithms in most cases. But um what is actually the case is if you have already sorted data, then with a simplified implementation of quicksort, you usually would have a degradation to O of n squared. So, it's a lot slower

um where sometimes other sorting algorithms would be better. Of course, what you can do is um we can sort the you know, the the array first by randomly sorting it before going through uh quicksort which solves some of those problems, but still if you want to have the best um you know, um algorithm for example for your specific use case, this is how you would find out

with by making sure that your data is the same. Um so, it you know, you need to make sure that if your data in production that you get is random, then make sure it is in random order as well. If it's mostly sorted, but just you know, one or two elements are different, um use of course, you know, this as a basis to benchmark. And then also

um yeah, just like I said. then step two is we are finally adding the benchmark. So, we have the basic um my benchmark.java file which is generated by the first step that we ran. And you can see here, it's pretty basic. So, essentially every benchmark that we want to add is we have a simple method and it's annotated with @Benchmark. So, it's actually pretty similar to writing

a JUnit test, for example, except that you add @Benchmark instead of @Test in front of it. So, you might be thinking now, well, so we can simply call the method now in this benchmark method and um that is it, Cool. Unfortunately, it's not that easy. As I already hinted on earlier, um Java compiler is really smart and it does really cool things, which is sometimes a bit

annoying during microbenchmarking. So, for example, in this case here, the Java compiler realizes that this is just a line of code that is there. It has no side effects. So, you know, why should it keep executing it? So, it just optimizes it away and this is called dead code elimination. So, this is one of the pitfalls that you need to be wary of. And you can see

here, um there's a link again for dead code elimination, which shows you like um a more detailed example of this. But, essentially, um every time you have code that could be optimized away because it doesn't have side effects or it's not needed it would strictly not be needed in this application, um then, you know, this benchmark would essentially run through really, really quickly. Um and, you know,

you would get wrong results, of course. So, if you want to just perform one of operations, so let's say you want to just test um uh you know, the band performance of um you know, decoding uh in one specific string, what you can do is you can simply, instead of uh returning void, you return string in this case. And then you just return whatever you um want

to measure. And then by returning it, we kind of trick the Java compiler into telling it, "Oh, you know, I'm using this because it's returned." So, it will no longer optimize it away. That's kind of the easiest way. Um if you need to kind of return multiple results, so if you have, you know, just like in my case, you have a for loop, as I mentioned, and

um you know, because we had lots of different strings we had to decode at once, then um what you can do is you can um like here I added a for loop, and you can add something that is called a black hole. A black hole is just an object provided by JMH, and you can simply use the black hole, and then all the black hole call consume,

and then by calling consume on that, um you will let, you know, because this is something that is in JMH optimized to essentially not have the um compiler optimize it away. Um you will make sure that uh this will be effectively run because, you know, it will realize, "Oh, I cannot remove this here." And um by the way, um just uh a word of caution. So, one

common mistake I see people making with micro benchmarks is they want to test different kinds of data, but they are only interested in knowing how the performance is of one individual string, um but with different kinds of data, which is different from, you know, having a use case where you have lots of different strings that you have in production that you need to decode at once. So,

if you have in production in your code just the case where you have to decode one string, and but you want to test different data to see if a longer string, for example, makes it slower, um then what you should do is um you should not do it this way by using this for loop because if you do this for loop, it will essentially measure the performance

of all of the different um you know, um snippets of text that you have. Instead, um you know, if you have a use case like this, click on this loops link, and then it will bring you to an example showing you how to uh do this on your own if you want to. so, if you in your production context are using a for loop and decoding HTML

uh you know strings like here, then make sure, you know, if you just have one statement, then use the same just one statement in your benchmark. But if you have for loop, then use a for loop in the benchmark as well. Um but yeah. And exactly, make sure you do not use loops to test different data. And again, here link to how you can do this. All

right. And then we also have a case where usually microbenchmarks are executed in multiple iterations. So it's usually not enough to just run a you know code once because you know, usually we say like measuring once is like measuring you know never kind of. So usually we say like in science you have to at least measure three times. Uh with microbenchmarks you have to go way beyond

that usually. So if you have state like this as you can another problem is that the JVM could perform an optimization which is called constant folding. And with this you know, it's just that the JVM realizes that it can just reuse the same result in each iteration because once it computed it once, it can essentially just you know replace it by calling the result that it computed

over and over again. Because here we don't have side effects as I mentioned. We have the same kind of strings. You see we have an array of strings. So after the first iteration, the JVM already knows what the result was of this method call. So it will just replace that with a constant. So your first iteration will be measured correct, but then the following iterations will be

very very fast because it just replaces it with one kind of um you know, constant. And this will make it a lot you know, very inaccurate of course because it seems like the code is a lot faster than it is. So don't do it like this. What I recommend you to do instead is there is a so-called at state annotation. So you can see here you add

this at state and there are different scopes. Um you can read um under the at state link specifically what kind of scopes there are, but in this case we want to use a thread scope. And here we just define a public static class that is, you know, you can call it whatever you want, but here I called it thread state. But what is important is you can

see in the method below that I had pass in the thread state instead of just passing in the black hole. And then now we can use the state and then JMH behind the scenes has some optimizations to make sure that it is not um you know, constant folded away by the JVM. So, again here um you will be safe if you do it this way. And my

clicker seems to Ah, okay, good. All right. So, and then we have different benchmark modes. There are tons of them, but I mostly use two and those are the ones that I want to show you today. Um so, one of them is a so-called benchmark mode which is uh mode.throughput. And this measures how many executions it can perform in a given time unit. So, for example, if

you set the time unit to nanoseconds, it will show you how many you know, times it can execute that method in, for example, 1 nanosecond. And be careful because um almost all of the benchmark modes um a lower score is better, but this is the only kind of mode where a higher score is better. So, make sure that if you um are using this, you're not misinterpreting

the results. And then we have another benchmark mode which is called average time which I'm I use a lot as well. And this measures the average time it takes to execute your code. So, you know, if your code is in the order of nanoseconds, then usually it is better to use throughput, but usually if you have something that is in the order of, I don't know, milliseconds

or hundreds of milliseconds or seconds even, then average time is usually And um but you know, it really depends on what you prefer or what you want to show in a graph for example, as I've done. Uh again, you have here that the lower score is better just like every other benchmark mode except for throughput. And again, here you have a link to the benchmark mode in

case you're curious to find out more about them. So, there are some other annotations I recommend you to add as a starting point that um usually for most, you know, basic cases you want to micro benchmark, give you a really good starting point. So, I already added these in the template that I provide you when in the link. So, when you get the email, when you click

on the template, this is already pre-added for you, so you don't need to add it yourself. Otherwise, if you run the command from earlier, you It seems the founder doesn't like my presentation. >> All right. So, and then what you should do is if you notice when running the benchmark, um it will calculate an error. So, how much um the, you know, the measurement can deviate between

the different iterations. And if you notice that the error is too high, then make sure that you just increase the amount of iterations. It will take longer to execute, but you will have a more accurate result, so. And um if you have some long-running operations, so let's say you have some IO, so let's say I don't know, you want to specifically test how fast saving to a

database is, um or so if you have some complex computations, um then uh what I would do is increase the time of the iterations to at least the time that it takes, but maybe a bit more is usually, you know, a good practice. Um you will see when you run it what the error is, um but you know, definitely make sure in those cases to increase it.

For most other things, 1 second, that is the default as I specified here, should be pretty good as a starting point. So, then we get to execute the benchmark. So, for the most accurate what I would recommend you to do is you should turn off um you know, CPUs have like a mechanism where it has like a base frequency that it performs at. And then when it

needs more when it has more load, it then starts up, you know, to a clock to a higher frequency, which means the processor will, you know, be faster. And the problem with that is um of course if you run a benchmark, um if in one iteration you have it at the base frequency and then it just during the benchmark starts ramp up to the maximum frequency, then

your benchmark will obviously also be a little bit skewed. Um and uh if you can in your BIOS, these are often called power saving or turbo boost. Um and then also what I recommend you to do is restart your computer because more often than not you have like tons of stuff that is running in the background and you know, this could of course also influence the result.

And I recommend you to really try, you know, close all applications you have, all background processes if you can. Um your browser, IDE. Uh the ideal point is really just having a small terminal open and running the command there. Of course, you might be thinking, "Well, that is a lot to ask." It is, but you just need to keep in mind this is if you really want

the most accurate results. You will see when it executes what kind of error you have and you can always decide is this error good enough for what you want to measure or not. Um I just want to tell you if this scares you away, don't let it scare you just running a benchmark even without paying attention to all of those things is better than not running one

and you will see that in a lot of cases it will be enough if you don't do all of those things. It's just these are things that can help you get a lower error and especially if you have operations that are really really fast, um you know, or really close together, then this could help you to um get the most out of it by having the least

amount of error. So, what you're doing is you're essentially running these two commands in order. Pretty simple, right? You just run them in your terminal window. And then you just wait. As I mentioned, this takes a while sometimes. Um some operations, they you know, some benchmarks they might take, I don't know, hours even to execute. Um it really depends on how many iterations you have and how

many things you want to test and so on. And this is what the result will be at the end. So, you can see here, um we have different kinds of libraries and their score that they have and you can also see the error. And you can see the is pretty good. So, it's within a margin where we can still make, um you know, good decisions. Maybe the

Jsoup and the Apache Commons Text one are a bit close together. But um you know, since the other some of them are lot, you know, faster, um this doesn't really make that big of a difference in this case. I actually like to visualize it because I'm a very visual person. And if you are, then do this as well. You can see we even have uh the error

here um with uh those whiskers here. Um but you can see and you can see uh surprisingly uh Jsoup has um you know, is the slowest in this case. So, and Spring is the fastest. So, this was a bit surprising to me, but I mean, of course, I was kind of happy because I was thinking, "Well, great. So, I can use Spring." >> But here is a

word of warning. Be careful because just because Spring is faster, that doesn't mean that Spring is necessarily better than Jsoup. Or it also doesn't specifically mean that you're able to use Spring, for example, to do this here. It could be that, for example, Spring is so fast because um it is just treating less edge cases, maybe. Maybe it just focuses on one specific set of things that

it does. And maybe Jsoup is much more thorough takes care of many more um use cases. So, this is why you shouldn't only perform a micro benchmark, but you should also do a unit test with all of your data that you expect to have and uh make sure that it also works because it might be that So, in my case, I can tell you uh I tried

it with Spring and it fulfilled all of the use cases we would have on production, which is great. But, you know, your case might be different. And especially if um you know, you have a case where your unit test fails. Um if you want to know more or if it's difficult to test with unit test, you can always uh especially if it's open source, uh look at

the implementations of those methods and see what they do differently. Find out, you know, is maybe one more thorough than another. Um and this really helps you to decide which one to use. But, of course, if you like in our case have not as I don't know, heavy requirements or whatever and um you know, Spring implementation is fine, this will be the fastest in this specific case

that I have. Of course, this doesn't mean that um you now when you do HTML entity decoding, you should just go ahead and use Spring because you heard I said it is the fastest. This just means that it is in my [clears throat] specific case that we had in our customer project, it was the fastest. But, it doesn't mean that it is the fastest in your case.

So, make sure you really try this out and find out what is the right one for you. And then um at the end of each run of JMH, there is the following thing that is printed, which many people might ignore. So, they mention that uh to gain reusable insights, you need to follow up on why the numbers are the way they are. This kind of points to

what I said earlier with, you know, if you see that this there is such a surprising difference between, for example, Jsoup and Spring, ask yourself, how come there is such a huge difference? There must be a reason for it, right? Either you maybe made a mistake in writing the benchmark, that's also definitely possible. Um or you know, just like I said, maybe one of them is more

thorough than another. So, keep those things in mind. Try to question why the results are the way they are. And that's also why it's useful, like I mentioned in the beginning, if you think about in the beginning what kind of things you want to measure, what you know, expect them there to be. Because the subs compare with what you expected, and then you can see, you know,

is this what I expected or maybe it's not, and you know, it makes it easy for you to follow up and decide what to do with the data. And it also mentions ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. So, this is also of course great advice. You know, if you have a really sensitive use

case, it always helps to ask some other people, you know, people that have experience. You know, you can also write me an email. So, if you scan the QR code, you know, you also get an email from me, so you can just send me a message, and I will be glad to help you out, and you know, review your microbenchmark if you ever write one, and have

a look and tell you, you know, if I see anything obvious. And of course, just ask anyone, you know, like is this how you would expect, this to be in our production code base, for example. And yeah, make sure really that you do this if you can. Of course, again, [snorts] don't let this deter you if you run it, and you know, you you see like, oh,

it it looks reasonable. You know, you don't necessarily have to do this, but it really makes the results even better if you can. So, again, this does not mean that it will perform the best in every kind of scenario, but only in this specific context that we have. And then also make sure to verify the functions that you are benchmarking if they work correctly. So, for example,

if you need tests. And you should have a clicker that doesn't die in the middle. And also what you need to keep in mind is that microbenchmarking results will be different depending on the hardware that you have and other factors. So, for example, um if you develop a microbenchmark and then you send your friend um your code, for example, or they check out the repository, and they

add another benchmark, and then let's say your microbenchmark takes hours to run, so you think, "Oh, well, let's just optimize it. Let my friend just execute this one, you know, new method that my friend added." And then, you know, we'll just compare his results with the results I got. Uh this could be easy to think, but uh this is not something you should do or, you know,

can't really do because your friend might use different hardware. I mean, if they use the exact same hardware, then that might be something But uh again, you know, they might have different settings or, you know, maybe they have some more applications running, whatever. So, really you really want to run a full run with all of the benchmarks that you want to compare, you know, on one specific

machine and not on multiple different ones and compare them. Um yeah, so keep this in mind. In a fast machine, the results might look totally different. Or also, if they use, for example, ARM64 as the architecture instead of AMD64, um the results might also look totally different. Um so, keep this in mind. [clears throat] That's why I also recommended uh test on your production hardware if possible,

or maybe as close as possible to it, as then you get the best results. And also, what I recommend you is there are lots of samples that are provided by JMH, and they teach you about all of the different um pitfalls that you can run into. I've shown you the ones that are most um you know, that most microbenchmarks that were poorly written that I've seen run

into or that I frequently run into um but in this talk. But of course, there are others, but they are more niche. Um So, if you really want to learn it in the best way, this is how you should do And then, if you want to learn more, whoops. If you want to learn more, I have tons of other links here. So, on the one hand, you

have the samples, but you also have, you know, some blog posts that people have written that I can recommend you because they are really well written. Um and again, you know, you can get this list of links by scanning the QR code. So, and then I'm already at the end. And um also, by the way, as a little goodie, um I have written an ebook on how

to fix flaky tests. I don't know who of you has experienced the case before where, you had a test that, for example, you know, it passed and then you run it again and it failed. Or, for example, that passed locally and then CI it failed. Yeah, that's like at least half, maybe all of them. So, um if you scan this QR code, you will also get um

this ebook I've written for free. So, it can also help you fix flaky tests. And also, who of you is using AI coding agents to code Okay, that's quite a lot of you. So, maybe half of you. So, um I noticed that AI coding agents are really terrible at writing optionals. So, what I built was I built a skill to have it write optionals right from the

get-go. Or, you can also ask it to clean up your code base with it. And you will also get a link to this by email. I'm currently writing one that is the same, but for streams, because uh it turns out they are even worse at writing good streams. So, if you're curious about that, just um you know, uh scan the QR code. And otherwise, I still have

7 minutes time for questions. Are there any questions? Oh, yeah, in the front. Mhm. Mhm. Mhm. That is a good question actually. So, what he said was um if I ran for example this benchmark on different hardware, isn't it the case that probably, you know, kind of what I understand is more like the the relative difference is between the different libraries are the same. So, kind of

Spring will probably still be the fastest one and there will still be, you know, similar amount of difference between the different libraries. Is that what you asked? So, that's a good question. Um I would say the answer is yes and no. So, yes in especially if the architecture stays the same, most of the time it probably will. But, um you know, especially if you have different architectures,

it there is a good chance that maybe, you know, some of the things that you do in your code is just more optimized in that architecture in a way and then maybe some other library is faster because it maybe uses some instructions that are more optimized for a certain architecture or the other way around. Maybe, you know, in a arm architecture you have to run, you know,

emulation um to get something working which slows it down maybe for example while if you run it on an AMD64 architecture it might be, you know, faster because it can run natively. So, this really depends. So, it's not an easy question to answer, but you're always the safest if you try it on your target hardware or as closest as possible to it. I mean, sure again as

I said, it's better to test, you know, anything than nothing, but you know, if you want the best results, that's what I recommend. Does that answer your question? All right. Let me have one more question in the back. Yeah. Ah, okay. Another good question. So, he asked um can you account for the differences in the garbage collection in your benchmark? Um so, I would say this depends.

I mean, of course since this is just a regular, um you know, Java application that is running when you're running the benchmark, um it will also do garbage collection if you're, um, if you're, uh, if if what you're running is fast enough. Um, so it will be, um, not fast enough. I mean, if it takes a long long enough that the garbage collector has time to do

it, sorry. And, um, so it will be more realistic because in production your garbage collector will also be running. So, of course, if your garbage collector is already really busy with other things, you know, this is a bit more difficult to simulate. But, um, most often they're not. This is, you know, good for most use cases. And if your garbage collector is doing lots of other things

already, then maybe I would first focus on this part to reduce this it's not a great thing to have anyways. Um, but, yes, I mean, you can also turn off the garbage collector, which is, you know, something I've seen somebody ask before at another conference. Um, but I would not recommend this because of obviously it's not as accurate because in your production use case you will have

a garbage collector that will do work, and you would want to include those things in your measurements as well, so. Yeah, you're welcome. Are there any more Um, yeah. >> [snorts and clears throat] >> That's another good question. So, he asked the sweet spot for the warm-up, um, in terms of how many iterations and how long, for example. So, I really recommend you to start out with

the basics that I gave you in an earlier slide. Um, those are usually, in my experience, you know, good enough as a baseline. And then, you know, if you notice that the error is too high, then tweak it by increasing it, and then try to see if it's, uh, if it's lower. Uh, but, you know, usually there will be diminishing returns. So, if you get towards 100

or whatever iterations, you know, usually you will not get that much reduction in the error. But, yeah, I would really recommend you start out with what I give you as a basis and then try to, you know, increase it if you if the error is too high or if you want to make the benchmark run faster and, you know, you can afford for the error to be

a bit higher. Yeah, especially if you have a let's say a benchmark that takes 5 seconds anyway. Um, you know, you can run less iterations as well if you want. Uh, yeah. Okay, so we asked, um, what if you want to simulate a system that hasn't warmed up? So, that is a bit more difficult. I never heard someone ask that question, I think the question is what

use case is there where measuring this would make sense? I don't know, can you come up with a use case where, you know, this could be I mean, is there maybe something you're thinking of or Oh. Okay, he was just thinking of it. So, I think for most use cases measuring with warm up will be, um, you know, more precise. So, by the way, warm up refers

to first, you know, running the code a few times because, you know, then you make it closer to the conditions that you have in production when you have run the code multiple times. You know, for example, if you have a Spring Boot application and you have an endpoint, it will usually get called, you know, I don't know, hundreds of times, you know, over a day or whatever.

And then, um, in after a while it will be warmed up and then when you make a call, it will be more accurate to measure it already warmed up. That's why I recommend doing it by default. Um, but if you want to measure it without it, I I mean, you could measure it without the warm up flag, but I'm not sure. I never tried it out if

that would really give you good results because yeah, that would be really difficult. You would So, I'm not sure if JMH already has built-in, you know, mechanisms to deal with that, to be honest, but yeah, maybe if someone of you has a use case for that, let me know. I'm kind of curious. Yeah. Did that answer your question? Thank you. Any other questions? All right. [snorts] So,

since we are almost out of time, um I really want to thank you a lot and make sure to grab some Swiss chocolate and uh some stickers uh when you're going out. And thank you so much for coming to my talk here today. >> [applause]