Nelson Vides: A Murder of Crows: Actors, Instrumentation, and Load-testing as Part of the TDD Cycle

In this talk, the speaker, Nelson Vides, discusses performance tuning in software development, specifically focusing on load testing and performance metrics within the Test Driven Development (TDD) cycle. He shares experiences from his work with scalable systems in Erlang and Elixir, emphasizing the importance of efficient code to enhance user experience and reduce latency. The talk highlights relevant statistics, such as the impact of increased latency on revenue for companies like Amazon and the necessity for rigorous testing of software systems to ensure optimal performance. Vids shares his insights on benchmarking, load testing, and stress testing, explaining how they help identify issues before they become critical problems. Drawing on real-world examples and his projects, he demonstrates how to analyze and enhance performance metrics effectively.

[Music] ladies and Gentlemen please welcome our next speaker Nelson Vibes presenting the topic a murder of crows actors instrumentation and load testing as part of the TBD cycle hello yes all right I cannot is this it's pronounced vides it it it comes from wine so I'm proud of it anyway that's a quote from mbth it will have some meaning later are you all familiar more or less

with mag Beth like a a guy wants to get some power he betrays his family kills his brother if I remember correctly his wife is behind him telling him all the time what to do and how to get more power and it gets like very like T NOA but Shakespeare anyway M of crows actors instrumentations and part of and load testing as part of the tdd cycle

I work Inland Solutions a company that specializes in scalable full tolerant and distributed systems uh with all the keywords you can imagine fancy projects uh very interesting um clients we specialize mostly in two languages erang and elixir you all use these languages even if you don't know for example WhatsApp is 100% Aang in the backend Discord is all El the backend to Classics plenty of others uh

this is my GitHub if you want to find me somewhere and my two main babies a messaging server and a load testing tool that we are going to get to and you can find me by that picture everywhere in social media except LinkedIn because I was told that I have to look professional in LinkedIn so in LinkedIn I have something cool but otherwise that's that's me my

career you could say that started more or less this day I was a programmer before this but like this is the important day and there was a photographer that day so I I'm very fortunate to have this picture I was at this point uh meid junior meid developer in C in telecommunications and I was uh frustrated with one specific problem we had a software that was to

be deployed in servers my company was resp responsible for the whole stack including the hardware so we were selling the hardware with the software installed everything was like one unified box and we had this challenge that software would would get slower if you put it on a bigger machine which was counterintuitive the thing is that now bigger machine doesn't mean a faster core it means more cores

and you started reaching loog contentions so for some time uh I was implementing I was trying to deal with this in C until I met this guy and I discovered that he already did what I was trying to do but like with my incredible lack of experience he's one of the creators of Aang I met him this day by accident I didn't know he was there somebody

told me hey we cannot use robber building and my head was like wait what then I moved to Eran that's where I code now through him I met my company uh let's get to some topics on that's a good old statistic from Google is from 17 if yes 17 bounce is uh the probability that somebody that enters your website would abandon the website it increases uh dramatically

the slower your website is and Google still today uses this to penalize your website is if it's low it's going to just put it further down because Google wants to you know the users of Google want the fast experience so if your website this low people are going to blame Google so you know that's how they make money uh almost 20 years ago Amazon found that every

100 milliseconds of latency would cost them 1% in sales in context please check the source uh somewhere there uh by today that would be a few billion dollars a year for Amazon in general imagine 1% of your Revenue whatever your business is doing imagine that you would just lose 1% of it can hurt a lot I mean maybe it's not back rupy but it can make a

big difference uh broker for trading could lose uh $4 million in Revenue per millisecond so that ends that builds up very fast if their platform is 5 milliseconds behind the competition because it's beaing so it's who makes the faster bet an example a very innocent example from the mailing list this is from Jan this has been making my merge window miserable by like he says you have

been warned this is very polite by today standards by lenus thals January 2024 something was making the build very slow he he went from 22 seconds to 44 drove him mad he found the pull request that the the merge that introduced this slowdown and he said something in your merge is going to be reverted Saturday 13 just a couple of days later he says that we have

lost power and internet thanks to a winter storm and then he goes to make fun of what people in Oregon call Winter he's from Finland so that's just three days later Emer window is posted the next day this little tiny fix is released it was just like a just calculating some margin like was a no-brainer but lenus was out of uh connection so this took a few

days to be taken care of what if this happened this was only for the compilation but what if something like this happens to a database for example we all use databases under the hood uh if it happens to a video game engine they you know frames per second and you get to buy so much expensive Hardware to play this game and then the game is slower not

that good uh for your business so let's talk about performance ready I'm still don't know how fast I'm going to go through this you only care if you have a match kit end of the talk that's the most important point if you can ignore me from now please don't but this premature optimization is the root of all evil we have all heard this quote I hope by

the one and only Donald G the father of algorithms this quote has some context we should forget about these things 97% of the time it's it's root of all evil yet is that critical 3% and why not it's very tempting I'm obsessed with performance I tend to overlook for this 3% and I'm blame for it often so I'm also reminded of the whole thing Focus benchmarks you

only care about code you have benchmarked right don't forget that most important point I had this is playing well yes fantastic good internet uh I had a very nice uh physics teacher in in high school we were studying at some point flu in mechanics and he gave us this example I'll never forget it it was impressive that's a bridge in the Tacoma Arrow North uh west US

during construction it was behaving weirdly but just a couple of days after construction this happened uh the bridge was evacuated there was no human casualties only one dog inside of that car that was the only casualty of thing uh this if you look it up in Wikipedia like it's not that crazy thing the the wind was something like 25 miles per hour don't ask me in kilometers

I'm a great supporter of the International System but Wikipedia I in miles I don't know what is it like 40 kilometers per hour maybe it's not a lot so the question is what the hell happened like it's not like crazy Force no don't play it again sorry the arrow things that I want to play again now yeah we went one day from these kind of bridges from

you know the Roman Empire big solid pieces of stone that you could just hammer down and nothing happens and one day we went to these kind of bridges that are way bigger but actually weigh a lot less and they are flexible they are hanging then they become susceptible to new forces like for example we used to go from these kind of computers that I could punch with

all my anger and I would probably just break my hand and we went to this world or to this laptop that I have that if I punch it presentation is over there are new forces that make a difference now that I didn't use to make those computers were used by like three guys in the lab that just knew how to use it and they took turns and

this this is used by everyone you never know in which ways you all heard like God knows what your client is going to do with your software so we went from a bridge that it was like in destructible because like it was just one big solid thing but now gravity is pulling everything to like three points of uh concentration similar analogy we went uh to a single

massive computer used by three people to very strange Cloud deployments used by sky the limit so these things we need to test them and we had one very important luck I'm going to talk about computers not about bridges that was my last analogy I'm not a bridge engineer but we have one amazing advantage that Architects or Bridge Engineers don't have we can make copies of of our

program and see what happens and just just break it just do whatever ugly stuff you cannot build two Bridges and bring break one to see no no chance but we have that luck is virtually free of course consumes Hardware but you get the idea so we need to test things I'm going to give a few examples based on this uh little baby of mine I mentioned it

before my it's not mine it's my team but I'm very proud of it it's very close to my heart um it's a messaging server written in aine extensible uh scalable I mostly like to emphasize the the extensibility we can customize it so some examples are going to build on it you can have a look robots scalable extensible instant messaging server I am comes from instant messaging and

we are going to run a little performance test in one of the things that this uh does uh addresses in this chat application it uses the xmpp protocol so it works the address is something like an email but you just get that extra resource that's how you distinguish more than one device connected to an account and that's how you can Federate servers when you know you send

a message from when you send an email from Gmail to your friend that has an account in Yahoo that they you send a message to each other you Federate the servers so a very common operations is to pass this thing and turn it into a data structure operate on your data structure and vice versa the parsing needs internationalization like your URLs to defend against uh you know

uni code that makes it look similar so it has some string and I use this Library called Beni to test one all implementation that we had written using a if that means a natively implemented function a lang is an interpreted language so sometimes for performance reasons you want to implement something in C because it's like a very is perfect so you do it very fast and that

was our all implementation and then we wrote a little one in pure airline code run rly generate like three uu IDs let's say that that's the the username the server and the resource and let's see how passsing uh happens to them and turns out my pure C code was two times lower and a tiny little bit more memory use like 40 bytes doesn't matter but two lower

so this is a very single algorithm like you you saw how complicated this can be like you just need to find the the ad the Slash that's crazy but hey pure code was faster easier to maintain get rid of the C code benchmarked another example uh bit more complicated the protocol works on XML is the markup language so that's another operation that is very hot code you

do it for every single message that you route everywhere you need super fast XML encoded on decoding so you try it in c u then try it in pure Airline make a a comparison with like different sizes I just make here a few different inputs this time systematically for all of them the C code was way way way faster and also consuming a lot less memory but

on average you see somewhere between 20 30 times faster there is this one that makes a big difference all the others were between 20 to 30 times lower on the pure airine code this one is only five times lower what happened here in in this size I run this on this machine that I'm showing you and I'm doing this presentation you get the specs that's uh cache

size so at at exactly five kilobytes the word didn't fit in cash and there was some memory so this did you get a curve but then when you get something very big cash doesn't make a difference anymore compared to the whole encoding and decoding so that's very good to know like this slums in my performance because I can optimize for what uh will be my usual packet

size how big are your messages do you send to your grandma messages that big 5,000 characters it's usually hello how are you and I you know I can optimize a bit uh this was wrong with Beni a little framework that gives you a lot of nice statistics is written in Elixir I know the developer and he's lovely full open you can also have a look here you

saw generally the like very intuitive you give it some inputs a bunch of functions you can also calculate memory time running in parallel etc etc and now a little bit of uh low test I checked a few linear algorithms now let's see what happens when I want to connect and going to use this little Library I get the spec for two users so randomly create a username

password whatever uh connect them both uh and have them both do this thing send create a stanza send it stanza is a message in terminology and the other user waits for the stanza and verifies that it is correct and you just do it you know for Alice to Bob and Bob to Alice so you check you load test uh this kind of scenario also this runs in

a loop uh 60 times and you wait 10 seconds so Alice and Bob send a message to each other wait 10 seconds do it again runs in this uh infrastructure uh low testing Master node and two worker nodes two mongos nodes using postgress all of this is also um archived 100,000 users one every 5 milliseconds and as I said every 10 and uh sens 60 message in

total and these are some results for a comparison I wanted to compare like a pool request to master with some important changes so we wanted to make sure that at the very list doesn't get worse so here we get how it was before with the new underground under the hood Library Nisha CS no need to worry about the details uh same stuff a little bit the medium

and the the 95s is a little bit better but maybe doesn't make a big difference when it comes to like the the user experience If the message took three or four milliseconds to arrive the user doesn't notice that his phone the internet is probably slower anyway so I'm good to go my new code is doesn't it's not making my whole system worse this was used by my

other little baby that I introduced at the beginning amok that spawns all these users and connects them all and runs my the test as I want to do it and there is one last kind of test stress test for this one I'm not going to give you any fancy example because it's very hard to give one what I want to do in a stress test is to

kill it I don't want to see how it works when I connect 100,000 users I want to see how it dies that's when you what you cannot do with a bridge you can copies uh I will share with you inside on how to do it but I will skip an example because they can be very convoluted to generate so we get three kinds of blow test really

load is the one in the middle you want to see how it behaves at high low but you get like the they ual microeng marking so performance at minimum load you are like loading one thing and the stress you see how it dies that's the three performance test that you should probably run there is a lot others one that is also a very good idea is a

long-term test you know some things uh when you have a memory leak you only see it when it has been running in production after a month so run a lad test and keep it running in the background for a month to see if it theg grades this is more complicated but these three are relatively low hanging fruits that you should be aware of them and start doing

so we know our test now we need to know the purpose test driven development everybody knows this right I don't need to argue I I don't need to try to convince you I hope it's clear you you know you know how it works you write a test for example this is Eran code by the way uh name of my test I'm going to run a story escalus

is a helper Li for the xmpp clients I will give it to users and I will tell it to create two users to completely fresh so there is no I can run them in parallel and execute the story the story let's go through it just like in the load test before create a message have Ali sends that message to Bob Bob waits for the message and verifies

that is correct and vice versa so my usual functional test I can send a message and as I say you wrap it into a story escalus automatically generates the clients and does the whole thing cool fails because tdd now I have to make it pass not going to show you the source code cool green it's you know okay you also of course get the needs and you

know for initialization excellent refactor make sure the code is great repeat next functionality you get the idea you probably already knew the idea and I only repeated what you already knew now very important this this is how long my test took 075 seconds that's how long it took to create two completely fresh users connect them have them send the message disconnect them and say the test is

green really what I want to test is that I can send the message in a different place I already tested that connected but this is how long it took thing so let's bench mark it with the previous tool this time I don't I I only want to know how f is the message delivery I don't want to you know this previous time was taking into account connecting

the user and so on this time I'm going to consider that they already connected so before each scenario I'm going to generate two connected clients and after each scenario I'm going to disconnect it so these two operations are not part of my scenario and in the scenario I'm going to do exactly the same that in the previous two examples send two messages receive them verify Etc this

as I said before before you generate them them the message generate the message verify when you receive them when it's done disconnect the users and repeat in my computer this one again in the Mac this time is running on a separate box but it's running on the same network it's a Linux with this specs and the scenario is been run on this MacBook that I'm presenting with

so you get the specs same them before on average uh my average is uh 600 Nan I'm very happy about this uh this is the amount of operations per second that I can do here you have some statistic the worst case you get the 155 milliseconds but here I'm sending I'm running one test at a time there is no parallelism in a real server I sell you

a messaging server and you know you are going to get Christmas and everybody rites like crazy what happens that day so going to try to run it in parallel because Beni also gives me uh tools for this all the same logic than before but this time I'm going to run the test first of all for longer not the default 5 seconds but let's see about 30 and

the parallel is a flag that I'm going to tweak I'm going to get this results the parallel is use one you know it before parallel is 12 both of my computers both this one and the other Linux box has 12 course so that's more or less where I would expect that things starts getting interesting at there I get the average is 1 millisecond worst case is 7

millisecond still is fantastic but then I see that when I grow I just try up to 64 the the grow is much less than linear when here is uh quadratic so I'm very happy about my server I have benchmarked and I have a baseline for future changes Now define the specification I want to sell you a server that delivers message but that delivers them immediately this is

the first point I want to make the relatively immediately is part of the specification of the product that I want to sell you and that is very important so I need to define the slas latency versus throughput and expected Behavior slas if I want the you know for the statistics not every message is going to be the same so some people tell you the 200 milliseconds uh

message delivery 200 milliseconds what the the median the 99 the maximum you know to Define these things you define your expected your for expected worst case and really really bad this should never be tolerated so that's your maximum same for HTTP first content delivery that's where Google is going to penalize your website latency versus throughput and also if I when it dies I want it to die

by full death or partial death this comes from the electric grid terminology when they disconnect part of the grid but not part of the other uh or if I want to do a brown out which is like lower the voltage that is I'm going to take longer to deliver the message I'm going to lower the quality of the service this is important because for the electric grid

you cannot do a brown out or any kind of blackout to a hospital for example that like it's better to disconnect some neighborhood and keep the hospital fully connected so it depends on your use case you need to Define these things and have it test them have have them tested now this is the Baseline the previous test that I have in this machine but if what if

I run this test in another machine I was showing you wall clock um times but this has limitations the first funniest one that usually it never happens until it does but really such an edge case it's just important to know the time warps time Corrections because I don't know a fluctuation in the Sun and the Earth took a nanc less to rotate and there are atomic clocks

can that can detect that everything is corrected real time your computer eventually gets the correction hey get your clock 0.76 NS behind your computer does this jumping in time so your measurements of time can suffer this it's really rare but it's important also they are expensive and they make a lot of noise this Le forwards or backr in time expensive because it's a global variable in a

computer with 64 cores is a unique thing you you need memory barriers to access and it makes a lot of noise this is affected by voltage just to name one thing so Alternatives performance counters do I need to introduce perf anybody doing some kind of lowlevel or sometimes not even low level but like concerned with performance you should be familiar with this tool for measuring things perf

is really a frontend to Kernel counters database counters you want to test how long the database took to make an operation if you put it on a different Hardware is going to take a different kind amount of time you want to know if your query is correct you check the database counters all databases postgress for example offer you this kind of things and interpreted counters for interpreted

languages Java is one example um not familiar with all the languages in the world of course Eran has ones has their owns the test that I show you I show you time because you're not familiar with aan but Eran has a concept of unit of work is what I actually use when I want to test if my algorithm is correct and I know that the units of

work will get worse or better as I change things and how long this takes is really Hardware dependent but I have a reliable noise-free counter deal with noise what kind of noise depends on what are we measuring you're all familiar with throughput latency and memory uh throughput and latency is usually a trade-off you cannot have both you can optimize a lot of both but they tend to

fight each other uh and you will say that for your test you want a static threshold for example I want my test the message delivery to be 5 milliseconds the the pure message delivery the static threshold has some disadvantages it's manual work for every single test I have to decide and how many tests are there there is no context there can be for example if my threshold

was 5 milliseconds I'm running currently on three and make some pull request that goes to 4.9 is still green but the next pull request only changes the r me and because there is noise the Ry change is red there's no context and background violence noise and so I need automatically applied thresholds that I don't need to manually decide that are aware of the context and that they

are resilience get variance so we are going to use a little Statistics uh what is variance the red thing has a very little variance that means you know you have your average but all the population are close to the average so that is little variety and the blue one has the exact same average but a lot of varieties of variant and the square root of variance is

called the standard deviation and there is this rule the 68 95 999 that says that all the population of a normal distribution are going to be this far away from the static from the standard deviation so when you get your average 68% is going to be one standard deviation around the average and 99.7% that's pretty much the entire population is going to be three standards deviations so

okay track how long all the last M runs and you need to Define them and you're are going to Define that red something that went too far away from three times the variance everything including noise should get a little bit more a little bit less but if it gets way too far that breaks the normal distribution this is not a normal distribution anymore so you could consider

that rate you also need one more thing this is the coefficient of variance you could have all your tests slowly getting slowly getting worse and a little bit worse and a little bit worse and a little bit worse so if you track M runs if you track all of them you will see that the variance eventually breaks but if you track the last 50 or the last

100 your average is growing so you will not detect that it's getting worse because nothing ever grows outside of the accepted variance so you need to take into account the coefficient of variance uh there are tools that calculate this and the coefficient of variance will tell you if the variance is changing and also when you run a test usually there is noise so you run the exact

same test end times let's say three and pick the best result because uh the average result is going to have a lot more noise uh one example of how this works I get all my tests that takes this amount of time so you graph them you know the points are my discrete test this line is the median so that's my Baseline all my code should take around

this time what I'm testing you get one standard deviation up and down like 2/3 of the population is going to be in this range but almost everything is going to be three standards deviations up and down if anything thing gets out CI should scream and then I can verify it went down no it got faster do I know why it got faster if it gets faster and

you don't know why you should check anyway I mean consider yourself lucky but check because it can get worse and then you will also not why so with this now you need to know your tool how am I doing with time all right uh I want to spark a little bit of um competition because I like it uh there are two tools that you're going to need

one is the usual benchmarking I already talk about Beni your language will have the appropriate ones there is plenty of shell uh scripts but a bit more fancy is the load test and eventually the Str stress test or the when you need to keep it alive for a month and see if it dies this is when things get complicated and there is a bunch of tools that

are open source and in my opinion the best one that I like the most was k6 by grafana amazing job one very important thing that I consider very useful is that what I want to test is an actual programming language there is a lot of tools that the tool tells you asks you okay you want to send the HTTP request to this server with this body but

what if I don't want to send an HTTP request and like I I want to have a full programming language because I know what I'm testing I'm going just just to write so that's something that I find very favorable and not a lot of tools do it so that's fantastic scales really well so I tried this test in my other server I had a echo server TCP

send and return the same packet written in pure rust using Tokyo like I just want to make sure that it's like the simplest logic in the best performance so I want to test the low testing tool not the other and repeat like get a connection uh repeat a 100 times in a loop generating a uu ID sending and waiting for the answer and verifying that the answer

is the same and then I wrote the same using my own tool called amok a murder of crows uh murder of crows in English uh group of animals uh they have different names depending on the animal when you have crows the group of them is a murder because they are waiting for you to be dead to eat you that's what my server does it's waiting for your

server to be dead to eat your server so I wrote the same scenario in my server this time isan code uh starts some like Telemetry this like for metrics uh at the end I'm going to just like print the the statistics uh the same connect generate the u u ID send it wait for the answer check that the answer is the same repeat a 100 times disconnect

exact same scenario the result was like like I'm very proud of this because amok is like the the work for of a good friend of mine and me and is on average uh five to 20 times faster to do the exact same load so I I met people working in grafana somewhere I don't know if they're in the public but I met people through the corridors not

in the k16 but I told them please I let's park some competition this was fun to do and let's get a different scenario this time in my other server there is a rust Tokyo best performance I'm just checking um files and I'm going to just read some static file and check that the answer is and hey where is my arrow move to the next all right is

it maybe a runner of battery okay anyway uh same thing this time I written I have written this test in Elixir because Elixir had a very nice uh where is it here a very nice library to do the HTTP request Elixir has lots of web uh tooling so I wrote my scenario in Elixir and again this time I was uh five times faster to run the same

thing so I was very happy about this uh I'm going to say one thing in favor of uh k6 I'm really jealous of the user experience they did an amazing job so again I'm not ranting on k6 it's it's awesome if you if you're a go or JavaScript developer you don't know what to do it's fantastic and it also integrates with raana so but when you're running

low tests you are like burning a lot of Hardware you need to generate a massive amount of noise so if I want to test 20 million collections connections because it's Christmas or it's Black Friday in my e-commerce and I want to test that it survives 20 million connections then I'm going to pay a massive difference in AWS credits running this test so I just need to continue

working on the configuration and user friendliness but anyway uh this stop working uh one more thing that you will need in your low test and this is uh all the configuration was very user friendly with k6 but this is one thing that I I like in my tool uh you will need things like throttles and coordinates so one mechanism is a throttle that I can configure to

change grad from this amount of request to this amount of requests in per minute this time I can change it to two different rates in steps and my actors so the connections if you are familiar with Aang it's an actor model so I can just spawn a virtually unlimited amount of them that's the advantage uh another thing is that I may need to coordinate for example if

I'm testing uh an e-commerce and everybody's going to buy something different but a few people are going to buy the same thing and I have five units but eight people want to buy that so I will want to coordinate my actors to tell each other okay you8 try it at the same time and let's see randomly which one is the one that gets it so I will

need um to create plans so to pick like every two actors run a function or every five or when you get all of them and then actors as they are running they are adding to this plan and then later they will be told what to do and one little last thing I need to finish because we're running out of time Gathering data instrumentation is not metrics almost

but it's uh so frustrating that I keep clicking here and it doesn't uh perf for example is just a front and to counters instrumentation is all the stuff the kernel and the CPU are doing perf is the the one that gathers all that data and builds it into and transforms it into metrix and this classic example of the kernel is what all your programming languages do with

their performance counters and then you use pretus same idea so in my tool I do something very similar when it matters I scream something happen and then the back end that needs it picks up this data and transform it this is one way uh every actor will run this function but this is run inside of Telemetry span Telemetry span automatically measures the time my function took to

run and then it triggers this event when it starts when it stops and when it stops if it stop with a failure and here I can gather the data I know the time stamp at the beginning how long it take and I can calculate all my metrics so again I expose this in my library so and with some basic Gathering but you can write your own and

one last quote uh this is from the other creator of airine that I didn't have the chance to meet because he passed away he said when you write something first make it work second make it beautiful and then if you really have to make it fast and then it didn't fit in the slide the continuation because most of the times when you make it beautiful it's already

fast but uh This what I said at the beginning you only care about code you have M marked Benchmark is part of your offering is part of the functionality is the key notes that I the the key points that I have for you that is all from me and I believe we have no times for questions right or oh perfect good all right questions thank you very

much yes that's a very good point now I three and so I was telling you that you know to deal with noise you do this like Gathering statistics the last and entries and so on but this only works when you have more than one run so what do you do with the first run I would say like very often you already have existing code so manually Define

some baseline and then let it build let it heal Define a baseline that is way too green Define something over optimistic and then Implement that mechanism when you write code from new and you get the chance to like write in your tdd cycle think about uh how long it should take what what is the contract of your offering like you have a website that will deliver the

static content in 5 seconds or within 100 milliseconds for a single operation you're not even doing load yes there is a little bit of U manual decision at the beginning do that with a critical part of your code and let everyone every other part of the code uh build its own uh statistics as as you run and that's how you define if the test is green or

not you need to define the scenario the the question is that how do you define the Threshold at the beginning is manual right any more questions all right oh there is a hand there yeah yeah I will repeat the question what kind of uh uh testing do I recommend in the cicd pipeline I mentioned at the beginning the three standard ones is Benchmark and load and stress

Benchmark and load I believe are the low hanging fruits low stress is a bit more complicated and then the one month long okay in CI for all the little libraries we do have benchmarking so for the XML parer the the Jets the identifiers base uh encoding Bas 64 Bas 16 we do have benchmarks for all of that I am still working on the self-generating statistics because I

need to play with like saving things in cash on CI so then it's fetch on the next one that is a work in progress uh I'm working mostly right now on that with uh scram DXL parser and U distributed in memory database that we have uh that for performance test the the usual Benchmark that is the easiest thing we do it a lot far from perfect is

work in progress but I have an idea how to do it there is a few ones that are already done the right way uh for low test we do have low test in the big server in mongus IM it was automatic at some point then uh anible change uh versions and stuff started breaking and we need to get back to that but at the moment it's manual

unfortunately sorry yeah yeah thank you