Pepe Cano: Performance Testing Workshop with k6

This talk focuses on load testing and performance testing using the tool k6, which the speaker has been developing and promoting for several years. The session begins with a general introduction to load testing, explaining its significance and how it differs from other types of testing, followed by an overview of k6, its functionality, and future directions. The speaker presents different examples and a simple demo environment named quickpixza, which allows participants to run their own tests. Key concepts such as virtual users, duration, scenarios, and iterations in k6 are discussed. Additionally, the presentation covers various load testing strategies, performance metrics, and the importance of thresholds in defining test success. The speaker emphasizes the extensibility of k6, enabling users to customize their testing processes by incorporating various JavaScript libraries and managing test results effectively.

[Music] hello uh and welcome back so today we have PPP from Spain working from um G and he'll be uh attending uh this presentation and talk about PC P6 uh and making a workshop for us so Pepe the the stage is yours uh the and and shine this okay thank you Ed thank you thank you for the intro I'm going to start the presentation sharing my screen

thank you everyone for joining today and good morning since I am sharing my screen now so the workshop or presentation is about low testing and performance testing using KC casc is the tool I have been working the last years just a short intro about me I joined the team as a engineer building the product and I have recently moved to a more educational role helping documentation explain

how to use the tool and talking about alling about performance testing this practice so the topic covered these three main topics first an intro about this practice not too much in details but to explain based on some testing background what is low testing or their difference and overview about the tool that I'm going to be presenting k6 why we build the tool and what is the future

Direction and then to make it more fun a demo I will go through different examples using a demo environment um the title of the presentation is a workshop I would have Lov to to run the workshop together with the different participant but with this format was not possible the good thing is that you can run the same Workshop that I'm going to be doing today um at

home your office with your college here are the two links that I'm going to be based this presentation let's short briefly we have here a 2 three hour Workshop touching most of the topic that I am going to be presenting today so just follow the workshop it will ask you some question so that you try to implement your own test and as I say it includes topics

in this presentation and yeah even more same for our demo environment let's see first our simple demo environment quick pixa is a very simple website you can order pixa recommendation and that's it the good thing about this project have different setup and even has also more example if you go over the casic folder we are going to be running some of the sample on this Foundation folder

a few of these um as I say I hope that I pick the Curiosity and you can also try on your own Let's Get It Started With the intro testing if we go to Wikipedia then description say that is the process of putting demand or load in a system and measuring its response as a developer I like to see other type of testing we have all done

some type of testing only testing integration testing and it going to work on the same way we are going to veryify different conditions in our system and the difference is just the other word load we are going to be testing under load and we have commonly two ways to set the load we could set the load in term of users imagine that someone say we are going

to be launching this website and we are expecting 5,000 concurrent users during the lunch then you will create a load test and set the load in term of this uh virtual user number on the other side we have setting the load in term of request per second STP do it more common for example if we are monitoring some of our service and we have identified that some

API starting to fail or perform badly whenever it reach a particular traffic normally our monitoring solution or observability solution will provide this traffic information in term of request per second then if we are thinking like that we may want to set the load of our test in t of request per second the other thing that will change with other type of testing that we are more used

to is what we say that is a a pass or a fail our pass fail criteria we are used to define too many assertions any number of assertion and if one fail my test will fail but when we design our production system we are not looking for 100% reliability and we are normally accepting a percentage of failur so the for example if we measure things we are

going to be measuring in term of percentile 95% of my request 99% of my request or when we Define ouro we are setting a particular error budget so we will have these things in consideration whenever we set the past fail criteria in our low test we will see later on this is a simple casic test you write your test in JavaScript that is probably familiar like to

many of us so we are doing a post another post to a login and the load we could express the load like here in term of virtual users running for 2 minutes let's see four basic concept about how the tool k6 we have already seen V virtual users we have talked about duration duration means my test will be running for one minute we have here the S

scenario the S scenario is the user flow or or the scene that we are going to be testing we are going to put there our test logic so we have a scenario view duration and the other concept is iterations iterations in case K 6 is just one execution of my S scenario that means if we tell k6 to run 200 100 iterations it will run 200 times

the SC function with this parameter we will set different uh way to set the load just to get an overview this is how case it will work Casey will schedule views that will be continuously running our a scenario until our condition does not hold true anymore it could be until my test has run for one minute or in the other case until it has run 200 iterations

let's see one example with quick pixza we have already see quick pixza okay where is this Local Host series ser3 quick Pi SI running here we are going to be testing the backend so we are going to see what is the request that we are doing with the buend and we see here that we are doing a post to the pixa end point with some payload and

then it's return a response so our first test going to do basically that one here we we have our first test this is the quick fixa project we go to the casic folder foundations and here we have the number one test in our test we import some casic modules and it's pretty simple we have options we have already s see that one view durations we just do

the post here to API pixel we pass the header we pass some data restrictions we check that the respon is 200 and we log the respon and we pause the V execution for one second let's see let's run now the test KC is a CLI tool it's released as a binary and you can just download the binary or use some of the different uh package Management Solutions

the command to run our test is casic run we give the name of the test and that's all so now K get started running all the view have been executed and loging the response and at the end of the test it display the results of our test for example this was the status of this assertion and some met some information um some Metric information here for example

latency here we have the number of failor request the percentage we will see this later on and then it give me more information about the test five virtual user running for 5 seconds that was a very basic test through the different example we will be just adding a little more uh information to this test since to consider when we are planning our testing we cannot expect that

our system will perform the same with maybe 10 user or 10,000 users that is why we commonly Define different type of lowest depending on the traffic the most common one is the lowest or what we call the average lowest we create a lot that simulate the traffic but it may be moment in the day that our system receive much more traffic it could be maybe during the

morning when everyone connect to our application and it may have 10 times more traffic than our average traffic so we may have in one side the average traffic and on the other the big traffic or stress traffic other common case is a spike of traffic something that goes very quickly from very low traffic to high traffic it is commonly uh expected when we launch some marketing campaign

but it happened also more often that people usually expect we didn't have the so or long duration test we may be running a test for hours two hours four hour and see how our system responds on those conditions or or to define a test to find the limits of our system we increase the load increase the load until a moment that we know that our system will

fail and we want to identify exactly what is the breaking point and at that moment we may stop the test as you can see here we are defining the what we call ramping uh load so we start from some virtual user to another then we have some stable load and then then we ramp down our load let's see this example in casc you will use the stages

options here we are telling KC run two seconds from zero to five five virtual user is the target then stay with five virtual user for 3 second and then 2 second you could Define as many stages as you want normally we recommend something simple you ramp up then you stay stable for a time and then you ramp down because if you start doing like very crazy uh

ramping scene then it's going to be hard to interpret the results let's see just this this second test we run on the same way number two stages and the differ is just that going to be running for 7 second three stages we won't see big difference because it's very similar load so we see here number of iteration is higher done before and we got again the same

type of in information about our test options then on the other side what we are going to be measuring and this depend on what you really want to measure the most common Cas is the latency or how it perform how fast our system respond this is why performance and low testing are also very related but it could be for example the availability of my system we may

want to run test much more frequently in very short period to see if our system is up like synthetic monitoring or synthetic testing common the reability of our system normally given by the error rate but it could be any reability metric in your product or your SLO definitions then resilient we may want to fail our test or even to mix with some chaos engineering practice and to

evaluate how our system perform when it's recovering from a failure other common case there testing the scalability policies we may have defined some scalability policy in system and we want to see how system perform whenever it's scaling up or scaling down the resources that was the intro let's go with with some of the reason why we build a kic or the principle of the tool there is

four principles that I wanted to share with you the first one I don't know if you have done load testing before but the one of the most of the most popular tool has been gmet being a tool that had been created 20 years ago and is a GUI oriented tool so when we launch we as a developer wanted to write our Lotus as code because as any

other testing Suite we are going to see how our testing Suite will grow over time and we want to modularize our code something that is going to be very common when we do low testing is re the reusability of our test for example we may want to run the same test or the same test scenario with different load again different environment like you can see here and

maybe I have a goal for every different environment maybe on the pre-release I want to check the slos and in production I am just focusing on checking if the performance has changed and I don't need to run with a very large low test at the end we are going to be starting to modularize our test this is very common pattern you create a low test that pick

some a scenario with some load options and you are going to have some common test code between different scenario for example the log the loging logic accessing my site the second principle was that we wanted to build we build an open source tool and we also have a commercial service Cloud we wanted that independent of the choices that you choose to run your test that is three

common choices you could run the same test without any friction there is three way to run casc test we have seen the first one what we call local test case it will run in one single inance but you may want to scale your test and that you need to distribute uh your test across different machine so you could do that with a kuber net cluster that is

a k6 operator you pick the same script and you distribute the load across different poorts or you can just scale the your test in our Cloud so we wanted that as I say you choose the option that you want and you don't need to change your scrip that sometime adds some complexity in our side when we are creating new API because it not the same some API

to work on a single instant that working across different machine for example but it was one of the principle that we brought from the beginning then as you have seen you write your test in JavaScript but the tool or the magic is writing in goand for performance reason we wanted to be able to generate a decent amount amount of load with a single machine so that you

don't need to distribute the load across different machine which is sometime a little more complex the things to be aware here is that KC does not run on no GS and even casy can reuse import JavaScript library it cannot use JavaScript library that rely on no GS because KC does not run on no GS and the last thing I wanted to share about the KC extensibility we

build a tool five years ago and I think quickly the community like the principle of the tool and is starting to request new new capability like for example we see here three very common case I want to store the test results in this back end or I want to use KC for Brower testing for fail injection testing or I want to do a performance test to my

database or to promis or to one particular buend so we realize that we won't be able to provide everything and we needed to provide a plug-in architecture we call extensions in casig and extension in Cas allow you to extend the capabilities of kig there is many extensions from graph alap and also from the community you can see here how extensions work let me show you example here

we have the SQL extension and an extension will make some method available in JavaScript as we can see here as any other JavaScript API but extensions are writing in go for performance reason so for any go method following our convention you will have a JavaScript wrapper so whenever you call your JavaScript function it will be converted to a Goan code and return the response as a user

you don't need go as a extension developer yes the good thing is that you can import any Javas any go library that you may know extensibility at the end the new option allows us also to test our system in new ways traditionally we have been used to test our system when we have been doing performance testing end to end from public end point to covering different part

of my system but now because we could use new protocol for example we could be focused on particular component I may want to focus just on this storage system and I want I have defined a scalability policy and I just want to for example test this particular component or cluster or we could be focused on some integration in test I know that it's the Q system that

is starting to perform badly when we have put too many of our queue maybe I just focus on these two or three component and it's going to just facilitate me the testing so I don't need to go for uh end to end test just to really verify some internal part of system let's go with the basic I will go through to show now different casic features and

go with different example the first thing that I wanted to show are casic option and we have already seen do you there is many different options you could find here some are option at the test level some are specific for particular apis and what I wanted to show here is that you could set the option on the script or you could also overwrite using some environment variable

or via CLI flag let's see the previous example we have for example stage we are going to tell just run now this test with one iteration and this iteration option has overwriting my AIC options as we can see here it has run only one iteration here casic has also a setup and thir down function which is familiar to many testing framework the setup function if Define will

be executed before our low test start and is commonly used to interact with our testing environment or to fetch data that we want to use in our test at the end of the test if it is present Casey will execute the thir down function let's see the number three life cycle example we have defined we have now added the setup function we just check that the system

respond if not this throw an exemption to not execute our low test and on the third down function we just add a new loog message we run the this test and and okay it was it passed this check that quick pixza return at 200 spawn and it had execute the test whenever it had finished it locked the response the the message that we add it on the

third on function that's it the third one Matrix we have already seen some of the metric the first thing that is important to know is that KC provide two type of metric what we call the building metric metric that KC automatically collect depending on the k6 API that you will use let's see some of those for example information about the check information about the data sent or

data received iterations information view information if you use the HTTP module you will get information about the connection the latency percentage of fail failing request there is different modules so if you use this module like the browser module it will collect browser metrics if you use the web socket module it will collect web soet metric same with grpc also one extension could also collect building metric and

then it will be automatically presented then you can also Define your custom metric to measure anything that you want in your scenario code it may important to know that in case there is four type of metrics and depending on what you want to measure so uh learn those and then pick the one that you may need a thread metric just collect the different data points like for

example here HTTP Rec duration is latency and it's going to be collecting request all the latency for all the request at the end of the test it present the different statistics a rate metric for example HTTP Rec fail it give me a percentage of success or error and give me also the number a counter is just is counting what we had on in the metric for example

here this one HTTP Rex is counting the number of frequest and last we have gouch that g provide just the last result stored in metric let's see another example we go to number four metrix and we have added the metric CL class we are importing here for a trend metric and a metric then we instantiate the metric outside of our SC scenario function we have a counter

to count the number of pixza that it respond we have one that it will track the number of ingredient of the pixel so we just instantiate and then it will be shared by all the views that will be executed it add a new pixza whenever it go A respond it store the ingredients let's example number metrix the only differ is that at the test it will display

the metrix information and here it is quick pixza ingredient the trend provide statistics number of pixza 26 one pixza for each iteration same iterations assertions in k6 we have already seen checks they are just like common assertions we add just a um a condition and they will track if it was true or false with a name the important thing to know is that check by theault don't

fail our test does it different as I said at the beginning with other type of testing why because imagine that you have 1 million of iteration or 10 million of iteration a large lowest or even 10,000 of iteration are smaller looted because one check fail I may have 20 checks in my test checking different things because one check failed that does not mean that my system is

failing as I said at the beginning we are accepting or expecting a small percentage of error and that's normal if we want to Define our P fail criteria in our test we will use thresholds threshold are defined on based on the metric let's example we Define thresholds on the options and we tell K to check a particular metric information like here we are telling I am expecting

the rate should be I am expecting the failure rate should be below 10% in this other stold we are telling casc the latency 95% of the latency should be below half a second 99% of the latency should be below half as H one second here let me check this one we can also change we can also set a stressful based on a custom metric quick pix ingredient

and we are telling Cas the average should be below a abort on fail abort if whenever checking this condition H the thresold has not been met let's see one example okay see number five yes now we are running the test we have already set the threshold the differ will be as we will see the check mark that we have see here on the information so this check

this threshold pass this threshold pass this old stress pass we have here our test as passed Cas six casing now return or it has already return before a zero error code telling to the tool that it has H the passed if if we change a threshold like this one we are going to see how our test will fail let's run again the test and now it has

quickly failed because we told k6 abort on failure it had checked that the average is not below two that that was let's see that was our setting average should be below average was six then here the stress code fails and k6 return now non zero error code 99 that will allow you to integrate KC in your cicd jobs or pipelines what's about thresholds different ways to set

the load in k6 we have already seen the constant view like View and duration we have seen ramping we initially saw the iteration we can s Cas a fixed number of iterations in total or view and um last way to set the load in case it at was it's called arrival rate executors and arrival rate EXE utor allow to set the load in term of iterations per

second in casic and that is a little tricky let go through example I am jumping now to example eight and let me show with the example case we tell okay Cas runs 20 on a rate of 20 per a Time unit normally per second 20 iterations per second this is the type of executor or a scenario that you will use if you want to set the second

in case you cannot specify a request per second uh the load in term of request per second because Cas doesn't know in advance how many requests your test will execute it could be that you run two request three request four request or even you have defined many request in your loaded some of them could be executed or not so casic doesn't know in advance how many request

your test will do but k6 knows how many iteration will be running and it could scale up or down the number of running iteration so you will have you will set the load in term of iteration per second and if you know how many request will be done with that way you are going to know exactly how to set the load in term of request per second

let's say this example for example we want to reach 20 requests per second in our low test I am going to Define 20 iterations per second and I am going to just do one request in this is what we did in this case we set 20 iteration per seconds and I am I make sure that we just do one request in our lowest and we don't tell

Casey to P executions let's see this eight we see at the button that 20 iterations are running per second so casc is scaling up the number of views to reach this goal and here at the end of test we have the information about the iterations 2011 were executed and the rate of iteration was almost the 20 goal that we have if we go to the request we

have executed 202 request with almost this 20 iterations per so if you need to define the load in term of request per second you will use the arrival rate executor also to command you could Define multiple s scenarios in your lowest and one two as many as you want each s scenario could be configured with different load pattern like some user are doing this fun this action

in my site these other user are doing other um are interacting differently with my website so that give much more uh flexibility the way that we want to set up our performance tces yeah wrapping up um I think those are the two last topics so script recorder and test results a script recorder in are a common thing when you are doing some low testing so script recorder

will facilitate H you to autogenerate a test a a test that you don't know exactly how it works a sorry A system that you don't know exactly how it works and you don't know how many request have been done so what you could do is as a user you could record yourself interacting with the service I'm going to be I go to this page I navigate I

do this this this action and then from there we are recording your session like for example that is two different ways in caseing you could use a brow extension or you could record yourself with a proxy then from the on the proxy Cas you are interact with your site the pro will record all your request it will be converted to a hard file which is a format

that track all the request done with a system and then from this hard file you are going to convert a casic test with the browser session the process is the same so you just record you you start recording here then you your you are now recording you will interact with your website you will do as many different step as you want and then at the end of

the test it will auto generate a KC script for you other common case for example is to use the the Chrome the Chrome inspector tool that it could also convert all the request in a hard file so at this stage we have some generated K script that it will fail because probably you are using a particular a part particular session ID for example or some data because

it's going to just provide this hardcoded information what you are going to need to do is to pre-process this script you may have a script with many many different request it has facilitated this job and now you need to do what we call the correlation for example correlation between request I pick up some information from one request I pass to a variable and then I pass to

the other request so you find those variable that are use with all the different request and also you could parameterize the data for each view so that each view for example use different information like I am user one and I want to use this credential user two will use another credential and so on so after this after editing your test fixing this situation you will convert the

autogenerated test in a proper functional um loadest and the last topics are results briefly we have already seen on the terminal information casc also by default has H the option to visualize the results with a web interface let's see this example copy the command that is yeah it's required more chain here we have so we are telling K to enable the web dashboard and okay now I

am passing the duration of 30 second to get more data we run the number test and it's telling me the web dashb bar is available under running on my Local Host it already here and we can see now now we don't see only the summary that it was shown on the terminal we have this information here as we have in the terminal but we can see also

the different metrics during the execution of my test and we could see maybe at the moment that something have changed so that is the way that you normally want to visualize your performance uh result but in both cases we we don't have the data and it going to be very common that you may want for example to compare the result from one test to another test because

you between one test and another test you have changed something so for that you need to store the information of your test result somewhere that you can also um compare then Result One result be or another common case I want to see if something have changed over time over longer period of time you may want or you may need to to store the result for that case

it also present different choices and yeah we wanted to like with extension to Pro to provide the option that you may want um and the option that it may fit better for your particular case so we have implemented a few options other are have been created by the community you could create your own extension to Output the casic results where you want here there is a a

few example in that case we are going to pass the Das o or Das Das output parameter with the name of the of the output then we could visualize with any visualization tool here we have with the quick pixa project also running a promis instant also a grafana instant that will F this promis information to visualize the cas result let me show you this example I will

copy the command now we are indicating k6 to Output the results to the promis remote right and point point so this endpoint is running on the default setting you could change the settings and here it is so cas6 say output sending the results to this end point let's see the promises and let's see and here are the casic metric for duration let's see the graph here are

the different k6 metric now just send we can see all those metric that we have seen before at being a store in this promis instance you probably won't use promis you to visualize the result probably you won't make to use something like grafana that can f the data and visualize the result there is some dashboard from the there is dashboard in grafana for many different options here

I am using one of dashboard and it visualize the casic results we can see here it allow also to select the if we run different test ID and we can see those visualization that are in that case very similar okay request information information information about my checks information about the aggregated information it was very similar to our dashboard we have designed both dashb so we try to

find similarity okay that was all for today and as I say at the beginning I hope that you also the work yourself thank you everyone more information on the website and thank you uh thank you Pepe unfortunately we don't have time for questions although we have a question if you can answer in the chat so thank you uh folks and