About this talk
This talk explores novel approaches for optimizing test case suites using metahuristics, a branch of algorithms distinct from traditional methods. The speaker begins by discussing the challenges of testing complex systems like self-driving cars, where the number of potential test cases can be astronomical. Traditional approaches like pair-wise testing and boundary value analysis are compared to the metahuristics, which provide a smarter alternative that can significantly reduce the number of test cases while improving their quality. The session includes two demonstrations, one focusing on a library that generates test data and cases, and the other showcasing the integration of this library with managed automation workflows. The speaker emphasizes the importance of experimentation and statistical validation in developing effective testing strategies, concluding with insights on using metahuristics for problem-solving in automated testing.
Full transcript
Thank you for joining the session. Um in these 30 minutes I will try to show you a different way for optimizing uh test case suites. Um and uh before we begin with the agenda and uh so forth I want to mention that usually during talks when I'm on conferences for me it's more valuable to get inspired about the different projects. It's not always about the code and
the library but about the different approaches the methods and uh I think this will be the most valuable thing for you. Um I will touch briefly on what metauristics are because I'm sure that it's not very popular. This is like a totally different branch of software uh algorithms uh that are different than uh the regular ones. But also I will show you two demos uh that you
can get inspired to try something like that uh in your particular context. Um so first I will start with defining the problem uh compare it to how we solve it with traditional approaches in their limitations like pair wise testing boundary value analysis and so forth and then we'll proceed talking about how metahuristics can help us what they are in short because you know even the introduction to
uh this um field is like at least 4 hours lecture so I I will try to explain it with simple words and then I don't want to bother you bore you with lots of theory. So there will be two demos one of the library and then another one of how we can use it with MCP servers in AI agents and such uh automation workflows uh and also
since this is like uh as I will mention um in a minute um I'm not selling you a tool or anything it's open source project and actually it's um a result of my let's say scientific work or curiosity whatever and actually uh I'm actually very happy because yesterday it was uh approved and it will be published in a couple of weeks as a scientific article in one
popular uh Switzerland scientific magazine. Um so yeah it's a scientific work with a lot of experiments and math behind it. So let's begin. So first what is the core testing challenge? Um to give you an example I will use self-driving cars. Imagine that when you have one self-driving car, how do you test such a system AI system? It's very hard because there are like u astronomical number
of test cases. Imagine that you have the different types of roads like highway, uh mountain road, city road. Then you have different weather conditions like sunny, foggy, it can rain, can snow. You have different road conditions like uh pedestrians or different cars from the left from the right. Signs might be missing. You have at least 20 sensors that have uh various configurations and maybe at each particular
moment one of them can break and something might happen just for 10 parameters with five value each. If you do full uh combinatorial testing, you'll end up with more than 10 million possible test cases. And uh some of you here QAS you know that one of the even maybe the first testing principles uh is that exhaustive testing is not possible. We cannot do uh full combinational testing
and with the case with the um self-driving car you can imagine there are billions billions of test cases. It's not possible. We need a smarter way. And uh recently in the last few years I'm reading a lot of papers scientific papers about testing of AI AI and QA etc. And such problems usually are solved with one traditional approach. It's combinatorial approach uh is called pair wise testing
or all pairs testing. This is a combinatorial approach for those of you who are developers that actually minimizes significantly the number of test cases uh while it preserves only test cases where at least uh in one test case each pair of inputs uh is present. And in order to explain that imagine a simpler uh example you have this web form in this web form there are some
fields that we don't care about them like the color picker etc. But let's imagine that we have five six of them with three possible values like um we'll discuss in a minute the ranges but for example you can put three names we can limit to three names three emails three phone numbers you'll end up with all combinations uh around uh 5,000 uh 15,000 um combinations if you
use pair wise depending on which tool you use in technology you will end up between 50 maybe 100 test cases and the approach that I will show you with metahuristics we can further optimize the set maybe in half but uh you'll receive like maybe between 20 and 70% better test case quality in a minute I will explain what this test quality means why traditional uh methods are
not working very well uh in pair wise because It's combinatorial approach. We are just combining all of the values. You don't care whether they uh at the edges of a particular range or not. Because if we uh if we are here imagine that for the full name in the database you have a constraint like the name can be between 10 symbols and 100. For the age you
it can be between uh one and 120. And uh for the QAS you know that we have uh boundary value analysis meaning that we are testing at the edges of the ranges. Why this is important? uh because for the developers you know that when we are writing code we write with conditions right if conditions whatever and many often we use uh greater than equal then uh and
if we miss the requirement if we miss the equal sign there will be a book there will be a book exactly at uh some of the edges of the range this is why it's um this technique exists uh we usually when we have a range between 1 and 120 20 um we we take the valid boundary which means 1 and 120 then 0 and 121 etc. Equivalence
partitioning is much simpler. We usually group the input parameters uh for example valid invalid security parameters whatever and um this horistic says that when we use one of the input parameters from one group then we don't need to test the rest however this in practice doesn't work very well because there are interdependencies between the parameters and this is usually why we combine it with boundary value analysis
and as I said pair wise testing is just combinatorial approach. It doesn't care about ranges. It doesn't care about equivalence partitioning. And also there is one more huristic usually that it's used by QAS. Um and this huristic is that especially when we write unit test automated tests, we don't combine multiple invalid parameters in a single test because this will cause the so-called buck masking. Meaning that for
example here if you want to check uh all of the validations rules and if you put I don't know wrong full name email phone etc you won't know if you test with all invalid values whether you're not busking buck masking masking another problem. This is why the huristic is that we test only with one invalid value. And this is why um actually now it's the time to
explain how this metauristics uh work and then I will tell you how I combined all of those together uh to optimize further the pair wise testing. So metauristics are very interesting they are different than standard algorithms. Uh the standard algorithms for example sorting etc. they can be proved with math. uh they find um most of the time if you use the right algorithm the most optimal solution.
Metahuristics on the other hand they cannot be proved very easy with math uh they're nature inspired. What this means you heard terms like uh evolutionary algorithms by Darin evolution uh you have genetic algorithms. Um there are some algorithms that are inspired by animals like the base algorithm that I used is actually called artificial bee colony. It's inspired by uh how the bees are working together because there
are three types of bees. Um we have for example algorithms about ants and uh other animals. Uh but again I promise not to bore you with theory. Uh I'm sure that actually on a daily basis you're using a lot of apps that are behind the scenery using metahuristic algorithms. If you're using Google maps behind the scenes when you click find me find the optimal route uh to
go somewhere it's it's not using a regular algorithm uh you know calculating all of the optimal roots. is using some kind of a meturistic to find the near optimal very quickly. This is the um the base difference. It's doing it very quickly but it's not finding the most optimal solution. It's finding near optimal which most of the cases is just fine for us. And uh this usually
happens because under the hood these algorithms they're using a random element and this is why it's very hard sometimes to prove it with math. Um I will give you another analogy. When you go to the Vituska street and you want to find if you go straight and you want to find a very good restaurant, you can go just straight like normal algorithm. However, if you are met
heristic algorithm, sometimes there will be a random element that will cause you to go left or right and sometimes you'll find a very great bar that it's somewhere in the small streets. Uh this is exactly how this mini search algorithms are working in metahuristics. Uh another apps that probably you are using are for example YouTube, Spotify, Netflix. These algorithms that they suggest you sometimes they're using AI
but before AI they were using meturistics right. Um and another example if you um working in a or if you went to a university and I'm sure that all of you have the schedule to make a schedule for your university this is extremely hard problem because imagine you have thousands of students 20 buildings thousands of rooms with different equipment with uh hundreds of professors with all different
requirements. No professor can be double booked. No student can be double booked. And the next uh next um uh hour that you need to take, it needs to be in a building that it's nearby. Imagine how hard it is to build a program like that. And usually this is uh done uh via meta heristic algorithms. And now it's time to show you uh really quick a short
demo. Um here on the page uh you can see the testimise at least a short version of the snippet of the library. Um it's doing three things. It generates the test data. It's generating the test cases. I mean using pair wise and optimizing it further and after that is generating the test themselves. Like here we're using this testimize generated test case attribute and when you build the
project it will generate from the right you can see the generated test cases automatically with all of the data and everything behind the scenes. um how we do that um first we have the fluent interface where this is why I said that um it's very extensible which means that for each input imagine again the web form you have a text field form here we are mentioning the
ranges like the valid range where we can put it so behind the scenes it will generate the boundary values but also um if we go to this configuration we have the so-called equivalence classes meaning that it will generate very interesting cases for emails for phones uh with a particular precision and we have that for many different fields uh however there is slight problem as I told you
um metoristic algorithms they are not deterministic which means that because of this random element we are finding near optimal solution Ju just a second because I'm not showing you the code actually. Sorry about that. Um yeah so we have this configuration here JSON configuration where we have this uh interesting values right uh these are the equivalence classes uh that we can use and um yeah and as
I said we are using this uh ranges here to generate the mode um however as I said it's um not deterministic which means that every time there will be a slight change in the test cases generated. This is not very good for continuous integration, right? Because every time you have a little bit different test cases however this is the so-called I call this mode exploratory mode. It's
similar to exploratory testing because very often when you develop for example web API, GraphQL, unit test, whatever you want quickly to run something like that very quickly the library to generate the data, the test cases to run it to find some books and then you can use the so-called precise mode where uh this is very precise. I mean here I listed all of the different possible inputs
that I can place uh whether they are valid, whether they are boundary valid, what is the expected message etc. This is like an ultimate test. We can leverage again the library itself to generate everything. As you can see there are many fine tunings but in the end it will copy to the clipboard and to the console output. it will generate um all these attributes for you in
any format. I mean there are v various formatterers for example we can generate JSON, we can generate nunit, ms test, xml whatever format you want. Um and again once generated it will stay the same always. And now how actually this works under the hood maybe the next demo will be a little bit more interesting for you. Um this works in the metauristics many often they rely on
this fitness functions evaluations functions uh that basically give a score about one of the items. In my case, this evolution function is calculating the quality score of a test case. What this means? For example, if you have a test case with five input values, we need somehow to measure whether a test case is a good or bad. How we do that? We combine in this function the
different inputs. For example, if you have an input for email, we are categorizing it whether it's a boundary value, whether it's boundary valid meaning at the edge of the valid range or outside of it. And we are giving a bonus of it. For example, if it's just exactly one, which is a boundary valid, we give uh like a score of 20, which is very high. Uh however,
if it's inside the range, this means this is not interesting value, right? So we are giving it uh much smaller bonus. And also if you take at uh if you look at the beginning of the function uh if you have more than one invalid values we are giving um a big penalty minus 50 which means that if you have two invalid values most probably this test case
will uh will be dropped. It won't be used later. Um and yeah if we have if you see at the bottom if you have more than uh if you have a value that it's uh new to all of the test cases we will give a bigger again I understand that it's even this is the simplest part of the algorithm anyway um it's really hard to speak about
topics like that anyway um when you start with the algorithm You see here the UML diagram a simplified version of it. Um usually this artificial B colony starts with completely random uh generation of the inputs of the test cases. However, this is not very optimal especially for cases with millions of test cases. Right? This is why under the hood one of the things I changed this is
why it's called a hybrid approach because it's combining I don't know maybe six or seven different metaistic algorithms. Um so under the hood one of the changes is that we use pair wise generator the nunit pair wise generator and we are optimizing from there. After that we multiple iterations through the algorithm for example 100. Each generation we are calculating this score about each test case and we
take just half of it. I mean we take it's it's called elite population which means elite test cases the best scores the test cases with the best scores. And then for the next half we are mutating them. Probably you heard the term mutation testing. Maybe you can relate to that. The thing that we do here is that we are mutating the input values. Meaning we are on
random we are changing some of them with other values and we are recalculating the score. And this is the important part about metaristic algorithms. They are working with trial and error. But we need a way how we can measure whether the next thing is better than the rest. This is why this function here is at the core of the algorithm. It's the core logic how it works.
Now let's stop with the algorithm itself. Here uh you will see another important part. These are the different parameters that you saw that you can fine-tune how it works. Even if you used on a lowle you know that they have various par parameters that you can use and fine-tune based on the different context how they work. And this is very important about metahoristic algorithms. They rely heavily
on various different ratios. For example, here we have a mutation rate which we can change depending on what is the value whether we change 30% of the test cases or 40 it can sign significantly change the final output and another important part I think I mentioned it during the slide about meturistics is that we are not proving that with math actually we use statistics and variances etc.
uh however we do that with extensive experimentations for example in order here to find the optimal ranges I did at least 1,000 experiments with different settings you can imagine that and we use like math to prove that it's not happening on random again this is like a requirement to be a scientific work anyway um so here is the table a simplified version because here is with the
math uh But here you can see some of the parameters and why it's better. As you can see the improvement with um 67% we need to compare it like as you know in testing in performance testing etc. we need to have a baseline and something in our thing to measure against it. uh in my case I was comparing uh against the pure pair wise just like we
are taking half uh of the uh test cases again sorted by this evalation function uh compared to if you use the hybrid uh artificial beam uh algorithm and in many cases actually it's better like uh here you can see the math uh 15 different configurations like in half of them the improvement is over 15 to 20% in some cases like over 60 um and one of the
key things that I want you to remember that in metauristics we need to use um high math statistic variances to prove that some observation is not happening um by luck for example this p number at the where you see 0.001 001. This means that all of those experiments prove that this is uh there is 0.1% chance this to happen by luck. It's not happening by luck. And
usually over 0.5 is good. Uh below that uh and now I want to show you maybe the more interesting thing for you will be to see uh a demo. Actually, I incorporated this library uh in an in an um MCP server um and I combine it in a just a minute. So the demo that uh you will see um I'm uh basically using in Visual Studio Code
a couple of MCP servers. Uh the first one will read some requirements from Confluence uh about the different ranges, right? Uh then here I configured in my visual studio code with MCP servers Jura Zefire which is a test case management system. Here I built in C# the MCP server for and also I use playrite to automate uh the test cases at the end. um through experiments I
found that actually um you know AI is not very good uh in generating boundary values or pairwise it's like for 10 cases maybe it will do it but if you have more it totally doesn't work or you cannot prove that actually generated pair wise this is why we need MCPS or outside tools or libraries to do that uh so yeah in this demo I will the requirements
or reading the requirements from confence through the AI. I will instruct it to read it from confence. Then it will use testimize to generate 100 test cases optimized version of them with the test data behind the scenes directly in the AI. As you can see here, we can get the settings of the algorithm and we can change them. We can fine-tune them. By default here I listed
with um this particular tool the default values but we can change them in plate text. As you can see here I I uh I changed the mutation percentage how how much of the test cases we can um mutate. And now I have an assistant from the left uh that I'm showing right now with different steps. I use prompt engineering and prompt patterns uh to teach it how
to read the requirements. What is the template for the test cases that later in a particular step I want to create the test cases in ZFire then that we are going to start the browser. We are going to execute the test case in the browser. It will find the bugs and then it will create the automated tests with playright and generate not playright code but selenium in
Java. And as you can see here we are reading at the bottom the confluence page and now first confluence search get page then we are calling generate hybrid test cases with uh the testimize MCP server. As you can see here I visualized some of them again with all the data. Now it's starting with playright mcp the browser executing all of the scenarios in the browser and it
generated the selenium code it's not playright we are using playright mcp but but we generated selenium java code not typescript as you can see when we close the browser it closed the tap it will generate test cases for our approval when we approve them. It will in step four here it will create them in ZFire and like if you download um there are not there isn't official
uh Zfire MCP I forked one and improve it a little bit to include steps and expected results but yeah in a second you will see that we open ZFR and we will see that actually all the data all the test cases we generated with the testim were created um in Zephr in Jura uh with all of the data again usually this is extremely hard if you rewire
just an AI this is why MCP servers like that are very important as you can see the data that we generated with faker under the hood using testimize and even in the end here uh I'm using another MCP server for loading slack and in slack I will post all of the ids for uh the test cases that we generated. This is fullblown scenario with this assistant and
you can combine it. You can imagine uh what can happen if you for example use tools like cursor or cloud code, right? This is just visual studio code which is like not the smartest of demo. Um, yeah, just a second. All right. Um, something happened. Um all right so I have a little bit of time but a couple of things that I want you to remember. first
met heristics that they are very important again you can use these algorithms in to solve very complex problems especially if time is essential for you as I said uh they're used um in apps where time is essential and you don't have the computing power uh to do it like on on your phone you cannot calculate all of the roots right uh and you need to remember that
we are finding near optimal solution Not like sometimes we need the best solution then you will use standard algorithms but when near optimal is fine then you can use them also remember when you use AI LM charts etc they are very bad when they generate data boundary values or pair wise uh for sure you need to double check that doing the right job but in most cases
they don't uh and this is why we need outside tools um like you can use this MCP server or another but you need to remember that you need an outside tool for that um also one key thing in metauristics is that don't trust assumptions for example here I did many different experiments to find for example the different weights if you remember the function here and you you
are going to ask me actually one of the professors that uh proofreaded uh this paper asked me how you prove that this should be 20 and not 50 which is an accurate question and actually to prove that I did like 7,20 7,200 experiments with different configurations with five different schemes of how you calculate that and in reality I was thinking initially okay maybe when you change these
numbers, they will really change the final output, right? It's obvious. However, it's not obvious because all those experiments proved that when you change these val values actually doesn't change much the final output. It's not common sense. Why? This is why we do experiments. Actually, the only thing that matters here is for this boundary value if where where you see it when when we give the bonus to
be much higher uh than uh the one for the valid values. That's the only thing that actually matters. Um actually it turns out that the whole structure, how we group, how we give bonuses or weights is more important than the actual values there which is not common sense and you need to find that through lots of experiments and again using math to statistically validate that. Um and
yeah you can check the GitHub prepo even if you don't use it it's very interesting uh to see how different design patterns are applied for example fluent builder um and others like strategy etc. It will be just you can be inspired just to see how uh this architecture works so that you can extend it further. Thank you. Okay, do we have any questions? We have time for
one, two questions. Hey Tony, what made you choose the artificial B colony and not something like the artificial uh ant colony or another genetic algorithm? Um to be honest I read some papers that this particular algorithm is very good in um generating tests. Initially I read such a paper and I had to pick one of them to develop something and this is why I said okay maybe
I will just experiment and see how it goes. I want to implement something myself and um this is how I did it. Actually after that when I was doing the scientific research read many other papers and it turned out that most of the people were actually using this particular one because most of the algorithms they are very good in particular area right >> and that's >> thanks
anyone else >> yeah we can talk later in the breaks if
