Mesut Durukal: The Future is Today: Leveraging AI in Software Testing

This talk discusses the integration of machine learning into the software testing lifecycle, highlighting its necessity in a rapidly evolving tech landscape. The speaker, a quality assurance automation engineer, explains how machine learning can optimize various stages of testing, such as defining test cases, automating test code, and maintaining the quality of applications. The session covers specific applications, including the use of natural language processing for bug classification and the automation of test data generation. The speaker presents practical examples and tools, emphasizing the role of machine learning in alleviating common challenges in software testing, including time constraints and system complexity. The presentation also touches on the future of software testing and the collaboration between human testers and machine learning technologies.

[Music] hi everyone uh welcome to today's session um I'm going to introduce me um who's speaking about the Futures today um he's an experienced quality assurance QA um testing software engineer and uh he's going to be talking about how machine learning um is actually implemented in the the software supply chain or sorry in the testing cycle so Mees introduce yourself there and uh we're super excited thank

you so much Nigel yeah so if I have my slides on the screen yes I will add it to the stage okay cool thank you so much so again let me briefly introduce myself my name is misal and I'm a software uh Assurance quality assurance automation engineer so I'm basically doing the test Automation and today of course we will talk about test automation but from a different

perspective like using machine learning in software testing because in software testing we have several stages like we have different activities is some of them are related to the defining the test cases most of the time we are working on automating implementing the test code so and sometimes it is just managing the bugs the issues we have so all these are different parts of sare testing and the

quality assurance activities verification and validation activities and in each of them somehow we can use machine learning so today we will discuss about this how we can use them or what kind kind of maybe I will share some tools as well so this is not marketing any of them promoting any of them but just give some ideas about the usage of them and not only the tools

but also I will share some ideas approaches to use machine learning for example I will share one practical example as well which I applied in my project so what I will do is I will start with first of all let's discuss the need and the background like first of all why are we talking about using machine learning in software testing what's the relevance why is it needed

what's the requirement for that and then after this problem statement and why are we doing this we will discuss about different ways to use machine learning in software testing and eventually in the third part this is what I was just explaining I will share you with a case study where I did classification of the box in my project for example in all projects we are reporting some

bus right we have bu reports and of course we have the description the title of the bu and we assign to some people but one very important thing is we assign the seity levels all the priority levels like some boxs are uh critical seity levels some are some of them are less criticality or the seity levels so I did this classification like whenever we have a newb

description then based on this description based on this text by it will be something like uh NLP the natural language processing and try to guess what uh priority level or the separ level it should be based on the previous experiences the previous learnings so this is why we call it machine learning right so we will discuss all these stages one by one and let's get started with

the first one the introduction part which is why are we software testing machine learning is already very hot topic right it is everywhere like it is in my smartphone it is in my Smart TV all the elect electronic devices even in iot devices we are having collecting data everywhere and of course whenever we have data we can do some analysis on that but once we are talking

about software testing again of course as I already explained box are already a huge data source for us for example we can already analyze from what kind of aspects we are having some issues weaknesses before going production so these kind of thing analysis we can do not only that but also we can improve our efforts and maybe we can reduce the time we are spending on these

activities right because we have too many challenges software testing nowadays is not easy the systems that we are testing are really very complex and complicated we have too many Integrations it is not only just desktop application right we are generally deploying to different platforms and those applications are talking communicating to other applications too because sometimes there are some apis one application is sending some request to another

one and getting some responses from some other servers and showing on our dashboards or the charts on our application this means there are too many Integrations and interfaces so whenever we are developing our application we have to ensure the quality or we have to test all these interfaces Integra Integrations and executions on different platforms for example if it is a web application we are not testing on

a specific browser but we are trying to do a compatibility testing right we are executing a lot of platforms on a lot of environments so this means software testing really needs some for and comprehensive planning and and efficient working time is very precious like there is of course the time pressure the testing windows are very limited sometimes we have only a couple of days before going production

before the deployment we have to understand if there's a critical issue or not so if you cannot catch these kind of issues critical issues before going then maybe the customers will see those kind of issues in production Environ so it's a critical process which means it feels little bit stressful right there is too many challenges the time pressure the comprehensive scope the difficulty of automation everything these

kind of challenges sometimes we might feel desperate too many difficulties too limited resources not too many people not too many head count not too many time too much time so it might feel desperate but these kind of difficulties and challenges encourage us to find some new Solutions some new Integrations so one idea is maybe we might get some help from the machines right if we have too

many challenges why don't we cooperate collaborate with machines as human beings can they help us actually they are already helping us in a lot of different ways in our daily life right as we discussed in the very beginning of the presentation like whatever we are doing sometimes even if we are aware or not we already can encounter machine learning algorithms in lots of places in our daily

lives for example whenever I'm reading an article the system is already recommending me some similar articles or maybe some other articles that I might be interested in how does he know most of the time it's very accurate right how does it know because it observes my interests it knows my previous patterns so it learns who I am on what I might be interested in or on what

I'm working so it knows my patterns my interest and my inputs and outputs these kind of relations so it basically observes and learns these are the training data for the machine and then after the model is generated for me like it will be M model for me so what kind on top of what kind of inputs I can generate what kind of outputs so this is the

model so the the B main mentality is like just observing getting a lot of data so training data is very important and then try to predict the upcoming reactions to these kind of incoming data so we will see it will work in a very similar fashion for the software testing I'll come to that point but with for now we can just accept that machines are already helping

us in lots of different ways so why not in software testing too they can also help us to implement or uh to perform conduct our verification and validation activities so actually my title of this presentation was feature of software testing when I start presenting this topic a few years back but now I'm not presenting this anymore as a feature of software testing because it is not the

feature anymore it is already we are already there right we can easily use a lot of machine learning algorithms in seconds it's very easy there are some Cloud platforms as we all know we can just ask some queries and get very accurate answers in in very short times for example I I'm sharing this movie it is The Minority Report movie I think it was almost 20 years

ago uh released and when I saw it it's it was just a fiction right it's science fiction so who believes that like it was just imaginary but now we are almost there for example like the machine learning algorithms can be used in lots of different areas and industries in law as well like in that movie the criminals were being predicted before they even commit this crimes uh

so again we can see that some of course not the same usage but in lots of different ways in law in in medical care in it in cyber security Manufacturing in lots of different places we can see machine learning can be utilized so this is just as a general introduction so I'm uh I will focus more on the software testing too uh when we will come to

that point but again uh for example we all I believe we all use in different chat GPT or some other platforms to perform our queries right for example even if I ask some similar questions some related questions like what kind of tools I can see in software testing industry it can list a lot of different things to me so not only the tools but of course even

if I describe a problem then it can understand on top of the previous experiments the previous observations and it can generate some solutions and most of the time it's very accurate so let's little bit remember quickly remember the uh working mentality and the idea of machine learning again so again we collect the training data and we build our model and we start evaluating the performance so if

we are satisfied with the performance then we can already deploy but otherwise if we feel like we can further improve our accuracy of the our model then we can again maybe modify or do the fine tuning with the uh arguments or the maybe other options of the model and we can try to achieve the best performing uh configuration so this is the very basic background and again

the learning mechanism is very similar to what happens in our biological organisms for example we as human beings we also collect and several we have interceptions right we have several uh Integrations interactions with the outer system and we collect this data these are the learnings for us for example let's talk about a concrete example as a tester you might ask me to test your system which is

let's say an online sales system right you have a web application over where I can navigate to your system and choose some items add into my shopping basket shopping cart and then prove my order so the delivery will start after the payment so this is a very uh simple online payment system or online shopping system so if you ask me to test your system I can do

that right I can predict more or less what I have to do like I have to select some items and I should add them into the basket and then I can VI inside the basket to see what kind of items there are and I can maybe add even more or remove some of them modify my baskets and then go to the payment system so I can predict

that but this is the first time that I'm using your system right so how can I know that of course due to my thanks to my previous learnings previous observations I did not use your system before but of course I use some different systems which is working in a similar way so if they have those kind of systems maybe I can predict that in your system more

or less there will be some scenarios which is having those steps like uh modifying the basket like adding some items or removing them and then confirming that and go to the payment system so this is my prediction but of course I have to see it it will be an exploratory testing right so in that regard the machine learning algorithms will work in a very similar fashion that

our by logical learning happens so first observations do some modelings and then eventually whenever we have a new system whenever we have a new upcoming input we will make our predictions so this is where basically how it works so after this introduction let's little bit talk about how we can use what we can do by means of machine learning let's see what we can do so in

software testing we can start by analyzing the requirements right first of all we have to understand the system and then after doing this analysis we have to design our test cases to cover the requirements so this is the traceability between specifications requirements and the test cases and after we do this planning after we do this uh test design we have to execute the test cases that we

designed we developed how we execute it can be either in automated ways or in manual ways right if we are doing the automated test execution then first of all of course we have to implement the code which can be automatically executed and then after the implementation done done we can just start the execution or if it is the manual test execution after the steps are ready we

can just start executing them and after the test execution is done the final stag is the maintenance the test cycle closure which can be by maybe raising some issues opening some bu tickets or maybe it might be some refactoring algorithms uh refactoring activities on our test code because test codes might be flacky right sometimes there might be some issues on the test code that we developed so

we might need to fix them to refactor them so it is a part of the maintenance activities that we can do in the last stage so for all this St States I will show how we can use machine learning let's start from the beginning the first is we analyze the requirements so we should design or Define some test cases test case definition stage the very one so

for that again I will start with an example I guess it will be easier to understand over a concrete example so on this example I have an API query which is basically it might be I guess a library API so you have a application for your library so the users can query the books inside the available books they can rent or maybe they can as the admin

user they can add some new books by by performing a post or put query so this is the how the entities looks like for example in each book entity we have the ispn number uh represented in a ispn field and and then there's the price attribute of the book entity and then there is the year which states on which year the book was published so these are

the basic attributes or the features of the book entity or the book object so what we can see here is by an simple observation I already know the attributes of a book when I'm talking a book like an object which is a book for me it should consist these three at least by observing this and not only that but also the format of this data I can

already see how they should look like for example ispn has a special format like first there's a digit and then there is the dash hyphen and then there are some three more digits and going on like that there's a special format like probably it's a standard for uh expressing the ispn numbers and when coming to the price I can see that there's a probably a double value

and for the year it is probably a positive integer right the year probably theoretically it might be a minus value too but in our case if we are building a library probably it will be the year will be something some positive values right so these are the things that I can already observe and I can model my data and my system so at this point what I

can do is I can already generate some test data for example since I know what attributes I have to include in this entities and I know what kind of data I should include I can only generate some random values by injecting different numbers or values and intentionally I can inject some negative scenarios as well for example whenever a string with a special format is expected in ispn

number what if I pass it and with an empty string for example if it's a m field what happens if I pass just an empty string so this might be a negative scenario so all these kind of observations and uh on top of those expectations I can already do some injections and I can generate some test data so generating the data itself is one step and then

generating the test Steps step by step test definition is another step so on regard of course like when we talk about using machine learning it is coming to a point like almost like equal to using chat GPD but of course it is not only chat GPD but we can develop our custom approach as well right because most of the time okay chat CH it is very convenient

but most of the time we have some security concerns like if I'm conducting some company activities of course I cannot share everything to chat GPD so there might be some sensitive information so I cannot send everything so which means I can somehow develop my own custom approach as well my custom machine learning algorithm too but this is again just to give you an idea what in what

ways we can use machine learning so I'm just going over some different ideas so generating the test steps doing stepbystep test definition by using machine learning is another option that utilize what else after we Define the test cases developing the test code is the next step right and how we can develop our test code first of all we might figure out what kind of operations we need

in our algorithm we we will have a problem and we need a solution right to going to this solution from the problem for this purpose what kind of algorithms I need or operations mathematical operations or some other operations I need to figure out this we can again utilize the input output relationship of our model for example I have one example on this slide I have an input

and by applying some mathematical operations I have to get this expected output so what kind of mathematical operations I can do for us it might be a little bit tricky to immediately figure out maybe some of you already figured out but it is not easy to understand immediately but formations it is not that much a big deal so the expected algorithms in this case are all the

input values are multiplied by four for example I have minus 17 at the beginning of the array so when I multiply by 4 I have minus 68 here so all the values are multiplied by four multiplication is the first algorithm I need and then I can see only the negative values in the output right normally there are the negative values and the positive values but the positive

values are filtered out so filtering out is the second algorithm I need operation I need and then I need a sorting operation too like starting from biggest smallest those are ordered sorted uh in this array so this kind of like multiplication uh filtering out sorting these kind of operations are the things that I need to build my algorithm so After figuring out this the uh applying the

relevant code is not the difficult part because whenever I need I figured out that I need a multiplication operation it is just a very simple function right just taking an input and returning by multiplying it by four so that is not the difficult part the the real challenge was figuring out the steps or the operations that I need to algorithm so again one example is I guess

it's a commercial one but it is developed by Microsoft deep coder it is using this approach and you can investigate more to see how it works and of course again another approach approaches just doing the NLP just asking the your steps like in my application I have to go to this URL navigate to this URL and select this locator or like very some elements on the page

uh type some inputs and click some buttons so just describe your problem and you will have the code that you can use most of the time of course it gives you a code you can use sometimes of course you cannot just copy paste but very with very slightly modifying the code you can use it so this is just a way to get some ideas so even if

you uh might have to modify some piece of the code you might already get some idea about in what way you can Implement your code this is open AI example and again from another family from the same family another example is the chat GPT where when you ask your problem like describe your steps log into this application do the steps and uh eventually do this assertion do

this verification then it will let you know code for example here I guess this one is a yeah Cy is a Cypress code so in this example I'm sharing a very specific like give me a solution which can work in Cypress test automation framework but you can change the framework like you can ask for a playright solution solution too you can ask for selenium code too so

depending not depending on the language or the framework you using it will generate the code that you need moving on another example is you can even turn the API queries that you do over Postman so first of all you have some queries on Postman which is mostly done to do the API testing and then after after doing this uh code uh doing this queries you can convert

the relevant query to a curl comment and even the Cur command you can convert to different programming languages like if you use JavaScript you can convert to a fetch query or if you are using Java you can use some other libraries like HTTP request or something else so these are most the uh test automation or the implementation steps where you can somehow use machine learning in different

ways like most of the time it can be a nature language processing like you can ask your problem in the natural language that we use in daily life and get the response or in some other ways too and again just as a reference I'm sharing too many uh tools that you can see in the software testing industry in the marketplace again some of them are I guess

in this one uh yes some of them are open reposter but some of of them are commercial ones just to give you an idea you can just go and see what kind of approaches they utilize to build their models but I wanted to share this because especially for UI automation I feel like machine learning is very important because normally traditionally how we do our UI automations we

need some locators we need some selectors right because while we what we are doing when we are doing the manual testing Cas we just select some elements or do some interaction with the elements but what does it mean in terms of coding like whenever I click a button what does it mean for example the very early very primitive test automation Frameworks were uh working on coordinate system

like for example on what uh spot you are clicking it was checking the coordinates of the window but of course it doesn't work anymore right because whenever you minimize your window or increase the size then this is not the same coordinate anymore but then instead of those primitive test automation tools the next generation tools we're using not the coordinates but the elements in the Dom structure like

the the elements were defined by locators or the selectors like you can pass the type of the element like it can be an input or button type element and you can can pass some other attributes like you can pass the ID or something else but even that one it it is already better than the Primitive approaches but still very problematic right because most of the time the

developers are changing the layout of the page so all your Dom structure is already broken like you can pass the class pad or the relative location or of your element but whenever they just move slightly the elements on the page because sometimes they are doing some AB testing activities and they can replace the elements on the page right whenever they do this kind of changes on the

layout your code is broken so what can we do instead of all these kind of traditional methods what we can do is even don't use the locator just describe what element you are looking for for example I'm looking for a button with at to basket text for example the I just describe it verbally right so this is just again and NLP or I can just uh describe

what kind of Icon I'm looking for and then uh the model that I built might try to visually find the item for example I'm looking for a basket icon which can be used for an online shopping cart then whenever I tell you a basket for shopping application you might already visualize something in your mind right so this is exactly what happens in the uh machine learning model

as well so it might try to relate to the trained data of course we need some different images for the basket icons but after training this model with all these images then it might try to find the most similar one on screen so in this way whenever we try to find them visually even if all the Dom is changed still no problem right because I'm trying to

find it visually so you can do whatever you want in the back of scenes so this is one example of training such a model so I'm passing different versions of a shopping cart icon so whenever you have one similar of one of these then I will already detect that as the icon that I'm looking for because I already trained my model by including all these possible or

potential versions and this is from an open repository you can just go and check uh it is called as test.ai when you go to test.ai you can publicly see this reposter moving on again I'm sharing some more examples for example this is now we are in the execution stage we complete our implementation now we can do our execution so what can we how can we use machine

learning during the execution we can collect several metrics right for example execution duration whenever I do my execution after a number of times uh times of execution I can collect the execution duration for example one test case can take in average 30 seconds for example so this if I learn this if I observe this and if in one of the next executions if it is taking not

30 seconds but 2 minutes then in terms of machine learning terminology it's an outlier for me right because normally I was expecting something around 30 seconds but this time it is 2 minutes so the model can already notify me then there is something here please go and check so there might be an an anomality in the services that I have in my product so maybe some of

them are not responding properly or not responding uh on correct time so I can just go and check so instead of manually continuously tracking all these executions I can set up this model and it can already notify me automatically which will already fasten my processes right I can already do the same activities with way less effort and moving on so the last stage was the maintenance after

we Implement execute maybe we can do the refactoring of the code so one type of Maintenance activities is uh doing the code reviews for example after checking all the best coding practices or similarly the anti practices that I want to avoid then I can do the relevant refactoring activities so somehow if I can teach machine learnings machine learning algorithms the good practices that I want to follow

or the anti practices that I want to avoid then I can only let it run and it can try to detect the anomalities that I want to avoid in my framework and selfhealing is another type of Maintenance activity that whenever a test case is failing then it can try to uh list the potential so first of all it can try to detect all the similar previous failures

and on top of the previous Solutions it can try to generate a potential solution for this uh specific test failure so it's called as self-healing so the test fails but it already gives me some solution ideas it will be kind of selfhealing and mation is another type like if I want to migrate my code from one programming language to another one or from one framework to another

one again I can just ask these platforms to do the migration for me and finally I want to share my case study uh which I applied in my project because again it is not only using chat GPT right as I told beginning using machine learning does not equal to asking questions to chat GP you can develop your own custom algorithm or custom application where you can develop

some machine learning algorithms so this is a super simple one but I will show you just to give you an idea about what you can do for example in this study I try to classify the bux so bux are reported on issue tracking system in our case it was jira and I just exported all this previously reported box so actually it was not a huge data set

normally in a real uh actual application of course we need way wider data sets but in this one I wanted to use the real data from the production environment so I had four almost not four 900 tickets and I exported them and over that I had three different SE levels like high medium and low Let's uh suppose like that and my aim is after analyzing all these

because this previously reported has the descriptions and we already mark them as high medium or low right we already know that se levels so after analyzing this after doing this study for an upcoming bu I will have the description and the label will be predicted by the system so this is our problem this is a task so let's see how we can build this system so first

of course whenever I extract the tickets from jira it will be extracted into an Excel spreadsheet so I will have just some text some values so the first step I do is just preprocessing like cleaning the data like I remove a punctuation because commas uh dots or similar Punctuation is not important in terms of the context right in terms of meaning I mean it might be important

but in this case it's it might be removed so I just do the cleaning and then there are some other irrelevant data like the text coming from the HT ml queries or the responses so all this kind of thing cleanings are the prep processing step for me this is the first step and then cleaning the second step is the feature extraction like from strings from ER text

values I have to extract some numeric arrays numeric values so this is called as the feature extraction so there are different ways to do that but what I followed in the study is count the words in each sentence like for example I have a sentence and like this words is uh happening two times in the sentence or the other word is happening only one time in this

sentence so I just count I just measure the frequency of these words in each sentence so this is called as back of words b o v b w and with this method I just have the numeric values right for each word so I can just convert this string into a binary number which is called I I might already have the features and after I have the features

the next step is the classification so for classification we can follow different approaches like KNN nabas KS clustering all these algorithms are some examples and after that we can just perform or measure the performance of our model so for that for example I uh did uh after I do the classification I checked the accuracy rates so this is the confusion Matrix and initially I had 73% accuracy

it was okay because it is a difficult problem right even for ourselves sometimes whenever we discuss the sity of the box for some people it is high sity but for some other people it is medium right so it is a difficult problem even for ourselves so for a machine learning it is not really easy to align with the previous results but there was still one Improvement point

that I could do because let me little bit go back because in this case there were three different sey levels but most of them most of the examples that I had was clustered around one of the classes which means there was a bias on one of the classes so I tried to eliminate this bias so I tried to combine three and four which are the medium and

the low priority levels what is the meaning of that it made sense for me because for me in my system if Buck has the high SE level then it was a release blocker but if it is medium or low it is not so medium or low means I can combine medium or low these two together and decide if I have a high priority or not or not

means the others two three and four so after combining this three and four the bias was removed the distribution was more balanced and this time I had the 82% accuracy again it is arguable like maybe I cannot directly use the results coming from this algorithm but I can already do the first classification by letting machines run right and then after getting the responses I can just do

some slight modifications so this already helps me to reduce the time that I'm spending on classif classifying this box and not only the classification but also I can do the clustering too to group similar boxs together so I can assign to same teams or maybe the same members in different teams so I'm wrapping up a we discussed using machine learning in different stages of software testing which

starts by analyzing the requirements and then implementing the test code automating the test code and finally doing the maintenance activities but which might be refactoring the code or maybe trying to build some selfhealing algorithms or maybe just managing the Box by doing some classification or clustering algorithms and the final discussion like in the future will we lose jobs should should we be afraid of losing our jobs

to machines or robots so my question would be if you have two options like hiring a person who does not have a PC just doing all the task manually this is the first option or your second option would be having a PC where you don't have a person to operate it so these are the two options which one which one which one would you choose of course

you would choose a PC with a person right you would choose a collaboration so why are we trying to compete these two but not try to make some collaborations between machine learning algorithms and the manual activities that we have to complete complete in our projects so thanks a lot for listening to me and if there are any questions I will try to answer them now that that

was amazing session M thank you so much um at the moment we don't have a question Q&A for those attending feel free to ask a question if you have something on mind um otherwise again he's put his contact details here on this slide so again I'm pretty sure you're open to to question side of the chat itself but no this was a a really valuable session so

I'm just going to give another second or two to see if any questions going to pop up in the chat and if not I guess we are good to wrap up if there's anything else you want to end on if you felt rushed and there was a a closing note you wanted to add feel free that you have thank you so much Nigel for the moderation it

was great and it was my pleasure to be here thanks no amazing so look with that I guess we'll wrap up thank you again for the session I'm just gonna put the closing um conclusion part so thanks