Mesut Durukal: 4 Elements of Successful Test Automation

This talk focuses on the essentials of successful test automation, presented by a software quality assurance automation engineer. The speaker discusses the importance of test automation in verifying features before they reach production. Four key pillars of successful test automation are outlined, which include mindset, tooling, architecture, and continuous monitoring. The mindset emphasizes aligning automation efforts with customer perspectives and holistic testing approaches to address testability issues. Tooling involves selecting the right frameworks and libraries to support automation efficiently. Architectural considerations focus on creating maintainable and reliable test cases, while continuous monitoring is essential for ongoing improvement and reducing technical debt. The speaker emphasizes the need for collaboration among team members to enhance the quality of software delivery.

[Music] uh hello uh welcome to the next uh talk uh from method about uh test automation four elements of successful test automation this is a title method is going to cover uh common problems and fundamental solutions for test automation so if you're ready the stage yours sure thank you so I hope my slides are on the screen and yes yes okay so let me get started hello

everybody my name is missal I'm a software quality insurance automation engineer and as a test automation or today I want to talk about some basics of a successful test automation project or process because nowadays test automation is very important uh if we are trying to verify and validate the features that we are trying to deploy to our production environments first of all we have to complete our

testing activities but if we don't have automation of or if you don't have the proper automation of course we would be having a lot of troubles because automation helps us to complete our activities on time with an efficient process or activities and if we only have manual testing activities of course we would not be have enough resources to complete in a short time because we are racing

in time right time is very precious we are developing our features and we are going to do trying to go to the production as soon as possible because the customers and users are waiting for them and if we if they are waiting too long for bugs to be resolved or maybe new features to be implemented maybe they will already give up using our products right so in

these terms in terms of our product quality too test automation is directly affecting our results or the outcomes so we will discuss four elements four pillars of a successful test automation how we can manage that how we can maintain this so I believe most of the audience know what four elements I'm referring to on this slide you can see it's an ancient Theory ancient philosophy which says

that everything in the universe is composed of the these four elements so they are fire water air and soil Earth and everything in the universe is somehow composed of these four and if we lack one of those we cannot maintain a sustainable life so this is the very basic idea like if we are missing one of those elements then we will not be able to maintain our

uh life so what we will discuss is what is these four elements in terms of successful test automation how what kind of four basic pillars we need to sustain a successful test automation life cycle so on this slide you can see a funny movie in which I guess the original movie was the Bru's Fifth Element movie and this one is a funny movie which is making fun

of that original movie and in this scenario I guess somehow they are trying to save the planet from some other danger or some other planets or aliens something and somehow they are trying to combine these four elements and again if they cannot make one of those then the mission would be failing so all of these four elements are very important and based on this analogy let's see

what these four pillars the very important four pillars are for our so those would be first of all the mindset and the holis thing testing because if we are talking about test automation it is not only automating the test cases it is not only implementation but also doing this Automation and implementation on the correct track in the correct direction so it will be aligning ourselves with the

customer perspective and it should be aligning our automation coverage in the correct scope we will discuss in details and the second one is the tooling like if we are talking about automation of course the tools libraries Frameworks platforms or devices whatever we are using in terms of the tooling are very important if we choose or if we make some wrong choices wrong decisions then it might directly

affect the performance of of our automation because of wrong toolage maybe we might have some not reliable test cases or very flucky test cases due to wrong tools or the libraries or maybe we will be having some coverage gaps because the tools that we are using are not supporting the features that we need so tooling is very important we will discuss about that as well the third

pillar is architecture and the framework structure some uh coding practices some best coding patterns because if we are trying to build a test automation framework of course the way that we Implement our test cases is very important like if we are building structuring them in a framework where they can be maybe automate executed together with some other cases or maybe they can be executed even independently from

all the irrelevant from all the other test cases so all these kind of dependency issues and of course the reliability issues as are very important because if we have some reliability issues what does it mean we will have to cope with these kind of failing test cases which are reporting some wrong results right they are not reliable which means maybe they're reporting a failure but it is

not a correct failure it is a false alarm so we will these kind of this uh reliability issues and the tmls and the fourth pillar is the monitoring activities and the continuous monitoring and continuous Improvement because if we try to do a continuous maintenance this is how it works right whenever we Implement a test case it is not done like we cannot say that I automated this

one okay I'm done I'm going to home this is not the case we have to continuously update maintain our test case why because sometimes the features that the test is covering is being updated or maybe sometimes even without feature updates we need some refactorings to improve the efficiency or just to fix the test failures in terms of those reliability issues I just discussed so these are the

four basic fundamental pillars those four vital elements of our successful test automation life cycle what happens if we don't try to maintain them if you don't try to continuously improve them first of all the technical depth will increase raise in time right for example let's suppose test cases are failing a lot they are not reliable so in time whenever we Implement some new features on top of

those framework that we have the newly implemented test cases will be also FY because we did not uh fix the root causes so the test cases or the test code implemented on top of those uh Library codes or the common methods functions that we previously implemented will be also depending on those flakiness or the reliability issues so this is a technical part this is the technical side

but on the other hand there's a psychological aspect as well if people start that our test case are not reliable after some time they will start ignoring the test results right for example sometimes we set up some alarms or notifications which is reporting our test cases to our select channels right I A lot of times experience that no one is seeing those notifications anymore after some time

maybe they just mute the selection Channel because they know that it is not quite correct if the test cases are not reliable and they just don't care even if the test cases are failing but suppose if there's a real real issue not a false alarm but a real bug then even if the report is there since people already start ignoring them they will not be notified by

a real bug alarm right so it is really affecting our directly so these two aspects are how these four elements are improving our processes so it's important to continuously maintain them if we do not continuously take take an eye on them then the all the technical depth and the uh other aspects of quality will all increase in time so let's try to see them in detail one

by one starting with the first pillar which was the mindset right how the automation is related with mindset let me explain you for example let's say that I have 100% automation does it mean that everything cool maybe not right because maybe the test Suite is not complete like okay all the defined test cases are automated so automation coverage is 100% but how about the test coverage itself

like maybe the test cases were not defined properly in the first place so even if you try to automate all of them by the by the way 100% automation should not be our ultimate goal right sometimes you might uh choose not to automate some of them depending on the criticality depending on the frequency that you execute or even depending on the difficulty like you don't have to

100% automate all the test cases but even if you do sometimes it doesn't mean that the everything is done in terms of quality assurance so we have to ensure that the test scope the test coverage is correct and then if we have some test coverage or coverage gaps we might try to understand the root causes most of the time it might mean that we we have some

testability issues because somehow there are some features but they might not be testable due to some reasons it might be like the nature of the product for example it might be a microservices product and uh they're maybe working asynchronously so we might have some timing issues for example sometimes for some feature it they are working overnight right for example the users are doing something and some reporting

features are collecting these actions user actions cumulatively and after one whole day then we can see the reports but think about that how can how can I verify this uh feature because first of all I will do a user action and then the analysis feature will collect all the results for a day and then I can see the result after one whole day 24 hours so this

is a testability issues issue right of course there are there might be some work arounds like maybe I can already generate some user actions with the previous times and I can try to see the result but this is just an example so you know what I mean that there might be some testability issues stemming from different reasons it might be timing issue or it might be any

other technical issue so what we can do is we can collaborate on them with developers or product owners quality Insurance members so everyone in the team all the parties in the project and do a holistic testing approach let's do a brainstorming and how we can cope with this testable issues how we can try to open pave the way for testability how what kind of additional features we

can Implement to generate an environment where I can test these products because previously they were not due to some reasons so we can find some solutions but of course it is possible with a Hol with a whole team approach if we are trying to uh ensure the quality only with individuals only in the quality assurance teams it will not work it will not be possible and again

we will of course talk about this our test cases reliable for example we might automate again let's say 100% automation is done succeeded but are they reliable for example I automated all of them and I executed but how about the results so these are the aspects that we should keep in mind and try to continuously keep an eye on to ensure the reliability of our activities for

example let me a little bit further go into the customer perspective because we are saying like maybe we might automate all the test cases but they might not be the correct scope right what I mean by this for example I'm sure most of the people know this famous cartoon it is it's funny but it's quite correct most of the time like we can see sometimes the product

owner or the project manager understands in the wrong way what the client or the customer is explaining and most of the time the customer himself does not know what he needs and then after those kind of requirements or the specifications a development is done which is again maybe totally different and then we test in a way based on the documentation we we have which is most of

the time no documentation at all and of course it will be way different than what the customer needs so he needs something he or she needs some features and then they explain in a different way and then it is developed in a different way so how can I be verified or validated so having the customer focus looking to product from the customer perspective is very important so

what can I do as a quality insurance member like if I'm talking only with the product owner then I might have a limited perspective but sometimes I may try to find ways to communicate with the end users as well to understand what kind of use cases they have or what kind of features or extra implementation they need so in this terms I can raise some bug reports

or maybe some change requests in terms of the box for example it will be very costy or expensive whenever we cannot find them on time right because if I don't have the customer experience or customer perspective then I will already let the features go in production I will already let the deployment but the end users themselves will start reporting these bu buxs so what's the problem about

this eventually if we have this back reports we can solve right yes we can solve but it will be way more expensive then catching on time before going production because if the product is already in the environment then it will be very difficult to apply those fixes on working features maybe refactoring will be very difficult since the feature is already there so what we can do is

we can think again from the end users perspective so on this slide we can see for example those guys are trying to cross the small river right so what is meant in the story is as testers and developers we will be using these tools or these features that we already implemented previously a lot of times so we are familiar and we know how to use it but

for the customer or the for the end user it is not the case it will be the first time they are using this so fail because they are not expert on this which means maybe we might widen our perspective to think more about the usability to think more about how the documentation is done how the accessibility is achieved so all these kind of quality aspects we can

all already keep in mind if we have the customer perspective so one approach which can help us to improve a a customer perspective is personal based testing for example if we think about different characters which might or different profiles which might use our system it can really help us to think about different scenarios for example some profiles have some different strengths or different weaknesses or maybe they

have different priorities for some of them maybe accessibility issues are very important but for some others performance is super important for example so for each every other uh profile or Persona like the different characters which might use our system a different aspect of the quality might be important so it will be it will help us to improve our perspective and think from different aspects of the quality

so I advice using the personal based testing if we have different categories or Prof of profiles who are using our systems or features let's move on in terms of mindset of course not only the customer perspective but also the collaboration within the team is also very important for example most of the time we are doing some testing activities in different levels right we have the unit test

cases we have the integration tests and then we have maybe the system acceptance or n2n tests so we have different levels but if there are some duplication which means some feature is already tested in unit level and we are trying to test it again in the end to level so it is not super efficient right duplication but why don't we try to prioritize the features which are

not tested yet instead of the features which are already tested a lot of times in the unit test or integration tests so this means we need a decent collaboration and communication within the team like if I don't have any connection with the developers I cannot know what unit test have or sometimes I cannot get support get help from them right or like in the other way around

sometimes they need my help the development team sometimes they ask some questions to the quality team sometimes they need the quality coach perspective so if we don't have this communication or connection then of course it will directly affect the performance and the efficiency of our team because again I'm going back to the first point that I raised we are on the same boat if we are trying

to ensure the individually by only the members in the quality assurance team it won't work we have to work all together as a holistic approach so I'm moving forward to the second pillar the first one was the mindset the customer perspective and the collaboration within the team how about the second one this is a little bit more technical part so we will talk about the tooling that

we can use while building our test framework so on this slide this is me but a little bit confused me because there are too many options we can use right for example whenever you are starting a new test automation project which programming language do use is it Java is it JavaScript python which one is the best or let's suppose you okay you decide JavaScript because Bas most

of the time you are doing uh front end automation web application automation then what is the best framework you can use is it Cypress is it playright test cafe or selenium or anything else so obviously the answer is there is no clear answer right there is no single correct answer there might be several different options and we should choose the correct one for example if I choose

a tool which does not support what I need then of course obviously it is not the correct decision but how can we make decision of course there are this different decision criteria like the speed execution speed some tools are slower than others some of them are faster than others ease of coding some of them are easy to implement but some of them are working asynchronously so you

have to create manually some promises and wait for the responses uh properly and then continue the next steps but in some of them you basically put just a waight keyword at the beginning of the line and that's it so some of them are super easier than others flexibility like some of them you can easily capture the request and intercept the request that you are sending to the

servers and you can easily manipulate what you are trying to do those have those flexibilities to modify your test steps some of them have decent documentation and community supports while some of them lack uh documentation and of course in some of them some features are paid Services while uh some others are totally free so these kind of things are different decision criteria you can choose whatever is

important for you for example if you say that I don't mind what features they are supporting I I'm testing a very easy very simple product so only very basic elements components are enough for me I never work with cross domain issues I never execute my test case on a mobile device only uh executing on Chrome and only finding very basic components are enough for me so if

this is the case maybe you might choose the speed like which one is fast then go with it but otherwise if in your application what you are testing has some complex and complicated features then you might search for the one which supports those features which can be implemented easily so the decision criteria might depend but please beware of your requirements and then try to make the choice

choice based on those requirements so there's no clear answer for me play rights might be the best tool but for you Cypress might be the best one for another person selenium might be the best one so please define your requirements and do an analysis to understand which one is covering those requirements but just as a reference of course you can do a literature survey to see what

other people are using or maybe to see uh which uh Trends are in the marketplace nowadays like for example this graph I took it just today and we can show that the download rates over mpm repositories for the major test Frameworks like Cypress Night Watch play Puppeteer selenium test CFE web driver I here and we can show that a on playright which is the green line there's

a decent increase nowadays right this is the last one year graph so we can see how from where it started now it is one of the most downloaded one with Cypress too Cypress has some stability for some time because some time ago it was the mostly downloaded the most commonly used one but nowadays we can say that Cypress is not the only one anymore but sharing the

first place with play so again this is just a reference but your choice might be a different one but what I advise you is to have a developer experience what it means is whenever you do this analysis please try all these kind of options and try to implement some test cases with all these options so this is what I did on this screenshot you can see some

implementation I have done with all these options this was for a benchmark study and I picked one test scenario which was a easy test scenario and I just tried to implement with all options to see what kind of uh bottlenecks or the difficulties I have while using different options and of course eventually I measured the execution speed for each of them like 100 times I executed with

all options and I measured the average execution duration on different platforms like what happens if I execute locally what happens if I push to my cicd platform and I execute on the uh remote machines too so all this kind of analysis you can do and you can choose the best which is serving to our your goals and this is just a functional Matrix like which functions are

supported by each option so this second part was basically the tooling where you might need a benchmark study to choose the best option because again there are too many options starting from the programming language as well even starting from there how can you choose the program language some sometimes you try to align yourself the quality assurance team to the development team because to increase the collaboration sometimes

you may want to use the same language but not necessarily right whenever the developer teams are using another language you can use another one you can go with another one but you can choose the uh same too depending on the team structures or the other decision criteria but just list your requirements and try to understand which one is best for you because if you ask me what

is the best tool testing tool there's no clear answer for that it depends as always right the answer that we like the most is it depends element number three for us for maintainable successful test automation life cycle the design practices the Imp implementation coding approaches and for this I will mostly highlight the importance of the duplication whenever we have some duplicated code then it will result in

some trouble or some critical uh refactoring or maintenance issues so let's see how they affect our test implementation and the maintenance efficiency so I want to share you a concrete example from real life scenario for example on this slide we can see different uh headers in our web applications so I'm working for a job seeking platform so the users can go uh navigate to these web applications

and search for some jobs they for example put a keyword like engineer and they put a location like find the jobs in Berlin find the jobs in London for these kind of queries they can make but there are three different web applications we have all the targets are different for example one is for the part-time jobs the other is for a different Target people like for engineers

and the other is for the health care people for example but we all have the same header because sometimes you might realize this is the case right in your development teams there's a uiux team they are developing these kind of common Frameworks or components and then they share with different development teams so that they can use in their applications so how does this relate with testing activities

if these kind of common uiux teams are developing common Frameworks then as the testing team we can also develop a common test code for example when I'm testing these applications normally I have three separate repositories where I keep my test code but if I rewrite the same code again from scratch in these three different repositories it is duplication right so I'm doing the same implementation from scratch

again and again while it was already written in repository so it is not only the duplication of the effort because I'm trying to implement from scratch but also it results in some maintenance issues as well for example whenever there's a duplicated code and whenever I have to fix something on this piece of code what should I do I go to each uh point where this code is

used and I have to fix right but if I already use only one single Library where I have this code and I use this single Library as a dependency in all these three different repositories then what should I do is what I should do is just go to the library and do the fix on a single point not go to each repository and fix one by one

because I did not duplicate I just used one single code and I reused in multiple places so the key Point here is while implementing our test code please try to make them reusable please try to execute all the parameters or the user option arguments from the test code itself you don't have to write everything in the test spec files try to use helper methods helper functions and

try to exclude the common steps into some separate designated classes in this way you will be reducing the duplication and the maintenance activities the refactoring activities will be way easier than before for example when at what times do we need these kind of refactoring or maybe fixed activities for a while I'm refering referring to false alarms right what does it mean it means for example whenever we

have a failure in the test cases whenever the test report is failed but normally when the feature was working properly when there was no real bug it means it's a false alarm because test case is reporting a bug reporting a failure but it's not a bug it's a false alarm on the other hand if test case is passing which means it does not report a failure but

if there is a bug in the feature at this situation it means test case is not capable of finding the bu because it's not reporting anything it is just passing but there is an issue in the future so it's a silent horror case so these two cases are the issues which might uh result in some critical consequences in the later stages like the Escape bu what kind

of reasons may result in this kind of tests there might be several different reasons for example there might be asynchronous weights or there there might be extra load or maybe some objects which are created which were supposed to be cleaned up after the execution but which were not so they might be resulting in some conflicts since the objects are already there before creating again or there might

be several reasons so I will not list all of them but I will just warn you to be aware of these kind of test SMS there might be lots of different categories of potential reasons for the test SMS and of course there are different solutions to cope with these kind of root causes please try to uh understand why your test cases are failing or maybe not failing

while they were supposed to and try to resolve all these kind of root causes let me share you one example from my project again it's a practical example from real life so what we tried to do is it was the case the the product that we were testing was a microservices architecture product and there were some issues so I needed some retry mechanism in the test cases

because once you query for the expected result it might not be there but if you put call like periodically check until the last accepted timeout then you might have the correct answer so I did a benchmark study like what happens if I do not imp implement this retry mechanism smart polling algorithm but what happens when I add this additional supportive implementation to my test course so you

can see easily how your uh preventive actions are contributing to your reliability on the left hand side you can see there is too many failures but on the right hand side there's only once maybe it is really the uh product issue maybe it did not respond in time so by this kind of continuous monitoring activities you can see how your actions are helping you to cope with

the testm so third pillar was building the architecture in a proper way by trying to remove the duplication and trying to encourage using reusable test implementation and test code now the fourth pillar the fourth element for our test automation environment which is the continuous maintenance which is very directly relevant with the continuous monitoring activities so why do we need that why do we have to do or

why would we do continuous monitoring because it might help us to evaluate our current situation like how much progress we have already made or how much more we have to reach to our goals so this is the current evaluation but with a future estimation like if you go with the same Pace what percentage will be remaining so these kind of future estimations too we can do so

these kind of monitoring activities help us from two different dimensions and whenever we try to do this main monitoring activities of course we will be collecting some metrics right but what kind of metrics we can collect there are too many metrics we can collect like on this slide I have maybe I guess around 30 40 matrics but which one should we choose again the answer is depends

right because you have to again prioritize your needs and list all the metrics which should be or could be useful to represent these priorities or the requirements and then after listing all these metrics you might have to modify your execution environments or the implementation environments because sometimes whenever some project issue or the issue tracking systems you are using if you go with the default workflows it is

not visible enough like there are only three four state transition diagrams and it is not enough but you can modify it to e additional steps additional Transitions and it will be much more visible on which specific stage you are spending time or on which specific transition for example from an implemented task to executed maybe you are spending too much time it means why don't you start executing

after just it is implemented right so you can try to understand the root cause for this kind of waste of times so for a better visibility for a better transparency please consider modifying the default workflows and then of course doing monitoring manually is an option but you can do automated monitoring as well like try to build some code and try to execute them within a pipeline within

a job automated job scheduled job so you can already collect these matrics by sending these queries automatically over code so these are some best practices for doing the monitoring activities right Define some metrics and do the Automation and modify your environment and then after collect all these metrics numbers are great they will be explaining us a lot we can collect several insights from them but reading the

row numbers is not easy all the time so what we can do we can transform these numbers into some visuals right if we use some dashboards or maybe monitoring graphs charts it will be way easier to understand visually we can see how the trends are increasing decreasing or the distribution of different charts let me again give an example from my real life project for example this is

a chart where we can see distribution of bugs across different dimensions for example across the distribution across their status what percentage of bugs are still open and what percentage are still closed or still waiting to be closed or the percentage across uh their criticality or the priority like most of them are SE buxs or less 70 bus so you can see the percentage of different separ levels

too and not only the bux or or the historical data but also you can collect data from the production environment too for example at what times is the application responding slowly maybe for example in the mid nights maybe my web application is very slow the response time is uh very high so these kind of insights I can collect and after I get collect these kind of insights

I can try to improve them by uh building some strong quality Gates right if we do not stop the risks or the threats in the first place if you just let them go then we will have those troubles in the later stages which will be again more expensive more difficult to fix in production environment so it is better to stop them in the first place by some

Gates so I'm wrapping up we went over four different pillars four different life cycle which were basically the mindset and the customer perspective and the holistic testing uh trying to improve the testability issues trying to make a collaboration with them with uh the development team and product team and the quality insurance team and then we discussed about tooling doing a benchmarking Tool uh tool benchmarking and then

uh making the correct Choice and then we discussed about having the test reliability which is depending on the architecture and the approach or the patterns that we are following to implement our test cases and finally we discussed about continuous mon monitoring to collect some different metrics and try to get some insights about our progress right how much coverage we have how much uh or how many Bucks

we have of course only some specific numbers will not be helpful so you know monitoring should be uh implemented in a comprehensive way in a smart way if you just count the number of bucks I have 20 bucks I have 30 bucks so what it doesn't mean too much right so you have to uh collect the historical data and you have to do comprehensively analyze this historical

data so thanks a lot for listening to me if and if you have any questions please do not hesitate to reach out and if I have time I guess I yes thank you for for for your talk uh we have very few time left but may ask a couple questions first of all about flaky tests uh you mention it like like common common tools to mitigate like

a test like introduce ret trying uh so on but they don't don't always help so what should we do if you have FL T but and ret tring doesn't doesn't work so so what yes so every fness should have a reason like in my case it was the asynchronous weight so I tried to add some retries but in some other cases the root cause for the fness

might be something different like it might be uh the structure of the web page like if some components are not loaded on time or anything else so depending on the root CA there might be a different solution it is not always adding some retri but uh applying the correct fix would help to cope with the testal sure but sometimes we just cannot find the solution can cannot

detect what is root CA I mean if we apply the correct monitoring we should be able to find the result the the reason or the solution but if we not there might be some alternative ways like we can do parallel testing so if in any of those pipelines if the test is passing then we can take that into account I think that might be the worst case

okay and probably the last question uh how to test uh uh microservices I mean like complex system composed of multiple microservices which are hard to monitor Al together and even if you do monitor them it's hard to combine all this data together what happens here exactly since uh due to this fact that sometimes it is difficult to collect everything into a single platform uh contract testing is

a very commonly used approach for uh doing the microservices testing because in this way you you can uh test the microservices isolated from the others because you know the contract how it should communicate with the other services so in this terms you can apply the integration test and then eventually collect the whole system into a single platform okay so I suggest to test every micros service independently

it's absolutely good idea but often people say that no no no we want to test the entire system yeah but this is why we have a test pyram right you start with the integration test or even the unit test and eventually you have the system test where all the components are all mhm uh okay thank you I think we has we have finished these questions and yeah

thank you for your talk sure here we will have some lunch break probably about an hour and see you after the yeah thank you very much thank you