Filip Hric: Blink and You’ll Miss It: Stories From Discovering Most Annoying Bugs and Test Flakes

In this talk, Philip Critz shares insights and strategies for reducing frustrations associated with flaky end-to-end tests. He emphasizes that while the execution speed of tests might be a common focus, the maintenance speed is often more critical. The speaker provides several examples from his experience, particularly with the business intelligence software Metabase. Utilizing tools like Replay Dev Tools, he showcases how developers can debug tests effectively by gaining context around failures using features like time travel. He concludes that addressing the underlying issues in applications, rather than just test failures, leads to more reliable testing outcomes and highlights the importance of becoming a more technical tester in the evolving landscape of software development.

[Music] hello again and uh we're back with Philip critz um who started out as a tester in slido he's a Cypress Ambassador U was a keynote speaker on the apply tools frontend test Fest and his mission is to teach developers about testing and testers about development so he's going to do a topic uh that requires us to pray to the live demo Gods because it's full of

um a sequence of little demos uh to show us how to um track and find and debug the nastiest of bugs it's going to be very example driven and it's going to be more on the qualitative and not so much on the quantitative side of data Phillip over to you thank you thank you very much for the lovely introduction I'm really happy I'm really happy to be

here you see my voice is already breaking really happy to be here thanks for inviting me for for this wonderful event I have seen that there are people already active in the chat so I encourage you if you have questions throughout the presentation feel free to ask them and uh I'm hoping to get to those at the end of the of the presentation which I have called

blink and you'll miss it uh and I'm going to be sharing some stories from discovering the most annoying uh test flakes but what I like to do is to look over to the audience and we're not live well we are live but we are not in the same room together but when I'm in the same room with the audience I like to ask a bunch of questions

uh at the beginning and the first one is kind of a kind of obvious one right are you writing n to and tests I'm assuming if you came to this presentation the answer would in general be yes so there's a follow-up question to that one has your Ann test ever failed uh again going to jump into assumption that probably yeah if you haven't seen your test failed

uh you might not want to trust them actually so make sure you see your test failed um has your test ever failed and you weren't sure why it failed if we were in the room right now maybe fewer hands would uh would uh raise up uh above and maybe some of you would still hold your hand up um have you ever spent more than one hour debugging

your tests has that ever been the case for you I can safely say that it has been a case for me have you spent even more maybe two hours or more than two hours have you spent more than a day debugging a test flake or a test failure I have to admit I have been there and if you have been there too you maybe feel a little

sad right now I'm hope I'm not resurfacing some trauma but if you answered yes to the latter questions you may have experienced frustrations that come with UI n2n tests so this presentation is uh I'm hoping is going to be about possible ways of reducing those frustrations now at the risk of half people leaving my presentation I'm going to show the testing pyramid you probably know this model

I think like everyone knows this model it's been in every testing presentation for last 15 years and I'm showing it as well uh at the top the end to end tests are costly and slow unit tests are cheap and fast and today I'm going to be talking about the top of the pyramid but I believe that this model is kind of outdated uh we have much better

tools nowadays uh and especially if we look if you look at the top right corner with the dollar signs I think the cost of creating a test is not so high anymore and as a result what we have today is we have lot of endtoend test but let's stop at this little guy which is at the uh top left side and that's the speed of test we're

tend to We tend to think that entn tests are really slow and as I said we have some great tools nowadays uh but they all seem to be battling for Speed of execution we're looking at which one is the faster I have read countless block post comparing playright and Cypress and selenium and some other tools and they all seem to be focusing on how fast will execute

but if you answered yes to any of the questions that I uh answered at the beginning of this presentation you know that there's a difference between speed of execution and speed of Maintenance and the funny thing is is that we sometimes don't notice this right uh when it takes 30 minutes for your tests in the uh for your test to run in your pipeline we as developers

or testers are happy to spend a month of our time migrating from one testing tool to another so that we can take it down to 25 minutes right and we do all this work just because we believe that maintaining tests is just too much work uh we're still mentally living at the top of that pyramid we're still using that model and to be completely honest I think

in many ways we really are living on top of that pyramid so here's an example we have this so-called strategy IES of helping us live with the fact that our tests are flaky uh I've seen these strategies across many companies I work with uh strategies like muting a test or quarantining a test putting it aside maybe retrying a test adding a longer timeout reproducing a test locally

deleting a test or as I like to call them shock denial anger bargaining depression acceptance so that's a little side joke here but these strategies exist because we have limited we have limited visibility to what is happening on CI as they say what happens in CI stays in CI and many times what happens when our test fails fail on CI is that we're simply left with an

error message that error message may look something like this and the main problem when dealing with entn test is the lack of context right entn tests usually copy some kind of user story so when an element is found it tells us pretty much nothing about what exactly has failed and when and when tests end up with a error like this the problem is that the the problem

that actually caused the test to happen or caused this error that we see it may have occurred like a couple of steps before right so we are lacking context and I want to share a couple of stories uh with you that start with maybe some mysterious message like this and these are stories that I either encountered personally during my time as a tester or consultant or with

our clients at replay where I currently work at so one of those clients is metabase uh metabase is business intelligence software uh what's really beautiful is that they're open source and I really recommend you to uh take a look at their code and see how a large test Suite looks like the team has done amazing job there but uh as always when you have a large test

Suite you have a lot of flaky tests a lot of problems so we came in and helped them drive down their test flakes uh they have have hundreds of end to end tests and quite a big big of test flake and with integration of Replay we were able to drive it down to practically zero or zero on the main branch but like any new test flakes that

were introduced could be fixed quickly um so let me show you an example uh this is a fairly simple test uh the team wanted to make sure that their you that users with slower connections know what is happening so they would have this uh simple test that would make sure that the loader animation is visible uh and the test actually failed over and over which was really

strange because before it was passing and none of the codes that it was testing has changed and this was really frustrating for developers why is my are my tests suddenly failing um why did the spinner not appear so we recorded their test with replay browser we recorded it on CI and then started debugging so what you see here in this video is replay Dev tools uh it's

pretty much like having a chrome Dev tools but in CI and with time traveling capability so on the left side uh of this uh of this video you can see that as we Zoom mean you'll see this uh number in a in a blue in in Blue Square or something like that uh and that one indicates the number of executions so not sure how well you see

it but it shows 27 times so that line of code was executed during the test run uh it was executed 27 times so we can time travel to every execution of that line of code and as you can see there's a Comm click which you can use to go to that uh to a particular execution and there we can see and we can see that on the

right side in the view that there was clearly a moment where the spinner element appeared so this is kind of strange right why was the test failing when the element clearly appears uh was the test going too fast uh for the test to catch it or was the application going too fast in order to for the test to catch it and the answer is yes now metabase

was actually smart enough to think about this and they have implement this implemented this throttling strategy so while the spinner was waiting it was there to uh to wait for a response from a server so in order to catch that spinner uh metabase Team would implement this throttling strategy and and they would slow down their request by limiting kilobytes kilobits per second which is very smart but

there is one catch when the response uh comes from the URL it's going to be it's going to resolve in different time if the response is for example 100 kilobytes uh big versus when it has one megabyte so in the end when the response from this URL was small the test ran faster and the spinner showed for a shorter period of time in fact it showed for

such a short time that Cypress uh the Cypress script was not able to catch it so in the end the solution that was implemented uh to make the test stable was really simple instead of set throttle to limit the amount of kilobits per second uh they have set up set delay and then the text uh the test was fixed and this was largely due to the fact

that we were able to time travel to the line of code and we were able to record their test on CI where the failure actually happened because trying to reproduce that locally would always pass there would there would never be a problem if you try to record that locally all right there's another uh uh use case from metabase which I want to show you uh this was

a test flake that actually inspired the name of this presentation blink and you'll miss it in fact what I'm going to do now is I'm going to show you a screenshot from failing and passing test side by side so you see two images side by side one of them is the failing test uh and one of them is the passing test can you see which one is

which or can you even see the difference between those two if you see you have amazing eyes I actually want to show you uh in a video can you see it now can you see it going back and forth the difference is really minimal but this is what actually caused the test to fail so it was a simply a simple CSS change but what's really interesting is

how it actually affected functional side of things uh this was kind of most confusing part of it this is what the error message that we have seen uh the aor message there there was nothing that pointed to a problem with CSS in fact it really looked like a error sorry um because what we expected was some data we expected a certain date range right uh there was

a graph we did a drag drop and we were uh expecting that the date range would start on April 1st as you see over here but it actually started on May 1st so that was kind of a strange error uh to give you a better idea of what the test did here's the test code uh this is how the test looked like this is the original test

now the test tried to select a range and it did that with a series of of trigger commands it would do the mouse down then Mouse move and then Mouse up so sort of a drag and drop action in in three commands and it tried to select a certain range range and then it would try to validate whether that range was indeed selected and this is where

things failed now usually when you have a problem like this when you look at a uh data problem error the first instinct is to go uh to look into the network panel and since we recorded those tests on CI we were able to look into the network panel and confirm that the date range that was selected was indeed on May 1st so test was actually selecting the

range from May 1st so our test was doing something we didn't really expect it didn't behave the way we wanted and going back to the video comparison we can actually see why the test did not work work the way we want it the CSS changed uh the CSS change caused the whole layout to shift slightly uh so Cypress commands that relied on precise coordinates on the page

When selecting something were different after the page uses slightly Bolder for Bolder font not sure how well you can see that on this video but you can see the text is sort of changing from Boldt to thin and that is what actually caused uh caused the trouble now this would normally be super hard to debug but since we had that data pulled from the CI execution we

were able to find that fairly quickly in fact I believe this was uh just a couple of minutes of of debugging so that's sweet all right uh my next story is very common if you are testing uh if you're writing end to and test chances are you already seen an issue like this race condition so here's an example with a very simple e-commerce site uh it's a

uh it's a shop or e-commerce uh site where you can buy a sneaker and we written a test that attempted to buy that sneaker and it was flaky sometimes it will fail sometimes it will pass and when it failed we would see that we would click on the button too soon uh so on the left side you can see the code that we have used on the

right side you see the you see the page now we try to fix the test right by adding uh weight to this test which is you would say probably not the ideal solution right it's a slipper slope because we may be increasing the time but we are not really fixing the real problem so I'm actually I actually want to dive uh into this one with you and

show you how a debugging process might look like uh so let's debug This Together uh we're going to dive into the code and figure out what actually made this test flaky and here's what we know we know that test opens a page and clicks on add to card button we know that this test sometimes passes and sometimes fails uh we see the error the error said that

we wanted to find a card product added to card but what we actually had is a message that said cannot add zero items to card so Something Fishy is going on here uh test seems to click on the ads to C button too soon uh test runs locally fine everything is fine locally but it actually fails on Ci or sometimes it passes sometimes it it fails and

uh the application that we are using uses rest API to fetch data so there's some backend communication going back and forth all right so let's now jump into that uh so now I'm sharing my screen with you with the recording of that test I'm going to um make this slightly bigger so you can you can see that well but here I have a recording of my test

execution so I have a fail test and I have a passing test and I can take a look into the details of both the failing test and the passing test so here's what we have all right so um this if you uh if you have never seen this this is sort of uh this is the replay interface this are the replay Dev tools it's like having your

browsers in your uh browser Dev tools your CI and you can travel back in time and stop at any uh any point of time so here's a timeline where we can jump back and forth and see our test being executed we can play it like a video uh but what we want to do really is to debug what's happening so what I'm going to do is I'm

going to open this failed test and what I want to do uh if I take a look into into how this the how the test actually went I can see all of my commands I can see visit I can see contains I see my assertions I see the click and what I see also is my xhr requests so I'll head over to the right side and open

the network panel where we can see all of the apis that have been made during this test so I'm going to do go ahead and filter all the fetch requests so we got a bunch of those and I can see there's some Dev uh Jason and I have two API calls and one of them is passing it's giving us uh status 200 and one of them is

failing that one is saying add to card and we got a response cannot add zero items to card which is interesting this is probably our first clue into that something is uh is slightly off now you can see that I have a preview of my application from this exact moment so I can make this uh view bigger and see the exact moment when that click happened and

there are a couple of interesting things happening in here first of all we see that the sneaker is not available and the number of sneakers is zero and uh yeah and that's pretty much it so this is this is suspicious this is not something that we uh that we want to see so what we can do uh we can add comments but that's not what I wanted

to do now uh we can go ahead try to take a look at this click action that happened during our test and we can jump to code so when I click on this button jump to code what's going to happen is that these Dev tools are going to take me to the exact line of code that was executed when that click action happen so we have this

very nice connection between our test script and the application that we are testing so I'm going to hide this side panel for a moment and just focus on the code that's over here now I see number of lines uh and how many times they were executed so I already know that there is some problem with the add to cart API call right and I can see that

when I click this add to cart API call uh is what's being called right wait I wanted to here we go so when we click this add to cart function is called and our add uh to cart API call is being made so let's take a look at what this code is actually doing it's adding uh our items now it uses the method of post it has

some headers and here's the body of our uh of our API call so what I can do is that I see that there is this qty quantity body uh being shown so what I want to know is what this quantity actually was let's try to take a look now usually when you are using je tools you have your console available and If you're interested in what your

code is doing you normally add a conso log now in replay Dev tools you can do that as well by clicking on this plus symbol it's kind of the same thing as if you would add a console log into your code but now we're not modifying the code we're just looking at the recording of our test execution uh and our application so as you can see when

I add this plus symbol something is going to print out in my console now if I want to change that I can do that let's do hello world and now my console is going to say hello world but that's not really useful what I'm really interested in is this quantity so let's print out the quantity and now we can see what we are sending to the server

now this is rather curious we are sending number zero to the server are we trying to buy zero sneakers it seems that way and it seems that when I go go back to my passing test and look at the same execution now that number is one so let's annotate this a little bit more and try to see where this quantity is coming from so sent to server

quantity let's give it a purple color and now we have this uh conso lock nicely annotated all right let's go back to the fail test and keep on De so where does this quantity come from well why not search for it I'm going to type command F and look for quantity quantity should I if I don't make a typo it will appear and I can see we

have it appear in the counter that's where we are using it uh so that's fine but where is it defined it's defined at the very top of this react component so if you already familiar with react you know what this is if you're not it's fairly simple this quantity is like defining a variable but is actually a state variable uh so in react we Define a state

variable and then we Define a function that changes that variable so we have this um quantity and set quantity uh sort of pair of functions so this used State means what is the default state of that so that's set to zero so let's try to see where this set quantity is actually being called so I'm going to search for set quantity now and here I see that

this piece of code is actually calling the set quantity function so this is where that zero changes from zero to something else well how does it change well it passes on this data qty whereas the data qty come from it comes from this constant data how is that constant even made well is being fed by the data from API this fetch availability function where is that fetch

availability function come from well it comes from uh this definition what does it do it will check availability so let's go back to our Network panel and try to look at our fetch xhr to see what we have here so we got two API calls we got check availability and that response will say what the quantity is now this quantity is then fed to the set quantity

function to change the number of available items hopefully you're still with me all right and then we want to use that quantity and send it over over to server now it seems like this is not happening but let's take a look let's take a look at what this data qty was when this line of code executed so I'm going to do uh fetched from server actually let's

do quantity fetched from server and I'm going to add data qty and let's give it a green color all right now this is interesting but from the server we're getting back number one this is rather strange let's take a look now at our passing test if I go to my passing test I see that the quantity fetched from server is one but what's being sent to the

server is also one and what's really interesting is the order of these console logs because these actually appear in the order of how the lines of code were executed which means we do have a raise condition we are clicking on that button too fast our test doesn't really care whether that check availability API call has made it all the way back it actually just clicks on the

button as soon as as it is possible now what usually happens when we are dealing with CI executions is that we're saying oh the test is going too fast but what I think it really is doing is that it reveals problems we already have right if you take a fast execution on CI you can scale it down and take a look at a very slow machine that

a user has now everything is going to load very slow for them um so they might end up having the same problem they might click on the button way too soon and try to order zero sneakers instead of one um all right so let's uh let's continue so how do we fix that test well as I mentioned there's this counter uh uh component that's uh that's in

our application that uses this quantity that that was the plus minus uh symbol you might have seen there and then there's the at card button which we actually actually click on when we want to uh when we want to buy some sneakers so in order to fix that we can go ahead and when the number of quantity is anything else than zero we're going to add a

the is disabled property to the add to card button component now we should add it to the button component as well and then we can add an HTML attribute is disabled so that when we have zero sneakers available we will not be able to click on that button and then add some styling so we make this button semi-transparent and now our test is fixed so as a

result our code becomes much cleaner this is what we had before we had to add a Cy weight we had to wait for one second in order the page to be fully loaded uh but now we can take this out and this is what you should do as developer who owns test flakiness right you don't want the users to have the same problems as your tests have

and it accomplishes two things first of all it fixes hidden issues that customers actually have but never report and it makes their test automation more stable and therefore more reliable I'm very confident that more than 90% of what we call flaky test is actually flaky app it has nothing to do with the test code itself this is how our test should look like this is fine all

right let's go to one more example we all like our tests to be green right but there are cases where green test actually hides a real issue and I want to share a story like that you uh so here's a false positive flake that we have started with this is a this is how a test looked like we had a before each we visited a page we

waited for a loader to disappear and then we would have two commands inject X and check accessibility so this is a uh Cypress plugin that uh checks uh all the elements on the page and makes sure that the accessibility is uh that all the accessibility checks are passing now now what was really uh frustrating about this one that this test became flaky but what we actually wanted

is that test to fail all the time in fact when it passed that was the problem we had a false positive uh in our code base and we needed to fix that so let me show you what how that looked like again I have some uh I have some tests over here uh or recording of the of the test over here which I'm going to show you

now I have two tabs open over here I have the failed test and I have the passing test so uh let me take you to a to a place where you can actually see something here we go now visually you can see that they look kind of different there's this model window uh in the fail test and that model window is missing in the passing test and

some of you already uh kind of know what's going on here right so but let's let's break it down a little so here's our failing test and we can take a look into it into what the test was actually doing we see all of our Cypress commands we see that we visited the page we waited for the uh for the loader element not to exist then we

injected the accessibility script and then we did the accessibility checks now what's kind of funny about these accessibility checks is that when we get an error it looks kind of like this we have found two accessibility violations but we expected there to be zero of them which is kind of a non telling right and uh ending up with error like this it's not going to help you

really debug the test or even figure out what is going on uh the nature of Cypress command is that they sometimes hide what's going on uh behind but what we can do we can actually see this violations so we can go ahead and jump not only to the to the application code but also to the test source so when I do that it will take me directly

into the code of that accessibility plugin and we see that on this line of code it Returns the violations so what I can do now is is again to add a caal lock or uh to add a print statement to kind of reveal what was actually happening during that test so let me print it out let me print out violations and see what's in here all right

so now in our console we can see two violations we have button name impact critical buttons must have discernable text and it seems like this one is not not uh not having that here are all of the details about the problem so accessibility plugin does really good job of that so what we can do is we can take a look at the exact note the exact thing

that caused the problem and we can see this element over here it kind of doesn't have like too much information about it but we can see it's button class absolute stop so what we can do is to go jump to the elements panel and try to search for that as I search for that I can see that this is the element now in the viewer as I

click on this you'll see that element highlighted so this is the problematic element it contains a button that does not really have any uh any accessibility attributes and therefore it's it's inaccessible uh people with accessibility problems have have problem uh with uh with uh noticing that that button is there all right so now that we have searched the elements that's great we know what's going on uh

but what we can also do is to not only look at the elements panel but you can see there's a react panel as well so if your application is written in react you can debug even that and selectable uh or inspect element uh uh thing going on so I can go ahead click on my model and this will take me to the model component I can even

search within the components but here's the model component and that's that is kind of problematic I can see the properties that it is open and I can go ahead and jump into the code into the definition of that model component now here again I can see how many times this uh code was executed and let's make let's try to compare that to the passing test so when

I go to the passing test I can go to the dev tools and I can search for a model. TSX component and I can and this is actually what kind of hints what's going on here in the failing test we see this component being called in the passing test we actually don't see that so when our component uh when our model window our accessibility checks fail uh

when our test passes the model does not appear which means it is the model window that actually has the the problems now since we have recorded those tests we are able to see that um so now what we can do is we can uh now that we know what the problem is um is that we have uh that the test when it was testing it has done

the accessibility check too soon it has done it before the model appeared and that gave us the false positive so what we need to do is we need to make the sure make sure that the model is present uh we need to check the model presence first so we go model should be visible and then we'll do the accessibility check so these are just couple of stories

but there are definitely more uh and I could probably talk about them for hours but I only have limited time so I'm going to wrap it up now uh here's the thing about test speed as I mentioned we obsess over uh speed of execution of our test tools now I have shown a lot of examples about Cypress but you'll uh first of all replay works with playright

as well and most of these problems actually all of these problems would appear in playright as well there has been a very nice discussion on LinkedIn going on recently that testing tools are not actually the cause of flakiness which I partially agree with but this kind of demonstrates the point right it we can have the faster execution but test flakes will not go away as I mentioned

we obsessed uh over the speed of execution of our test tools but this is actually what we should be looking at how long does writing tests take how long does the debugging of your tests take if we want to drive down um the the if we want to make improvements to our speed we can do that only by driving down the debugging time uh that way we

can achieve real speed improvements all the test flags that I have shown you would normally take hours or even days to figure out maybe even maybe they would even be impossible uh to figure out but when you have the option to time travel you can take it down to minutes uh so thank you very much uh it has been again pleasure uh to be here to talk

to you about my favorite topic which is flaky test if you found replay interesting you can scan the QR code on the left to uh to give it a go if you're interesting go over going over to my slide it has links to those replays that I have shown you can scan QR code on the on the right and I'll be happy uh to answer any of

your questions if there were any uh and uh yeah see you around uh you can follow me on Twitter LinkedIn uh wherever and uh I'll be I'll be happy if you if you send me a message much thank you to Philip and um I have one question and I'm sorry that it's going to be about the AI hype again but how is testing profession going to change

according to your opinion with AI coming into the game because we already know how it's working out for developers yeah um yeah so I had uh I had a slide about this topic in my recent talk test like a developer develop like a tester and I uh made a appeal for the testing Community to become more technical because if you have interacted with Chad GPT or some

other AI uh thing you probably saw that it will give you an answer but you kind of need to check it you need to make sure that the code it's sped out uh is actually working is actually the thing you you need so there's a lot of not only human input but actually human expertise so I think yeah AI is going to take some of the jobs

away but if you want to make sure that it doesn't take your job away you need to become an expert at what you do and that way you can be the person that checks after the AI whether the thing it's spinning out is actually good because it's being modeled on some uh horrible code that's out there is being mod uh um built on top of some good

code that's out there but distinguishing between good and bad and distinguishing between the actual intention you have uh that will always be a human domain as as far as I can tell so uh yeah so that's that would be my advice get get more technical get more knowledgeable in your area of expertise and you should be fine yeah I I don't really think that AI will take

our jobs I mean I'm hoping it's going to take all the housework away from me but that's not where it's headed at the moment but maybe I think one of the things that it does well is Munch a lot of information to uh so that you don't have to so in in these very long hours of trying to find something I'm not sure it's going to be

a chat bot as such or like if you're going to have to become better at prompt engineering or something else but it's going to be interesting how it can help you track down issues faster yeah and yeah of course obviously the fact that a lot of the code is going to be AI generated is also going to testing yeah yeah exactly so I I I think we're

I think we're safe uh but we'll see what the future will be yeah thank you again for the great talk it has been very uh entertaining and informative at the same time and uh to the audience we now have uh uh 1 hour uh lunch break I think am I correct or do we have another topic yeah maybe another topic and then another hour lunch break so

have a nice break and see you later bye see you thanks for having