About this talk
This talk explores the role of large language models (LLMs) in testing, particularly within the context of test automation challenges. The speaker discusses the growing complexity and scale of software applications, which make traditional testing approaches increasingly ineffective. They highlight that shifting testing responsibilities 'left' does not resolve the issue of code complexity, as developers continually face challenges related to maintenance and prioritization of feature delivery. The session critiques existing AI tools, noting their limitations in modifying existing test code and generating reliable outputs. The speaker presents a new platform called Autonomous, which aims to combine AI capabilities in test authoring while maintaining stability and determinism during test execution. This innovative platform allows users to create tests in plain English without writing code, enhancing usability and efficiency across testing teams.
Full transcript
Hello everyone. Am I audible? Perfect. Hello everyone and I hope you enjoyed your lunch. Uh and today we're going to talk about LLM large language models in testing. And we're going to cut a little bit through the hype uh around LLMs. Uh if you have LinkedIn, which I'm sure you all have, you probably have anywhere from 20 to 50 links that you promised yourself you are going
to read at some point because someone wrote a very nice post about LLM's AI and what it can do for you, right? Personally, I have like 50 of those which I never had the time to uh found the time yet to to read them. So or I should point there. Perfect. Ah, like this. So we are living in exciting times where uh we're seeing explosion of innovation
powered by LLMs particularly around the application development and software or test application. It seems that every week there is a new announcement of flashy product or flashy demo that is promise us the word. But how much of it is really act actually useful for test automation uh and how much of it just in a hype in this session we will talk exactly about So let's start by
talking about the current state of test automation. As you all know in the recent years there has been an exponential growth in the complexity and the scale of out of uh software applications are getting bigger and bigger and companies building more and more of them. The user experience get more and more complicated and hence far more difficult to actually test them. On the other hand, teams are
releasing features or required to release features more faster than ever. Uh, and to do that, guess what? With less budget. Okay, I got this. Now, this is the button, not the left one. All right. So, and now I got it. So what we hear from our uh from teams where all around the world from our customers and from the research that we've done on oursel is that
actually traditional approach for testing doesn't scale. There are probably the number one uh uh factors for productivity loss, velocity loss and hence increased cost. As a traditional approach to test automation failed to keep up with growing complexity of modern software, the industry responded by shifting ownership left and left and left. It easy to see why the industry embraced this approach of shifting left. It promotes the value
of engineering building features that works. It promised earlier detection of bugs and tighter feedback loops. Encourage building testable application and also avoid int uh introducing automation breaking changes. Unfortunately, shifting left doesn't solve the core issue. Whether you are application developer or test engineer, at the end of the day, you still need to write a lot of code. which introduced us to the following four factors that writing
this to code first is being done by limited manpower people who know how to write code and have coding skills. Hence they have a major maintenance burden sorry ma um major maintenance burden because the fragile locators at the end of the day we need to maintain them down the road of testing. It creates a velocity loss. Every time that the developer is required to write test and
to do context switching from building the app to building the t to writing the test, it's actually time that they are not spending on writing new features. And it also introduced a never-ending conflicting in priorities because in most cases developers are measured by features delivery and not necessarily by testing coverage. So at the end of the day, we're are finding ourselves in a loop where we are
constantly chasing our own tail. As team discovered that shifting left doesn't solve the problem, they turned to the new wave of AI innovation that promised them the word. Putting aside vendors adding prompts to their legacy testing tools to check the marketing box that their tool is AI powered, most current AI tools focus on helping developers to write more code. But do they really solve the problem for
automation? When we review code ccentric tools such as GitHub copilot, playright, selenium mcp or llm powered IDE such as corsor or direct code generation uh uh platforms such as chedgd, claw, gemini and others. What we are seeing is that they are really good in generating uh they are not so good sorry in uh beyond unit testing. They are far better in authoring new code rather than editing
a code. They're really really struggle with assertions. They're doing first first grade uh uh schools uh assertions when most of the time and at the end of the day code generating tools demand constant developer oversight. If code is the problem, why not to use LLMs to run the test without code, right? The idea sounds very appealing. Let's describe the test scenario, put it into a prompt, and
get a tool to do it for us. Some tools and fancy demos promise just that, AI that drives the UI and validate functionality on its own given test goal. would you trust an AI to make up test steps live during each run? Would you trust it not to be not to hallucinate or skip important actions? Would it really catch bugs instead of working around them? Would you
accept no uh ability to debug or fix test failure? Would you accept multiseconds delay between test steps? And above all that, will you pay all those billions of tokens that it will consume? I guess the answer that for all of you is no. No chance. Let's be honest. We cannot compromise on the fundamentals of testing which are stability, determinism, and speed. They are not optional in automation.
They are essential for effective testing at And LLMs at the end of the day, they are statistical models. they do not provide the consistency and determinism that we need. So if traditional approach doesn't work, what should we expect from the next tool, the AI tools that we are going to see in the upcoming future? We would like tools that will allow us to attain near complete test
coverage. Nowadays, most companies uh admit that they coverage around 20% where we would like to have over 80%. We need tools that will lower the required skill set that anyone can use. Now with AI, we would like to drastically reduce the human involvement in the process and also the the reli to have a reliable and trustworthy platform. So at Appli tools, we thought really hard about how
we can take the good of both worlds and try to remove the the bottlenecks and challenges that I just described. And we came up with a platform called Autonomous that now allows you to have a full test development platform at at the at your fingertips. Sorry that was loud. If you are testing a website for example or a multi-page web application or a CSM or CMS sorry
with with our platform what you can do is to e with oneclick setup we will uh validate the entire page AI now can crawl across your your UI your website and validate each page on its own. How will it do it? It will actually look for regressions between current versions and new version. That's something that we can definitely do. It will also find for you broken links,
errors uh uh and network uh failures to test anything beyond the website. We also allow you to describe custom test flows in plain English. You can write test steps in English on the left hand side as you can see over here. and they will be automatically executed on the browser in the interactive browser. So you would be able to see what the AI is doing and to
monitor it. As you can see over here, we just say type Adam car me in your name field. No selectors or locators are needed and hence the test will be far more stable and less will require less maintenance down the road. It will allow us to speed up the authoring by interacting directly with the page or just uh perform the actions on the application itself and then
it will be translated to commands as well. Another important thing for any testing platform is the ability to debug it. Okay. So not only that you can write the test in plain English and now anyone in the organization can see it and know what exactly what it's supposed to do. We can also tell the platform to run this and that steps more edit them and you get
an immediate feedback about what the AI is doing. The more important thing is that we use AI only for the LLMs the large language models during the test authoring when the human is involved in the process. Once the test was executed or created every subsequent run it will be determined deter determine it determined steps each and every time it will not have elucinations. It will do exactly
what you are seeing while you are building the test. How did we do that? The platform is fully is 100% sus. So you don't need to install anything. There are no changes between your local machine to the machine that will run the test. Network changes, cookies and etc. that are the constant changes for those failures. As I said, LLM used only for authoring. LM using heavily in
the product but only when we're authoring the steps. [clears throat] because they operate on natural language, they perform much better than with test code. During test execution, no LLM is involved. Another thing that when it comes to functional testing and datadriven testing, autonomous also allows you to do anything a developer can do with your coded test. If you need to have assertions, of course, we can verify
that the text name field is capitalized or lowerase. You will have that in your hand. And you can see that I can just write it. I don't need to write any function. This is a simple uh example, not a complicated one. We can do test parameterization. as we can see over here, type name in your name field and that's it. We can capture values and variables, extract
values, ext execute JavaScript code and everything you need to do to have the full flexibility to h uh be creative and to get the test to do what you needed to do. Beyond textual assertions, Autonomous also allows you to have visual assertions powered by visual AI, which is by far the most advanced algorithms for visual testing. Visual AI is a large collection of computer vision algorithms that
mimic the human eye and human brain. The way that it works is that actually every time that you're running a test, we're going to take a snapshot of the browser of the app under test in different form sizes, browsers, and screen sizes and compare it with the last approved version. We have couple of customers in the cloud in the c in the crowd here that can justify
about it. So when you're going to run the test, it actually will allow you to test beyond the test beyond what you can imagine. For example, in this Wikipedia example, what we ask the test to do is to actually find regression on a static page. And it allows us to find here that the small and tiny icon had changed without that being noticed. Think about it. Who
on earth would write a test that will validate such a uh icon change? I guess no one will, right? Just because it's really hard to find those and to think about all of them. But with visual AI, if it changed from what we approved it, we will find it now. Okay. Okay. The next thing that we need to talk about is Wait a minute. It doesn't really
behave as I expected. Perfect. The next thing that we need to talk about in testing is the in in scalable testing is the maintenance. Whenever changes are do happen, we would like to minimize the the uh level of maintenance that we need to do. So here AI will be involved again and will mimic your decisions. Every if you will see an icon change and accept it as
the new the new behavior that you wish to see then it will automatically propagate this decision to other tests that had this change to reduce the overload that on on the team. Another thing that we've added into the platform is the API testing. We would like you to be able to initiate HTTP request. It is fully configurable uh and it's really easy to author those tests. So
you can see here send an HTTP request to this and that. And you will see in a moment how we can then debug the results, extract the results and use them and validate them across the test. Here it is. you can see the structure of the results and then build your keep on building your test upon that. Okay, again with no code at all and without selectors
whatsoever. You can just use that along the way. LLMs are also assisting you in authoring the test itself. First and foremost, there is an error corrections platform. It automatically corrects syntax and grammar mistakes. So if you say clack on something, it will change it to click on your name field. It also helps you with step simplification. If I will type uh below your name, click the field,
it will actually write it in a meaningful way that further later on anyone in the team will understand and change it to be to be click on the submit button. You can also write in other languages and it will also correct you. It also help in step disambiguation. Wow, that's a tough word. Automatically disambiguate step to clarify how the step should execute. And above all that there
is a smart flow reuse. So for example, if I saying fill out the form or clear out the form, it will actually call the function that we created before that that listed the fields that we need to clear to make sure that the team reuse the the test that they created. I need to remember that I need to click this way, not here. So yeah, pretty cool,
right? We can use LLMs, we can use AI without suffering from the hallucinations and the challenges that are embedded within it. How come we took the power of AI only for the step of where you are where the human being is part of the loop? When you author the test, once we author the test and we created it, from that point on the test is deterministic. It
will always operate the same way as long as the app doesn't change, of course, right? Which is then what you're want it to be. The main thing that I want you to take from it is that test automation doesn't need to be code first to be powerful. It's need to work not to be in code necessarily. let's look on what we've talked about today. The traditional approach
to test doesn't really scale the and the main bottleneck for it is the code itself. The fact that teams try to shift left to address this bottleneck, this code bottleneck and shifting ownership to the teams that are by by definition the more code oriented in the organization, the developers themselves is a proof for that. But in reality what we've seen is that it just created even greater
challenge because they are measured by other by other factors and they and they have other inspirations to do in mind. Then we've talked about the fact that the industry tried to react to this challenge by uh embracing AI tools that at the end of the day will just write more code which is not necessarily that good and we've we've heard about it. We've heard all about it
in all the conversations Why? Because we are in the testing mindset. We are all looking for those challenges in the code, right? AI is great in writing new code, is great in writing unit testing. It's not that great in authoring, in editing existing code, in interacting with complicated environments. It elucinates, which is the last thing that we can allow ourself in testing. We also talked about that
LLMs are not sufficiently reliable for executing tests at scale for all of those factors. And we've seen how taking the power of LLMs and removing the hallucination factors allows us to have uh efficient and reliable platform for doing testing and we are at Apple at this platform. We've been doing that for the last 13 years. We've been doing AI for testing. Believe us, it's hard, but it's
very much rewarding. Thank you very much for your time and for having me. [applause] >> Any questions? >> Um, hello, my name is Nikolai from Web Pros. Uh so what kind of approach do you actually use to make the test deterministic? Do you uh generate I don't know selectors x paths or something or do you point to the point on on the web page that needs to
be clicked or something? >> Amazing question. So the question I just repeat on it. How do you make it deterministic? Right. So remember that we've seen that the commands are saved in English. Click on this and that button. There is no selenium behind the scenes. There is no puppeteer or cyprus or anything like that behind the scenes. We're actually saving this the the intent such as click
on this button or fill out this form that you that you wrote and then each and every time we'll learn the page and we know what was the expected behavior in the last run and we implement that. So even if the selector changes we're uh or the location of the button is changed we're constantly learning the page we're for example if we're talking about click on the
submit button we are searching for the submit button on the page okay we're also using one sec one more argument we are also using the selectors behind the scenes and we have self filling behind the scenes but it's not uh selenium per se behind the scenes did that answer your question or >> partially U okay yeah so uh if you're talking about learning behind with the scenes
or whatever it implies that you are using some nondeterministic technology >> correct no okay go on go on >> basically if you're capturing intent you can use it to adjust your scenario later But uh you cannot use it to produce deterministic results because intent is not deterministic by its nature. >> I got your point. So the way that it works is that when we have the intent
or the command that you provided, we are learning what the button what let's say it was a click on a button. Okay, just for for >> for having a con convenient conversation. Okay, so we are seeing what was the element that was clicked and then we are trying to find in different ways. Okay, we have selectors in place that we have we have uh built-in uh uh
self-filling uh capabilities on that and we have other capabilities also visual capabilities that we have on top of that uh but it will always try to do the same thing for you. Okay. So it's some kind of deterministic heruristic. >> Yes. >> Okay. >> Thanks. >> We try to reduce or to reduce the uh degree of freedom. Okay. >> Okay. I think uh there isn't any more
questions but if any arise feel free to contact. >> We're at the booth. You're more than welcome. >> Thank you. Thank you very much. [applause]
