Berry Langerak - Elevating Quality: How AI Enhances Testing in Continuous Delivery Pipelines

In this talk, Berry Longstock, the CTO of Tila, discusses how the company leverages AI to improve the confidence and quality of their testing processes. He shares insights on the challenges of testing software, particularly with their application platform, Nexa, which competes with services like AWS Fargate. Berry emphasizes the importance of end-to-end testing over unit testing for acquiring confidence that the application is functioning correctly. He introduces Quinton, an AI agent developed to identify coverage gaps in their testing framework, highlighting the challenges faced during its implementation, such as defining features and managing the non-deterministic nature of AI models. The session concludes with Berry sharing valuable lessons learned and encouraging the audience to adopt similar strategies for enhancing their testing methodologies.

Is this thing on? Yes, [music] it is. So, it turns out. Well, this is a new experience for me because I've never done a silent disco. I've never been to a silent disco. But, I imagine that this is what it looks like but without the dancing. Right? So, hi everyone. It's really weird not to hear myself on the screen. Anyway, I guess I'll have to get used

to it now. My name is Berry Longstock. I'm the CTO of Tila. And I'm here to tell you today about how we're leveraging AI to increase the quality or increase our confidence in the quality of our tests. And before we do that, I have a confession to make. My confession is that I think testing is incredibly hard to do well. I'm curious to see what the audience

thinks about this statement. Do you think testing is hard just like me? Can I see some hands? Wow, that's everyone pretty much. Everyone who has their headphones on anyway. So, I guess that's why you're here. That's great. I want to share with you what I experienced, what I learned or what we learned while implementing Quinton the things that you can do with your code base to see

if you can elevate the quality of your tests just like we did. The reason why we think testing is hard is because even if you have a lot of tests, it doesn't mean that everything works as expected. As we've learned many times in in the past. Now, before I dive into how we're leveraging AI and doing the test, I want to share with you what we're testing.

So, what we're testing at Tila is Nexa, which is our cloud platform. Um basically, it's an application platform that is meant for development teams who want to focus on delivering software over managing their complexity of cloud providers. Um basically, it's a competitor to AWS Fargate or Google Cloud Run or Render or Railway. Um so uh that that's that's basically what what we built. Um this is not

a sales pitch, by the way, so don't worry. I won't get into the details. If you want the details, though, we have a booth. Uh so, please come by and say hi, and we can explain you more about that. Uh but, it is interesting context because uh this is the thing that we're testing. Um so, that's why I need to tell you this. Um now, the thing

that we um do when we're testing is that we have this pyramid. This is the testing pyramid. Um you're required to show the testing pyramid whenever you do a talk about testing, I've been told. So, that's why it's here. I think most of you already know this pyramid. Um and you probably already know the levels. I won't embarrass you by asking who doesn't, and you know, raising

hands. it'll be our little secret. It's fine. I'm going to go over the layers really quickly. Uh so, the bottom layer is unit Uh unit tests small, fast. That's basically the the summary. Then, you get integration tests, which test um more units in combination. And then, at the end, you get end-to-end tests. Now, the unit tests are is is wider. It's the It's the bottom, and that

is because conventional wisdom tells you that you need to have a lot of unit tests because they are fast, right? So, they have a very short feedback cycle. Integration tests are slightly slower, so therefore, you need a fewer. And then the narrowest would be end-to-end tests because those are slow. And that's absolutely true. However, we did find in in the last decades, I guess, that this is

about efficiency or how fast things are running. It's not about how valuable the tests are. Cuz if you're looking at value in the sense of gaining confidence on whether or not your application still works, the pyramid should be inverted because end-to-end tests give you a lot of confidence that the software is still working, whereas a unit test can pass while the application is broken. Um so, that's

that's an interesting idea. We spend a lot of time building end-to-end tests, as my colleagues can confirm. Yeah, shakes his head. Nice. This is the thing that we're testing. This is our application platform. In the center of it, you see a GraphQL API. That GraphQL API is the entry point to our infrastructure. the truth is more convoluted, right? So, the the GraphQL API doesn't talk directly to

the infrastructure. There's a queue in between and workflow engines and it's it's much more complicated than this, but that's beyond the scope of this talk, so I'll leave that. Um the interesting thing is that we have three clients that talk to the GraphQL engine, which is the Terraform provider that we have. We have a front end and we have a CLI. Um the subject under test is

the front end in in this case. So, the the thing that is interesting is we're building this software with what we would call modern principles, I guess. Which is ironic because it's not very modern. These things were already invented in the '90s. We do extreme programming. Which means that we do pair programming. So, we never tackle one feature alone. We do trunk-based development. So, we commit directly

to trunk. We don't use pull requests very often. Basically, we only use that as a last resort. For example, if there's no one to pair with because they're all on holidays. It happens. And we do continuous deployment. Continuous deployment means that software is deployed automatically on every commit unless the tests fail. I let that sink in. Unless the tests fail, the software is automatically to production. That

in combination with my previous admission that testing is really, really is a bit of a a shocker, really. Yeah, we're doing this because we enjoy being able to roll out features quickly and to solve bugs even quicker. It is a radical change to how we used to do software development, to be honest. Because I remember when I was doing software development 10 years ago we would do

commits to feature branches and then that feature branch would be merged into a release branch. That release branch was released after, I don't know, 2 weeks, 3 weeks, whatever. And then it went to the acceptance environment. And then from the acceptance environment, testers would take a lot of time testing the entire application manually. And then eventually, they would give a green flag or more realistically, a red

flag and they would report some defects that you had to fix before you could actually roll it to production. we don't do that at Atlassian. The reason why we don't do that at Atlassian is because we can. Uh the good thing is we're a product organization, so we build our own products, of which Next is Um so that means that we don't have a customer that needs

to do acceptance. Uh and that's kind of a privileged situation to be in. I realize that, but it's a really cool situation to be in. we also don't have any testers. I firmly believe that um developers uh are more than equipped enough to test their own work. Um uh if if if they're uh disciplined enough. Uh not not that I discipline them. If if they are have

the self-discipline, let's put it that way. Yeah, I'm not, you know, whipping, you know, so that's not what we do. and yet 2 months ago, I think, or maybe 3 months we had a bug report from one of our customers, uh a big customer. And the uh was basically they couldn't deploy any containers anymore, kind of the core business that we're in. So uh we were confused

because uh the monitoring screen was green. The tests were all green. Everything was green. So what happened? And uh we tried to figure out what happened, and uh we tried to reproduce it, and we couldn't. Um because we were just able to uh deploy a container. So we dove into the issue a little bit further, and we found that the problem was actually only there uh um

if a specific feature flag was turned off. We use feature flags extensively because we deploy every day, uh multiple times a day, and uh that means that we don't want to uh customers to see everything that we're still building. We only want them to see the finished product. So, we hide stuff behind the feature flag. As it turns out, there was a bit of functionality that we

had behind the feature flag, but the uh validation of a specific field was outside of the feature flag. And if you have a uh bit of validation that validates a field that the user can't even see, well, bugs. That's basically what happened. Um that that was a uh a learning for us because if we missed that, then what else did we miss? So, um we went into

into a room and uh we set out to figure out what our coverage was. Now, where unit tests are very easy to track uh coverage with because you can just basically uh trace the the code coverage, end-to-end tests don't have a system like that. what we did is manually uh figure out all the functionality that we have in our application, um and all of the variants in

uh those features. When I say variants, what I mean is parameters. So, uh for example, the feature uh deploy a container um has many variants because you can deploy a container with a health check or without the health check, with an ingress or without an ingress, multiple ingresses, um external connections, volumes, uh there's there's a lot of things that you can configure. And all of those are

variants. And um uh we listed all of the variants and all of the features that we have, and we um basically put that together with the tests that we have, and what we saw wasn't very nice. Because we missed some pretty critical uh variants that were untested. Uh Uh, the problem with this is that it's very, very hard to create this overview. We spent, well, few hours

um, to to build this in this this overview and to figure out where the coverage gap really was. Um, nonetheless, I mean, it was a it was a fruitful exercise. Uh, we learned a um, and we got more disciplined uh, because we understood that, you know, this is a problem. We need to do better. We need to test better and and that's uh, that's great. But, [snorts]

we wouldn't be Duda if we didn't figure out that we want to automate this because we automate everything. Uh, this is why we can handle 3,000 customers with 30 people. Um, and uh, so this is uh, Right. So, um, this is actually before we started automating it, we uh, I asked around to the development team, what do you think the confidence is in the test that we

have? Um, it was pretty high. It was 75%. Um, people did think that we missed a few things, uh, but they didn't actually think that we missed a lot of things. Um, after we did the exercise, that number was significantly lower. Um, we did know that we were missing this. We just didn't know which one we which ones we were missing. Okay. So, uh, enter Quinton. Uh,

this is the result of us trying to automate this uh, of bit of work that we that we did, this exercise that we did. And Quinton is an AI agent. An AI engine that understands uh, in in parentheses because, you know, understands is a is a difficult term in the in the language model. Um, if there's a coverage gap between the features and the tests. Um, in

case you guys don't know what an AI agent is, um, it's it's uh, it's a popular term nowadays. It's a loop. It's a loop that talks to an LLM until a specific condition has been satisfied. So, in this case, until the coverage report has been generated. Um and it won't stop talking to the LLM until it reaches that state. It's nothing more, it's nothing less. Don't get

excited. Or or do get excited if that's uh something that interests you. Uh but it's basically a while loop um until a specific condition has been met. All right. So, um we started building this and um it wasn't easy. Um we figured, you know, we'll just use an LLM because everyone does, right? Everyone uses AI nowadays, so this should be easy. Um well, it wasn't. We ran

into quite a few challenges. Um and that was a bit of a reality check for me. Uh I assumed it was pretty easy to do. It wasn't. Um so, these are five challenges we had to overcome um in order to to get something productive out of it. The first is, what is a feature? Um this is an interesting question uh because we talk about features every single

damn day. Um we talk about features, we talk about uh values, um but we've never defined what a feature really is. And and looking at the application, um it wasn't trivial to figure out what a feature is. But you have to tell the LLM very specifically what you want it to analyze. Um so, uh eventually we set out um and and we said uh uh a feature

is a component that changes the remote state. So, basically, every component that does an API call. Which works for us in this particular uh use case because it's a it's a dashboard which does GraphQL API calls, and that's it. Um so, this was a definition that was kind of low-hanging fruit. Um, and and it works for our uh for our use case, but it wouldn't probably work

for your use case if you're building an API or whatever. Um, but it does work if you if you're creating a a front end on top of an API. So, the second challenge is the one that you probably already saw coming, the context window problem. Our code base is relatively new. Um, I think a few years old. Um, it's Vue.js based. Um, it's well written. I hear,

anyway. Um, and uh um it's about 200,000 lines of code, which is not even close to the biggest uh code base that I've been working with. Um, but it is uh big enough to overflow the memory of the GPU. Um, if you if you're sending 200,000 lines, that's a multitude of that in tokens. And um, you know, tokens are expensive. Luckily, we have our own GPUs, so

that's not actually a problem, but we did get problems where the uh memory was exhausted, basically. >> [gasps] >> Um, so we started thinking about how can we fix this? How can we solve that problem? Um, and we came up with uh uh a software which is called Bitlane. Bitlane is an MCP, a model model context protocol thing, um, which is really really nice. Um, it builds

an AST, an abstract syntax tree of your application, whether it's JavaScript or C or Python or whatever. Um, it can generate Oh, sorry. I again That was annoying for you, I guess. Um, it can generate an entire abstract syntax tree, um, which, um, uh, documents the functions that are in the application and what functions call other functions. So, it can, uh, basically trace an whole call stack.

So, what we started doing is we started looking at all of the, uh, usages of the Apollo client, which is the thing that we're using to talk to the GraphQL API, and then work our way back to the component that is doing that call. Because only those files are actually relevant for the definition of feature that we used. Um, and that really works well because, uh, this

means that we're not sending all of our code to the, to the LLM. We're telling the LLM, "This is where you can find the code. This is how you can find the code. Start with Apollo um, uh, start with a mutate call and the query call, and then work your way back to the component that is using it. Parse that component and find all of the Uh,

and and the and the values in the, the variants. and and that really worked out well because we didn't spend, um, I'm not sure about the number, but I think it was about 20,000 tokens on a run now, which is significantly lower than it than it was in the beginning. So, Pitlane, uh, awesome tool. Um, also, if you're just using LLMs for for local development, it's already

very awesome because it saves you a lot of, uh, data traffic and a lot of tokens, uh, if you're, uh, just developing software. the third challenge that we ran into is that LLMs by design are non-deterministic. So, if you ask the same question twice, you get different results. Um, we want to use this in our pipelines, and pipelines that fail sometimes and fill don't fill other feels

like not such a good idea. Um so we transitioned to using a specific testing format, well-defined format for both features and test analysis um that we could reuse um uh between every single run. And these files look like this. Um I don't know if you can see it. I hope so. Uh it's YAML uh because I found that YAML is a language that an LLM can parse

pretty easily and I can still read uh which is not the case for for for example JSON. Uh maybe it's just me, but YAML works for me. Um if it doesn't for you, well that's your loss, I guess. Um so what this does is uh it can it it describes the feature that we have. In this case, create the container. It also describes what inputs uh there

are for create container use case. So it parses the the form that we have in our in our web page to check what variants there are. So in this case, it sees that there's a name, there's an external connection, an external connection is influenced by a And and that's what the LLM is outputting now. Um and uh it does so uh quite well. Then when it comes

to the test coverage, you get YAML files that looks look a little something like this, a lot longer admittedly. Um and basically it reports this test uh file um uh covers these features with these variants and these feature flags. And that's really helpful because based on those two, you can generate a coverage gap. And then you know whether or not there's things that you left untested. So

you know that there's values that you haven't tested when the feature flag is off. So, that basically would prevent us from ever having that situation that we had 2 3 months ago. Challenge number four is that I had this luminous idea. If if we have all the all the parameters or all the variants and we have all the features, I can calculate all possibilities. And I did.

And the number was quite staggering. >> [laughter] >> Um and we had to figure out which of these do we want to test and which of these do we not want to test? Because if you have only 15 form fields just boolean inputs, the number of of options is I think more than 16,000. Um and that's based on boolean options and we have a lot of non-boolean

options where there's a lot more options than just true or false. Um so, if you combine that with equivalency classes, which is a well-known concept in testing, that would mean that your number of tests basically explode. And you need to start figuring out what do you want to test and what don't you want to test? What we came down to is that we started filtering by risk.

And our risk is that we didn't catch the feature flag So, the thing that we're testing now or the thing thing that we're looking for coverage gap is um if if we didn't test a specific feature or variant against a specific state of our feature flag. Um and that's the thing that is being reported on. Because that is the highest risk. challenge number five is this thing

is Um Quinton is slow, the LLM is slow. Might be something to do with the fact that I'm using our own GPU. Um so we at we have a lot of servers. Some of these servers have GPUs. We're using a local model on that GPU. So we're not using Cloud or we're not using whatever. Um we're using our own local model which might cause a a little

a little less velocity than you want. Um but it's also repeating the same work over and over and over again. Because every time we work on a feature, we work on a feature for a few days or a few weeks. And uh we don't change the other features usually. So if a few features are unchanged and the test cases are unchanged, then it it doesn't make sense

to re-analyze what we're what we're doing uh because the the previous state is still valid. >> [snorts] >> And uh to prevent this, uh I I basically um went with the very obvious solution, namely we do uh caching uh with checksums. So we we have a checksum of test files, we have a checksum of feature files, and we can check whether or not anything changed. If something

changed, it will re-analyze the feature, it will re-analyze the tests, and then uh generate the new coverage report based on those. Um the result of all of this is that um uh uh I'll admit first of all that we're not using this in CI/CD pipelines yet. Um so it's manual runs. But still, the reports that come out actually give us a lot of information about what we're

missing. And it really gives us a lot more information about how um uh things can go wrong. Which also means that our confidence as a team has grown significantly. So we went from that 75% before Quinton to 90 plus percent right Um which is which is very good because I I can trust that if we onboard new customers, they won't run into to issues um and and

they'll have a have a good experience. Uh even though we uh deliver um if we deploy multiple times a day. Oh, sorry. That was the wrong side. Yep. Um looking back, what we do differently um is first of all, I would start with a smaller um piece of the code base. So, I I actually started with the entire code base trying to analyze all of the features

and all of the variants and all of the tests which um if you're working with an LLM is waiting a lot. Um which was not very efficient. Um I think we spent a few weeks on on building Quinton and most of it was just waiting on the LLM to make up its mind. Um the second thing is to treat Quinton as a tool rather than an oracle.

I had high hopes uh for this for this tool to be amazing. I think it is, uh but it took took quite a bit of time. Um there were a lot of false positives. There was a lot of times where we had to adapt the prompt uh because it was just missing things. Um and every time you miss something, you get a you get a false positive.

A false positives in this case are very very um annoying because they break your deployment pipeline. Um in theory, if you would use it in Um so, that's that's what we would be doing differently, but uh the value is definitely there. Ultimately, I think AI doesn't make testing a non-issue, but it does make it easier. It does make it easier to get a feeling about the coverage

of your end-to-end tests. And that's the thing that we were going for. So, um we're very happy with the if you're inspired and and you think you want to you want to use this, uh you can reach out to me. I'll I'll happily share whatever prompts we have. Um not sure how how well they will match your specific use case. Um but um I'll I'll happily share

whatever information we have with you. Um but still start small. Um measure the comfort confidence before you you run it and after you run it. Um and don't just look at coverage because that is a uh false sense of security. that's what I wanted to tell you about Quintin. That was my my session. Thank you for joining.