Test Data Management: The Cornerstone of Reliable Automation - by Nikolay Avramov

In this talk, Nikolai Ramov discusses the crucial topic of test data management for automation engineers. He highlights the common challenges faced, such as test flakiness and resource wastage, emphasizing that without proper test data, automation efforts can become ineffective. The session outlines various strategies for managing test data, starting from basic approaches to more sophisticated, model-driven solutions. Nikolai shares insights on structuring entities and handling dependencies within tests, as well as how to clean up test data efficiently. He also introduces a solution that leverages APIs and relational data management, making automation scalable and manageable in complex projects. The talk encourages the audience to adopt a proactive approach in their test data strategies to enhance automation results.

Challenge accepted. [music] Challenge accepted. Challenge accepted. P2 [music] is always challenge accepted. Our next topic is one of those things every automation engineer complains about, but not everyone talks about enough. >> Test data. Because no matter how shiny your framework is, without good data, your automation is just a very expensive clicker. >> The next talk is all about making test data reliable, secure, and CI/CD CD friendly.

And our speaker knows exactly how to make that happen. He started his career at Talleric Academy, grew to the QA ranks and become a principal QA engineer, trainer, and consultant. He's now a test automation consultant at Automate the Planet, helping teams replace boring manual task with smart repeatable automation. >> He shared his knowledge on stages like Testcon Europe, QA Global Summit, Selenium Conf, and of course QA

Challenge accepted. He is passionate about teaching others and proud of the many automation engineers he's mentored along the way. >> He is also one of our proud nominees of QA of the year award. Please welcome NIKOLAI ARAMOGEECTED. Is this working? >> Can I get the mark to work? Okay, that's great. Thanks for the intro. You got my name right this year. Thank you. Let me do the

quick setup. We're extending. [laughter] Good one. Yep. Show presenter view. Okay, perfect. All right, thanks for having me. As you all heard, my name is Nicolola Ramov and I'm a technical consultant and director in in uh Automated Planet. I'm super happy to present this in front of you today because this was a talk I was delaying quite a bit because test data is something pretty big and

uh I really wanted to get enough context and enough experience with different projects so I can really wrap it up and show it in front of all of you. It's not a talk just about different strategies, pros and cons, and evolution, but also I'm trying to I'm going to try to give you some useful tips and actual product that you can use in your projects that I

spend some time to build. So hopefully it's going to be pretty helpful for you. This talk, I hope, will serve you as a as a road map. So in your daily work, you can see where you're at in the evolution of the test data management strategies. and how to get to the next step. Of course, if that is needed, if you don't have issues with it, that's

perfectly fine. And uh this is the kind of speaking of different types of talks, this is the one with the hands. So, please use your hands if you find yourself in any of these strategies and help everyone understand how many people are using them or not. Also this talk as you see the picture is inspired by um my son's favorite cartoon asterisk and obelix. I'm really curious

to see with hands how many of you have heard of it or have seen it. Okay, that's good. That's a good number. Okay, so you probably know this the story. Uh the action happens in the small village called Galia which refuses to to get to be defeated by the Roman Empire. It kind of reminds me of our national volleyball team who are as we heard now fighting

for the gold medals in the world championship. So with this metaphor I will move forward with the universal challenges of test automation. getting a step back. When we start in a project, we al always look for the people on the shield that are basically guiding the team into a different direction. And when they when we ask them, what are the problems you face? What are the issues

your team have related to test automation? You we often get some of these test flakiness, execution speed is too big, we don't have any autonomy of the tests, lots of false positives, meaning lots of maintenance, and the worst of all false negatives, meaning that we have lots of automation and we still get issues along Looking at the business pers perspective of this that means we're they're losing

lots of resources on maintenance where they're getting slow feedback from their tests and they're wasting lots of time on analyzing their runs. The false negatives of course are meaning that critical issues and defects are reported by clients. So that's pretty bad thing. But when we look at all these and we start sniffing around uh in the project see what was built we find couple issues that some

of them are easy to fix others are not fragile locators that's something everyone facing static weights all over the place meaning that we cannot uh react to slower environment we get lots of end to end BDD tests that means everyone is going with um behavd driven tests written in Girkin syntax and they're trying to create their data and everything using the UI reporting and analysis is also

a big chunk but it has ways to to improve it quite easily but in all of these you can find traces of the missing test data management and overall strategy about how to build it because all of these can be achieved and can be improved with a better test management strategy. So that's why we find that test test data management is not a problem. It's the biggest

problem that we see in most of our projects. So let's move to the evolution of test data strategies. You're going to see different approaches starting from the lower level to the higher end. And the bad thing about not good enough test data strategies is that they work for some time. You get some rocks over there, but at scale they start to fail and you have to rebuild

the whole thing. So the later you are in this process, the more uh painful it is to rebuild it. So starting up with the first strategy, I call it the nave approach. That's why I have Obelix here who's uh always willing to help. He's pretty powerful but he's not on the long-term thinking. So what is this method? It's basically using the existing data from the database. We

just go there use whatever it is write our automation scripts they they pass locally do one run and we p and we commit the code. The benefits quick implementation just go there and do whatever is needed and your job is done you get the story points it's all fine so where the problems comes there are plenty of them I just listed three of them but test conflicts

start to appear between different tests we get environment issues we get unpredictable test outcomes so all this flakiness all these timeouts everything's come together so how many of you have tried this at least when you're starting your journey can you raise hand yep when I when I teach at the academy we always start no matter of the age of the participant we always start automating stuff and

this is the first thing everyone is doing let's uh let's agree with that and it's not perfectly fine so that's the first step moving forward we we decide that instead of relying on existing data we can produce the data ourselves So we can arrange our army in the test environment. We can set everything up that we need and when our test start we can get the data

and just use it. That's perfectly fine. And this strategy works. It could work. It could work for many years and we have seen that in in some So where's the issue with that? The issue comes when you want to run on parallel when you want some independence from the test. So you need some flexibility in in the test. You start to get some data corruption which basically

erases all your data and you have to recreate it or even if you decide to store a backup and then restore it every night before the test come at some point the knowledge about how this test data was built is starting to disappear. It could be a colleague that is on leave that was creating that entity you need. It could be someone that's no longer in the

company that produce the data. So if something happens with that data, what can you do with it? How to produce it? So that breaks that breaks hardly and you have to go back to the drawing board and you have to rebuild it. What is the next step from this strategy? Actually, let's check how many of you have tried this. Okay, it's not a huge percentage, meaning that

all of you are doing great job. Getting to the strategy number three, I've got Asterics here. He's quick and easy and he's very fragile. So, this is the next approach where you decide to scratch all the data that's in there and start producing it yourself. So you find the API, the database that you need to use to produce your data. You generate it and then every test

can use your data uh that is independent that does not um exist because of another test and that allows us to do many more things with our tests. At some point you start seeing that your tests are getting more and more useful. People start asking what are the results? devs want to uh want to ask you to start your test on their environment just because you can

using that approach and that's that's great. That's the point you see you're doing your job well. The challenge when when the project starts to get bigger and we have more and more complex entity relations these projects they are still code and you have to manage it some way. So if you create a data with one API, you start adding methods for everything you do. So this whole

project becomes a bit messy and hard to maintain. So with complexity, the issue comes and you have to find a better solution, better architecture and strategy how to achieve it. And this is the point when you have to turn your big powerful beast on the left to the more structured and agile thing on the right. That's why uh we implemented a solution we call a modeldriven solution.

The core architecture of it is basically getting models to represent the data. So we can combine and collect everything we have in the on our project then depending on the scenario we need to achieve. Having this generic approach and uh setting and coding the entity relationships between models enables us to scale and basically incorporate this solution into any problem we have with this data. until we get

to the next point when you need it in a large scale. In order to represent it in a real world, I'm using this uh trail board. Uh probably most of you have heard of Trello. This is a very simple task management system. It consists of couple of items. We have the board, we have the and inside the list we have these cards. So this is a typical

representation of dependencies. The cards relate on list and list relates on boards. Pretty simple. So how do we build this foundation? We start with the models. As I said, define the relationships. See what are the dependency chains and how they relate to each other. Can one live without the another. Later on we create the models themselves as classes. Could be Java, could be C# or anything else.

And then we introduce two entities, two classes that help us to achieve all of that. The first one is the factory. Factory is uh has single purpose and it's meant to just build a default thing, a default model that can be sent directly to the database. That's everything it does. The next one is the repository which which knows how to create, update or delete data. This is

all generic. So you can use it for any models. These two you can imagine as as if you're you're break making a bread. So when you when you make a bread you mix the contents you leave it aside and you get some dough. So after you have it you can make any shape of it. You can make it any type of bread but then you have to

bake it. So we are separating these two processes in order to achieve maximum flexibility in u in and to achieve the test scenario that we need. So how do we use all of that? Let's see one of the simplest test scenarios here. We just need existing card in order to update it. But that card actually depends on the list and the list depends on the board. So

how do you get to the card? actually you have to create both. So what we are doing is we are building default models across the whole dependency chain. You adjust the data that you need no matter if it's on the list on the board or the cart and then you create the entity in the database all together. Then we navigate to the page and you get a

data that is fresh that's adjusted to your scenario and you don't have to worry about it. uh for the test scenario of course we can take any of these and use them. So after we're done with the test usually there is a mess that needs to be clean. Uh if we use the approach with the with the uh golden set of course we can wipe up the

database but here this is not required. we know what data we created so we can keep track of it and after the test finishes we can go ahead and delete entity by entity everything using the repository because it knows how to do Cool. So having all of this it looks like we have some flexible approach and we can start shaping it according to our scenario. Now we

can try to face the big battle. So you might wonder what's the secret source behind this. I prepared a really short demo. Hopefully it's going to be presented here as well. Let me see if that works. I'm going to duplicate. Okay, I got it. So a brief intro. Uh this is a Java project obviously built on top of Bellatrix but I have spent some time to make

it possible to extract it because we have it as a module and this can be built into any project. So looking at the code uh looking at the code uh we have a very similar approach for the different entities we had uh back on the slides the board which is the model we use the repository and then the factory using some hierarchy that we built in into

this project. We're keeping all this super simple and keeping their duties really limited to what they're intended to do. The model has just a couple of um methods that we need for the whole infrastructure to work and inheriting a base entity. The board repository, the one that knows how to create, update, delete, has been built here. All the logic is inside of the base classes and you

just do the the stuff you override the stuff that is specific for that endpoint. In this case, you cannot delete a board. You have to set it to closed and that's The refactory itself is it has just one method that's required. It's supposed to build a default model that we can send directly to the database. So that's all we have in um when we create a card

we need the rest of the models created. So we have to incorporate these dependencies between models. How do we do that? We just create one one field that is of type list or whatever entity is the dependency and we denote that this is a dependency. So we cannot create this model without it. This way the module knows that it has to create the other one first and

this is how the whole thing works together. Looking at the test I'm going to show you how this is benefiting your tests. So when you start one test, you're just calling the factory using the card factory. I want the default card. At this point, this is not created in the database. So we can adjust it. Then you you call the method create with dependencies. It's being created

through the database with the default data. So I didn't didn't say that I need a board, but it knows because the models is are built that way. After that when I assert I can get from the cart to the list to the board and then get its ID and verify it. So that flexibility is super useful when you have complex scenarios and the whole idea actually uh

with these dependencies started from this how do we clean up the data. So when you have these dependencies as I said you can call one method and it goes all the way through the chain and it's cleaning up everything it produced. Cool. So when you want to modify stuff, you can just produce your data, get it into from the from the factory, set your uh set your

properties depending on the scenario and create it. And as you can see, this is not a web test. This is more closer to the API test, but it's not. And I build it like that because this one can be reused in either API test, desktop test, mobile tests or or web tests. With this module, you can power all these different testing um types. So we have also

example for web test. I'm going to just briefly show you getting to the boards. So it's basically the same flow. The difference is that instead of hard coding some stuff all over, you have to produce your data and just use it in your web test after you navigate to the So this is uh this is a public repo. I created it for you so you can use

it and take a look after the lecture. Let's move back. Extend. Oh god. Again. Getting there. Cool. So you might think this is a bit of a overkill for such a small project. It's three entities. I can create them right away. It's not a complexity, but this is a picture that I really like. This is a real picture from our one of our projects. And everything you

see, don't try to read it. It's blurred. Uh everything you see, every single line is representing an entity that is connected to another entity in the system. So we build this on top of our models that we built in the project and we use an external tool to draw this graph. So when you have that complexity what else can you use any of the on any of

the other approaches simply won't work. So getting to this what do we get after we implement such approach it's even better if you have it from the beginning but if you implement it you get great configuration management it's environment agnostic data isolation all of this has been done you can optimize your API calls so you don't create it every time or reuse it or whatever you want

basically the maintenance and debug debugging has been significantly re has been significantly ly reduced. We're talking about 15,000 regression tests, but not manually automated. They run from 24 hours and we have to give feedback from these results basically every couple of days. So, uh in this scale anything else is pretty much impossible. It has integration with the Belrix framework, but I believe it can be built with

any other framework as long as it has proper architecture in place. metrics, stabil, stability, execution speed, all these you have control of. And also this makes sense to developers because they use similar patterns and approaches in their production code. Usually you can get the modeldriven solution from this GitHub repo. I'll keep the slides just for example. So for a second I try to produce some documentation and

uh give you getting started guide. So um if you have questions always you can ping me for this. And that's pretty cool I think but it sounds a bit 2024 to to create your models to create your uh repositories classes and all of that stuff by hand. After all, it's the year FAI. So, the the example I built um I use cursor and uh ruot to help

me with the the uh with this example that I showed. What did I do? I didn't ask cursor, hey, let's create some test data management solution that works for everyone across the globe. I had everything in my head. I had the module. I had the documentation. So it was about providing the right context to the model so it can build the entities the way I want it

to be created. The relationship detection the customization it most of it came from the model but I gave it the open API document specification for that endpoint and it figure it out from that context. Without it would be pretty hard to get. for anyone who wants to to try it out. I produced this root prompt that you can you can use. So it will it will get

you the whole structure of the project according to the to the uh to the use case you have for any API at least on theory. So now we have something even better. We can start kicking butt with this approach and uh basically get your automation projects out of this world. In summary, not having test data strategy is not good strategy. Different different strategies hide issues along the

road. So you have to plan and you have to be informed about what's ahead and where you're staying in the m in the moment. It's perfectly fine to use a simple approach if you don't have that scale. But if you plan to do it's better to have it in mind. implementing flexibility is paying off big time after after um couple of years and after some time goes

and AI of course is here is here to help and we should use it for the right things so we can generate the boilerplate code and focus on the test scenarios we have. Thank you very much. [applause] Thank you, Nikolai. Thank you very much. Uh, do you have time for questions now? >> Sure. >> Uh, quick two questions. Yes. Uh, uh, Mike here for our our friend.

>> Good. Because I have two questions. Um, no, actually joking. I I wonder how you deal with um injecting data that you cannot simply generate uh and and and put into database just just to give you example. So if we have data that you need to encode and you don't have access to keys to encode it because of like security reasons and you don't want to expose

it or another example you have a service internal or or external one that must process data and then only then inject it into database. So you cannot just directly put it into database. How do you deal with this? >> Yeah. Yeah, that's that's a great question. I actually had a slide about this uh but wanted to keep the story line um straight. So basically you have cases

when you cannot produce your data and that's uh the usual case because you work with lots of third parties and this is the place where mock data comes in. So you want to create some mocks that simulate an API or external service that is producing the data. uh I can I can share the risk that I have seen uh using this approach and that data the data

structure and responses change with time. So basically if you keep it static keep your mock you can miss some issues just because the data has changed and basically you're not uh reacting on it. So um does that answer your question? >> Okay. Yeah, mocking is a valid approach in test data management. >> So thank you very much. Is there any other question for now? There's one raised

hand. Last question for here. The other question will be asked in the speakers corner. >> Hi. Hello. My name is Dante and I have a question. if I've understood correctly from your presentation is that you encourage to use the data regeneration and then cleanup over um storing the same and the same data, right? >> Mhm. So do you have a solution for then regenerating data is uh

taking too much time and slows down the test >> but also reusing the already existing data is um not sufficient because it can break other tests as well. >> Yeah. Yeah. That's also good question. Um usually when when you deal with a big system you have to draw a line between data that should stay and data that we need to produce. These are the so-called nomenclatures or

enumerations. This is the these are configurations on the system that should stay. So if you produce everything on the system from scratch every time it's obviously get uh really long. So what we usually do is we draw a line where do we stop creating and we rely on existing data that is needed for the system to function and what data we produce ourselves. So uh this is

a dynamic process. It's different for every project but yes it's needed because if you create everything from scratch every time it's going to take lots of time. >> Thank you very much. You receive t-shirt. You receive t-shirt. You can ask your organized questions in the speakers corner where when where where Nikolai will be waiting for you during the next break which is your >> my favorite break.

This is a lunch >> Let's hand off Nikolai with a big round of applause. Nikolai Amo.