PandorAI's box - What have we opened!?! - by René van Veldhuijzen

This talk explores the multifaceted implications of artificial intelligence (AI) within the quality assurance (QA) domain. The speaker introduces the concept of Pandora's box as a metaphor for the complexities and unpredictabilities associated with AI's integration into testing. He emphasizes the necessity for testers to adapt and evolve their skills in response to AI's capabilities, including its potential to generate test cases and analyze code. Relying on AI, he argues, presents risks of over-reliance and a decline in exploratory testing skills, making it essential for QA professionals to remain critical thinkers and maintain human intuition. The speaker shares the ACME method, which consists of anticipating, controlling, managing, and expanding the role of AI in testing to enhance efficiency while maintaining creativity and effectiveness in the QA process.

Challenge accepted. [music] Challenge accepted. Challenge accepted. P2 [music] is always challenge accepted. Welcome back everyone. Coffee break is over. Unless you're an AI, in which case you just didn't need uh such a break. Ladies and gentlemen, AI is everywhere. It is used in hospitals, in your phones, in your kids' homeworks, and yes, even in scripts to write jokes that aren't so funny always. >> It's debated in

parliaments, whispered about in classroom, and shouted ac across boardrooms. >> It is a friend, but also a foe. It's a blessing but also a curse. >> It is a tool but also a weapon. >> It's a creator but also a destroyer. >> It is a dream but also a nightmare. >> It's a remedy but also a poison. >> It is the beginning but also the end. >>

Okay. [clears throat] Okay. Okay. Let's invite our next speaker who knows better than us what AI is. >> He is well known across the testing world. He's been on the stages of Euroar, Testcon, Hustf Tapost and many more. >> He's also the creator of fundriven testing. He's an organizer, a writer, and a true test community architect, spreading knowledge and good vibes all the way from the Netherlands.

Please give a warm and curious welcome to the end to the one and only >> Rene So, wow. I like the introduction style. That's great. you're not in Kansas anymore. You're on Pandora. Ladies and gentlemen, respect that fact in every second of every day. Sorry, that was of course a famous quote from Colonel Mice Quartridge from Avatar from 2009. So, if your your AI assistant told you

my talk was going to be about Avatar, you will be disappointed. Um, no. But welcome to the age of Pandora AI. Um I'm going to tell you a story about hype which AI is uh hope and the bugs that we brought with us because we made this well maybe not the people here in the room but we as a humanity. I'm Reneelausen. I'm a strategic consultant at

Squareist and I call myself the test positivo because are we in the most great job in IT or what? Yeah, let's be proud. I'm going to talk about Pandora's box because AI is out of the box. Uh we didn't just open a box. We triggered a cosmic popup window filled with brilliance, chaos, which was mentioned by Vippin, and everyone's favorite cat memes. Um, AI didn't sneak in

quietly. It arrived like a party crasher. Confident, charming, and slightly terrifying. let's rewind. Long before AI, there was Pandora. And now we the testers, the QA specialists are left holding the lid wondering is this a feature, a bug or a philosophical crisis in Python. no. Pandora opened the box, not out of malice, but out of curiosity, and out came chaos, toil, despair. basically your first beta release.

Uh but at the bottom of the box, there is hope. Sounds familiar because we open environments, pipelines, and backlogs, and chaos just spills out, right? But somewhere in those logs, there's always a glimmer of insight. And I want to share that insight with you and talk about Pandora's box. Now let's do a update, a Greek update because it was Greek mythology. And actually Pandora was the first

woman created by the god Heestos, the god for smitting and fire. Uh and he she was sent by Zeus, the supreme god, uh to the mortals as punishment. uh because um the human or the the the mortals that were on Earth uh brought disaster after Prometheus, the first man uh and one of the titans stole fire from the heavens. And um we'll get back to Prometheus in

a while, but she was bellayed with gifts from the gods. Athena gave her intelligence, talent, manners, and some colorful clothing. Aphrodity gave her grace and divine beauty and a host of other gods gave gold and put flowers in her hair and stuff like that. And finally Hermes and sorry ladies about this she gave her speech. That's okay. Shameless thoughts um and a deceitful nature resulting in curiosity.

Um okay. they gave that well they offered um uh Pandora to Prometheus which is the guy in the middle the big one but he was like no I stole the fire and a gift from the gods never goes without consequences so not pass. Um but then his younger brother, Epimetheus, what he was not so bright and he didn't listen to his big brother. So he said, "Yeah,

okay. Bring her on." And >> [snorts] >> um the gift from Zeus that Pandora got contained all the misfortunes that I just told you about. And it came in the form of a jar. A jar? Wait. Uh I was talking about Pandora's box, wasn't I? But I have to rename it to Pandora's jar. [snorts] Um so um mythology is wrong just like AI is hallucinating sometimes. So

we have been hallucinating for many millennium. Um raise your hand if you have ever opened your test environment and something that you didn't expect. So why are we so shocked that AI does the same thing? Um, so what happens if this box I I'll keep calling it a box because jar doesn't feel right. Um, isn't mythological, but it's algorithmic. Um, and introducing AI, our new mischief maker

after decades of development, because AI has been around for at least 70 years. You should look it up. Um but after decades of development, tweaking and different forms, we've seen Kasparov lose to deep blue, stuff like that. Uh some AI won Jeopardy and um AI suddenly showed up. Well, for us humans, yeah, with Jet GPT and all the other LLMs, it's friendly. It says hi and says

nice things. Hey, great question. Ah, good for you. Go on. And um but it's in unpredictable and we like that as as test specialists as QAS and it is convinced it can fix everything with a motivational quote. It writes you haikus in your test logs. Um it suggests refactoring for your login flow into a choose your own adventure. Um and it insists that every test case needs

a dragon. Um is it helpful sometimes? Is it mischief often? Confusing always. It's like Clippy. Do you know this one? Yeah. It got a PhD and now it wants to refactor your test suite. Okay. Now, AI is everywhere these days. It's from your EDI to your CI pipeline. It's like glitter. Once it's out, it's everywhere. For those of you who have kids, you know what I'm talking

about. Um, and um, QA, we as the QA community are feeling the tremors. Our roles are shifting. Our skills are evolving and suddenly exploratory testing feels like jazz in a big techno dance. Um, but is it a feature or a bug? When AI suggests a test for a feature that doesn't exist yet, is it a visionary thing or is it just confused? LLMs and Genai are the

new players in our testing arena. They generate tests, suggest code, analyze logs. Cool. And occasionally they hallucinate. The ambiguity is real. Is this brilliance or just be beautifully formatted nonsense? That's what we are here to find out. So let's peek inside the box and see what what's actually crawling around because it's not just tools. It's a real mood thing because we have automation. It helps us but

we had this before. You know who of you have managed to work with robotic process automation? Are there any RPA specialists here in the crowd? It's a form of AI. You know our automation tools. It's a form of AI. We don't call it AI. But now the tool specialists, the companies like I don't know, let's call some mediocre company like Tricentis. They claim it works on AI.

Okay. Um, it helps us with acceleration. This is my good friend Wy Coyote. He will be later on because I have the Acme methods. And for those of you who know the Road Runner uh comics, Acme is a part of that. Um but he will be back faster than you think. Just like our Alexa acceleration thing, there's anxiety that came out of the box because it can

be overwhelming at sometimes. Your boss is asking, "You're on top of the AI game, right? People from your family who are not in IT are saying you are the AI specialist in our family. we turn to you just like when they turn to you when their printer is broken, you know. Yeah. Um and then there's ambiguity because we don't actually really know what's going on, you know.

Um we experience fear of replacement. Will it take over my job? Will it take over a job? Who thinks it will take over his or her job? Okay. Well, could be. Um, and we are losing some of our craftsmanship because when testing becomes autogenerated, where is the art? Where's the nuance? Where's the fun? So, we risk trading human intuition for If we trust the machine blindly, we

stop asking weird questions. And that's where the real bugs hide, right? And number two, testing isn't just execution, you know, it's exploration. It's feeling things and the courage to say, but what if? And speaking um of curiosity, you know, um let's talk about the awkward first date between QA and AI because AI wants to help us and we as QA just wants to build trust. And when

your test suite starts referring referencing to unicorns, sandwiches, and the login form that requires a spirit animal, you know it's time for a serious conversation at the dinner table. It's like dating someone who's finishing finishing your sentence. Actually, there are some LLMs that actually finish your sentence. So, um, okay. But when it finishes your sentence, there's always some kind of plot twist, right? I wrote 500 test

cases overnight. Great. But why are they all about medieval siege weapons? no. The the tension is there. So, we're on the date and we are experiencing real risks. There's over over reliance when we don't write the things ourselves. There's no second-guing your work. There's lack of transparency where it where is it all coming from? Is it okay to use? Is it legal? Because it's it's ripping off

things from the internet, the free internet where people have written many thesis, documents, studies. And there's erosion of our exploratory skills. No more exploration means no more adventure means no more fun. So anyone here ever reviewed an AI generated test and thought this is brilliant but also completely wrong. Yeah. Or worse, this is brilliant and I have no idea why. So what do we do? We can't

close the box. It's out there. It will be out there. But we can stop poking it blindly. It's time to swap panic for a plan and maybe a snack. Um, so we need a plan. Here my friend Wy is. And well, let's stop reacting and start strategizing. AI isn't going away, but neither is our critical thinking, our curiosity, and as I said, our ability to ask what

if, you know, think like a user. We need a plan that blends machine speed with human weirdness. A strategy that respects the chaos but doesn't surrender to it. And the first thing that comes to mind, are we going to be obsolete? There were a few hands. You know, my job is going to go away. And uh I put on my my tin foil hat for that. Testers

are not obsolete. People are going to need us in the workspace now [snorts] more than ever because we are evolving. You know, we are learning learning with AI, learning by AI, and learning by ourselves by just talking at conference like this. You know, it's good that I'm up here, but the real conference content comes from talking to each other, right? And we need to test smarter. So,

we have to evolve even faster. And we need structure because that's why we have ISTQB TAP stuff like that. Structure, right? and I said, well, let's have some snacks. So, I put in some snacks. Maybe you recognize some, but still. Okay. Um, Pandora gave us Chaos. Okay. But, oh, before I explain the plan, actually a helpful method. Funny thing. When I flew over here, the Bulgaria Air

magazine mentioned artificial intelligence. Meet maybe you know the guy Peter Stf and he's the chairman of well here the Sophia Tech Park and also a manager for something really great the largest scale supercomputer of Eastern Europe. So kudos for you right? It's somewhere here. We're probably not allowed to look at it, but okay. Uh, and uh, it's number eight in Europe. Um, if you're not counting the

Czech Republic at number seven and Slovenia number six as Eastern Europe. I don't know where I draw the line, but okay. Um, thinking people are needed. It's the next thing that I read about. This what this man said, you know, uh, we need thinking people. So I put on my thinking and started helping and he mentioned the five sins of AI refusing to obey because sometimes AI

just doesn't do what we are telling it. Um the second one it makes us dumb. No I think it makes us smart but we have to use it wisely. Sounds familiar like a music quote. You can come up to me later if you know it. And um it destroys the free internet. But there was another thing. The five sins are yeah there's energy crisis in depth. But

I was really concerned. Who of you lives in Sophia? Okay. And has a house? You are rich people because you are having the most unaffordable housing. If you can buy a house here, it equals one house in San Francisco and four houses in Singapore. Wow. So, I'm a good company here. Thank you. I will accept checks and uh visa, of course. Um, no, but seriously, let's let's

go to the Acme method. I promised you to talk about it. And uh Acme of course the largest nonexisting company uh selling uh rockets and uh uh high energy roller skates and stuff like that to coyotes to uh catch a fast running bird. Uh and actually it's a pneumonic which is always is a kind of a tongbreaker. There's a movie with Kiana Reeves about it. Um and

there are a couple of famous um mononics that I would like to discuss with you like SF depot you know San Francisco does it properly on Tuesdays or Thursdays you can switch for that um and the other one is hiccups and that's for evaluating test oracles do you know these anyone nobody good because I will do the acme method and after this you know you don't know

these But you will know the ACME method. So ACME what's it all about Renee? Well the A is for anticipate. You know we are going to deal with AI. You need to anticipate that it will be in the workflow in your environment. It starts with awareness. You know be aware is it know look it up find it. So detection is impossible uh is is important. You have

to detect where it is and you have to understand its ideology. Why is AI doing what AI does? What does it mean for me? What does it mean for my co-workers and the company that I work for? Educate yourself. You know, read about it. That's the only way that you can understand the ideology. And the most simple thing, just start using it. See how it works for

you. See if you can train it a bit that it comes be more you let's move on to the C which is control we are aware now now we can control the thing we can't but we can give ourselves a kind of feeling uh about it after meeting Fippin for the first time my metaphor with cars is slightly gone but um you know if you you think

you control the car but you don't Yeah. Um, important is adaptability. We always say this, right? It was on top of many books. We have to be adaptable because the today's knowledge, tomorrow it's obsolete. You know, it goes really fast. So, you have to adapt and you can use zero AI. There are many good zero AI tools out there and I know conferences use it to filter

out all the abstracts who are built with AI. um just to um divide real stuff and Um there's training there is a lot of training out there for by many smart once you are trying to do the control thing you can do embedding and you can get certified. Yes, we can. There are a lot of certifications out there just like there are LLMs out there. There are

a lot of certifications out there and that's the learning that's the training part. Yeah, I don't suggest that you have to do a certification. That's an ongoing discussion in our community as well, right? Do I have to get an IHTQV certification? No, but it helps. Yeah. And that people want money and then they can do great conferences as well. Um from control we move on to manage

and manage managing with AI managing to cope with AI starts with getting it a part of your culture your company culture your personal culture you know so we talked about embedding that's in infrastructure in your system and we have that well but this is for your you know this is the personal part your put it in your way of working yeah for all you agile lovers out

there, um, make it a part of architecture. It is software. It can be put anywhere in your architecture. It has to have a place there. So smart people in the company know when we do this, it affects something that we do with AI. If we update to the next Jet GBT version, it does something with all the other things. You know, it it's just like any normal

software or any tooling that you buy. But also for our own testing architecture, it's important where do we put it in between our automation tools or maybe together with our automation and making it a part for the scrum masters and product owners in here of your definition of done already or whatever you call it and now you've managed. So you know we are aware we are in

control we are managing it and now we can expand just like AI and LMS are expanding. [snorts] Stay informed. Come to conferences like this. Listen to crazy people like me and learn. Read blogs. Read articles and studies. And I said use it. And that's your trial and error part. You know, it will fail, it will go wrong, you will be disappointed and then you will have success.

You know, it's like with Thomas Edison, he had to use 10,000 times to make this kind of lighting possible. So, it's okay to have errors and to fail. Be open about it. Talk about it. It's not a filthy thing. It's a good thing. You know, it's just another step in our IT evolution. And this is what Richard Bradshaw always says, use it early and often, you know,

like Tito, test early, test often. AI early, AI often, you know, uh, it's on your phone, it's on your laptop, it's on your desktop. It is out there. It's probably on your TV as well. And my famous one, my my my favorite one, testimisms. Uh, what? Well, you saw that I call myself the testimist uh from pessimist optimist like the Americans do big bigger American or these

days big bigger Trump or something. Um and of course this is a play on euphemisms you know it's it's being positive about what we are talking about this this is great right this gives us more work this gives us more fun so I'm always almost in the red zone so it's time for me to wrap things up so brace yourselves and I saw Philippines Um, slide. We're

on the same page here. Anyone know this? The guy is from Game of Thrones. Yeah, he's Jean Bean. He gets killed off in every movie and every series in almost the first part. Um, and the music. Anyone knew the music? I have a whole um, uh, presentation with all kinds of movie quotes and sounds. This is just one. So, anyone knew recognize the the team? Lord of

the Rings is like going to a Star Wars convention and calling out Star Trek. Uh it was of course Game of Thrones, the Winterfell theme, but okay, I'm the I'm in the only nerd here on stage probably. Um let's let's let's do the summary, right? Um I I've told you about um opening the box of AI of I have told you about the parallel between the both

worlds. Uh [snorts] what came out an awkward first date, you know, QA AI, we are in a dance together. Um I've explained the Acme method. It's it's just my idea of course, but use some of the things that I talked about. It will help you make your work a lot more easy and more pleasurable. And the most important thing, testing off with, by or for artificial intelligence,

because there is no escape anymore. It's it's here. I know automation for some of you was a hard thing to adapt to. Uh but it was there as well and you have to work with it even when you're in Excel. It's kind of automation. Um but um AI is here and we are going to have to use it. And now um you know you have to um

you know do some audience interaction. So I'll give you something to interact with. Thank you very much. So, if you have any questions, don't hesitate to ask it. Uh, you will receive a t-shirt for the question. >> Question. >> There there's one question. >> Yeah. Only if it's a good question, >> Yes. Okay. >> There are no bad questions. >> Sure. If this is a good question.

>> Okay. [laughter] >> But I hope it's an interesting one for everyone here. Uh, I'm Camila. I just wanted to ask you for a manual PA who's never integrated the AI in his work, right? What would be the practical steps to start? I know that you said that you need to read about it right but you cannot just simply open chat GPT give the documentation and ask

for the test cases right so how you would suggest to >> yeah we talking about an LLM that is not on premise right because if when it's bought by the company that's kind of safe so then you can use it but if it's open yeah um don't put your test data out for your company because everyone will enjoy it and like uh all the cat me movies

once it's out there it will be out there forever. Um but um you know like using test data um don't use the real thing you know so scramble your information but stay close to what you want to test just to get acquainted with it and you can you can think of uh real life stuff like from companies that are already out there doing some things you know

use that as an example and then you can go back to the the workplace face.