About this talk
In this talk, David discusses the emerging concepts of vibe coding and vibe testing, highlighting the risks associated with AI-generated code. As chairman of the W3C's browser testing and tools working group and head of developer relations at BrowserStack, David provides insights into how these AI tools can inadvertently compromise coding standards and security. He presents real-world examples, such as incidents where AI tools caused significant data loss due to failures in coding practices. David emphasizes the paradox of speed versus understanding, noting that while AI facilitates rapid coding, it can lead to a lack of foundational knowledge about the code being utilized. He argues for the necessity of understanding the full context of coding and the potential pitfalls of relying solely on AI, urging developers to maintain a robust knowledge of the fundamentals to ensure software reliability and sustainability.
Full transcript
All right. So, today I'm going to be talking about vibe coding, vibe testing. It's kind of the new terms that everyone's talking about and how these AI powered shortcuts can actually create new problems. Um, so for those who don't know me, uh, my name's David. Uh I chair the W3C's browser testing and tools working group. So this is the working group that kind of creates web driver.
The web driver biodai uh does all the work with all the browser vendors to get things uh into a space that you can use it. Um I work on selenium. I've written books and now I head up uh the open source program office of browser stack and also head up uh developer relations. So the agenda for today is I'm going to talk about the AI paradox that
I think a lot of people are seeing but don't fully understand. Uh I'll talk about vibe coding, the rise of vibe testing and some real world pitfalls. Um and along the way I will try to give you real world examples because I think it's important uh that like it's not just theory that we see what's actually happening out there. So in July this year um this story
came out. Two major AI coding tools wiped out the user data after making cascading failures. And I really like what Gemini said to the user. It was like I have failed you completely and catastrophically. Right? It was so like one of the the first one that was um like made it on there so was Gemini. The other one was Replet, right? Replet's getting a lot of good
news at the moment uh for their work. Um and what Replet did was they deleted uh the production database for the person that they had created. It was all the sales leads. So suddenly they had no sales pipe, couldn't make any sales in their company uh because the person who was doing it was vibe coding. Um and that's quite scary. Like we're we're all taught never do
anything in production. uh like you have all these guard rails, but the world of Vibe coding and things like that is slowly but surely eroding the guardrails that we've set up for years, maybe even decades to protect our users, protect our data, uh and protect our company. So the AI paradox is speed versus substance. So if we look at like all the tools out there, Gemini, Chat,
GPT, Claude, things like that, the speed that you can write code at nowadays is unprecedented, right? We saw um Hamza's graph earlier of like the the rise of AI, right? how quickly it is going up that way and how quickly it is kind of democratizing all the roles that we can do and suddenly we've got rapid prototyping and as we all know prototypes go into production and
then they stay in production for forever right uh we then have to fix them um and the speed is like phenomenal right I'm sure we've all been following the the work that Jason Huggin is doing about Vibium, right? His vibecoded selenium um and all the work that's going on there and it's moving so fast. But the problem that comes with it is suddenly we lose all understanding
of what is happening underneath the code. Suddenly we can uh write code very fast. We have people who have written code really fast but they are product managers and don't understand how the code works. They are designers who don't understand how the code works. And in the same way that like and this is going to show my age. When I started in the industry and I was
using Dreamweaver to write to build websites, I had no idea what the HTML would look like. And then as I got better and I looked at it, I was horrified. So, and suddenly like the foundational knowledge that we all need is gone, right? We have moved our thinking into another tool at the expense of speed, at the expense of being the first to market, the first to
do anything, which I think is really important to like to make sure that we understand this. So, um, I've done a talk in the past where, uh, Selenium is not the the reason why your your tests are flaky. The reason why your tests are flaky is because you don't understand how browsers work or your application works. And it's not your fault because we've been taught to focus
on the tooling, the frameworks, the the next step. understanding the getting that becomes the thing that you need to do next. So, vibe coding um for those who don't know what vibe coding is, it's kind of writing or accepting AI generated code and you just ask a prompt. I would like you to build out this feature and or I would like it to look like this. I
vibe code um mostly around things that are front end uh because I understand the fundamentals of CSS, but I can never get the positioning right. Right. And so the code kind of starts to feel right and I don't always spot the um that are there because it kind of it's placed it in the right place. Going back to my Dreamweaver um like example is that I've kind
of instead of being able to drag and drop, I just go use a natural language prompt and go, "Hey, I want this over here now, please." And I want it to be mobile friendly while while doing that. And the problem that we then kind of hit is that we stop anticipating failures as testers, as developers. Uh the problems that we always tend to focus on is the
happy path. Testers then have this unique and amazing ability to find every edge case. They find race conditions. They find obs obscure bugs that no one ever thought of. And like as I put at the bottom there, if you don't know why the code works, you won't understand why it fails. You don't have that fundamental knowledge underneath. and going to kind of back to edge cases, right?
As testers, we know that there is no such thing as an edge case because everything suddenly becomes an edge case. We can change our code. We can do things like that, but the edge case will still be there or we just keep adding if loops or if statements. So, I've talked about the basics of Vive And I'm keep seeing kind of the the phrase being used a
lot which is what is vibe testing? Um and vibe testing is this kind of incredibly reactive approach. Again it's like can I quickly write up a playright script selenium script to do what I want? And the coal the goal is always make it or make it work good enough, right? And we all have this kind of role in our jobs every day of kind of the feature
that we've been given needs to be released the next day, yesterday, um things like that. And so we we just go, well, it's good enough and we should focus in on getting it right. Um, but if we keep doing the bare minimum of trying to focus on speed, we never make sure that our code is actually robust. Can it scale? Can it take um bad data inputs?
Right? The thing I used to enjoy doing um like early on in my career was just putting a uh uni-ode snowman as text. Right. A lot of code is written to be pure ASI. I was working doing a lot of Python at the time. Python is awful or was awful at the time for kind of focusing on uni-ode. You put in a snowman, your code will crash.
So that was my way of doing it. And it would always like it's a little thing, but it would check the robustness of the But if we're vibe coding, our unit tests become incredibly shallow, right? Is it doing what we want? Security checks that we might have written as unit tests or things like that because we can do that. those get dropped quite quickly um because we
don't understand how the code is accepting data from a user. Easy ones for this are kind of cross-ite scripting. Hopefully, if you're using a framework, the framework takes care of that. But not everyone understands how frameworks should work. So, if you're vibe coding and go, I want a website, is it going to be using the latest front-end framework, uh, or is it going to be kind of
just writing all in JavaScript or do you actually understand the difference between those two points? Not everyone does and it's okay because like our industry is designing things in a way that we move fast, we break things. The other thing that kind of really crops up and and this is where the human side of things becomes incredibly important. Performance regressions suddenly crop up. No one notices because
the code is passing. You have your automated tests. They are passing. No one spots kind of the performance of everything. My favorite thing when I kind of say to people like when they say, "Oh, I should automate everything." I go, "You shouldn't." Because a good automated test will log into your application, do the thing that it wants. It won't notice that moving from the login page to
the next page took 10 seconds. It can't do that, right? Because we go, "Oh, Selenium's flaky. Puppeteer's flaky. Cyprus is flaky. Let me add a more timer. More things like that. And again, it's just because we're trying to move as fast as possible in an industry um that kind of goes first to market is best to market. So as I said kind of I was going to
talk about real world examples. I couldn't this one I couldn't show because I like I was working on something internally. Um but I knew because I had the underlying knowledge, the underlying craft, I was able to spot the problems which was I was trying to create uh an AI tool to kind of process user data um cuz I head up like the developer relations and I wanted
to go we've got all this information uh we need to see uh who are people constantly coming back to our meetups uh because browser stack runs um I think at 400 meetups a year at the moment around the world, constantly growing and we want to just find the people who are constantly coming back, promoting us, doing things like that. So we could send them something nice, right?
It take like they're giving to us, we should give back to them. And so I wrote some code and I said, well, I want to do this. Um, I was vibe coding and suddenly it was like I needed and I was doing it all in JavaScript because I was trying to use uh APIs for a website to pull the data quite quickly and suddenly I like found
that I was being able to firstly use non uh standard APIs. I worked in Mozilla. I still use Firefox. I don't use Chrome unless I have to. Um and suddenly like I spotted these things. Um I also then spotted that I could do um an injection of code into the website. Yes, I could be using dev tools but I could still inject into it to then get
past other things to get more data. In this case I shared the information uh with the person and they were like oh okay we'll go fix that. But in this case, they weren't able to recognize the injection case. They just assumed that like it's safe because the JavaScript is there. And um they were able to patch it. And so this is from a well-known uh website for
uh doing meetups. The other side of it is code assistants are making more security problems. Right? In this case, uh a security firm said this, right? This is not just me saying it, but uh they've analyzed, it says at the bottom there, analyzed code from tens of thousands of repositories uh with several thousand developers um at Fortune50 enterprises they're moving fast. They're injecting security problems into their
code. Anthropic CEO last year said that um they did like they expected developer 50% of code to be written by developers 50% to be written by AI. This year their belief is that uh 75% of code will be I think we're kind of where we are in the hype cycle is that next year maybe the year after uh we'll hit 100% of AI and very shortly afterwards
we'll do a reversal because companies will because hacking and like security problems is not a if statement it's a when statement right and so we'll get to a point where these code are creating larger problems, getting hacked, people's data being stolen, and then companies being sued for it. Again, this is not your fault. It's not AI's fault either, right? AI is just kind of a decision engine.
and it tries to figure out where the next bit of code comes from and it tries its best and it does really really well. But if you don't understand the underlying craft, you're in for a bigger problem. The other one is kind of I this one I kind of I I played around with and you can do it too, right? Is that if you get um you
ask AI to process something, right? Give it a large data set. I'd like you to write some code, right? Uh because that's what like AI is not going to process it for you, but it will create other bits of code or other bits pieces to kind of generate to process it. And again, if you have this prototype, prototypes go into production um rarely changed um before they
move on. And so this case, you can do it too. You can just go here's a like generate a thousand emails, email addresses, tell it to process it in any way like sorting, things like that. And a lot of the time it will go for kind of what it sees other people are doing because it takes the data that others have done. So it doesn't necessarily generate
the most most performing code. So in this case I kind of got it to um oh there's a typo uh n squ um algorithm uh when kind of it was simple like the size of the data set should be as quick as it processed. Now going back to my original thing where the person like a person from replet lost his data he only had a thousand records
right so it wasn't too big but it was his entire business a thousand records I'm sure we can all agree in a database is tiny is what we would expect like at a um high school level if you're building something just to prove in a computer science class. Um, but we need to be talking kind of billions of rows if you want a large enterprise application, if
you're processing anything that comes in at high volume, event data, um, things like that. And the world is getting into a state where like every year we double the amount of data we collect in it like as a kind of humanity. So if we're now trying to process this information and we don't understand the the fundamentals, we can't scale. The other side of it is that like
can never be 100% correct, right? Open AAI earlier uh this year admitted like they wrote a white paper. This is the news story that came out um couple of last month, but earlier this year they wrote a white paper going hallucinations are here to stay. We can't get over it. We can't beat the hallucinations. But if you understand why or what is happening, we can get there.
My favorite story about hallucinations, I'm sure we've all seen it. His a lawyer in the US uh kind of was doing his work, got chat GPT to write up, uh everything he needed, uh site case law, things like that, so he could send it to a judge. Uh the jud judge read it. The judge kind of in the perfect sense of a tester went and said, "I'm
going to check everything. I'm going to go through it. Looked at all the cited case law. None of it existed. The lawyer never even looked it up himself, right? He was like, I've got I've got to get hit my billing hours. Got to do this. I got to go send it. Uh the judge threw out the case straight away. Didn't said like, if you're doing this, what
else have you done wrong? actually before I move on is that like the um the other side of it is that uh AI can be poisoned right AI prompts can poison a data set in itself um because you can kind of manipulate how things are and we're also getting to a point uh through MCPS things like that where it's um it becomes a lot more dangerous. I
think this is fairly self-evident, We've always been taught that as the sooner you find a bug, the easier it is to resolve. So if a developer while they're coding, it's an instant fix. If it goes to a tester, it creates a loop and it's a little bit slower. uh it gets into kind of a uh develop um a testing environment, it becomes even harder because then you
have to think about deploys, right? Uh it gets into production, it's the most expensive point. So, but if you never understand that the technical debt is there, how do you even know how much it's going to cost you later? one of the like things that like I'm a big advocate for kind of uh low code automation because I see it as a a really good way to
get into the industry. You can just low code do everything you want. The thing that it always generates is code that can't be maintained. It's getting better, but it's still a long way off from being maintained. And because if you don't have the it becomes technical debt. Now, sometimes it's cheaper just to re-record and go, right? Because you've had a look at it. And I see that
a lot in different companies, and I think that is an acceptable way to do it. But if you're trying to scale up your testing, scale up everything, make sure you have this long-term plan, technical debt will kill a company. if you're constantly having to rewrite stuff. I know like Hamza was saying earlier like uh a billion tokens is um $10. But if you kind of expand uh
the context that a prompt is in, you can blow through that $10 in 10 minutes because your prompt is getting bigger and bigger and bigger, right? So, like if you go, I'd like this thing. Great. This thing is in the wrong place. Can you move it? Right. It takes the previous thing and this thing and does it. And those like tokens burn through very quickly. Um, and
it becomes far more expensive to resolve later, but it can be resolved. Like it's not u the end of the world. Now um I was speaking to a colleague um I've just launched a podcast at browser stack called browser stack talks. This the latest episode I was talking to a colleague called Dan uh Marov. Now Dan's one of our product managers super smart guy uh and he's
like described AI in that talk like when we were doing it about AI being the great aer. So if you don't understand something, it averages you upwards. And that's amazing, right? We get more people understanding things, doing things, moving on, which is great. and his analogy was if you're a musician, right? My wife's a musician. She's brilliant at music. My kids are because of my wife are
brilliant at music. I can play three notes on a piano. Right? If I got chat GPT to write compose something for me, it would do something that I would never be able to do. And I think that is amazing because it gives allows me to open up creativity that I've never had that opportunity to do. The problem is I then can't go put it in on like
a piano and play it. Pick up another instrument, play it. so if you understand what it was, if I could play the stuff and do that, I'd be a much better musician, right? Because I understand the fundamentals that are below that to be able to do it and I'd be averaging up again. Um, and moving every like because the with averages, you want everyone to move up.
Um, you're not going to like have a Beethoven come along and go, "Let's do create music through chat GPT because they understand the craft. They want to do things. They will do things that because that have never been done before." We get different types of music. Mathematical music sounds really weird and then you get like really romantic music that sounds amazing. And so we need to think
of going from this crutch where we like go the tool's going to make me amazing to actually going it's a tool. So if we get it to write code for us, get it to test. Don't just paste it. Don't just leave it in the code, right? See what it's actually doing. Read it. If it doesn't look right to you or you don't understand it, go learn understand
it. understand the documentation that it's picking it from. The other problem is like chat GPT might be using old outdated ways um that that framework doesn't support anymore or when you go to upgrade the framework later on you've got to rewrite intentionally make breaking changes so that you can walk through the code again understanding the craft behind it so you can go oh it's doing this doing
that why is it doing this weird looping here it doesn't need to do that and then that way we can all learn we all move um upwards in doing it we should um we should always dedicate time to understanding the fundamentals, right? I keep repeating that over because I think it's so important. Um like if your tests are flaky in selenium, one of the key reasons that
you your tests are flaky is understanding the event loop in the browser. When do things get executed? How do they get executed? When is it going to send network requests to pull more data from the database? things like that. Understanding that becomes really important, right? In the past, we like if you're a C++ developer, you had to think about memory management. Then we moved to other languages
where we didn't have to think about it. We had garbage collectors that just did our job for us, right? Um, but don't be afraid to kind of use it, see what the problem is, fix it, or if it generates the code and generates the tests, add another test that you think it probably wouldn't be able to do, or if it like that doesn't make sense. If you
ask AI to fix the test, make sure that it's not rewriting the test that it actually rewrites the code. I've seen that a lot of the time where people go fix this test and then it deletes the test. It's like that's not fixing the test that's deleting the test. Challenge AI, right? It's a tool. For years, we had like if you're a developer. Um, like when I
got started, like the tool of choice was always resharper to get what I needed right and it was very good at making me move fast, but it wasn't always right. And I was able to challenge and kind of learn things better. I think for testers we need to always focus on the boundaries right wherever boundaries are there are bugs contracts will be as good as kind of
the data that it's expecting. So give it the unexpected. Does it suddenly crash the entire system? uh if you're using microservices uh what happens if you spin up a previous version of that and the front end needs to speak to a previous version or kind of things like that does is the backwards compatibility is the kind of the contracts that everyone understands always up to date or
are we slowly but surely like not understanding what's there? also, and I know this is contrary to a lot of my talk, is ask AI if it were to test this, how would it test it? And then go, "Oh, okay. I hadn't thought of that." I think a lot of our job is nowadays to use our creativity. Humans are an amazing creative creatures, right? We created the
internet. We've created AI. We've got done all of this. Um, we've been able to do all these. Let's get it to do some of the hard work for us. in summary, like AI is just a tool. It's a tool that makes us faster if we understand its limitations. Um, and it allows us to kind of be better at our jobs. We as an industry need to master
our so much. Understand like how asynchronicity works. How like the language that your company is using to write their application. How is it actually putting these things together? When it run like you know when it compiles, what is it doing? Like one of my favorite languages that I like to play with. it's only playing with is Rust and being able to go with Rust, you know, there's
no garbage collection because like once But if you're just going AI write me this code in Rust and you don't understand that and then you make a change and it's like well I need to actually extend the life of that variable because Rust has these thing called lifetimes you if you don't understand that you're not going to be able to do it and build for sustainability right
get get it like if you're gaining AI to vibe code like you're vioding get it to do smaller things. Get it to explain why it's doing it, how it's doing it, and then building it out. The more you build for sustainability, and to me to be sustainable also means writing incredibly boring code because boring code is easy to read and easy to maintain. You build for sustainability.
No one wants to be rewriting things in six months. And with that um I've come to the end. Um it's about two minutes but if there are any questions if not I will be at this networking area. You can ask me more questions there. >> Okay. Thanks. Thanks David.
