Andrea Arcuri: Fuzz Testing Web APIs: Overview of Existing Tools

This talk covers automated API testing, focusing on the importance of testing REST APIs and the challenges associated with it. The speaker, Andrea Akuri, discusses the prevalence of APIs in modern applications and the difficulties in creating comprehensive test cases for numerous endpoints and varied input parameters. He introduces the idea of automating test case generation to improve testing efficiency, highlighting various tools designed for this purpose, such as Fuzzers like Evo Master, SchemaFuzzer, and Cats. Akuri also elaborates on the potential for leveraging AI techniques in fuzzing, exploring the ability to assess API behavior based on generated test cases and responses. The session offers practical insights into setting up these testing tools and the significance of utilizing schemas for effective test case generation while touching upon collaborative work with industry players.

And I'm very pleased to have Andrea Akuri here to tonight with us. He's a professor at the Cristiana University of Applied Sciences in Oslo, Norway. And he's doing the research and very innovative stuff as well as practical stuff. As part of his work, he did the Evo master and API automated API testing tools, but he will talk about automated API testing in general here and we will

here learn a lot about different tools and different methodologies. Please welcome with me Andrea Akuri. [clears throat] >> Thank you very much. It's a pleasure to be here today. So I'm going to speak about fats testing web APIs. APIs like REST APIs are everywhere. You use your mobile an app that's going to connect to a remote server that's going to be an API. You are using a

web application in a browser most of the time that will make communication with an API somewhere. And it's not only a single API modern enterprise system often are built with microservices system. So they can be hundreds or thousands of APIs and I read some recent statistics saying something like around 70% of the traffic on internet is about API calls. So they use it everywhere and uh because

this is a testing conference of course the question is how do we test them? Uh how many people here in the audience write test cases for APIs? Can you raise your hands? So quite a few. So you know what is the problem? Uh you have several endpoints. You might be tasked, oh please can you test for me this API with 100 endpoints. You need to start to

create test case for every single endpoint. And there are quite a few decision that you need to make. Example you might have query parameters. I mean it's not sometimes if you're lucky an endpoint has only one or two query Some other times maybe it has 15 parameters and then you need to test them and possible combinations path What about body payloads? You're sending a post with some

complex structures as input like a JSON uh object with tens and tens of fields and you need to fill them up. So this takes time and uh sometimes even set up the environment of API like possible test data into the database when you want to test these APIs you might have different kind of objectives in mind. For example uh you might trying to maximize code coverage on

the source code of API. You might want to maximize uh coverage of a specs mean trying to call every single endpoint at least once with different status codes as returned. You might want to find faults and so on. And of course if you do these things by end especially for large APIs it's going to take quite a bit of time. So what if what if we could

generate those test cases automatically with automation? I'm not talking about the execution but the actual generation of the test cases and making all of those choices like which query parameters to use, which uh body payloads to employ to do these things automatically. And of course it's a hard problem. I mean if it was easy it would already been solved decades ago. So let's assume you can generate

those test cases. Okay, you have a tool, you have an API, you get as output some test cases out of it. You don't know if are good or bad, but what are you going to do with them? What are you going to do with those test cases? There are at least two possible things you can do. If a test case tells you, hey, I found a bug

here, you say, oh, let's investigate it. Let's debug it. Let's give this test case to a developer and say, "Oh, this test case says that there is a bug here. Can you double check it?" That's an option. But then you need to have a way to tell you if a bug was found. And there are different kind of bugs that you can automatically check. Of course, you

cannot find all possible type of bugs, but there are some categories that can be predetermined that you can check out. A typical example is uh if an API is returning 500 server errors but you can flag it. What if you get a response that is very different from what's specified in the schema of API or there are also other kind of uh security vulnerability security properties that

you can automatically Then let's say that you don't find faults but you still have a set of test cases possibly as output. And one thing that you can do especially for developers you can use them for regression testing. Maybe they don't find a fault today but when I rerun those test cases tomorrow after the API has been updated maybe might break some existing functionality. So fodzers fadsers

are tools that can automatically generate test cases and this is nothing new. This has been done for decades. I mean fuzzing at least 20 30 years in a lot of different domains especially in parsing and unit testing because one of the easiest uh way that you can do is just generate test cases at random and then if you have an oracle that tells if a fault is

found or not like a crash then you can use that automatically and then you can leave a fer running for hours generating possible millions of test cases and then if any of those crash then you have a test case that is crashing that then you can give you to the developers and the point is that you don't need to babysit those tools I mean you can run

them on the background you can run them in the night you can run them on continuous integration servers once you start them then you can come back a few later a few hours later so what about APIs so what about fudsing APIs that's something that has started a bit more recently in the last four or five And there are quite a few tools out there that you

can use. So I'm aware of at least 25 open-source fodzers on GitHub that you can use. Uh some of them are kind of now dead like dread it has been discontinued. Some others are just uh academic proof of concept. So you cannot really uh use I mean you can use them if you want but uh be warned. However uh I'm going to briefly mention at least four

of them. The top four fodzers that I'm aware of wrestler schemais cats and evo master. And uh the choice is just based on popularity on GitHub. It's just literally uh sorted by number of stars on GitHub. And as I said, these are open source project. Those are not commercial tools and they're all on GitHub. So Wrestler uh the most popular at least according to GitHub stars uh

was actually made by Microsoft research and that has been open source since 2020. So these are not really something uh new has been around for some years. Written in Python not super friendly to use. I mean you have to uh uh make a clone of git repository and install it locally. Uh but but it works. You can check it out. You can [snorts] use it and try

it Another popular fodzer uh is called schema physis but is even older from and written in python as well and still uh in continuous development which again is a fad that can generate test cases for rest api as well as graphql and so Another popular one cats uh made by company called endava.com uh again from 2020. So these are not something from this year or last year.

They have been for some years around. In this particular case, it was is written in Java. But you as a user how it is written is not so important. But just to show you if you are look interesting to look into the source code because as I said all of these things are open source. And the last one uh which is master that I'm actually one of

the authors and is actually the oldest and again these things has been on for quite a while but we started nine years ago. So that it was already in 2016. So how does toast tool work? The point is that given an API you need to have a schema typically an open API schema because if you want to generate test cases and you only know uh the URL

of API it's not much point to send random bytes on a TCP socket. You want to send calls HTTP call that make sense from a point of view of API. So you need to get an open API schema that tells you which are the endpoints, which are the query parameter and what is the expected structure of the body payloads. And again uh open API is the most

uh used and most famous way to define schemas for rest API but in theory it could also be other schemas approaches. How many of you have ever heard about pet store? Okay, some hand. It's just a trivial example API that is used for the dactic reason that was introduced by the outsource of swagger of open API just for showing an API and is actually online. So for

example that on that uh URL and you can actually get the open APA if you want to look at it. The question is what happens if I trying to fudge this API? what I'm going to get out of it, what should I expect? All these tools first will download the schema because they need to know how to structure the messages. As I say, sending random bytes on

HTCP connection is kind of pointless. So you want to send properly formatted HTTP request with the proper endpoints and all other data structures toward VPI. How you are going to do that? What strategy is going to be used to decide which inputs to send? That's an internal detail of a tool. The most trivial could just do things at random. But then there is a lot of research

and a lot of techniques using for example AI. I mean that is the buds of the day but as I said it has been things doing for decades. And it's not only LLM. It's not when we're talking about AI is just LLM but there are a lot of other techniques like neural networks reinforcement learning genetic algorithms and so on. Then it will they will start to send

messages in a loop depending on how long you want to keep a fzer running. You can keep you can keep it running for 30 second. You can uh leave office at 5:00 in the afternoon and leave it for running 12 hours and pick up the result the morning after and so on. We will generate test cases. we will try to make HTTP calls possibly millions of calls

and if for any reason a bug is found they can detect it they can say hey I evaluated uh five millions test cases and out of those five millions I got these 40 test cases showing these bugs so of course as a user you don't want 50 millions test cases as output that will be quite a long uh file you want a minimized test suite And the

outputs could be in different formats which again it depends on the tool. Could be code like in Java, Python, JavaScript could be supporting some other tools like Postman or Bruno. It depends different tools will have different supports for different formats as So let's say that we want to uh fuds that pet store that is online you uh when you use a tool like in this particular case

I'm using evo master as an example I don't even need to install it if I run it from docker then I need to provide some parameters the most important is where is the schema like in this particular case https pet storebagger.io your slashv2vagard.json then I might also need to provide some other parameters like as I was saying for how long do I want to run the fzer

in this particular case is just an example let's say okay 30 seconds I'm fine with that uh but you might also want to put some rate limiters for example if you're testing an API on your machine you you want to generate as many test cases as possible but if I'm just using as an example something online I don't want to do a denial of service against that

API. So I'm limiting myself as okay I want to test it but let's not do a denial of service attack. So here I'm saying just six 60 calls per minute. So I have a short video in which this is run. I don't like live demo for a very simple reason. Those are the best way to do testing because if something is wrong, you will find it immediately

when you do a a demo which is great for the you but not maybe great for the audience. So here I'm just register. It's just short. It's just two minutes just to show you that you can just copy and paste a command like that run it on a shell and that what it's going to do is start to make call toward that remote API with different strategies.

And of course it will take some time. It's not immediate. But the point here these kind of things can be run on an old laptop. You don't need any uh expensive uh GPU or cluster of GPUs. An old laptop from 10 years ago will still be fine. So in this particular case, it's running for 30 seconds. Once it's done, then it does some minimization phase. And the

point here is that this is automated. You can have it on the background. You can you can have it in a continuous integration. Uh it's not that you have to check every single step. Of course, depending on where the API is, uh you might generate more or less test If it's some an API remotely on the other side of the world where it's going to be some

delay. If you're running the API on your same machine or inside a testing environment, then of course you can it will be faster. Then once it's finished, we'll generate uh some test cases. Now let's see here. For example, on by default generate test cases in Python, but again there is other outputs as well. And then of course you might generate 50 or 100 test cases depending. But

then you also want to get some test reports out of it. Some way to visualize uh which parts of API has been covered and which parts have not been covered yet. Yeah. And I think this was a V. Yeah. And then of course you can look end point by endpoint. So the point is that a fader might generate 100 or 200 test cases. is not going to

give you millions of test cases as output, but you also need a way then to get an overview of what has been tested. So yes, so I actually run this thing uh on the 29th of September. So that is the result. But again, APIs online change. So if you do the same things today might get some different results. And the point is what kind of test cases

are you going to get? This is an example just calling an endpoint with some query parameters based on the schema uh with some random inputs. This is fine. All it go all goes well. You get a 200 status uh code back. You put an assertion on it. Happy day scenario. And of course you might want to improve it. I mean it's an executable test cases as a

tester you can take it expand it use it as it is modifying it and so on but that's in another another interesting uh here is making uh a a post call uh with a body payload. The body payload is structurally correct. Random inputs when it comes to strings but technically correct. All is good. You get the API when you send that payload tells you all fine 200

status code. Either for some reason that API thinks that the 200 is not allowed for that API in the schema. It says oh I'm only uh expecting user errors. And of course this kind of mismatch can be automatically flagged. Now here the bug is not uh in the API the bug is in the schema I will assume. But this is a type of things that you can

automatically check. You don't as a user you don't need to do anything. Then of course uh you also want to get a general picture of what happens because okay you run a fadser for five hours you get as output let's say 120 test cases minimized okay you don't want to get millions of test cases as output but how much of the API was covered then of course

most of these tools will provide some kind of high level overview of what was tested like in this particular case only 13 out of 20 endpoints, it was possible to generate a test case that was successful, returning it to under And then you might also want to go into more details and check each endpoint one by one and see which status code did I get on this

endpoint, which faults were found on these endpoints and what are the test cases that can show those pet store is a trivial example. You take a fer, you point toward an API, you get some result. But as you know, most APIs need some sort of authentication, username and password. And that information has to comes from somewhere. mean uh even if you have a direct access to a

database in which the authentication information is stored usually password are hashed. So you cannot uh even if you have a tool with direct access to the database to reverse engineer it. So somehow as a user you need to provide some extra information like one, two or three uh valid users with credentials uh with as I say for example username and password which must be real based on

the contact of your test environment and then of course you have to say oh what which one is the endpoint for authentication. If I send a payload with my username and password uh to get an authentication token, where is that token? How do I extract it? How I how I'm going to reuse it afterwards? So there is a bit of information that you need to provide can't

escape and there might be different ways to provide this information and approach uh now that is used by the master and but also other tools like schemas and cats are going to support uh is to provide yaml files or toml if you prefer toml with some basic information and the point is that when you have this information. When you provide this information to the fzer, it can

use it uh it it can make a call by first making a call to the authentication endpoint, extract the token and then use that token in the following request to the and uh based on the output that you want to generate your test cases, different code will be generated based on that authentication information and that's why it's important to have that authentication information in a configuration file

and not in code because this example is in Java. But then what if uh you need an output in cotlin that's exactly the same code using the same information but in cotlin. Now if you really really have to to do it in JavaScript yes it's also possible to do it in JavaScript if you really have to. But hopefully maybe Python is more common. But the point is

that all of these test cases are exactly the same just different programming languages. And the point is that that authentication information that you only need to provide once then can be used and embedded directly in the generated test cases because then each test case is self-contained and can be uh used uh by itself and uh I don't have time to go into all the low-level details of

security and penetration testing. But there is more interesting things that this foder can do automatically once you provide such Now let's say uh you are as a tester you want to check some access policies and you have two different users a user bar create a resource what resource it is doesn't matter just creating a resource then you want to test that another user uh is not allowed

to delete it another user cannot start to delete resources created by others. So you check that actually you get the 403 not authorized but what if the delete say not authorized but you can do whatever modification you want with a put. So that will be kind of weird. Will raise some flags say oh I cannot allowed I'm not allowed to delete a resource from other user but

I can do whatever modifications I want. That's a bit fishy. There might be some access policies out there in the wild in which this is perfectly fine. But what if the developer just forgot [snorts] uh a check on that endpoint? The point here this can be automatically flagged. That's a trivial example and this kind of access policy violation when you use this fzer can automatically be flagged

and the fer itself can generate a test case in which there is user bar creating au a resource a user fu trying to delete it and getting a 403 and then trying to modify it successful with a note a this most likely is a bug please check it >> [snorts] >> So those were kind of basic example uh blackbox testing. Uh what if you have access to

the source code? If you have access to the source code, oh you can do much more much more interesting things. You can do things like whitebox testing. You can use evolutionary computation. You can use genetic algorithm. Lot of fun stuff. So what if you're dealing with a SQL uh you can actually when you do uh whitebox testing you can automatically instrument the code of API intercept every

single call to the database automatically analyze every single commands to the database and check if there was any for example select that didn't return any data and the point is that from the point of view of testing maybe you want to at a path in the code in which data is returned because most likely that returning data will impact what is executed in the PI. Let me

be a bit more concrete. Now this is a testing conference. So I'm not sure how much of you have experience with Java and Spring Boot. But this is just a trivial endpoint. A very very simple trivial endpoint that can be fit in a slide in which an endpoint receive two numbers as input and is check if those two numbers are in the database. If yes 200 if

not 400 as a tester you might want to check both scenarios. But the point is when you do white box testing this become trivial. It's easy to automatically generate test cases that can insert data directly into the database and then call the endpoints by matching the same data and that we do it by analyzing the interaction with the database and that's a type of test case that

you can get as output. Then there is even more sophisticated techniques. One very important technique is called taint analysis in which you analyze how variable are traced in the execution uh of a system that you're testing. You can trace the inputs from for example from HTTP calls and see how those inputs are used inside API and in different part of the code. So for example here again

maybe not the best example for a testing conference but is short just an endpoint taking three strings. Uh so no information about the actual value in the PI and the first the date is parsed into a date object. The second is parsed as a number and the third is parsed as an enumeration. And then there is an if statement a conra check that looks at the year

of that date saying that it should be 2019. Look at the number and say it should be 42 and then checking that that value is part of enumeration. From the point of view of blackbox from the point of view of a schema you only see three strings but those are very specific properties. And here for example you want to test the epidday scenario that okay again the

solution here you can use state analysis in which you can trace all these variables are used. So you can trace that uh the input from the HTTP request uh a string date when was matched to a library that parse it as a date and so on. But that is not enough because you have some constraints like the date must and in that case when you use techniques

like genetic algorithm to solve it. Long story short something like this is It's very easy to test automatically and as a user you don't need to know how a genetic algorithm can solve that constraint there. You just get at the 2019 is not out of context is because this example is from 2019. So this things has been around for quite a while now. So this is at

a high level how things work and what you can get out of it. Uh so let me discuss a bit of uh application and success stories. I mean does this thing work only on pet store or you can actually use them in practice. So I'm uh as I say the author of one of these tools Eva master which has been around for nine years now I mean

we started in 2016 and uh we applied I mean I'm a scientist so when we develop these techniques we do quite a few experiments we do analysis we do comparison and one way to check things is to apply them for example on existing API like open source And nice thing that uh 9 years ago was maybe difficult to find interesting API open source but today it's not

so difficult. Uh a lot of countries around the world uh in their public service they put a lot of their code as open source. I am from Norway and a lot of uh departments uh in Norway actually have a lot of their API open source on GitHub. So those are interesting case studies because of course you want to start from something simple like pet store but you

also want to test uh things that are more realistic and this is the type of things we do. We develop new techniques do they work or not? We run experiment and then we also compare with other techniques in the literature. So for example, this is from one of the most recent studies in which we applied. We compare six of these fads on 36 APIs and I say

some of those APIs are example, some others are real world from uh uh department uh government departments. So you can do uh these kind of uh studies uh in the lab on open source API but that tells you only one part of the story. You can measure things like code coverage. You can measure things like which bugs were found but how testers are actually going to use

them. Are these test cases useful for them or just a waste of time? Does it matter if you get a test case a test suite with 60% coverage but you cannot read it because it's comprehensible these kind of things you cannot answer them in the lab and so that is why we work with industry we have some collaboration with some companies and as I say uh in

which we apply these techniques in their company and see what kind of results we get out of it and as I say I'm a professor I work at university I'm not selling uh tool licenses. We do experiment, we do invent novel techniques and uh so we collaborate with some companies. Uh one for example is mate one is anyone from China here? No I guess not but in

case that's one of the most famous company in China. uh and actually we use these techniques uh they use it daily on their microservices and this is the type of studies that we do uh they needed some techniques uh to how to reuse existing test cases so say okay yes let's trying to integrate it let's see how can we reuse existing test cases but then we needed

to have access and run experiment and say oh is this technique actually working so we did a study in this particular case we didn't need thousand of API Okay, 40 APIs were enough and that is the kind of experiments in which then we check how much coverage was covered, how many faults were automatically found. Another collaboration that we have is with Volkswagen in Germany. Uh in that

case we do blackbox testing of REST API. uh and there uh the tools were open source you could use them but there are quite a few features that we really needed to be able to use them uh properly. So what we did we say okay we can implement these features but then we want to study how does this thing work. So from us then we just as

long we were allowed to report on this experience then was for us was fine. So we made a study and check what was all the things needed to be able to apply it at Wswagen and what is still a challenge what are the open problems and yes there is a lot of challenges I say uh in academia there is a lot a lot of research with AI

especially with AI techniques on how to improve things like code coverage and fault detection we are far away from 100% code coverage or finding all faults and we will never get there but we can improve step by Uh faults there can be a lot of different kind of faults that you can think of that you can try to automatically detect. So there is quite a bit of

research and say what more can we found. We will never be able to replace a tester but the more things we can automatically check the more we can help And another important thing is test readability because let's say you generate you get as output 200 test cases can you read them? Can you understand them? Especially when it's a complex workflow with an API putting some state and

a second endpoint using that state and so on. So the understandability of a generated test case is a very important topic. Then of course this is the year of LLMs and the question is why don't you use an LLM? I mean you have an open API schema and say oh charg get me some test cases out of it from this open API schema. Yes, you can. That

will work. Result will suck. Uh and I know because we did we did the studies. And the point is that uh if the only information that you have is the schema, you are constrained by the schema. You want to exploit as much information as possible if you want to generate better test cases. And one of the main thing you can exploit are the responses from API. Every

time that you try a new test case and you get a response, you can use and exploit that response to generate better test cases next time you generate a test case. So yes, you can use LLMs but will be quite limited in functionality compared to a specialized foder. [snorts] So in conclusion, there is many success story about fuding. say not just rest api libraries unit testing and

this has been going for at least 20 years web APIs like rest is a bit more recent and let's say I'm the author of your master so I'm biased when I speak about it but you don't need to try it you can try schematis you can try cats and all those tools are open source you don't need to buy a license to try them out and as

I show you if you don't need to deal with authentication is trivial to try it out. You just need to know where the API is running and access to the schema. And I think this is mainly from my side what I wanted to tell you today. As I say, there are at least 25 uh tools out there, but the four main ones are those. I'm biased about

EVO master, but you can try the others like schema fees and cats. Um, if you if you try it out, if on Monday when you're back at work, you try it out and it work great. Excellent. If it sucks, still excellent. Tell me, I'm really want to hear from users like you that use these tools in practice. As I said, we do quite a few collaboration with

companies. So, we actually want to hear feedback. And so, if you want, you can just reach me on LinkedIn. And uh, thank you very >> [applause] >> Thank you so much Andrea. This was very interesting and the most technical talk in this track. So I very much enjoyed it. So many details and so many APIs. >> Let's see what people are asking here. So the first question

is any options for API neighbors like websocket messages. Uh, as far as I know, no I don't know any fuzzers that deal with websocket at the moment >> or websocket. I don't even know if there's a swagger definition for >> a sync API. >> Okay. Yeah. >> So, there is a schema uh to define things for websocket and so on. But I am aware of no foder.

>> Today then who knows next year. >> Okay. Then let's get to the next question. Oh, it's just changed the voting process. Don't vote so fast. so far for me. [laughter] Uh, is it worth to test scenarios that um are unproducible on GUI? Eg sending post with random strings that on GUI can be selected from a list only. >> Uh, how do you know that that GUI

will never change? How do you know if that API is going to be called by a GUI? So, if some inputs will never be called from a GUI, then maybe it's not so useful. But you don't know what will happen tomorrow. >> Yes, I'm not even quite sure if I understood the question correctly, but uh that's uh what? Yes. Well, let's get to the next question. Maybe

a parallel topic being professor. What is it move? Would you professor? Would you advise getting a PhD or master's degree? >> Career advice? >> Yes and no. Uh it was a lot of fun being a PhD. It's a very good way uh to uh go to another country if you want to uh start a new life in a new country. One of the easiest ways to do

a PhD there because finding good PhD candidates is kind of hard and we all university hire from all around the world. But the point is do you want then to be in academia? If the answer is no, maybe not. So it depends. But again the PhD is only three years so it can still be interesting. Here someone wants to go to academia. This is this you're raising

their hands or just want to go to inter academia. If you if anyone wants to go to academia, please raise your hand. >> Who wants to go to AC? Oh, nobody wants [laughter] I'm not surprised. Very positive. >> Yes, please. >> I need a I I did a PhD to get into an expert in my field. >> So, [clears throat] I think it's a good way to

go if you want to become an expert. specific field then a PhD is a good way to do it. >> Okay, PhD is a good way to become an expert. Applause for the expert, please. So, next question. Do we have do we have time for one more question? Uh, cool. So, no, this one is the blue one. What is your experience? Do you find more bugs in

the API or in the open AI specification? That's [laughter] >> I was saying in the open API uh is insane the number of errors there. uh because the people that write the open API schema manually they might not be expert and don't know all the details so very easy to make mistakes there and a lot of tools that generate automatically the schema from code have a lot

of issue because don't have entire picture so right now if find uh practically any endpoint you run a fer to you will find issues with the schema >> okay there are more bugs in documentation than in code So to avoid don't write documentation maybe >> I'm not sure I agree with that with >> okay then >> uh can you use API fuz testing against an API without

a schema? Yes, [sighs and gasps] >> you need to infer I mean if you just throw things at random on a TCP socket you get nothing out of it. Somehow you have to infer that schema. There might be ways if you have for example some uh production logs. I mean if that API is called from a GUI even without the ski if you can trace production logs

then you can trying to reverse engineer a schema for it. But starting from zero it's very unlikely. >> Yes. Interesting. So the next thing is how to handle complex data requirements. [sighs] It dep [laughter] it depends if those uh requirement are expressed with some constraints in the schema then the fer will sample data according to those uh requirements sorry according to those constraints. The point here is

that although the uh the requirement might be complex if you run a fer for hours it might millions and millions of test cases. So in those you can still maybe find uh the cover of those requirement. Otherwise if you have something very very complex there are new ways to uh provide information like uh not only open API there is a new specs called aratso that can be

used to define complex workflows but uh I don't think any fads right now support them because it's something very recent it came version 1.0 came out last month. Arat arato arato I need to look into this because that's a big problem I I had when the APIs depend on each other you need to call consistent the sequence of APIs with consistent matching of ids that you return

and send back >> and that is uh that is very difficult >> yes so I will look into this let's the next thing do you consider any hybrid fuzzing modes combining LLM with traditional fuzzers >> yes uh some fuzzer do not evo master because for each technique you use uh what it's good at. Uh one thing that an LLM can be pretty good in this context is

when you have constraints expressed in natural language in the schema. Let's say that you have a string in the schema and you have a complex constraint but that is expressed with English text. A fader will not be able to analyze it but an LLM can [clears throat] trying to infer constraints that can be used. So yes, Ebre approach are the future. >> Okay, uh that's it. No

more time for the last question. You need to be dropped. But uh give us a warm applause for Andrea for this great talk.