Anil Kumar Krishnashetty - Bots & Humans: Maintaining AI agent and automation in an AI-bot web

In this talk, Anil Kumar, a technical product marketing manager at BrightData, discusses the evolving challenges of web scraping in a world where websites increasingly block bots and automate anti-scraping measures. He highlights that the biggest consumers of documentation tools are often agents rather than developers. The session covers various issues faced while scraping large volumes of data, especially in competitive environments like e-commerce. Anil introduces BrightData's Scraper Studio as a solution that allows users to create APIs from websites effortlessly, bypassing common obstacles such as CAPTCHAs and IP blocking. He presents a live demo, showcasing how users can quickly extract valuable data from an e-commerce site, emphasizing the tool's self-healing capabilities and how it can adapt to changes in website structure. The presentation concludes with an invitation to try the platform, which offers significant support for those needing scalable web data solutions.

Hi. Welcome everyone. Um I just want to start uh this talk with giving you uh a quick uh number insight. How many of you here are using Cloud? Just raise your hands. Okay, that's great. Uh this is a fresh uh insights from Mentify. It's a documentation platform. Uh they published in March. Just one month data which has Cloud, just Cloud itself as nearly 199 million requests to

the documentation which is way, way greater than a Chrome on the Windows. So, what I wanted to share is your biggest readers, if you're building a dev tools, is not a developer anymore. It's the agent which developer is sending you to your documentation. I'm co- I'm myself Anil Kumar. Uh I'm a technical product marketing manager at BrightData. Uh before that, I have a decade of experience at

building large-scale B2B applications at enterprise companies. Uh if you don't know what BrightData is, BrightData is a world's uh number one web data infrastructure platform. At least more than 70% of world's AI labs are our customers. Just to give you a sense of what BrightData is. In this talk I'm going to give you a set some of the landscape and what are the challenges. And second part

of my talk, I'm going to do a live demo today. Want to pick one website. Wanted to build an API instantly using the approach uh using a Scraper Studio approach. Uh with that I want to start on with this saying a statement The web we have today now is a closed web. What it means? It means bots are by default are defended everywhere. AI crawlers are blocked

by default. What that means is how are we going to manage this? How are you going to access this? There's a two contrasting things, right? When I'm saying that I give you a first example. Cloud code was able to kind of get the data from the documentation. And whereas here I'm talking about the web is closed. Uh to to give you a a glimpse of what web

is closed, uh you might be familiar with a CAPTCHAs. Right? It is checking it out whether you're a human or not. Uh just want to give you a one ma'am How do we know if it's a human or if it's a bot? is answer is like, "Hey, you just got to pick the a traffic lights just to verify you are a human, right?" So, wherever you're trying

to access the content it's kind of blocked. Uh so, there is a block for bots. Taking those uh I just want to quickly run through the original version of web was more of a open. Uh you you could just get an HTML and you can just uh by just using a simple curl. And later in 2010 you could just build a apps using the social media which

were free and lot of billions of uh uh apps were built uh with the free free API. But uh later later in 2018, uh these platforms realized that data itself is a product. These platform realized that data itself is the product. And they start closing that and kind of charging for that. For example, maps uh start charging more, Twitter like that. And now now we are in

the era where web is kind of closed. What I mean by that is for example, uh Cloudflare uh introduced last year a one-click AI AI crawl blockers. Many of the millions of sites enabled this uh in last year July. And just in 5 months, they were able to block nearly more than 400 billions of bots. I'm talking about billions. So, here's the thing. It's getting tighter and

tighter. And on the other side, you're seeing AI agents are able to access the content. It's getting more and more. It's getting more nearly kind of equal to humans now. But how are we going to How are we going to manage this? Like for example, uh I just want to state this quote and move to more interesting part where currently we're in a situation where it's both

bots and humans, right? Are able to access the content. We built a web for human consumption. For a long time, we are going to share this space. Machines acting like a humans, clicking buttons, scrolling PDFs, doing OCS on the fly. That's what's look like many years until the transition is complete. So, this is the current state. I'm going to touch up on this uh in the demo.

let's say I want to get a data of a thousands of millions of uh products uh from Amazon. I want to do I want to get a pricing information, rating information. I'm building a competitive uh application. I want I'm going to I'm going to I'm also e-commerce company where I want to get the data. If it's one one a couple of tens of website pages, it's okay.

I can just ask LLM, "Hey, get me these data." Or I can use some other solutions available in the market, uh like Apify or friends there in the booth. Uh and also there are many other solutions out there like Firecrawl and so on. what about I want to scrape large scale? Not just under tens of pages. I want to scrape millions and billions of pages. If you

ask me why, that's required like for example, if you're building a rag or AI application where you want to train your AI model with the latest data, not with something with a 6-month data, uh which is outdated. And you just can't just train it with a one or two or tens of websites. You need a millions of fresh real-time data. How do you do it? one approach

like we already talked about it, right? Like asking LLMs, uh like asking our AI agent like Claude, "Hey, go and get it for me." Well, it's it's initially I felt it's a good idea, but the problem is um there are some limitations. My Claude can just go do one task at a time because it's also like human, right? Uh it has to open a browser and click

on those pages, uh navigate and get the data for me. It it does one task at a time. So, imagine I got 10,000 request pages, it it's going to take days. It's not it's not feasible. And also cost. Think about the cost. For each each task, it's going to take multiplied by a number of pages, it's millions. Think about the cost. It's going to scale there. And

other most important thing, it's also going to work with browser. The more requests it does, it's also going to experience the blocking. It's also going to feel the CAPTCHA. So, you have to solve that. So, what do I do generally in that case? We as a builder and developers, we start building our own way to solve this. The first approach is I start with a automation frameworks

like Selenium or Playwright. I I start with this. It works well for a 10 or even up to 50 pages. But I start getting issues. My IP is noticed. My IP is going to be blocked. Then I look for a a proxy proxy solution. Again, with proxy, I can do this rotations of IP so that I can be able to But still, there's one more issue. There's

going to be still going to be blocked by CAPTCHAs. Then I'm going to look for a CAPTCHA solvers. How can I do those? Solve by CAPTCHAs. Again, there's going to be one more issue, which is TLS finger fingerprint going to recognize our handshake even before the request. And I will also be trying to do some mimicking my mouse interactions. Basically, I'm implementing or simulating the behavior of

the human behavior. what I'm trying to do, my initial goal was to get a data, just to get a simple But now, I'm basically building a anti-bot infrastructure. But if question is, is that what we want to build? No. But think about you're having a database. I just want to query. I just want to get a data. Or you have a CDN. You just want to configure

in a Cloudflare or a Fastly and you just request and you get a data. And I also want to get that same thing. I just give a URL. Uh I just want to get the website data at a large scale. So, this is a this is a problem. And uh there's another problem. Even though if you build an infrastructure, you know, the website you want to you

can also break. Their DOM structure can change. With that change, it can like Again, you have to fix it. You have to fix it manually. You have to identify it. It can get even worse. Like if you use a framework like um Beautiful Soup, uh you just don't even get a data. You just get a null value. That's you're going to feed it to your AI models

or your data pipeline. Which is not good. Your downstream data Yeah. So, fundamentally what I wanted to trail is it's not a bug. It's the fundamentally how you're thinking about the architecture. Initially, what I showed you pre before, it's do-it-yourself as a developer. Uh we're trying to do it. And we also going to see that it's going to break. It's brittle because the selectors can DOM structures

can change frequently. And imagine you as a e-commerce or any other data provider you wanted to get is you have to If you want to add new source, it's going to take time. It's not going to happen instantly. You have to build it. You have to understand the DOM structure. It's going to take time. And since you know that this code is keep changing, you also don't

want to touch it. And with that with that, now I'm going to show you a demo. Hopefully, wish me a good luck because I want to show you real time. Uh pick a one uh e-commerce site from um uh from the Netherlands and scrape it um quickly. Um uh So, the idea is very simple. You just give me what you want the data and just give me

a URL and I'm going to give you a API that you can use it and put it in your Not just one website, you can actually scrape thousands, millions of pages. So, let's get started with that. Um So, for that, I'm going to I'm going to switch my squaring option. So, I'm just going to change my sharing option to be uh extended display. Oh, it's loading? Okay.

I can just show Okay, let's try to do. I have some issues with that. Okay. Uh Hope you can see that. Uh I'll try to do um the website I chose was like Coolblue.nl. I just spoke to one of the attendee, what is the most common uh e-commerce site here in uh Netherlands? So, let's go here and I'm let's going to pick some um Uh I'm just

randomly going to pick some Uh let's just pick some uh smartphone. Uh I just want to pick uh Uh I just need that one product page. So, my goal is now This is a website. I'm a e-commerce uh competitor and I want to extract a a data um like a pricing, uh name, some description about the product. So, let me just go and as I said in

the beginning I'm just going to give the prompt and a URL and I just going to go list to get an API so that I can scale it up to get thousands of pages So, I'm going to start scraping. Uh this tool is going to analyze the website which I shared you shared it now going to give me a schema that I can either approve or reject

it. So, let's see if uh if I take couple of uh seconds of analyzing it. And the goal is uh for me the goal is to get uh most important information in my the e-commerce site. I want to get some most likely those product name names, uh pricing, uh how much quantity or if it's sold or available, uh some description. All those details I'm interested. once I'm

able to get this, so it's I can see here it's trying to extract the uh in a in a few seconds. Uh okay, cool. You can see here I didn't I'm not coding here. I'm just gave the URL. It is intelligent enough to get me, "Hey, you need a product title, price, rating, review counts." Uh for this demo purpose, I'm just going to remove I just need

only the review counts for now. I just don't need this. Uh availability, I don't need this. I'm just going to remove them uh for now. Uh okay, description is okay. Brand, I Uh for that, let's keep this and approve. Now, it's [snorts] going to create me a a code or a collector that it's basically preparing me a API. That API I can call get any of those

thousands of pages of this cool blue e-commerce site instantly. Let me take it. My preparing its collector. So, cool. Wow. I'm impressed with this even with this Wi-Fi. It's working like a great and so let's try it out. It's asking me to And important, you see here I can keep on adding URLs which are shared, but I'm not going to for this for this demo I'm just

going to keep it one. Or I can also upload thousands of URLs a product URLs here and it can able to get get the data. So, let's start with start first and it's going to basically go and get Yes, it's ready and I can see it's on the I can download a JSON or CSV for this demo. Let's take a JSON one and let's save it to

my downloads. That's okay. And let's see if it looks. Yes, it works. I can see the pricing I asked for title, ratings. Cool. What's interesting is now what I what you show is just only the manually I showed you. What's interesting is you can now I can get an API, initiate by API. This is This was our goal. This was our goal like just prompt an URL,

get an API. Why API? You can completely automated. So, I'm going to just going to go and take this API code I'm just using a Postman collection just to demonstration purpose, but you can use any rest client. Of course, I'm going to delete my API key later. So, now I can also initiate this by calling an a API. Uh so, don't know why it's taking time. basically,

I can just put it here and get get get the make a call. Uh let me see why it's taking time. Okay, cool. And now you see here on the body, let me beautify this little bit. Uh you see here, I'm passing the URL which we which we took it from the cool >> [applause] >> Now, uh I just want to make sure my authorization is correct.

I'm just going to use uh a API key which I got it from there. Uh for this demo, I'm going to use my uh I'm just going to use my uh API key. And let's make a request. See if we can get the data quickly. Uh okay, we got a collection. Uh uh I can use this collection which basically ID. I can use this collection and uh

get a data. Uh for this, like let let me just change this value with this value. I'm going to Oops, I'm going to save this. I'm going to make a request. You see here, it says it's still building. It should be able to get it ready. Uh I can make a one more request again. Cool. You see here, now I got the data. That's great. I just

want to Okay, this is great and you can scale it up. Like what what do you pass is only a one set of um value. Uh one URL. But you can think about millions of pages. You can just scale it. You can just get a API instantly. Okay, that's the hardest part. That is part. Okay, I it's showing a demo. That's great. It works. But what is

the maintenance? What if something broken? Uh your scraper is broken, the DOM structure is changed, or you want to get a different data. Like initially, I didn't care about the ratings, but now I want a rating information. I can get it easily. Uh, for example, here I Okay, I have a rating. Uh, but um, like let's say I want to remove the ratings. I don't need it

for my AI application because I want to save the tokens. Uh, all I can do is I can just go and I can go to my uh, coding mode. Uh, I can go to my agents. I can just say, "Hey, remove ratings." Again, I'm not coding. I'm just saying it as It's going to start looking into it at DOM uh, the the schema and it's going to

remove and ask me, "Hey, is this what you want to do?" Or this I'm just showing you this one example, but even if your scraper is you can actually just ask, "Hey, can you go and fix the scraper?" It's going to fix it for you. And meanwhile it's running, I just want to also talk about it. Here, you get your code. It's not like it's a black

box. Uh, you see here? It is already showing me it is removing just removing the code it generated, and I I'm just for the demo purpose I'm just going to apply uh, accept this change, and also going to save this to production so that when I run an X scraper, it's I I will not be able to I'll not be able to get the uh, ratings information.

Let's see uh, quickly. okay. If I I I can quickly go and run it Uh, it's here. It's a You see it's a second version. and should be available shortly. Uh, let me just reload it. Yes, it's available. Inputs. uh, let me just change it. You see here? Now I'm going to download it again. And this shouldn't contain this shouldn't be containing my ratings. Sure. Uh so

let's me see if I can get ah okay. I'm just going to get my And you see here the ratings is missing Cool. And the last thing I want to show you is uh before I wrap up you got everything you need. You own the code. You can make changes. If you prefer you are a coder you can go and edit it and you can make changes.

It's up to you what you want to do it. And uh I want to switch back to the the presentation. What I showed you demo is just a one it's a one approach. Like your pipelines AI is going to write for themself. What you saw is a self-healing capability. And sites can be changed broken scrapers can be detected and patched. And again you want to scale it

up to get a data from 10 other sources? No problem. Just get those sources and scale it up in in a matter of few days. And fix any scraping issues or DOM basically the DOM or a broken scrapers instantly with a self-healing And more importantly we as a developer we want to have more control on what we What what's readable? What's not readable? So we have that

control. The versioning system is available. The coding is available. So I believe this is one of the way where we can get scalable web data in in way. Uh with that, I want to invite you What do you What if if if it's inspiring, if it's interesting for you, I want you to invite you to give it a try. Uh, just log in to uh sign up

to our platform, and you're also going to get a $50 uh credit uh with this uh QR code. And give it a try, and more importantly, if you want to building applications or you want to get a web data at a large scale, uh give it a try, and it can really enhance the way you're managing and building applications at a large scale. Uh with that, I

want to say thank you very much.