Fabien Vauchelles: Live-Coding: Mastering Anti-Ban & Web Scraping with Scrapoxy

Fabian Vushelle discusses web scraping and introduces Scrapoxy, an open-source proxy management tool designed for the web scraping industry. He provides insights on bypassing website protections, using proxies, and managing HTTP requests effectively. The presentation features a practical demonstration where a hypothetical student, Isabella, seeks to scrape data from Trekkyreview.com for creating travel recommendations. Fabian showcases how to overcome security measures including CAPTCHA and user-agent restrictions while utilizing frameworks like Scrapy and Playwright for data extraction. He emphasizes the importance of properly managing proxy types, including data center and residential proxies, to ensure successful web scraping.

Okay. Hello everyone again and this time it's turn for Fabian Vushelle. Did I did it? Yes, I did it. It was already the most difficult part of this really. Uh so Fabian is an antibban expert with over a decade of experience in web scraping. His passion for code and technology enables him to bypass various protections. He's the creator of Scrap Proxy, a major free and open-source proxy

uh waterfall tailored for the web scraping industry. And right now he will do a bit of life coding fine. So I really wish you good luck and I hope that the gods of life coding will be with you. So good luck man. Thank you very much. So, first of all, how many of you are doing some web scraping stuff? Yes, some of you. Uh, how many of

you just get a capture this morning? None. Yeah, some uh because we are in the proxy country. That's one of the biggest providers of proxy here. So, we had quite a lot of talk about capture today. So, that's nice to talk about web scrapings. So I'm Fabian. Uh I'm doing web scrapping for many times. I'm also the creator of Scrapoxy. So what's Scraposy? It's free and open

source. If you want to start in web scrapping, it will help you to manage all your proxy. So it's able to manage and root proxy um your traffic everything through cloud providers and proxy services. you can handle cloud provider like GCP whatever proxy services like IP royale which is a literian one by the way um and we will see that later but before we enter to our

discussion I'd like to share with you a little story so enter Isabella so Isabella is a finite year student in IT school she has a lot of energy and a source for travelings Every years she embarks to a random country for a one month's backpacking journeys. But here is the twist. To prepare this one month journey, it took her an entire year of preparations and she couldn't

help but notice there is a gap in the market. Why in a digital era pump with LLM AI everything there is no such a tool doing that so perhaps it's time for her to create kind of business but she realized to create this kind of business she need a lot of data yes this data will be used to train a large language model to create the ultimate

trip for that but Isabella knows she must be take care about the legal aspects. She understands she can't do too much request to the website. She can't overwhelm the website. She also understand she can't download personal data like emails, names, this kind of stuff. She only collect public data. Also, she's not signing any terment conditions. So, she's not logging. So she is free from contract and because

she can go web scraping. Now I would like to introduce you the website Isabella choose to scrape which is trekkyreview.com. So let's have a look on track review. So what's Treky review all about? So Track Review is your go-to place for searching any accommodations in the city you want. So don't try venues only Paris work. If I'm clicking on search I've got 50 result here and I've

got a lot of accommodations. What I can do is to click on one accommodations and I will get the names location contact phone description whatevers and I will get also some review. Isabella is interested about review because if you analyze a review perhaps checking all the bad review you can extract what are the main issue of the authors. That's clearly what I'm doing with Airbnb checking every

review and extracting the bad one to find oh I shouldn't get to place. So let's get back to this website. Oh, I forgot this website is super secured. We have the same website in different versions. Here there is a levels and we have the same website with different protections and Isabella we try to bypass all these protection during this presentations. So let's have a look of the

website. I will open the inspector here on Chrome going to network request. I will uncheck preserve log filter on dock and reload. So when we start we are on the homepage on the level one. If I'm clicking on Paris here I've got the list of 50 accommodations and if I'm clicking here I can get all information inside the response of the request. Here you can view the

HTML. So perhaps there is information inside. If I'm looking at the bottom, I will get uh the hotel description. That's nice. But I want to collect this review comment. So Isabella want to collect this part. Of course, she doesn't want to write everything from scratch because it's HTML. She need a parser. So she doesn't want to create that. She needs also to make all the request. Sometime

when you're doing web scrapping request fail or you need a retry strategy you need to manage also concurrency that's why she will use a frameworks and these frameworks is scrappy who knows scraping some of you yes some of you that's cool so it's a 10 year of this framework this year um it's an open-source framework in Pythons uh there is a lot of add-ons extensions it manage

all the parsing of expas CSS reg x you don't like to write reg x that's why they're handling that they manage request cookie concurrency whatever most of the company doing crawling scraping at scale use scrapey and scrapy run spiders so basically it's a python class you give it a cool name like tricky spider review um you gave it also a name here and you've got methods. First

one is used to define the URL that you want to collect and when the engine get the response you can parse the content. Here we have a CSS extractor. We get the names and we can also start further request and doing other parsings. So let's have a look to a real spider. So same spider here you've got you've got uh the start request when you get all

the information here we are going on the different page of the pageation you know a website at different page pagination is like flipping the page of a catalog this kind of stuff when you got the list of items you parse each hotel links when you get the hotel informations you extract information with magic CSS parsers that's cool we get the email, name, review, whatevers. So now what

we will do is to run these spiders. So you can see I'm connecting very quickly to the website, no protections. And I've got here 50 items. Nice. And also the beauty of scrapey is it's cracking in a CSV form or whatever format you want. And you can have this kind of data used for your machine learning algorithm, connecting to an API, connecting to an MCP server, whatever

you want. So that's perfect. But in the real life, it doesn't seems like that when Isabella is trying to go to a serious website. So let's move to the next level. So level two. And she when she start the spider what's happening is a lot of error. First one is unknown browsers. Yeah. It's because when you're connecting to a website uh your clients which is Python client

which is a JavaSc uh client a browser like Firefox Chrome whatever they are sending a lot of informations. So let's have a look of what Chrome are sending. So Chrome here if we are looking to the headers we've got the request headers and Chrome are sending a bunch of errors here especially what we called the user agent at the bottom so we are declaring that we are

a modular on Linux and we are using Chrome that's perfect also Chrome add because it's Chrome additional security header this kind of stuff so to confirm this information so if you're writing that you're a user agent on Chrome and you're not adding this kind of information, you will be rejected. And of course, by default, scrappy send, hey, I'm scrappy. Please ban me this kind of stuff. Of

course, the website won't let you in. So, we have to modify this kind of informations. So, let's have some modification of the parameters. So, I will open the defa headers. I will add the user agents. Oh, I am on Windows. That's perfect. I'm on Linux, by the way. So, it's completing with Copilot. Copilot would save your life for that. And I've got some consistent eders here. So,

if I'm started that again, if I'm running the spider, this time I can bypass the protection and I've got the 50 items. So that's the most simple protection that you can find on internet. You sometime have to change the user agents. If you're doing randomly, don't forget to add additional letters. But my advice is use the latest one with the common signature because everyone is using these

signatures and don't randomize it. But clearly that's not what happening when you're connecting to a serious website. If you're starting the crawlings, what will happen? You will get few requests here. But at the end, you've got a lot of retry, retry, retry, and that will fail because we're doing too many requests. Do you know why? Because I'm using my laptop. So, my single IP address from the

IP address from this Wi-Fi. And from this IP address, I'm sending a lot of requests and the server is just red limiting me. So I need to do something. You see, I just collected 12 items. So what should I do? Any idea? Change the IP with proxy. Yes. So what is a proxy? A proxy is a system which can relay your request from your spider to the

website. So instead for the server instead of seeing one IP address sending thousand of request you will see multiple IP address sending a few request and there is multiple type of proxy. First one very simple what we call data center proxy. So basically you go and address you install a squid proxy and you get a fresh IP. You can do that on whatever cloud providers. But the

thing is it's easily detected because an IP address of center or an IP address of whatever machine you have is bind to an IP range. And when provider like Amazon is buying IP range, they pay that for a lot of money. There is not so much IPv4. They can use IPv6 too. But they are tied to what we call an autonomous system number. And there is database

for that. You can find free or paying database for that which match the IP address and link that to the autonomous system number. So I know that using this kind of IP range is Amazon or GCP whatevers. So I need to trick that a little bit. What I will do first is to use what we call ISP proxy. So ISP proxy stand for internet service provider proxy.

So how does it work? Very easy. Imagine that you are a proxy services and you want this kind of IP. So you will call a mobile providers, an internet box provider and you will say okay please rent me some IP range because they have a lot of unused IP range every times and they will rent you for that of course for a lot of money and you

will bind your server with this IP. You can do that on Amazon whatever. You can bring your own IP address. It's possible. You're not forced to use the IP address from address and you will be mixed with other user like mobile internet box etc. and you will pass but still it can be detected by antibot because you're renting a full IP range. So what we will use

today is resentials. Who use residential? No one. Okay, let me reformulate. Who have installed a free application on his mobile phone? Yeah, you can. Everyone's stance. So basically imagine that you are of developers for getting some money for apps because it's never free. Nothing is free in this world. You can sell subscriptions. If you don't want to fail subscription to unlock some features, you can put some

advertising like a video to watch and after it unlocks some features. But there is a last thing that you can do is to sell the bandwidth of the user. Yeah, you can do that. There is a lot of FDK we can do that of course with your agreement which is hidden in the bottom line of the term and So now that said we will try to use

some proxy but if we are doing seriously proxying we can't use single provider we need to use multiple provider we can't rely just on only one what we will do first is to install script proxy or infrastructure to manage all our proxy strategy so let me give you an example imagine that you want to collect amaddio data um you're connecting directly on their website and you're just

blocked because clearly they have a good antibot systems and based on your IP address whatever they will detect you. So what you will do first is to install scrapoxy. Scrapboxy is also able to manage at desk browser. So that's the cool part and capoxy will connect to every cut provider API SDK ask for some instance or allocation IP address and all the traffic will be root through

that and that's the cool part you will get a lot lot of IP address uh with that so now what we can do is to start scrapboxy so I will just start by docker file here so it's just docker on my computer. You can insert that also in tub test and I will connect to my local host here. I've got already one project. I can click on

that and I don't have any uh provider. You can add whatever the provider you want. There is multiple ones. It's just a list of connector. If you want to add address, you just have to create one with your credential, you just add that one and it will handle all the management orchestrations. So I already add some address credentials and I will start 10 machine just now. So

I'm starting the connector. If I'm going to my AWS account, you see I don't have any instance. But in the few seconds you will see that instance are coming. So incoming. Perfect. I've got 10 instances and it's really really fast with a T4G nano. It's already up and ready. And Scrapoxy will install all the proxy inside. And now I can go on Scrapoxy check my proxy. You

see I've got the status a lot of informations. uh request number of data sent and received the IP address and I will see in a few seconds a lot of new IP address which are located by the way in the Dublin data center so I can confirm with this map. Perfect. So now I need to plug my spider to Scrapoxy to go to an address and collect

data from the website. So what I will do I will use these credentials and get back to my spiders. So let's just copy paste a small code I will show and explain you. So on the custom setting I will add this add-ons and some parameters of my local instance. And now what I can do is to spider. And now I'm not red limited because if I'm going

back here on the proxy you can see that I'm just sending some request with trapoxy. So I'm going to island and getting all the data. And of course here I will get my 50 items. So now Isabella is very happy. But sometimes see she's trying to have some issue. So let's me show you that. So if she moving to the level four she running again the spiders

she is directly blocked because the data center IP are just detected as I told you this data centers is detected because it's written Amazon 02 so I can't use this kind of IP address so I need to switch to more advanced kind of proxy what I will do. So I will use a Lithuanian one I already told you. So IP Royal they provide a lot of proxy

and I will configure that in Scrapoxy. So first thing I will stop scrapoxy I don't want to pay for my instance and that's also the beauty of Scrapoxy is just warming up and down the proxy only during your scraping session. So you reduce your cost by 80% using a free tool. Um I will add a connectors here with IP royal credential already configured and what I will

do I will use on United State ones. Okay, perfect. Let's start them and just wait a few seconds. So you can see that scrapboxy is stopping the instance is everything are stopping and I won't pay for that. So now I've got nearly ready 10 allocated IP on IP real. Let's have a look. It take a little bit times and I can check if everything are up and

ready. Okay, I've got some IP there. Perfect. Uh this one. Yeah, I should get rid of this. Sometime it happens. and automatically it will re rotate. Also, you can configure your spiders to automatically uh remove an IP. It will be replaced immediately. So, I've got everything in the US. What I can do is to start again the spider. So, I didn't touch anything on the spiders, just

added a new providers. So, you can see now I'm not blocked. I'm passing everything through the US. It's a little bit slower because I'm doing a round trip, but I've got most of the items. So, 50 items. That's perfect. So, that's nice. So, that's what you can do with proxy. So, now let's imagine something more complex. If I'm jumping directly to the level six running that again

I'm connecting to the website but got harder issue which is no fingerprint. Yeah. So what's the fingerprint is? So let's have a look deeper on the website. So now I will go on level six. What I will do is remove the filtering refreshing. So you can see here a lot of request. So we're doing getting some HTML stylesheet images. We're downloading a lot of information for rendering

cool execution on the websites. So a lot of get request but the thing is I'm also have post request and not only one I think I will add very soon anothers and on this post request I can check what are my sonic so I'm not only downloading information from the website I'm uploading informations and if I have a look I can check the payload and what am

I sending I'm sending the real platform from the real-time zone and the user agents. So I'm executing JavaScript. The website asks me to execute JavaScript to gather information and to send that to the web server. So if I'm not executing JavaScript, I can't send this information and relying only on scrappy now doesn't work. So I need to change the way of getting the data. So that's why

I will use another framework combined with scrapey because I love scrappy which is playright. Who knows playright some of you? So playright is used for end to end testing but perhaps you know it it's mostly used for web scrapings. So what's pray all about? play right it's free and open source maintained by Microsoft and it can start a browser Firefox Chrome whatever execute some JavaScript so it

will handle all the logic of browser executions so how can I set up that in my scrappy spiders so let's get back to my spiders here so I will open another file which is this one which is adapted for playright basically it's the M spider you can see I'm sending the set request we are getting all the city the all the page on the city the hotel

link we are passing that when we get the hotel information we're extracting name email review and having we'll have at the end structured informations the only difference is the way of getting the data so I'm not using the basic endler download endler of scrapy I'm asking to play right to open the browser here and to handle the logic. So if I'm starting that this file playright. So

now I've got scrapey asking to play right to open many browser. You can see connecting to homepage downloading the CSS JavaScript whatevers sending the payload. I'm granted and now I can quickly go on each hotel page and you can see it's very fast. So for the demo you see the browser but of course you can write that in a headless mode in a container docker containers and

you will get all informations. It's a little bit slower uh than doing direct request but for most antibot solutions that can bypass them the simple one of course. So if we have a look we should be close to the 50 items. Yeah, here. So, it's not exactly 50 because I didn't correctly implement in retry mechanisms. So, you should, but you get bypassed the protections. So, let's have

a look at something quite more difficult. So, let's move to level seven and run it again. So, what we will have So, now Scrappy is asking to play to open again the spiders. Playright will open 10 browsers connecting to the website getting the HTML, CSS, JavaScript, whatevers, sending the payload. But you see I've got some errors here. I've got an error which is unconsistent time zones. Yeah,

that's because they are not only checking the JavaScript executions, they're also checking what you're sending. So the payload because we are using US IP address. So there is a time zone from this IP and we are using also the browser time zone. So we have America for IP and VUS for the browsers. So two choice I can change perhaps the location of the IP get a European

ones but that's things I don't want to do because perhaps the website is blocking blocking me based on geol locations sometimes if you go to Brazilian website you get blocked so you need Brazilian IP address and on other countries that's the same Japan it's too um so I need to change the setting of play rights and of course you can do that. So what I can do

here is to force the time zone and we will put I'm listening but I will America. Yeah, let's do that. So if I'm again now, so Chrome will open. We'll go to the homepage. downloading every information, executing the JavaScript, collecting the fingerprint, sending the fingerprint with this time the correct time zone of the browser, aligning the the time zone with the IP address. And you can see

I now going to the different hotels. It's very quick. Our goal is not to last on hotels. Also, I'm not rendering the image. You can say, okay, on this page, I don't want to rendering the image. So you save a lot of bandwage especially if you're using residential proxy because you pay by gigabyte. So you don't want to download useless images only those within cash and at

the end you will get all information. So you can see I just bypass the protections. Okay. So that is a generic way to go on website and execute JavaScript and try to bypass some protections. But sometime when you're going to commercial website, you will have more advanced protections. So let me show you. I will move to another level which is level eight. So let's have a look

on the different request here. So of course I'm downloading on this level a lot of information. So you know them but this times I've got a post request here which is not sent by FP. It's something random. Okay. So let's have a look to this payload. And we've got an encrypted payload. I can tell you it's not a 64. If you're decompiling with B 64 you will

get the following. So this one, so it's encrypted. Of course, if the payload is encrypted, it's very difficult perhaps to mimic the post information because something I want to do perhaps is not to start playright, which can be very slow. If you want to do million of requests in one hour, you perhaps you want to emulate the antibot payload and send that. But to emulate the antibot

payload, you need to understand what is inside the payload and how to emulate it and fake it. So what we can do is to go back on the network. We have this payload. We can go on initiators and see the script. We sending the payload. So basically it's a oneliner like this. And if you script it this JavaScript yeah so everyone can understand that. Yeah you can

do yeah of course of course not you can't it's obfuscated and what we need to do is to deiscate the code. So here are some techniques of the obiscations. So let's take this kind of stuff that you have here. What you can do in the office? Imagine you have this kind of code. The first technique is to replace what you can do and to decipher the text.

You cannot understand the text on top. But what you can do is to use this functions which is used here and decipher directly the text. So let's do that. It's called string concealing. That's the operation of opiscations. So now we've got barely the text in a mode. Of course you can see there is a lot of constants and the con are used here. So what we will

do we will replace the constant by the value. So it's called constant unfoldings here. We've got of course something more readables. I will join the string. Okay. And I will replace the uh string notation by the dot notations. And we know now that the antibbot is looking about the screenwrit ugly language. So that's a fun part. And I only show you what you can do with strings.

You can do a lot of bad things with uh code. So you can have a lot of techniques. So let's do that now. Um if I'm going back to the code, what I will do is to copy this code and to put that in this file. Perfect. So this is the code and I worked a little bit before uh this presentation. I wrote the obfiscator specifically for

this payload. So I'm using bevel bevel is a transpilers and I'm doing this kind of operation I just explained you. So constant unfoldings I'm doing also string joints whatever converting string notation to dot notations. So you see I'm chaining the operations. So now if spiders, not the spiders, the deop offiscators. So okay, it's quite fast today. So let's open the deop offiscated source code. Okay. So I've

got now the deopiscated source code. So as you can see, we still have some creepy name on the function. Yes, we cannot find these names because when you're doing obfiscation, you're losing part of the informations, but we can still understand what the code is doing. So my advice, take this code, put it in CL code will just get the correct names. That's work like a charm. So

now let's try to understand what the code is doing. So first we have this function. I don't know what is it. We have this one. Okay, there is RSS stuff. So, we're doing encryptions. So, if we are speaking about encryption, there is a key. This key is here. Perfect. Next functions. Um, okay. We have a post method with content type. We're managing error. So, this is a

function sending the payload. Okay. So, we need to find what we are sending and where the payload is built. So, let's have a look on the next one. Oh, we are gathering some WebGL informations, renderer information. Oh, yeah, that's it. So, we are collecting the vunder and the renderer information from the GPU models. So, we have to fake this informations and after the everything is changed. This

is the main function sending the build the payload. So now what we will do is to mimic the same behavior and to collect the informations. So I will open another spiders this one. So this is the first spider same one you have here the tricky spiders. You can see there is the start request. We are also passing the different pages. We are getting the hotel links. And

when we get the hotel information, we are tracking name, email and review. Same spiders. But the only difference is what we do first. First we are going to the homepage and we are sending a payload. So a post request to this URL like the website the browser and we are building the payload here. So I can go here. So this function is building the payload. I need

to replace that to create the correct payload and I need a public key. So let's first do this. We were speaking about renderer vors. Let's say it's intel on these computers. Okay. I need the key now. So the key were at the top. I will copy past the encryption key which is a public key and I will get back to my here and put the public key.

Perfect. And now if I'm running the spider this one I'm sending the key and now I've got all the request and you see no user play right and I've got the 50 items. So the question is why should I deopuscate or use playrite? It's a matter of time and money. Perhaps it will take three months to deopiscate the source code of the antibbot systems. Perhaps not. Perhaps

it's more costly to use I don't know thousand of VM with Chrome inside. You don't know. Okay. So that's all for me. Um, download Scrapoxy, add a GitHub star. It will help a lot this opensource project. And I've got a lot of cool sticker if you'd like. So, don't forget to get this one. And perhaps I can get three four questions. Yes. So, yes, we even have

uh already one. Uh, but maybe first I will ask if is there someone in the audience who have some questions? Oh, don't be shy. Okay. So we start with someone who is anonymous. Uh if we are trying to scrape a website that has location specific content, is there any other option than buying proxy from that country? No. So next uh yeah and sadly that's no there is

VPN but oh yeah VPN proxy but to be honest all VPN address are known. Don't use that. So, so, so the moment we are just uh trying to scrape a website that knows what it's doing when it comes to preventing that uh the VPN won't work, right? Yeah. Yeah. Because on the database uh you can have this kind of website IP info and they will tell you

okay this is a to this is a VPN outbound IP address. Um so you will get this information on the website. you pay this informations because there is subscription but that's something which is flagged really quickly and we have another one how do you scrape cloudflare ah that's a good question so if you want to collect data from cloudflare um I'm speaking not the enterprise version but

the free versions you can uh let's have a check on this website this solutions. So this is a patch Firefox. The guy who creates that is amazing. So it use the letters patch to anonymize Firefox and you can mimic uh the fingerprint. You can mimic a lot of stuff including WebGL information, WebRTC. So this is something you're leaking the information normally. So you can set the proxy

address with that. That's one of the solution to bypass cloud right. So it hless play right but that works. Okay. And we we have another one. How and why did you start your journey and scrape and scraping and scrap uh at scraper screw scraper? Yeah, I was collecting websites eight years ago and I had a lot of issue with IP address and you know at this time

I I was very poor so I can't afford any proxy proxy services were very expensive now it's quite affordable but at this time it was very tricky to get a lot of proxy the only solution to start stop machine I created some script to do that do after I did that on Azure other provider users start to use cropoxy uh and now I've included a lot of

proxy services. So it's helping a lot of company in the world and we have one more. So how do you scrape Google to get thousands of websites of a specific type? Yeah. uh so what you're looking for for it's uh what we call search SER APIs so there is specific provider who provide you API for SER uh you can use this service because they are targeting especially

uh search engine like Google's so this kind of SER API can help you on that but it can be very tricky to do that at scale having perhaps I don't know 10 requests is okay but can be difficult if you want to scale Google search this kind of stuff. Okay. And one more. How many websites are usually on level eight and it requires the office? Yeah. Um

a lot. Um because when you're collecting website it depend of your business some business collecting various website. So to give you uh how the web is split, you have what we call the longtail which are most of the website uh website but usually not script and you have a small part which are social network uh e-commerce travel uh website and everyone wants the data from this website.

So they are really well protected and they have this kind of um protection. Of course you can find guys on internet selling services uh to do the deopiscations that's how company are working and there are still more how to bypass captures uh to bypass capture there is uh this cat and moose game of course uh you have AI capture solver which are really fast but I saw

a talk just before which has amazing about how to detect AI capture soldier. So we will see the next step. Okay. And I think that's all. No one from the audience. Okay. Cool. Thank you. So thank you so much. Huge applause.