Fabien Vauchelles: Mastering Web Scraping: Unleash Your Data Extraction Wizardry!

In this talk, Fabian Vell shares his expertise in web scraping, specifically discussing the challenges of avoiding bans from websites during data collection. He introduces his open-source tool, Scrapy Oxy, which is designed to manage proxy routing effectively through various cloud providers. Vell demonstrates the process of using the Scrapy framework for web scraping, including modifying user agents and managing requests to prevent detection by anti-bot measures. He elaborates on different proxy types such as data center proxies, residential proxies, and mobile proxies, highlighting their advantages and use cases. Further, he discusses advanced scraping techniques, including handling JavaScript-rendered content with Playwright. The session culminates with practical insights for effective scraping while emphasizing ethical data collection practices.

[Music] hello hello we are at day of the dev days devops Pro cyber wise conference and my name isina vasila I'll be hosting the data track today and our first speaker is Fabian Vell uh who is an expert at not getting banned while scraping data and he's going to show you how to become an expert just like him uh by also using a tool that he has

created an open source open source called scrp oxy over to you Fabian thank you very much deina that's a very nice opportunity to speak at this conference so um my name is Fabian relle I've been deeply passionate about web scrapping for years my enthusias let me to explore this fascinating world of proxy and antibot systems I work at wmine an amazing company specializing in Revenue management within

the transportation industry our work at w mind involve handling million of prices on a daily basis which requires substantial investment in web scrapping Technologies and our setup for data collection is pretty big mainly built on open source and also they use scrap boxy so I'm the creator of scrap oxy scrap oxy is a free and open-source proxy waterfall proxy aggregators it allow you to manage and rout

traffic through cloud provider and proxy Services it support major Cloud providers such as AWS a gcp or digital oceans whatever cloud and also it supports proxy services like bri Rite IP ra and many others since I started the version 4 in 2024 over 200 users have installed boxy they exchange 70 terab of data re 1 billion request and use 1 million proxy but before we dive into

our discussion I'd like to share with you a little story Hunter Isabella Isabella is a final year student at I school she has a Briant mind a lot of energy and thir for traveling every year she unbs for the one month backpacking journey to a random country but here is a Twist this level of preparation consumes her entire year in preparation for just one month of plannings

of travels Isabella couldn't help but notice there is a gap in the market why wasn't there such a tool in a digital era pumped with AI this could be a ticket to a successful business she realized she needs vast amount of data to create such tool this vast amount of data would train her large language model to create her ultimate trip however Isabella is very careful in

her approach to business before she starts strapping data she makes sure to consider all the legal aspects she knows it's important not to overwhelm the website by making too many requests too quickly she also respects privacy she only collects information that is already public like reviews and doesn't take any personal details like names she doesn't sign the website terms and condition teers she is free from contract

now that everything is clear she's ready to collect the data so let me introduce you the website she choose to scrape try review.com so I will connect to try review so what's try review all about Trey review is you go to place where to find any accommodation in the city that you want visit imagine that Isabella is looking for accommodation in Paris she just have to click

on the search button here take a little bit time yeah and she will find 50 accommodations she if she clicks on one accommodations she will get all information related to the accommodations like the name the description the locations also the review Isabella is interested in reviews it is all about analyzing those review to understand what the customer are thinking about the hotel if we are using large

language Motel to query this review we can extract the main feeling about the hotels so we need to gather this part so now let's understand how the website is structured let me open the inspector in Chrome so I will inspect the pages go to the net work I will go back here just clean a little bit Yeah so I will preserve log filter only on htmls and

data so if I'm on the homepage here so I'm on the homepage I on this URL if I'm looking for Paris accommodations here I will get another request here so I can find all accommodation here on Paris if I'm clicking on one accommodations yeah let me get back yeah I've got all the informations and the request is here I did that twice and if I'm clicking on

the response here I've got all the informations from the accommodations it's a little bit slow but I've got all informations like the review Etc yes I've got the review here and that's what Isabella want to extract but the thing is Isabel uh newcomer to web scraping so she doesn't want also to handle all the infrastructure stuff logic stuff so um sending the request doing the passsing also

managing concurent request and doing all the retry so she will use a framework for that let me introduce you the Scrapy framework so the Scrapy framework is a very popular framework in pythons used everywhere Scrappy can handle to of request passing of responses the retri the cook cookie management the concurrency with requests delay between work requests and also it's open source and maintained by the large community

from over 10 years and perhaps more so let me show you how Scrapy is working Scrapy use spiders so let's break down how Scrapy spider work so we start with a python class and give it a cool name so here we have try view spiders next up we've got our initial method start request where you add the URL you want to scrap then in the P method

we use a special tools called CSS select or to extract the information from a DF tag and that's it so now let's open a read spider it's here yeah so I just wrote the same spider let me show you so I've got the spider here the start request I'm going to the homepage when I've got the homepage I'm searching for the list of accommodations or sending this

requests so when I've got the response of the list of Hotel I'm passing also the pagination it's like flipping the page of a catalog and also when I've got hotel link I'm following the hotel link and when I've got the response of the hotel link I can extract all information like here name emails review so let's run this spider here as you see the spider run very

quickly and I've got my 50 items I can check that also that it's correctly stored it's extracted in the CSV file and if I'm opening that you can see that I've got the name the email the review we can extract more of course but the thing I didn't tell you is that the website also is super secure I introduced different challenges and Isabella Will bypass the different

challenges during these presentations so now let's switch to the next levels here you can see there is different levels so I will move to the level two it's the same website but with added protections if I'm running again this spiler this time I've got a big error which is unknown browsers yes there are some things that you should know when you are connecting to website with your

browser Chrome Firefox browser you sending a lot of information to the server prior to collect the response you sending a lot of HTTP headers and in this HTTP headers there is informations that said who are you so let me show you that if I'm going back to the adders here you can see the request adders I'm sending a lot of request adder and if I going to

the end I've got to user agent add so I'm saying on Chrome that I'm Chrome working on Linux this is my Chrome versions so the web server is aware of the type of of clients and if I'm getting back to my spiders by default the user agent is creepy so the website know that it's not a real user agent a real human so he will block me

me what I can do is to change the user agents so if I'm going here on custom setting I can change the agent so thank you copilot to helping me so now I'm on Windows and Chrome on Windows perfect let's run that so now I've got another error so I already solved the user agent issue the thing is you cannot now in a modern world in 2020

for S only the user agents because there is some consistent also adders as you see I've got the user agent informations but I've got also this kind of head xus X mobile and platforms so they are consistent with the user agents and if it's in consistent or inexistence the website will reject me so let's add headers so if I'm adding this header default request say CH user

agent mobile than F platforms and running that again I can try and I've got now perfectly my 50 items sorry it's here yeah you see the request and I've got my 50 itm so I bypass anti protections on these basic protections let's jump to the next protections on the level three so if I'm running this time the spider I can retrieve only 20 items so not the

50 unfortunately and I've got a lot of Errors I've got too many requests Al so again and again and again it is because I'm sending all these request from the same laptop the same server the same IP address so I need to add other IP address to bypass the red limit which is obviously by so I need to introduce to the concept proxy so what is a

proxy a proxy is a system running on the internet it relays request to a server and the server believes that the request is coming from the proxy from the from the client not the real source so instead of seeing one IP address sending you million of request you will see a lot of IP address sending a few request and you will pass and of course there is

plenty type of proxy the first type is data center proxy this kind of proxy run on a Aur gcp data centers this is the first serious proxy that you can find on the internet they are fast cheap and reliables however they can be easily identified by antibot solutions to explain you the IP address of the proxy is inside an IP range and the IP range is associated

with an autonomous system number ASN and the name of the autonomous system number can be Amazon Microsoft whatever so it can systems but there is a trick to get around it and this is called internet service provider IP proxy so let's talk about IP proxy and how work these proxy are set up in data centers but they don't use the IP addresses from those data centers instead

they rent IP addresses from a clean autonomous system number like mobile carrier or Internet Box such as verone or Comcast and they will get a bunch of IP addresses the proxy will use just one of them so this Pro uh this mean when you're using the proxy your activity get mixed with all the mobile IP address and keeping new ens between the internet box or mobile carriers

and there is the last type of proxy residential proxy so the IP address come from a real device which can be a laptop or a mobile phones so how does it work when a developer want to earn money from his application he has three solutions first he can sell subscriptions like monthly or annual subscription which unlocks features second he can add advertisings like having an ad to

the bottom of the applications or a video to watch before unlocking features every video games of that and third he can share the bandwidth of the device of course with agreement that's where the residential come from this type of proxy is very powerful because the IP address can be the same IP from a real user and there are million of endpoints available so now now we will

use scrap proxy the super proxy aggregator to manage our proxy strategy so let me switch to a terminal here and start scoy to start scoy just write a Tok light and it will run in a second so it's already up and ready so chrome interface here I can connect to UI so my basic password yes and I already got one project ready if I'm clicking on project

I can add a lot of connectors so there is data center connectors there is IP Pro provid there is residential provider there is Hardware providers you can choose one existing Provider from the market place and if I want to add an NE um connectors it's very easy you just drop your credential here and scrap oy will enter everything installations St and stop of the VM and also

all the the routing also all the traffic is encrypted so it's super secure so if I'm moving to connectors you will see that I've already added one aess connectors and I can start it so in a second it's very very quick you will see that scrap oxy just start 10 instances on ad so Scrapy is connecting to the SDK of AD doing all the instant Creations installation

also and install the softwares you can see the name of the instance it will be the same on your UI of EDS you have some metrics also and you have the real IP address you see that everything is B in based in irand I can confere that with the coverage map so if I'm clicking on this part you can see that every Pro here are B in

delins so that's perfect and also I've got some Metric um on Scrapy so traffic send valid request invalid request is also analyzing traffics so let's now integrate scoy into our spiders so if I'm switching back to my spider here I can add that may we just copy paste some credential yeah let me show you how it's working so on the settings if I'm adding a middle rare

so it's this one and also some credential here now the spider is ready to connect scoy so I didn't modify all the part of settings I can run now the spiders and you will see that I'm not blocked anymore and I've got my 50 items if I'm going back to scop boxy you can see that I send uh some request in SCP boxy so all requests are

correctly sent to every proxy and I can also check the success rate so I've got 100% success rate perfect so let's get back uh to The Spider and increase the security level website if I'm moving to level four and now I've have got some error and the error is a very classic one the data center are forbidden as I explained you uh using Data Center IP address

can be very detected because you he here's the ASN Amazon is blacklisted so I need to use a more advanced proxy so let's use a commer one so if I'm showing you uh Bri data connections so Bri data is very famous proxy providers they provide ISP proxy mobile proxy and of course resal proxy they have also product I already add some credential of Bri data so let

me get back to scxy ah just something to show you so the request before moving to a rest here was block so you see the SE s is lower so now I will add the bright connectors connectors I will stop a bre I don't want to pay for my instances so scoy will shut down the instances I can now add a connector here so I create one

connector here add 10 proxy using the US Zone and just start the proxy if I'm going to the list of proxy you can see that uh scxy is stopping the a instances and starting the Bri data IP proxy I can checka on the coverage and you will see a lot of proxy in the US so that can be good for you SCS so let's get back to

our spider as you see I didn't touch anything on the spider spiders it just very click and configurations on scoy if I'm starting again the now all the requests are working and at the end I will get all my 50 items that's perfect I bypass these protections okay let's move to the next level so level five so now now we're starting to get some more tricky stuff

so if I'm running now a spider I direct got one figure which is too many IP for a sessions yes so SC uh the website are monitoring that you are not doing the request on the Run Robin way so if you are using the same session so one you connect to the homepage add some cookie on to track your sessions so that you are the same user

if you are navigating on the different pages clearly you should not use different IP it's not normal for the user you will stay on the same IP for this session so this cookie so clearly I need to do the same on my spider so doing some request perhaps 10 20 requests on the same IP address after doing all requests on the another IP address and so on

so I can change that in the spider first so I just modify an another spider so it's quite the same spiders but what I will do first is every 10 request I will collect a new here so if I'm running that here perhaps it will work working let's see ah no I have to modify very important stuff I have to go to scrap oxy and say on

the settings and you know the very insightful functions which is SK the same proxy with cookie injections so if I'm using these options now scrap oxy will intercept all the requests in inject a cookie and when I'm using always this cookie I'm using the same output proxy and you can see here in the connector we'll see in the in a minute so we have 7666 here okay

let's go back to our spiders and now if I'm showing you that you can see that some of them will increase a little bit oh I did too much request on this one yeah perhaps oh yes I started the the wrong one yeah so it's this one that's perfect yeah I send all the request here and you see stoy monitor the success R it was not cool

so here I did a sessions here also here also here also so I didn't send all the requests on every proxy at the same times I use the proxy one by on and if I'm coming back here I've got my items it's not working perfectly because sometime I get bned and I need to introduce some retry mechanisms that I didn't but clearly I'm close to the 50

items so that's very useful functions when also you are using some browser headless browsers so let's move to the next levels so now if I'm using U on this one the level six and I starting the spiders I've got one blocking error which is no fingerprint now we are using more advanced antibot systems so let me explain to you what is a fingerprint so if I'm going

back to the website try reviews and going to the network aner not filtering on anything here if I'm going to the homepage and going this time to level six okay that's perfect so you will see that now on the this levels I added protections there is a lot of get request but sometimes there is a post request and another post request here and clearly there is a

lot of post request that will happen another Etc so let's dive into the post request so if I'm checking the content of the post request you can understand that we are sending information to the server at regular intervals we sending the platforms which is Linux the time zone EUR Paris and the real user agents so if you are not correctly faking the user agents it will fail

if the consistency is checked so now the website is executing JavaScript and with the JavaScript it's collecting information on the browser and sending that at regular interval to the web server so it's a fingerprint most of the time time this payload is encrypted and the code is OB fiscated but for the demo it's not OB fiscated and you can read that so now you understand that we

cannot stay with the basic spiders we need a real browser to execute the JavaScript part so we need some framework which is controlled by Scrapy so we can correctly write a spiders and execute JavaScript request collect CSS and whatevers so let me you this framework which is playright so playright is headless browsers but for the demo we will make it head FS it can execute JavaScript it

work with chrome Firefox Edge and Safar and the beauty of that it's open source and maintained by Microsoft so let's see how we can adapt our spider to use that so if I'm opening this time play right so it's quite the same spiders so here I just changed some information but I've got the homepage request the different passing of the hotel Etc and at the end I

can EXT clearly the name so I didn't modify a lot of things to a play right I just modify the download header so instead of using basic python request we will use uh a request coming from the browsers so it's specify here so let's run that on the level six so I will use the play right part yeah so now Scrappy is ordering to the browser to

open so it's connecting to the homepage sending the and as you see after the fingerprint is sent I can do all the request and I can also uh open many browser as you see there is a lot of browser opening so because I created sessions and make a few request you know it was about all the cookie so I need to reset uh the cookie and I'm

getting a lot of here sometime it's not working sometime working but there is no issue on that if you have a proper retry mechanisms okay let's stop from there but you can see that I can collect all the information and I don't have the fingerprint error so let's move now on to the rate seven if I'm starting again the spiders so we will order to Chrome to

open so play right now will open a chrome fingerprint but this time it stopped and this is because there is a error and consistent time zone so yes the server is collecting fingerprint so testing that you are executing JavaScript but they are using this inform information they are not just collecting the informations so they check that the scent fingerprint is also consistent between all the parameter of

your browser layer so they are checking consistency between the time zone of the browser and the time zone of the IP address here we are using a Time Zone on the browser in Paris but we are using us IP address so there is six hour difference how can it be so let's correct that um to correct that I can change the IP address locations but I want

to keep them in us because they are correct IP address um I can change the browser settings so how to do that I can for that in playr settings so here I can set up the time zone oh no it's not your is America Chicago let's do that yeah in the middle so yes perfect so now if I'm running against spiders Scrapy order to play right to

open the chromium chromium is opening the homepage we are sending the fingerprint with the correct time zone so in US server is checking the consistency with the IP address and you will see that I can this time uh getting all the information so yes I can connect and the website didn't block me so that's perfect and when you are working with fingerprint clearly you try to find

all the monitor signal and to find which signal you show to emulate or modifies I will startop from there perfect so that all for me I hope that you enjoy the session so thank you very much down scoy add star on GitHub and join my company if you want and I will be very happy to answer some questions thank you very much for the very exciting session

Fabian it's good to speak about scraping in a uh a non-threatening way uh is it not being like the most illegal thing that you can do do and you know uh just get rid of some of the myths around it um so one question for me is um in your scraping experience what is your fort nox the most difficult system that you had to from yes that's

a very good questions um you know uh scraping is a cat and moose game so you find out the onti both block you and the next day they are the protections and it's a never ending game so that the most difficult part to track uh all the different change because if you uh send in production some scraping systems and you rely on it and the customer uh

expects that correctly get the data but tomorrow they add a new protection you don't know how much time it will take you uh to to to get uh to bypass protections so that can be clearly annoying technically speaking uh let's talk about the future of protections uh and that's where uh the most Fork KN stuff is coming um now the introduce uh encrypted payload I didn't show

you but most of the time the encrypt the code so you have to decate it to understand what they are doing to correctly emulate the different signals but now they introduce new encryption layer which is difficult to do it's called Uh JavaScript virtual machines so it's kind of java virtual machine the Java virtual machine execute bite code and you can understand by code so B code is

like asbl ASM you can understand that but you more clear way so you can TR back to the source code but now what they are doing when you de get JavaScript virtual machine you get the code of the virtual machine and the by code but clearly you don't have any documentation like in Java to get from the buy code to the source code because it's private it's

undocumented proprietary so it's a very very challenging task to do that and yes it will make me sweet sometimes yeah and I I think they are trying to pass also some laws to make some of the scraping illegal how do you feel about that do you think that this is going to actually happen and there will be not just the technical challenges but legal challenges as well

yes that's very good questions um it's not um technical U cat and moose game it's also legal cat m games um and also marketing Katan m games if I'm speaking on the marketing part uh they are trying to protect their Monopoly by sending a lot of message don't do web scrapping it's illegal no guys you are just protecting your Monopoly um there is a lot of company

built on data you know the most famous one which is Google so no Googles no searching engines and to be honest no internet if you cannot collect the data the Google engine is connecting to your website every days so if you forbid that you don't you cannot have customer on your websites so REM remind that speaking uh after that um clearly they are trying to send this

message but also they are working on the legal part and that why we also contact on this part if you collect public data you collect nonpersonal data yes you're good to go uh you can clearly uh create a new business and also we saw in the US a lot of low care lawsuits and sometime it's on the big company the big company wins but uh sometimes it's

a scraping company wins and yes good for us uh the last time the someone wins it was when weing companies nice nice it's a it's a good thing to see happen gives us a bit of Hope U there's a question from the whole um is it possible to dynamically set the time zone per IP address if our proxies are located in different regions yes you you can

do that uh you can set the browser you must have different um browser instances because once the browser is uh instanced and started you you cannot easily change the time zones so my advice is to to have perhaps in scrap boxy different project one by uh GEOS so one for perhaps Europe one for East Asia one West Asia one Us North or South uh but quite the

same Ty Zone but different locations um and you set up connectors uh which are very consistent in each project so if you are speaking about the project in Europe so you have all the connector with all the proxy in VMS and you use browsers which is connecting to this project so we will use always IP address from yours and you can set the time zone per uh

in Paris whatevers uh in the balcon and it will be consistent if there is not too much difference they are not tracking that okay good answer thanks for that and can you tell me what's on the road map of scr oxy do you have any new functionality that you're planning to implement yes there is a cool functionality uh which uh most user asked um is uh smart

um IP allocations um so let me explain you when you try to bypass protections it can be a very Co expensive games because uh if you always start with residential proxy or mobile proxy which are the most expensive s pieces it can be very costly so the strategy is more you start with cheap IP address and if it's not working if you are blocked because protections are

not all the same on the different website so if you are blocked perhaps you switch or more advanced so you start with data center proxy okay it's not working you go to IP proxy okay it's not working these times you go to residential proxy okay it's not working so you retry with um mobile proxy okay and let's say to swxy okay soxy is starting and you don't

say just do a random assignation of the IP address just give me a DC IP or give me an IP IP and you will select on his list the best success rate or the best IP from this type of proxy and of course you can add also filtering on Geo because you won't use DC strategy with DC in France IP in the US residential in China and

mobile proxy perhaps in South America you will be clearly detected by the antibot system if you're using IP address from all over the world so you want to stay consistent in the same area so it's very L to the previous questions uh so you have this kind of strategy and yes that's something which is ask uh from user to to be very efficient in Productions and it

will be relas um clearly uh around September and before at the end of the years because it's a major features yeah sounds like a really good addition to the product okay well thank you very much for the lovely talk Fabian I think it was very useful to everyone piece of advice for the audience on their road to web scraping yes if you start with to scrape uh

you can easily install scrip boxy it's adapt for new or for experts uh so you will have immediately a new IP address so you never scrap on your own IP address so the next time you use your Mobile mobile phone at home or your family won't be blocked with your IP address always use proxy thanks thank you very much all right um thank you very much to

our uh audience as well um and uh we'll be moving on to a break and then the next session bye thank you cheers bye