Paul Conroy: Digital Cat-and-Mouse: Strategies to Outsmart Scrapers, Phishers, and Thieves

In this talk, the speaker explores strategies for protecting online businesses from scrapers, scammers, and thieves, drawing from personal experiences in the property website industry. With nearly 30 years of web development experience and as the CTO of a digital agency, he discusses the challenges faced against aggressive competitors employing scraping tactics to capture valuable listings and user traffic. He details a creative technical approach to fend off scraping attempts, such as manipulating property data to confuse potential buyers and undermine the credibility of competitor platforms. The speaker also touches on broader themes, including AI bot traffic issues and the need for regulations concerning web scraping. Throughout the session, he emphasizes the importance of having a multifaceted defense strategy against digital threats.

Uh thanks for coming along, folks. Uh this this is a cool venue, isn't it? They're being in a cinema like this rather than sitting in a conference chair somewhere. Um I was saying to one of the guys earlier that after college I briefly worked in a cinema. Uh so the last time I stood in front of a screen like this talking to a room of people was

it was a long time ago, but it was the opening night of Lord of the Rings where I stood in front of 300 people telling them that unfortunately the projector had eaten the last reel of film and they wouldn't get to see the rest of it. So, I'm hoping the vibes in the room when I finish talking today are a little bit more positive than the the

last time that I did it. So, we'll see how we get on. But uh yeah, so thanks. So, today we're going to talk about uh digital cat and mouse, the strategies outsmarting scrapers, fishers, and thieves. Uh when you have built up an online business, a lot of content, a lot of users, that sort of stuff, and someone else is coming along and basically trying to take it

from you in one form or another. So, a little bit about me before we dive into it. So, I'm from Dublin in Ireland. I have my background is as a developer. I've been working with the web in one form or another for uh nearly 30 years now back when notepad, GeoCities, and FrontPage were absolutely the the cutting edge. I am the CTO at Square One. We're a

digital agency. We do a lot of work with online publishers, classified websites, that sort of stuff. And a lot of the the stories today will kind of come from that sort of experience. Now, when we're talking about protecting your your online assets, normally these type of talks are around SHH SSH security problems, uh zero-day exploits, people trying to get at your database, you know, that that type

of serious infrastructure stuff. Today's conversation is is not so much about that. It's as I said, you're you've built up an online business and people are coming along, pirates, scammers, thieves, and they're trying to take your your digital booty away from you, as it were. Um uh a side note, Googling for uh safe-for-work pirate booty image turned out to be a lot more difficult than I than

I expected. So, uh ChatGPT came to the rescue there. The first story we'll talk about today is back in the mid-2000s. I was working in a popular property website in Ireland. Now, in Ireland at the time, the market was effectively a duopoly. We had one competitor who were previously the largest one in the country. They were well-funded. They were backed by the largest newspaper publisher and a

number of estate agents. And then there was us. We were effectively a scrappy startup, a lot of guys in our first jobs out of college, that sort of stuff. But we're doing well and we'd recently become number one in terms of the volume of of listings. And property websites in Ireland are an enormous business. These would be some of the most popular websites throughout because property is

kind of a an obsession in Ireland. You know, even if you're not buying or renting a house, you want to go and see, well, what does a kitchen look like in a half million euro house? Or what what what do the bedrooms look like in that house down the road? That sort of stuff. So, there's a lot of curiosity and it was it was big business. And

the big business meant that a lot of new competitors were being attracted This meant that every week we would see on the side of a bus or a billboard somewhere some snappy new brand that popped up. They launched their website. It would look beautiful, but they wouldn't have a lot of content. So, they would tend to vanish after a couple of months. And the reason they wouldn't

have content is that at the time, there wasn't a central portal where an estate agent would enter their information and it would syndicate everywhere. You had to kind of enter it into every single website. So, this was a nice moat for ourselves and our other main competitor. Uh and it meant it was very difficult for a new company to come in and get any kind of traction

because, you know, you have a chicken and egg problem. Why would you go and spend all of your time putting content on this other website when there's no users there yet? Then one day, uh we saw this uh Shamrock Shamrock Island was going to be built in Dublin Bay. There was a planning application went in with the city council. And a video went out and went massively

viral. It was basically going to be like the Palm Islands in Dubai, but in in Dublin Bay. There'd be massive skyscrapers, apartments for 50,000 people, light rail system. They're going to have the the world's first giraffe-only zoo. They'd uh they're promising everything in this island. And it got a huge amount of attention. It got a lot of coverage on the national broadcaster. And it turned out to

be a viral ad campaign that uh had been set up by a new competitor in the the property market. And this was quite an expensive one, you know, it would have cost the equivalent 70 or 80,000 euro in today's money. So, it was a serious serious amount of money being spent. And it's being spent by a company called Funda. Now, Funda are a very large, very well-established

property website in the Netherlands. So, for us, this was a bit of a concern because now here we had a competitor who had the deep pockets and potentially the ability to to come in and take a big chunk of the market. Now, just a quick detour for those of you who aren't massively familiar with property classified like this. You'll find a property you like. You click through

into a detail page. You like it again. You contact the estate agent, and hopefully you end up buying or renting the thing at the end What Funda had decided to do was they would have agents put properties on their website, but they didn't have a huge amount of content. Again, they had the chicken and egg problem here. So, they went out and they scraped our listings as

we had by a country mile the most property listings in the country at the time. And what they would do was on their website, they would show all of the listings from both our site and their site. And if you clicked on one that was from our site, they would send the traffic back to us, which would ultimately go on to the estate agent. But their plan

was as their website grew more and more and more that they would ultimately be able to go to the estate agents say, "Look, we have all of the audience. You don't need to work with those guys anymore. Come work with us." Push us out of the picture, and on we go. So, again, this was this was quite a a threat for us. And what are our options

at this point? Well, one option is to just ask them nicely to stop. Uh this didn't have a huge amount of success as you might expect. Sometimes works if you're a larger player when you can kind of threaten someone smaller coming in, but wasn't wasn't going to work with these guys. Another option would be to go to court and try and tie them up in in in

legal cases. This was the approach that our competitors took. Uh this wasn't an option for us for a number of reasons. One, I mentioned scrappy startup, you know, very expensive. We didn't have a lot of a high-powered lawyers on call. The case law in Ireland as well at the time wasn't really settled in terms of the legalities of screen scraping. Uh and it could also take 18

months, 2 years to actually settle, by which point it might be a moot point if they've grown significantly in that time, and then the court tells them to stop in 2 years, doesn't matter where we're dead, and uh, it's not not a big issue for them anymore. So, our final option was looking at technical countermeasures. So, first thing we did was we went to our logs, and

we know that within our logs, we would have uh, a lot of requests coming in every day, and there would be a suspicious batch of requests coming in from Dutch IP addresses, which is very unusual for an Irish property website aimed at Irish audiences. We looked up these IP addresses and found that they belonged to a company whose website said they specialize in web scraping, and listed

one of their main clients as Funda, so we thought these are probably probably the guys we're looking for. So, first thing we did, we saw the IP addresses, put them into our firewall, and said, "Nope, don't let these guys in anymore." They rotate IP addresses, they come back in, so it's this kind of cat and mouse every day of of trying to find them and block them,

and find them and block them, and it was a bit of a pain. So, we decided to take a slightly different approach. We had a function in our code where we would be able to log all of these IP addresses, and if you came to the website from these IP addresses, we would mess with the listings just a little bit. So, instead of maybe a two-bedroom house,

it's maybe it's a three-bedroom, maybe it's a one-bedroom, maybe the price is 10% higher or lower. So, if you came to our website, you might see or sorry, if you went to their website, you might see a listing that would look something like this. And if you think, "Yeah, that looks good, the price point is good, the bedrooms are good, that's all fine." They click through to

our website, and suddenly notice, "Oh, hang on. Price is different here. Or the beds and the baths are different here." So, there's there's a discrepancy between the two of them. And the thinking behind this was that people would be going to the first the the Funda website, they would see property details, they come to our website, there's a difference there. The idea is that they would then

ultimately go and complain to the estate agent saying, "This this is wasting our time, we're very angry, and and this is just a big waste of time here." The estate agent would then go to Funda and say, "Listen, you guys are putting wrong details up, you need to take all of my listings down. And this kind of worked. Some agents made the complaint and Funda took some

of their listings down. But in other ways it backfired on us quite a bit because at the end of the day people were looking at the two websites, they saw they were different. They didn't know which one was right and which one was wrong. So they would just complain to everybody. So our support team were getting a lot of blowback on this as well. So we needed

a a slightly different approach here. And we thought about this and said, "Okay, so our problem here is that can't really tell which is the more credible website between the two." What we did was set up a system where not only would we mess with the the price and the bedrooms, but we'd also mess with maybe the addresses and the photographs as well. What we needed to

do it in a way that would be semantically sound. So as far as a machine was concerned, it still looked okay, but to a human it would be obvious nonsense. So it would be easy for people to look at a website and see this isn't this this isn't a website we can trust. So we would then have addresses like 10 Downing Street in Dublin. We would have

Willy Wonka's chocolate factory in County Cork. We'd have 1600 Pennsylvania Avenue in in Leitrim. And then we would have the Emerald Castle Oz in in in County Galway. Internally we called this our project Yellow Brick Road. So we we had a lot of Wizard of Oz references shoehorned in there. So we got this done, put it live, and then the next day noticed that on the Funda

site, you suddenly started to see some legitimate results in there, but an awful lot of junk. Awful lot of junk spread through it. But junk that's very easy to detect for a human looking at this, less so for a machine. This was a great great result for us. You can see this this kind of problem. As I mentioned, you know, property was a a bit of a

national obsession in This led very quickly to people taking screenshots and sharing it on social media that, you know, look at these crazy joke listings. This is crazy stuff on this website, which again helped us as it was continually chipping away at the credibility that they had. So they would delete all of the listings, they would try changing their IP addresses, coming again. We'd catch them and

and every day they would have the same sort of thing like this. This went on for uh maybe a week or two that they they kept trying this before eventually they had to look at this and say they had to throw their hands up and say this this is this isn't going to work for us Because it would need so much manual effort to clean this stuff

up that it's it's just not it's it's not in their road map. It's not in their business plan, the investment plan that they have. So, they gave up on us and then decided to try scraping more our competitor who were tying them up with legal cases at the time. But, the point was they left us alone. So, it was a absolutely wonderful victory for us. So, we're

very very very happy with this that it worked. We didn't have to spend a lot on lawyers. Nice little technical work around there did the job. Now, as I was a flawless victory, it was almost a flawless victory. So, our code looked a little bit something like this when one of our developers was testing it locally. We would check if you are coming from one of these

known scraper IP addresses, then we would go and fake the address up. The more technical amongst you in the audience can see the problem here straight away. Uh this little stray or true here means that this code will be executed all of the time regardless of whether you're coming from those IP addresses This is a kind of a it's a lazy way for someone to test this

locally. And our developer was testing this and you know, coming back to my mention of scrappy startup, a lot of people on their their first jobs. This was a long time ago before GitHub was even a a glint in its inventor's eyes. So, we had weak or non-existent code review practices. Uh the deployment was effectively someone just FTP'd it to production and then a happy days job

was done. Our problem here was that this was pushed to production. And while we were all sitting around very smug and satisfied with ourselves in the technical team about 2 minutes after it went live, someone from the sales team came sprinting down to us going, "Guys, has someone hacked the website? It's full of absolute nonsense. What's going on?" Uh so, we had effectively turned our our lovely

gun on ourselves at this point. So, this was a horrifically embarrassing. Uh but, it did mean I guess the silver lining here was it led to a very rapid improvement in our development processes, code reviews, uh deployment, and so on. You know, there's there's a saying in industry that a lot of regulations are are written in blood after some disaster or other. And it's not quite that

serious, but certainly in this case our our dev process is where written in the the screamed obscenities of a very angry tech lead. So, this went live, this worked, and uh technical countermeasure this was for us this was this was quite a while ago when we were we were working on this website, but it's still quite an active conversation today. You know, if you go to social

media, you you're seeing an increasing number of posts like this where site owners are talking about the fact that they they have a site and AI bots in particular are just relentlessly coming at these sites and hoovering up as much content as they can. So, site owners are seeing increased in bandwidth bills for for no real benefits. The AI bots are sucking up the content and trying

to build something impressive that is not really the value isn't really flowing back to the the site owners and site creators. Cloudflare have announced a product called their labyrinth. And what this does is it will detect an AI bot coming in, and instead of serving the content from your website, it will return an AI generated page. And it will be again semantically sound, but generally nonsense, and

it'll keep the AI bot busy reading it. And it will also be littered with links internally. So, the bots will start to follow these links, which also go to AI slop pages managed by Cloudflare. So, the idea here with Cloudflare is just to keep the bot running around in circles internally and just wasting its time rather than actually going and incurring resources on your site. So, if

you you're a Cloudflare customer and you have any kind of issues like this with AI bots traffic this is certainly something checking out. But what both of these approaches kind of have in in common it's effectively like a tar pit. We're just trying to waste the time of the people who are coming after us and see if we can just slow them down enough we're no longer

really a useful target for them. It's a bit like you know, you might have heard the story about the people who are on safari in the savanna. And then a lion gets loose and it's chasing after them. And the guys are there, and then one guy bends down and he starts tying his laces. And the guy beside him is going, "What what what are you doing? Uh

you're never going to outrun the lion if you're just standing sitting there and tying your laces to to get ready." The guy looks up at him and says, "I I don't need to outrun the lion. I just need to outrun one of you." So, you know, it's one thing you can remember here is that you don't necessarily need a flawless victory against these guys. It's just enough

to be annoying enough or painful enough that they then turn their sights to someone else. In our case, they went off to their competitor and happy days. We were we were done. We were in in the clear there. And this kind of idea of being just annoying enough to to force your your adversaries away can also come back to say bandwidth bandwidth hogs. So, if you have

uh a rich site, you know, a lot of uh maybe user-generated content, a lot of imagery. Again, using a property site as an example here. You're probably going to have a lot of images. And images can make up 60 to 65% of a site's payload, generally speaking. And bandwidth can get quite expensive if you have if you have pop popular images uh on your site. Uh pattern

that we saw a lot was one of these upstarts would come along. They'd scrape all of our content except for the images. So, they would have all of the the text and the details of the property. And then they would just hotlink to the image. So, hotlinking meant that they were using our URL for the image there. So, we would incur all of the costs of serving

the image, whereas they would have all of the content on the page. So, we didn't like this a lot. One strategy we could take is if someone's going to embed one of our images in their page, we'll just stick a big fat watermark in the middle of it. So, it becomes obvious that it's it it's it's from our site and not theirs. So, this the company was

uh it's called daft.ie. It's this kind of orange house with a blue circle around. Big fat watermark smack in the middle of it. It's very obnoxious. It's very obvious. And uh when you see it on the other sites, then it's uh it's it's it's it's pretty clear that this image is not native to that site. And the problem we had with this was estate agents absolutely hated

it. So, the estate agents as a group were not always the most technically sophisticated groups. So, what would be a common pattern would be they would upload an ad onto our system, all of the images, and then in maybe a month when they need to generate a brochure or send a newsletter or something like that, they wouldn't want to go run around in the office checking, "Oh,

what USB key do we have these images on?" Or what share drive was it on? They try and take them back out of our platform. So, this this watermark was painful for them, so they they wanted us to to at least dial it down a bit. So, we said, "Okay, let's go for a slightly more subtle watermark in the bottom corner here." And our competitors responded by

saying, "Well, maybe we'll just chop off the bottom 10% of the image. No one really notice. It's a nice nice wide screen format." So, we started shaking it up a bit and said, "Okay, well, maybe it's on the bottom. Maybe we'll stick it up the top." These guys said, "Okay, well, maybe we'll just stick a button over the top." And we started shuffling it up then where

it would randomly change depending on the the image. So, okay, maybe sometimes it's left, sometimes it's right. They weren't going to know which one it was before it generated. So, they says, "Right, well, we can stick another button there." And another one. And then eventually they ended up with just these black spots. It was kind of fun to watch them commit this UX harakiri on themselves, but

uh ultimately it's this kind of arms race back and forth. So, we There are more technical solutions we could follow that were relatively straightforward. One of them was using Engine X as a web server. So, if any request came in for an image, we would check if the referrer was coming from our website. If it wasn't, it was probably hotlinking and we could serve back the same

stock image regardless of what you asked for, we'd serve back the same image, which was this big fat "Please go view this on our website." Which was a very uh subtle subtle message. Um But again, this was annoying enough for these guys that they they stopped or moved on to one of our competitors. So, again, we we managed to to annoy them just just sufficiently to to

get them out of the way. Again, sort of didn't outrun the lion, but managed to outrun someone else in our our our group Now, a challenge with this in more recent years is that the referrer information isn't always sent as much as it used to be. Browser security has changed quite a bit. So, a pattern we see a lot more now is this type of signed URLs.

So, you'll have an image and you'll have some kind of signature at the end. The signature's generally a hash of uh some shared secret and then maybe a timestamp. So, this is valid for maybe 5 minutes. And when you have these kind of signatures in place, you don't care massively if someone does copy these URLs because if they copy all of them now, put them into their

database and their website, in 5 minutes they're going to be useless. So, it's a it can be a very effective strategy for this. Now, the the challenge here is when a request comes in with this signature, how can we validate it? So, a naive approach here might be to say, "Well, our our Rails app or our Laravel app in PHP, whatever is that generates these signatures in

the first place, we'll just use that to validate them and then proxy through an image or or whatnot." Now, the problem is at scale, booting up your application all of the time to serve these static files is not going to be great from a performance perspective. You you can end up slowing down the experience for your legitimate users all because you're adding this little protection against these

uh these pirates that are trying to do something sneaky on on the side. Unfortunately, if you are using Nginx, uh there's there's a lot of boxes on the screen here, but ultimately all it's checking is when a request comes in, Nginx can check, "Okay, we cached this already. If we have, send it through. If it's been cached in the last 5 minutes, that's fine. If not, within

Nginx uh Nginx uses uh or makes available Lua that you can do some very basic scripting. So, directly within Nginx, you can load the shared secret, validate the signature, see if it's in a the right timestamp. If not, show an error. If it's good, show the image, pass it through, and send it through all without booting up your application. So, you can keep this kind of nice

responsive uh uh performance for your your static images here while still having the security of of the signatures. You do get a lot as well. Uh the likes of Cloudflare and Cloudinary will also offer one-click out-of-the-box check this and do hotlink protection as well. So, again, certainly worth looking at that before diving into your own infrastructure if you're a customer of those type of companies. Uh sometimes

when we talk about this as well, when we talk about the signature and how it's built, we sometimes get the suggestion that, "Well, when someone comes to our website, maybe we could include something extra like maybe their IP address in the signature so we can validate it definitely is them um along And we would typically recommend against that. One of the reasons is that it's important to

remember that one IP address is not one person. We had a problem here where this little red dot down here in the the bottom left here was in our logs doing all kinds of weird wonderful things. It was logging into 27 different accounts and it was doing things all over the country, very suspicious activity. And we found out that it was the one of the early mobile

networks in Ireland was just sending all of their their mobile data for all of their users in this region through this one one endpoint. So one IP address is not is not equal to one person. You know, you think about shared Wi-Fi here for example. So just something to keep in mind when you're designing these kind of protection strategies. We talked a little bit about protecting our

assets, but we also need to protect our users and sometimes protect them from themselves. So uh really really common scam that used to happen on property websites that had say rentals would be that you would see property like this. You would have a beautiful photograph which would be copied from booking.com or Airbnb or somewhere where they'd professional photographers that put a bit of time into it. It

would be city center and the price would be obscenely low. It would be maybe half the price or less of what would realistically be. So what happens is you look at this, it's a there's a housing crisis, you're desperate for somewhere to live. It seems too good to be true, but maybe just maybe it's a bargain. Contact the owner. And I as the owner will get back

to you and say, "Oh brilliant, yes. Unfortunately, I'm out of the country at the moment or I'm looking after this for my nephew who has left to go to college in another country. So I'm I'm not available right now, but you seem trustworthy. I like I like the cut of your jib here. So let's strike a deal. If you send me two months worth of rent through

Western Union, I will get the keys to you immediately and this is yours." And Western Union is the key part of the the scam here because then with that code, you can take that money out of a Western Union nearly anywhere in the world. Disappear into the ether and that money is never coming back. So this was a common scam. when we would uh to fight against

this on the website, sometimes it was uh easy enough to spot these dodgy ads when they came in. So, this is is a beautiful looking uh sitting room here, uh supposedly in Dublin. But, if we look a little bit closer, we can see these little roundy plugs. Uh so, in Ireland and the UK, we're horrifically awkward and different from the rest of Europe in that we have

square plug sockets with the the three prongs on them. So, you can see that this is clearly not anywhere near Dublin city center. Uh so, this was a a subtle one, but sometimes uh the scams are a little more obvious. So, we had another photo for a Dublin city center apartment once, looked a little bit like this, and uh if you look out the window, you can

see the obvious problem here is that there's a blue sky with no rain clouds in it, which is very very unlike Dublin. Uh so, we can have this, but then we could also add in things that uh we would let users report these ads to us automatically if they saw a dodgy things going on. And we would have heuristics internally that would say, "Okay, the scammers, they

tried to place this on a Friday evening of a bank holiday weekend in the hope that support would be thin on the ground. If we get 10, 15, 20 reports, whatever it is, take down the ad automatically, lock the account, and and so on." So, this is one type of heuristic we could do, and and lean on our users to help us. Uh but, we could also

do other things like say, "If you are a a private landlord placing your first ad, then it needs to go through manual verification." It adds some sort of friction to the process, but it does eliminate this kind of scam quite quickly. But, the problem is in say the property website case, one group of users were estate agents. So, they would pay uh a a much larger fee,

they would have access to much more functionality in the system, and they would also have credibility. Like, when they place an ad like this, there would be agent branding all over it, and you would see the 50 other ads that they'd placed. So, you know, you would you would trust the ad even more. So, if you're someone trying to run one of these scams, it's really attractive

to get the login details of an agent that you can kind of run it under their banner. So, you try to to fish them. And uh typically, coming to our website as an agent, they log in, they get their platform access, and all is good. What the the scammers would try to do is set up a page that looks identical, trick the agent into going to it

whether through email or text or some other means. When the agent tries logging in, they get an error message. Maybe they get redirected to our website and they think, "Oh, I typed that wrong." They go again and everything looks okay, but the scammers now have those credentials and then they can log in as the agent and and work away and and place ads to their heart's content

until the the agent notice there's there's a problem, probably a couple of days later. When the scammers have copied these websites, they would look visually identical. There were sometimes ways we could fight against this. So, sometimes the scammers are very lazy in that they would copy almost all of the HTML across, change where the form submits to, but copy pretty much everything else including hot-linking back to

our CSS. So, you can see here this V123, it's the same on on both of them. So, we can say, "Okay, on our side, we just going to change this to 456 and this is the normal CSS, that's all fine. So, now the only place in the world that anybody would be looking at this 123 version is on the fished website. So, we can release a a

separate CSS file targeting that that particular URL that does nothing other than say, "Display don't don't don't display anything here." So, our page continues to look fine and suddenly on the fisher's site, it's completely blank and completely dead. So, this was this was a nice way of using their willingness to hot-link us against them. Now, after a while, they get a little bit smarter and they figure

out that they need to copy the CSS and JavaScript into their own fished fished websites, but they didn't always catch everything. So, we had an example once where we had Intercom tags on the page. So, this uh black button at the bottom here, for those of you who don't know Intercom, you'll see these kind of chat buttons on a lot of websites. You press that, you can

talk to an operator or an AI agent or something like that, but it also allows you to target your users. So, if you can set up a rule in Intercom to say, "If someone is on my {slash} brochure page, pop up a message telling them for the next 24 hours this discount or whatever it is." And what we were able to do in this case was check,

"Okay, if someone is on this scammy fishing site, pop a big warning saying, "Danger here. Danger here. Do not log into this website." Now, this is a really weird experience. If you think about it as an agent, you've clicked a link that's going to log you into a portal you log into all of the time, and suddenly the portal is telling you itself, "This is dangerous. Stay

away from me." Uh this led to a lot of support emails and calls from confused agents asking what the hell is happening. So, there was a bit of a support overhead, but it also meant that they weren't getting fished, and we weren't seeing an increase in scams. So, it was it was a net positive for us. And the thing with these fishing websites is that when you

discover them, the immediate tendency is, "Okay, there's damage here. We want to smash them. We want to take them down. We need to get rid of them as quickly as possible." Uh but sometimes we followed a slightly different strategy here and went for more of a a honeypotting We would have credentials that looked real, but didn't belong to any user, and we would go to the fishing

site as if we were an estate agent, put in these credentials, and now the attackers will be the only ones who have these credentials. And back on our site then, we would have some code that checks when someone tries to log in if they're logging in with these uh honeypotted credentials, we will get all of the other accounts that they've logged into from that IP address, browser,

and and try and figure out what else they possibly logged into. And for each of those accounts, we could proactively lock them and then have our account management team reach out to the agents. And this was a big thing in helping us to figure out people who had maybe already been compromised and didn't know it. So, this was this was a very useful way of doing it

using this kind of a honeypotting strategy. And now, there are other strategies to educate your users, you know, don't click on URLs that clearly aren't us. Two-factor authentication is by a country mile the most effective way of uh fighting against these scams. It is also the most effective way by a country mile of swamping your support desk with complaints from users. Uh I mentioned non-technical user base.

They're used to having three people in an office passing around the post-its with a username and password on it. Now you're saying, "Well, now that a text will be sent to this phone. Bit of a challenge to go through, but uh we we got there in the And uh another resource here, have I been pwned.com. Um you may or may not have come across this site before.

This is a free resource where they go through and collect all of the password leaks that appear on the dark web. And there's an API where you can call this and say, "Okay, the password I'm being given here, is this one that's known to be incredibly weak or vulnerable and and is in all of these attacks?" This is uh something you put into, say, your your registration

process or someone is changing a password, you can do this kind of check and give them a warning. It's kind of baked into the core of uh of Laravel, uh for example. And a few other services. So, if you haven't come across that, it's it's an interesting uh interesting thing to add to your your security toolkit. Now, we talked about the phishing sites and what we do

when we find the phishing sites. So, there's always the question, well, how do you find them in the first place? And uh one email here, you might If you're an estate agent, uh you might get an email that looks a little bit like this when somebody makes an inquiry on your property, and there'll be a link there that you have to click and log into. Very, very

attractive target for the the scammers to try and send something like this and trick you into clicking onto their their system. I mentioned uh scrappy startup, the nice styled emails like this were not even in the top 10 of anybody's lists. So, our emails tended to look a little bit more like this. Very crude, plain text. It was a it was get it out the door and

get it done. Uh so, we discovered a phishing site at one point because an estate agent got onto us and said, "Uh guys, I've been using the site for a while. Uh it works well for me, big fan. Uh however, I got the email on the left here. And when I saw it, I knew this looks way fancier and more professional than what I normally get from

you guys. So, I knew there was something dodgy about it." Um which Okay, don't go all all feedback is good feedback. Um so, I'm I'm not saying to be conspicuously in professional unprofessional in some parts of your communication, but I'm not not saying it cuz you know, it worked worked for us in in in one sense there. Um so, you know, we've talked about a couple of

cases here where we're very reactive and responding to these kind of threats. And the the next thing I want to talk about is slightly different. It's a story I love to tell cuz it's it's a little bit more strategic and a little bit more long-term. And it's a the Black Sunday hack. So, the Black Sunday hack happened in 2001 in the States. So, the the environment the

background to this is in the '90s pay TV took off in a big way. Uh sports and movies mostly uh football, boxing, pay-per-view, that that sort of stuff. You want this uh system at home. You get this decoder. You get a card that goes in. It connects to your phone line. Satellite signal comes in. You pay $80, $100 a month, whatever it is. If you're running a

bar, you're going to pay a lot more than $100 a month. But you make your money back by charging people 20 quid ahead to come in and watch the game on Sunday. So, it's uh it's a thriving industry. The way it would work is the signal would come down. The box is basically a a dumb box that just relays the signal onto the card. Card has a

small micro process processor on it. And then it will check things like does this card have access to to this stream or whatever. Uh now, this encryption got cracked and people started producing counterfeit cards. And they figured out that if you took the phone wire out of the back, there wasn't really any way for the box to proactively communicate back to uh DirecTV in this case. And

if you had the cracked card, you could put into the micro process processor routines like uh if we're ever asked, does this person have access to this channel? Just always answer yes. Don't even do any any calculations. So, these these cracked cards were sold for a couple of hundred dollars one-time fee. If you're running a bar and you're managing to charge 150 people 20 quid ahead and

you can get away with a one-time fee rather than paying a couple of grand a month, you know, could be very tempting. So, they were very popular in in a lot of the bars as well. But the cable company tried to fight back against this. So, they would regularly from the satellite send new signals, new kind of countermeasures. The cracked card makers, uh if the countermeasure matched

the signature they'd seen before somewhere, the card just ignored it. Uh otherwise, they would work on a an adaptation to support whatever the change was in the software update was. So, it was a was a thriving ecosystem, but it started going round in circles quite quickly. You know, the cable company would push an update that would break everything. Pirates would get notified of this. Many times within

15 or 20 minutes, they have a workaround in place, and it's being sent out on the IRC channels, and people are adding it in the with their serial ports and so on. Cable company go again, and this just kind of goes on and on and on and on. So, again, back back to this type of arms race we talked about earlier. And this went on for a

while and then changed in the sort of end of 2000, where suddenly the the pirates noticed that the volume of these updates increased significantly. They were coming out much more regularly. But, the code that was being sent down, it wasn't it wasn't destructive in the same way it was before. A lot of it seemed to be um inert functions and stuff that wasn't really doing much of

anything. So, it wasn't threatening, so the pirates weren't blacklisting it. They'd let it through. They didn't really know why this was happening. One theory was that the cable company were trying to just annoy them out of existence and get them to go and focus on someone else by just constantly pinging these updates they needed to worry about them. Uh another theory was maybe the cable company had

just hired a bunch of juniors and their QA had gone, and they were just sending out junk, whatever it But, in any case, cuz they weren't really messing with the card anyway, the pirates were happy enough to just let these little blocks come through and get executed on the microprocessor. So, you have your card. Cable company sends something down, and okay, it gets waved through. They send

something else down, and it gets waved through. And there's still no obvious pattern going on here. And over time, more and more of these things start to come through. And into uh all the way through December and into January, and then suddenly in February, 1 week before the Super Bowl kicks off, the last bit comes in, and now, boom, they have a problem. Cuz what had happened

was the cable company had with the last update plug together all of the other little inert bits they had, and now suddenly there was a program that could execute on the card. And what it would do is it would check the certain hardware signatures, and it would be looking for things that only exist on legit cards and patterns that only exist on the dodgy cards. If it

found a dodgy card, what it would do is it would write into part of the the memory and right at the start of the memory would override it and it would override it in such a way that it effectively breaks the boot loader. So, these things are going to an an infinite loop and they so they can't be easily hooked up and and overwritten. And it meant

that effectively 98% of the pirate cards were bricked straight away. They would need to be physically replaced, they couldn't be easily done. So, this was enormously successful. Now, if you're a publican or a bar owner who has one of these dodgy cards and you're a couple of days out from the Super Bowl and you've already sold 200 tickets at $20 a head, you're you're starting to sweat

a little bit at this point. So, DirecTV's sales team had a a record week with a publicans who all of a sudden discovered the need for for a legitimate pay TV. But this what I like about this story is that it is a uh it it's a testament to a longer-term strategic thinking here. You know, you're going from a pattern of just arms race back and forth,

back and forth, back and forth. At some point somebody in the cable company went to their boss and said, "Listen, for the next 2 to 3 months, we're just going to allow the pirates to keep working. They'll grow their market share, they'll get more people in. Okay, fine. But we're putting a big bet on the fact that if we can do this right, we can knock out

a lot of them very very quickly. And you know, the it worked very well. It was a very technically sound fix. So, the developers did take a bit of a a bit of a victory lap here. Uh I mentioned it over at the start of the boost uh the boot code. The first eight characters in boot code uh read game over afterwards. So, that was a little

bit of a a little bit of a victory lap that the developers took for when the uh the pirates were decoding this. This happened back in uh 2001 and as we know, uh there has been absolutely no piracy since 2001. So, it was a resounding and lasting success. Uh unfortunately not. Like you look at the at the news over the last few weeks, Fire Sticks, IPTV, all

of this sort of stuff. There's so much money in TV and movies in particular that the pirates gathered themselves up again, found more countermeasures, and kind of got back on the horse. We see a lot of the the strategies here. Now, they seem to be increasingly going for a legal legal approaches rather than the the more technical ones. Um so, it's kind of to to loop back

on the the strategies we've we've talked about today. Not so much legal here, but the the the tarpitting I think worked very very well So, if you have the opportunity to slow your adversary down, whether your your your code is responding slower or you're just giving them stuff that makes it very slow for them to get through, that's that's one way to go. Honeypotting works very well

and can be a great way of getting a sight on the wider world of damage that might be happening in your system you might not know about with these dodgy users. The fake data as well, this one worked very well in the past where you have semantically sound data that looks good to machines, but humans would need to spend a lot of time on it and it

sucks up a lot of time. Much more challenging with a with AI. Uh I think you probably need to get AI to help you create data that will uh trip up other AIs, so you're back into kind of an arms race there. But ultimately, whatever strategy you're you're taking when people are coming after your business like this, you're kind of looking for a Swiss cheese defense because

no one strategy on its own is is ever going to be bulletproof. Uh you want to have just enough strategies built up together that you you ultimately manage to catch a lot of these uh these attacks that are coming The DirecTV example, what I like about that was a patience can also be a virtue. You know, you want to respond to these things immediately, you want to

wipe these guys off the face of the earth, but sometimes taking a deep breath and coming up with a a longer-term strategy can can really pay off. So, just uh coming up on time here. So, just the main sort of takeaways uh I'd like to leave you with today is to uh to know your adversary. Like, what is it they're trying to do? What are they trying

to take from you? Your monitoring internally should give you an idea of exactly how they're doing this and what kind of countermeasure you can put in place. Do we want to uh just immediately cut them off or do we want to follow them around and maybe mess with them a little bit and have more substantial counterattack. Really really really really really really really important point here is

knowing the law. Uh at some point, if uh let's say someone is trying to fish your website, someone in your team may come to you and say uh we found their server, we did a port scan against this and uh we can find vulnerabilities. So we can shell in there and we can absolutely destroy this site very very quickly. Hypothetically, somebody might say that to you. Know

the law in your jurisdiction as in many cases if you do go through with that sort of thing, you can legally be in more trouble than the the pirates that are attacking you in the first place. So you need to be very careful to know what you can and can't do in in your own jurisdiction. And then ultimately, like the big thing to take away here is

you don't need total victory. Again, you just need to be a little bit less attractive of a target than the other guys trying to run away from the the lions at the same These slides I'll share them afterwards. So there's a little bit further reading here if you're interested in any of the stories we talked about today. The the Dublin Island you can tell how old this

is as they're still running HTTP website. No one's touched it in about 15 years. So they're they're up there. And just leaves it with me to thank you all for your time and attention today. My blog's up there. QR code. Feel free to link in with me on the socials, LinkedIn, anything like that. And enjoy the rest of the conference. Paul, thank you very much. That was

very interesting. We have couple of questions. We still have 5 minutes so or so. And questions will be regarding the IP use. And the question The first question is I want Whenever you showed you were catching the Dutch IP address, right? And the question is why didn't the scrappers hide their IPs? For example, you know, use VPN or I don't know, get your IP in Dublin, you

know, and use your IP. Why Why not? Yeah. We did get a little bit lucky in this case as you know, this this was happening quite a while ago where the the state of the art now is very trivial now to to fire up a bunch of these VPNs and and rotate between IPs a little bit more easily. We got lucky in that the adversary you were

dealing with at they they pitched themselves very openly as a scraping company. So they they would be very clear about this. So we were lucky in that sense that yeah, obviously we if we had a more let's say if we had a sneakier adversary who was rotating and and bouncing around, it would have been more challenging to do the same thing and we would have had to

look at much more much more difficult ways of detection. You know, we would do subtle things like maybe you're tweaking the the description subtly. There's an extra full stop here or not there when you see suspicious traffic and then seeing if you can follow that back through to to their website. Much longer process, much more error-prone that sort of stuff. So yes, long story short, we we

got very lucky. Very lucky at the time as yeah, it is it is definitely if you're dealing with these scrapers now, it would be a little bit more challenging to to fight these sort of things. Okay, the second question is also very similar to the first one. What insights do you have on detecting scrapping bots that invade detection? For example, rotating IPs and browsers, fingerprints, or use

residential proxies. Yeah, it's it's a very tricky one when you don't have that consistent whether it's IP or user agent or whatever following them through a journey. And which case you're you're in you're in a tricky zone here of trying to sort of reverse engineer from your logs people who are going through a certain path. So you know, maybe you'll be doing something like they're they're going

through different IP addresses every time, but you might do something like you you remember if you you were on the web a few years ago, any Microsoft site you went to, they would always have this kind of session ID in the URL, something like that. You could do something like where you can then in your logs track through and say, "Okay, even though these four requests came

from radically different IPs, they were close enough in time, we can kind of sense, okay, this is probably a pattern coming from this guy." And then the IP is is important, but you're also looking at the pattern of what exactly it is they're trying to do on the website. You know, I talked a little bit about knowing your adversary, knowing what it is that they're trying to

do to you. You know, whether it's they're they're trying to attack your contact form or they're looking to find the quickest route possible to to get a full list of all the images on a property, for example. You know, this once you get a better idea of how they're trying to attack you, you can then kind of come back from the logs and say, "Okay, what's our

protection against someone who does this?" You'd be surprised sometimes at how efficient some of these guys can be. You know, we were being scraped by a different competitor at one point, and they managed to find routes through our website where we had a we we did like a property listings like the search results. And we used to have all kinds of different variations of this. There was

a map one, there was one that was just large images, there was all kinds of weird and wonderful ones. And there was one we'd kind of forgotten about, which was primarily image-based, but had little image carousels, and wasn't linked on the site anymore. It wasn't anywhere, but somehow these guys found us through a combination of URL parameters. Nobody else Nobody else was using this. So, this was

a very obvious tell for us when we saw this light up in the logs. Suddenly, this weird search URL has come up, and then it's followed by a bunch of requests to to the images on those pages. That that sort of detection came in. So, it's it it certainly tricky. There's a bit of a sort of detective game of digging into and trying to figure out what's

going on. But, yeah, it can be challenging. Okay. Last question regarding the, you know, regulation. Do you think web scraping should be regulated or anyway restricted in EU or UK? Yeah, I'm I'm I'm not sure about the the web scraping in and of itself, because you know, we do have a lot of laws around copyright and things like that in the first place, which you know, so

I'd be wary of throwing new laws that are specific to one particular type of technology as, you know, as we've seen with the likes of AI over the last year. technological means can change really really quickly. And and regulating closely against that I I don't know. I think it's probably more clear clearer guidelines and clearer enforcement of the existing laws. Copyright, for example, you know, what what

happens if I go to your website and I copy 80% Not all of your content, but 80% of it. You know, where's the fair use line? How much can I copy? What's the difference between me copying it for a reference versus me copying it and then trying to charge someone or build it into my own business, you know? I I think clearer guidelines around that and then

consistency across the European states, I think is important as well, because a lot of these cases when they get decided at at at member level, they're not always fully consistent with each other. So, I think that's probably what I would look for more than uh let's write a bunch of new rules that say you can't run a curl command that does XYZ. Okay. So, the current copying

regulation, do you think it's enough or we should add uh the web scraping uh regulatory framework? I I I think I think the the copyright regulations that we have, there's there's enough in there. Um you know, there there's there's copyright uh sort of precedent in uh say magazines copying information from each other like other media than say digital media. And in a lot of cases, digital media

is treated as just this separate entity that needs its own rules and its own special stuff where there's a lot of this ruling there where you just need clarity to say, "Okay, if in in in a magazine, for example, if I go to Time magazine and I take these three pages and I republish them, I can't do that. That's that's that's just not that's that's not acceptable."

And it's clear that's not acceptable. And and I think it just needs to be clarified from from the digital point of view as well that digital isn't necessarily this special little world over here that's you know, the rules don't quite apply to or maybe they do, maybe they don't. Uh you get this kind of wild west of a let's do whatever we can until the laws catch

up. Uh I I I'm not sure that's that that needs to be the case. Okay. All right. Do we have any questions? No? All right. So, thank you very much. Great. Thank you.