About this talk
This talk addresses the growing challenge of AI crawlers and scrapers that are impacting websites, particularly those built on Drupal. The speakers, Matt West and John Brandenburg, provide insights into how these automated tools operate, their effect on website traffic, and the performance issues they cause. They discuss the types of crawlers, including those that extract structured data for AI training, and how traditional methods of monitoring and mitigating these bots are becoming inadequate. The session also covers various detection and mitigation strategies specific to Drupal sites, such as implementing firewalls, using CDN services, and setting up tailored configurations to limit bot traffic. The speakers emphasize the importance of being proactive in identifying crawling patterns to maintain website performance and accessibility.
Full transcript
All right. Good afternoon, everyone. Um, how do I sound? Am I too loud? Not too bad. Everybody can hear. I see thumbs up. All right. Fantastic. Thank you everyone for coming here this afternoon. We're going to talk about AI crawlers and the damage that they're doing to all of our websites. Um, let's get started. Um, my name is Matt West. I'm uh down here from Rochester, New
York. I've been developing for the web for like I see like 20 years and working with Drupal for about 17 according to Drupal. Um when I'm not on my laptop, I uh enjoy being a part-time farmer and working on old cars. Um I'm co-presenting today with my colleague John Brandenburg. Um I'll let him introduce himself. >> Hi, John Brandenberg also with Forum One. Um I've been doing
web development probably yeah 18 years. My drupal.orgverversary orversary is 14 years in one week. Um, and my hobbies, u, I'm a big Warhammer 40k lore nerd. So, if you want to talk exactly about why horse did a heresy later, I'm down. And, and yeah, back to you. >> Cool. Thanks, John. Um, John and I are engineers at 4M1. 41 is a digital agency that partners with missiondriven
organizations in government and nonprofits. We specialize in building and protecting high impact digital experiences. Um if you have an interesting project, we'd love to hear about it. Please reach out. Um I did leave a card on each desk, each table here, so feel free to grab it. Um and um that's a great way to reach out to us. Um otherwise, Slack, LinkedIn, uh drupal.org, great ways to
reach us. Okay, so here's what we plan to cover this afternoon. Um, we're going to provide some background on the rise of AI crawlers and scrapers, why they're crawling sites so aggressively. Um, we'll talk about how we do crawl detection and mitigation to um, control access from these bots. Um, we'll spend a lot of time talking about some Drupal specific strategies that we've developed. And, um, we
do hope to leave time at the end for questions. If we do run long, John and I would love to chat with anybody out in the hallway. So, um, you we hope to have time for questions. We'd love to hear about your experiences with any of this problem. Um, but if we run out of time, let's, uh, meet out in the hallway. All right, let me, uh,
set the stage here. So, what we would like to talk about this afternoon is, you know, how automated crawls are just destroying different website traffics and infrastructures. Um, we'll talk about a few different things. A lot of these kind of all fall under the same terminology of either a bot or a crawler, scraper, fetcher. There's a lot of different um terms for these tools. Uh, most commonly
crawlers, scrapers are kind of what's used interchangeably these days. Really a crawler is just the bot that does the discovering and navigating kind of like Googlebot, Google uh spider crawler that I think a lot of us are familiar with. And then the the web scraper is the actual script that does the extraction that might be us that might be run as a different process and that's the
part that is often so taxing. That's what pulls all of the HTML text images and other structured data. Um you know crawlers themselves they're they're not necessarily new. you know, Google bot is the most common um I think one that most of us are familiar with and that's been crawling websites since the late 90s. The the crawler bots uh from Google, Bing, and you know, other SEO
bots, those themselves were not too problematic. There were times years ago where they did cause um a lot of problems with server loads, but for the most part, they've sorted themselves out. Um and those bots, they had a clear purpose. They were there to index websites so that you know help their services and most importantly they were relatively polite and they would crawl websites at reasonable rates
and come back at reasonable intervals. And then a few years ago we started to see kind of a change in behavior on traffic the automated traffic that was coming to our clients websites. We saw this rise in like web scraping products from companies that were pulling the structured data. And in the last few years, it's just been an essentially all out allout scramble from what essentially seem
to be AI training processes. And these bots, they're seem to be just trying to ingest the entire internet. Um they're they need to use this content to train their large language models. And it means that they crawl as aggressively as possible. And the traffic volume that that generates is just massive. Especially for a lot of the sites that Drupal is used on like government and nonprofit. These
types of sites are perfect for structured crawls. They're authoritative. They are well organized. They have meaningful data. So it kind of becomes a perfect target for these automated crawlers. So that's why we're seeing sites that have this sudden surge in traffic spikes from Box um that just don't behave like traditional search crawlers. Um the web scraping portion of this has really shifted the landscape of the internet.
Um the internet itself is just experiencing a largecale web scraping problem. It's no longer a niche thing. It's driven by well-funded companies in AI, finance, um SAS providers. They all have these data technologies that they're working on and it's all driven by scraping. So, this technology is now in a phase where it's made the internet in a situation where the vast majority of internet traffic right now,
unfortunately, is automated. It's no longer us and our clients. It's automated scripts basically requesting our websites and crawling across all of the pages. The scraping tech itself is now also pretty sophisticated. You know, leverages cloud hosting services, proxy networks, automated browsers. Um, and the a lot of the communities around developers that are working on web scrapers, a lot of what their discussion topics now are, they're more
they're less about how to actually crawl and scrape websites. They've figured all that problem out. Now they're all working on how to evade detection and how to evade detection at scale. So now all of these automated scrapers are doing things to act and look like real users, real web traffic. I'll call it like the scraping community. Um there is a subreddit that if you're ever curious, go
read our/web scraping. It kind of gives you some insight into I'll call it the other side. Um a lot of the discussion topics are just how to evade being blocked. So, one of the techniques that they'll use is to spoof their user agents. I see some head shaking in the audience, probably because you've already looked at your web server logs and noticed that uh user agents that
look legit are maybe not necessarily legit. And if you looked at your web server logs, you probably have also noticed that these requests come through proxy networks or rotating proxy networks. And that allows a crawl request that somebody initiates to go through a proxy service that will split up their request and use one IP for only two or three different requests. And this will turn into a
crawl event on your website that may come from several hundred thousand different unique IP addresses. This makes it very difficult to detect at like a high level if you just look at your logs. Um, so we'll talk about that a little bit more in a future slide, but I do want to mention that they are in addition to um, spoofing their user agent and, you know, hiding
their source IP address that they're routing through, they're also doing things like leveraging compromised internet devices, which is not necessarily a new thing. I mean, botn nets have been a problem for it almost 20 years now. Um, so they're now wrapped rolled up in this effort. Um there are also uh device farms like in this image. This is just a Raspberry Pi device farm. This is kind
of an example of a very small device farm that somebody had created to run their scraping operation. Um there are also more sophisticated startup companies that provide these types of infrastructure for um these tools. One is a company called Bright Data. They supply a mobile ad network. If you've ever pulled up like a mobile phone game, they have those kind of silly annoying ads. They allow that
service to also be part of a proxy network. Something you may not realize. Your phone might be caught up or people's phones might also be caught up in this. So, this has been difficult to detect. and really the impact of this just cannot be overstated. I I think that this problem has impacted pretty much all of our clients at Forum One. Um I get the sense that
many of you are experiencing this as well. Uh we've probably seen several articles just on different blogs and news articles about different open source communities complaining about how their infrastructure is occasionally crushed by some crawling traffic. Um you know, these bots, they just cause severe performance issues. They'll hit every possible URL combination. You know, if you have a site with exposed facet filters, you know that that's
a large vector for problems. Um, they'll generate thousands of unique URLs. Those unique combinations will bypass caching. Um, so the actual result is the web server infrastructure load goes like 10x in a matter of minutes. And this can last for several days sometimes. Um the biggest effect is clearly a very slow experience for the users of the website. So your content editors, your site maintainers, your actual
visitors of the site that may be bypassing the homepages and other commonly cached pages, they just have a terrible experience. Um so this is of course unacceptable. um you know the high volume of this traffic because these crawlers are often spoofing their user agents it will cause uh analytics to be polluted. So even though Google Analytics does attempt to filter out bot traffic it can't get at
all. So now we are in a situation where it's difficult for us to understand real user behavior on our client websites and difficult for us to make informed decisions. Um, and you know, of course, the hosting overages associated with all the increased data that's served out. This has caused a number of headaches for several different hosting companies because of the support requests for billing and the overages
and all sorts of things like that. Um, you know, this is in effect a a denial of service attack. It's just like a distributed denial of service attack as far as the consequences. Um, you know, from our side at forum 1, it's been a huge distraction for our engineering team. Our developers now are distracted from doing actual feature development and working on product road maps for clients
because we will have to spend hours in the morning trying to figure out why the site is down and working backwards from that. Um, it's been a huge loss in velocity for some of our projects and Okay. So now we talked about impact. Let's talk about how we try and catch aggressive crawlers early. Um the tricky part for us has been spotting the problem before the site
breaks before our clients report problems. So in this next section uh we'll dive into a little bit of the tools and methods that um we've been using to try and detect bot crawlers crawlers excuse me. Um, you know, in a previous slide, I mentioned how web scrapers are working to just evade detection and this does make it tricky to try and find out the source of what's
going on. Usually, we get tipped off that there's a problem, very low tech, a user, somebody from, you know, the the client says, "Hey, the site feels slow this morning." And um, you know, they're someone that we always listen to. You know, they are in editing their site, working on their site, editing pages. they have a sense for what the Drupal site performance feels like. You know,
I know that the backend experience of a very large Drupal site is maybe not the most performant, but they they have a sense of what feels slower than normal. So, we try to respond um quickly when we hear that report um that just something feels off. We also, you know, fortunately, we have access to some more sophisticated actual monitoring tools. You know, we have access to New
Relic and Blackfire.io IO through our hosting providers with Pantheon and Upson and Aquia and others. So we have created rules uh with like that can monitor transaction times and load levels and send us pings either on Slack or through email. That's one way that we kind of try to get ahead of things. We also have generic server status monitoring and that's been useful during the middle of
the night when one of these crawls might happen and the performance of the site is degraded. We'll get those pings and then when we arrive at our desk in the morning we can see that something is probably going on um hopefully before a client somebody else notices. Um we have had to really be uh deliberate at spending a lot of time in our raw web server logs.
They're difficult to parse especially if you're on a load manage or load balance environment. We have had to become very intimate with um Apache and EngineX web server logs so that we get an understanding one when there is a crawl event what's going on but beforehand to know what the baseline actual usage of uh our client sites are. Tools like go access are helpful for this. They'll
allow you they'll parse the the the raw logs and give you a visual representation. John has a few slides that he'll share with you later on that um articulates this a little better. But those types of tools have been very helpful for us so that we can do uh more accurate detection. What I really want to hammer home here is that we try very hard to be
proactive understanding how our logs look normally, checking them somewhat regularly, you know, as time allows. It does take time to do this, but if we can capture some odd behavior early on, um it just prevents compounding problems because maybe like you, it's not always just one crawl event that's a problem. Sometimes there's several happening at once and it really makes it challenging to juggle when a site's
being crushed. Here is an example of something that we commonly see in our web server logs. I see some head shaking. I'm not surprised to see this. This seems to be a very common attack vector. Um, this is tough to read in the slide up here, but what this represents is a web request to a search API and to a search page that has facets installed. This
is something that has kind of been resolved in the version three of facets, but there's still a lot of activity out there from store requests. What this is showing is a get request that has a query string that's incredibly long and complex. contains complex combination of search filters. Um, this is specific to to versions previous before version three of facets and um, this just has a number
of different combinations of facet filters and it's all the way down on page 12 of the search results. So, we know we look at this that this is an odd request. We just from this particular client site, we know that most clients tend to not select this many different combinations of filters. they certainly don't go to page 12. Um, because we have an understanding of roughly what
our logs normally look like. This immediately stands out as an outlier. Um, so this is an this is a very common example of a bot that's like stuck in a bot trap with you know essentially an accidental bot trap um when it's trying to crawl a faceted search page. So what's important about this is because of the parameters it'll bypass any CDN caching and because there's dozens
hundreds thousands of these coming in per hour. This is what really loads up uh a lot of our CL this is an example of one thing that really loads up our clients sites. Um this is what that request would look like exploded just to maybe help illustrate what's happening here. Um these are all these different filters that are selected. Again, we know that on this particular website,
most users just do not behave this way. Um, you might also notice that if if you were to look at something similar in your log, you might see filters here that don't even exist. Um, certain sorts that aren't available to the user. Um, so a lot of this just comes back to being familiar with what your website looks like normally and what actual legitimate users look like
because um, as we'll talk about here in a bit, we can no longer rely on IP address, user agent coming some of these traditional um, flags and in web Here's one that I wanted to illustrate uh, on a separate slide. This is a trai what I'll call traditional DDoS. Um this comes up sometimes um you know how is this different than a traditional DDoS? Sometimes we've had
people on our team uh you know look at pre like the example of a previous slide where a site is being crushed by uh requests to different facet filters and they will think that the site is under attack being DDoS. We have some nonprofit clients that are very sensitive to um you know these types of security attacks. So we have had to be a little more mindful
to inform them like no you're not necessarily being attacked although it's the same consequence. Um so this request here is an example of from one of our actual clients one of actual uh one of our actual projects of what an actual DOS looks like. Again it's just a bunch of parameters. They make no sense but they'll bypass it and cach it. Okay so with detection cover let's
talk about mitigation. Um, we have a few different defensive strategies that we use. Um, I'm going to cover a first uh a couple of slides in this section and then I'm going to pass it off to John. John has a lot of Drupal specific mitigation techniques that I think are very useful. for us the first step in the process is having some firewall, some CDN if possible.
I mean, the the best way to prevent a load on your server is to prevent a request to that server. Um, I'll try to not make this an advertisement for Cloudflare, but Cloudflare is fantastic if you have access and the ability. I, you know, Cloudflare has some user interface tools that allow creating rules, see what's going on, analyzing requests coming in from different networks, ranges, types of
clients, and where they're headed. very easily, very easy, very, very quickly. And so a lot of our ability to respond to these crawl events and reduce the load on our infrastructure has been to quickly switch over to either deploy Cloudflare if it isn't already deployed for a client or jump onto Cloudflare and uh review rules and modify rules as they see fit. This kind of depends on
your client and your authorizing environment. Um, you know, we don't always have access to Cloudflare as maybe Drupal engineers. We maybe don't have access to the team that manages Cloudflare if there is a Cloudflare account. Um, there are a lot of uh, you know, potential walls to getting this set up. Um, however, uh, John has some solutions for that uh, from the Drupal side of things. But
if there is a tool in place, whether it be doesn't have to just be Cloudflare, Fastley, Okami, any W is really your first line of defense. Um, you know, Cloudflare is handy. They have that are you a human challenge. The nice thing about that tool is it doesn't necessarily block requests. You might have visitors that are in a region that's producing a lot of, let's call it,
malicious automated traffic, and we can't just block them, unfortunately. So, um, the manage challenge is a quick way to really snuff out, uh, or at least slow down, at least for now, slow down a lot of this automated traffic. Um, when creating rules for these, I mentioned earlier, we can no longer create rules for specific IP addresses or even subnets because of the availability of rotating proxy
services that a lot of these web scrapers use. Um, if you feel that your website is being crushed with traffic, it's just running very sluggish, and you were to take your web server logs, parse it by an endpoint or a search page that you feel is being hammered over and over again, and sorted by IP address, you would get nothing useful. Um what is more useful is
to try and identify either regions or specific networks that are uh at least the current source of whatever's happening on your site. Uh this is where a tool like cloud for is very helpful because they have tools built in that'll help you identify which network and they identify by ASN number. Uh so you can very quickly create a rule um to manage traffic at that level. Okay.
One of the ways that the CDN services and the W services can actually effectively uh deploy these tools is they do a thing called threat profiling. Um and threat profiling is essentially just taking one of the ways they do this is they take the all the request headers that are coming in and they finger they hash them and create a fingerprint and um this is also sometimes
known as TLS fingerprinting. This is handy because user agents are now meaningless. Um, and there are more there's more information in the request header that can be used to develop patterns. So, um, Cloudflare is very handy. They have a very rich database of fingerprints, but you know, Okami Fastley, they're all having to develop these same tools and capabilities. Okay, I'm going to hand it over to John.
John's gonna talk about some more Drupal specific tools. Thanks, Matt. Uh, so really this boils down to two goals. Uh, reduce the opportunities bots have to crawl your site and make it as cheap as possible to serve requests that do get through. Uh, I'll start with one of the classics, a a well-crafted robots txt file. Um, it's still critically important, even uh, more so now. Uh, for
well- behaved bots, it tells them where and how they can crawl your site. Um, you'll want to customize the one that ships with Drupal core. U, I have a few specific recommendations for this. Uh, add a crawl delay of 30, which is the highest value that's recognized by some crawlers like Bingbot. Um, and add disallow patterns for uh, dynamic pages. Uh, facet search listing pages, pages with
URL parameters, anything that can generate a huge number of unique URLs. uh and include your sitemap so bots have a structured uh cheaper alternative to crawling everything directly. Uh note that sitemap references need an absolute URL uh not relative ones starting with a slash. Um it's worth mentioning robots txt won't stop every bot. Uh a lot of what we've dealt with over the past year involve what
CDN providers call browser impersonator bots. Uh by their nature they don't follow the rules. Uh that's why you also need to enforce these boundaries uh at some infrastructure level u which we'll uh cover in the next several slides. Um okay page cache. Um what can I say? I mean this is a a Drupal conference with a lot of developers. I think you should know this but um
make sure it's on. Make sure you know what your max age is set to. Um I know it sounds basic but uh it's easy to inherit a project and you don't know if best practices were followed um there. So, it doesn't hurt to check. I could literally talk about caching for hours, so I'm going to just leave it at that. Um, even with caching in place, uh,
bots can do damage to your site. Um, generating a large number of unique URLs. Uh, facets search with link style facets is the classic example. Every combination of filters produce a distinct URL and bots will crawl all of them. Uh, one real example, a client of mine had a related content block at the bottom of every one of their article pages. Um, this block had a pager
on it, uh, that had no limit. Uh, so we were getting bots crawling, you know, page 1,00 on every single individual article. So that just, uh, exploded the number of requests there. So, uh, we removed the the pager and, you know, that helped a lot. Um, but there's a pattern that we've noticed where uh these URLs tend to get cached and they'll come revisit them over time,
so even months later. Um, I think we had a site that I had fixed one of these problematic components back in August and we were seeing it get hit in January. Um, and one last tip, review your access log regularly. Uh the Go Access uh tool has a a no query flag that will aggregate URLs um with different query parameters together. Um this makes it much easier
to spot which pages are tracking bot traffic. Um since we've mentioned go access a couple times now, I just wanted to show a quick example. Um you can just run that command on your log file and it will spit out a report. Um it comes with tons of options like generating a CSV instead and you can use that for some other analyses. Um, and here's an example
of the difference running uh, go access with a no query option can do. Uh, both reports are from the same site in the same time frame. Uh, the first one doesn't seem like there's any problem at all. Uh, the second one runs a report using that no query option and uh, because the requests were so distributed across unique variations, they didn't appear in the top 10 results.
U, but when you strip out that problem, the the problematic pages become clear. Um, there's a funny anecdote about this site in particular. Um it was a high school uh and someone had renamed that events page to state baseball uh to recognize their baseball team had gone to finals at the state and uh that created another version of that page that now the bots were cataloging all
the URLs from that page. So we changed it back but um that's example where we you know after we changed it back we saw a crawl event happen um on that now 404 page uh months Um and here's a little trick I use. I mean if you're a a contract developer uh like us running sites on many different platforms um then you'll uh you know go access
will break uh if their log format configuration isn't correct. Um, so what I do is I just have these different versions of Go Access RC files, which is how you store your your configuration. Um, and then I just copy it over to uh Go Access RC and that lets me uh parse the log for that platform uh when I need it. Um, yeah, and if anyone else
has any tricks using Go Access, uh, we'd love to hear Um, so now I'm going to dive into some purely Drupal specific tools for handling bot traffic. Um, I do want to say that a W CDN tool or service should be your first choice here. Uh, but I understand that's not always an option. Um, bed ramp compliant versions are very expensive and and if you figured out
how to get exceptions for that, I'd love to hear it. Um but even so these uh uh even so these work with in great in addition to those services like Cloudflare because uh maybe that's in place but like Matt just mentioned it might be difficult to get a hold of the IT team who manages your client's Cloudflare so here's where Drupal Config matters. Uh like I mentioned
earlier this boils down to two categories. uh make serving anonymous requests as cheap for you as possible. Uh and fix any places that provide too much variation on the site. Uh one thing to know is a lot of hosting providers are counting hits to your site based on the response you provide. So generally 400 level responses don't count towards your entitlements. Um but that will vary by
the provider. So be sure you know what your your hosting um is is doing. Um now these tactics aren't specific to AI bots per se. uh much of the problem we're seeing are from these browser impersonation bots um which we're assuming are out there collecting data to train models but that is an assumption I think a good one but um so I'll be covering some general tips
for maintaining a wellp performing site um even when it's getting hit by flood traffic I don't want to spend too much time on the performance side here uh since many of these are all the same lessons that have always applied to making Drupal a wellperforming site Right. And I'm sure there are other presentations that cover that. Um, performance best practices. What I mean here, um, I'm not
talking about your Lighthouse score. Um, Drupal bootstrapping Drupal is expensive. Um, and when handling dozens of requests a second, uh, really to scale, you need to be either serving from cache or finding ways to respond to requests as quickly as possible. Um, I recommend occasionally checking your site's time to first bite uh, to see how long it takes to build a page. Uh, you might be surprised.
I was surprised the other day when I learned one of my client sites had a two second time to first bite for anonymous users. And it turned out that there was this interaction between Cloudflare and their F5 security layer um that was causing this. Um small things could be breaking your page cache and you don't even realize it. U I once discovered that the search bar, which
we render on every page on one site, doesn't get its render cache reused uh between pages. And I think that might even still be an issue. I'm not sure. haven't looked into it. Um or perhaps uh you have some special access rules to a page that appears in your menu. Um that could be breaking your render cache. Um this can happen with modules like domain access. Um
that's why it's important uh to use the dynamic page cache module in Drupal. Uh but also make sure all the components you think are caching really indeed are. Um now from here I'm going to go over some modules to use. Um, perhaps the cheapest way to handle a bot is just to send them a hard-coded access denied page. Um, there's several ways to do this, actually banning
the the IP address using the the core band module that ships with Drupal Core. Um, this is being removed in Drupal 12, and to be honest, the core band module um is a bit too simple for this. Uh, you can only use it to ban individual IP addresses, and this is kind of a losing battle of whack-a-ole. Um, that's why I recommend the advanced ban module. Um,
this allows you to ban IP ranges and lets you set an expiration to those bands so they aren't really permanent. Um, I'm on the fence about permanent bans really. Um, if attacks are coming from these temporary proxy IPs like Matt mentioned, uh, you might not really be banning the actual bot source. Um, uh, you also need to be careful when using any IP address bands. Um, if
you have a proxy like Cloudflare set up, uh, you could accidentally ban Cloudflare instead. Um, and then no one could see your site. Uh, so if you're using a CDN or a W service, uh, make sure you have that set up in settings.php. Uh, in this example, I should note that this should only be used if you trust that remote address value. Um, some setups might be
locked down and only accepting uh, connections from their CDN or proxy. Um uh but uh you can also use IPs uh hardcoded here as well. Um even if you aren't using bands uh make sure you have this set up accur uh make sure even if you aren't using this to not ban Cloudflare um it's good to have this set up correctly because your access or your Drupal
logs or web form logs or submission logs uh will be correct. I learned about the perimeter defense module uh last year and it quickly became one of my new favorite modules. Um it allows you to set a list of paths that would be hit by bots probing for vulnerabilities and ban them after a certain threshold. Um we've all seen this in our access logs before. Um why
are these WP admin paths, you know, popping up on my Drupal site, right? Um if you ever looked at access logs and you know this has just been happening in the background for years. Um, but even if if your site is secure, the constant 404s that these would be generating is expensive. Um, so I wouldn't have worried about this like a few years ago, but uh now
my my patience is getting short and uh I'm clamping down on on this kind of stuff. Um, on the screen is an example uh or a sample of the list I use with this module. And um I've actually have you know my personal blog site I've posted a list of the full uh perimeter defense configuration that I use. Um, oh, and u I found that generative AI
is pretty good at combining these into the regular expressions uh to to make a shorter list. Uh, crawler rate limit will allow you to set uh limits per for IPs and bots uh letting you respond with a 429 or too many requests uh code. Um, it handles bots differently as it defines those as specific user agent and IP combinations. Um, it'll even let you do ASN or
country level rate limiting, which I I don't even think that's a feature that a lot of CDN or W services can do. Um, but that's if you use uh use it with the GOIP database integration. Um, I've used the free version of the MaxMine database before and I've had some limited success with that. Um, I I'm guessing the free version just isn't as complete or uh up
to-date as a as a paid version. Um, my only feedback on this module is that I wish it gave you some visual metrics on the back end uh of it so you can see like how well it's performing. Um, otherwise you need to review your access logs to see what percentage of of requests uh receive a 429 response which is kind of an unusual uh code to
see coming out of your uh Drupal application. So here are a couple of modules that I have not used myself. So I didn't feel like I could uh dedicate whole slides to each of them. Uh but they do seem interesting. Uh challenge mitigation. Uh one of my co-workers is using it and uh they said that they've had solid success with it and that is a direct quote.
Uh it appears to generate an allow list of IPs uh once a user passes some sort of challenge. Um and the autoban module also seems intriguing. It works by matching patterns in your database log um and then translates that to to banning and that IP address. I haven't quite thought of a way to integrate that into my own sites, but I just felt like it was worth
mentioning. Um I think I'm essentially doing what I would be doing with that in the perimeter defense module. Um you know, where a 404 to some vulnerable path or potentially vulnerable path uh would become a band. Um, and if you have any modules that that you like to use in this area, we'd love to hear about that during uh Q&A. Um, now this brings me to a
couple of my own modules. Um, the facet bot blocker module was a a creation to handle the issue of bots endlessly crawling your search facets um or search facet pages. Uh this module allows you to define a a limit to uh set for the number of facets that can be used at once and will block any requests that meet or exceed that limit. Um you do need
to be careful here and ask like how many filters would a real human use. Um personally like I would only ever use a handful. Um and like the log example earlier, bots can get this number pretty high. I think the highest I've seen was like 23 or 24. Um, honestly, the fat bot the facet bot blocker module was a bit of a hasty and blunt solution to
an urgent problem that came to a head about a year ago. Um, if you haven't migrated to facets version 3 and move to placing your your search facets in an exposed form. Um, I really recommend you prioritize that. Um, I've done that uh on a couple sites now and it wasn't, you know, quite as heavy of lift um as I was dreading. Um it did take some
you know retheing of those updated uh versions of those uh forms but you know it it wasn't that that bad. Uh I should also note again that even if you remove these components uh on a page that that produce these variable URLs uh you can still see traffic for many months afterwards um since they seem to cach these URLs and revisit them later. Um it seems like
these bots Yeah. Um, it's definitely still worth doing though, um, as eventually I think these requests fade away. Uh, the bot blocker module attempts to fill the gap between um, this is likely a request from a browser impersonation bot u, but they aren't stacking our stuck in our faceted search page links. Um, because you know bots can crawl the rest of your site too. Uh the module
allows you to block requests that contain specific substrings uh within their user agent header uh as well as blocking requests from uh self-identified uh from as older browser versions uh according to their user agent. Um since we've observed a pattern of of these requests coming from very old browsers. I think the record for the oldest I've seen was Chrome 23 which came out in 2012. Um, if
you're curious about the question like what browser versions should I block, uh, the answer is a bit more nuanced than you might think. Um, I did some research with some AI assistants recently, um, just to answer that, uh, exact question and I'll come back to that in a moment. Um, but right now I wanted to point out, uh, that, you know, this module does, uh, seem very
effective at hardening sites against DOS attacks. Um, case in point, uh, and I have a bigger version of this graph on the next slide. uh uh the client site I used to develop the block bot blocker module on uh just happened to be experiencing a DOS attack when I was going to deploy it. Um it was uh you know literally I I pulled up the the Aqua
dashboard saw that we're experiencing a crawl event and you know normally in cases like this I would abort the deployment but since this was a bot mitigation module I thought hey why not go for it and uh yeah so I I went with it and the attack pretty much stopped immediately. Um you can see from the bottom graph where uh there was a spike in the 400
level responses and then that's the bot blocker module taking effect. Um and the request just stopped altogether. Um in fact that traditional DOS attack example Matt showed earlier uh was from this incident. Um all from serving a cheap hard-coded 403 page uh to requests coming from old browsers. Um and here's a closer view of that chart. Uh you can see the 200 responses actually went lower to
even before the incident. Uh so I think there's a lot of background bot traffic uh that also ended up getting Uh now I was going to I said I was going to come back to this and I want to acknowledge the challenge of false pauses with this approach. Uh what bot what bot blocker does is blunt. Um and answering the question of you know what browsers are
are used by real people is nuanced. Um, in this day and age of automatic updates, you might think that people are likely receiving those. Uh, but there are some legitimate use cases for people being on older browsers. Um, Safari is pinned to the iOS version. Uh, many of these are still supported. Uh, Chrome 119 I think was the last version available for Windows 7 and there might
still some people on out there on that. Um, so this is what led me to consider a new approach for the bot blocker module. And last evening I pushed a 2.0.x branch uh to the modules repository that attempts to look at more data data points that identify bots. Um, so bot blocker 2.0. Um, I've had this version of bot blocker running uh on one of my client
websites for the past two weeks or so. Um, and this site I've already migrated to facets 3. So I thought that the bot problem had largely been addressed. Um I can say from the dashboard metrics that uh you know it was blocking about 13% of requests 13.7 on there. Um however while rehearsing this presentation this morning I checked in on the the metrics there and that jumped
to 40%. >> Wow. >> And that was including the data that started on 311. So, uh, you know, bringing that average up, uh, to 40% from 13% inclusive of that time frame, uh, I think there was just a very significant crawl event that happened since I last looked. Um, and since I've been preparing for this presentation, I haven't had time to actually look into what what was
going on here. Uh, I was asking one of my colleagues at back at Forum1 to to take a look at it for me. so yeah like I said this this module works this version of the module works at uh looking at a wider range of data points um and building a threat score based on those uh similar to what uh some of these W CDN services might
be doing. Um here's a sample of the data points that the 2.0 version of the module is looking at and an example of how the admin form looks for that. Um you can toggle which items are enabled and assign a score to each. um they come with a set of defaults. Um you ask me how I came up with those defaults, it's AI. Um so for example,
the the pragma header is present when you do a hard refresh of a a page. Um this would give you five points. So not enough to block you altogether, but you know, we're building up a score here. Um the threshold is set to 50 by default. Uh but you can change that if you like if you feel like it's blocking too many legitimate bots. Um, a couple
of items are disabled by default, like having proxy headers, uh, which you would expect if you are using a CDN. Um, but, uh, and I expect that to be case in most setups. Um, uh, it's still just a dev branch, not a full release. Um, I'm hoping to get some feedback before tagging a a 2.0 release. Um, sadly, there are several items that you just can't do
in within Drupal or PHP. Um, TLS fingerprinting Matt mentioned earlier. Uh really that needs to be done where HTTPS is being terminated. Um that's either going to be like in your actual web server itself um or your load balancer or CDN or you can do that in a lot of places. Um and identifying IPs that are part of like a internet as a service provider like one
of these bot networks that that Matt was talking about. Um you just need access to a large and current database of IPs. uh which you may not um be able to you know and you might be able to get that with like the paid version of that MaxMind database um but who knows uh the 2.0 version of the module does utilize a a plug-in system so uh
maybe someone can make a module Um so I want to take this opportunity to announce another module that I've been experimenting with over the past year. Um I'm calling it ban network and I'd like to give some thanks to Mr. Claude for the help on all this. Um the idea is that if you uh manage several sites uh you may want to share the IP band list
between them. Um ban network sets up an API that allows you to do this while keeping the band list itself secure um as it does require an API key. Uh so despite this meme, I do still highly recommend a W service uh because even though we've been talking about handling these bot requests uh by serving an access denied page or making sure your site performs well, um
what would stop someone from making a million requests to your, you know, varnish cache, the varnish cache version of your homepage? Um and that, you know, if you're responding to with a 200 code to those, like that still counts towards your entitlements according to a lot of uh hosting providers. Um, so my apologies for ending this section of the presentation on a more pessimistic note, but I
do believe in the adage that attacks against you only get better over time. Uh, my concern is that bots will be wising up to how we're blocking them and fixing those gaps. Um, so I think we really need to reconsider um that and how our sites are are dealing with, you know, interacting with these bots. So I'm going to pass it back over to Matt who's going
to talk precisely about Okay. Um, so we've spent a lot of time talking about keeping bots and crawlers out of your website. Um, but I want to pivot a little bit the few minutes that we have left and discuss kind of a tension that we have across this whole topic, which is actual content accessibility for some of these tools that we might want to uh access our
clients sites. Um, this is like attention. We we have content on a lot of our client sites that is a high value. These are state, federal government, nonprofits, and we want those to be the source of truth that's consumed and used in some of these large language models. It's clear that this is what everybody wants to use. They have a lot of power. um we just have
to kind of be strategic on how we define discoverability, not necessarily just be defensive. So, it's kind of a back and forth. One thing that we didn't really talk about during this session was like good bot versus badbot, good crawler versus bad crawler cuz it's all still kind of up in the air on who's who. Last summer, Claudebot was destroying several of our client sites and now
we love Claude. Um, so these things can change and you know that's an example of a bot that is identifying itself. A lot of these are not. They're offiscating their tracks through proxies and other services. Um, so I just wanted to make a point to mention that like we try to think about our clients and their the citizen phasing content and and the the content that they
make available. We just want to make sure that we're not totally closing the doors and our clients and our stakeholders that we talk to, they are very mindful of this. Um, this causes a number of challenges both with addressing of load problem that's coming from swarms of automated traffic. Some of our clients are resistant to apply any type of rules or traffic shaping to begin with um
because they don't want to just shut the door on some of these technologies. So we've had to learn to use certain vocabularies and techniques and you know a lot of that is really like if the website is not accessible for anybody because it's being crushed by automated traffic that's doing no service any good. Um, so you know there's also a small bit of data that suggests that
some of these AI generated overviews like the Google generated search results actually boosting site traffic back to client sites. Some of that data is still up in the air, but let's assume that it does help to allow our sites to be crawled by some of these automated crawlers, some of these AI tools. Um, you know, I would just say that we're just trying to be very selective
with our tools. We don't just block countries, regions. We try to apply rules based on what's happening today. Check it again in a week, massage the rule, loosen it if it's no longer an issue. Um, so again, key is being selective. Allow access to stable and capturable stuff. Maybe block or apply rules to specifically search pages that are a common vector for attacks. That's been um helpful
for us to just not apply blanket rules. Um, but I'll I'll acknowledge it's been a real challenge to communicate the strategy sometimes to uh clients, stakeholders, and other decision makers. So, okay, we've covered a lot of ground today. Um, before I open it up to questions, I'm going to quickly just recap what we've covered. Um, some of the things we'd like you to take away is, you
know, the bottom line is it's unlikely these crawlers are going to just stop and go away anytime soon. Um, we I haven't found or we haven't found that the source of this is necessarily malicious, but it's the same essentially as a as a distributed denial of service attack at times where this traffic is just so overwhelming on a lot of our client sites that it's a disruptive
service for them and the other actual legitimate visitors. Um, so just be mindful your website, your client's websites, they require defensive architecture. This means that any layers of caching and sophisticated bot management is important. You want to be intentional with information design, specifically how you expose certain links and if you're going to use faceted search, be mindful of that. Um, finally, I'm just going to say again,
try to be proactive. be be aware of on a normal day, a normal traffic day, what load on your machines look like, what types of visitors are requesting your site, what legitimate actual traffic looks like, so that you can respond quickly. And if you need to make uh if you need to talk to like your DevOps or security folks, if they're on a separate team, um you
can do that intelligently. And if you're struggling to, we've offered you some Drupal configuration tips that should hopefully help you kind of manage the fire on your side. until some other action can be taken. Okay, so we have a minute or two. Um, I'm going to open the floor for questions for a couple minutes. We are at 50 minutes, but I don't think anybody has to come
in here until 3:00, so we can hang around for a bit. If we got to leave, then we can. Um, okay. I see one hand up. If you could just please try and say it loudly and I'll >> repeat the question and I'll repeat. >> You guys mentioned uh views of pages. Uh have you had any experience with uh infinite scroll or like the infinite load pages?
Uh how what's that behavior look like with bots that they scroll inherently and find more? They don't get parameters, right? They can't load it that way. I I don't understand if or how they interact with that. >> I don't know if you have an answer to this. I personally haven't noticed anything specific to an infinite scroll. Um the result is the same, you know, in our request
logs and we see those the example that I had up from a facet request. It's on like page 12. Um it would be the same result, but it hasn't I haven't found that to be an example of a an actual um called a honeypot. I guess unintentional honeypot. Have you is this something you've seen? >> Nick and I have some. I've just I haven't noticed anything detrimental.
>> I have also mentioned one of the first security roles I did because group of sites I just inherently blocked anything with WP admin in it because I saw no point in even entertaining the traffic. Um but uh Cloudflare there's the Cloudflare module that you can install that restores the original IP according to its description. >> I did the settings PHP way you did. I don't know
if there's any advantage to either way of doing it. >> Anything that we can manage at Cloudflare is the best. I'll add that if you have access to Cloudflare and its WFT tools, the core CloudFlare rules uh for like mitigating um like I forget the the the fuzz scanning, they often will capture some of those WordPress exploits that are you know it wouldn't block access to WP
admin of course, but it would uh block some of those fuzz scans that um like the perimeter module could also help with. for the advanced IP band >> so the question was for the advanced IP ban module what length of time would you recommend for IPs um I don't know if you have any insight but a week a week is fine um the trouble with the IP
banning is that it's often coming from well I shouldn't say often in my experience it often comes from a proxy service And the easiest way to kind of make an informed decision is do a who is lookup. Get a sense of where it's coming from, whether it's residential or not. Um, and again, know your traffic. If you are managing a website for the state of Virginia and
the request is coming from Brazil. >> Yes. >> You know, uh, do you have any sense of what hosting companies might be doing to provide a better service to block these for clients? Is that something they're just leaving up to clients? I know some offer a CDN for pay, which a money maker now, but do you have a sense of what's going on there? >> Um, I
don't know if you have anything to add. The question was, what hosting companies, what are hosting companies potentially doing to help with this? I don't know if we have any Aqua or Pantheon folks in the room. Um, we were speaking with a Pantheon rep yesterday and this is been no short of a pain for them. um they are trying to evolve and roll out some tools on
their end to make this a little easier so that I think their support staff is not just quickly refreshing varnish varnish rules um to kind of play whack-a-ole on regions or not. Um I don't know if you have >> Yeah, I think the they're just as interested in in seeing modules like these uh get out there as we are. Um, I was I I was talking to
our Pantheon rep yesterday and uh she was asking if we could uh create a WordPress plug-in uh for the bot blocker module and I'm like I'm not a WordPress guy, but um I I maybe we'll talk to one of our folks at form one. >> Um Yes, sir. Uh my understanding uh with like these WordPress uh URLs is by blocking them in the ht access file, we're
not even going to bootstrap Drupal. So that's going to help with your performance. So that's kind of been my process. Uh during the upgrades, I look at the page not found. Anything that I know is should never be in to Drupal, then I move that to HD access. Does that seem like a good process? So I think the question was um there's a lot of blocking that's
done in varnish based on seeing 404s in watchdog and the question was whether that's good or not. I think that's absolutely good. um you know if you don't have access quick access to manipulate uh W rule tweaking a varnish rule if you if you're on like Upson you can quickly change sort of quickly change the fastly VCL anything that is blocked before it gets to the Drupal
request is fantastic based on any any rules any logic if you want to block certain commonly requested endpoints like something that's a WordPressbased route if you're clear if you're hosting a Drupal site that'd be great if you want to block um you know a a request to like a search page that was previously getting hammered and you've changed the URL path. Anything that you block anywhere before
it gets to Drupal or mitigate before it gets to Drupal is is good. just just to add I mean for HD access specifically I you're talking about like a single file and you know with a lot of hosted providers you need to you know deploy that via git or or some other uh process and so uh and you can't really like you know put in a million
IPs in your HD access file to to handle that. So I think there's a scalable issue uh with managing that managing that in ht access specifically but yeah again like using a w service where you can you know change the configuration on the fly uh there is is going to help you a lot um the idea with a like bot blocker and and that module you can
you know make a lot of changes again on the fly uh that way um yeah it'll be managed through like exported configuration and deployments in Drupal but you can change those if you absolutely need to at at that moment. Uh go ahead again or >> just for for nonprofits. Uh well uh Cloudflare does have something called Project Galileo which does provide free business tier service for certain
nonprofits. So it's worth pursuing if it's like if cost is an issue. >> Cloudflare like you said uh provides a nonprofit tier um for users. I know I work with Smithsonian and they uh recently moved to to Cloudflare and like a lot of my other federal clients can't use Cloudflare because the federant client version is way too expensive for them to use and uh I haven't figured
out how to get an exception for them in those cases but just in that kind of area between you know federal agency and institution and you know I think they it's more gray with them but um yeah yeah good point um I saw some other hand >> this man in the back here >> you trieding the AI lab >> uh a cloudflare now well they have announced
the tool called AI labyrinth which is like a honeypot the question was whether we've used it I have not used it has anybody in the room used the that tool yet I'd be curious to hear if it's working or I actually don't know if it's even publicly available >> is it yeah >> yeah my my my cloudflare clients are managed by their IT teams which I don't
get to talk directly and that's a churn as a contractor ctor like you're dealing with the your clients and you can't get a hold of the people that are control these things easily. So, uh we we've been working with uh one of my clients to enable turnstyle um on one of their sites which is like a a service that Cloudflare uses that actually rotates uh different strategies
for dealing with with uh bots and other malicious traffic. Um, but uh that's that's one tool. But yeah, Labyrinth is is interesting. Yes. >> I just want to say thank you for confirming uh the things that during my investigations were coming up like you know I was in a bubble the whole time probably like five months ago. I was freaking out. And so, um, I ended up
using the W and I ended up just like four weeks ago noticing that the user agents were ridiculous and I was like, what? And once I latched on to that, it was like game changer, right? So, >> you can really go down. So, that there wasn't it was just a comment about how this is been a headache for a while. You could really go down a rabbit
hole trying to understand the user agents in your web requests. >> Absolutely. I don't know how much value there is to it at this point. Um you know, John has done some really interesting investigation to just kind of lay the landscape of what current user agents are used what and where. But because there's so many different internet devices out there like TVs that don't get updated, who
knows what, you know, as soon as that batch gets compromised and becomes part of a botnet, it's >> my light bulbs are connected to the >> and it's a moving target. >> It and it can be spoofed. I mean, that's the limitation of that layer of technology is that the user agent string much like the referral string and um can be spoofed. So >> that's around rate
limiting a lot too. Of course, you know, it's it's a sensitive thing. So, you got to understand what the CMS needs in terms of like um well, parameters even, right? The amount of parameters that are being sent around. >> So, preI when SEO was important, um the meta description tag is what you jam your 180 characters in to to get, you know, search result. Is there a
way to I'm asking from the perspective of an authenticated website where we payw wall a lot of our content. Is there a way to spoon feed AI the gist of your content without giving it the payw wall content? I don't know. I feel like there's no standard yet really to it. But I don't know you've seen a trend where there's a way to give AI here's the
gist of the article without giving the article away. >> Yeah. So I think the question is is there a way to feed a preview to AI for an application that maybe is paywalled or loginwalled. Um there does seem to be a few different initiatives that are floating out there. LLM.txt is sort of being I don't know it that that is a something that's being maybe promoted loosely.
There's markdown extensions to files. That is another potential route instead of your URL that's normally accessible and read and parsed by a web browser that we browse. Um, some sites are just adding markdown support. So, the same web request followed by MD. It's just a markdown version of the actual content of the site. No HTML or anything. I don't know if John, if anything. >> Um, I'm
working with a couple of my clients to add schema.org or uh markup to their sites. Um so there there's a a a schema.org metatag module uh that you can add and that complements the existing metatag module to add uh JSON LD I forget what that stands for. It's like a some flavor of JSON uh to your your uh metad information. Um, and I I think the idea
is that like that isn't being read directly b by the AIS, but the tools that are being used to feed and train AIs um do understand that stuff. Um, but that's that's just a theory. I don't really know how they they're working under the hood. >> It was easy and understandable SEO days and now there's so many flavors of AI. I assumed there wasn't a standard but
I assume someday >> we are actively trying to figure this out for our clients because you know we've kind of spent the last decade or two trying to optimize all of our projects to be SEO friendly so that our client sites can rank highly and now the new modern frontier we think is trying to optimize our sites for these large language these other products that people are
using like claw and chatgbt. Um but it the landscape definitely seems to be evolving almost weekly. Yeah, we're uh at forum one we're offering new services like doing like AI I don't know if it's called you call it SEO anymore but you know AI optimize optimization you like testing where we will periodically like go check uh you know certain questions posed to LLM to see if we're
getting back responses that contain references to our client sites um and those kinds of audits. I I don't know how we're phrasing or calling those. Um, >> cool. Uh, in the back, sir. >> Yes, I I I love fast 404. It's a great module. Um, I didn't mention it in the presentation. I think I had one bullet point on one slide somewhere that included it. But yeah,
fast 404 is great. Um, it was actually a big inspiration uh for the uh bot blocker and facet bot blocker modules because of how it handles like an early uh runtime uh like event listener uh to insert this basically a hardcoded response uh that's cheap for you to produce. So yeah, fast 44 is great. I love But I need to install it on my own site uh
for my blog because I migrated to really cheap hosting last year and now it falls over at the drop of a hat and I have to go in and like restart my SQL all the time and uh I think it's from generating a bunch of 404s on these bot pages. Um cool. Well, if anybody else wants to chat about this, please see John and I here online
in the hallway anywhere else. Um, we really appreciate you all coming uh this afternoon and thank you for giving us an opportunity to present
