About this talk
This talk examines the rise of AI-powered crawlers that scrape public government websites, highlighting their impact on server performance and infrastructure. The speakers explore how these crawlers operate, the patterns to identify, and the potential consequences when a site's resources are overwhelmed. They provide actionable solutions to detect, throttle, or block these crawlers using content delivery networks, web application firewalls, and server-level protections, with specific guidance for configuring Drupal sites to manage caching and crawl behavior effectively. Additionally, the session addresses the policy implications of blocking such crawlers and offers strategies for communicating with stakeholders about the associated risks. Attendees will gain valuable insights and a checklist of protective measures to maintain site integrity while ensuring transparency.
