--- title: "How to Block AI Crawlers & Safeguard Your Website (2026)" description: "Block AI crawlers from stealing your website content. Complete guide: Cloudflare one-click blocking, robots.txt setup, Nginx rules, and AI Labyrinth. Updated for 2026." date: 2026-07-22 categories: ["tools"] tags: ["ai","cloudflare","security"] --- import Button from "../../components/widgets/Button.astro"; import { Picture } from "astro:assets"; import Notice from "../../components/widgets/Notice.astro"; import ListCheck from "../../components/widgets/ListCheck.astro"; import Accordion from "../../components/widgets/Accordion.astro"; import Tabs from "../../components/widgets/Tabs.astro"; import Tab from "../../components/widgets/Tab.astro"; import YouTubeEmbed from "../../components/widgets/YouTubeEmbed.astro"; AI crawlers now account for over 20% of all web traffic, and with 170+ AI bots actively scraping content, **blocking AI crawlers** matters for every website owner. Whether your content is training AI models without compensation, you're losing referral traffic to AI-powered search, or your VPS bandwidth is disappearing, this guide covers every method available in 2026, from Cloudflare's free one-click toggle to server-level Nginx rules and the AI Labyrinth defense. The approach that works is layered defense. No single method stops all bots. I'll walk through seven methods, from the simplest (one click in Cloudflare) to the most granular (server-level configs and new web standards), and show you how to verify each one actually works. ## Why block AI crawlers in 2026? Three reasons this matters more now than it did in 2024: **Bandwidth and cost.** AI bots are responsible for over 20% of all HTML requests across Cloudflare's network. Aggressive crawlers like ByteDance's Bytespider accessed over 40% of Cloudflare-protected sites. Anthropic's ClaudeBot saw an 800% volume increase in late 2025. If you're running a VPS with bandwidth caps, these bots are eating into your allocation for zero benefit. **Content ownership.** Your content is being used to train commercial AI models without compensation or attribution. The visitors don't come to your site, don't subscribe to your newsletter, don't click your affiliate links. The AI company profits; you get the bandwidth bill. **Legal context.** The EU AI Act (Article 53, in force August 2, 2025) now requires General-Purpose AI providers to implement copyright compliance policies and respect machine-readable opt-outs like `robots.txt`. If you're in the EU, your `robots.txt` file now carries legal weight. More on that in Method 6. Not all AI crawlers are harmful. Search-indexing bots like OAI-SearchBot and Claude-SearchBot may drive traffic to your site through AI-powered search results. Training crawlers like GPTBot and ClaudeBot won't send you a single visitor. This guide helps you decide what to block and what to allow. ## The AI crawler landscape in 2026: who's crawling and why The AI bot ecosystem has exploded since 2024. The community-maintained [ai-robots-txt repository](https://github.com/ai-robots-txt/ai.robots.txt) (4,000+ GitHub stars) now tracks 170+ AI crawlers. For practical purposes, you need to know about three categories, which map directly to Cloudflare's new classification system: These bots scrape your content to train AI models. They provide zero referral traffic. Block these. | Crawler | Operator | robots.txt token | |---------|----------|-----------------| | GPTBot | OpenAI | `GPTBot` | | ClaudeBot | Anthropic | `ClaudeBot` | | CCBot | Common Crawl | `CCBot` | | Google-Extended | Google (Gemini) | `Google-Extended` | | Meta-ExternalAgent | Meta (Llama) | `meta-externalagent` | | Meta-ExternalFetcher | Meta | `meta-externalfetcher` | | Bytespider | ByteDance | `Bytespider` | | Amazonbot | Amazon (Alexa) | `Amazonbot` | | DeepSeekBot | DeepSeek | `DeepSeekBot` | | cohere-ai | Cohere | `cohere-ai` | | Diffbot | Diffbot | `Diffbot` | | Applebot-Extended | Apple Intelligence | `Applebot-Extended` | These index your content for AI-powered search results. They may send referral traffic. Consider allowing. | Crawler | Operator | robots.txt token | |---------|----------|-----------------| | OAI-SearchBot | OpenAI (ChatGPT search) | `OAI-SearchBot` | | Claude-SearchBot | Anthropic (Claude search) | `Claude-SearchBot` | | PerplexityBot | Perplexity | `PerplexityBot` | | DuckAssistBot | DuckDuckGo | `DuckAssistBot` | | Gemini-Deep-Research | Google | `Gemini-Deep-Research` | These act on behalf of a specific user in real time. Not bulk crawling. Consider allowing. | Crawler | Operator | robots.txt token | |---------|----------|-----------------| | ChatGPT-User | OpenAI | `ChatGPT-User` | | Claude-User | Anthropic | `Claude-User` | | Perplexity-User | Perplexity | `Perplexity-User` | | MistralAI-User | Mistral (Le Chat) | `MistralAI-User` | ### The Perplexity stealth crawler warning This is the most important cautionary tale for anyone relying solely on `robots.txt`: In August 2025, Cloudflare caught [Perplexity using stealth, undeclared crawlers](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/) that masqueraded as Chrome on macOS. When blocked via their declared `PerplexityBot` user-agent, they switched to a generic browser fingerprint, used rotating IPs across multiple ASNs, and in some cases didn't even fetch `robots.txt`. Cloudflare observed 3-6 million daily stealth requests and de-listed Perplexity as a verified bot. If you're interested in how Perplexity's AI search works and its approach, check out our article on [Perplexity's approach to AI search](/perplexity/). The lesson: `robots.txt` is a polite request, not a security barrier. Technical enforcement at the WAF level is the only thing that actually stops bad actors. ## Method 1: Block AI crawlers with Cloudflare one-click toggle (recommended) This is the simplest and most effective method. Cloudflare introduced a one-click toggle in July 2024 that blocks known AI scrapers and crawlers automatically, no WAF rule creation needed. It's available on the **Free plan** and Cloudflare updates the bot fingerprints as new crawlers emerge. **Step 1: Enable the Cloudflare proxy** Make sure your site's DNS records have the orange cloud (proxy) enabled, not just DNS-only. If you need help setting up Cloudflare for the first time, see how to [deploy your site on Cloudflare](/deploy-astrojs-cloudflare/) or [set up a blog on Cloudflare's free tier](/build-astro-blog-free/). **Step 2: Navigate to Security > Bots** In the Cloudflare dashboard, go to **Security > Bots**. You'll see the **"AI Scrapers and Crawlers"** section. **Step 3: Toggle "Block AI bots" to ON** Enable the toggle. That's it, Cloudflare handles the rest. **Step 4: Verify it works** Wait 5 minutes, then test: ```bash curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.0" https://yoursite.com # Should return 403 or 406 ``` You can also check the **Security > Analytics** dashboard after a few hours to see how many AI requests have been blocked. Cloudflare has blocked over 416 billion AI bot requests since July 2025, and 2.5 million websites have enabled this protection. This is the fastest path: one click and you're covered. Cloudflare automatically updates bot signatures as new crawlers appear. ## Method 2: Cloudflare granular AI bot policies (Search, Agent, Training) If you want fine-grained control over which types of AI bots can access your site, Cloudflare's new three-category system (launched July 2026) gives you exactly that. This replaces the older "Block AI bots" managed preset. Navigate to **Security Settings > Configure AI bot policies** in your Cloudflare dashboard. You'll see three categories: | Category | What it controls | Default (from Sept 15, 2026) | |----------|-----------------|------------------------------| | **Search** | Crawlers indexing content for AI search (OAI-SearchBot, etc.) | Allow | | **Agent** | User-triggered agents acting in real time (ChatGPT-User, etc.) | Block on pages with ads | | **Training** | Crawlers collecting data for model training (GPTBot, ClaudeBot, etc.) | Block on pages with ads | For each category, you can choose: - **Block**: block on all pages - **Block on pages with ads**: middle ground (new default for Training and Agent) - **Allow**: let them through The "Block on pages with ads" option is clever. It lets AI bots access your content pages (which may drive traffic) while blocking them on monetized pages (where they'd cost you ad revenue without contributing). To check your results, go to **Security > Analytics** after 24 hours and review blocked vs. allowed requests per category. The old "Block AI bots" preset deprecates on September 15, 2026. If you're currently using it, migrate to the new three-category system before that date. The old method used a WAF custom rule with the expression `(cf.verified_bot_category eq "AI Crawler")`. This still works and doesn't consume your custom rule slots differently, but it's being superseded by the new three-category system which offers finer control (block training separately from search). If you already have this rule deployed, it's fine to keep it, but the new system at Security Settings > Configure AI bot policies is the recommended path going forward. ## Method 3: AI Labyrinth: waste bot resources instead of blocking Cloudflare's AI Labyrinth (launched March 2025) takes a different approach: instead of just blocking bots, it serves them **AI-generated decoy content** through hidden links that no human would ever click. This does three things: 1. **Wastes the crawler's compute resources** processing fake content 2. **Acts as a honeypot**: if a crawler follows the hidden links, Cloudflare knows it's a bot 3. **Feeds Cloudflare's ML models** to identify new bot patterns faster It's available on the **Free plan** as an opt-in toggle at **Security > Bots** (same page as the AI Scrapers toggle from Method 1). AI Labyrinth works alongside blocking, it's not a replacement. Enable it after turning on Method 1 for maximum defense-in-depth. The hidden links are invisible to human visitors, so there's zero impact on your site's user experience. This is the kind of defense I like: it doesn't just reject bad traffic, it actively makes the attacker's job harder and more expensive. ## Method 4: Block AI crawlers using robots.txt (complete list) `robots.txt` is the web standard for declaring which crawlers can access your content. It's now legally significant in the EU under the AI Act (more on that in Method 6). But keep in mind: `robots.txt` is a request, not a barrier. Well-behaved bots respect it. Bad actors don't. Always combine this with Cloudflare or server-level blocking. Here's a purpose-annotated `robots.txt` that covers the major AI crawlers: ```txt # === AI TRAINING CRAWLERS (BLOCK) === # These bots collect data for model training, no referral benefit User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Applebot-Extended Disallow: / # === AI SEARCH CRAWLERS (CONSIDER ALLOWING) === # These may drive referral traffic, uncomment to block # User-agent: OAI-SearchBot # Disallow: / # User-agent: Claude-SearchBot # Disallow: / # User-agent: PerplexityBot # Disallow: / # User-agent: DuckAssistBot # Disallow: / # === AI USER AGENTS (CONSIDER ALLOWING) === # These are user-triggered, not bulk crawling # User-agent: ChatGPT-User # Disallow: / # User-agent: Claude-User # Disallow: / # User-agent: Perplexity-User # Disallow: / # User-agent: MistralAI-User # Disallow: / ``` If you just want to block all known AI crawlers in one shot without deciding per-bot: ```txt User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Gemini-Deep-Research Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: CCBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: cohere-ai Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Diffbot Disallow: / ``` The [ai-robots-txt community repository](https://github.com/ai-robots-txt/ai.robots.txt) (4,000+ stars) maintains an even more comprehensive list with 170+ bots. It also provides ready-made configs for Nginx, Apache, Caddy, and HAProxy. robots.txt is a polite request, not a security barrier. Perplexity was caught ignoring it entirely and spoofing browser fingerprints. Always combine with Cloudflare (Methods 1-3) or server-level blocking (Method 5) for actual protection. Common causes: - **Crawlers cache robots.txt** — it can take 24-48 hours for major crawlers to re-fetch and respect changes - **File not at the root URL** — `robots.txt` must be at `https://yourdomain.com/robots.txt`, not in a subdirectory - **Syntax errors** — a malformed file may be ignored entirely. Use Google's robots.txt tester in Search Console to validate - **Your robots.txt is blocked** — if you have a WAF rule blocking all bots, they can't read the file either (the Cloudflare toggle handles this correctly — it allows robots.txt access) ## Method 5: Server-level AI crawler blocking (Nginx, Apache, Caddy) For self-hosters running their own VPS, server-level user-agent blocking adds defense-in-depth below the WAF layer. If you're self-hosting on a VPS like [Hetzner Cloud](https://go.bitdoze.com/hetzner), this is your last line of defense when bots bypass or don't go through Cloudflare. The ai-robots-txt community repo provides ready-made config files for all three major web servers. To [secure your VPS against malicious traffic](/crowdsec-secure-server/) more broadly, combine this with tools like CrowdSec. **Option A: Inline if block (simple)** ```nginx # Add inside your server {} block if ($http_user_agent ~* "(GPTBot|ChatGPT-User|ClaudeBot|Claude-User|CCBot|PerplexityBot|Bytespider|Amazonbot|meta-externalagent|Google-Extended|OAI-SearchBot|DeepSeekBot|cohere-ai|Diffbot|Applebot-Extended)") { return 403; } ``` **Option B: Community include file (maintained)** Download the community-maintained config and include it: ```bash # Download the latest list curl -o /etc/nginx/conf.d/nginx-block-ai-bots.conf \ https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/nginx-block-ai-bots.conf ``` ```nginx # Add in your server {} block include /etc/nginx/conf.d/nginx-block-ai-bots.conf; ``` After either option, test and reload: ```bash nginx -t && systemctl reload nginx ``` Add to your `.htaccess` or virtual host config: ```apache RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|ClaudeBot|Claude-User|CCBot|PerplexityBot|Bytespider|Amazonbot|meta-externalagent|Google-Extended|DeepSeekBot|cohere-ai) [NC] RewriteRule .* - [F,L] ``` Add a route block in your `Caddyfile`: ``` @blocked_ai_bots { header_regexp User-Agent "(GPTBot|ChatGPT-User|ClaudeBot|Claude-User|CCBot|PerplexityBot|Bytespider|Amazonbot|meta-externalagent|Google-Extended|DeepSeekBot|cohere-ai)" } handle @blocked_ai_bots { abort } ``` Server-level blocking is your last line of defense. Bots can spoof user-agents, so this shouldn't be your only method. Combine with Cloudflare (Methods 1-3) for complete protection. Check these common issues: - Run `nginx -t` to verify your config has no syntax errors - Ensure the `if` block is inside the correct `server {}` context (not inside a `location {}` block) - Check for conflicting `location` blocks that might override the server-level rule - Verify nginx actually reloaded: `systemctl status nginx` - Test locally first: `curl -A "GPTBot/1.0" http://localhost` before testing through Cloudflare ## Method 6: TDM Reservation Protocol and machine-readable opt-outs Several new web standards have emerged to give publishers machine-readable ways to declare AI usage rights. These complement `robots.txt` and carry increasing legal weight, especially in the EU. ### TDM Reservation Protocol (W3C standard) The TDM Reservation Protocol lets you declare rights reservations via HTTP response headers. Add this to your Nginx config: ```nginx # Add to your server {} or location {} block add_header tdm-reservation "1"; add_header tdm-policy "https://yourdomain.com/tdm-policy.json"; ``` For Apache: ```apache Header set tdm-reservation "1" ``` This is legally significant under the EU AI Act — GPAI providers must detect and respect these signals. ### Cloudflare Content-Signal If you use Cloudflare's managed `robots.txt`, it now automatically prepends a Content-Signal directive: ``` User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / ``` The `use=` parameter signals content usage levels: `immediate`, `reference`, or `full`. This is Cloudflare's way of making opt-outs machine-readable without requiring manual config. ### Other emerging standards - **ai.txt** — Proposed standard for AI usage permissions (no-training, no-inference, Allow-RAG). Placed at site root or `/.well-known/ai.txt` - **llms.txt** — A Markdown file at site root that gives LLMs a curated summary of key content. Not a blocking tool — it's an AI SEO tool for controlling how LLMs understand your site - **RSL 1.0 (Really Simple Licensing)** — Open standard (December 2025) supplementing `robots.txt` with licensing categories and contribution payment models. Backed by Yahoo, Ziff Davis, and O'Reilly Media If you're in the EU, a properly configured `robots.txt` carries legal weight under AI Act Article 53 (in force since August 2, 2025). The Hamburg Higher Regional Court confirmed in December 2025 that natural-language opt-outs in terms of use are insufficient — opt-outs must be machine-readable. In the US, `robots.txt` has no direct legal enforceability under current law. Either way, technical enforcement matters more than legal threats. ## Method 7: IP blocking and supplementary methods IP blocking is the oldest trick in the book but the least practical at scale — AI companies use cloud IPs that rotate frequently. Still, it has a place as a supplementary method. If you're seeing heavy traffic from specific IPs, you can block them at the firewall level. Check [OpenAI's bot documentation](https://platform.openai.com/docs/bots) for current IP ranges (they change over time). ```bash # Example — verify current ranges before running sudo ufw deny proto tcp from 23.98.142.176/28 to any port 80 sudo ufw deny proto tcp from 23.98.142.176/28 to any port 443 sudo ufw deny proto tcp from 40.84.180.224/28 to any port 80 sudo ufw deny proto tcp from 40.84.180.224/28 to any port 443 ``` **Caveat:** IP ranges change. Check the docs for current values. Also note that if you're running Docker, containers may bypass `ufw` rules — see our guide on [firewall rules that actually work](/docker-bypasses-firewall/) with Docker. **Identifying the IPs:** Check your web server logs for AI crawler traffic: ```bash grep -E "GPTBot|ClaudeBot|CCBot|Bytespider" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn ``` If you're on Cloudflare, go to **Security > WAF > Tools** to add IP Access Rules. You can block specific IPs or ranges without consuming WAF rule slots. Enter the IP, select "Block," and add a note for why. This is useful for blocking specific abusive IPs you've identified in your logs, rather than trying to block entire AI company IP ranges. ### Other supplementary methods **Meta tags.** The `noai` and `noimageai` meta tags are still valid. Add them to your HTML ``: ```html ``` Compliant AI companies check for these, but they're easy to ignore technically. **WordPress plugins.** If you're on WordPress: - **Block AI Crawlers** (by bobmatyas, updated November 2025) — generates `robots.txt` blocking common AI crawlers and adds `noai`/`noimageai` meta tags - **Known Agents** (formerly Dark Visitors) — tracks the AI crawler landscape and auto-generates your `robots.txt` For more WordPress-specific options, see our guide on [WordPress anti-scraping plugins](/best-web-scraping-plugins-for-wordpress/). **Netlify users.** Netlify offers a **User Agent Blocker** extension (Edge Function-based) that blocks AI crawlers from the project dashboard — worth checking if you host static sites there. **DNS-level blocking.** You can also [block unwanted traffic at the DNS level with NextDNS](https://go.bitdoze.com/nextdns), which adds another layer of protection before requests even reach your server. This is expected. IP blocking has fundamental limitations: - AI companies use cloud infrastructure (AWS, Azure, GCP) with IPs that rotate - Bots can use residential proxies or VPNs to change IPs - User-agent spoofing means the same IP can appear as any browser - Blocking IP ranges may accidentally block legitimate cloud services This is why IP blocking is supplementary — not primary. Use Cloudflare (Methods 1-3) for reliable blocking. ## How to verify your AI crawler blocks are working Don't just enable protections and assume they work. Here's how to confirm: **Test with curl using AI bot user-agents:** ```bash # Test if GPTBot is blocked curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.0" https://yoursite.com # Should return 403 or 406 # Test if robots.txt is still accessible (it should be) curl -s -A "GPTBot/1.0" https://yoursite.com/robots.txt # Should return your robots.txt content # Test ClaudeBot curl -s -o /dev/null -w "%{http_code}" -A "ClaudeBot/1.0" https://yoursite.com # Should return 403 ``` If the curl commands return `403`, your blocking is working. If they return `200`, something isn't configured correctly — check your Cloudflare toggle, Nginx reload status, or `robots.txt` syntax. **Check Cloudflare Security Analytics:** Go to **Security > Analytics** in the Cloudflare dashboard. After 24 hours you should see blocked requests in the "Bot Traffic" section. Filter by bot category to see training, search, and agent traffic separately. **Search server logs:** ```bash # Check for AI crawler hits in the last 24 hours grep -E "GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot" /var/log/nginx/access.log | tail -20 # Count blocks vs. gets grep -E "GPTBot|ClaudeBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c ``` **Note:** If you updated `robots.txt`, compliant crawlers cache it for 24-48 hours before re-fetching. Don't expect immediate results from `robots.txt` changes. ## The future: Pay Per Crawl and AI content monetization Cloudflare introduced **Pay Per Crawl** (private beta, July 2025) as a third option beyond "allow" or "block" — **monetize**. Instead of blocking AI crawlers outright, publishers can charge per request. It uses HTTP 402 (Payment Required), Web Bot Auth with Ed25519 signatures, and Cloudflare acts as Merchant of Record. The idea: AI companies get access to quality content, publishers get paid, and Cloudflare handles the payment infrastructure. For solo operators who've spent years building content libraries, this could eventually become a revenue stream. Pay Per Crawl is in private beta as of July 2026. You can configure it from your Cloudflare dashboard, but widespread adoption depends on AI companies signing up. We'll update this guide when it becomes generally available. ## Conclusion: build a layered defense against AI crawlers No single method blocks all AI crawlers. The Perplexity stealth crawler scandal proved that even well-configured `robots.txt` can be ignored. Here's the defense stack I recommend: Combining Cloudflare (Methods 1-3) with `robots.txt` (Method 4) covers 99% of threats for most sites. Self-hosters on a VPS should add server-level blocking (Method 5) as a safety net. If you're serious about taking control of who accesses your content, the layered approach works. For broader protection of your online presence, you can also [block unwanted traffic at the DNS level](/block-ads-malware-dns-protection/) and explore [self-hosted privacy solutions](/searxng-self-host-privacy-search/) that keep your data under your control.