Is your website blocking AI crawlers?

There are two completely separate ways a website blocks AI, and almost everybody only knows about one of them. The one people know about is easy to check and easy to fix. The one people do not know about is invisible to every standard SEO tool and is the reason a business can pass an audit and still be unreadable.

Worth saying up front, because the category is full of scare marketing: outright blocking is not the common case. Depending on which slice of the web you sample, somewhere between about 10% and 25% of sites block OpenAI's crawler. It is real, and when it is happening to you nothing else you do matters, which is why it is worth ten minutes of your afternoon. But it is not why most businesses are invisible.

Block one: robots.txt

Every site has, or should have, a plain text file at yourdomain.com/robots.txt. Open yours right now in a browser tab. It is public, so you do not need a login.

You are looking for a block shaped like this:

User-agent: GPTBot
Disallow: /

Disallow: / means the whole site. If you see that against any of the agents below, that crawler has been told to stay out.

The eleven worth checking

This is the list our test checks, and it is longer than most people expect because the major AI companies each run more than one crawler for different purposes.

• GPTBot – OpenAI's training crawler
• OAI-SearchBot – what builds ChatGPT's search index
• ChatGPT-User – ChatGPT fetching a page live, mid-conversation
• ClaudeBot and Claude-SearchBot – Anthropic
• Google-Extended – Gemini and Google's AI Overviews
• PerplexityBot – Perplexity
• CCBot – Common Crawl, which feeds many models indirectly
• Applebot-Extended – Apple Intelligence
• meta-externalagent – Meta AI
• Bytespider – ByteDance

The distinction between those first three matters more than it looks. Blocking GPTBot keeps your content out of training, which is a legitimate choice a publisher might make on purpose. Blocking OAI-SearchBot or ChatGPT-User keeps you out of the answers, which for a local business is simply self-harm. Plenty of sites blocked all three at once because a plugin offered a single tick box labelled something like “block AI”, and nobody realised the three did different jobs.

How this usually happened to you

Rarely on purpose. In our experience it is almost always one of: a security or privacy plugin that added the rules during an update, a hosting provider that ships an opinionated default robots.txt, a theme or site builder with an “AI protection” toggle switched on at setup, or a developer who added it in 2023 during the first wave of publisher panic and then moved on.

None of those people were wrong to consider it. The mistake is that it was decided once, for a general case, and never revisited for a business whose customers are now asking assistants for recommendations.

Block two: your host, which robots.txt never shows

This is the one almost nobody checks, and it is our favourite finding because it is invisible, consequential, and usually a five-minute fix.

robots.txt is a request. It is a note left at the door asking well-behaved crawlers to respect certain rules, and they do. But your host, your CDN or your firewall sits in front of that door and can simply refuse a connection before robots.txt is ever read. Bot-protection features on managed WordPress hosts, aggressive security rules, and some firewall defaults all do this, and they often classify AI crawlers as unwanted automated traffic by default.

The result is a site with a completely clean robots.txt that still returns a 403 to an AI crawler. Audit it with any standard tool and it passes. Ask it for a page while identifying as GPTBot and you get a wall.

How to check it yourself

If you are comfortable with a terminal, this compares the two requests directly. Run both and look at the status code that comes back.

# as a normal browser
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0" https://yourdomain.com

# as OpenAI's crawler
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://yourdomain.com

Two 200s means you are fine. A 200 followed by a 403 means your host is blocking AI crawlers and robots.txt will never tell you.

If a terminal is not your thing, the test on this page does exactly this comparison for you as part of the technical teardown, alongside the robots.txt check, and tells you in plain English which of the two is happening.

One caution on reading the result

A timeout, a rate-limit response or a server error is not the same thing as a refusal, and we are careful about this in the test because accusing a host of blocking AI when it was actually just slow is the kind of mistake that discredits an entire report. Only a genuine 4xx that is not a rate limit is real evidence of a policy. Anything else, we say we could not tell rather than guessing.

Unblocking, in order

1. Fix robots.txt first, since it is the one you control outright. Remove the Disallow lines for the crawlers you want, or replace the file with one that allows them explicitly. If a plugin generates it, change the setting rather than the file, or the plugin will put it back on the next update.
2. Then test at the host level using the comparison above. If the second request fails, the setting is in your hosting or CDN dashboard, usually under bot protection, firewall rules or security. Allow the AI user agents there.
3. Re-test both. This is the step people skip. A plugin update can quietly restore the block weeks later, and nothing tells you.
4. Then move on, because unblocking only gets the crawler through the door. What it finds once it is inside is a separate problem, and usually the bigger one.

That last point is worth dwelling on. Unblocking a site whose pages do not state what it does or where is like unlocking an empty shop. Necessary, and on its own not enough. The overview covers what happens next.