AI crawlers and your robots.txt: are you blocking the assistants?

Why this is the first thing to check

Most AI visibility problems are gradual. You are named sometimes, by some models, for some questions. A crawler block is not like that. It is binary and silent: the assistant cannot read your pages, so it recommends somebody whose pages it can read, and nothing anywhere tells you this happened.

The uncomfortable part is how often nobody chose it. A security plugin added a bot rule. A developer copied a robots.txt from a template written in 2023, when blocking GPTBot was being discussed everywhere. A CDN shipped a new default. We regularly find businesses spending on content while quietly refusing the agents that would quote it.

The crawlers, and what each one actually does

The single most important distinction is between crawlers that gather material for training and agents that fetch a page live, or index it, so that an assistant can answer a question with it. Blocking the first is a policy decision about your content. Blocking the second removes you from the answer.

Agents that fetch or index pages to answer questions

• OAI-SearchBot (OpenAI) indexes sites for ChatGPT's search features. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User (OpenAI) fetches a page when a person asks ChatGPT or a custom GPT a question, or when GPT Actions call out to a site. OpenAI notes that because these actions are initiated by a user, robots.txt rules may not apply.
• Claude-SearchBot (Anthropic) crawls and indexes content to improve the relevance and accuracy of search results.
• Claude-User (Anthropic) fetches sites when a person asks Claude a question. Anthropic says blocking it can reduce your visibility in user-directed web searches.
• PerplexityBot (Perplexity) is the indexing crawler that surfaces and links sites in Perplexity results. Perplexity's own documentation recommends allowing it.
• Perplexity-User (Perplexity) visits a page when a user's question needs it. Perplexity says it is not used for crawling or training, and that because the fetch is user-triggered it generally ignores robots.txt rules.
• Googlebot (Google) is still the crawler behind Google's AI Overviews and AI Mode, because those sit inside Search. Google's guidance is explicit that there are no additional requirements to appear in them.
• Bingbot (Microsoft) builds the Bing index. Microsoft's controls for chat answers operate on content already in that index, which is why Copilot visibility and Bing crawlability are not separable.
• meta-externalfetcher (Meta) fetches individual links when a user asks for them, and Meta says it may bypass robots.txt because it acts on a user request.

Crawlers and tokens about training

• GPTBot (OpenAI) crawls content that may be used to train OpenAI's foundation models.
• ClaudeBot (Anthropic) gathers public web content that may be used to train Anthropic's models. Anthropic treats a block as a request to exclude your future content from training datasets.
• Google-Extended (Google) is not a crawler at all. It is a standalone product token with no separate user agent, controlling whether crawled content can be used to train Gemini models and for grounding in Gemini apps and Vertex AI. Google states it does not affect Search inclusion or ranking.
• Applebot-Extended (Apple) is also a token rather than a crawler. It controls whether material Applebot already collected may train Apple's foundation models. Apple says disallowing it leaves your pages discoverable through Spotlight, Siri and Safari.
• meta-externalagent (Meta) crawls to train Meta's foundation models or to index content for its products.
• CCBot (Common Crawl) belongs to a non-profit archive rather than a model vendor, but its crawl is a widely used training corpus, so it tends to appear in these discussions.

How to check your own file in under a minute

Open your domain with /robots.txt on the end in a browser. Read the whole thing, not the first group. Rules are grouped by user-agent, matching is case-insensitive, and when a crawler matches no named group it falls back to the group for the asterisk.

Three patterns should stop you. A blanket block, which on a live site is almost always an accident left over from staging:

Named blocks on the answering agents, which is the costly case described above. And a robots.txt that returns a server error rather than a file: RFC 9309 tells crawlers to assume a complete disallow when the file is unreachable with a 5xx status, while a plain 404 means they may access anything. A missing file is safer than a broken one.

Check every subdomain separately. The file governs one host, so shop.example.com and example.com each need their own, and a blog on a subdomain is a common blind spot.

A robots.txt that allows everything

For most local service businesses this is the correct file. It is short on purpose.

You do not need to list every AI agent by name to let them in. A crawler that matches no named group uses the asterisk group, and if no rule matches, the path is allowed. Naming them is only useful when you want to treat them differently from everyone else.

Allowing answers but not training

This is the position many publishers want: be quotable today, without contributing to the next model. It is achievable because the vendors separate the tokens.

The answering agents are not named, so they fall through to the permissive group. If you prefer to be explicit, add an Allow group for OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot. The behaviour is the same; the file is just easier for the next person to read.

The block that happens before robots.txt is read

Robots.txt is a request for cooperation from a well-behaved crawler. A CDN or firewall is a gate, and it closes first. If a bot rule returns a 403 or a challenge page, the agent never sees your file and never sees your content, however welcoming the file is.

Cloudflare is the common case, and its defaults have changed more than once. It offers a managed robots.txt that writes AI directives for you, and separately a set of AI bot policies that actually enforce blocks. Cloudflare's own documentation is clear about the difference: the file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level. Check the live dashboard rather than trusting what you remember configuring.

The same applies to WordPress security plugins that block unfamiliar user-agents, bot protection on a managed host, aggressive rate limiting, and any rule that challenges traffic from datacentre IP ranges. Most of the vendors publish the IP ranges their crawlers use — OpenAI, Perplexity and Microsoft all do — which is the reliable way to allowlist them and to confirm a hit was genuine rather than a spoofed user-agent. The honest test is your server logs: search for the agent names above and see whether those requests returned 200.

When blocking is a reasonable choice

Some businesses should block, and the argument is not unreasonable. A publisher whose revenue comes from people arriving on the page is giving away the product when an assistant summarises it and the reader never visits. A membership site, a research archive or a stock library is handing over inventory. Some organisations have a licensing deal that depends on not giving the content away for nothing. Others simply object to their work training a commercial model and are entitled to say so.

A local service business is in almost the opposite position. A joiner in Mansfield or a dental practice in Leeds is not selling the words on the page. The page is an advertisement, and being summarised by an assistant is distribution, not theft. There is no content to protect and a customer to gain, so blocking costs something and returns nothing.

If you are between those two — an agency, a consultancy, a firm with genuinely valuable guides — the split file above is the sensible default. Let the answering agents in, keep the training crawlers out, and revisit it when the licensing market matures.

What to do this week

Read your robots.txt on every host you own. Check your CDN and firewall for bot rules. Grep your logs for the agent names in this article and confirm they are getting 200s. That is the whole job, and it is usually an hour.

Then move on to the things that decide whether an assistant has a reason to quote you: what assistants actually read on a page and whether your structured data helps a machine parse it. If you want a baseline before you change anything, our free check will tell you whether the models name you today.