The AI crawlers, and how to let them in
Who the crawlers are
| Crawler | Powers | You want it? |
|---|---|---|
| GPTBot | ChatGPT browsing and answers | Yes |
| OAI-SearchBot | ChatGPT search results | Yes |
| ClaudeBot | Claude answers | Yes |
| PerplexityBot | Perplexity answers and citations | Yes |
| Google-Extended | Gemini grounding and training | Usually |
These are separate from classic search crawlers like Googlebot and Bingbot. Allowing or blocking an AI crawler has no effect on your normal search rankings; it only decides whether that engine can use your content to answer questions. For almost every business, being answerable is the point.
How sites block them by accident
Three ways, all common. A robots.txt copied from a template that disallows unknown agents. A security product or firewall rule that challenges any bot it does not recognize, which now includes the AI crawlers. And a well-meaning but outdated decision, made in the panic about AI training, to block everything, which also blocks the crawlers that would put you in the answer. The result is the same: you are missing from AI answers and nothing in your analytics explains why.
The robots.txt that works
Explicit is better than silent. This allows the major AI crawlers and points them to your sitemap:
If you have reasons to keep a crawler out, that is a real choice, just make it deliberately rather than by inheriting a template. The tradeoff is always the same: control over training versus presence in answers.
How to verify
Reading robots.txt is not enough, because firewalls block below that layer. The reliable check is to confirm the crawlers actually reach your pages, then confirm you appear in the answers they build. Our AEO Visibility Audit checks crawler access as its first step and measures whether you are cited across every engine. Read next: how to get cited by ChatGPT and the llms.txt guide.
Common questions
Should I block AI crawlers?
Only if you do not want to appear in AI answers. Blocking GPTBot, ClaudeBot or PerplexityBot removes you from those engines. Most businesses want in.
Does blocking Google-Extended hurt my Google ranking?
No. It only controls AI training and grounding use. Classic Googlebot and rankings are unaffected, though blocking it can reduce your presence in Gemini answers.
How do I know if I am blocking them?
Check robots.txt for Disallow rules under the AI agents, and check firewall or security rules that block unknown bots. Both silently remove you from AI answers.
Check what the engines can reach
The AEO Visibility Audit confirms crawler access and measures whether you are cited across ChatGPT, Perplexity, Gemini and Claude.