The AI crawlers, and how to let them in

Who the crawlers are

These are separate from classic search crawlers like Googlebot and Bingbot. Allowing or blocking an AI crawler has no effect on your normal search rankings; it only decides whether that engine can use your content to answer questions. For almost every business, being answerable is the point. Bingbot is the odd one out: it also doubles as the retrieval layer behind Microsoft Copilot, which is why getting cited by Copilot is really just Bing SEO, measured for the first time.

How sites block them by accident

Three ways, all common. A robots.txt copied from a template that disallows unknown agents. A security product or firewall rule that challenges any bot it does not recognize, which now includes the AI crawlers. And a well-meaning but outdated decision, made in the panic about AI training, to block everything, which also blocks the crawlers that would put you in the answer. The result is the same: you are missing from AI answers and nothing in your analytics explains why.

The robots.txt that works

Explicit is better than silent. This allows the major AI crawlers and points them to your sitemap:

If you have reasons to keep a crawler out, that is a real choice, just make it deliberately rather than by inheriting a template. The tradeoff is always the same: control over training versus presence in answers.

How to verify

Reading robots.txt is not enough, because firewalls block below that layer. Access is also only half the question: the crawlers that do reach you never run your JavaScript, so a page that loads its content client-side comes back empty even when every bot is allowed in. The reliable check is to confirm the crawlers actually reach your pages, then confirm you appear in the answers they build. Our AEO Visibility Audit checks crawler access as its first step and measures whether you are cited across every engine. Read next: how to get cited by ChatGPT and the llms.txt guide. For local businesses specifically, access is only the first gate: what actually correlates with a citation once reviews enter the picture is thinner evidence, covered in reviews and AI search.

What each crawler does with your content

They are not all the same job. GPTBot and ClaudeBot fetch pages so their models can read and, when a user asks, summarise or cite them. OAI-SearchBot and PerplexityBot are closer to classic search crawlers: they index for a live answer engine that shows citations back to your site, which means they can send you real referral traffic. Perplexity is the awkward case here, because it also runs a second agent, Perplexity-User, that its own documentation treats differently under robots.txt; the Perplexity guide unpacks that and how to verify access against the IP ranges Perplexity publishes. Google-Extended is a control switch rather than a crawler, governing whether Google may use content it already has for Gemini grounding and model training. Knowing which is which matters, because the ones that drive visible referral clicks, the search-style bots, are the last ones any business should be blocking.

Training versus answering: the distinction that matters

Most of the anxiety about AI crawlers is really about training: the worry that a model will ingest your content and give it away for free. That is a legitimate choice to weigh. But answering is a different thing, and it is where visibility lives. When Perplexity or ChatGPT search builds an answer and cites sources, being one of those sources sends interested readers to you. Blocking every crawler to prevent training also blocks the crawlers that would have recommended you, which is usually a bad trade. The precise move is to decide training and answering separately, allowing the answer engines even if you restrict training, rather than swinging the whole door shut.

How to monitor that they actually reach you

Robots.txt is a request, not proof. Beyond letting crawlers in, you can hand assistants a direct line to your data with an MCP server. A firewall, bot-management product or aggressive rate limit can block a crawler below the robots layer while your file looks perfectly welcoming. The reliable check has two parts: confirm the AI user agents appear as successful requests in your server or CDN logs, and confirm you actually show up when you ask the engines your own buyer questions. If the logs show the crawler arriving but the answers never name you, the problem is content and entity clarity, not access. If the logs show the crawler being turned away, the problem is infrastructure. Our audit checks both, in that order.

The robots.txt mistakes that cost the most

Two are expensive and common. The first is a blanket disallow left over from a staging site or a template, which quietly keeps every crawler out while the visible site looks fine. The second is the opposite error of over-specifying: adding a long list of AI user agents with Disallow rules copied from an alarmist blog post, then forgetting them, so months later you cannot understand why you never appear in answers. The safe pattern is short and explicit, allow the crawlers you want, name your sitemap, and revisit the file deliberately if your stance on training changes. Set it once, correctly, and check it whenever you change hosting or add a security product, because those are the moments access silently breaks.

A quick access checklist

Four checks settle it. One, open your robots.txt and confirm there is no Disallow under GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or a blanket rule that catches them. Two, confirm your firewall or bot-management tool is not silently challenging them below that layer. Three, check your server or CDN logs show those agents fetching pages successfully. Four, ask the engines your own buyer questions and see whether you appear. Pass all four and access is not your problem; fail any and you have found exactly where visibility is leaking, before spending a cent on content.

Sources

Crawler behaviour changes often, so check the primary documentation before you edit robots.txt: Google's crawler and fetcher overview, OpenAI's crawler documentation, and Anthropic's guidance on ClaudeBot.

Common questions