The short answer
Based on the first-party documentation available as of October 2026, blocking GPTBot should not affect your traditional search rankings. GPTBot is OpenAI's training crawler; it is not Googlebot, and Google does not use it. What blocking it does affect is whether your content can be collected as training data for OpenAI's models. Whether that trade-off suits you is a legitimate business decision — this guide lays out both sides rather than telling you blocking is always wrong.
Two separate systems that get conflated
Traditional search and AI answers are built by different pipelines, run by different companies, using different crawlers. Google Search is indexed by Googlebot. AI assistants are built by OpenAI, Anthropic, Google and others, each with their own crawlers and their own rules. Blocking or allowing one crawler says nothing about any other.
The confusion is understandable because the companies overlap. Google is both a search engine and an AI company, and it separates the two concerns with different crawler tokens: Googlebot handles Search indexing, while a separate token, Google-Extended, lets site owners control whether their content is used to train Google's AI models — without affecting Search. OpenAI similarly separates GPTBot (training) from OAI-SearchBot (search indexing) and ChatGPT-User (live retrieval). These distinctions come from each company's own crawler documentation, checked in October 2026.
The practical consequence is that "did I block AI?" is never one question. Your robots.txt might disallow GPTBot while allowing OAI-SearchBot, say nothing about Anthropic at all, and leave Google-Extended untouched — four separate decisions, each currently set to whatever your file happens to imply. Reading your own robots.txt with this list in hand is the only way to know which decisions you have actually made, as opposed to inherited from a plugin default or a hosting template.
What blocking GPTBot does — and what it does not do
A robots.txt rule disallowing GPTBot tells OpenAI's training crawler not to fetch your pages. OpenAI's documentation states that it respects these rules, so the practical effect is that your content stops being collected as training data for future OpenAI models.
What it does not do, based on the available documentation:
- It does not remove content that was already collected and trained on before you added the rule.
- It does not block ChatGPT from fetching your pages when a user asks about them — that is ChatGPT-User, a different crawler with its own rules.
- It does not block OpenAI's search indexing — that is OAI-SearchBot.
- It does not affect any other company's crawlers. Anthropic, Google, Perplexity and others each have their own user-agents and their own rules to set.
- It is not enforced by the protocol itself. robots.txt is a convention; it works because the major AI companies have committed to honouring it, not because anything technical compels them to.
Legitimate reasons to block training crawlers
Blocking training crawlers is not an anti-AI gesture; it is a content-licensing position, and there are sound reasons an owner might take it. Publishers whose business is selling subscriptions may not want their reporting absorbed into a model that answers questions without a visit. Companies with proprietary documentation may prefer it stays inside their own support channels. Some owners simply want the negotiation to happen explicitly — several large publishers have signed paid licensing deals with AI companies, and blocking the training crawler is what creates the leverage for that conversation.
Bandwidth is a more mundane but equally legitimate reason. Crawlers consume server resources, and a small site on shared hosting may reasonably decide that training crawlers — which fetch heavily and send no visitors back — are a cost without a return. Retrieval agents are different in kind here: a ChatGPT-User fetch happens because a person asked about you, and the answer can carry a link back to your site. Blocking training crawlers while allowing retrieval keeps the benefit and drops the cost.
There is also a cost to weigh on the other side. If your content is absent from training data, assistants are less likely to know about your brand when answering general questions in your category. Whether that matters depends entirely on how your customers find you. A local trades business and an online publication are in genuinely different situations, and the right answer is different for each.
A middle position exists: block the training crawlers but allow the retrieval and search agents. That keeps your pages available when a user explicitly asks an assistant about you, while keeping them out of training pipelines. How to let ChatGPT and Claude crawl your site has a robots.txt example that does exactly this.
How to check your current status
First, look at what your robots.txt currently says:
curl -s https://yoursite.com/robots.txtLook for a User-agent: GPTBot group. If there is none, check the User-agent: * group — a Disallow: / there blocks GPTBot too, along with everything else. If neither group disallows anything, GPTBot is allowed.
Second, check whether anything upstream blocks the request regardless of robots.txt:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
https://yoursite.com/A 200 means the page is served to that user-agent; a 403 means a firewall or CDN rule is refusing it. Sites are frequently surprised by this second test — the robots.txt was edited months ago, but an edge rule nobody remembered is still blocking the crawler. Our methodology describes how the audit runs both checks together, and if your site is behind Cloudflare, the free Cloudflare AI bot check runs the same comparison without an account.
How to block — or unblock — GPTBot safely
The mechanism is a named group in robots.txt. To block GPTBot:
User-agent: GPTBot
Disallow: /To unblock it, remove that group. If a wildcard User-agent: * group carries Disallow: /, removing the named group is not enough — add an explicit empty group for GPTBot, because an explicit group with no Disallow lines overrides the wildcard for that crawler. The same pattern applies to ClaudeBot and every other named agent.
Two cautions. First, robots.txt changes are not instant: crawlers cache the file, so a change can take days to be noticed, and content already collected stays collected. Second, check the layers above the file. A CDN bot rule that blocks GPTBot will keep blocking it whatever robots.txt says, while a CDN rule that allows it does not override a Disallow — compliant crawlers check robots.txt first. After any change, test the actual response with curl as shown in the previous section, and re-test a few days later.
Finally, write down what you decided and when. Crawler names and policies at the major AI companies have changed before and will change again, and the decision someone made two years ago may not be the one your configuration still expresses. A one-line note in your runbook — what is blocked, why, and who approved it — saves a forensic exercise later.
What this does not settle
The honest limit of this guide is that the long-term consequences of either choice are not yet measurable in a rigorous way, and anyone who claims certainty — in either direction — is overstating the evidence. We do not know how much training-data presence contributes to how often an assistant mentions a brand, and we will not pretend otherwise. What you can control today is narrow but real: knowing your current status, making the allow/block decision deliberately rather than by accident, and making sure the layer you configured is the layer that is actually doing the blocking.
If you want the broader frame for deciding where this fits in your priorities, AI visibility vs AI readiness explains the difference between being mentionable and being readable. And if you want your own site's current status checked across all the major AI crawlers at once, AI-Primed's free audit reports exactly that.