How to let ChatGPT and Claude crawl your site

The short answer#

To let ChatGPT and Claude crawl your site, do two things: make sure your robots.txt does not disallow their crawlers, and make sure your server, CDN or firewall actually returns a 200 response when those crawlers request a page. The second step is the one most sites miss. robots.txt is only the first gate — a firewall or CDN rule further down the line can still block a crawler that robots.txt explicitly allows. This guide walks through each layer in order: what a crawler needs, how robots.txt rules apply to named AI crawlers, where edge rules override them, and the exact commands to test your own site.

What a crawler needs in order to fetch a page#

Before any crawler — AI or otherwise — can read a page, a chain of things has to go right. Your domain has to resolve, the TLS certificate has to be valid, and the server has to return a successful status code with the page content in the response body. For crawlers that respect robots.txt, there is an extra step at the front: they fetch /robots.txt first and check whether their own user-agent is permitted to fetch the URL they want.

Each of these is a separate failure point, and they fail independently. A site can have a perfectly permissive robots.txt and still return 403 to every AI crawler because of a web application firewall rule. A site can also allow a crawler and then serve it an empty page because the real content only appears after JavaScript runs. Checking one layer tells you nothing about the others.

How robots.txt applies to named crawlers#

robots.txt is a plain text file at the root of your domain — https://yoursite.com/robots.txt. It is organised into groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules that apply to that crawler. A crawler looks for the group that most specifically matches its own name; if none matches, it follows the User-agent: * group. If no group matches at all and the file has no wildcard group, everything is allowed by default.

The major AI companies publish the names their crawlers use. OpenAI's crawler documentation lists three separate agents: GPTBot, used for collecting training data; OAI-SearchBot, used for indexing content for search features; and ChatGPT-User, used when a ChatGPT user asks about a specific page and the system fetches it in real time. Anthropic's documentation lists ClaudeBot for training, plus Claude-User and Claude-SearchBot for retrieval and search. These names and roles come from each company's own documentation, checked in October 2026 — the companies can and do change them, so treat the documentation, not this article, as the authority.

Because these are separate user-agents, you can allow some and block others. A rule that disallows GPTBot has no effect on ChatGPT-User, and vice versa. This matters, because the three agents do very different things.

A robots.txt that allows the major AI crawlers#

Here is a complete example that allows the crawlers OpenAI and Anthropic document, while leaving the rest of your rules untouched:

# AI crawlers — explicitly allowed
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Everything else
User-agent: *
Disallow: /admin/
Disallow: /cart

Sitemap: https://yoursite.com/sitemap.xml

A subtlety worth knowing: if your existing file already has a User-agent: * group with a broad Disallow: /, adding Allow: / under a named crawler is not enough on its own in every parser — the safest form is an explicit group per crawler, as above, with no Disallow lines in those groups. An empty group for a named crawler means that crawler may fetch everything.

Retrieval agents are not training crawlers#

Allowing a retrieval agent and allowing a training crawler are different decisions, and you can make them independently. A training crawler (GPTBot, ClaudeBot) collects content that may be used to improve future models. A retrieval or search agent (ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot) fetches a page so an assistant can answer a specific user's question right now.

Some site owners want assistants to be able to quote and link to their pages when asked, but do not want their content used as training data. That combination is expressible in robots.txt: disallow GPTBot and ClaudeBot, allow the retrieval agents. Other owners want nothing to do with any of it, and can disallow all six. Others are happy to allow everything. All three are legitimate positions. It is your content and your choice — the point of this guide is to make the choice an informed one, not to push you toward any particular answer.

When robots.txt allows a crawler but the server still blocks it#

robots.txt is a convention, not an access control. It asks compliant crawlers to stay away; it does not and cannot grant access. Actual access is decided by your server, your CDN and your firewall — and those layers frequently contradict the robots.txt file.

The common culprits, roughly in order of how often our audits encounter them:

• CDN or edge bot-management rules that challenge or block requests whose user-agent looks like a bot. Cloudflare, for example, offers AI-bot blocking controls, and its defaults for new configurations have changed over time — check your own dashboard rather than assuming.
• Web application firewall rules that block by user-agent string or by IP range.
• Server-level rules in nginx, Apache or .htaccess that return 403 to unknown agents.
• Rate limiters that return 429 to crawlers fetching many pages quickly.

Because these layers sit in front of your application, your application logs may show nothing at all — the request never reached it. This is why editing robots.txt so often appears to 'do nothing': the file was never the thing blocking the crawler. If your site is behind Cloudflare, our free Cloudflare AI bot check tests exactly this layer — no account needed.

How to test#

You can test the whole chain from any terminal. First, confirm what your robots.txt actually says:

curl -s https://yoursite.com/robots.txt

Then request a page while presenting yourself as each crawler, and look at the status code that comes back:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  https://yoursite.com/

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  https://yoursite.com/

A 200 means the page was served. A 403 means something is actively refusing the request — go looking in your CDN and firewall rules, not in robots.txt. A 301 or 302 means a redirect; follow it with curl -L and check where it lands, because redirect chains are another place crawlers get lost. Test a few representative pages, not just the homepage: templates often behave differently.

One caveat: spoofing a user-agent does not perfectly reproduce a real crawl. The real crawler comes from the company's IP ranges, and some firewalls key on IP rather than user-agent. Your server or CDN logs are the ground truth — if GPTBot's requests appear there with 200 responses, it is genuinely getting in. Our methodology page describes how our audit combines both approaches.

What this does not fix#

Letting a crawler through the door does not mean it can make sense of what it finds. If your main content only appears after JavaScript runs, if the HTML is a tangle of divs with no headings, or if there is no structured data, a crawler that successfully fetches the page may still extract nothing useful. Those are separate problems with separate fixes — How to check if AI can read your website walks through them.

And to be plain about the limit of all of this: allowing AI crawlers does not guarantee that any assistant will mention, recommend or link to your site. Crawl access is a precondition, not an outcome. What it does is remove a silent, common and entirely fixable reason for being absent.

AI-Primed's free audit checks both layers described here — robots.txt permissions and the actual response each major AI crawler gets — in one pass, if you would rather not run the curl commands yourself.