Allow AI agents through your firewall

Resolve the access barrier before changing files or structured data.

Guidance, not a guarantee. These steps never ask you to turn off your firewall or bot protection — only to let verified AI crawlers read your public pages.

What your site returned

Run an audit, or use Re-check access below, to see exactly what your site returns to automated visits.

Allow verified AI crawlers in your provider

Find the entry for our request in your firewall, CDN or hosting security log, using the fetch time and HTTP status above. Identify the exact rule that blocked it.
Prefer a verified-bot list if your provider offers one. Otherwise use the IP ranges or reverse-DNS checks each AI company publishes for its crawler.
Add a narrow exception for those verified crawlers on public pages only (GET and HEAD requests). Keep logins, checkout, admin pages and rate limits protected.
Only if nothing better is available, match by user agent with the self-hosted rules below — and understand that this is weaker.
Use Re-check access below to confirm.

The AI user agents to allow

These are the 15 AI user agents the audit checks. Where your provider offers a verified-crawler list (one that confirms the visitor really comes from the AI company), use it instead — a user-agent name alone can be faked.

ai-user-agents.txt

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-Web
anthropic-ai
PerplexityBot
Google-Extended
CCBot
Applebot-Extended
meta-externalagent
Bytespider
Amazonbot
YouBot
cohere-ai

Allow-listing verified AI crawlers lets agents read your site. It does not let anyone else in. If you are unsure, ask whoever manages your CDN — and test with a re-audit afterwards.

Self-hosted? nginx and Apache rules

If you run your own server without a CDN, these rules exempt the same agents from your bot blocks on public pages only. User-agent matching is weaker than a provider's verified-bot list.

Firewall allow-list for AI agents

nginx and Apache variants, commented block by block. Paste only the section for your server.

firewall-rules.conf

New · 75 lines added

Added: + # firewall-rules.conf — AI agent allow-list for your-domain.com
Added: + # Generated by AI-Primed. Guidance, not a guarantee.
Added: + #
Added: + # WARNING: user-agent matching is weaker than your CDN or firewall provider's
Added: + # verified-bot list. Anyone can send a request claiming to be "GPTBot".
Added: + # These rules therefore only exempt AI agents from bot BLOCKS on public
Added: + # GET/HEAD pages. They never bypass logins, admin pages, rate limits on
Added: + # forms, or your other security rules. Prefer provider-level verified bots.
Added: +  
Added: + # ======================================================================
Added: + # nginx
Added: + # ======================================================================
Added: +  
Added: + # 1) Recognise the AI agents. Put this inside the http { } block.
Added: + map $http_user_agent $agentready_ai_agent {
Added: +     default 0;
Added: +     "~*GPTBot"  1;
Added: +     "~*ChatGPT\-User"  1;
Added: +     "~*OAI\-SearchBot"  1;
Added: +     "~*ClaudeBot"  1;
Added: +     "~*Claude\-Web"  1;
Added: +     "~*anthropic\-ai"  1;
Added: +     "~*PerplexityBot"  1;
Added: +     "~*Google\-Extended"  1;
Added: +     "~*CCBot"  1;
Added: +     "~*Applebot\-Extended"  1;
Added: +     "~*meta\-externalagent"  1;
Added: +     "~*Bytespider"  1;
Added: +     "~*Amazonbot"  1;
Added: +     "~*YouBot"  1;
Added: +     "~*cohere\-ai"  1;
Added: + }
Added: +  
Added: + # 2) Only count read-only requests (GET/HEAD) as AI-agent reads.
Added: + map "$agentready_ai_agent:$request_method" $agentready_ai_read {
Added: +     default       0;
Added: +     "~^1:(GET|HEAD)$" 1;
Added: + }
Added: +  
Added: + # 3) If you block bots with your own variable (e.g. $bad_bot), make AI-agent
Added: + #    reads evaluate to 0 by listing them FIRST in that map — nginx uses the
Added: + #    first matching pattern. Example (keep your own patterns below):
Added: + # map $http_user_agent $bad_bot {
Added: + #     default 0;
Added: + #     "~*(GPTBot|ChatGPT\-User|OAI\-SearchBot|ClaudeBot|Claude\-Web|anthropic\-ai|PerplexityBot|Google\-Extended|CCBot|Applebot\-Extended|meta\-externalagent|Bytespider|Amazonbot|YouBot|cohere\-ai)" 0;
Added: + #     "~*(your-existing-bad-bot-patterns)" 1;
Added: + # }
Added: + #    and then, where you currently return 403 for bots:
Added: + # if ($agentready_ai_read = 0) { set $agentready_check $bad_bot; }
Added: + # if ($agentready_check = 1) { return 403; }
Added: +  
Added: + # 4) Keep sensitive paths protected for everyone, including AI agents.
Added: + # location ~ ^/(admin|login|wp-admin|checkout|account) {
Added: + #     # your existing protection stays here unchanged
Added: + # }
Added: +  
Added: + # ======================================================================
Added: + # Apache (2.4, .htaccess or vhost; needs mod_setenvif)
Added: + # ======================================================================
Added: +  
Added: + # 1) Recognise the AI agents.
Added: + SetEnvIfNoCase User-Agent "(GPTBot|ChatGPT\-User|OAI\-SearchBot|ClaudeBot|Claude\-Web|anthropic\-ai|PerplexityBot|Google\-Extended|CCBot|Applebot\-Extended|meta\-externalagent|Bytespider|Amazonbot|YouBot|cohere\-ai)" agentready_ai_agent=1
Added: +  
Added: + # 2) Only count read-only requests (GET/HEAD).
Added: + SetEnvIf Request_Method "^(?!GET$|HEAD$)" !agentready_ai_agent
Added: +  
Added: + # 3) If you block bots with an environment variable such as bad_bot
Added: + #    (e.g. "Require not env bad_bot"), clear it for AI-agent reads.
Added: + #    This line must come AFTER the lines that set bad_bot.
Added: + SetEnvIf agentready_ai_agent "^1$" !bad_bot
Added: +  
Added: + # 4) Keep sensitive paths protected for everyone, including AI agents.
Added: + # <LocationMatch "^/(admin|login|wp-admin|checkout|account)">
Added: + #     # your existing protection stays here unchanged
Added: + # </LocationMatch>

Verify it worked

After you apply the change, re-run only the access checks: a browser-style request, each AI agent's request, and your robots.txt rules.