How to check if AI can read your website

The short answer#

Four things determine whether AI systems can read your website: whether their crawlers are permitted and actually let in, whether your real content is present in the initial HTML, whether you provide structured data, and whether key files like robots.txt and your sitemap are reachable. You can check all four manually in about half an hour with a terminal and a browser. This guide gives you the exact commands.

Test 1: are AI crawlers permitted — and actually let in?#

Start with robots.txt. Fetch it and read it:

curl -s https://yoursite.com/robots.txt

You are looking for two failure modes. A User-agent: * group containing Disallow: / blocks every compliant crawler, AI included. And named groups — GPTBot, ClaudeBot and the rest — may have been added by a plugin, a host or a previous developer without anyone noticing. If you find Disallow rules you did not intend, How to let ChatGPT and Claude crawl your site explains the fix.

Then test whether the server honours what robots.txt says. robots.txt is advisory; the firewall is actual. Request a page as each major crawler:

for ua in \
  "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0)"
do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$ua" https://yoursite.com/)
  echo "$code  $ua"
done

You want 200 for each. A 403 means a CDN or firewall rule is blocking that user-agent — very common on Cloudflare and similar platforms, where bot controls can contradict a permissive robots.txt. A 301 or 302 means a redirect; check where it leads. Run this against a few representative pages, not just the homepage. If your site is on Cloudflare, our free Cloudflare AI bot check runs this comparison for you — no account needed.

Two other codes are worth recognising. A 429 means rate limiting — the crawler is nominally welcome but being throttled, which slows the discovery of new pages. A 503 with a challenge page means a bot-protection interstitial: a human clicks through it, a crawler cannot. Both look like "the site is up" in a browser while meaning "machines get nothing", which is exactly why the browser is the wrong tool for this test.

Test 2: is your content in the initial HTML?#

Most AI crawlers do not run JavaScript. They read the HTML your server sends in the first response. If your page is built by a client-side framework that renders content in the browser, the crawler may receive an empty shell: a div and a script tag where your content should be.

The check takes ten seconds. Fetch the raw HTML and search for a sentence you know is on the page:

curl -s https://yoursite.com/pricing | grep -i "per month"

If the sentence appears, that content is in the initial HTML. If nothing comes back — but you can see the sentence when you open the page in a browser — your content is being rendered by JavaScript after load, and most AI crawlers never see it. In the browser, compare View Source (what the server sent) with the inspector in DevTools (what JavaScript built). The difference between those two views is exactly what crawlers miss.

The fix is server-side rendering, static generation or pre-rendering for your key pages — a framework-level change, not a tweak. It is the single most impactful fix our audits surface, and also the one that most often requires a developer.

Test more than the homepage. Check a blog post, a product or service page and your pricing page — the pages a potential customer's question would actually land on. Sites frequently server-render the homepage for performance while leaving interior templates client-rendered, so a homepage pass can hide a sitewide problem. If you know your stack, the framework is a clue: a plain single-page application renders everything in the browser unless someone deliberately added pre-rendering, while Next.js, Astro, Nuxt and similar frameworks server-render by default but can be opted out per page.

Test 3: do you provide structured data?#

Structured data — JSON-LD markup in your page head — tells machines explicitly what a page is: an organisation, a product, an article, an FAQ, with named fields rather than guessed meaning. Check whether your pages carry any:

curl -s https://yoursite.com/ | grep -o 'application/ld+json' | head -1

If that prints nothing, there is no JSON-LD on the page. If it prints, view the block itself and check it describes the right entity. A minimal organisation example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Books",
  "url": "https://examplebooks.co.uk",
  "description": "Independent bookshop in Edinburgh."
}
</script>

Schema.org defines the vocabulary; the search engines' own documentation describes which types they use. Structured data is not a ranking trick — it is a way of removing ambiguity about who you are and what the page contains, which benefits every machine reader, AI crawlers included.

Syntax matters as much as presence. A JSON-LD block with a trailing comma or an unclosed brace is ignored entirely, silently. After any change, paste the block into a validator — Google's Rich Results Test and the Schema.org validator both parse JSON-LD and report errors. And match the type to the page: an Article block on your homepage, or an Organization block with no name field, creates confusion rather than removing it.

Test 4: are your key files reachable?#

Finally, confirm the files machines look for actually exist and return 200:

for path in /robots.txt /sitemap.xml /llms.txt; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "https://yoursite.com$path")
  echo "$code  $path"
done

robots.txt and sitemap.xml are the established ones; llms.txt is a newer, unproven convention — see What is llms.txt? for an honest assessment before you bother. A 404 on the sitemap is worth fixing: it is the map traditional and AI crawlers alike use to find pages your navigation does not link prominently.

While you are there, open the sitemap itself and sanity-check its contents. It should list the canonical URLs of pages you want found — not redirected URLs, not pages behind login, and not thousands of near-duplicate filter or tag pages. A sitemap that cries wolf teaches crawlers to distrust it; a tight one gets your important pages recrawled faster.

How often to re-check#

These are not one-off checks. Every deploy can change what the initial HTML contains; every plugin update can rewrite robots.txt; every CDN rule change can start blocking a crawler that was fine last week. Re-run the four tests after any migration, redesign, hosting change or security-plugin update — and on a schedule even when nothing has "changed", because the crawlers themselves change. The user-agents and behaviours in this guide reflect first-party documentation checked in October 2026.

If manual re-checking sounds tedious, that is precisely the gap scheduled monitoring fills — re-running these checks and alerting when a crawler that could read your site suddenly cannot. But the manual version above will always be free, and now you have it. Bookmark it, and share it with whoever maintains your site; the checks only help if someone actually runs them.

What these checks do not tell you#

Passing all four checks means your site is technically readable. It does not tell you whether any assistant will mention, recommend or link to you — that depends on the user's question, the assistant's other sources and factors no site owner controls. It also does not tell you exactly what each assistant does with your content, because their behaviour varies and changes without notice; everything in this guide reflects first-party documentation checked in October 2026, and the companies involved update their crawlers and policies regularly. And it does not measure quality: a readable page with thin content is still thin content. These checks establish that the door is open and the lights are on — what visitors find inside is a separate, harder question.

What these checks do give you is a solid answer to the question that has to come first: can machines get in and read what you published? If you would rather not run the commands yourself, AI-Primed's free audit runs all four — plus the firewall checks that catch the cases curl misses — and reports the result page by page.