State of AI Readiness
Access blockers
Front door locked: a firewall or bot shield refuses agents, or the site doesn't respond — before a single page is read. Usually unintentional.
Content & discovery gaps
Reachable, then scored on what agents can understand. Across all sites, 14% have no homepage structured data and only 6% publish an llms.txt.
Most low scores aren't poor optimisation — the door is shut. Fix access first; content fixes only count once agents can get in.
of scored sites get an F for agent readiness
Most of the web is effectively invisible to AI agents — hard to crawl, hard to read, or impossible to quote.
fail outright — blocked or unreachable, agents get nothing
The homepage refuses automated requests or doesn't respond at all, so there is nothing to score. See the explainer below.
have no structured data on the homepage at all
JSON-LD is how an agent knows your page is a product, an article or a company — not just text.
have any structured data at all
Even a single Organization or WebSite block puts a site ahead of most of the web.
block GPTBot in robots.txt
3% block at least one AI crawler — often without realizing it.
publish an llms.txt file
The emerging standard that tells language models what a site is and where its key pages live.
of online stores have complete Product schema
Based on 17756 stores. Without name, image, description and offers, agents can't quote your price.
What "fails outright" means
When a site's firewall or bot protection refuses our request — or the site simply doesn't respond — there is no homepage to read, so there is no score. The audit is recorded as a failure, because an AI agent hitting the same wall gets the same result: nothing.
This is now one of the most common problems we find. In most cases the site owner never chose to block AI agents — a default "block bots" setting in a firewall or CDN does it silently. The fix is usually to allow-list the major AI crawlers, not to turn protection off. Our guide to allowing AI agents walks through it provider by provider.
Most common critical problem
Homepage could not be fetched
Found on 64% of audited sites.
- 2No JSON-LD structured data on homepage14%
- 3Homepage blocks automated requests5%
- 4Content is likely JavaScript-rendered4%
- 5Major AI agents blocked: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended2%
How the whole web compares
The average scored site gets 21/100, median 66. Here's how the grades are distributed:
- A
- 92–100
- B
- 80–91
- C
- 65–79
- D
- 50–64
- F
- < 50
Where does your site land?
Free audit in under a minute — see your score against these numbers.