Guides to making your website readable to AI
Practical, evidence-based guidance on AI crawler access, llms.txt, structured data and AI readiness — everything measured, nothing promised.
Featured report · live data
State of AI Readiness
Live anonymized figures from every site we've audited — failure rates, most-blocked agents and grade distribution, with explicit sample limitations.
Guides
How to let ChatGPT and Claude crawl your site
What AI crawlers need to fetch your pages, how robots.txt applies to them, and how to test that nothing upstream is still turning them away.
8 min read
Does blocking GPTBot hurt your SEO?
What blocking OpenAI's training crawler does and does not affect, why some owners legitimately choose to block it, and how to check your current status.
7 min read
What is llms.txt and do you need one?
What the llms.txt file is, where it goes, what a useful one contains — and the honest caveat that it is a proposal, not a standard, with unproven effect.
7 min read
How to check if AI can read your website
Four manual checks — crawler permissions, content in the initial HTML, structured data and reachable key files — with the exact commands to run.
8 min read
AI visibility vs AI readiness: what's the difference?
Visibility is whether assistants mention you; readiness is whether your site can be fetched and understood. Why the distinction matters and which to solve first.
7 min read
See what AI can read on your site
Run a free audit — no sign-up needed — and get an evidence-based score with specific fixes.
Run a free audit