The short answer
llms.txt is a proposed convention: a Markdown file placed at the root of your website that gives AI systems a curated, plain-text map of your most important content. It is not an established standard. As of October 2026, no major AI assistant has published a commitment to read or honour it, so its effect is unproven — we cannot guarantee it changes anything. It costs very little to add, which is the main argument for having one.
What the file is
The idea was published by Jeremy Howard in September 2024 at llmstxt.org. The proposal addresses a real problem: when an AI system fetches a web page, it gets the whole page — navigation, sidebars, cookie banners, footer links, tracking markup — and has to guess which parts matter. HTML is a presentation format, not a content format, and a lot of what is in it is noise to a machine reader.
It helps to be clear about what llms.txt is not. It is not a standard ratified by any body. It is not an access control — it cannot allow or forbid anything. It is not a ranking signal that any search engine or assistant has announced. And it is not a replacement for making your actual pages machine-readable; it is, at most, a well-labelled map to pages that must stand on their own.
llms.txt proposes that site owners publish a simple Markdown file listing the pages that actually matter, with one-line descriptions, so that a system that chooses to read it can skip the guesswork. Think of it less like robots.txt — which gives instructions — and more like a table of contents you hand to a visitor: here is what this site is, and here is where the good stuff is.
Where it goes and what a minimal version contains
The file lives at the root of your domain: https://yoursite.com/llms.txt. The proposed format is Markdown with a small set of conventions: an H1 with the site or project name, a blockquote summarising what the site is, and then H2 sections containing lists of links with descriptions.
A minimal but genuinely useful version looks like this:
# Example Books
> Independent bookshop in Edinburgh. We publish staff
> recommendations, event listings and a monthly reading guide.
## Recommendations
- [Staff picks](https://examplebooks.co.uk/staff-picks): Current recommendations from the team, updated monthly
- [Reading guide](https://examplebooks.co.uk/reading-guide): This month's themed reading list
## Visiting
- [Events](https://examplebooks.co.uk/events): Author readings and signings, with dates and booking links
- [Find us](https://examplebooks.co.uk/visit): Address, opening hours and accessibility informationTwo things make this useful rather than decorative. Every link has a one-line description that says what the page contains, so a reader — human or machine — can decide whether to fetch it. And the list is short. A llms.txt that dumps all 4,000 of your URLs recreates the problem it was meant to solve. Curate: ten to thirty links that represent what you most want understood about your site.
Choose links the way you would choose what to show a new customer in five minutes: the page that explains what you do, the page that proves it, the pricing or contact page, and the handful of resources you are proudest of. If a page would embarrass you as a first impression, it does not belong in the file.
How to add one
On a static site, create a file called llms.txt in whatever directory your web server serves as the document root. On most frameworks there is a public or static assets folder whose contents are served at the root — put it there. If you would rather not write the file by hand, our free llms.txt generator builds one in your browser — no account, nothing stored. Either way, the test is simple:
curl -sI https://yoursite.com/llms.txt | head -1
# Expected: HTTP/2 200Framework specifics: in Next.js, Astro, Vite and TanStack Start projects the file goes in the public directory, whose contents are served at the root. On WordPress, use a file manager or an SEO plugin that can serve a root-level text file, or ask your host to place it. Wherever it lives, the requirement is the same: https://yoursite.com/llms.txt must return 200 with the Markdown content.
Serve it as text/plain or text/markdown. Make sure it is not blocked by your robots.txt (it will not be, unless you have a blanket Disallow: /) and that your CDN does not redirect or challenge requests for it. Then keep it current — a stale llms.txt pointing at dead URLs is worse than none, because it actively misdirects anything that trusts it.
Why its effect is unproven
This section exists because most articles about llms.txt skip it, and you deserve the unvarnished version. robots.txt became effective because every major crawler committed to honouring it. llms.txt has not yet reached that point: as of October 2026, we are not aware of any major AI assistant that has published a commitment to fetch or follow llms.txt files. Some AI-adjacent tools and documentation sites have adopted it, and the convention may well spread — but adoption by tool-makers is not the same as adoption by the assistants your customers actually use.
The honest position is deliberately modest. The file is cheap to produce, harmless when curated, and plausibly useful if the convention takes hold — a reasonable bet, not a proven lever. We do not claim it will change how any assistant treats your site today, and you should be suspicious of anyone who does. If a major assistant publishes a commitment to read llms.txt, that will be visible in its first-party documentation, which is the only source worth trusting on the question.
The llms-full.txt variant
The proposal also defines an optional companion file, llms-full.txt, which contains not just links but the actual content of your key pages, concatenated in Markdown. The idea is that a system could ingest your documentation in a single fetch instead of crawling page by page. For documentation-heavy sites this is attractive; for most marketing sites it is overkill, and it doubles the maintenance burden, because now two files can go stale instead of one. Start with llms.txt. Consider the full variant only if your site's main purpose is documentation that machines need to absorb completely.
Keeping it honest and current
A llms.txt is a claim about your site, and like any claim it can rot. Pages move, descriptions drift, sections get renamed. Build the file from the same source of truth as your navigation or sitemap if you can — a small build script that regenerates it beats a hand-maintained file that nobody owns. If you cannot automate it, put a review reminder wherever you keep content chores, and treat a dead link in llms.txt with the same seriousness as a broken link in your main navigation.
Keep the descriptions factual, too. "The best project management tool for modern teams" tells a machine reader nothing; "Feature comparison and per-seat pricing for teams of 5–50" tells it exactly when the page is worth fetching. The file is read by systems deciding whether to spend a fetch on you — give them information, not adjectives. A useful test: read each line aloud and ask whether a stranger would know what is on the other end of the link. If not, rewrite the description until they would.
What this does not fix
llms.txt is a signpost, not a door. If AI crawlers are blocked by your robots.txt or firewall, a perfect llms.txt changes nothing — they cannot fetch the file or anything it points to. If your key pages are rendered by JavaScript and arrive empty, the signpost points at blank pages. The fundamentals come first: crawl access, content in the initial HTML, structured data. How to check if AI can read your website covers those checks in order, and How to let ChatGPT and Claude crawl your site covers access specifically.
If you would like the fundamentals checked before you spend time on conventions, AI-Primed's free audit tests crawler access, content visibility and key files in one pass — and includes the llms.txt generator if you decide the file is worth having.