What Is llms.txt? An Honest Guide From People Who Ship It
llms.txt is a machine-readable index of your site for AI tools. What it does, what the data says about adoption, and whether your business needs one.
llms.txt is having a moment: agencies sell it as an AI-visibility silver bullet, skeptics call it dead on arrival, and most explanations are copied from each other. We ship llms.txt files for clients and on our own site, so here is the honest version: what it is, what the measurements actually show, and when it's worth your time.
What is llms.txt, exactly?
llms.txt is a plain-markdown file at your site's root (yoursite.com/llms.txt) that gives AI systems a curated index: who you are, what you do, and annotated links to your most important pages. Where robots.txt tells crawlers what they may not read, llms.txt suggests what they should read: a menu instead of a fence.
The idea was proposed in September 2024 by Jeremy Howard, co-founder of fast.ai, as a response to a real problem: language models work with limited context windows, and a typical webpage buries its substance under navigation, scripts, and boilerplate. A clean, hand-curated index lets a machine find the substance directly. The format is deliberately simple: an H1 with your name, a summary, and H2 sections of annotated links. You can read ours at viziblty.com/llms.txt.
Does anything actually read llms.txt?
Here's the honest part. Ahrefs studied 137,210 domains with llms.txt files and found 97% of them received zero requests in May 2026: nothing fetched them at all. Among files that were fetched, actual AI retrieval bots made just 1.1% of the requests. Meanwhile 28% of studied domains had published one. Adoption by websites: high. Adoption by AI companies: thin.
And Google is explicit that its AI features need no new machine-readable files: its May 2026 optimization guide's core message is that AEO and GEO 'is still SEO'. So anyone selling llms.txt as a ranking lever for Google AI is selling something Google itself disclaims. The realistic case is narrower: some AI tools and agents do read it, it costs an hour, and it can only help the machines that choose to look.
Should my business have an llms.txt file?
Yes, on the same logic as printing a menu: cheap, harmless, occasionally decisive. But it belongs at the end of the checklist, after the things with proven weight.
| Priority | Work | Why it outranks llms.txt |
|---|---|---|
| 1 | Crawlable, server-rendered site | If bots can't read your pages, nothing else exists |
| 2 | Consistent business facts everywhere | Engines verify before they recommend |
| 3 | Schema markup | The established machine-readable layer engines do parse |
| 4 | Deep, evidence-rich content | What actually gets absorbed into answers |
| 5 | llms.txt | One hour, low odds, non-zero payoff: do it last, then forget it |
How do I write a good llms.txt file?
Start with an H1 of your business name, then a one-paragraph summary a machine could quote verbatim, then H2 sections grouping annotated links (services, locations, pricing, guides), each with a one-line description of what's on that page. Curate hard: ten essential pages beat eighty indiscriminate ones, because the whole point is pre-digested relevance.
Two cautions from the field. Keep it synced: a stale llms.txt that contradicts your site is worse than none, and the Ahrefs study found bad actors already probing these files for prompt injection, so treat yours like code: versioned, reviewed, minimal. And don't confuse it with schema markup: the two do different jobs, and the technical SEO layer needs both. Writing it is part of our standard AI search optimization scope; seeing whether you have one is part of the free audit.
Frequently asked questions
Is llms.txt an official standard? No. It's a community convention with a published spec at llmstxt.org: no standards body governs it and no AI company is obligated to read it.
Does llms.txt affect my Google rankings? No. Google states its AI features require no special files, and llms.txt plays no role in classic rankings either way.
What's the difference between llms.txt and robots.txt? robots.txt restricts: it tells crawlers what not to access. llms.txt recommends: it points AI tools at your best content. They coexist at your site root doing opposite jobs.
Can llms.txt hurt my site? Only through neglect: an outdated file that contradicts your live site feeds machines wrong facts, and an uncontrolled one is a small prompt-injection surface. Keep it current and minimal.
Sources
- 97% of llms.txt files received zero requests in May 2026; AI retrieval bots made 1.1% of requests to fetched files; 28% of 137,210 studied domains publish one: Ahrefs llms.txt study (June 2026)
- llms.txt was proposed in September 2024 by Jeremy Howard, co-founder of fast.ai: OptimyCloud guide
- Google's AI features require no new machine-readable files; AEO/GEO "is still SEO": Google Search Central, AI optimization guide (May 2026)