Write one if you like; it costs ten minutes and hurts nothing. Just don't book it as a visibility strategy. Server logs from 137K domains show 97% of llms.txt files were never fetched in May 2026, and a preregistered study of 3,048 websites found cited sites no more likely to have one. Google Search ignores the file.

The idea, and why it spread

llms.txt is a Markdown file at the root of a site that sums up what the site is about and links to its important pages. Jeremy Howard of Answer.AI proposed the format in September 2024 with a reasonable argument: web pages are built for people, and an AI agent that has to strip out navigation, ads, and scripts to read them wastes time and tokens. A clean summary in one predictable place is cheaper to read.

The proposal found an audience. In Ahrefs' sample of 137K domains using its web analytics, 28% published an llms.txt by mid-2026. The major AI labs publish the files for their own developer documentation, and in late May 2026 Chrome's Lighthouse added an llms.txt check to its experimental agentic-browsing audits. This is a real format with real adoption, not a fringe experiment.

Evidence 1: 97% of the files never get read

If the file worked the way generator landing pages describe, AI crawlers would request it constantly. Server logs say otherwise. Ahrefs analyzed server logs and live traffic for 137,210 domains using its analytics products. The headline finding: 97% of published llms.txt files received zero requests in May 2026. Nothing fetched them at all.

The 3% that did get fetched tells its own story. Most requests came from bots of one kind or another (96%). Named AI tools accounted for 19.5% of fetches, with GPTBot first and Claude-Code second, ahead of every AI search bot. Another 12% came from generative-engine-optimization tools, llms.txt checkers, and researchers: the industry reading its own homework.

The sharpest finding sits near the bottom of the report. AI bots made zero requests for llms.txt files that didn't exist. They never go looking. Generator pages that promise a file "built to the spec ChatGPT, Claude, and Perplexity look for" are describing a lookup that doesn't happen in the logs.

Evidence grade: large-sample observational log data, single month (May 2026), one analytics provider

Evidence 2: cited sites don't have the file more often

Fetch logs measure crawling. The second study measures the thing people actually care about: being cited. Matt Timmermans of Timmermans Media ran 400 non-branded questions through the ChatGPT and Gemini APIs in September 2026, three runs per question, collected every website the answers cited, and checked those sites for an llms.txt. The comparison group was websites from the Google top 10 for the same questions that did not get cited. 3,048 websites in total.

The result: 22.4% of the checkable cited sites had the file. So did 25.4% of the checkable uncited ones. The intervals overlap, and if anything the direction runs backwards. Looked at the other way around, 1,033 of the 1,331 checkable cited sites (78%) had no usable file at /llms.txt. Sixteen of the twenty most-cited sites, including the Dutch consumer association and the national government's site, had none at all.

The study deserves credit for its methods, because they are rare in this market. It was preregistered, the data is public on Zenodo, and the author separates what he measured from what he didn't: this is a descriptive comparison, not a causal one, and it cannot show whether adding the file would change a site's citations. He re-weighted the two groups for site type and reported that the small gap shrinks further when he does. A null result delivered with that much care is worth more than a positive result delivered with none.

Evidence grade: preregistered descriptive comparison, single measurement day (September 26, 2026), Dutch-language questions, ChatGPT and Gemini APIs only

What Google says

Google's optimization guide for generative AI features, published in May 2026, has a section called "Mythbusting generative AI search," and llms.txt is first on its list of things you can ignore:

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

The guide adds that creating one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Ahrefs reports that when Lily Ray pressed Google's John Mueller on why Chrome's Lighthouse team would audit for a file that Search ignores, he called llms.txt "not done for search," a "temporary crutch, perhaps to save some tokens" for AI coding tools reading developer documentation.

That description matches the file's actual use case. It is a format for coding agents reading developer documentation, not a channel into search or chatbot citations.

Where the big claims come from

None of this has slowed the marketing. Generator pages promise that once you publish the file, "the next time ChatGPT, Claude, or Perplexity touches your site, it gets a clear map instead of guessing." Plugin landing pages tell you that without the file, AI systems "rarely cite or recommend you," and one carries a testimonial claiming traffic from Perplexity "tripled." Checker tools scan for a file the major engines mostly never request.

The pattern repeats across the category. The claim is stated as fact. The supporting link, when there is one, points to another vendor's post. The studies that actually control for anything come back empty. We walked through the same pattern for JSON-LD separately (what schema markup really does to AI citations): vendor pages quote big multipliers for markup changes, while the controlled tests find nothing. When a vendor quotes a large lift for a markup tweak, check whether the number comes from a controlled study or from the vendor's own benchmark. So far, markup lifts have all been the second kind.

So, should you write one?

Here is the verdict we use ourselves.

Worth the ten minutes:

  • You publish developer documentation. This is the one use case with logged evidence behind it. The fetches that do happen come mostly from coding tools: GPTBot and Claude-Code were the two most active AI fetchers in the Ahrefs logs, and Mueller described the format as a crutch for exactly this audience. If your product ships an API, this includes you.
  • Your CMS or SEO plugin already generates one. Check it rather than ignore it. Auto-generated files are often empty or stale, and an empty file adds nothing for the rare agent that reads it.
  • You want a canonical, human-readable summary of your site in one predictable place. That has plain documentation value even if no crawler shows up.

Not worth the budget:

  • Buying llms.txt as an AI visibility strategy. Both of the best available datasets point the same way: the file is not the lever.
  • Hand-curating and re-submitting a bespoke file every month. In the Timmermans study, most cited sites had no file at all and got cited anyway. The habits those sites do share are the ones worth your hours: being named across the web your buyers read (why being named beats ranking), and pages that answer the category question completely. If you haven't measured whether AI names you at all, the manual test takes one afternoon and no tools.

Where we fit

We run these checks for a living. In the CiteGap diagnostic, llms.txt sits in the hygiene tier: we check whether yours exists and says something useful, and we don't pretend it moves your citation rate. The checks that can actually cost you visibility sit upstream: whether AI crawlers can reach your pages at all, whether robots.txt blocks the bots that answer users, whether your pages say plainly what your product is called. If you want a first look for free, the instant check runs against your homepage in seconds and shows each result on the page.

Run the free instant check

Every number in this piece was checked against its linked source on November 4, 2026. This market moves fast; re-check before you repeat any of it in a client deck.

Sources