Do I need an llms.txt file?

Someone on your team read a thread about llms.txt, or an audit tool flagged that you are missing one, or it showed up on an agency proposal. It takes ten minutes to add, so the question you are actually asking is not whether it is cheap. It is whether it does anything.

The short answer is no. The longer answer is more useful, because llms.txt is the cleanest available case study in how a plausible-sounding idea becomes conventional wisdom without anyone checking whether it works. The verdict matters less than the walk to get there — the next proposed AI standard is already on its way.

What llms.txt actually is

llms.txt is a proposal published by Jeremy Howard on September 3, 2024, for a Markdown file at the root of a domain — https://example.com/llms.txt. Its stated purpose is to provide “information to help LLMs use a website at inference time.” The reasoning is sound. Model context windows are small relative to a whole website, HTML pages are padded with navigation and boilerplate, and a curated, clean map of what matters on a site would in principle save a model from guessing.

The format is deliberately simple: an H1 with the site name, a blockquote summary, then H2 sections holding lists of links with short notes, plus an optional section for material a model can safely skip.

# Acme Analytics

> Acme Analytics is self-serve product analytics for small SaaS
> teams. Event-based pricing, no seat limits.

## Docs

- [Quickstart](https://acme.example/docs/quickstart): Install the
  SDK and send your first event in about ten minutes.
- [Event reference](https://acme.example/docs/events): Every
  event type, property, and limit.
- [Pricing](https://acme.example/pricing): Plans, event caps,
  and overage rates.

## Optional

- [Changelog](https://acme.example/changelog): Release notes.
- [Security](https://acme.example/security): SOC 2, data
  residency, retention.

That is a reasonable artifact. If you were building a retrieval system and every site offered one, you would probably read it. Which is exactly why the idea spread — it is the kind of proposal that sounds correct before you check anything. The concept is not the problem. The problem is the claim that attached itself to it: that publishing one improves your visibility in AI answers.

The evidence: 137,210 domains and their server logs

The most direct test of whether AI systems read llms.txt files is to check whether anything requests them. Ahrefs did that, publishing a study on May 2026 data covering 137,210 domains and the actual request logs for the file.

Read the middle number carefully. It is not that the files were fetched and ignored. Ninety-seven percent of the 38,360 published files were never fetched by anything at all during the study period. For a file whose entire function is to be read, that is close to a complete answer on its own.

The remaining sliver is where it gets interesting. Of the traffic those files did receive, 96% was bot and 4% human. And of that bot traffic, AI bots accounted for just 19.5%. The rest was SEO and GEO audit tools, generic crawlers, and research bots built specifically to discover llms.txt files in the wild.

So the main thing reading llms.txt files is the tooling that checks whether you have one. An audit flags a missing llms.txt, you add it, and the next audit comes back to confirm it is there. That loop generates activity, dashboards, and completed tickets. It does not generate citations.

If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration. … The cons outweigh the pros right now.

Ahrefs, llms.txt study, May 2026 data, 137,210 domains

What Google says in its own documentation

Google's Search Central documentation is unambiguous here. In its guidance on AI features in Search, Google states: “You don't need to create new machine readable files, AI text files, or markup to appear in these features.” The same page says a page simply has to be indexed and eligible to appear in Google Search to be eligible as a supporting link in AI Overviews or AI Mode.

Google went further on May 15, 2026 (updated June 24, 2026) with a dedicated guide, “Optimizing your website for generative AI features on Google Search.” It debunks five myths, and the llms.txt myth is the first one listed. That position is convenient for Google, and you should weigh it accordingly — but on the narrow factual question of whether Google reads an AI text file, Google is the only authority that exists.

You will also see llms.txt statements attributed to individual Googlers circulating in roundups and LinkedIn posts. Some are real, some have been paraphrased into something firmer than what was said, and most are three aggregations deep by the time you read them. You do not need them — the documentation says the same thing and does not shift meaning each time it is re-quoted.

The argument from silence, stated carefully

Here is the strongest negative evidence available, with its limits attached. None of the major AI operators' own crawler documentation mentions llms.txt anywhere:

• OpenAI documents GPTBot, OAI-SearchBot, ChatGPT-User, and OAI-AdsBot, including published IP ranges and how robots.txt applies to each.
• Anthropic documents ClaudeBot, Claude-User, and Claude-SearchBot, including crawl-delay support.
• Perplexity documents PerplexityBot and Perplexity-User, and is candid that the user-triggered one generally ignores robots.txt.
• Google documents Googlebot, Google-Extended, Google-CloudVertexBot, and GoogleOther, and separately documents its AI features.
• Meta, Mistral, Apple, and Amazon each document their crawler families — Meta-ExternalAgent, MistralAI-Index, Applebot and Applebot-Extended, Amazonbot and Amzn-SearchBot.
• Common Crawl, whose archive feeds training data across the industry, documents CCBot and how to verify it.

Every one of those pages goes into real operational detail — user-agent strings, robots.txt handling, IP ranges, propagation delays, what happens to your inclusion if you block a given token. Not one of them mentions a Markdown file at your site root.

Absence of a claim is not proof of non-use. An operator could read llms.txt without documenting it, and documentation lags behavior everywhere in this industry. Say that out loud rather than treating silence as a verdict. But set it next to the server-log data and the two independent lines point the same direction: nobody says they read it, and almost nothing is fetching it.

It is a small file, so why not just add one?

This is the fair version of the counterargument and it deserves an answer rather than a shrug. Three reasons, none of which is that llms.txt is dangerous.

1. It is a second copy of your site's structure. Hand-maintained, it is accurate the day you write it and wrong three months later, after the pricing page moves and two posts get merged. If something ever does start reading it, it reads the stale version.
2. It displaces effort. The hour spent drafting and reviewing an llms.txt is an hour not spent on crawler access, index coverage, or one page that answers a question customers actually ask. Those have evidence behind them.
3. It substitutes for measurement. “We added llms.txt” is a satisfying line in a status update. It feels like AI work and it closes the item, and it produces nothing checkable afterward — which is exactly why it is comfortable.

A test to run on the next proposed AI standard

llms.txt will not be the last file, tag, or format proposed as the key to AI visibility. Rather than judging the next one on whether it sounds sensible — llms.txt sounds extremely sensible — run three questions against it.

1. Has an operator committed to it in its own documentation? Not a conference remark, not a quote in a roundup. The crawler docs, the developer pages, the search documentation the company maintains and updates.
2. Do server logs show it being fetched? Requests are cheap to measure and hard to argue with. If a thing is meant to be read, you can check whether it is — on your own domain, today.
3. Has anyone measured an outcome? Not a mechanism that sounds plausible. A study with a sample size, a date, and something to compare against.

A proposal that fails all three is not necessarily wrong. It is unevidenced, which means the honest description is “we do not know yet” and the honest budget is close to zero. Two adjacent claims you have probably heard score about the same way.

Schema is the interesting row, because it has a real job. It still drives classic Google rich results, it is cheap, and it plausibly reduces ambiguity for any parser. Keep it. But “schema markup drives AI citations” is a separate claim, and as of mid-2026 there are no peer-reviewed studies on it and no public technical statement from OpenAI, Anthropic, or Perplexity about whether they parse it at all. One analysis, from Search Atlas, found no correlation between schema coverage and citation rates.

The seeded-mention claim is weaker still. Google's May 2026 guide lists chasing inauthentic brand mentions among its five myths and states that its spam systems filter them — an operator saying, in writing, that the tactic is actively countered.

What to do instead, today

Everything below has either operator documentation or a measured outcome behind it, which is more than llms.txt currently has.

1. Check crawler access first. Open your robots.txt and confirm you are not blocking the retrieval bots you want: OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Googlebot. OpenAI states plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
2. Confirm you are actually indexed. Google's documentation is explicit that a page must be indexed and eligible in Google Search before it can appear as a supporting link in AI Overviews or AI Mode. Index coverage is the prerequisite, not a file.
3. Check your own access logs. If you already have an llms.txt, filter the last 30 days for requests to /llms.txt and group by user agent. Two minutes, and it tells you more about your own site than any study can.
4. Write question-shaped pages. Ahrefs' February 2025 analysis of 1.4 million ChatGPT prompts found cited page titles had higher cosine similarity to the prompt (0.602) than uncited ones (0.484), and higher still against fan-out sub-queries (0.656).
5. Use natural-language URL slugs. In that same study, pages with natural-language slugs were cited 89.78% of the time versus 81.11% for non-natural-language slugs. Free at publish time, painful to retrofit.
6. Measure something monthly. Pick five questions a real customer would type, check who gets cited today, re-check in thirty days. That number moves. llms.txt does not.

The verdict on llms.txt matters less than the method that produced it. A file nothing requests is a clean, checkable example of a whole category of AI-visibility advice: plausible, widely repeated, never measured. Run the three questions on this one and the next one takes five minutes.