Do You Need an llms.txt File? What the Crawler Data Actually Shows in 2026

No — not for AI search citation, not yet. llms.txt won’t get you cited by Google, which has ruled it out twice on the record, and Perplexity’s own crawler documentation doesn’t confirm it either. Three independent studies found no measurable citation lift. Its one confirmed use is developer tooling — Cursor, Claude Code — not ChatGPT or Perplexity answers.

That’s the short version. Here’s the actual evidence behind it, and the one situation where implementing it is genuinely worth your time.

What Is llms.txt?

llms.txt is a plain Markdown file placed at the root of a domain — yoursite.com/llms.txt — that lists a site’s most important pages for a language model to read, along with a one-line description of each. It’s a curated index, not a permissions file: it doesn’t tell a crawler what it’s allowed to fetch, it tells a model (already permitted, per robots.txt) which pages are worth fetching and why. Think of it as a table of contents written for a machine reader with a limited context window, not a security gate.

Who Created llms.txt, and Why?

Jeremy Howard — co-founder of Answer.AI and fast.ai — published the original proposal on September 3, 2024, hosted at llmstxt.org alongside Answer.AI’s own site. The problem he was solving was narrow and specific: LLM context windows are too small to hold most full websites, and converting a real page — navigation, ads, JavaScript and all — into clean text a model can use is slow and error-prone. His own FastHTML project served as the reference implementation.

That origin matters for how you should read the rest of this article. Howard built llms.txt to solve a documentation-retrieval problem for developer tools. He didn’t build it as an SEO or GEO mechanism, and nothing in the original spec claims it moves citation rankings. Most of the confusion since has come from treating a developer-tooling convention as if it were a search-visibility lever.

llms.txt vs. robots.txt vs. llms-full.txt: What Each File Actually Controls

These three get conflated constantly, and they do genuinely different jobs.

If you only fix one thing after reading this article, check that AI crawlers can actually reach your site via robots.txt first. That’s the file every documented AI crawler respects. llms.txt is optional polish on top of access that robots.txt already grants or denies.

What Does an llms.txt File Actually Contain?

The spec is deliberately minimal:

• An H1 with the site or project name
• A blockquote with a one-sentence summary — this is the line a model will lean on most heavily if it reads the file at all
• Optional context paragraphs
• H2-grouped sections, each a bulleted list of links with a short description after a colon

That’s it. No JSON, no XML, no required schema validation. The low barrier to entry is exactly why implementation takes under 30 minutes once you’ve decided it’s worth doing — which is the harder question.

Does Any AI Platform Actually Use It?

This is where most of the existing content on this topic goes vague or, worse, confidently wrong in both directions. Here’s platform status, checked against primary sources rather than secondary blog claims.

Google

Settled, and settled publicly. Gary Illyes, an Analyst on the Google Search team, said as much at Search Central Live in July 2025, comparing llms.txt to the old keywords meta tag — a signal site owners could set, that Google simply doesn’t read. John Mueller reiterated the position separately later that year. Google’s own AI optimization documentation, updated in mid-June 2026, states plainly that no new machine-readable AI file is required to appear in Google Search or its generative features. In December 2025, Google briefly added llms.txt files across a number of its own developer and documentation sites — then pulled them within 24 hours. Mueller attributed it to a sitewide CMS update rather than a policy shift. If your target is Google AI Overviews, this file does nothing for you. Full stop.

Perplexity

This is the one platform where the internet genuinely disagrees with itself, and it’s worth being precise about why. A number of GEO blogs from earlier in 2026 state flatly that “Perplexity reads llms.txt.” Perplexity’s own published crawler documentation, though, doesn’t say that — it points to a documentation index for AI agents, which describes Perplexity’s own llms.txt file for its own docs, not a commitment to fetch and weigh other sites’ files. Other independent trackers report finding no such support anywhere in Perplexity’s crawler documentation. The honest position: unconfirmed. Not denied, the way Google denied it — just never actually stated by Perplexity, despite how often the claim gets repeated as settled fact.

ChatGPT / OpenAI

No public commitment. Worth knowing the actual mechanics here, because “does ChatGPT read llms.txt” is really three separate questions. OpenAI runs three distinct crawlers: GPTBot (training data collection), OAI-SearchBot (the one that actually powers what gets cited in ChatGPT search results), and ChatGPT-User (fetches triggered by a live user request). None of OpenAI’s published documentation for any of the three mentions llms.txt as an input.

Claude / Anthropic

Genuinely nuanced, and this is where most competitor content collapses two different things into one claim. Anthropic ships an llms.txt for its own developer documentation and llms.txt shows up as a convention Claude Code and MCP-based tooling check for when working with a codebase or docs site. That’s real, and it’s Anthropic-confirmed. What isn’t publicly documented is ClaudeBot — the crawler behind what Claude.ai cites in chat answers — reading and weighting other sites’ llms.txt files for citation purposes. “Anthropic supports llms.txt” is true for developer tooling. It is not the same claim as “Claude cites pages because of their llms.txt file,” and treating them as interchangeable is the single most common error in coverage of this topic.

Coding agents and MCP

This is the one place llms.txt has real, confirmed, everyday use. Cursor, Windsurf, Cline, and custom MCP servers check for an llms.txt file before ingesting a site’s documentation, because it saves the agent from guessing at a URL structure or crawling blind. If your actual audience is developers integrating with your API through these tools, this isn’t a maybe — it’s infrastructure they’re already looking for.

What Three Independent Studies Found

Individually, each of these gets cited somewhere. Almost nothing cross-references all three, which is a gap worth closing, because they converge on the same answer from three different angles.

Three different methodologies — adoption correlation, citation-source tracing, and raw server logs — landing in the same place. That’s not one weak study you can wave off. It’s convergent evidence.

Here’s the position I’ll take and stand behind: if you’re implementing llms.txt hoping it moves your Perplexity or ChatGPT citation rate, you’re spending implementation time on the wrong lever. The same datasets these researchers built show FAQ schema and answer-first content structure move citation frequency measurably. llms.txt, on current evidence, doesn’t. If a client asks me to prioritize it as a citation tactic, I’ll say so plainly, even though “just add this quick file” is an easier sell than “restructure your content.”

This kind of gap between hype and confirmed platform behavior is common across AI search advice — for the complete picture of why a brand might not be showing up in AI answers at all, see the full AI search visibility diagnostic.

So, Do You Need One? The Actual Decision Tree

Your audience uses coding agents or MCP tooling to work with your docs or API → Implement it. This is the one confirmed, real-world use case. Low cost, direct benefit.

Your goal is AI-answer-engine citation — ChatGPT, Perplexity, Google AI Overviews — and your site is content-heavy editorial or marketing material → Skip it, or treat it as a 20-minute hedge at most. No platform has confirmed using it for this purpose, and the data above shows no measurable lift. Spend the hour on FAQ schema and answer-first chunking instead.

Your site is primarily product or e-commerce pages with minimal editorial content → Optional, low priority. There isn’t much for the file to index either way.

You want near-zero-cost future-proofing regardless of confirmed use case today → Fine, add it. It won’t hurt anything. Just don’t report it internally as a GEO win — the honest label is “hedge,” not “lever.”

Common Mistakes When Implementing llms.txt

1. Treating it like a sitemap. Listing every URL on the site with no description defeats the purpose — a model gets the same guessing problem the file was meant to solve.
2. Vague link labels. “Click here” or “Read more” tells a model nothing about what it will find. Every link needs a description that stands alone.
3. Letting it go stale. A file pointing at pages that have moved, merged, or been retired actively misleads a model that does read it — worse than having no file at all.
4. Building llms-full.txt with no token budget in mind. Inlining every page into one giant document produces a file too large for practical use.
5. Selling it as a confirmed ranking factor. As of mid-2026, the data above doesn’t support that claim to a client or a stakeholder.

A Minimal llms.txt You Can Hand to a Developer

If you decided yes above, here’s the structure — copy this, fill in the brackets, and pass it to whoever manages your site’s root files.

# [Site or Organization Name]

> [One-sentence description of what the site covers and who it's for.
This is the line a model leans on most — be specific about category
and audience.]

[Optional: 1–2 short paragraphs of additional context.]

## [Section Name — e.g. "Documentation"]

- [Page Title](URL): [One-line description of what's on this page]
- [Page Title](URL): [One-line description]

## [Section Name — e.g. "Guides"]

- [Page Title](URL): [One-line description]

Required: the H1 site name and the blockquote summary line. Everything else is optional.

For your developer: this is a plain-text file that goes at yoursite.com/llms.txt — same root-level placement as robots.txt. It needs no server configuration beyond being uploaded and served as plain text rather than wrapped in your site template.

FAQ

What is llms.txt?

llms.txt is a Markdown file at the root of a website that lists its most important pages for a language model to read, with a short description of each. It was proposed by Jeremy Howard of Answer.AI in September 2024 to help AI tools navigate a site’s content without parsing full HTML pages. The format is deliberately minimal: an H1 site name, a one-line blockquote summary, and H2-grouped sections of linked pages. It’s a curated index, not a crawler-permission file — robots.txt still controls whether a bot can access your site at all, regardless of what llms.txt says.

Do I need an llms.txt file for AI search?

Not if your goal is showing up in ChatGPT, Perplexity, or Google AI Overviews. No major consumer AI answer engine has confirmed using it for citation, and three independent studies — covering roughly 300,000 domains, 94,614 cited URLs, and 137,210 domains’ server logs — all found no measurable citation lift from having one. You do need one if your audience works with your site through coding agents like Cursor or Claude Code, where llms.txt has real, confirmed use as a navigation aid for retrieving documentation.

Does Google use llms.txt?

No. Google has said this explicitly and repeatedly. Gary Illyes, an Analyst on the Google Search team, said so at Search Central Live in July 2025, comparing llms.txt to the old keywords meta tag — a signal site owners could set that Google simply doesn’t read. John Mueller reiterated the same position separately, and Google’s own AI optimization documentation, updated in mid-June 2026, states plainly that no new machine-readable AI file is required to appear in Google Search or its generative features. It plays no role in Google Search or AI Overviews.

Does Perplexity use llms.txt?

Unconfirmed. Perplexity’s own crawler documentation doesn’t state that it fetches and uses third-party llms.txt files for citation purposes — it references Perplexity’s own llms.txt for its own documentation, which is a different claim entirely. A number of GEO blogs assert that Perplexity supports the format; none of them point to an official Perplexity statement confirming that claim. Treat this as a genuinely open question rather than a settled yes, and don’t build a content strategy around an unverified assumption about any one platform.

What’s the difference between llms.txt and robots.txt?

robots.txt controls access — which bots are allowed to fetch which parts of your site — and every major documented AI crawler, including Googlebot, GPTBot, ClaudeBot, and PerplexityBot, respects it. llms.txt is a curated index for content a bot is already permitted to read; it’s optional, and no major consumer AI answer engine has confirmed reading it for citation. You need robots.txt configured correctly regardless of what you decide about llms.txt — it’s the file actually doing the access-control work that determines visibility.

How do I create an llms.txt file?

Write a Markdown file with an H1 for your site name, a one-sentence blockquote summary, and H2-grouped sections listing your key pages with short descriptions after each link. Save it as llms.txt and place it at your domain’s root — yoursite.com/llms.txt, the same location as robots.txt. It requires no special server configuration beyond being served as plain text rather than wrapped in your site template. Confirm it loads as raw text at that URL rather than through your CMS’s HTML wrapper, since that’s the most common implementation mistake. The template above is ready to copy and hand directly to a developer.

If you want the fuller picture on whether AI crawlers can even reach your site in the first place, how AI crawlers access your site walks through the robots.txt and CDN-layer checks that actually gate visibility — before llms.txt becomes a relevant question at all. For the broader diagnostic this spoke feeds into, the full AI search visibility diagnostic covers all eight signals.