llms.txt: What It Is, How to Create One, and Whether It Actually Works (2026)

llms.txt is a proposed markdown file, hosted at yoursite.com/llms.txt, that gives AI models a curated map of a site’s most important pages. Jeremy Howard of Answer.AI proposed it in September 2024. As of mid-2026, adoption sits under 9% of top sites, no major AI provider has confirmed using it to inform answers, and the cost to add one is close to zero. Write one in the next 10 minutes, then stop thinking about it.

What is llms.txt?

Jeremy Howard, co-founder of Answer.AI and formerly of fast.ai, published the llms.txt proposal on the Answer.AI blog on September 3, 2024. His argument: large language models pull information from websites to answer questions, but a model’s context window is too small to hold most sites in full, and converting a JavaScript-heavy page full of navigation and ads into clean text is neither reliable nor cheap to do at inference time. His fix was a plain-text index a site owner writes by hand, pointing a model to the pages that actually matter.

The spec, maintained at llmstxt.org, is short and specific. A valid /llms.txt file is markdown, and it follows this structure:

• An optional byte-order mark at the very start of the file.
• An H1 with the site or project name — this is the only required element.
• A blockquote directly under the H1, giving a short summary a model needs to understand the rest of the file.
• Optional prose sections in plain markdown with extra context, not link lists.
• One or more H2 sections, each containing a markdown list of links in the format [name](url): description.
• An optional H2 titled “Optional,” reserved for secondary links a model can skip if it’s short on context.

That’s the entire file. No JSON, no schema, no required fields beyond the H1. In practice, most published examples also ship a second file, llms-full.txt, at the same root path. Where llms.txt is a curated table of contents, llms-full.txt is the actual content: full page text, concatenated, for cases where a link list isn’t enough and a model needs the material inline. Anthropic’s own developer docs publish both — llms.txt at docs.claude.com and platform.claude.com, and the matching llms-full.txt versions with the complete documentation text.

It’s worth being precise about what llms.txt is not. It is not access control. It is not an inventory of every URL on a site. It is an editorial judgment call by whoever runs the site, expressed as a short list: these pages, in this order, are the ones worth reading first.

Does llms.txt actually work?

The honest answer, based on the data available in mid-2026: unproven, cheap, and worth doing anyway as a hedge, not because there is evidence it moves citations.

Start with adoption. Rankability’s June 2026 tracking of the top 1,000 sites globally found llms.txt on 8.7% of them — and that figure only reaches 15.8% among the roughly 549 sites in that sample that were actually reachable and testable. A separate, larger study from Ahrefs, drawn from server logs across 137,210 domains in May 2026, found a higher publication rate — 28% of domains, around 38,360 sites, had a valid file — but a much more telling number sat underneath it: 97% of those files received zero requests during the entire measurement period. Of the small share that did get requested, most of that traffic was from non-AI bots; AI crawlers specifically accounted for under 20% of requests to the file, and AI retrieval bots — the category that would actually matter for showing up in an answer — made up about 1.1% of all requests. Ahrefs’ own conclusion: if the goal is showing up in ChatGPT, Perplexity, or Google’s AI Overviews, an llms.txt file is “largely decoration.” Some of the adoption growth is also mechanical rather than deliberate — Shopify pushed llms.txt by default to every store on its platform in spring 2026, which alone accounts for a large share of the sites that now technically have one.

Then there’s what the AI providers themselves have said, which is close to nothing. Google’s John Mueller addressed it directly in a June 2026 discussion, and his position was blunt: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” He called the file’s current status “purely speculative,” compared it to the old keywords meta tag — a claim a site owner makes about their own content, unverified by anyone reading it — and said he prefers Google’s WebMCP work instead, because it has “clear goals & processes.” His practical recommendation, and the most useful line in the whole debate: “When an AI platform that brings you clients complains that it needs the file for your site, then I’d recommend taking the time to create one.” In other words, wait for a specific request, not a trend piece, before treating this as urgent.

OpenAI has not published anything stating that ChatGPT, GPTBot, or OAI-SearchBot parse llms.txt or give it special weight. Some site owners do see those crawlers request the file in their logs occasionally, but a crawler fetching a URL is not confirmation that anything downstream reads or prioritizes it. Anthropic publishes its own llms.txt and llms-full.txt files for its developer docs, which shows the company finds the format useful for its own documentation — it is not the same as Anthropic confirming that Claude fetches and prioritizes other sites’ llms.txt files when answering a question. Perplexity’s documentation site also ships both files, and some industry reporting says PerplexityBot reads llms.txt during indexing passes — but that claim traces back to secondhand blog posts instead of a clear statement from Perplexity itself, so treat it as plausible, not confirmed.

Put together: no provider has said it reads other sites’ llms.txt files as an input to what it cites, server logs show AI retrieval bots barely request the file where it exists, and the SEO practitioner closest to Google’s own thinking calls it speculative. None of that means the file is harmful or pointless — it costs a few minutes to write, it can’t hurt crawlability, and if a provider does start using it, an existing file means one less thing to scramble for later. Treat it as a low-cost insurance policy, not a growth lever.

llms.txt vs robots.txt vs sitemap.xml

These three files get lumped together because they all live at a site’s root and all relate to how machines read a site, but they solve different problems and none of them substitute for another.

File	Purpose	Format	Audience	Enforced by anything?	Origin
robots.txt	Access control — tells crawlers which paths they may or may not fetch	Plain-text directives (User-agent, Allow, Disallow)	Search and AI crawlers	Yes, by convention — most major crawlers respect it, though compliance is voluntary	1994, the Robots Exclusion Protocol
sitemap.xml	Inventory — lists every URL a site wants indexed, with metadata like last-modified date	XML	Search engine indexers	No, it’s a hint for crawl scheduling and discovery, not a guarantee of indexing	2005, introduced by Google
llms.txt	Curation — a short, human-edited list of the pages worth reading first, for a model with limited context	Markdown, with a required H1 and blockquote	LLMs and AI assistants, if and when they check for it	No — no provider has confirmed reading it, and there’s no enforcement mechanism at all	September 2024, proposed by Jeremy Howard / Answer.AI

The practical distinction: robots.txt says what a crawler is allowed to touch, sitemap.xml says what exists, and llms.txt says what matters most, in the site owner’s own opinion. A site can have all three, and having one says nothing about whether the others are configured correctly — a common mistake is polishing an llms.txt file while robots.txt still blocks GPTBot from the pages that file links to.

How to create an llms.txt file

Building one takes longer to plan than to execute. The file itself is short; the work is deciding what belongs in it.

1. Inventory the pages a model would need to accurately describe the product. For a SaaS site, that typically means the homepage, the core product or feature pages, the pricing page, developer or help docs, a handful of comparison or alternative pages, and any review-platform profile worth pointing to — a G2, Capterra, TrustRadius, or Spotsaas listing page, since those often carry more third-party credibility than a page the company wrote about itself.
2. Write the H1 and blockquote first. The H1 is just the product or company name. The blockquote is one or two sentences that would let a model unfamiliar with the company understand what it does and who it’s for — treat it like the summary a new hire would need before reading anything else.
3. Group links under H2 sections by purpose, not by internal site structure. “Product,” “Pricing,” “Documentation,” and “Comparisons” reads better to a model than a section literally named after a URL folder.
4. Add descriptions after each link. A bare URL forces a model to fetch the page to know what it is; a link with a one-line description after a colon saves that round trip.
5. Put secondary material under an “Optional” H2. Changelogs, old blog posts, and legal pages belong here — visible if a model has room in its context, skippable if it doesn’t.
6. Save it as plain text at the site root, served at exactly /llms.txt, with no login wall and no redirect chain in front of it.
7. Consider a companion llms-full.txt if the site is documentation-heavy — it’s the same idea, but with full page text pasted in instead of just links, useful when a model needs the actual content instead of a pointer to it.

Here’s a complete, copy-paste-ready example for a fictional SaaS product, Acme Analytics:

# Acme Analytics

> Acme Analytics is a web and product analytics platform for B2B SaaS teams. It tracks user behavior across web and mobile apps, builds funnels and retention cohorts, and exports data to a company's own warehouse without sending it through a third party.

## Product

- [Overview](https://acmeanalytics.com/product): what Acme Analytics does and who it's for
- [Event tracking](https://acmeanalytics.com/product/events): how events, properties, and identities are captured
- [Dashboards](https://acmeanalytics.com/product/dashboards): building funnels, retention curves, and cohort views
- [Integrations](https://acmeanalytics.com/integrations): supported data warehouses and app frameworks

## Pricing

- [Pricing plans](https://acmeanalytics.com/pricing): current tiers, usage limits, and what's included in each
- [Enterprise pricing](https://acmeanalytics.com/enterprise): custom contracts, SSO, and data residency options

## Documentation

- [Getting started](https://docs.acmeanalytics.com/quickstart): install the SDK and send a first event
- [API reference](https://docs.acmeanalytics.com/api): full REST and SDK method documentation
- [Data model](https://docs.acmeanalytics.com/data-model): how events, users, and properties are structured

## Comparisons

- [Acme Analytics vs Mixpanel](https://acmeanalytics.com/vs/mixpanel): pricing, data ownership, and feature differences
- [Acme Analytics vs Amplitude](https://acmeanalytics.com/vs/amplitude): pricing, data ownership, and feature differences

## Third-party reviews

- [Acme Analytics on G2](https://www.g2.com/products/acme-analytics/reviews): verified customer reviews and ratings
- [Acme Analytics on Spotsaas](https://www.spotsaas.com/software/acme-analytics): SpotScore and verified reviews

## Optional

- [Changelog](https://acmeanalytics.com/changelog): recent product updates
- [Blog](https://acmeanalytics.com/blog): product announcements and analytics guides
- [Status page](https://status.acmeanalytics.com): uptime and incident history

Once it’s live, confirm it’s actually reachable before moving on. Two checks cover it:

• Fetch the URL directly — curl -I https://yoursite.com/llms.txt should return a 200 status with no redirect and no login wall in the way.
• Check server logs a week or two later for requests to that path from known AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended are the names to filter for. Given the Ahrefs data above, don’t be surprised if there’s nothing there for weeks; that’s the current normal, not a sign of a broken file.

llms.txt generators

A handful of tools will build a first draft automatically, which is a reasonable starting point and a poor finished product.

Firecrawl runs a free generator at llmstxt.firecrawl.dev, and the same functionality is open-sourced on GitHub as firecrawl/llmstxt-generator: point it at a domain, it crawls the site, and it produces both llms.txt and llms-full.txt using a small model to summarize each page. On WordPress, both Yoast SEO and Rank Math added native llms.txt generation as a built-in feature during 2025 and 2026, and AIOSEO offers the same, automatically excluding pages marked noindex or nofollow. There’s also a dedicated Website LLMs.txt plugin in the WordPress plugin directory built specifically for sites that don’t want to add a full SEO suite just for this one file. Shopify goes further than any of these — as of spring 2026, it generates and serves an llms.txt automatically for every store on the platform, without a merchant opting in.

The honest caveat: every one of these tools is good at the mechanical part — crawling a site and listing its URLs — and bad at the part that actually gives llms.txt its value, which is judgment. The blockquote summary, the choice of which ten pages matter out of the two hundred that exist, and the grouping into sections that reflect how someone would actually explain the product — a generator can approximate all of that, but it defaults to a flat list of every page it found, sorted by whatever the crawl order happened to be. For a marketing or docs site with a clear structure, running a generator and then editing the output by hand — cutting it down, rewriting the summary, and reordering by importance — takes less time than most people expect and produces a file that’s actually worth reading. Hand-writing a short file beats generating a long one, every time this has been tested by people who’ve done both.

What matters more than llms.txt

Given everything above, the file is worth ten minutes and nothing more. Three things matter more, and none of them are speculative.

First, crawler access. An llms.txt file pointing to a set of pages is useless if robots.txt blocks the bots that would read those pages in the first place. Check that GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot aren’t disallowed on the pages that matter for AI visibility — product pages, pricing, comparison pages, and documentation. This is a five-minute audit and, unlike llms.txt, it directly controls whether an AI crawler can see a site at all.

Second, answer-first content structure. Models pull answers from pages that state a claim clearly near the top, back it with a specific number or fact, and don’t bury the point under three paragraphs of throat-clearing. This is a content and formatting problem, not a file-at-the-root problem, and it applies to every page on a site regardless of whether llms.txt exists.

Third, and the one with actual evidence behind it: presence on the review platforms AI models already cite. SE Ranking’s research on review platforms in AI Overviews found that despite these sites losing the large majority of their organic search traffic over the past two years, they still account for a disproportionate share of AI citations — G2, Capterra, TrustRadius, and Gartner Peer Insights together make up the bulk of review-platform links inside AI answers, with G2 alone responsible for between roughly a third and three-quarters of review-site citations across ChatGPT, Perplexity, and Google’s AI Overviews depending on the query set. That’s the mechanism working today: an AI model citing a review platform because the platform has structured, third-party-verified data about a product, not because a company told it what to read via a text file at its own root.

Spotsaas is one of those review platforms — 2M+ buyers use it every year to evaluate software across 24,578 products and 419 categories, backed by 12,400+ verified reviews and a SpotScore rating out of 10 for every listed product. Getting listed takes a few minutes at spotsaas.com/get-listed, and unlike an llms.txt file, a review platform profile is a channel AI models are already citing, today, at measurable volume. For a full breakdown of how AI visibility actually works and where the citation volume comes from, see Spotsaas’s AI visibility guide.

Frequently asked questions

What is llms.txt?

It’s a markdown file at yoursite.com/llms.txt that gives AI models a short, curated list of a site’s most important pages, plus a one-line summary of each. Jeremy Howard of Answer.AI proposed the format in September 2024. It’s a voluntary convention, not a ratified web standard, and no browser or server enforces its structure.

Do ChatGPT and Google read llms.txt?

There’s no confirmation that they do. Google’s John Mueller has said directly that no AI service has confirmed using it, and that server logs show AI systems largely don’t even check for it. OpenAI hasn’t published anything stating GPTBot or ChatGPT parse the file specially. A May 2026 Ahrefs study of 137,000+ domains found AI retrieval bots made up about 1.1% of all requests to llms.txt files that existed — evidence the file is rarely fetched by the crawlers that would need to use it.

Where do I put the llms.txt file?

At the root of the domain, served at exactly /llms.txt — the same level as robots.txt and sitemap.xml. It needs to return a 200 status with no login wall, no redirect, and no authentication in front of it, since a crawler won’t sign in to fetch it.

What’s the difference between llms.txt and llms-full.txt?

llms.txt is a curated list of links with short descriptions — a table of contents. llms-full.txt is the actual content behind those links, concatenated into one file, for cases where a model needs the full text instead of a pointer to a separate page. Anthropic’s own developer docs publish both versions at docs.claude.com, and most generator tools produce both files at once.

Is llms.txt worth it for a SaaS site?

As a ten-minute addition, yes — it costs almost nothing, can’t hurt anything, and covers the case where an AI platform eventually does start reading it. As a growth strategy or ranking lever, no — there’s no evidence it currently affects what gets cited, and the data available says AI crawlers barely request it. Spend the bigger effort on making sure crawlers can actually reach your pages and on getting listed on the review platforms AI models are already citing.

Keep reading

• AI visibility for SaaS — the full guide to how AI models decide what to cite, and how review platforms fit in.
• The software marketplace in 2026 — where buyers actually research and compare B2B software today.
• SaaS statistics 2026 — the current numbers on SaaS spend, adoption, and buying behavior.

Sources

• Answer.AI, “/llms.txt — a proposal to provide information to help LLMs use websites,” September 3, 2024
• llmstxt.org, the /llms.txt specification
• Rankability, “LLMS.txt Adoption: 8.7% of the Top 1,000,” June 2026
• Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read,” May 2026
• Search Engine Journal, “Google Says LLMs.txt Is Purely Speculative for Now,” 2026 reporting on John Mueller’s comments
• SE Ranking, “Despite 90% Traffic Loss, Review Platforms Top AI Overview Citations,” 2026
• Firecrawl documentation, llms.txt generator
• WordPress.org plugin directory, Yoast SEO, Rank Math, and Website LLMs.txt listings

Last updated: July 17, 2026