llms.txt: A Markdown Convention for Handing Agents the Right Pages

The Problem llms.txt Addresses Is Token Waste, Not Discovery

A coding agent that needs one API signature should not have to download a documentation site's navigation, ad slots and JavaScript bundle to find it. The proposal states the case plainly: HTML wraps information in markup that is difficult and imprecise to convert back to clean text, and while context windows have grown, they remain too small for most websites in their entirety. Every wasted token costs time and money. The proposed fix is a small Markdown file at a known path that describes the site and links to detailed pages, with the detail fetched only when an agent decides it needs it. The audience is anyone whose site agents read on a schedule: documentation teams, API providers, and to a lesser degree businesses publishing policies or individuals publishing a CV. The repository itself is a proposal and a set of examples, not a library you install. Its primary language is Jupyter Notebook, which reflects that it is a document project built with nbdev rather than a runtime package.

Two Files, One Path Convention, and a Link Relation

The mechanism has two halves. The first is the llms.txt file itself: a Markdown document that lives at the site root or at any subpath, and covers the URLs beneath that path. Where more than one file applies, the proposal says agents should use the most specific one, so /docs/llms.txt governs /docs/ and everything under it. The second half is the content the file points at. The proposal asks that pages agents might need also be available as clean Markdown at the same URL, either with .md appended (page.html.md) or with the extension replaced (page.md). URLs without file names get index.html.md or index.md. To make those files findable without guessing, the proposal recommends standard link relations: rel="alternate" type="text/markdown" for the Markdown version of a page, and rel="describedby" for the llms.txt that covers it. These can be HTML link elements or an HTTP Link response header. The header form is the more interesting one because it works for non-HTML resources and can be set in web server or CDN configuration without touching any page. The proposal gives this example: Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby". The data flow is therefore: agent reads llms.txt, picks a link, fetches Markdown, stops. The file stays small enough to fit in context by design.

The File Format Is Deliberately Rigid

The specification fixes the section order. A file contains an optional byte-order mark, an H1 naming the project or site (the only required section), a blockquote summarising the project with the key information needed to interpret the rest, then zero or more Markdown sections of any type except headings, then zero or more H2-delimited file lists. Each file list is a Markdown list whose entries are a required link plus an optional colon and note. The mock example in the proposal is short: an H1, a blockquote, a details paragraph, a section named for its content with one linked entry and a note, and an Optional section. That Optional section is a convention rather than a rule: it holds secondary links an agent can skip when context is tight. The proposal is explicit that the choice of Markdown over XML is intentional, because the readers are language models, but it also claims the format is precise enough for parsers and regex. That dual claim is the part worth scrutinising. A parser can rely on the H1, the blockquote position and the H2-delimited lists, but the free-form sections in between are only as structured as the author makes them, so a tool that expects machine-readable metadata will not find it here.

What You Actually Create, and Where It Goes

There is no install step in the material. You write a Markdown file and place it at the path you want it to govern: /llms.txt at the root, or something like /docs/llms.txt for a documentation subtree. The FastHTML project is cited as a working example, with its file at /docs/llms.txt and a corresponding HTML page at /docs/tutorials/by_example.html alongside the same URL with .md appended. If your docs are built with nbdev, the proposal notes that all nbdev projects now generate .md versions of pages by default, and that Answer.AI and fast.ai projects using nbdev have had their docs regenerated with that feature. For a fastcore example it points to the Markdown version of the docments module page. The server-side piece is the Link header, which the proposal says can be added in web server or CDN configuration. That is the whole deployment surface: one Markdown file, a Markdown rendering path for existing pages, and one header. The cost is not the file. It is the second rendering of every page you choose to expose, kept in step with the first.

The Failure Mode Is Drift, and the Spec Does Not Prevent It

Nothing in the proposal keeps page.html and page.html.md consistent. If the HTML changes and the Markdown does not, an agent that follows llms.txt gets a confident, well-formatted, wrong answer, and it has no signal that anything is stale. The format has no version field, no timestamp requirement and no hash to compare against the rendered page. Compare that with a sitemap, where lastmod at least gives a crawler a hint. The proposal also depends on agent behaviour it does not control. It says agents are expected to view or search llms.txt, but expectation is not enforcement, and a client that ignores the file loses nothing. The link relation approach softens this slightly, since rel="describedby" is a hint a general-purpose crawler might follow, but the proposal does not describe a discovery path for agents that never look for the header. There is also a scope question the material leaves open: the file covers URLs under its path, and the proposal says agents should prefer the most specific file, but it does not say what happens when two files at the same depth make conflicting claims about the same page. If your content changes hourly, or your pages are generated per user, llms.txt is the wrong tool, because the Markdown mirror becomes a second system to operate.

How It Differs From an agents.txt or a Sitemap

The nearest alternative in the agent-tooling space is an agents.txt style file, which typically states permissions and access rules for automated clients. The difference is the payload. An agents.txt tells a crawler what it may do; llms.txt tells a reader what to read, in what order, and where the clean version lives. They answer different questions and can coexist. The older comparison is sitemap.xml, which enumerates URLs for indexing without saying anything about which ones matter or what format suits a language model. A sitemap for a documentation site might list four hundred pages; an llms.txt for the same site is meant to fit in a context window, which forces the author to choose. That editorial act is the real value and the real labour. The proposal also sits alongside the Markdown-mirror convention itself, which is separable: you can serve page.md files and advertise them with rel="alternate" without ever writing an llms.txt. If your only goal is to stop agents parsing your HTML, the link relation alone gets you most of the way.

Adoption Signals and Maintenance Cost

The v2 document was written after two years of adoption and cites several signals: documentation platforms such as Mintlify generate an llms.txt automatically, Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, and OpenAI, Anthropic and Gemini publish files for their own developer documentation. Those are adoption facts, not quality claims, and they should be read that way. For a maintainer, the recurring cost is the Markdown mirror. If your generator already emits .md, as nbdev now does, the marginal cost is writing and occasionally revising one index file. If your site is hand-built HTML, you are adding a rendering path and a synchronisation habit. The repository is Apache-2.0 licensed, which permits commercial use and modification of the specification text and any accompanying code, subject to the usual attribution and notice conditions in that licence. The licence covers the repository's contents; it says nothing about your site's content or about how a model vendor handles what it fetches, and this is not legal advice.

Editorial conclusion

Adopt llms.txt if you maintain software documentation or any site where agents repeatedly need precise answers, and you can serve Markdown at stable URLs. Do not adopt it if you cannot keep the linked Markdown in sync with the HTML, because a stale llms.txt is worse than none. Before publishing, verify three things: that every URL in your file list returns Markdown rather than an HTML wrapper, that the file sits at the path covering the pages it describes (for example /docs/llms.txt for /docs/), and that your server emits the Link header or rel attributes the proposal recommends. The specification itself is short enough to read in one sitting at llmstxt.org.

Official sources

1. AnswerDotAI/llms-txt on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases