TL;DR
Every 2026 implementation guide defines llms-full.txt as a complete text export of your site. That definition is the most expensive sentence in the format.
This is the first asset a publisher ships whose value falls as it grows. It is not stored in an index; it is read inside a finite attention budget that degrades long before it fills.
The instrument here is the answer-density shelf: a token budget, a per-page density score, and a mean-crossing admission rule that tells you exactly when to stop adding pages.
The reference implementations run near 500,000 tokens. On the published long-context evidence, that is past the point where every measured model has already started to fail.
Version 2 of the spec, published 10 August 2026, quietly converts llms.txt from a destination file into a routing layer attached to every page you own.
Open any of the 2026 implementation guides for llms-full.txt and you meet the same definition, phrased four ways: a complete text export of your entire website, every page concatenated into one large Markdown document. Ship it, the guides say, and an AI system can take your whole site in a single request.
The definition is not a lie — large publishers have done exactly that. It is worse than a lie: a specification that fails in the direction people naturally push. Give a content team a rule that says include everything, and they will include everything, and the file will get worse every week the site grows.
Look at the exemplars everyone is copying. A February 2026 survey of the format by Kubicek put Anthropic’s llms-full.txt at close to 500,000 tokens, Vercel’s at roughly the same, and Cloudflare’s at as much as 3.7 million in aggregate. Those are the reference implementations, and on the published evidence about how language models handle long inputs they are well past the size at which the systems meant to read them stop reading well.
Every other asset a publisher ships gets more valuable as it gets bigger. A sitemap with 12,000 URLs beats one with 400. A profile with 900 referring domains beats one with 90 — the whole logic of how link building compounds rests on accumulation. llms-full.txt inverts this, because it is not stored anywhere. It is read: inside a finite attention budget belonging to somebody else, at a moment you do not control, in competition with everything else loaded alongside it.
Accept that and the exercise changes character. You are not exporting a site; you are packing a context window for a stranger. A packed window has a maximum useful size, and past it adding genuinely good content makes the file worse. What follows is the instrument for finding that size, the mechanics for building to it, and an honest account of who is on the other end of the fetch — in 2027 a narrower and more valuable audience than the format’s marketing suggests.
The two files, and what each one is really for
llms.txt is a proposal, not a standard. Jeremy Howard of Answer.AI published it in September 2024: a Markdown file at your domain root with an H1 site name, a blockquote summary, and H2 sections of annotated links. No standards body sits behind it and no engine is obliged to read it. In June 2026 Google added a note to its AI optimisation guidance confirming that Search does not use these files and that maintaining one will neither harm nor help visibility.
llms-full.txt is the companion. Where llms.txt lists and describes, llms-full.txt inlines — the actual text of the linked documents, in one response. The distinction is not size; it is what the reading system must do next.
What is llms-full.txt?
llms-full.txt is a single Markdown file that contains the full text of your chosen pages rather than links to them, so a machine that fetches it needs no further requests. It is the ingestion companion to llms.txt, which is a curated index that tells a machine where to go next.
That difference produces two opposite design constraints, and almost every failed implementation comes from applying one file’s logic to the other. llms.txt is a routing document, read to make a decision, so it should be short — the 2026 practitioner consensus puts it under 10,000 tokens, with some arguing for as little as 2,000. llms-full.txt is a payload, read to produce an answer, so it has to survive being read alongside a user’s question, a system prompt, a code file, and whatever else is already in the window. Routing documents can afford to be generous. Payloads cannot.
The only asset you publish that gets worse as it grows
The mechanism has a name. In July 2025 Chroma Research published Context Rot: How Increasing Input Tokens Impacts LLM Performance, by Kelly Hong, Anton Troynikov and Jeff Huber. Context rot — measurable quality loss as input grows — was tested across 18 frontier models, and every single one degraded as inputs lengthened. Not most. All of them.
Two findings matter for anyone sizing a file. First, rot is not overflow: a 200,000-token window can show serious accuracy loss at 50,000 tokens of input. Zylos Research’s January 2026 analysis puts the general gap at 30 to 40 per cent below the advertised limit, so a 200K model becomes unreliable near 130K. For million-token flagships, observers put the visible onset around 300,000 to 400,000 tokens.
Second, position matters independently of length. The lost-in-the-middle effect, documented by Liu and colleagues and replicated across six model families, gives accuracy a U-shape: highest at the beginning and end, 20 to 30 points lower in the middle. A June 2026 TMLS review separates the two failure modes and reports a controlled study in which reasoning accuracy fell from 0.92 to 0.68 as inputs grew from a few hundred to three thousand tokens, with ten of twelve models below half their short-context score by 32,000 tokens.
Put those two together and the arithmetic on a 500,000-token export becomes brutal. Everything you concatenated between roughly page four and roughly the last page sits in the region where models attend worst, at a total length where they reason worst. You did not publish a library. You buried a paragraph.
Why this is not a problem you can wait out
The obvious rebuttal is that windows keep growing, so the constraint is temporary. It is not, for a structural reason: standard transformer attention spreads a fixed softmax budget across every token pair, so as a sequence grows, the weight available to any single token shrinks. Position encodings help; they do not repeal the arithmetic. The industry has moved the other way, and the 2026 default is hybrid — retrieve a bounded slice, reason over that. Hamel Husain’s summary of the practical implication is the one to keep: the posture is not to fit it all in, it is to give the model the smallest relevant context that answers the question. Your file is competing to be that smallest context. Bulk is a disqualification.
Key takeaway
Size is not neutral here. Length degrades reasoning, middle position degrades attention, and a concatenated export maximises both. Treat every token added as a token subtracted from everything already in the file.
The answer-density shelf: an admission rule for llms-full.txt
The instrument replaces include your best content — not a rule, because everyone believes their content is their best — with a test that produces a different answer per page and a hard stopping point for the file.
The answer-density shelf
1. The shelf (S). Your token budget: the smallest effective context among the tools that fetch you, times the share a documentation fetch can reasonably claim. For a 200K-class coding agent already holding a codebase, 10 to 15 per cent is defensible — a shelf of 20,000 to 30,000 tokens. Derive yours; do not inherit mine.
2. Answer density (D). Per candidate page: self-contained answers delivered, divided by length in thousands of tokens. Self-contained means a reader holding no other page of yours can act on it. Five answers in 1,200 tokens scores 4.2; a 3,000-token essay resolving one scores 0.33.
3. The admission rule. Admit a page only if its D exceeds the file’s current mean D. Sort candidates by D, add from the top, and stop at the first page that would pull the mean down or breach the shelf, whichever comes first.
4. The consequence. The mean rises with every admission, so the bar rises as you build and the file closes itself — on most sites between 15 and 40 pages, nowhere near a site export.
This is not a scoring rubric with weights to argue about. It is a single comparison, and it is self-tightening: add your densest page and every subsequent page faces a harder test. That is the opposite of how content inventories normally behave, where the hundredth item faces no test at all.
Counting self-contained answers without fooling yourself
The one soft number is the answer count, so define it strictly. A self-contained answer states a question’s resolution and everything needed to use it: the figure, its unit, its date, its scope, and the condition under which it stops being true. Our pricing is transparent is not an answer. The Starter plan is £29 per seat per month, billed annually, with a 5,000-call daily API ceiling as of August 2026 is one answer, and it will still be one answer after a machine has stripped every heading around it.
Count with your own headings hidden. If a span only makes sense because of the H2 above it, it is not self-contained — it is a fragment adjacent to context, and adjacency is what concatenation destroys. This is the discipline that governs writing passages that win featured snippets, and for the same underlying reason: a system that extracts spans rewards spans that survive extraction.
| Page type | Tokens | Answers | Density (D) |
| API reference page | 1,100 | 6 | 5.45 |
| Pricing and limits table | 700 | 5 | 7.14 |
| Definitions and glossary | 1,400 | 9 | 6.43 |
| Original data study | 2,600 | 7 | 2.69 |
| Step-by-step tutorial | 2,900 | 3 | 1.03 |
| Thought-leadership essay | 3,200 | 1 | 0.31 |
| Customer case study | 1,900 | 1 | 0.53 |
| Company about page | 800 | 2 | 2.50 |
Illustrative profile for a mid-sized B2B publisher; answers counted under the self-contained test above.
The discomfort arrives immediately. The pages that win are the ones nobody puts in a portfolio: tables, limits, definitions, thresholds. The pages that lose are the ones the business is proudest of. Your best essay, the one that earned forty referring domains, may be the first thing the rule cuts — and it should be, because the qualities that make an essay linkable (argument, sequence, voice, deferred payoff) are the qualities that make it a poor payload when a machine has one shot at extracting a usable span.
This relocates the essay rather than demoting it. An essay is an earned-link asset, written to be cited by humans with publishing power; the reference page is a retrieval asset, written to be quoted by a machine with two seconds of attention. Confusing the two is how sites end up with a 400,000-token file of prose no system can use.
A worked example: Ravensworth Data, 612 URLs, eight weeks
Ravensworth Data is a Leeds-based payments-compliance data provider publishing 612 URLs: 148 API documentation pages, 61 regulatory reference pages, 340 blog posts and 63 marketing pages. Its customers are engineering teams integrating a sanctions-screening API, increasingly with a coding assistant in the loop. Here is what the admission rule did to its file.
Week 1. The existing llms-full.txt, auto-generated by a plugin the previous December, ran to 511,000 tokens — the whole site, careers page included. Server logs showed 22 fetches in six months, 19 of them from crawl-audit tools.
Week 2. The team set the shelf. Integration users run 200K-class assistants that arrive already holding thousands of lines of the customer’s own code, so S was set at 12 per cent: a 24,000-token shelf. Nobody argued for more once the figure sat on a whiteboard beside the phrase that is what a stranger will lend us.
Weeks 3 to 5. Density scoring across all 612 URLs. Regulatory reference pages scored highest — one page listing 14 jurisdictions with their screening thresholds and effective dates returned a D of 9.1. API pages clustered between 4 and 6. Of 340 blog posts, four cleared 2.0; 71 scored a flat zero because every answer in them pointed elsewhere.
Week 6. Admission. Sorted by D and added from the top, the file closed after 31 pages and 21,400 tokens — 4.2 per cent of the original, at a mean D of 5.6. The 32nd candidate, a 2,400-token integration walkthrough scoring 1.9, was the page that would have pulled the mean down, and the page the content lead had been most confident about.
Weeks 7 and 8. Rewriting for extraction rather than cutting further. Eleven of the 31 admitted pages had answers that dissolved when headings were stripped, so each was rewritten to restate scope in-span. That added 2,900 tokens and lifted counted answers from 119 to 147, taking mean D to 6.1 without admitting a new page.
The honest outcome, twelve weeks after publication: fetches rose from 22 in six months to 141 in three, 96 of them user-triggered agent traffic rather than audit tools. No measurable change in AI Overviews citations. One thing did change and could be attributed: the assistant told me the wrong rate limit support tickets fell from a running average of nine a month to two. That is a real return, and it is a support return, not a search return. Anyone selling you the search return should be asked for evidence, as you would interrogate any claim in a backlink data set before repeating it.
Key takeaway
The admission rule closed a 612-URL site at 31 pages and 21,400 tokens. The second-order win came from rewriting admitted pages so answers survived heading removal — 24 per cent more counted answers for 2,900 tokens, no new pages.
Who is actually on the other end of the fetch
Curation only pays if somebody reads the result, and the honest 2026 answer is that the reader is not who the format’s promoters imply. The engines generating consumer answers are not the constituency. The tools that write code are.
The clearest evidence is what the model vendors do with their own domains. Anthropic, OpenAI and Google’s Gemini team all publish llms.txt files — for their developer documentation. Documentation platforms ship it by default: Mintlify generates one for every site it hosts, as do Fern, GitBook, Vercel Docs and Supabase Docs. Google’s Chrome team added a retrieval check for the file to Lighthouse’s Agentic Browsing category while Google Search’s guidance says the file is ignored. Two teams, one company, opposite directions, because they serve different readers.
The coding-side mechanism is concrete. LangChain’s mcpdoc is an open-source MCP server — MCP being the Model Context Protocol, a standard interface between assistants and tools — that exposes a user-defined list of llms.txt files to hosts such as Cursor, Windsurf and Claude Code, and hands the assistant a fetch_docs tool for reading URLs listed inside them. The documented pattern: list sources, fetch the index, reflect on the links, fetch only the relevant pages, write the code. Every step is a decision about what to spend context on — which is why a bloated payload loses to a curated one at selection.
| Reader | How it reads | What it wants | Size that helps |
| IDE coding agent | Fetch and read whole | Exact syntax, limits, errors | Small; competes with code |
| MCP docs server | Index first, then pages | A routable index | Index tiny, pages scoped |
| RAG ingestion job | Chunk and embed | Distinct, non-duplicated spans | Large is tolerable |
| Consumer answer engine | Crawls HTML instead | Nothing from this file | Irrelevant |
| Crawl-audit tool | Fetches to score you | Presence, not content | Irrelevant |
Three of those rows are the reason to build the file. The fourth is why most people build it, and it is where the file does nothing. That mismatch explains almost every disappointed 2026 write-up — and it is separate again from how a model decides which sources to trust when training, which no file at your domain root settles.
Does llms-full.txt improve AI search rankings?
There is no published evidence that either file increases citations in consumer answer engines, and Google states plainly that maintaining one neither harms nor helps visibility in Search. Build it for coding agents, documentation tooling and ingestion pipelines, and treat search visibility as unproven upside rather than the business case.
Narrower than the format is usually sold on, and still worth taking. A tool that repeatedly hands developers a wrong endpoint or a stale rate limit is manufacturing a factual error about your product at scale — the same failure class as any other hallucinated brand claim you have to correct — with the difference that here you own the corrective surface outright, so the fix is a publishing decision rather than a monitoring-and-escalation exercise.
Mechanics that survive contact with an agent
Once the admission list is set, six build decisions determine whether the file works. None is stylistic; each maps to a documented failure mode.
Order by density, not by navigation
The U-shaped attention curve is a constraint you can exploit. Put your highest-density pages first and your second-highest last, letting mid-range material occupy the middle where attention is weakest. Sites that mirror navigation order — company, product, docs, blog — spend the two strongest positions on an about page and a footer.
Make every span survive decapitation
Assume every heading above a passage is removed before the passage is used. Restate scope inside the span: not the limit is 5,000 per day but the Starter plan’s API limit is 5,000 calls per day. It costs tokens and buys answers, and at Ravensworth it was the highest-return work in the project.
Strip repeated furniture ruthlessly
Concatenation multiplies boilerplate. Thirty-one pages carrying the same blurb, call to action and legal footer produce thirty-one near-identical passages in one document. In a fetch-and-read path that is wasted budget; in an ingestion path it is worse, because near-duplicate chunks compete, crowd out better evidence and disagree when one copy goes stale. The TREC RAG work on MS MARCO deduplicated its corpus precisely because near-duplicates degrade retrieval accuracy and reduce diversity. Ship each recurring element once, if at all.
Serve Markdown, but for the right reason
Markdown is the correct serialisation and the token argument is real: Cloudflare’s February 2026 measurement recorded one of its own blog posts at roughly 16,180 tokens as HTML against 3,150 as Markdown, near an 80 per cent reduction. But be precise about who benefits. Profound ran a randomised controlled test across 381 pages on six sites in early 2026; the Markdown arm led by about 16 per cent on mean bot visits, driven almost entirely by already-popular pages, with the median page gaining roughly one visit — consistent with noise. Markdown saves the reader’s budget. It does not, on current evidence, buy citations.
Date every claim inside the text
A concatenated export is a copy of your site that ages independently of it, and a machine reading it cannot tell a current figure from one you changed in March. In-text dating (“as of August 2026”) is the cheapest defence and doubles as the marker that lets you audit the file later. Skip it and you ship an artefact contradicting your own pages — the quiet start of the drift that eventually needs deliberate recovery work to fix.
Publish the smaller companions too
The spec permits subpath-specific files, so a large site can serve a scoped file per section rather than one monolith — /docs/llms.txt for documentation, /research/llms.txt for original studies. Scoping keeps density high without abandoning coverage, and it fits how an assistant actually works: it identifies which part of your product owns a question before it fetches anything. Where structured feeds exist, the same discipline applies to machine-readable API feeds, which solve the freshness problem a flat file cannot.
Key takeaway
Order by density, restate scope inside every span, deduplicate furniture, serve Markdown for budget rather than citations, date claims in-text, and scope by subpath instead of building one monolith.
The objection that nearly kills this rule
Here is the strongest counter-argument, and it is genuinely strong: nobody stuffs your whole file into a context window. Retrieval systems chunk documents into passages of a few hundred tokens, embed them, and pull only the top matches at query time. Under that architecture size dilutes nothing, because attention never sees the file — it sees three retrieved passages. A hundred more pages is simply a hundred more chances that one of your passages is the match. On this reading the admission rule throws away free upside, and the guides are right: export everything.
The objection is right about the mechanism and wrong about the conclusion, for two reasons.
First, it does not describe the path that actually reads these files. The fetch-and-read path — an agent calling a tool that returns your document, then reasoning over it in the same window as the user’s code — is what mcpdoc and its equivalents implement. No retriever stands between your file and the model’s attention. Size costs you directly, and the objection does not apply.
Second, in the chunked path the cost changes form rather than disappearing. Empirical work on top-k retrieval finds that dense corpora return overlapping and near-duplicate chunks, and that redundancy crowds out diversity in the retrieved set. Every extra page is not only a new lottery ticket; it is a new competitor to your own strongest passage, in a system that surfaces a handful of results and then stops. Adding a thousand mediocre passages raises the chance the retrieved set is three weak variations on one point rather than the span that resolves the question. Chunking configuration shifts retrieval quality as much as embedding-model choice; a corpus built by concatenation is a chunking decision made by accident.
So the rule survives and its justification splits: in the read path the cost is attention, in the retrieval path it is self-competition. Only the tolerance changes. If your consumer is definitely an ingestion pipeline, the shelf can be generous. If you do not know — and you do not — build for the tighter constraint, because a file sized for the read path still works when chunked, while a file sized for chunking is unusable when read.
This is the same asymmetry that makes conservative decisions correct elsewhere: you size for the constraint that fails badly, not the one that fails cheaply. It is the logic behind keeping internal link structures deliberate rather than sprawling, and behind every manual action recovery that starts by removing rather than adding.
And beyond: version 2 turns a file into a routing layer
On 10 August 2026 Jeremy Howard published version 2 of the spec — the first revision since 2024. Read superficially it is housekeeping. Read properly it changes where the format lives.
V2 supports two URL patterns for Markdown versions of pages: appending .md to the full filename, as before, or replacing the extension, so /docs/tutorial.html serves as /docs/tutorial.html.md or /docs/tutorial.md. More importantly it adds two link relations: rel=“alternate” with type=“text/markdown” points from a page to its Markdown twin, and rel=“describedby” points to the llms.txt file covering it, since one file can describe every page under its path.
<link rel=”alternate” type=”text/markdown” href=”/docs/tutorial.md”>
<link rel=”describedby” href=”/llms.txt”>
Link: </docs/tutorial.md>; rel=”alternate”; type=”text/markdown”
Those relations ship as HTML link elements or as an HTTP Link response header. The header route is the significant one: it works on files that are not HTML and can be added at the server or CDN — the content delivery network in front of your site — without touching a template. The syntax is still open for feedback, so treat it as stable enough to implement and not yet stable enough to hard-code into a product.
What changed in llms.txt v2?
V2 adds a second URL pattern for Markdown pages and two link relations, rel=“alternate” and rel=“describedby”, that let an agent discover a page’s Markdown version and its governing llms.txt file. Existing implementations need a minor addition rather than a rebuild.
The strategic reading: before v2, discovery ran root-first and a machine had to guess that /llms.txt existed. After v2, every page announces its own machine-readable twin and its own index at the moment of contact. The file stops being a destination and becomes a routing layer — the entry point is wherever an agent lands, and your discovery surface scales with page count instead of sitting in one place. That is far closer to how agentic browsing actually generates value than a root file ever was.
What is plausibly next, flagged as speculation rather than plan: pricing metadata attached to referenced URLs, so a high-value page sits behind a payment handshake. Nothing in the spec supports it today — watch it, do not build for it. Expect too the pressure that follows adoption of any self-declared file: as more sites publish these, the incentive to overstate what is in them rises, which is where signed manifests stop being a curiosity. The same arms race already played out in the detection of manipulated link signals.
What this changes for link building specifically
This file is not a link-building tactic and should not be sold internally as one. What the exercise delivers is an unusually honest audit, because the density test is close to a quotability test: a span a machine can lift and use is a span a journalist can lift and use. If your whole library scores below 1.0, you have learned something uncomfortable about why your digital PR and outreach keeps producing coverage without citations: there is nothing in your pages that anyone can quote and attribute cleanly.
Three consequences follow. First, the assets that score highest are the ones worth commissioning more of — original figures, defined thresholds, dated tables. These are the same assets that earn links unaided, which is why original data with a defensible method outperforms opinion in both channels at once, and why a data-led interactive feature should ship a plain-text companion carrying its findings as prose. Second, density scoring gives you a ranked inventory to point outreach at; running it across a rival’s public pages is a cheap variant of competitor asset analysis that tells you which of their claims are portable. Third, high-density spans are what machine-assembled listicles pull from, so rewrite-for-extraction is the one intervention that helps every channel at once.
Keep the accounting clean. Measure fetches, agent-triggered requests, and support tickets caused by wrong machine answers. Do not measure it in citations or referring domains, and do not let it borrow credit from work that earned those — the attribution discipline you would apply to entity authority measurement or to any claim about AI Overviews and links. Tools that score your site on file presence are scoring a checkbox; the tooling that matters is whatever reads your logs, and most of the standard link-building stack does not touch this layer at all.
The Monday checklist
Eight steps, roughly a fortnight of part-time work for a site under 1,000 URLs.
- Fetch your current file and count its tokens. If a plugin generated it, assume it is a site export until proven otherwise. Record the number before you change anything.
- Pull six months of logs for the file’s path. Separate audit tools from user-triggered agent requests. For most sites this baseline is close to zero.
- Set the shelf in writing. Name the smallest effective context among your realistic readers, choose the share you can claim, and put the token figure where the content team can see it.
- Score every candidate page for density. Count self-contained answers with headings hidden. Expect most blog posts below 1.0, and expect that to be uncomfortable.
- Apply the admission rule. Sort by density, add from the top, stop at the first page that lowers the mean or breaches the shelf. Record where you stopped, and why.
- Rewrite admitted pages for decapitation. Restate scope, units and dates inside each span. Re-count answers afterwards; this usually beats admitting more pages.
- Ship the v2 relations. Add both link relations via an HTTP Link header at the CDN, so no template changes are needed, and scope subpath files where sections differ.
- Diarise a quarterly re-score. Re-run density on anything published since, re-apply the rule, and confirm every dated claim in the file still matches the live page.
Name the failure threshold in advance. If, two quarters after publication, agent-triggered fetches are still in single digits and your support log shows no change in machine-caused errors, the file is not your constraint. The cheaper fallback is a hand-maintained llms.txt index at a few thousand tokens plus Markdown twins for your top twenty pages — perhaps two hours a quarter, retaining most of the option value. On a site whose real problem is authority rather than machine-readability that is the correct allocation, and no amount of file engineering substitutes for the citation and mention footprint that actually moves it.
Into 2027 the direction of travel is clear enough to plan around. Discovery moves from the root to the page. Payloads get smaller and more scoped, not larger. The reader that matters is a tool with a job to finish and a budget to spend, and it will always prefer the source that resolves its question in the fewest tokens. That standard is reachable this quarter, and it rewards the writing worth doing anyway: precise, dated, self-contained, short enough to quote whole.
