Let me save you some time.
If you’ve been hearing buzz about llms.txt and wondering whether you need to bolt one onto your link building site — here’s the honest answer:
It’s a near-zero-cost optionality bet. Worth doing, but not worth losing sleep over.
That’s the short version. But there’s a longer story here that matters if you’re building topical authority in 2026, because the gap between what llms.txt enthusiasts claim and what the actual data shows is bigger than almost any other topic in AI search right now.
I’ll cover what llms.txt is, where it came from, what the 2026 adoption data actually says, and exactly when it’s worth the effort for a link builder versus when you should ignore it and ship something else. There’s a working example you can copy at the end.
Let’s get into it.
What is llms.txt? (In plain English)
llms.txt is a markdown file you stick at the root of your website — yoursite.com/llms.txt — that gives AI models a curated map of your best content.
That’s it. That’s the whole concept.
Think of it like this. Your sitemap.xml says: “Here is every page on my site.” Robots.txt says: “Here is what bots are allowed to crawl.” llms.txt says something different. It says: “Out of everything on my site, here are the pages I actually want an AI model to use when answering questions about my niche.”
It’s a curated recommendation, written in plain markdown, that lives at a predictable URL so any AI system that wants to read it can find it.
The proposal was made by Jeremy Howard of Answer.AI in September 2024. His reasoning: AI models have small context windows, websites are messy (HTML, JavaScript, ads, navigation, footers, cookie banners), and asking a model to parse all that just to figure out what your site is about wastes tokens and produces worse answers. A curated markdown file solves both problems at once.
Important: llms.txt is not a blocking tool. It can’t restrict any crawler. It can’t prevent any AI system from reading your content. It just tells AI tools where the good stuff is. If you want to block AI crawlers, you do that in robots.txt — which I’ll cover later in this guide because it actually matters more than llms.txt does right now.
What an llms.txt file actually looks like
The spec is genuinely simple. Here’s the structure every llms.txt file follows:
- An H1 heading with your site or project name. Required. Only one.
- A blockquote summary — one to three sentences describing what your site is and who it serves. Required.
- Optional paragraphs of extra context (scope, conventions, anything that helps an AI understand the site).
- H2 section headings that group your links — “Guides,” “Tools,” “Statistics,” whatever makes sense.
- Under each H2, a markdown bullet list of links. Each link has a short description after a colon.
- An optional “Optional” H2 section at the end for nice-to-have content. AI models with limited context windows are advised to skip this section if they need to.
Here’s what it looks like in practice. This is a working example for a link building site:
# Link Building Journal
> Link Building Journal is the UK’s topical authority site covering
> link building strategy, tactics, statistics, and tools for SEOs and
> in-house marketing teams in 2026.
We publish in-depth, data-backed guides on every aspect of link
building, from fundamentals through to AI search and digital PR.
## Foundations
– [What is Link Building](https://linkbuildingjournal.co.uk/what-is-link-building/): The 2026 definition of link building and why it still matters.
– [15 Link Building Strategies](https://linkbuildingjournal.co.uk/link-building-strategies/): Tactics that work in 2026, ranked by ROI.
## Tools and Data
– [Best Link Building Tools](https://linkbuildingjournal.co.uk/link-building-tools/): Reviewed and ranked link building software.
– [Link Building Statistics](https://linkbuildingjournal.co.uk/link-building-statistics/): Up-to-date 2026 data on link building performance.
## Optional
– [About](https://linkbuildingjournal.co.uk/about/): Who we are and our editorial standards.
Notice what’s NOT in there: every blog post on the site, every category page, every tag page. That’s the point. It’s curation. If you list 800 pages, a model with a tight context budget will run out of room before it finds anything useful. Twenty to fifty links is plenty. Cloudflare, Anthropic, Vercel, and Coinbase have all published llms.txt files at roughly this size, and that’s the working norm in 2026.
There’s also a sister file: /llms-full.txt. Instead of just links, it contains the full text of your documentation flattened into one markdown file. It’s mostly used by developer-focused sites — Anthropic and Stripe both ship one — and it lets developers paste a single URL into ChatGPT or Claude to load the entire documentation set into context. For most marketing sites and link building blogs, you don’t need llms-full.txt. Just /llms.txt is plenty.
The honest 2026 adoption picture (this is where it gets interesting)
Here’s where most llms.txt guides go quiet. Let me give you the data straight.
Adoption among websites: still tiny
SE Ranking ran a study across 300,000 domains in early 2026. The result: 10.13% adoption. That sounds higher than you’d expect, until you read the second finding — among the 50 most-cited domains in AI responses, only one had an llms.txt file.
Another study from SEOScore.tools analysed 10,000 websites across industries and found 96.8% had no llms.txt at all — a 3.2% adoption rate using a different sample. The two studies disagree on the headline number, but agree on the direction: it’s still very much an early-adopter file.
AI crawlers requesting the file: even tinier
OtterlyAI ran a 90-day test. They deployed llms.txt on a test domain and watched the server logs. Over the test period there were 62,100 AI bot visits to the site. Of those, 84 requests targeted the llms.txt file directly.
That’s 0.1% of all AI crawler traffic.
OtterlyAI subsequently removed llms.txt from their GEO audit checklist, on the grounds that it was “drawing attention away from factors that actually move citation frequency.” That’s a fair criticism.
Traffic impact: nothing measurable
Search Engine Land tested llms.txt across nine sites and reported eight of nine saw no measurable change in traffic after implementation. Google’s John Mueller commented publicly that “none of the AI services have said they’re using llms.txt — and you can tell when you look at your server logs that they don’t even check for it.”
That was Mueller in late 2025, and the data through Q1 2026 backs him up. AEO Engine’s monitoring across 200+ client websites came to the same conclusion independently: zero correlation between llms.txt presence and reduced or increased AI citations.
But: some signals are real
It’s not all bearish. Profound, which specialises in GEO tracking, reported in early 2026 that Microsoft and OpenAI crawlers are actively fetching both /llms.txt and /llms-full.txt files. Anthropic, Cloudflare, Vercel, Coinbase, and Astro have shipped public llms.txt files of their own. The Mintlify documentation platform added auto-generation in November 2024, which means thousands of dev-tool docs now have llms.txt files by default.
What’s actually working today is the developer-experience use case. If your audience uses Cursor, Claude Code, GitHub Copilot, or similar AI coding assistants — those tools really do retrieve docs in real time, and a well-formed llms.txt measurably improves how they answer questions about your product.
What’s not yet working is the SEO/link building citation case. There is no documented citation lift, traffic lift, or ranking lift from having an llms.txt file. Not yet.
So why should link builders bother?
Fair question. If the data says it doesn’t move citations today, why am I writing 3,500 words about it?
Three reasons. They’re soft reasons, but they’re real.
Reason 1: Optionality
If even one of the major AI providers — OpenAI, Google, Anthropic, Perplexity — officially adopts llms.txt as a ranking or citation signal in 2026 or 2027, adoption will explode overnight. The sites that already have one will benefit immediately. The sites that don’t will scramble.
Building an llms.txt file takes maybe 30 minutes for a small site. That’s the cheapest insurance policy in SEO. You pay 30 minutes now to be ready if the signal turns on.
Reason 2: The curation exercise is genuinely useful
Here’s something nobody warns you about. The hardest part of writing an llms.txt isn’t the markdown formatting. It’s deciding which 20-50 pages on your site are actually worth promoting.
Most link building sites have 100, 300, or 500+ posts. When you sit down to write your llms.txt, you have to answer a question you’ve probably been avoiding: “If an AI was going to cite ONE page on my site for each major topic, which page would I want it to be?”
That question forces you to face up to which posts are pillar content and which are clutter. You’ll find pages you forgot you’d published. You’ll find topics covered in three half-finished posts that should be one definitive guide. The exercise is worth the time even if no AI ever reads the file.
Reason 3: AI coding assistants and IDE agents really do use it
This is the use case that’s working today. If any portion of your audience uses Claude Code, Cursor, GitHub Copilot, or similar tools, an llms.txt makes your content meaningfully easier for those tools to retrieve and cite.
For a link building site that’s a smaller portion of the audience than for a SaaS docs site, sure. But “a small portion of my audience uses AI coding assistants” is still more than zero benefit for a file that takes 30 minutes to make.
If you’re still building out your content foundations, work on those first. A solid set of link building strategies that work in 2026 will move the needle ten times more than any llms.txt file in May 2026. Don’t skip the basics for the new shiny thing.
When you genuinely shouldn’t bother
I think llms.txt is worth shipping for most sites. But not all. Skip it if any of these apply:
- Your site is under 20 pages of indexed content. There’s no curation problem to solve. AI models can read your sitemap.xml and figure it out themselves. The juice isn’t worth the squeeze.
- You don’t have any pillar content yet. llms.txt is a way to amplify already-good content. If you don’t have a small set of authoritative cornerstone pieces yet, build those first. The file will still be there in three months.
- You’re running on a CMS that won’t let you publish raw markdown at the root. Some older setups serve everything with content-type text/html. You need /llms.txt to be served as plain text or markdown. If your stack can’t do that without a custom solution, your time is better spent elsewhere.
- Your traffic is already healthy and your AI citation visibility is fine. If you’re already getting cited at the rate you want, llms.txt is unlikely to move the needle further. Spend the 30 minutes on a refresh of an underperforming post instead.
If none of those apply to you, ship one. It’s worth doing.
How to write your own llms.txt (step by step)
Here’s the workflow I’d actually use for a link building site.
Step 1: Make a list of your 20-30 best pages
Open a notebook or a doc. Write down every page on your site that’s genuinely your best work — the definitive guide on each major topic, your most-cited statistics page, your tool comparison hub, your most-shared case study. Keep going until you have between 20 and 50 entries.
If you can’t get to 20 entries, you don’t have enough cornerstone content yet. Stop, build more cornerstone content, then come back.
If you have over 50, you’re going too broad. Cut it back. The whole point is curation.
Step 2: Group them into 4-6 sections
For a link building site, the natural groupings tend to be: Foundations, Strategies and Tactics, Outreach, Tools and Data, AI Search, and Case Studies. Pick the groupings that match your content. Don’t force a section just to hit a number.
If you want a structural starting point, the way we organise content on Link Building Journal mirrors our four hub articles — what link building actually is in 2026, the 15 strategies that work, the best link building tools, and our running 2026 link building statistics. Those four hubs anchor the rest of the site, and they anchor the llms.txt too.
Step 3: Write a one-line description for every link
This is the part most people skip — and it’s the part that actually matters. Auto-generated llms.txt files almost never include per-link descriptions, which is exactly why they don’t work as well as hand-curated ones.
Each description should be one sentence. Tell the AI what’s on that page and who it’s for. “Tactics that work in 2026, ranked by ROI” is better than “A blog post about link building tactics.”
Step 4: Write your H1 and blockquote summary
H1: your site name. Blockquote: one to three sentences saying what the site is, who it serves, and the year (because freshness matters in 2026 AI search). Keep it tight. This is the first thing an AI reads, and it sets the tone for everything else.
Step 5: Publish at /llms.txt
Put the file at the root of your domain. Use the URL yoursite.com/llms.txt. Not yoursite.com/seo/llms.txt. Not yoursite.com/static/llms.txt. The root. AI tools that go looking for it look at the root first.
Make sure your server serves it as plain text (content-type: text/plain or text/markdown). If your server serves everything as text/html by default, you’ll need to add a rule. On WordPress, the Yoast SEO plugin now auto-generates and serves an llms.txt file with one click. On static sites built with Astro, Next.js, or similar, you typically drop the file into the public/ or static/ folder and you’re done.
Step 6: Update it quarterly
Don’t leave dead links in there. Every quarter, when you do your normal content audit, take ten minutes to review your llms.txt. Remove pages you’ve deleted. Add pages you’ve shipped that deserve the spot. Update descriptions if the focus of a page has shifted.
That’s it. The whole thing should take you 30 minutes the first time and ten minutes per quarter after that.
What to include in YOUR llms.txt as a link building site
Generic llms.txt advice tells you to include “your best content.” Useful, but not specific enough. Here’s what works for a link building site specifically.
- Your foundational definition pages. What link building is, what anchor text is, what a backlink profile is, what DR/UR/DA mean. These are the queries where AI models reach for citations most often.
- Your statistics and data pages. AI models love citing concrete numbers with named sources. A well-maintained statistics hub is one of the highest-value entries you can include.
- Your tool comparison and review pages. Comparison queries (“Ahrefs vs Semrush”) often trigger retrieval. A well-structured comparison page that you’ve earned editorial authority on is a high-yield include.
- Your tactics and strategy hubs. Listicles of tactics — “15 link building strategies,” “7 outreach templates,” “10 content formats that earn links” — are extracted near-verbatim by Gemini and ChatGPT and travel further than long-form essays do on AI citation pipelines.
- Your case studies with named numbers. “How we built 47 links for a UK fintech in six weeks” is more citation-worthy than “Lessons from a recent campaign.” Named, dated, sourced case studies are catnip for AI citations.
What to leave out: thin posts, gated content (AI can’t access it anyway), category and tag archive pages, author bio pages, and anything you wouldn’t want a model to quote as authoritative. Your llms.txt is a hits collection, not a duplicate of your sitemap.
The thing nobody tells you: robots.txt matters more than llms.txt
Here’s the practical truth in May 2026. The file that actually controls how AI systems treat your site today is robots.txt — not llms.txt.
AI crawlers respect User-Agent-specific rules in robots.txt. They mostly do not respect llms.txt yet. If you only spend time on one of the two files, spend it on robots.txt.
Here’s the configuration that’s currently considered best practice for link building sites that want maximum AI citation visibility (i.e. you want to be cited, you’re fine with that):
# Allow real-time AI fetches for citation purposes
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Allow Google’s AI search crawler
User-agent: Google-Extended
Allow: /
# Block bulk training scrapers if you want to opt out
# (remove these lines if you want maximum training inclusion)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
That config does something specific: it lets the on-demand search bots fetch your pages when a user asks ChatGPT, Claude, or Perplexity a question (so you can be cited), but it blocks the bulk training scrapers (so you’re not feeding your content into the next training run for free).
If you don’t care about the training-data question and just want maximum visibility, allow everything. If you’re a publisher with editorial concerns about training inclusion, the dual config above is the current consensus position among UK and US publishers in 2026.
One more note: ClaudeBot’s crawl volume increased roughly 800% at the start of 2026 as Anthropic scaled its web search API, and Google-Extended is now the largest single AI crawler globally, accounting for around 31.6% of all AI bandwidth share. These are not small fish anymore. The robots.txt rules you set for them matter.
How to track if your llms.txt is actually doing anything
Here’s the test that takes five minutes and tells you almost everything you need to know.
- Open your server access logs (or your Cloudflare analytics if you’re on Cloudflare).
- Filter for requests to /llms.txt and /llms-full.txt.
- Cross-reference the requesting user-agents against the known AI bot list: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Meta-ExternalAgent.
- Count how many requests you’re seeing per week from each.
If you’re seeing weekly fetches from at least two named AI crawlers, your file is being read. That’s not the same as being cited because of it — but it’s the precondition for any citation lift to ever happen.
If you’re seeing zero fetches after 60 days, the file is sitting there as pure optionality. That’s fine — but don’t expect any tangible benefit yet.
A simple trick that works well: embed a unique honeypot link inside the llms.txt that points to a URL nothing else on the site links to. If anything ever follows that link, you know an automated reader has parsed your llms.txt. It’s a cleaner signal than just counting hits to the file itself.
For server log analysis, most teams use the same tooling they already have. If you don’t have a dedicated bot analytics setup, Cloudflare Analytics breaks down bot traffic by user-agent for free, which is enough to get started. We cover broader options in our review of the best link building tools available in 2026, a number of which now have AI bot tracking modules layered on top of their classic SEO functionality.
llms.txt vs robots.txt vs sitemap.xml — what each one actually does
These three get confused all the time. Here’s the cleanest comparison I can offer:
| File | What it does | Audience | Priority in 2026 |
| robots.txt | Controls which bots can crawl which pages on your site | All bots (search and AI) | High — biggest lever right now |
| sitemap.xml | Lists every URL on your site for search engines to index | Search engines (and some AI crawlers) | High — table stakes for SEO |
| llms.txt | Recommends your best pages to AI models as a curated map | AI tools, IDE agents, some AI crawlers | Low-medium — optionality bet |
| llms-full.txt | Flattens entire site content into one markdown file for AI ingestion | AI tools loading full context | Niche — mostly developer-docs sites |
| schema.org markup | Adds structured per-page metadata for entity recognition | Search engines and AI retrieval pipelines | Very high — measurable citation lift on Claude and Gemini |
If you’re prioritising in 2026: robots.txt first, sitemap.xml second, schema markup third, llms.txt fourth. That ordering reflects current measurable impact, not future potential. If your audience includes developers using AI coding tools, bump llms.txt up to third.
The honest verdict
llms.txt in May 2026 is a low-cost, low-yield bet with clear optionality.
It will not move the needle on your AI citations today. The data is clear on that — every independent study run between Q4 2025 and Q1 2026 has found either zero impact or negligible impact on citations and traffic. John Mueller is right that the AI services aren’t really reading it yet. OtterlyAI is right that it’s drawing attention from things that matter more.
But it’s also a 30-minute job. And if even one of the major AI providers turns it on as a citation signal between now and end of 2027 — which several of them might — the sites that already have a well-formed llms.txt will benefit immediately and the sites that don’t will be playing catch-up.
My recommendation: ship one. Spend half an hour. Curate your 20-30 best pages, write one-line descriptions, publish at /llms.txt, add it to your quarterly content audit. Then move on to the things that actually move citations today: schema markup, earned editorial coverage, Knowledge Graph entity work, and a robots.txt configuration that lets the right AI bots in.
If you want the foundational framework that underpins all of this, our breakdown of what link building is and why it still matters in 2026 is the right place to start. AI search visibility is a layer on top of solid traditional link building, not a replacement for it.
Don’t believe anyone telling you llms.txt is a game-changer in May 2026. Don’t believe anyone telling you to ignore it either. It’s just a small bet with asymmetric upside, and the cost is half an hour of your time.
Ship it. Move on.
FAQ
Is llms.txt an official standard?
No. It’s a community-driven proposal published by Jeremy Howard of Answer.AI in September 2024. It is not currently a W3C or IETF standard. The IETF launched an “AI Preferences Working Group” for related standards in 2025, but llms.txt itself is not part of that effort. Compliance from AI providers is voluntary.
Does Google read llms.txt?
No, based on John Mueller’s public comments in 2025 and the absence of any documented Google adoption through Q1 2026. Google has explicitly said its search systems do not read or act on llms.txt files.
Does Anthropic’s Claude read llms.txt?
Anthropic has not formally confirmed that Claude reads llms.txt in production. However, Profound’s 2026 research reports that crawlers from Microsoft and OpenAI are actively fetching llms.txt and llms-full.txt files. Anthropic has published its own llms.txt at anthropic.com, so they clearly see value in the format — they just haven’t publicly committed to reading other people’s files when generating responses.
Will llms.txt improve my Google rankings?
No. Google’s John Mueller has confirmed publicly that no Google Search system reads or acts on llms.txt. It has zero impact on classic organic search rankings. If a tool or guide tells you otherwise, they’re guessing.
How long should an llms.txt file be?
Keep it under 200K tokens — roughly 150K words or about 700KB of file size — so an AI model can ingest it in one go at typical context window sizes. For a link building site, you’re not going to come close to that ceiling. 20-50 curated links with one-line descriptions is the right size. If you find yourself listing 200+ links, you’ve stopped curating and started dumping your sitemap.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a curated navigation file — a markdown list of links with short descriptions, designed as a map. llms-full.txt is the deep version: it contains the full text of your important content flattened into a single markdown document, so an AI tool can load your entire documentation set into context with one URL. For link building sites, you typically only need llms.txt. For documentation-heavy sites like SaaS products, both are useful.
Can llms.txt block AI from training on my content?
No. llms.txt is an inclusion file, not a restriction file. It cannot block any crawler. If you want to block AI bots from training on your content, you need to use robots.txt with User-Agent-specific rules for GPTBot, ClaudeBot, Google-Extended, CCBot, and Meta-ExternalAgent. That’s the lever that actually controls AI access today.
Do I need to update llms.txt every time I publish a post?
Only if the new post belongs in your curated set, which most posts won’t. The whole point of llms.txt is that it’s a hits collection of your 20-50 best pages. A new blog post is only worth adding if it joins that list. Quarterly updates are plenty for most sites.
Should small business sites bother with llms.txt?
If your site has fewer than 20 pages of indexed content, probably not — there’s no curation problem to solve, and AI tools can read your sitemap and figure out your structure on their own. If you have a properly developed content library with at least 20-30 pages you’d happily promote as definitive, yes, ship one. Half an hour, low yield today, low risk, real optionality.
Are there tools that auto-generate llms.txt files?
Yes. Firecrawl’s generator is the most popular standalone tool. Mintlify auto-generates llms.txt and llms-full.txt for hosted documentation projects. Yoast SEO’s WordPress plugin now includes one-click llms.txt generation. Treat any auto-generated output as a draft — auto-generated files tend to be too long and skip the per-link descriptions that make the file actually useful to a model.
