llms.txt in 2026: What It Does, Doesn’t, and How to Create One

llms.txt is one of the simplest files you can add to a website and one of the most oversold. In 2026 it is still a useful convention for giving AI systems a clean map of your important pages. It is not a secret ranking cheat code, not a training opt-out, and not a substitute for pages that actually answer questions. This guide covers what the file is, what it does and does not do, and how to create a good one with a generator or by hand.

Direct answer: llms.txt is a Markdown file at the site root (https://example.com/llms.txt) that lists your project name, a short summary, and curated links to the pages AI systems should treat as canonical. Create it with an H1, a blockquote summary, and H2 sections of Markdown links. Use the llms.txt generator to draft it, then edit ruthlessly. For citation strategy beyond the file format, read the AEO pillar: AEO in 2026: What Actually Gets You Cited by AI Search.

1. Where llms.txt came from

In September 2024, Jeremy Howard and Answer.AI proposed llms.txt as a simple, public convention: put a Markdown file at the root of your site so language models can read a curated overview without fighting your layout CSS, cookie banners, and JavaScript chrome. The canonical reference remains llmstxt.org.

By 2026 the idea is widely known in developer and SEO/AEO circles. Documentation platforms and some CMS tools can generate a draft automatically. Chrome Lighthouse documentation has discussed agentic-browsing checks that look for the file. None of that makes llms.txt a formal internet standard with mandatory behavior. Treat it as a polite, structured hint.

2. File anatomy (the parts that matter)

The proposed structure is intentionally small:

H1 — site or product name (required).
Blockquote — one or two sentences summarizing what the site is.
H2 sections — groups such as Documentation, Product, Policies, Blog.
Links — Markdown links with short descriptions: [Title](url): why it matters.
Optional section — lower-priority links that are nice but not canonical.
Optional pointer — to llms-full.txt if you maintain a longer Markdown export.

UTF-8 plain text. Served at the domain root. No HTML wrapper. No tracking pixels. If your CDN requires a rewrite rule so /llms.txt hits a static object, add that rule explicitly and test with curl.

# Example.com
> Example.com builds billing APIs for SaaS companies. This file lists the canonical pages for product, docs, and policies.

## Docs
- [Getting started](https://example.com/docs/start): Install SDK and first API call
- [Auth](https://example.com/docs/auth): API keys, OAuth, scopes

## Product
- [Pricing](https://example.com/pricing): Plans and limits
- [Status](https://example.com/status): Uptime and incidents

## Optional
- [Blog](https://example.com/blog): Product updates and tutorials

3. What llms.txt does in practice

Curates attention. You choose which URLs represent the product truth.
Reduces parse noise. Markdown is cheaper and cleaner than full HTML for many pipelines.
Supports agentic browsing. Tools that look for a site map optimized for LLMs can start here.
Documents intent. Teams align on "these are the pages we want quoted."
Complements AEO work. It points at pages that should already contain direct answers and evidence.

Think of it as a receptionist, not a lawyer. It greets and directs. It does not argue your case.

4. What llms.txt does not do

Claim you might hear	Reality in 2026
"Guarantees ChatGPT/Perplexity citations"	No major public guarantee. Citations still depend on page quality, entity clarity, and retrieval.
"Blocks model training"	Wrong tool. Training preferences use other signals (robots, product terms, publisher controls). llms.txt is not robots.txt.
"Replaces sitemap.xml"	No. Sitemaps help discovery at scale. llms.txt is a curated overview.
"Fixes thin content"	No. Pointing at weak pages just helps models find weak pages faster.
"Is required for SEO"	No. Useful hygiene for AI-era sites, not a Google classic ranking factor checklist item.
"Should list every URL"	No. Curation is the feature. Dumping 5,000 links recreates the noise problem.

5. llms.txt vs robots.txt vs sitemap vs llms-full.txt

robots.txt tells crawlers what they may fetch. It is about permission and politeness at request time.

sitemap.xml enumerates URLs you want discovered, often thousands, with lastmod hints.

llms.txt narrates the few dozen URLs that define the product for a language model reader.

llms-full.txt (optional) can hold longer Markdown versions of key docs for models that want depth without HTML. Keep it updated or do not ship it. A stale full dump is worse than none.

Healthy setup: robots allow the AI crawlers you want, sitemap covers the indexable corpus, llms.txt highlights the canon, and key pages themselves contain answer-ready content.

6. How to create llms.txt manually

List the 15–40 URLs a knowledgeable employee would send a new AI intern.
Group them into 3–7 H2 sections by job-to-be-done, not by your nav labels if nav is messy.
Write one honest blockquote summary. No slogan salad.
Add a one-line description per link that states what question the page answers.
Save as UTF-8 text at /llms.txt.
Fetch it with curl -sI https://yoursite.com/llms.txt and confirm 200 and text content-type.
Re-read quarterly or after IA changes.

7. How to create one with the generator tool

Use the free llms.txt generator when you want a structured draft quickly.

Enter site name and one-sentence purpose.
Add your most important URLs with short descriptions.
Generate Markdown.
Delete anything promotional or duplicate.
Verify every link returns 200.
Publish at the root and re-fetch.

The generator accelerates formatting. It cannot invent which pages are canonical. That editorial judgment is the actual work.

8. Quality bar and anti-patterns

Good signs: stable URLs, descriptions that name the reader question, policies included, pricing included if public, docs versioned carefully, optional section truly optional.

Anti-patterns:

Listing every blog post ever published.
Including noindex or login-walled URLs.
Using tracking query parameters.
Writing descriptions like "click here" or "learn more."
Contradicting on-page titles.
Forgetting to update after a migration.
Shipping only marketing pages and omitting docs where answers live.

9. How llms.txt fits AEO (without restating the pillar)

Answer Engine Optimization is about becoming the source models quote. The file is a pointer layer. The quote still has to be earned by clear entities, direct answer blocks, evidence, and maintainable pages. Do not write a second AEO essay inside your llms.txt project plan. Do the format work here, then improve the destination pages using the pillar: AEO in 2026.

Practical pairing: after you publish llms.txt, pick the five linked URLs most likely to be cited and ensure each has a 40–80 word direct answer near the top, visible evidence, and updated dates.

10. Validation checklist

[ ] URL is exactly /llms.txt on the primary host (watch www vs apex).
[ ] HTTP 200, not 301 into HTML soft-404.
[ ] Content-Type is text-ish (text/plain or markdown-friendly).
[ ] Valid Markdown; H1 present.
[ ] Every link resolves.
[ ] No secrets, staging hosts, or internal IPs.
[ ] Descriptions unique; no copy-paste spam.
[ ] Owner assigned for updates.
[ ] Referenced from internal docs so relaunches do not drop it.

11. Monitoring without mythology

You may see low fetch rates from some bots. That does not automatically mean the file is worthless. It means you should not build a board deck on llms.txt traffic alone. Monitor:

Fetch success in CDN logs for /llms.txt.
Whether AI answers cite the pages you listed (spot checks monthly).
Broken links after deploys (add to CI if you can).

If models cite a random thin page instead of your canon, the fix is usually on-page clarity and internal linking, not a longer llms.txt.

12. Team process that keeps the file honest

Assign a single owner (often docs or web). Require a PR checklist item: "If you add a canonical docs page, consider llms.txt." Review in the same meeting where you review sitemap changes. When marketing launches a microsite, decide explicitly whether it belongs in the optional section or not at all.

13. Multi-product and multi-locale sites

If you run multiple products on one domain, section by product first, then by doc type. If you run locales, prefer one primary language file unless you truly maintain parallel canons; duplicated half-translated sections create contradictions. For subdirectory locales, absolute URLs with locale prefixes beat relative ambiguity.

14. Security and privacy notes

Do not list admin URLs, internal runbooks, or customer subdomains. Do not include tokens. Remember that anything in llms.txt is public by design. If a page should not be highlighted to machines, omit it even if it is indexable.

15. Migrations and redesigns

When URLs change, update llms.txt the same day as redirects. A file full of 301s is acceptable short-term; a file full of 404s is not. If you move docs to a new subdomain, decide whether the apex llms.txt points across hosts (allowed) or whether the subdomain gets its own file (often cleaner for pure docs sites).

16. Worked example: SaaS docs site

A 40-person SaaS company ships billing APIs. Their first llms.txt listed 120 blog posts and buried authentication docs. Models quoted outdated tutorial posts. They cut the file to 28 links: auth, errors, idempotency, rate limits, pricing, status, security, and five evergreen guides. Within a month, internal spot checks showed answers citing the auth and idempotency pages more often. The change was editorial, not magical.

17. Generator output vs human edit (before/after pattern)

Weak generated line: [Blog](https://example.com/blog): Latest news
Strong edit: [Idempotent requests](https://example.com/docs/idempotency): How to safely retry POST payments

Always prefer the page that answers an expensive question.

Frequently asked questions

Is llms.txt required in 2026?

No. It is optional hygiene. Skip it only if you will not maintain it; a rotten file helps no one.

Will llms.txt stop AI from training on my site?

No. Use appropriate robots/publisher controls for training preferences. llms.txt is about inference-time guidance.

Do I need llms-full.txt?

Only if you will keep a high-quality Markdown corpus updated. Otherwise skip it.

Does it replace classic SEO?

No. Technical SEO, content quality, and entity clarity still matter. See the AEO pillar for citation-focused work.

What is the fastest correct path?

Draft with the generator, edit to 15–40 canonical links, publish at /llms.txt, verify with curl, improve the linked pages.

Sources

llmstxt.org — canonical proposal
Chrome Developer documentation on agentic browsing / llms.txt checks (verify current docs)
I Love AI Agents. AEO in 2026: What Actually Gets You Cited by AI Search
I Love AI Agents. llms.txt generator

18. Which content types deserve a place in llms.txt

Prioritize pages that resolve expensive questions: authentication, pricing rules, limits, compliance, core workflows, status, and canonical definitions. Deprioritize ephemeral campaigns, paginated archives, tag pages, and thin listicles. If a page exists mainly to capture a long-tail keyword and collapses under scrutiny, do not promote it to AI systems as canon.

For product companies, docs usually beat blog posts. For media companies, the opposite may be true for evergreen explainers. Match the file to how trust is earned in your category.

19. Writing link descriptions that models can use

Each description should complete the sentence: "Use this page when you need to know ___." Include entities and constraints. Bad: "Our flexible pricing." Good: "Monthly and annual plans, seat minimums, and overage rules for the Growth tier."

Avoid claims you cannot defend on the destination page. The description is a promise. Broken promises train models and humans to distrust your site.

20. Implementation notes for static sites and SPAs

Static sites: drop the file in the publish root beside robots.txt. SPAs: ensure the host serves the raw file without rewriting to index.html. Many "soft 200" SPA bugs make /llms.txt return the app shell. Test with curl and check the first bytes for # or plain text, not .

On Vercel, a public/llms.txt (or repo root for static exporters) is usually enough. On Nginx, add an explicit location if your try_files catches unknown paths.

21. CDN caching

Cache llms.txt like other plain text assets, but keep TTL moderate if you change IA often. Purge on deploy. Do not cookie-vary the file. A personalised llms.txt defeats the point.

22. Versioning and changelog discipline

Keep llms.txt in git even if your CMS is elsewhere. Review diffs in PR. If marketing edits live in a GUI, export back to git on a schedule. Historical diffs answer "when did we stop listing the security page?" during incident reviews.

23. Competitor patterns (what to copy, what to ignore)

Copy: short summaries, docs-first sections, absolute URLs, honest optional sections. Ignore: wall-to-wall adjective spam, duplicate links, QR codes, and embedded HTML. If a competitor lists fifty near-identical feature pages, do not imitate the bulk; imitate the clarity of their best three lines.

24. Metrics that matter more than file vanity

Share of AI answers that cite your canonical URLs vs random posts
Support deflection on questions already answered in linked docs
Time-to-find for new hires using only public pages
Broken-link rate in the file after releases

Vanity metric to downgrade: raw bot hits on /llms.txt without outcome measures.

25. Tie-in to your content calendar

When you publish a pillar, decide the same day whether it enters llms.txt. Most should not. Pillars that define category language often should. Product changelog spam almost never should. Make the decision in the publish checklist beside "sitemap" and "internal links."

26. Common objections, answered

"Models already browse the web." Yes. They also benefit from cleaner maps, just as humans benefit from sitemaps and nav. "Our sitemap is enough." Sitemaps are inventories; llms.txt is a guided tour. "We will wait for a formal standard." You can wait. The cost of a good file is low enough that waiting is usually ideology, not ROI math. "We tried it and saw no traffic." It is not an acquisition channel. Judge it as documentation infrastructure.

27. 90-day maintenance plan

Week 1: ship v1 with 15–40 links.
Week 2: fix any 404s; align titles.
Month 1: spot-check AI answers for your top 10 queries.
Month 2: improve destination pages that fail citation checks.
Month 3: prune links that never deserve canon status; add new product truths.

28. Closing

llms.txt is a small file with a narrow job: show AI systems the pages you are willing to stand behind. Create it carefully, maintain it like docs, and invest the real effort in the destinations it lists. Use the generator for speed, the checklist in this article for quality, and the AEO pillar for citation strategy.

Deep dive note 1: operational detail

Treat llms.txt like an executive summary of your information architecture. When a new product surface launches, decide within the same sprint whether it is canonical. If two pages compete for the same question, pick one winner in the file and demote or redirect the loser. Ambiguous canons create ambiguous answers. Keep descriptions concrete: name the audience, the task, and the constraint the page resolves. Re-read the file aloud; if a sentence could apply to any SaaS company, rewrite it until it could only apply to yours. After each major docs release, spot-check five AI answers and record which URL was cited. Feed misses back into either the file or the page. This feedback loop is the actual AEO practice; the file is only one instrument in it. Prefer stable paths over campaign URLs. Prefer primary research pages over syndicated reprints. Prefer policy pages with dates over undated FAQ blobs. If legal pages change, update the file the same day — models quoting outdated ToS is a brand risk, not a curiosity. For open-source projects, link to contribution guides and security disclosure pages explicitly. For marketplaces, link trust and safety rules before promotional collection pages. For API companies, link error catalogs and idempotency guides before generic "why us" pages. These choices sound obvious in workshops and get skipped in launch week. Put them in the publish checklist so they do not rely on memory.