llms.txt: What It Is, How It Works and How to Set It Up

If you have heard about llms.txt in the last year but are not clear on what it does or whether your site needs one, this guide is for you. It covers what the file is, what the AI platforms have actually said about it, how to write one, and the mistakes that make it useless.

Short version: llms.txt is cheap to set up and useful for some audiences, but no major AI search platform has confirmed it uses the file to decide what to cite. Treat it as housekeeping, not a ranking lever.

What is llms.txt?

llms.txt is a Markdown file placed at the root of a domain, at https://yourdomain.com/llms.txt. It gives large language models a short, curated summary of what the site is and links to its most useful pages.

The format was proposed by Jeremy Howard of Answer.AI in September 2024. It is not a W3C or IETF standard. It is a community convention, and adoption by the big AI platforms is patchy.

People often call it “robots.txt for AI”. That comparison is misleading. robots.txt controls access: which bots may crawl which URLs. llms.txt controls nothing. It is a reading list, not a rulebook.

What the AI platforms have actually said

This is where most llms.txt guides go wrong, so here is the position as it stands in September 2026.

• Google: Gary Illyes has said Google will not use llms.txt, and that ranking in AI Overviews needs normal SEO. John Mueller compared it to the keywords meta tag. Google’s AI features draw on the same search index as classic results.
• OpenAI, Anthropic and Perplexity: their crawler documentation covers user agents and robots.txt. None of them says llms.txt is used to decide which sources get cited.
• Server logs: independent log audits have found the main AI crawlers rarely or never request the file.

Where llms.txt does get used is in developer tooling. Coding assistants and documentation tools read llms.txt files to pull clean context about a library or API, which is why so many software documentation sites publish one.

So why bother?

Three honest reasons:

1. It costs almost nothing. On most WordPress sites it is a setting you switch on.
2. It helps AI agents and tools that do read it, particularly if you run a SaaS product with documentation.
3. Writing it forces a useful exercise: deciding in one paragraph what your business is and which ten pages matter most. That same clarity is what makes your actual pages easier for AI systems to understand and quote.

What it will not do is get you cited in ChatGPT or AI Overviews on its own. That comes from crawlable, clearly structured content, consistent entity information and third-party mentions. We cover how those fit together in our GEO vs SEO guide.

The llms.txt file format

The proposal defines a simple structure:

1. An H1 with the name of the site or project (the only required part).
2. A blockquote (a line starting with >) giving a short summary.
3. Optional plain paragraphs with extra context.
4. H2 sections, each containing a list of Markdown links, with an optional note after each link.
5. An optional section headed ## Optional for links that can be skipped when context is short.

# Example Store

> UK online retailer of workwear and safety clothing for trades and construction teams.

## Products
- [Hi-vis clothing](https://example.com/hi-vis/): Jackets, vests and trousers to EN ISO 20471
- [Safety boots](https://example.com/safety-boots/): S1P and S3 rated boots and trainers

## Help
- [Delivery and returns](https://example.com/delivery/): UK and EU delivery times and costs

## Optional
- [Blog](https://example.com/blog/): Buying guides and PPE regulations explained

Some sites also publish llms-full.txt, a single file containing the full text of their key pages. It is mostly useful for documentation sites.

How to write an llms.txt file

Step 1: pick your most important pages

List the pages that best represent the business: core product or service pages, key category pages, the about page and your most useful guides. Ten to thirty links is plenty. This is a curated shortlist, not a sitemap.

Step 2: write the summary

The blockquote summary should say what the business is, what it covers and who it serves, in one or two sentences.

Weak: “We provide world-class digital marketing services to help businesses grow.”

Better: “UK SEO and generative engine optimisation agency working with eCommerce and SaaS brands on Google rankings and visibility in ChatGPT, Perplexity and Google AI Overviews.”

The first could describe any agency anywhere. The second names the service, the audience and the country.

Step 3: group links under H2 headings

Use headings a human would understand: Products, Services, Guides, About. Give each link a one-line note describing the page, in plain words rather than keywords.

Step 4: publish it at the root

• WordPress: Rank Math and Yoast both now generate llms.txt from their settings, and let you choose which post types and pages to include. You can also upload a file by SFTP to the site root.
• Shopify: check Shopify’s current documentation or your theme and app options, as the method has changed more than once.
• Static and custom builds: put the file in the public root and deploy.

Step 5: check it works

Open https://yourdomain.com/llms.txt in a browser. It should return a 200 status and display as plain text, with no theme or HTML around it.

robots.txt matters far more

If you only fix one thing for AI search, make it robots.txt. A blocked crawler cannot read your pages, however good your llms.txt is.

The user agents to know:

• OpenAI: GPTBot (training) and OAI-SearchBot (ChatGPT search)
• Anthropic: ClaudeBot
• Perplexity: PerplexityBot
• Google: Googlebot crawls for Search, including AI Overviews. Google-Extended is not a separate crawler, it is a robots.txt token that controls whether your content is used for Gemini training.

If your robots.txt has a blanket block like this, every one of them is shut out:

User-agent: *
Disallow: /

To keep private areas out while allowing public content:

User-agent: GPTBot
Disallow: /account/
Disallow: /checkout/
Allow: /

Also check your CDN or firewall. Some bot-protection settings block AI crawlers before robots.txt is ever read.

Common mistakes

Putting the file in a subfolder

It has to be at /llms.txt on the root domain, not /blog/llms.txt.

Serving it as HTML

If your CMS wraps the file in a page template, it is no longer Markdown. Check the response is plain text.

Treating it like robots.txt

Allow and Disallow rules do nothing in llms.txt. Access control belongs in robots.txt.

A vague summary

Name the services, the audience and where you operate.

Letting it go stale

If links 404 or services have changed, the file is actively misleading. Review it when you change your site structure or positioning.

Does llms.txt improve AI citation?

There is no public evidence that it does, and Google has said it does not use it. The work that moves AI visibility is the same work behind good SEO: crawlable pages, content that answers questions in clear standalone passages, accurate structured data, a consistent brand record across the web, and being mentioned by sources AI systems already trust.

llms.txt is a small, sensible addition to that setup. Do it, spend twenty minutes on it, and move on to the things that matter more.

If you want help with the bigger picture, that is what our generative engine optimisation service covers.

FAQ

Is llms.txt required?

No. It is optional and Google has said it does not use it.

Does llms.txt block AI crawlers?

No. It has no access-control function. Use robots.txt to allow or block crawlers.

Who created llms.txt?

Jeremy Howard of Answer.AI proposed it in September 2024.

Can WordPress create llms.txt automatically?

Yes. Rank Math and Yoast both include an llms.txt option, and dedicated plugins exist too.