What is llms.txt and should your website have one?

Hephaestus Technology7 min read

llms.txt is a proposed plain-text file, written in Markdown and placed at the root of a website, that gives AI tools a short summary of the site and a curated list of its most useful pages. Most B2B websites do not need one to appear in AI answers: Google says it does not use the file for Search, and 2026 data shows most llms.txt files are rarely requested, though it can help software companies whose documentation is read by developers and coding agents.

Key takeaways

  • llms.txt is a proposed Markdown file, placed at the root of a website, that gives AI tools a short summary and a curated list of the site's most useful pages.
  • It was proposed by Jeremy Howard in September 2024; it is a community proposal, not a web standard, and no AI provider is obliged to read it.
  • Google says it does not use llms.txt for Search, and that such files neither harm nor help your visibility there.
  • An Ahrefs study of 137,210 domains found that 97% of llms.txt files received no requests at all in May 2026.
  • For most B2B websites it is a low-cost, low-impact extra; it makes more sense for software products with documentation that developers and coding agents read.

What is llms.txt, exactly?

llms.txt is a convention, proposed on llmstxt.org, for a Markdown file that helps large language models find and read the most important content on a site when they are answering a user's question. It was written by Jeremy Howard and published on 3 September 2024; the site now carries a revised version dated 10 August 2026.

The idea is simple. Web pages are full of navigation, scripts and layout that make them awkward for an AI tool working with limited context. A short, structured index pointing to clean versions of key pages saves the tool from guessing.

It is important to be clear about status. llms.txt is a community proposal, not a standard from a body such as the IETF or W3C, and no AI provider is obliged to read it. As Common Crawl puts it, "llms.txt is not an access-control mechanism, and nothing obliges any crawler to read it."

How does an llms.txt file work?

An llms.txt file follows a fixed order of Markdown elements, according to the proposal:

  1. An H1 heading with the name of the site or project. This is the only required part.
  2. A blockquote with a short summary.
  3. Optional detail in paragraphs or lists, without headings.
  4. H2 sections of links, each entry a Markdown link optionally followed by a colon and a note.
  5. An "Optional" H2 section for secondary links a tool can skip when it needs less context.

The file usually sits at /llms.txt, though the current proposal also allows it in a subpath such as /docs/llms.txt. The proposal also suggests offering a clean Markdown version of each important page at the same URL with .md added, which tools can discover through a rel="alternate" link.

A minimal, illustrative example for a B2B services firm:

# Example Engineering Ltd

> Irish engineering consultancy designing control systems for food and pharma plants.

## Services
- [Control system design](https://example.ie/services/control-systems.md): scope, typical timelines, price ranges
- [Validation support](https://example.ie/services/validation.md): GAMP 5 documentation services

## Company
- [About and team](https://example.ie/about.md): founded, locations, certifications

## Optional
- [Blog](https://example.ie/blog.md)

How is llms.txt different from robots.txt and sitemaps?

The three files do different jobs, and only one of them is a standard.

File Purpose Status Who reads it
robots.txt Tells crawlers which URLs they may fetch Internet standard, RFC 9309 (2022) Search engines and most named AI crawlers
sitemap.xml Lists URLs for discovery Widely supported protocol Search engines
llms.txt Curated summary and reading list for AI tools Community proposal Some AI tools and agents, inconsistently

If you want to control which AI systems may use your content, robots.txt is the tool, as we explain in AI crawlers and robots.txt choices. llms.txt cannot block anything.

Does llms.txt help with Google AI Overviews?

No. Google's optimisation guide for generative AI features says you "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", that Google Search does not use them, and that creating them for other services "will neither harm nor help" your visibility in Search. Google's AI features page makes the same point. The same is true of special schema, which we cover in schema markup for AI search.

What does the data say about llms.txt in 2026?

The data says adoption is growing but use is thin.

  • Requests: Ahrefs analysed 137,210 domains in its web analytics product in May 2026. About 28% had an llms.txt file (an upper bound, since its customers skew technical), and 97% of those files received no requests that month. Named AI bots made 19.5% of the requests that did occur; the top AI fetcher was GPTBot, and Claude-Code, a coding agent, came second.
  • Contents: Common Crawl analysed 584,107 non-empty files from its July 2026 crawl. Only 49.90% had the full specified shape, 68.27% were templated, and Wix alone produced 41.34% of the corpus. 22.56% contained no links at all.
  • Misuse: the same analysis found files carrying rate limits, copyright notices and even prompt-injection attempts, none of which the proposal defines.

Ahrefs stresses that a fetch does not prove a bot used the file, so even these low figures are ceilings.

Should your website have an llms.txt file?

For most B2B service websites, it is optional and low priority. Use this guide:

Your situation Recommendation
Brochure or services site, under 100 pages Optional. Fix crawler access and page content first
SaaS or API product with developer documentation Worth doing, alongside Markdown versions of docs pages
Site built on a platform that generates one automatically Check it is accurate; edit or remove misleading entries
Large content site with clear sections Reasonable as a curated index, if someone owns its upkeep

The strongest case is documentation. Coding agents and AI assistants used by developers fetch documentation directly, and a clean index plus Markdown pages makes their job easier. If your product exposes tools to AI agents, the Model Context Protocol is likely to matter more than llms.txt.

What are the costs and pitfalls?

The cost is small: an hour or two to write the file, plus upkeep whenever key pages change. The pitfalls are more important:

  • Stale links. An outdated llms.txt points AI tools at wrong prices or retired services.
  • Using it as a policy file. Writing "do not train on this site" in llms.txt has no defined effect. Use robots.txt.
  • Expecting traffic. No published study shows llms.txt increasing citations or visits.
  • Neglecting the basics. Time spent on llms.txt is better spent on answer-first pages, until those are done.

Frequently asked questions

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which URLs they may or may not fetch, under an internet standard. llms.txt is an optional guide that points AI tools towards useful content and does not grant or block access to anything.

Does Google use llms.txt?

Not for Search. Google's optimisation guide says you do not need AI text files or Markdown to appear in Google Search, that Google Search does not use them, and that creating them for other services will neither harm nor help your Search visibility.

Will llms.txt get my business cited by ChatGPT or Claude?

There is no published evidence that it does. Ahrefs found that only around 3% of sites with the file received any requests for it in May 2026, and fetching a file is not the same as using it. Citations depend far more on crawler access and the quality of your pages.

What is llms-full.txt?

It is a related convention in which a site publishes the full text of its key pages in one Markdown file. Common Crawl looked at it in its July 2026 analysis but warned that its sample was deliberately enriched, so there is no reliable adoption figure. Treat it as optional.

How long does it take to create an llms.txt file?

For a typical B2B site with a few dozen important pages, writing a good file by hand is a short job: a title, a one-paragraph summary and grouped links with one-line notes. Keeping it accurate as pages change is the ongoing cost.

If you are deciding what your site or product documentation needs for AI tools, book a free 30-minute call with our team.

Cover photo by Lukas Blazek on Pexels.

Hephaestus Technology

We help founders and businesses validate, build and launch products across software, AI and hardware, from offices in Ho Chi Minh City, Dublin and Delaware. See how we work.

Tell us what
you're building.

A free 30-minute call. You'll get an honest view of the idea, the risks we see and what we'd do first, whether or not we end up working together.