EN
What Is llms.txt

Agentic web positioning · Guide

What Is llms.txt

August 25, 2026

Web positioning is entering a new stage with the rise of generative LLMs, after decades of an already highly specialised SEO. llms.txt is a direct example of that stage: a 2024 proposal to declare which content on a site is worth an LLM reading and citing. This article explains its origin, how it differs from robots.txt, what real adoption looks like so far, and how this very site implements it.

Ficha del conceutu →

A new stage in web positioning

SEO (Search Engine Optimization) has spent more than two decades being refined: since the 1990s, search engines like Google have kept publishing —and industry professionals kept decoding— increasingly precise signals to determine which page deserves to appear first in a results list. Inbound links, page speed, structured data, user experience… the ground is already well trodden, with reasonably stable rules and mature tools to measure them.

With the rise of generative LLMs —systems that don’t return a list of links but a synthesised answer, citing or not citing the sources behind it— that well-trodden ground opened up again. It’s not that SEO stops mattering: it’s still the basis of indexing and visibility in classic search engines. It’s that a new space appeared, where there are still no settled rules or equivalent mature tools —GEO— and where every actor, from websites to LLM providers themselves, is still testing which signals actually matter.

llms.txt is a direct example of that early stage: a concrete proposal, not a consolidated standard, that tries to solve a real problem —how to make a website declare which content is worth reading— without it yet being certain whether the large providers will adopt it in a widespread way. This article explains what it is, where it comes from, and what real adoption looks like so far.

What llms.txt is

llms.txt is a text file in markdown format published at the root of a website (/llms.txt) to explicitly declare which content is meant to be read, chunked and cited by generative systems based on language models —GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and other AI crawlers. It has no single formal standards body: it’s a proposed convention, adopted increasingly by sites that practise GEO.

The short definition works as a self-contained answer: llms.txt is the file that declares, at a site’s root, which content is worth an LLM reading and citing, without the agent having to crawl and work it out on its own.

Where the proposal comes from

llms.txt is not a standard imposed by a web consortium or an LLM provider: it was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024, published at llmstxt.org. The starting motivation was concrete: LLM context windows are small compared to the total volume of a website, and HTML pages —with navigation, ads and JavaScript— are costly to process for a machine that only needs the actual content. The proposed format is deliberately minimal: an H1 title, a summary paragraph as a blockquote, H2 sections with links to resources, and an optional section for secondary content.

Analogy and difference with robots.txt

The name and location (at the domain root) deliberately echo the classic robots.txt, but the function is different:

  • robots.txt declares crawl permissions: which routes a given crawler can or cannot visit. It’s a list of access rules.
  • llms.txt declares readability and citability: what content exists, what’s worth reading and how it’s organised to be summarised or cited accurately. It’s not a list of access rules, it’s a machine-oriented index.

Both files can coexist on the same site without conflict: robots.txt keeps controlling what can be crawled, and llms.txt shows, within what is allowed, what’s most relevant to cite.

The llms-full.txt extension

Alongside basic llms.txt (a short index with links and summaries), a second convention emerged: llms-full.txt, which includes the site’s full content embedded directly in markdown format. The practical difference is the query cost for the agent: with llms.txt alone, an LLM has to make an additional HTTP request for every page it wants to read in full; with llms-full.txt, all the content arrives in a single response, with no additional round-trips.

How much real adoption there has been

Real adoption of llms.txt has been measured several times since it was proposed. By mid-2025 there were already thousands of websites with a published llms.txt, including well-known technical companies like Anthropic, Stripe, Cloudflare and Vercel; the sectors with the highest adoption are those that most depend on technical precision —artificial intelligence, developer tools, SaaS. There are even community directories that catalogue sites with llms.txt implemented.

But adoption of the format is not the same as its actual being read by LLM providers. In June 2025, John Mueller (Google) publicly stated that no AI system uses llms.txt in any confirmed way, and that server logs show AI crawlers don’t request it in any significant volume. None of the major providers —OpenAI, Google, Anthropic— have published documentation confirming they consult it at inference time. This doesn’t invalidate the practice —a cheap declarative file, with no real automated maintenance cost, remains reasonable— but it does temper any promise that exposing an llms.txt will, on its own, measurably improve a site’s citability in the short term.

Real case: implementation on Xiringase-web

This very site implements both files in production, generated automatically from the site’s actual content (not maintained by hand):

  • /llms.txt generates a summarised listing: Xiringase’s own projects, blog articles (with a link and short summary) and indexable pages (manifesto, support).
  • /llms-full.txt embeds the full content in markdown: the entire body of every blog article, the project features and the home page’s editorial sections, all in a single plain text file.

Both are served only in Asturian (the site’s canon language); the translations into Spanish, English and French live under /es, /en and /fr respectively, without being duplicated in llms.txt.

Relationship with GEO

llms.txt is one of the concrete practices that define GEO: it’s literally one of the “machine-oriented feeds” cited as a practical pillar of that discipline —files that help an automated agent know what content exists on a site and where to find it, without having to crawl the whole web to work it out.

It’s worth drawing the line clearly: llms.txt is not the source of a site’s citability, it’s only an index layer on top of it. A site with well-structured content —atomic, with structured data, factually verifiable— remains citable even without exposing an llms.txt. And conversely, exposing an llms.txt without quality content underneath doesn’t improve real citability: it only makes it easier to discover content that, on its own, either has or lacks value worth citing.

Sources