What is llms.txt?

llms.txt is a proposal, not an established web standard. It takes the same basic idea as robots.txt and sitemap.xml, a plain-text file at the root of a domain, and applies it to a different problem: instead of telling crawlers what they can access or listing every URL, llms.txt is meant to give a language model a short, curated, markdown-formatted overview of a site, with links to the pages that matter most and brief descriptions of what each one covers. The intent is to help an AI system understand a site's structure and priorities quickly, without having to crawl and interpret every page from scratch.

The format itself is simple by design. It is typically a markdown file, at the path /llms.txt, that opens with an H1 naming the site or organization, a short summary blockquote, and then a series of H2 sections grouping links by category, each link accompanied by a one-line description. Some implementations also publish an llms-full.txt, a longer version containing more complete page content inline, intended for systems that want to ingest more of the actual text in one pass rather than following links.

It is important to be direct about adoption. As of now, no major AI provider, not OpenAI, not Anthropic, not Google, not Perplexity, has confirmed that it actively crawls or gives special weight to llms.txt files in retrieval or citation. It emerged from the developer community, most visibly associated with Answer.AI's Jeremy Howard, as a proposed convention rather than something backed by the companies that would need to consume it for it to have real effect. Some crawlers associated with specific AI tools have been observed requesting the file, which suggests limited, inconsistent uptake, but it is not comparable to robots.txt or sitemap.xml, which are universally respected by every major crawler.

That does not make llms.txt worthless, but it changes what it is reasonable to expect from it. Because it costs very little to create and maintain, publishing one is a low-risk, low-effort addition for a site that already has clean information architecture, and it may provide marginal benefit for any AI system that does choose to consume it now or in the future. What it is not is a substitute for the things that reliably affect AI retrieval today: crawlable pages, clean HTML, fast rendering, accurate schema markup, and clearly written, directly stated content on the pages themselves.

There is also a meaningful difference between llms.txt and the technical fundamentals it sometimes gets confused with. robots.txt controls crawler access and is respected by essentially every legitimate bot. sitemap.xml gives crawlers a complete list of URLs to consider indexing and is a foundational technical SEO asset. llms.txt does neither of those jobs. It does not grant or restrict access, and it is not a comprehensive URL list, it is a curated, opinionated summary that assumes an AI system is already willing to fetch and read it, which is exactly the assumption that has not been confirmed at scale.

For a business deciding whether to invest time here, the honest framing is proportional effort. Creating an llms.txt file for a site with a handful of core pages, a clear site structure, and existing technical SEO in good shape takes very little time and carries essentially no downside, so there is little reason not to have one. Treating it as a priority ahead of fixing crawl errors, slow rendering, missing schema, or vague page content would be a mistake, because those fundamentals have demonstrated, consistent effects on retrieval and citation, while llms.txt currently does not.

The space is still moving. It is plausible that adoption grows if enough AI providers find value in a lightweight, standardized summary format, especially as the web gets larger and full-site crawling gets more expensive to run at scale. It is equally plausible that it stays a niche convention used mainly by developer-focused sites and documentation platforms, where the audience publishing it overlaps heavily with the audience proposing it. Anyone claiming certainty about which of those outcomes will happen is guessing, and any vendor selling llms.txt implementation as a guaranteed AI visibility fix is overstating what the file currently does.