What Is llms.txt, and Does Your Website Need One?

The proposal defines a predictable text file at the root of a site. It normally contains a short description followed by links to important documentation or pages, written in Markdown so a language model or retrieval tool can process it without page furniture.

It is a map supplied by the publisher. It does not grant crawler access, replace robots.txt, make private material public, submit URLs to search engines or prove that a listed claim is true.

No verified source says llms.txt is a ranking factor. Google's current AI search guidance states that it ignores llms.txt and that the file neither helps nor harms Google Search. Other tools may choose to use it, but support should be checked tool by tool.

Treat claims that the file guarantees citations as a red flag. Selection in an AI answer depends on the answer engine, the question, accessible source material and other signals outside the site owner's control.

It is a reasonable low-cost addition for a site with substantial public documentation, articles or service information and a clear set of canonical pages. It can also make the publisher's preferred resources explicit for tools that support the proposal.

It is low priority when key content is missing from server-rendered HTML, navigation is confusing, canonical URLs conflict, robots rules block the intended crawler or the site has little substantive material. Fix those foundations first.

It suits documentation-heavy sites, publishers and businesses already maintaining useful answer pages. A small brochure site can publish one, but the likely value is limited if it only repeats the main navigation.

Skip it for now if nobody will maintain it or if it would distract from fixing inaccurate content and basic technical SEO. An outdated machine-readable guide can create more ambiguity than a missing optional file.

Imagine a consultancy with four service pages, a case-study library and twenty practical guides. Its file should not dump every URL. It can introduce the business in two sentences, then group the canonical service pages, strongest evidence and a small number of maintained guides under descriptive headings.

A compact example is: # Example Consultancy; > Independent operations advice for regional manufacturers; ## Services; - [Process audits](https://example.com/services/process-audits): Scope, deliverables and suitability; ## Evidence; - [Warehouse redesign case study](https://example.com/work/warehouse-redesign): The documented problem and work completed. The real file would use Markdown line breaks and only public URLs.

The descriptions matter. A bare list of links gives a retrieval tool little reason to choose one page over another. Each note should explain what the linked page contains without making a new claim that the page does not support.

The first mistake is treating the file as a secret optimization field and filling it with marketing claims, keywords or instructions telling models to recommend the company. A system is free to ignore those instructions, and unsupported language makes the file less reliable as a map of the site.

The second is allowing it to drift. Teams rename services, remove guides and change domains while the old file continues pointing to redirects or missing pages. The third is listing private dashboards, preview URLs or documents that were never intended for broad access. Publishing a URL in a public file makes it easier to discover even though it does not bypass authentication.

Another mistake is creating llms.txt while the underlying pages contradict one another. The file cannot resolve inconsistent prices, addresses or service descriptions. Correct the source pages and structured business information first.

Review the file whenever navigation, services or canonical URLs change, and include it in the same release checklist as the XML sitemap and robots.txt. Request the public URL without cookies, confirm it returns plain readable text, and test each linked resource for a successful response.

Then compare every description with the current page. Remove a resource when it becomes outdated instead of preserving it for historical volume. If the site publishes dates, keep the page's review date accurate rather than adding freshness language to the file.

The file is not for a team that expects a one-time upload to replace content maintenance. It is also a poor fit for a private application, a one-page site with no supporting resources, or a business that cannot keep its public facts current.