llms.txt

llms.txt is a proposed standard file, served at a site's root, that gives large language models a curated markdown index of the site's most important pages.

For documentation sites, llms.txt acts like a sitemap written for AI. Instead of forcing a crawler to infer structure from navigation and HTML, it lists the pages that matter, with short descriptions, in a format language models parse easily.

The proposal also includes llms-full.txt, a single file containing the full text of the docs in markdown. That gives an AI assistant the entire documentation set in one fetch, without HTML boilerplate, navigation, or scripts.

llms.txt is an emerging convention, not a ratified standard. Publishing it does not guarantee that a particular AI tool will fetch or use it. Check the behavior of the tools your audience uses.

Questions

Is llms.txt like robots.txt?

No. robots.txt sets crawler access rules. llms.txt proposes a guide to useful pages and how they are organized. It does not control access or guarantee that a tool will read them.

Do AI companies actually use llms.txt?

Adoption varies by tool. Test the tools your readers use rather than assuming that publishing llms.txt makes your docs available to them.

llms.txt

Questions

Related reading