llms.txt
llms.txt is a proposed standard file, served at a site's root, that gives large language models a curated markdown index of the site's most important pages.
For documentation sites, llms.txt acts like a sitemap written for AI. Instead of forcing a crawler to infer structure from navigation and HTML, it lists the pages that matter, with short descriptions, in a format language models parse easily.
The proposal also includes llms-full.txt, a single file containing the full text of the docs in markdown. That gives an AI assistant the entire documentation set in one fetch, without HTML boilerplate, navigation, or scripts.
llms.txt is an emerging convention, not a ratified standard. Major AI vendors do not officially commit to reading it, but many docs platforms generate it anyway because the cost is low and adoption keeps growing.
Frequently asked questions
Is llms.txt like robots.txt?
They are complementary. robots.txt tells crawlers what they may not access; llms.txt tells language models what they should read first and how the content is organized.
Do AI companies actually use llms.txt?
Adoption is uneven. No major provider guarantees it is read, but publishing one costs little, and some AI tools fetch it when present.