Crawlable docs, llms.txt, or MCP: which should you use?

Your documentation may be available through several paths: a public website, an llms.txt file, or an MCP server. They solve different access problems.

Public docs give people a canonical source and make that source available for web access. llms.txt can give tools a curated map of that documentation. MCP lets compatible AI clients access your documentation while a developer is working.

Choose the path your readers and tools actually need, then verify that it still points to current documentation. The table below helps make that choice.

Choose the access path, then test it

Reader situation	Access path	What it provides	What it does not guarantee	Verify before rollout
A person searches or browses public docs	Crawlable documentation page	A canonical page available without authentication	Indexing or retrieval by a particular crawler	Load the canonical URL without a signed-in session and check its rendered task steps
A tool benefits from a curated map of docs	Scoped llms.txt file	Background and links to LLM-friendly material beneath that path	That a crawler or agent will discover or use the file	Confirm the file scope, links, and maintenance owner after a docs release
A developer asks a compatible AI client while working	MCP server	A configured client can connect to a server that exposes approved capabilities	That the client will connect, select a tool, or use returned context for every question	Connect the target client and test one canonical task against the configured source
A procedure is absent, unclear, or stale	Improve the documentation first	Better source material for every path	A delivery layer that invents missing instructions	Repair the canonical guide, then repeat the applicable checks

Start with the canonical documentation page. Add llms.txt or MCP only when they solve a specific access problem for your readers or tools.

Start with public documentation

For public product documentation, the canonical web page should usually be the source you maintain first. Readers can retrieve that page without an authenticated application session. A particular crawler may still handle it differently because robots controls, rendering, and crawler policy vary.

Write each procedure on one page with a specific title, task steps, prerequisites, and links to related guidance. That gives a human reader a source to check when an AI answer is uncertain. For a related content audit, see how to optimize documentation for LLMs and AI agents.

llms.txt can provide a scoped map

The llms.txt proposal describes a Markdown file that gives agents background information and links to more detailed LLM-friendly content. A file can live at /llms.txt or under a path such as /docs/llms.txt, where it covers URLs below that path. The proposal recommends that those links point to clean Markdown versions of the detailed pages when available.

Treat llms.txt as an optional convention, not a crawler directive or an AI-ranking guarantee. It can help a tool find a maintained map of an API reference, getting-started guide, or account-management documentation. It cannot make a missing procedure useful, require an agent to fetch a linked page, or establish how an AI-search product ranks content.

Use a scoped file when its owner can explain which docs it covers and update it after information-architecture changes. A small, current map is more useful than a second copy of the documentation.

MCP connects compatible AI clients to your docs

The Model Context Protocol specification defines an open protocol for connecting LLM applications with external data sources and tools. MCP servers can offer resources, prompts, and tools to a compatible client.

An MCP server is useful when developers want product documentation inside a compatible AI client. The client still has to be configured to use the server, and it may not call the documentation tool for every question. MCP changes how the client can access your documentation. It does not fix missing, outdated, or unclear source content.

Biel.ai provides a hosted documentation MCP server that exposes documentation indexed in a Biel project to compatible AI clients. See the MCP server setup for current requirements, authentication options, and supported clients. For a Claude Code example, see how to connect documentation to Claude Code with MCP.

Keep the source and its access paths in sync

Do not maintain a separate explanation of the same procedure for the page, llms.txt, and MCP server. Keep one canonical guide, then make each path refer to that source.

After changing a procedure, run this short check:

1. Load the canonical public page without signing in and verify the changed task steps.
2. If you publish llms.txt, verify that the scoped file links to the current LLM-friendly page or its maintained equivalent.
3. In the configured MCP client, ask the same task and check the returned context against the canonical guide.
4. Record the date, source URL, client used, result, and owner of any repair.

This check tests access paths after a change. It does not test every crawler, benchmark answer quality, or replace a security review of an MCP deployment. For a pre-launch answer-evaluation record, use how to evaluate a docs chatbot before you ship.

Frequently asked questions

Does llms.txt improve AI search rankings?

There is no general guarantee that publishing llms.txt changes an AI-search ranking or makes a system use your documentation. Use it when a maintained map of LLM-friendly content helps a real reader or tool access your docs.

Should llms.txt link to HTML pages or Markdown alternatives?

The current proposal recommends links to clean Markdown versions of detailed pages when they are available. If you do not publish maintained Markdown alternatives, do not generate a map that points to stale or duplicate content.

Start with the path your readers use

Publish and maintain the canonical documentation first. Then add llms.txt when a scoped map is useful and MCP when a compatible client needs the configured connection.

If you want to make indexed documentation available to compatible AI coding assistants, review the Biel.ai MCP server setup.