AI crawler
An AI crawler is a bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web content to train language models or to retrieve sources for AI-generated answers.
AI crawlers matter to docs teams because they determine whether answer engines can see the documentation at all. If the crawler is blocked, cannot execute JavaScript, or gets lost in navigation chrome, the docs will not appear in AI answers.
There are two distinct kinds: training crawlers, which collect content for future model training, and retrieval crawlers, which fetch pages in real time to answer a live user question. Sites can allow one and block the other in robots.txt, since each uses its own user agent.
Most AI crawlers do not render JavaScript. Documentation that only exists after client-side rendering is invisible to them, which is one reason server-rendered docs sites and plain HTML content get cited more often.
Frequently asked questions
Should I block AI crawlers from my docs?
For product documentation, usually not. Docs exist to be found, and blocking retrieval crawlers removes your product from AI answers your users are already reading.
Which AI crawler user agents should I know?
GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (Google AI training) are the common ones. Each can be allowed or blocked individually in robots.txt.