AI crawler
An AI crawler is a bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web content to train language models or to retrieve sources for AI-generated answers.
AI crawlers matter to docs teams because they determine whether answer engines can see the documentation at all. A blocked crawler or a page that requires JavaScript can prevent a tool from retrieving the documentation. Test access and parsing for the tools that matter to your readers.
There are two distinct kinds: training crawlers, which collect content for future model training, and retrieval crawlers, which fetch pages in real time to answer a live user question. Sites can allow one and block the other in robots.txt, since each uses its own user agent.
JavaScript support varies between crawlers. Serve important documentation in accessible HTML or a documented text format, then check what each crawler can retrieve. Readable content alone does not guarantee that an engine will cite it.
Frequently asked questions
Should I block AI crawlers from my docs?
Decide separately for training and retrieval. Public product documentation may need to be available to search and retrieval tools, while your policy on model-training crawlers may differ. Verify each provider’s user agents and controls.
Which AI crawler user agents should I know?
Check the providers your readers use. Their official crawler docs distinguish training, search, and user-requested retrieval, and list the current user agents and access controls.