AI Crawler Types, Actually Explained

Same Company, Different Bots, Different Jobs

The single most common mistake site owners make with AI crawlers is treating "AI bots" as one thing to allow or block. In reality, a single AI company typically runs several separate crawlers, each with a different name, a different purpose, and a different consequence if blocked. OpenAI alone operates at least three: GPTBot, which gathers content that may improve future model training; OAI-SearchBot, which indexes pages so they can be surfaced and cited in ChatGPT's search feature; and ChatGPT-User, which fetches a single page in real time when a live ChatGPT user's question requires it.

Blocking all three with one blanket rule solves a training-data concern but also removes a site from ChatGPT's citations and breaks live requests real people are actively making through the assistant. Anthropic follows the same pattern with ClaudeBot (training), Claude-SearchBot (search quality), and Claude-User (real-time queries), and Perplexity, Google, Apple, and Meta all maintain comparable splits, though with varying degrees of transparency about which bot does what.

A single AI company often runs three or more separate crawlers under different names, each serving a distinct purpose.

Training, Search, and Agent Crawlers

Named Crawlers and What Each One Does

How to Set Rules for Each Type

Robots.txt Is a Request, Not a Lock

It's worth being clear-eyed about what robots.txt actually accomplishes. It's a plain-text file that compliant crawlers voluntarily honor, not an enforcement mechanism. Most major, named AI crawlers from established companies do respect it, partly because ignoring it carries real reputational and, increasingly, legal risk as publishers and courts pay closer attention to AI training data sourcing. But it's been reported in various contexts that some crawlers, including PerplexityBot in certain configurations and Bytespider, have not always fully honored disallow rules, and a malicious or poorly built scraper can simply ignore the file altogether.

For site owners who need stronger guarantees than a text file provides, firewall-level blocking based on IP ranges or verified bot detection (the kind Cloudflare's AI Crawl Control and similar services offer) provides actual enforcement rather than a request. robots.txt remains the right first layer for legitimate crawlers that behave as documented; it just isn't the only layer worth having if training-data control genuinely matters to a site's business.

Checking server access logs for actual AI crawler user-agent strings confirms whether robots.txt rules are being honored in practice.

Verification matters because user-agent strings are self-reported. Anyone can write a script that announces itself as GPTBot or ClaudeBot, and impostors do exactly that, either to bypass rules that only apply to specific named bots or to make suspicious scraping look like legitimate AI company traffic. The major AI operators publish the IP address ranges their real crawlers use, and OpenAI, Anthropic, and others document these ranges specifically so site owners can confirm that a request claiming to be a given bot actually originates from that company's infrastructure. Cross-checking a suspicious user agent against the published ranges is the most reliable way to tell a genuine crawler from a spoofed one.

The category distinction also affects how a site should think about the tradeoff between visibility and control. Search and citation crawlers are effectively the new distribution channel for a growing share of readers: when someone asks an assistant a question and receives an answer with linked sources, the sites linked are the ones whose content a search-oriented crawler was allowed to read. Training crawlers offer no comparable return. Once content is absorbed into a model, there is no per-use link, no referral, and no way for a reader to trace an answer back to the page that informed it. That asymmetry is why many publishers now treat the two categories completely differently, welcoming one and refusing the other, rather than applying a single policy to everything labeled AI.

Frequently Asked Questions