AI Crawler Rate Limiting Explained: How Sites Can Manage Machine Traffic

What Is AI Crawler Rate Limiting?

Rate limiting caps how many requests a client may make in a given period and answers extra requests with an HTTP 429 status instead of the page. For AI crawlers, it means slowing heavy machine traffic so your servers stay fast for people, without blocking the bots that bring you visibility.

The difference from blocking matters. A block says "go away." A limit says "slow down." A well-behaved crawler that receives a 429 with a Retry-After header waits and tries again, so the content still gets collected, just at a pace your infrastructure can afford.

Why Machine Traffic Needs Managing

Search engines have crawled the web for decades, and most sites learned to live with them. AI systems add more visitors, and they do not all behave alike. Some collect pages to train models. Some build search indexes. Some fetch a single page because a person just asked a question. Each has different volume, timing and value to you.

The cost lands in a few places. Dynamic pages that hit a database, internal search results, filtered product listings and calendar views are expensive to generate and easy to crawl endlessly. If you pay per request or per gigabyte at a host or CDN, the bill follows. If your site slows down under load, human visitors and Googlebot both notice.

Blocking is tempting but blunt. Blocking a training crawler is a policy choice about your content. Blocking a search crawler can remove your pages from AI answers, and blocking a user-triggered fetcher can make an assistant fail when a customer asks about you. Rate limiting lets you separate those cases. For the wider question of who these visitors are, read how to separate human visitors from AI agent traffic.

Sort Bots by Purpose Before Setting Limits

The same company often runs several bots with different jobs. Operators publish this in their documentation, and it changes, so confirm the current names before writing rules.

OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as separate agents controlled separately in robots.txt, and Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot the same way. Google lists Google-Agent among its user-triggered fetchers and says such fetchers generally ignore robots.txt because a person requested the fetch. So a rule that works for one group may do nothing for another.

Four Layers of Control

Think in layers, from politest to firmest.

1. robots.txt. A voluntary signal telling crawlers which paths to fetch. Good for scope, weak for speed.
2. Crawl-delay. A robots.txt line asking a bot to wait between requests. Anthropic's documentation says ClaudeBot honors it and Bing has long supported it. Google does not.
3. Response codes. Answering with 429 or 503 tells bots to back off. Googlebot treats these as a signal to reduce its crawl rate, and well-built AI crawlers do the same.
4. Network-level rules. CDN or firewall rate limits and bot rules enforce the limit no matter what the client wants.

Start at the top and move down only as needed. Most sites get far with robots.txt scoping plus a CDN rule. A snippet that scopes some bots and slows another looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot Allow: /

User-agent: ClaudeBot Crawl-delay: 10 Disallow: /internal-search/ ```

Robots.txt changes are not instant. OpenAI notes that updates can take some time to be reflected, so give it a day before judging the effect.

Setting Limits at the Server or CDN

For real enforcement you need a rule that runs on your side. Every CDN and web server has one. Here is an nginx example that limits only requests whose user agent claims to be a known AI bot, leaving everyone else alone:

map $http_user_agent $ai_bot {
    default "";
    ~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|CCBot) $binary_remote_addr;
}
limit_req_zone $ai_bot zone=aibots:10m rate=30r/m;

server { location / { limit_req zone=aibots burst=10 nodelay; limit_req_status 429; } } ```

The empty default key means non-matching visitors are never counted. The rule allows 30 requests a minute per IP address claiming to be an AI bot, with a small burst. Tune the numbers to your capacity and traffic. Nginx returns 503 by default when a limit trips, which is why the example sets 429 explicitly.

When you send the 429, add a Retry-After header with a number of seconds. It costs nothing and gives compliant crawlers a clear instruction. A user agent match alone is easy to spoof, so pair it with IP or signature checks where the operator supports them. Cloudflare's AI Crawl Control and similar CDN products let you see, allow or block AI crawlers by category and adjust responses without editing server config, and Cloudflare has also described a pay-per-crawl option built on HTTP 402 responses. Product details evolve, so check current documentation before relying on any of it.

Fix What Makes Crawling Costly

Limits treat symptoms. It is worth also reducing what is expensive to crawl.

• Cache aggressively. Pages that change rarely should be served from a CDN cache, so a burst of bot requests never reaches your application.
• Tame infinite URL spaces. Faceted filters, calendar pages and internal search can generate millions of URLs. Use robots.txt disallow rules, canonical tags and noindex where appropriate.
• Keep a clean sitemap. A sitemap that lists your real pages gives bots a shortcut, so they crawl less blindly.
• Serve static snapshots for heavy pages. A pre-rendered version of a slow report page can absorb traffic without database load.

How to Roll Out Limits Safely

1. Read your logs first. Identify which bots hit which paths and at what rate. Set limits above normal legitimate peaks.
2. Start in log-only mode. Many CDNs let you record what a rule would block before enforcing it.
3. Exempt verified search crawlers. Confirm Googlebot and bingbot through reverse DNS, and never rate limit them by user agent alone. Search Console's crawl stats report shows whether Googlebot is being slowed.
4. Watch for false positives. Check that customers using assistant features are not being refused.
5. Alert on 429 volume. A sudden jump means either a new bot or a rule that is too tight.

Test with curl by sending a request with a bot user agent repeatedly and confirming you get a 429 and a Retry-After header.

A Tiered Policy for a Mid-Sized Content Site

Picture a publisher with a few thousand articles, a slow internal search page and a hosting plan that charges for bandwidth. The details are hypothetical, but the reasoning applies widely.

Tier one: search and classic crawlers. Googlebot, bingbot and AI search bots such as OAI-SearchBot are allowed everywhere except the internal search path. No user-agent rate limit is applied to Googlebot and bingbot once verified by reverse DNS. The AI search bots get a generous limit that only trips during unusual bursts.

Tier two: training crawlers. The publisher decides it will allow some training crawlers to read public articles but not the archive of subscriber-only content. Robots.txt disallows the archive path for each named bot, and the CDN caps request rates at a modest level with a 429 and a Retry-After value.

Tier three: user-triggered fetchers. These are allowed, because they represent a person asking about an article. A per-IP limit prevents any single address from pulling thousands of pages, which no real user request would need.

Tier four: unverified claims. Anything that names a well-known bot but fails IP or signature checks gets a strict limit and, if it keeps going, a challenge or block.

The policy is a page of notes, three CDN rules and a robots.txt file. It also gives the publisher a sentence to say when someone asks why a bot was slowed: because of purpose, not because it is an AI.

What About Licensing and Pay-Per-Crawl?

Rate limiting manages load. It does not decide whether machine access should cost money. Some infrastructure providers have started offering ways to charge crawlers, for instance Cloudflare's pay-per-crawl approach, which uses the HTTP 402 Payment Required status to tell a client that a fee applies. Other publishers negotiate licensing deals directly. These options are new, uneven and mostly relevant to large content owners, so most small sites can ignore them for now.

What everyone can do is publish clear terms. Robots.txt handles crawling permissions, and newer proposals such as the Content Signals Policy that Cloudflare introduced in 2025 add lines describing how content may be used, for example for search, for AI input or for training. These are declarations of preference, not technical barriers, and support varies. We cover the options in how to create machine-readable policies for AI agents.

Common Mistakes to Avoid

• Blocking with 403 instead of limiting with 429. A 403 tells crawlers the resource is forbidden, and they may drop it. A 429 says "try later."
• Relying on Crawl-delay for Google. It has no effect on Googlebot.
• Rate limiting by user agent only. Spoofing makes it ineffective and can accidentally hit real crawlers sharing a pattern.
• Applying one limit to every bot. Search bots, training bots and user-triggered fetchers deserve different treatment.
• Forgetting policy. Limits manage load. They do not state what you permit. Pair them with a clear statement, which the guide to machine-readable policies for AI agents explains, and with rules about what actions bots may take, covered in AI agent permissions for websites.

Frequently Asked Questions