Guides · 2026-11-13 · By VSNARY | Emmanuel Orta
Block AI crawlers in nginx, Apache and Caddy (tested)
Server rules that answer 403 to AI crawlers by name, for nginx, Apache and Caddy, with the test we ran on real servers and the two mistakes that make such rules silently fail.
To block AI crawlers at the server, match the User-Agent header against a list of crawler names and return 403. In nginx use a map in the http block and an if in the server block; in Apache a RewriteCond with the NC flag; in Caddy a header_regexp matcher with an inline (?i) flag. We tested all three on real servers: named crawlers got 403, Googlebot, Bingbot and browsers got 200.
robots.txt asks. A server rule refuses. If you have decided a crawler should not get your pages, and especially if it is one of the fetchers whose operators say they may ignore robots.txt, the server is where that decision takes effect.
The rules below are short examples. The full, current lists are generated from our crawler catalogue and published on the ready-made policies page in three sets: training and corpus crawlers, every AI crawler, and SEO and ad crawlers.
How the rule works #
nginx #
The map goes in the http block and the test in each server block. A map is evaluated once per request and keeps the list in one place.
map $http_user_agent $cc_block {
default 0;
"~*(GPTBot|ClaudeBot|CCBot|Bytespider)" 1;
}
server {
if ($cc_block) { return 403; }
}Apache #
In .htaccess or a virtual host, with mod_rewrite enabled. The NC flag makes the match case-insensitive.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider) [NC]
RewriteRule .* - [F,L]Caddy #
Inside a site block. Caddy's header_regexp is case-sensitive by default, so the pattern carries an inline (?i).
@cc_block header_regexp User-Agent (?i)(GPTBot|ClaudeBot|CCBot|Bytespider)
respond @cc_block 403The test we ran #
On 13 November 2026 we loaded the published "every AI crawler" rule into nginx 1.24, Apache 2.4 and Caddy 2.8 and sent eight requests to each.
| User agent sent | Expected | nginx | Apache | Caddy |
|---|---|---|---|---|
| GPTBot (full string) | 403 | 403 | 403 | 403 |
| chatgpt-user (lower case) | 403 | 403 | 403 | 403 |
| archive.org_bot | 403 | 403 | 403 | 403 |
| meta-externalfetcher | 403 | 403 | 403 | 403 |
| Googlebot | 200 | 200 | 200 | 200 |
| bingbot | 200 | 200 | 200 | 200 |
| a desktop browser | 200 | 200 | 200 | 200 |
| archiveXorg_bot (near miss) | 200 | 200 | 200 | 200 |
The lower-case row checks case handling. The last row checks that the dot in archive.org_bot is escaped: an unescaped dot matches any character and would block names you never listed. We also publish a Cloudflare rule expression; we have not loaded that one into a live zone, so treat it as untested.
Two ways these rules fail quietly #
Case. Crawlers do not always send their name the way the documentation prints it. A case-sensitive match on ChatGPT-User lets chatgpt-user through, and nothing in your logs says the rule missed.
Forged names. A user agent is text anyone can send. A name-based rule stops crawlers that identify themselves; it does nothing about a scraper using a browser's name, and it will block a real person who sends a crawler's name by accident. Where the operator publishes address ranges, they are the stronger check. We mirror them in one file.
Before you block everything #
Blocking training crawlers and blocking answer engines are different decisions. The first changes what future models learn. The second removes you from answers people are reading now. The three published sets keep them apart so you can choose one without the other, and none of them touches classic search crawlers.
CrawlCheck’s dataset when this page loaded: 561,366 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How do I block GPTBot in nginx?
Add a map in the http block that sets a variable to 1 when $http_user_agent matches GPTBot case-insensitively, then return 403 in the server block when that variable is set.
How do I block AI crawlers in Apache?
With mod_rewrite: a RewriteCond on %{HTTP_USER_AGENT} listing the crawler names with the NC flag, followed by RewriteRule .* - [F,L], in .htaccess or a virtual host.
Why does my Caddy rule not block a crawler?
Caddy's header_regexp matcher is case-sensitive. Add an inline (?i) flag at the start of the pattern so it matches the name in any case.
Is blocking by user agent enough?
It stops crawlers that name themselves. A user agent can be forged, so where an operator publishes its address ranges, checking the address is the stronger test.
Will these rules block Googlebot or Bingbot?
No. The published sets list AI training crawlers, AI answer engines, and SEO and ad crawlers. In our test Googlebot, Bingbot and a normal browser all received 200.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.