Blocking spoofed AI crawler user agents does not clean your analytics

Credential scanners were hitting my site while claiming to be AI crawlers — /.env, /wp-config.php, the usual list — so I put two Cloudflare WAF rules in front of it. Stop them at the edge and both the abuse and the polluted analytics go away, or so I assumed.

A month of numbers says otherwise. Perplexity-User, which the rules were blocking the entire time, still accounted for 595 requests in an eight-day window — all of them 403s. And the scanning had moved onto Amazonbot and ChatGPT-User: names that real crawlers use.

Here is what I wish I had known before writing those rules.

Blocked requests stay in your analytics

Cloudflare's GraphQL Analytics API (httpRequestsAdaptiveGroups) aggregates the response the edge returned. A WAF rule does not remove a request from the data; it changes its status code to 403.

Perplexity-User makes this visible. Before the rule it reached the origin and got 404s. After the rule it got 403s — and the volume went up, not down.

The rule works. The 404s are gone, so nothing reaches the origin. But if you count "AI crawler traffic" by user agent, those 595 requests are still inside your number.

So: a WAF protects the origin, it does not fix a metric. If you want the contamination out of your reporting, you have to classify it where you collect the data.

The two rules

A static site on Cloudflare Pages, free plan zone. The rules go in through the API from a script that is idempotent — it matches on description, then updates or creates.

Rule one matches on path and ignores the user agent:

SCAN_PATH_TOKENS = [
    "/.env", "/.git", "/.aws", "/.svn", "/.ssh",
    "/wp-", "wp-admin", "wp-login", "wp-includes", "wp-config",
    "wlwmanifest", "xmlrpc.php",
    "secrets.json", "service_account", "/actuator/", "/api/auth/session",
    "phpinfo", "/.npmrc", "/.htpasswd", "id_rsa", "config.json",
]

scan_expr = " or ".join(
    f'(http.request.uri.path contains "{t}")' for t in SCAN_PATH_TOKENS
)

Rule two matches on user agent, gated on Cloudflare not having verified the client — that not cf.client.bot is the part that matters:

SPOOFED_ONLY_UAS = [
    "Bytespider", "cohere-ai", "anthropic-ai", "Perplexity-User", "Google-Extended",
]

ua_expr = "(not cf.client.bot) and (" + " or ".join(
    f'http.user_agent contains "{u}"' for u in SPOOFED_ONLY_UAS
) + ")"

Deploying them is a PUT against the Rulesets API:

API_BASE = "https://api.cloudflare.com/client/v4"
PHASE = "http_request_firewall_custom"

# read current rules -> replace the ones whose description carries my tag -> PUT the whole set
entrypoint = api(token, "GET", f"/zones/{zone_id}/rulesets/phases/{PHASE}/entrypoint")
rules = entrypoint["result"]["rules"]
# ... swap in the new definitions, keep everything else untouched ...
api(token, "PUT", f"/zones/{zone_id}/rulesets/{ruleset_id}", {"rules": rules})

The token needs Zone/WAF/Edit plus Zone/Zone/Read. An analytics-read token returns 403 here, so mint a separate one.

Gotcha 1: starts_with only matches the root

The zone already had a hand-written rule:

starts_with(http.request.uri.path, "/.env")

That catches /.env and nothing else. The requests actually arriving were /server/.env (18 of them), /admin/.env and /app/.git/HEAD, so the rule existed and the scanning continued anyway. contains catches the nested ones.

http.request.uri.path is evaluated after normalization, so %2e-style encoding tricks do not get around it either.

Gotcha 2: a path blocklist is always behind

The token list stops exactly what you wrote down. My 08-28 snapshot still had 300 requests from AI-claiming UAs ending in 404, hitting paths that are not in the list:

• /secrets.yml — I wrote secrets.json, not .yml
• /id_ecdsa — I wrote id_rsa, not id_ecdsa
• /.zshrc
• /@fs/proc/self/environ

config.json started as an exact eq "/config.json" match and got walked around with /runtime-config.json and /api/runtime-config.json. Before widening a token to contains, check that no real path in your sitemap contains that string — otherwise you start blocking things you actually serve.

The blocked name was kept, and real names were added

This was the finding that changed how I write these rules. Scan-classified requests, grouped by the user agent they claimed:

The blocked name stayed in use and unblocked names were layered on top of it. The paths did not diversify — the same scanning simply arrived under more names.

The problem is that Amazonbot and ChatGPT-User are names real crawlers send. Same eight days, split by verification:

Match those strings in a block rule and you also drop the genuine ChatGPT-User that fetched 216 articles. A blocklist keyed on user agent expires the moment the impersonation moves to a name you need.

Names that cannot exist are the opposite case and stay blockable forever. Google's crawler documentation states that Google-Extended has no HTTP request user agent at all — it is a robots.txt control token — so every request claiming it is fake by definition. anthropic-ai and cohere-ai get the same treatment.

Takeaways

• Blocking at the edge leaves the request in your analytics as a 403. A block is not a measurement fix.
• User agents that cannot exist (Google-Extended, anthropic-ai) are safe to match on the string.
• User agents shared with real crawlers (ChatGPT-User, Amazonbot) are not. Move that decision onto the verification result — cf.client.bot in a rule, verifiedBotCategory in your reporting.
• Write path rules with contains, and check new tokens against your sitemap before widening them.

How I now read a drop in that series (scans collapsed in a week where I changed no rules, while zone-wide 403s went up), and how blocking and measurement ended up in separate layers, are on Aulvem → Aulvem | Blocking spoofed AI user agents did not clean my analytics — a month of numbers