WordPress Plugins
Free Tools
Claude Skills
Resources
Pricing Support My Account Cart
Support / Royal AI Firewall / Page caches

Keeping blocked bots out of page caches

A page cache answers with a saved copy of the page before WordPress starts. That is what makes it fast, and it also means Royal AI Firewall never sees the request. Without help, a bot you blocked could still be handed any page a visitor already cached. The firewall closes that gap for four popular cache plugins automatically. For other caches, add one rule by hand.

Requires Royal AI Firewall 1.0.12 or later

See the current state under AI Firewall → Settings → Page caches.

Caches the firewall keeps in sync

The firewall writes the user agents of every bot you set to Block, Rate limit or Paywall into each cache's own "do not cache for these user agents" setting. Rate-limited and paywalled bots are included because the firewall stops them in WordPress too.

Cache pluginSetting the firewall updates
WP RocketThe user agents that never get cached pages. They are added through WP Rocket's own filter when its configuration and .htaccess rules are regenerated, so they do not show in its settings field.
W3 Total CacheThe rejected user agents list of its page cache
WP Super CacheRejected user agents, plus a matching rule for its expert (mod_rewrite) mode
LiteSpeed CacheDo Not Cache User Agents

The Page caches section in Settings names the supported caches that are active and how many user agents are listed. If it finds ForgeCache (until it reads the firewall's list), WP Fastest Cache, Hummingbird, Breeze, Cache Enabler or SiteGround Speed Optimizer, it shows Not covered automatically: add the bots' user agents to that cache's exclusion setting yourself.

The firewall's own answers are never cached

A 403, 429, 402 or paywall redirect is sent with no-store headers and a do-not-cache flag, so a cache never saves it and shows it to visitors.

Which user agents to use in a manual rule

Use the name each bot sends in its user agent for every bot you block, rate limit or paywall. Common training crawlers send these:

GPTBot  ClaudeBot  CCBot  Bytespider  meta-externalagent  Amazonbot

Match them case-insensitively, as the examples below do. Keep the list in step with your policies: when you block a new bot in the firewall, add it here too. Developers can add or remove names from the list the firewall writes to the supported caches with the raif_blocked_user_agents filter.

nginx fastcgi_cache

Add a map in the http block, then use it in the location that passes PHP to FastCGI:

# In the http { } block
map $http_user_agent $raif_bot_nocache {
    default 0;
    "~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent)" 1;
}

# In the server's PHP location, next to your other fastcgi_cache lines
fastcgi_cache_bypass $raif_bot_nocache;
fastcgi_no_cache     $raif_bot_nocache;

fastcgi_cache_bypass sends these bots to WordPress instead of the cache, so the firewall can answer them. fastcgi_no_cache stops their responses from being saved. Test and reload:

sudo nginx -t && sudo systemctl reload nginx

On managed hosting you may not have access to nginx settings. Ask your host to add a cache bypass for these user agents.

Varnish

Add this at the top of vcl_recv in your VCL (Varnish 4 or later):

sub vcl_recv {
    if (req.http.User-Agent ~ "(?i)(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent)") {
        return (pass);
    }
    # ... your existing vcl_recv rules
}

return (pass) sends the request straight to WordPress and skips the cache in both directions. Reload Varnish after saving the file.

Cloudflare

By default Cloudflare does not cache HTML pages, so nothing is needed. If you cache HTML with a Cache Everything rule, add a cache rule that bypasses the cache for these bots:

Create a cache rule

In the Cloudflare dashboard, open your domain, go to Cache Rules and create a rule. Choose to edit the expression and paste:

(lower(http.user_agent) contains "gptbot")
or (lower(http.user_agent) contains "claudebot")
or (lower(http.user_agent) contains "ccbot")
or (lower(http.user_agent) contains "bytespider")
or (lower(http.user_agent) contains "meta-externalagent")

Set it to bypass

Set Cache eligibility to Bypass cache, then deploy. Place the rule after your Cache Everything rule so it takes priority.

Cloudflare APO caches pages in its own way. If you use APO, check Cloudflare's APO documentation for bypass options, or block these bots in Cloudflare itself with a WAF custom rule using the same expression.

Check that it works

Load a page once in a private window so it is cached, then request it as a blocked bot:

curl -sI -A "GPTBot/1.2" https://your-site.com/

You should see HTTP/2 403 (or 429, 402 or 302 for rate-limited and paywalled bots) and an x-royal-ai-firewall header. A 200 with a cache hit header means the cache answered before the firewall: check the rule for that layer.

Still stuck? Two support paths

Community support (free): wp.org plugin forum

Post a new thread at wordpress.org/support/plugin/royal-ai-firewall/. The Royal Plugins team monitors the forum regularly.

Premium support (paid)

For direct one-on-one help with a 24-hour SLA, see Premium Support ($149/year).

Information to include (either path)

  • Your Royal AI Firewall version and your cache plugin or server cache
  • What Settings → Page caches shows
  • The full curl -sI output above
  • Whether Cloudflare or another CDN sits in front of the site