# Keeping blocked bots out of page caches

> Royal AI Firewall keeps blocked bots out of WP Rocket, W3 Total Cache, WP Super Cache and LiteSpeed. Copy-paste rules for nginx, Varnish and Cloudflare.

- Canonical: <https://royalplugins.com/support/royal-ai-firewall/page-caches/>
- Last updated: 2026-11-14
- HTML version: <https://royalplugins.com/support/royal-ai-firewall/page-caches/>

---

A page cache answers with a saved copy of the page before WordPress starts. That is what makes it fast, and it also means Royal AI Firewall never sees the request. Without help, a bot you blocked could still be handed any page a visitor already cached. The firewall closes that gap for four popular cache plugins automatically. For other caches, add one rule by hand.

Requires Royal AI Firewall 1.0.12 or later

See the current state under **AI Firewall → Settings → Page caches**.

## Caches the firewall keeps in sync

The firewall writes the user agents of every bot you set to **Block**, **Rate limit** or **Paywall** into each cache's own "do not cache for these user agents" setting. Rate-limited and paywalled bots are included because the firewall stops them in WordPress too.

| Cache plugin | Setting the firewall updates |
| --- | --- |
| **WP Rocket** | The user agents that never get cached pages. They are added through WP Rocket's own filter when its configuration and `.htaccess` rules are regenerated, so they do not show in its settings field. |
| **W3 Total Cache** | The rejected user agents list of its page cache |
| **WP Super Cache** | Rejected user agents, plus a matching rule for its expert (mod\_rewrite) mode |
| **LiteSpeed Cache** | Do Not Cache User Agents |

- **It stays in sync.** The list is updated when you change a bot's policy, when auto-block adds a bot, when a cache plugin is activated, and on admin page loads if the list changed. A write that fails is retried on the next admin page load.
- **Your own entries are safe.** The firewall replaces only the entries it added. Anything you typed into those settings stays.
- **Deactivating cleans up.** Deactivating Royal AI Firewall takes its entries back out.

The **Page caches** section in Settings names the supported caches that are active and how many user agents are listed. If it finds ForgeCache (until it reads the firewall's list), WP Fastest Cache, Hummingbird, Breeze, Cache Enabler or SiteGround Speed Optimizer, it shows *Not covered automatically*: add the bots' user agents to that cache's exclusion setting yourself.

The firewall's own answers are never cached

A `403`, `429`, `402` or paywall redirect is sent with no-store headers and a do-not-cache flag, so a cache never saves it and shows it to visitors.

## Which user agents to use in a manual rule

Use the name each bot sends in its user agent for every bot you block, rate limit or paywall. Common training crawlers send these:

```
GPTBot  ClaudeBot  CCBot  Bytespider  meta-externalagent  Amazonbot
```

Match them case-insensitively, as the examples below do. Keep the list in step with your policies: when you block a new bot in the firewall, add it here too. Developers can add or remove names from the list the firewall writes to the supported caches with the `raif_blocked_user_agents` filter.

## nginx fastcgi\_cache

Add a `map` in the `http` block, then use it in the location that passes PHP to FastCGI:

```
# In the http { } block
map $http_user_agent $raif_bot_nocache {
    default 0;
    "~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent)" 1;
}

# In the server's PHP location, next to your other fastcgi_cache lines
fastcgi_cache_bypass $raif_bot_nocache;
fastcgi_no_cache     $raif_bot_nocache;
```

`fastcgi_cache_bypass` sends these bots to WordPress instead of the cache, so the firewall can answer them. `fastcgi_no_cache` stops their responses from being saved. Test and reload:

```
sudo nginx -t && sudo systemctl reload nginx
```

On managed hosting you may not have access to nginx settings. Ask your host to add a cache bypass for these user agents.

## Varnish

Add this at the top of `vcl_recv` in your VCL (Varnish 4 or later):

```
sub vcl_recv {
    if (req.http.User-Agent ~ "(?i)(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent)") {
        return (pass);
    }
    # ... your existing vcl_recv rules
}
```

`return (pass)` sends the request straight to WordPress and skips the cache in both directions. Reload Varnish after saving the file.

## Cloudflare

By default Cloudflare does not cache HTML pages, so nothing is needed. If you cache HTML with a Cache Everything rule, add a cache rule that bypasses the cache for these bots:

#### Create a cache rule

In the Cloudflare dashboard, open your domain, go to **Cache Rules** and create a rule. Choose to edit the expression and paste:

```
(lower(http.user_agent) contains "gptbot")
or (lower(http.user_agent) contains "claudebot")
or (lower(http.user_agent) contains "ccbot")
or (lower(http.user_agent) contains "bytespider")
or (lower(http.user_agent) contains "meta-externalagent")
```

#### Set it to bypass

Set **Cache eligibility** to **Bypass cache**, then deploy. Place the rule after your Cache Everything rule so it takes priority.

Cloudflare APO caches pages in its own way. If you use APO, check Cloudflare's APO documentation for bypass options, or block these bots in Cloudflare itself with a WAF custom rule using the same expression.

## Check that it works

Load a page once in a private window so it is cached, then request it as a blocked bot:

```
curl -sI -A "GPTBot/1.2" https://your-site.com/
```

You should see `HTTP/2 403` (or `429`, `402` or `302` for rate-limited and paywalled bots) and an `x-royal-ai-firewall` header. A `200` with a cache hit header means the cache answered before the firewall: check the rule for that layer.

## Still stuck? Two support paths

### Community support (free): wp.org plugin forum

Post a new thread at [wordpress.org/support/plugin/royal-ai-firewall/](https://wordpress.org/support/plugin/royal-ai-firewall/). The Royal Plugins team monitors the forum regularly.

### Premium support (paid)

For direct one-on-one help with a 24-hour SLA, see [Premium Support](https://royalplugins.com/premium-support/) ($149/year).

#### Information to include (either path)

- Your Royal AI Firewall version and your cache plugin or server cache
- What **Settings → Page caches** shows
- The full `curl -sI` output above
- Whether Cloudflare or another CDN sits in front of the site

## Keep reading

[llms.txt, robots.txt and the firewall

Which layer suggests, which layer asks, and which layer actually enforces.](https://royalplugins.com/support/royal-ai-firewall/llms-txt-robots-txt-and-firewall/)
[GuardPress and Royal AI Firewall together

No setup needed. Which plugin runs first and where each one logs.](https://royalplugins.com/support/royal-ai-firewall/guardpress-and-firewall/)
[Dashboard shows zero hits

When a cache keeps bot traffic from reaching WordPress at all.](https://royalplugins.com/support/royal-ai-firewall/#dashboard-shows-zero-hits)

[Royal AI Firewall Documentation](https://royalplugins.com/support/royal-ai-firewall/)
[Premium Support](https://royalplugins.com/premium-support/)
