WordPress Plugins
Free Tools
Claude Skills
Resources
Pricing Support My Account Cart
Support / Royal AI Firewall / llms.txt, robots.txt and the firewall

llms.txt, robots.txt and the firewall: which layer does what

Three things on your site talk to AI bots, and only one of them can make a bot stop. llms.txt suggests what AI tools should read. robots.txt, including its Content-Signal lines, asks bots to follow your rules. Royal AI Firewall enforces your choice inside WordPress. Page caches and CDNs sit in front of all three and decide what reaches WordPress at all.

At a glance

LayerWhat it isCan it stop a bot?
llms.txtA suggestion file for AI toolsNo
robots.txt + Content-SignalA request that well-behaved bots followNo. Bots may ignore it
Royal AI FirewallEnforcement inside WordPressYes: 403, 429, 402 or a redirect
Page cacheAnswers with a saved copy before WordPress runsNo, and it can bypass the firewall
CDN / edgeAnswers or filters before your serverYes, at the network level

llms.txt: a suggestion for AI tools

/llms.txt is a plain Markdown file at the root of your site that points AI tools to the pages you think are most useful, with a short description of each. It is a proposed convention, not a standard. Some tools read it, many do not.

llms.txt does not allow or forbid anything. A bot that ignores it still gets every page, and a bot you block still cannot reach the pages it lists. Royal AI Firewall does not create or read llms.txt. When WordPress serves it (a plugin generates it), the dashboard groups requests for it under the discovery page type so you can see which bots fetch it. A static llms.txt file is served by the web server before WordPress runs, so those requests are not logged.

robots.txt and Content Signals: a request bots may ignore

robots.txt tells crawlers, by user agent, which paths they may crawl. The Content-Signal line adds how they may use what they read:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

Reputable crawlers follow robots.txt. Nothing makes them. A bot that ignores it gets the same pages as everyone else.

How the firewall uses your Content Signals

Royal AI Firewall never edits robots.txt. It reads the Content-Signal lines your site serves (the robots.txt file on the server, or WordPress's own robots.txt including lines other plugins add) and compares each bot's purpose with them. Each bot is compared with the robots.txt group that names it, otherwise the * group. If SEObolt Pro's Content Signals feature is on, its settings are used for every bot instead.

Every logged request is marked as honoring or crossing your signals, and Settings → Content Signals → Signals in effect shows which source is being used. To act on it, turn on Settings → Content Signals → Auto-block repeat violators: every 6 hours the firewall blocks any bot that crossed your signals 3 or more times in the last 24 hours. Major search engines are exempt, and policies you set yourself are never overwritten.

Royal AI Firewall: enforcement inside WordPress

The firewall checks each request at the start of WordPress, before themes and most plugins do their work. When a recognized AI bot arrives, its policy decides the answer:

PolicyWhat the bot gets
BlockA 403 with a short text body
Rate limitNormal pages up to the number of requests per minute you set, then a 429 with a Retry-After header until the minute ends
Paywall (402 / redirect)A 302 redirect to the page you choose (for example your licensing terms), or a 402 when no page is set. The paywall page itself loads normally.
Log only / Always allowThe normal page. The request is recorded.

Blocking also covers anonymous admin-ajax, admin-post and XML-RPC requests from AI bots. Googlebot, Bingbot, Applebot and DuckDuckBot are protected from accidental blocking when their identity is confirmed: by reverse DNS for Google, Bing and Apple, and by user agent for DuckDuckBot, which publishes no reverse-DNS method. These answers are sent with no-cache headers, so a page cache never stores them and shows them to visitors.

Page caches and CDNs sit in front

Page caches

A page cache answers with a saved copy of the page before WordPress starts, so the firewall never sees that request. Royal AI Firewall adds the bots you block, rate limit or paywall to the user-agent exclusion lists of WP Rocket, W3 Total Cache, WP Super Cache and LiteSpeed Cache. Other caches need one rule by hand. See Keeping blocked bots out of page caches.

CDN and edge firewalls

Cloudflare and other CDNs see the request before your server does. They can block at the network level and can verify bot identity there. Anything they block never reaches WordPress, so it does not appear on the firewall dashboard. The two layers do not conflict. If you use Cloudflare, the setup wizard lists which of its AI settings to turn down so the firewall can handle AI bots: see Cloudflare setup.

When to use which

You want to…Use
Point AI tools at your best pagesllms.txt
State your terms publicly, for every bot that reads themrobots.txt with a Content-Signal line
See which AI bots actually visit and what they readRoyal AI Firewall in Log only mode
Stay in AI search but keep content out of trainingai-train=no in robots.txt, plus the default policy Block training crawlers, allow retrieval bots
Make sure a bot gets nothingBlock in the firewall, plus the matching cache rule if your cache is not one of the four the firewall updates
Slow a heavy crawler without cutting it offRate limit
Point bots at your licensing termsPaywall with a redirect page
Stop abusive traffic before it uses server resourcesYour CDN or edge firewall

The layers work best together: robots.txt says what you want, the firewall makes it stick, and your logs show who listened.

Still stuck? Two support paths

Community support (free): wp.org plugin forum

Post a new thread at wordpress.org/support/plugin/royal-ai-firewall/. The Royal Plugins team monitors the forum regularly.

Premium support (paid)

For direct one-on-one help with a 24-hour SLA, see Premium Support ($149/year).

Information to include (either path)

  • Your Royal AI Firewall version
  • The address of your robots.txt (and llms.txt, if you have one)
  • The bot and the policy you set for it
  • Whether a page cache or CDN sits in front of the site