Agent Tier: two-path delivery for humans and AI agents
ForgeCache detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) at the pre-WordPress drop-in level and serves them a stripped-down markdown twin of your page at roughly 20x smaller byte counts. Human visitors still get the full-fat CSS/JS/lazy-load optimized HTML.
What is the Agent Tier?
A two-path cache: full-fat HTML for humans, clean markdown for AI crawlers, tier-partitioned so they never collide.
Every cache plugin before ForgeCache 2.1.34 built one variant of a page and served it to every requester. Humans got the CSS/JS optimized version. AI crawlers got the same thing, complete with loading="lazy" images they couldn't hydrate, deferred scripts that were never executed, RUM beacons they'd never fire, and full page chrome (nav, footer, cookie banner, sidebar) diluting the actual content signal.
ForgeCache 2.1.34's Agent Tier splits the cache. Detection happens at the pre-WordPress drop-in level via user-agent match against the current list of AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent, Bytespider, ChatGPT-User, Applebot-Extended, CCBot). Matched requests get routed to a separate cache slot containing the markdown twin of the page: the same content, hydrated as clean semantic markdown, without the human-optimized scaffolding.
The result: your site indexes cleanly in AI answer engines (ChatGPT, Claude, Perplexity, Gemini, Copilot) at roughly 20x smaller crawler bandwidth per page, without changing anything about how humans experience your site.
AI crawlers the Agent Tier detects
User-agent match at the pre-WordPress drop-in level. TTFB stays fast for all requesters.
GPTBot
OpenAI's crawler for ChatGPT and the OpenAI API knowledge base.
ClaudeBot
Anthropic's crawler for Claude, including the web-search tool.
PerplexityBot
Perplexity's crawler for its answer engine and Comet browser.
Google-Extended
Google's opt-in agent for Gemini and Vertex AI training data.
ChatGPT-User
OpenAI's live-fetch agent for user prompts that need web content.
Meta-ExternalAgent
Meta AI's crawler for Llama training and inference-time lookups.
Bytespider
ByteDance's crawler for Doubao and TikTok recommendation models.
Applebot-Extended
Apple's opt-in agent for Apple Intelligence and Siri knowledge.
CCBot
Common Crawl, the shared corpus that seeds many LLM training runs.
Three files, auto-generated
ForgeCache writes and refreshes three markdown surfaces at your site root, so LLM ingesters can find your content the way they expect to.
/llms.txt
A curated map of your most important content at your site root, per the llms.txt spec from Answer.AI. Points LLM ingesters at your top posts and pages, with links to their markdown twins.
/page/index.md
A markdown alternate for every cacheable page. Same content as the HTML, hydrated as clean semantic markdown. Fetched directly by crawlers or served automatically when a matching AI user-agent hits the HTML URL.
/llms-full.txt
Optional. Bundles your top posts as one concatenated markdown document for RAG (retrieval-augmented generation) pipelines that want a single-file ingest of your site's substantive content.
Two paths, one cache
Detection at the drop-in level. Tier-partitioned cache slots. No changes to the human path.
What the human path skips
AI crawlers do not need any of the scaffolding that makes a page feel fast for a human. The markdown twin strips it all out:
- No
loading="lazy"hints on images that crawlers can't hydrate - No
defer/asyncscripts injecting content that crawlers never execute - No preconnect / preload hints, RUM beacons, or Web Vitals collector
- No cookie banner, consent scaffolding, or A/B testing frames
- No nav, footer, or sidebar chrome diluting the primary content signal
- Schema.org JSON-LD intact so entity extraction still works cleanly
Request routing
How to enable the Agent Tier
Four steps, no config file editing, no external services.
Update to 2.1.34+
Update ForgeCache Pro from WP Admin → Plugins if you are on an earlier version.
Open Agent Tier
Navigate to ForgeCache → Agent Tier in the WordPress admin sidebar.
Toggle master switch
Enable the master toggle. Optionally opt individual crawlers in or out per-agent.
Verify /llms.txt
Visit yoursite.com/llms.txt in your browser. You should see your top posts listed with markdown URLs.
Common questions
The Agent Tier is a two-path delivery system inside ForgeCache. Human visitors still get the full-fat CSS/JS/lazy-load optimized HTML that ForgeCache has always produced. AI crawlers (identified by user-agent at the pre-WordPress drop-in level) get a stripped-down markdown twin of the same page at roughly 20x smaller byte counts. Both variants live in tier-partitioned cache slots so they don't collide.
The Agent Tier detects the major LLM crawler user-agents: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), Meta-ExternalAgent (Meta AI), Bytespider (ByteDance), ChatGPT-User, Applebot-Extended, and CCBot (Common Crawl). Detection happens at the pre-WordPress drop-in level so TTFB stays fast for all requesters.
llms.txt is an emerging web standard proposed by Answer.AI in 2024 and adopted by Anthropic, Mintlify, Zapier, Cloudflare Docs, and others. It sits at your site root (like robots.txt) and gives LLM ingesters a curated map of your most important content, pointing to markdown alternates when available. ForgeCache generates yours automatically from your published posts and pages, and refreshes it when you publish new content.
Roughly 20x smaller on content-heavy pages. A typical article page might be around 200 KB of optimized HTML (CSS, JS, lazy-load hints, RUM beacon, nav, footer, cookie banner, WebP images) and around 10 KB as a clean markdown twin. Byte savings depend on how much scaffolding your theme adds; sites with heavy chrome see the largest reductions.
No. AI crawler detection runs at the pre-WordPress drop-in level, before ForgeCache's normal cache lookup. Human visitors follow the same fast path they always have; only requests with matching AI-crawler user-agents get routed to the markdown-twin cache slot. TTFB for humans is unchanged.
Yes on both. The Agent Tier admin page has a master toggle and per-agent toggles if you want to allow some crawlers but not others. Individual pages can be opted out of markdown twin generation via a per-page setting or filter. If you disable the Agent Tier entirely, ForgeCache falls back to serving the full HTML variant to all requesters (its behavior pre-2.1.34).
Ship the Agent Tier on your site today
Every ForgeCache Pro license unlocks the Agent Tier plus every other cache and optimization feature. One license, one plugin.
Get ForgeCache Pro