ForgeCache detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) at the pre-WordPress drop-in level and serves them a stripped-down markdown twin of your page at roughly 20x smaller byte counts. Human visitors still get the full-fat CSS/JS/lazy-load optimized HTML.
A two-path cache: full-fat HTML for humans, clean markdown for AI crawlers, tier-partitioned so they never collide.
Every cache plugin before ForgeCache 2.1.34 built one variant of a page and served it to every requester. Humans got the CSS/JS optimized version. AI crawlers got the same thing, complete with loading="lazy" images they couldn't hydrate, deferred scripts that were never executed, RUM beacons they'd never fire, and full page chrome (nav, footer, cookie banner, sidebar) diluting the actual content signal.
ForgeCache 2.1.34's Agent Tier splits the cache. Detection happens at the pre-WordPress drop-in level via user-agent match against the current list of AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent, Bytespider, ChatGPT-User, Applebot-Extended, CCBot). Matched requests get routed to a separate cache slot containing the markdown twin of the page: the same content, hydrated as clean semantic markdown, without the human-optimized scaffolding.
The result: your site indexes cleanly in AI answer engines (ChatGPT, Claude, Perplexity, Gemini, Copilot) at roughly 20x smaller crawler bandwidth per page, without changing anything about how humans experience your site.
User-agent match at the pre-WordPress drop-in level. TTFB stays fast for all requesters.
OpenAI's crawler for ChatGPT and the OpenAI API knowledge base.
Anthropic's crawler for Claude, including the web-search tool.
Perplexity's crawler for its answer engine and Comet browser.
Google's opt-in agent for Gemini and Vertex AI training data.
OpenAI's live-fetch agent for user prompts that need web content.
Meta AI's crawler for Llama training and inference-time lookups.
ByteDance's crawler for Doubao and TikTok recommendation models.
Apple's opt-in agent for Apple Intelligence and Siri knowledge.
Common Crawl, the shared corpus that seeds many LLM training runs.
ForgeCache writes and refreshes three markdown surfaces at your site root, so LLM ingesters can find your content the way they expect to.
/llms.txtA curated map of your most important content at your site root, per the llms.txt spec from Answer.AI. Points LLM ingesters at your top posts and pages, with links to their markdown twins.
/page/index.mdA markdown alternate for every cacheable page. Same content as the HTML, hydrated as clean semantic markdown. Fetched directly by crawlers or served automatically when a matching AI user-agent hits the HTML URL.
/llms-full.txtOptional. Bundles your top posts as one concatenated markdown document for RAG (retrieval-augmented generation) pipelines that want a single-file ingest of your site's substantive content.
Detection at the drop-in level. Tier-partitioned cache slots. No changes to the human path.
AI crawlers do not need any of the scaffolding that makes a page feel fast for a human. The markdown twin strips it all out:
loading="lazy" hints on images that crawlers can't hydratedefer/async scripts injecting content that crawlers never executeFour steps, no config file editing, no external services.
Update ForgeCache Pro from WP Admin → Plugins if you are on an earlier version.
Navigate to ForgeCache → Agent Tier in the WordPress admin sidebar.
Enable the master toggle. Optionally opt individual crawlers in or out per-agent.
Visit yoursite.com/llms.txt in your browser. You should see your top posts listed with markdown URLs.
The Agent Tier is a two-path delivery system inside ForgeCache. Human visitors still get the full-fat CSS/JS/lazy-load optimized HTML that ForgeCache has always produced. AI crawlers (identified by user-agent at the pre-WordPress drop-in level) get a stripped-down markdown twin of the same page at roughly 20x smaller byte counts. Both variants live in tier-partitioned cache slots so they don't collide.
The Agent Tier detects the major LLM crawler user-agents: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), Meta-ExternalAgent (Meta AI), Bytespider (ByteDance), ChatGPT-User, Applebot-Extended, and CCBot (Common Crawl). Detection happens at the pre-WordPress drop-in level so TTFB stays fast for all requesters.
llms.txt is an emerging web standard proposed by Answer.AI in 2024 and adopted by Anthropic, Mintlify, Zapier, Cloudflare Docs, and others. It sits at your site root (like robots.txt) and gives LLM ingesters a curated map of your most important content, pointing to markdown alternates when available. ForgeCache generates yours automatically from your published posts and pages, and refreshes it when you publish new content.
Roughly 20x smaller on content-heavy pages. A typical article page might be around 200 KB of optimized HTML (CSS, JS, lazy-load hints, RUM beacon, nav, footer, cookie banner, WebP images) and around 10 KB as a clean markdown twin. Byte savings depend on how much scaffolding your theme adds; sites with heavy chrome see the largest reductions.
No. AI crawler detection runs at the pre-WordPress drop-in level, before ForgeCache's normal cache lookup. Human visitors follow the same fast path they always have; only requests with matching AI-crawler user-agents get routed to the markdown-twin cache slot. TTFB for humans is unchanged.
Yes on both. The Agent Tier admin page has a master toggle and per-agent toggles if you want to allow some crawlers but not others. Individual pages can be opted out of markdown twin generation via a per-page setting or filter. If you disable the Agent Tier entirely, ForgeCache falls back to serving the full HTML variant to all requesters (its behavior pre-2.1.34).
Every ForgeCache Pro license unlocks the Agent Tier plus every other cache and optimization feature. One license, one plugin.
Get ForgeCache Pro