The Edge Cache Partition: RFC 9110, Markdown Negotiation, and Preventing Agent Poisoning

🤖 Read Raw Markdown • 📄 Google Doc (Try: Tools/Audio/Listen to document summary)

Setting the Stage: Context for the Curious Book Reader

When an autonomous AI agent crawls a webpage, feeding it raw HTML wastes up to ninety percent of its context window on layout wrappers, tracking scripts, and styling bloat. The obvious technical answer is HTTP content negotiation under RFC 9110: allow synthetic clients to ask for pure Markdown while humans continue to receive rendered HTML from the exact same URI. Yet the moment you attempt this in production, you slam directly into RFC 9111 and the brittle mechanics of edge caching. Without strict cache partitioning and header normalization, shared CDNs will either poison human browser sessions with raw text or serve bloated HTML to token-constrained models. What begins as a quest for AI-readiness rapidly mirrors the decade-long migration to Responsive Web Design, cycling through temporary edge proxies and tactical fragment splicing before settling into native origin architecture.

Technical Journal Entry Begins

MikeLev.in: Tell me about RFC 9110 and 9111. When did they come out? Which one deals with Content Negotiation? How do they relate to RFC 7231? Which one are people going to be focusing on most for AI-readiness and why?

Gemini 3.8 Flash High:

Overview and Release Date

RFC 9110 (HTTP Semantics) and RFC 9111 (HTTP Caching) were both published in June 2022 by the Internet Engineering Task Force (IETF).

They represent a major multi-year effort to modernize and clean up core HTTP standards:

  • RFC 9110 defines the core architecture, concepts, request methods (GET, POST, etc.), status codes, representation metadata, and content negotiation that apply universally across all versions of HTTP—including HTTP/1.1, HTTP/2, and HTTP/3.
  • RFC 9111 defines HTTP caching behavior. It standardizes how shared caches (CDNs, forward/reverse proxies) and private caches (browsers) store, validate, invalidate, and serve responses using mechanisms like Cache-Control, Age, ETag, and conditional revalidation.

Which One Deals with Content Negotiation?

RFC 9110 handles Content Negotiation, detailed primarily in Section 12.

It outlines both:

  1. Proactive (Server-Driven) Negotiation (§12.1): Where the user agent advertises its preferences and processing capabilities via request headers (Accept, Accept-Language, Accept-Encoding, and weighted quality values like q=0.9), and the origin server chooses the most suitable representation.
  2. Reactive (Agent-Driven) Negotiation (§12.2): Where the server offers multiple representations (typically via a 300 Multiple Choices status or alternative links), and the client selects which one to fetch.

RFC 9111 touches negotiation only downstream—governing how caches handle negotiated variants using the Vary header.

Relationship to RFC 7231

RFC 9110 directly obsoletes RFC 7231.

In 2014, the HTTP specification was published as the RFC 7230–7235 series. RFC 7231 was titled Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. That grouping created architectural friction: semantics were explicitly tethered to HTTP/1.1 framing in title and structure, even as HTTP/2 (2015) and HTTP/3 (2022) were developed.

The 2022 reorganization disentangled the protocol layers:

                     RFC 9110: HTTP Semantics
                    (Absorbed RFC 7231, 7232, 7233, 7235)
                                  │
         ┌────────────────────────┼────────────────────────┐
         ▼                        ▼                        ▼
RFC 9112: HTTP/1.1       RFC 9113: HTTP/2         RFC 9114: HTTP/3
(Framing & Wire Syntax)  (Frames, Multiplexing)   (QUIC Transport)
  • Consolidation into RFC 9110: RFC 9110 absorbed RFC 7231 (semantics/content), RFC 7232 (conditional requests), RFC 7233 (range requests), and RFC 7235 (authentication) into a single, transport-independent document.
  • Decoupling Caching into RFC 9111: The caching specifications from RFC 7234 were extracted, updated, and consolidated into RFC 9111.

The Protocol Divide: HTTP Semantics Versus Edge Caching

Which One Are People Focusing on Most for AI-Readiness, and Why?

RFC 9110 is the headline focus, but RFC 9111 is the operational requirement that keeps it from breaking production.

1. Why RFC 9110 Leads: Token Efficiency via Content Negotiation

Autonomous agents, retrieval-augmented generation (RAG) systems, and LLM web scrapers struggle with standard web delivery. Delivering raw HTML to an LLM context window means 80% to 90% of the consumed tokens are consumed by navigational boilerplate, Tailwind classes, JSON-LD fragments, tracking scripts, and nested <div> wrappers.

Instead of building brittle scraper extractors or relying on separate, non-canonical API endpoints, architects are leveraging RFC 9110 Proactive Content Negotiation:

  • An LLM agent sends: 
    GET /insights/architecture HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.1
  • The server responds with pure, markdown-formatted text representing the core content (Content-Type: text/markdown), stripping all UI layout chrome.
  • The payoff: Lower token cost, smaller payloads, reduced latency to first generation, and preservation of canonical URIs and permalinks across both human and synthetic traffic.

2. The Operational Trap: Why RFC 9111 Cannot Be Ignored

Deploying RFC 9110 content negotiation immediately creates an edge caching problem defined by RFC 9111.

If a browser requests [https://example.com/article](https://example.com/article) with Accept: text/html and an AI agent later requests the same URL with Accept: text/markdown, an intermediary CDN operating under RFC 9111 rules will return the cached HTML version to the AI agent—or poison the browser cache with raw Markdown—unless the origin server strictly implements RFC 9110 §12.5.5 and RFC 9111 cache partitioning:

Vary: Accept
Cache-Control: public, max-age=3600

Without Vary: Accept, multi-representation edge architectures break. Furthermore, due to the volume of agentic crawling, properly setting conditional request validators (ETag, If-None-Match, and 304 responses under RFC 9110/9111) is the only way to keep LLM bot egress bandwidth and database query costs sustainable.

Implementing Normalized Content Negotiation Across Edge Platforms

MikeLev.in: Break down how to implement Accept: text/markdown negotiation with Nginx, Cloudflare, Akamai and Fastly please and thank you.

Gemini 3.8 Flash High: To safely serve both HTML (to browsers) and Markdown (to LLMs) from identical URIs under RFC 9110 and RFC 9111, you must avoid keying on the raw Accept header. Browsers send dozens of permutations of Accept: text/html,application/xhtml+xml,..., which shatters cache hit ratios and causes cache poisoning if a raw string is used.

The standard architecture across all platforms follows a three-step normalization pattern:

  1. Inspect & Normalize: Classify incoming Accept headers into a discrete binary enum (markdown vs. html).
  2. Partition the Cache Key: Include only that normalized value in the edge cache key.
  3. Emit RFC 9111 Headers: Ensure downstream caches see Vary: Accept alongside standard cache directives.

1. Nginx

In Nginx, an upstream reverse proxy or local static server uses the map directive to normalize the incoming header, then binds the resulting variable into proxy_cache_key.

# 1. Normalize the raw Accept header into a binary variant
map $http_accept $content_variant {
    default                  "html";
    "~*text/markdown"        "markdown";
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=API_CACHE:10m max_size=1g inactive=60m;

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://origin_upstream;

        # 2. Partition cache key by method, host, URI, and the normalized variant
        proxy_cache API_CACHE;
        proxy_cache_key "$request_method$scheme$host$request_uri$content_variant";
        proxy_cache_valid 200 302 10m;

        # 3. Pass canonical Accept header to origin to preserve upstream caching
        proxy_set_header Accept $content_variant;

        # 4. Enforce RFC 9110/9111 downstream compliance
        add_header Vary "Accept" always;
        add_header X-Cache-Status $upstream_cache_status;
    }
}

If serving pre-rendered static files (.html and .md) directly from disk, swap proxy_pass for try_files $uri.$content_variant $uri/index.$content_variant =404;.

2. Cloudflare

Cloudflare’s edge cache does not vary on arbitrary headers like Vary: Accept out of the box (it only honors Vary: Accept-Encoding by default). Handling content negotiation requires either Cache Rules or a Cloudflare Worker.

Method A: Cache Rules (Declarative / Dashboard)

  1. Navigate to Caching > Cache Rules > Create Rule.
  2. Under Cache Key, enable Custom Cache Key.
  3. Under Headers, add Accept.
  4. Set the Normalization setting for Accept with media_types: ["text/markdown", "text/html"]. Cloudflare will normalize all browser permutations to text/html and bot scrapers to text/markdown, maintaining two clean cache slots.

Method B: Cloudflare Workers (Programmable)

If you require dynamic origin synthesis or programmatic routing, partition via the Cache API using a synthetic cache key:

export default {
  async fetch(request, env, ctx) {
    const accept = request.headers.get("Accept") || "";
    const isMarkdown = accept.includes("text/markdown");
    const variant = isMarkdown ? "markdown" : "html";

    // 1. Create a synthetic cache key URL to partition Cloudflare edge cache
    const cacheUrl = new URL(request.url);
    cacheUrl.searchParams.set("__cf_variant", variant);
    const cacheKey = new Request(cacheUrl.toString(), request);

    const cache = caches.default;
    let response = await cache.match(cacheKey);

    if (!response) {
      // 2. Rewrite the header sent to the origin
      const originRequest = new Request(request);
      originRequest.headers.set("Accept", isMarkdown ? "text/markdown" : "text/html");

      response = await fetch(originRequest);

      // 3. Ensure response carries RFC 9111 Vary header downstream
      const mutatedHeaders = new Headers(response.headers);
      mutatedHeaders.set("Vary", "Accept");

      response = new Response(response.body, {
        status: response.status,
        statusText: response.statusText,
        headers: mutatedHeaders
      });

      // 4. Store in edge cache if cacheable
      if (response.ok && response.headers.get("Cache-Control")?.includes("max-age")) {
        ctx.waitUntil(cache.put(cacheKey, response.clone()));
      }
    }

    return response;
  }
};

3. Fastly (VCL)

Fastly executes Varnish Enterprise. While Varnish understands Vary, leaving Accept unnormalized causes cache fragmentation. Normalize in vcl_recv and append the normalized value to vcl_hash:

sub vcl_recv {
  #FASTLY recv

  # 1. Normalize Accept into a clean internal header
  if (req.http.Accept ~ "text/markdown") {
    set req.http.X-Normalized-Format = "markdown";
    set req.http.Accept = "text/markdown";
  } else {
    set req.http.X-Normalized-Format = "html";
    set req.http.Accept = "text/html";
  }
}

sub vcl_hash {
  #FASTLY hash

  # 2. Partition cache key by the normalized format
  set req.hash += req.http.X-Normalized-Format;
}

sub vcl_fetch {
  #FASTLY fetch

  # 3. Guarantee Vary: Accept is present in stored object
  if (beresp.http.Vary) {
    if (beresp.http.Vary !~ "(?i)Accept") {
      set beresp.http.Vary = beresp.http.Vary ", Accept";
    }
  } else {
    set beresp.http.Vary = "Accept";
  }
}

sub vcl_deliver {
  #FASTLY deliver

  # 4. Transmit Vary: Accept downstream to client caches
  if (resp.http.Vary !~ "(?i)Accept") {
    set resp.http.Vary = resp.http.Vary ", Accept";
  }
}

4. Akamai

Akamai edge servers treat responses containing non-compression Vary headers as uncacheable by default or trigger “Remove Vary Header” warnings. To make content negotiation cacheable on Akamai, use Property Manager rules with Cache ID Modification or EdgeWorkers.

Method A: Property Manager Rules

  1. Rule Criteria:
    • Match Request Header: Accept matches wildcards *text/markdown*.
  2. Behavior — Set Variable:
    • If matched: set user variable PMUSER_FORMAT = "markdown".
    • Else: set PMUSER_FORMAT = "html".
  3. Behavior — Cache ID Modification:
    • Under Include Elements, add Variable: PMUSER_FORMAT.
    • This isolates the cache space for each format under the same URL path.
  4. Behavior — Modify Outgoing Request Header:
    • Set Accept = %(PMUSER_FORMAT).
  5. Behavior — Remove Vary Header:
    • Enable Remove Vary Header for upstream origin responses so Akamai does not discard the cache entry, but enable Downstream Cacheability with a custom header to send Vary: Accept to the client.

Method B: Akamai EdgeWorkers (onClientRequest)

For code-based routing, modify the cache key directly via the EdgeWorkers API:

export function onClientRequest(request) {
  const rawAccept = request.getHeader('Accept')?.[0] || '';
  const isMarkdown = rawAccept.includes('text/markdown');
  const format = isMarkdown ? 'markdown' : 'html';

  // 1. Partition Akamai's True Cache Key
  request.cacheKey.includeVariable('PMUSER_FORMAT', format);

  // 2. Normalize header forwarded to origin
  request.setHeader('Accept', isMarkdown ? 'text/markdown' : 'text/html');
}

Platform Comparison Matrix

Platform Native Vary: Accept Support Recommended Partitioning Mechanism Edge Normalization Point
Nginx Yes (proxy_cache) proxy_cache_key with variable map $http_accept block
Cloudflare No (requires rule/worker) Custom Cache Key / Synthetic Request Cache Rule media allowlist or Worker
Fastly Yes (native Varnish) vcl_hash with normalized header vcl_recv regex matching
Akamai No (ignores/drops by default) Cache ID Modification / Variable Property Manager Variable or EdgeWorker

Verification and Testing

Verify deployment using curl against the same URI to verify both cache partitioning and correct header reflection:

# 1. Test Agent Path (Expect text/markdown, Vary: Accept)
curl -i -H "Accept: text/markdown" https://example.com/docs/intro

# 2. Test Browser Path (Expect text/html, Vary: Accept)
curl -i -H "Accept: text/html,application/xhtml+xml" https://example.com/docs/intro

Check the headers on both responses:

  • Content-Type must strictly match the respective request.
  • Vary must include Accept.
  • Second runs of both commands must return cache hits (HIT) without altering each other’s payloads.

Verifiable Edge Testing and Cache Poisoning Audits

MikeLev.in: Write an automated bash or Python test script that verifies cache isolation between text/markdown and text/html and checks for CDN cache poisoning.

Gemini 3.8 Flash High: This standalone Python script tests for edge cache isolation and detects CDN cache poisoning between text/markdown and text/html. It uses Python’s standard library (urllib.request), requiring no external dependencies like requests.

Save as test_cache_isolation.py:

#!/usr/bin/env python3
"""
RFC 9110 / RFC 9111 Edge Cache Isolation & Poisoning Test Suite.
Verifies that multi-representation endpoints safely partition HTML and Markdown.
"""

import argparse
import sys
import time
import uuid
import urllib.request
import urllib.error

# ANSI color formatting
GREEN = "\033[92m"
RED = "\033[91m"
YELLOW = "\033[93m"
CYAN = "\033[96m"
BOLD = "\033[1m"
RESET = "\033[0m"

BROWSER_ACCEPT = (
    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
)
MARKDOWN_ACCEPT = "text/markdown, text/plain;q=0.5"

class CacheTestRunner:
    def __init__(self, base_url: str, verbose: bool = False):
        self.base_url = base_url
        self.verbose = verbose
        self.failures = []

    def _execute_request(self, url: str, accept_header: str) -> dict:
        req = urllib.request.Request(
            url,
            headers={
                "Accept": accept_header,
                "User-Agent": "RFC-Cache-Audit/1.0",
                # Akamai Edge diagnostic headers (if enabled on property)
                "Pragma": "akamai-x-cache-on, akamai-x-get-cache-key",
            },
        )
        start_time = time.time()
        try:
            with urllib.request.urlopen(req, timeout=10) as resp:
                elapsed_ms = (time.time() - start_time) * 1000
                headers = {k.lower(): v for k, v in resp.headers.items()}
                body = resp.read(2048).decode("utf-8", errors="replace")
                return {
                    "status": resp.status,
                    "headers": headers,
                    "body_preview": body[:200].replace("\n", " "),
                    "elapsed_ms": elapsed_ms,
                    "error": None,
                }
        except urllib.error.HTTPError as e:
            elapsed_ms = (time.time() - start_time) * 1000
            headers = {k.lower(): v for k, v in e.headers.items()}
            return {
                "status": e.code,
                "headers": headers,
                "body_preview": "",
                "elapsed_ms": elapsed_ms,
                "error": str(e),
            }

    def _extract_cache_status(self, headers: dict) -> str:
        """Inspects vendor-specific headers for edge cache state."""
        # Cloudflare
        if "cf-cache-status" in headers:
            return f"Cloudflare: {headers['cf-cache-status']}"
        # Fastly / Generic Varnish
        if "x-cache" in headers:
            return f"X-Cache: {headers['x-cache']}"
        # Nginx (standard proxy_cache custom header)
        if "x-cache-status" in headers:
            return f"Nginx: {headers['x-cache-status']}"
        # Standard RFC 9111 Age indicator
        if "age" in headers and int(headers.get("age", 0)) > 0:
            return f"RFC 9111 Age: {headers['age']}s"
        return "UNKNOWN / MISS (No edge header)"

    def _assert_response(
        self,
        phase: str,
        result: dict,
        expected_type: str,
        forbid_type: str,
    ):
        content_type = result["headers"].get("content-type", "").lower()
        vary_header = result["headers"].get("vary", "").lower()
        cache_state = self._extract_cache_status(result["headers"])

        print(f"\n{BOLD}[{phase}]{RESET}")
        print(f"  HTTP Status  : {result['status']} ({result['elapsed_ms']:.1f}ms)")
        print(f"  Content-Type : {content_type}")
        print(f"  Vary         : {vary_header or 'MISSING'}")
        print(f"  Cache State  : {cache_state}")

        if self.verbose:
            print(f"  Body Preview : {result['body_preview'][:100]}...")

        # 1. Cache Poisoning Detection
        if forbid_type in content_type:
            msg = (
                f"CRITICAL: Cache Poisoning Detected in {phase}! "
                f"Expected '{expected_type}', but received poisoned '{forbid_type}'."
            )
            print(f"  {RED}{BOLD}✖ {msg}{RESET}")
            self.failures.append(msg)
            return

        if expected_type not in content_type:
            msg = f"Failed {phase}: Expected Content-Type containing '{expected_type}', got '{content_type}'."
            print(f"  {RED}✖ {msg}{RESET}")
            self.failures.append(msg)
        else:
            print(f"  {GREEN}✔ Correct representation received ({expected_type}){RESET}")

        # 2. RFC 9110 / 9111 Vary Compliance
        if "accept" not in vary_header:
            msg = f"Warning in {phase}: Response is missing 'Vary: Accept' (RFC 9110 §12.5.5 violation)."
            print(f"  {YELLOW}▲ {msg}{RESET}")
            self.failures.append(msg)
        else:
            print(f"  {GREEN}✔ 'Vary: Accept' header present{RESET}")

    def run(self):
        # Cache buster ensures execution against clean edge cache slots
        separator = "&" if "?" in self.base_url else "?"
        run_id = uuid.uuid4().hex[:8]
        test_url = f"{self.base_url}{separator}__audit_id={run_id}"

        print(f"{CYAN}{BOLD}Starting Cache Isolation Audit:{RESET} {test_url}")

        # Sequence 1: Prime HTML Cache, then request Markdown (HTML -> Markdown)
        print(f"\n{BOLD}=== Test Cycle A: HTML Prime -> Markdown Probe ==={RESET}")
        res_html_prime = self._execute_request(test_url, BROWSER_ACCEPT)
        self._assert_response("Phase 1: Prime HTML Cache", res_html_prime, "text/html", "text/markdown")

        res_md_probe = self._execute_request(test_url, MARKDOWN_ACCEPT)
        self._assert_response(
            "Phase 2: Markdown Request (Post-HTML Prime)", res_md_probe, "text/markdown", "text/html"
        )

        res_html_hit = self._execute_request(test_url, BROWSER_ACCEPT)
        self._assert_response("Phase 3: HTML Cache Revalidation", res_html_hit, "text/html", "text/markdown")

        # Sequence 2: Inverted Prime (Markdown Prime -> HTML Probe)
        run_id_inv = uuid.uuid4().hex[:8]
        test_url_inv = f"{self.base_url}{separator}__audit_id={run_id_inv}"

        print(f"\n{BOLD}=== Test Cycle B: Markdown Prime -> HTML Probe ==={RESET}")
        res_md_prime = self._execute_request(test_url_inv, MARKDOWN_ACCEPT)
        self._assert_response("Phase 4: Prime Markdown Cache", res_md_prime, "text/markdown", "text/html")

        res_html_probe = self._execute_request(test_url_inv, BROWSER_ACCEPT)
        self._assert_response(
            "Phase 5: HTML Request (Post-Markdown Prime)", res_html_probe, "text/html", "text/markdown"
        )

        # Audit Summary
        print(f"\n{BOLD}=== Audit Results ==={RESET}")
        if not self.failures:
            print(f"{GREEN}{BOLD}PASS: Cache isolation verified. No poisoning detected.{RESET}\n")
            sys.exit(0)
        else:
            print(f"{RED}{BOLD}FAIL: {len(self.failures)} violation(s) identified:{RESET}")
            for err in self.failures:
                print(f" - {err}")
            print()
            sys.exit(1)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Verify RFC 9110/9111 content negotiation edge cache partitioning."
    )
    parser.add_argument("url", help="URL of the negotiated endpoint (e.g., https://example.com/page)")
    parser.add_argument("-v", "--verbose", action="store_true", help="Print response body excerpts")
    args = parser.parse_args()

    runner = CacheTestRunner(args.url, args.verbose)
    runner.run()

How to Run the Script

Make the script executable and test your endpoint:

chmod +x test_cache_isolation.py

# Run against your production or staging origin
./test_cache_isolation.py https://example.com/docs/intro -v

What the Test Validates

  1. Test Cycle A (HTML Prime $\rightarrow$ Markdown Request):
    • Phase 1: Sends browser Accept: text/html,... to populate the HTML slot.
    • Phase 2: Requests Accept: text/markdown at the identical URL. If the CDN returns text/html, cache poisoning is flagged immediately.
    • Phase 3: Re-requests HTML to confirm the edge cache serves a cache HIT without being corrupted by the markdown request.
  2. Test Cycle B (Markdown Prime $\rightarrow$ HTML Request):
    • Tests the inverted path using a fresh, cache-busted query parameter.
    • Phase 4: Primes the edge cache with Accept: text/markdown.
    • Phase 5: Sends browser Accept headers to ensure humans receive rendered HTML, preventing markdown leakage into browsers.
  3. RFC Header Compliance:
    • Checks that Vary includes Accept on both variants (required under RFC 9110 §12.5.5 and RFC 9111).
    • Displays the edge provider cache hit indicators across Cloudflare, Fastly, Nginx, and Akamai.

CI/CD Integration Example (GitHub Actions)

Add this test directly into deployment pipelines to block CDN cache misconfigurations before they hit production:

name: Verify Edge Cache Isolation

on:
  deployment_status:

jobs:
  cache-audit:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Run Cache Partitioning Audit
        run: |
          python3 test_cache_isolation.py "${{ github.event.deployment_status.target_url }}"

MikeLev.in: Would you call these implementations heavy-weight or lightweight. What level of qualifications, prerequisites, certifications, general experience or otherwise must someone poses to implement RFC 9110 per the RFC 9111 details?

Would you say it’s an hour’s worth of work?

A day’s worth of work?

A week’s worth of work?

A month’s worth of work?

How big of a team would you estimate is required to carry out such an implementation? What sub-specialties are necessary in people on the team?

Does it get more complex when there are large amounts of real-time price changes on the order of tens of thousands or even hundreds of thousands per day?

What questions should I be asking here given the pattern of my questions but have not asked, and what are the answers to those?

Gemini 3.8 Flash High:

Lightweight or Heavyweight?

The code is lightweight; the operational blast radius is heavyweight.

  • The Code (Lightweight): The configuration itself is 20 to 50 lines of configuration (Nginx directives, Fastly VCL snippets, or Cloudflare Worker/EdgeWorker JavaScript). Syntactically, it is straightforward.
  • The Blast Radius (Heavyweight): Caching is stateful across distributed edge nodes. A one-character syntax mistake in proxy_cache_key, an unnormalized Accept header, or a missing Vary: Accept header will:
    1. Poison human browser sessions: Desktop visitors receive raw, unrendered Markdown files instead of CSS/HTML layouts.
    2. Crush origin servers: Unnormalized Accept headers fragment the edge cache into hundreds of microscopic buckets per URL, dropping edge hit rates from 95% down to 10%.
    3. Break SEO indexation: If a search crawler receives a non-HTML representation unexpectedly, indexation drops.

Prerequisites, Experience, and Qualifications

No specific paper certifications are required, but the implementer must operate at a Senior Infrastructure, CDN/Edge, or Senior Backend Engineer level.

Prerequisite Knowledge:

  • RFC Semantics & Wire Protocol: Deep understanding of HTTP state machines, representation metadata, conditional requests (ETag, If-None-Match, 304 Not Modified), and the interaction between origin headers and intermediate shared proxies.
  • Platform-Specific Edge Runtime Internals: Understanding how specific CDNs handle cache keys (e.g., Cloudflare ignoring Vary by default, Akamai stripping Vary unless configured with custom Cache IDs, Fastly VCL hash manipulation).
  • Defensive Edge Testing: Ability to write automated synthetic test harnesses to simulate race conditions, cache warming, and invalidation cycles before deployment.

Time Estimates

  • An Hour’s Work: Only for an edge engineer adding a single declarative Cache Rule in a Cloudflare staging environment for a read-only blog.
  • A Day’s Work (1–2 days): The standard timeframe for writing the edge routing configuration, deploying it to a staging environment, and running automated test suites (like the Python audit script) against both variants.
  • A Week’s Work (1–2 weeks): The realistic enterprise timeline. This includes:
  • Updating origin render pipelines to reliably spit out clean Markdown alongside HTML.
  • Staging integration and edge regression testing.
  • Canary deployment across global CDN points of presence (PoPs).

  • Setting up observability dashboards (splitting edge metrics by representation).

  • A Month’s Work: Only occurs if the backend CMS or monolithic legacy architecture has no decoupled data layer, forcing engineers to rebuild the rendering pipeline from scratch to generate Markdown.

Team Size and Required Sub-Specialties

This does not require a large team—typically 1 to 3 engineers:

  1. Edge/CDN Systems Engineer (DevOps / SRE): Configures proxy caching, cache key normalizations, CDN header policies, and edge runtimes.
  2. Backend / API Engineer: Implements the origin representation logic (ensuring Accept: text/markdown pulls from pure markdown templates or headless CMS fields, rather than doing hacky runtime HTML scraping).
  3. Technical SEO / Web Architect: Audits crawler behavior, canonical link preservation, and status code consistency to ensure search crawlers are not inadvertently misdirected.

Complexity Under High-Frequency Real-Time Price Changes (10k–100k+ / day)

Yes, the complexity increases significantly.

Under real-time price volatility, TTL-based caching (max-age=3600) is unusable because stale prices create legal and transactional liabilities. However, relying on max-age=0 destroys CDN offload.

To maintain content negotiation with 100,000 price mutations per day:

1. Dual Invalidation via Surrogate Keys / Cache Tags

Every price update must invalidate two representations simultaneously: the HTML page and the Markdown document.

  • Do not purge by URL alone if your CDN treats variants as distinct entities.
  • The origin must emit Surrogate Keys / Cache-Tags on both responses: 
    Cache-Tag: product-98124 sku-active store-us
  • When the pricing engine writes an update, it emits an event to the CDN API to purge product-98124, clearing both representations from the edge in under 150 milliseconds globally.

2. Origin Compute Spikes on Cache Misses

If an AI agent and a human both request an invalidated product at the exact same moment, the origin receives two concurrent cache misses for different formats.

  • Implement Origin Shielding (Fastly Shielding, Cloudflare Tiered Cache, Akamai Parent servers) and Request Collapsing (proxy_cache_use_stale updating in Nginx) so simultaneous misses for the same variant coalesce into a single backend fetch.

3. Client-Side Decoupling vs. Static Inlining

If HTML pages update prices dynamically via client-side JavaScript (fetch('/api/v1/price')), the HTML shell itself can stay cached for long periods. But an LLM requesting Markdown expects the price baked directly into the document:

  • The Trap: Markdown misses will bypass the CDN and hit your database directly while HTML hits the edge cache.
  • The Fix: The Markdown edge worker or origin template must pull from a low-latency key-value store (e.g., Redis, Cloudflare KV) to inline the current price without running full database queries.

Questions You Haven’t Asked (And Their Answers)

1. “Why not just use a .md extension or a query parameter like ?format=md instead of HTTP Content Negotiation?”

  • The Answer: Using [https://example.com/item.md](https://example.com/item.md) or ?format=md is far easier to operate and completely avoids edge cache-poisoning risks because every representation has a unique URI.
  • Why People Choose Content Negotiation Anyway:
  • Canonical Uniformity: Clean URIs adhere to REST principles (one URI represents the abstract resource; the headers represent its serialized form).
  • Agent Simplicity: Autonomous AI agents simply navigate standard hyperlinks found across the web without having to rewrite URI schemas or guess file extensions.

2. “How does Googlebot react to an endpoint negotiating text/markdown?”

  • The Answer: Googlebot’s standard request headers include */* or standard HTML Accept lists. However, Googlebot evaluates rendering through headless Chromium. If your edge detection logic misidentifies Googlebot as an AI bot and returns Markdown, Google will index the raw Markdown text. While Google can index Markdown, you lose structured microdata (JSON-LD), internal DOM link parsing, and layout cues.
  • The Rule: Strictly negotiate on the Accept header value, never solely on User-Agent. If an incoming request sends Accept: */*, always default to text/html.

3. “Where should Markdown generation happen: at the Origin or at the Edge?”

  • The Answer: It belongs at the Origin.
  • Some teams attempt to intercept HTML responses at the edge and run regex/DOM parsers (like Turndown or Readability) inside a Cloudflare Worker or Fastly Compute service. This introduces heavy CPU usage, latency overhead, edge compute costs, and frequently breaks tables, code blocks, or structured attributes. Delivering Markdown natively from the origin’s data layer is faster and preserves structured accuracy.

4. “How do we handle Conditional Requests (ETag / If-None-Match) across two representations of the same resource?”

  • The Answer: An HTML document and a Markdown document derived from the same data must not share the same ETag.
  • Under RFC 9110 §8.8.3, strong entity tags must be unique to the specific representation format. If the HTML and Markdown responses share an identical ETag, an AI agent revalidating a cached Markdown file with If-None-Match: "abc" could receive a 304 Not Modified from a cache slot that just updated its HTML formatting. ETags should be scoped by format: e.g., ETag: "abc-html" and ETag: "abc-md".

MikeLev.in: Is there any reason to remain with a pre-rendering scheme? What kind of pre-rendering schemes are out there? How do they work in principle that’s different from this? Are they scheduled? Can they do better than scheduling it? What about real-time price updates with pre-renders? Can you use HTML re-writes for that even after a pre-render is cached targeting updating just particular parts of the page through addressing? Would that have to be scheduled too? Are there options here to do better than scheduling?

How do such solutions compare to those suggested under RFC9111?

And then finally, it appears that the compute has to exist somewhere for the HTML-to-markdown conversion, if it is even a conversion. It seems that many systems should be able to produce Markdown natively just as they would HTML, and that “origin” of course seems like the best scenario but many companies will not be able to do that for years. It’s like the move to Responsive Design and Mobile-first, isn’t it?

But doesn’t Cloudflare just offer this as a service? You say “some people try…” but has it been reduced to a CDN switch to just flip on? And what about the JSON-LD and other structured data? Can that live on the Markdown pages? Does it do good there?

Break all this down pedantically but also for the newcomer, please and thank you.

Gemini 3.8 Flash High:

1. The Case for Pre-Rendering

Pre-rendering means generating the full response payload (HTML or Markdown) before a user or crawler asks for it, storing the result as a flat artifact in memory or disk.

Why Stick with Pre-Rendering?

  • Zero Origin Compute at Request Time: A database query, template engine, and markdown serializer can take 50–500ms. Serving a pre-rendered flat file takes 2–5ms from an edge cache.
  • Immunity to Database Crashes and Spikes: If an agentic scraper like GPTBot floods your site with 10,000 requests per minute, a dynamic origin will melt. A pre-rendered edge cache barely registers the load.
  • Cost: Running thousands of database round-trips for LLM crawlers is expensive; serving static blobs from edge object storage is negligible in cost.

2. The Spectrum of Pre-Rendering Schemes

Pre-rendering is not a single tool—it exists on a spectrum from completely offline builds to lazy on-demand generation:

[ Static / Offline ] ────────────── [ Hybrid / Edge ] ────────────── [ Dynamic ]
   Pure SSG (AOT)            On-Demand ISR / SWR          SSR + Dynamic Proxy
(Hugo, 11ty, Next Export)     (Next.js, Fastly/Cloudflare)    (Puppeteer / Node)
Scheme How It Works Freshness Mechanism Trade-Off
Pure SSG (Static Site Generation) Compiles the entire site into static files at deploy time. Full rebuild or scheduled batch jobs. Build times explode when managing 100k+ SKUs.
ISR / SWR (Incremental Static Regeneration) Generates pages on the first request, then caches them. Stale copies serve immediately while the origin quietly regenerates in the background. Time-to-Live (TTL) or Event-Driven Purge. Edge caches can briefly serve stale data during background re-renders.
Dynamic Headless Pre-rendering (e.g., Prerender.io) Runs headless Chromium at the edge or origin to evaluate JS-heavy SPAs and cache the DOM snapshot. Scheduled crawler runs or cache invalidation calls. Resource-intensive; introduces complex, brittle rendering infrastructure.

Can We Do Better Than Scheduling?

Yes: Event-Driven Invalidation via Change Data Capture (CDC).

Relying on scheduled cron jobs (e.g., rebuilding a catalog every night at 2 AM) is an outdated approach. In modern high-scale architectures:

  1. A price or content update commits to the database (PostgreSQL, ERP, Shopify).
  2. The database emits a transaction event (via Postgres WAL, Kafka, or a CMS Webhook).
  3. An automated worker calls the CDN’s cache purge API (targeting a specific URL or cache tag) and immediately fires a background warm-up GET request.
  4. Result: The pre-render is updated within 100 milliseconds of the database write—no cron jobs, no unneeded rebuilds.

3. Real-Time Price Updates on Pre-Rendered Pages: Edge HTML Rewriting

You do not need to discard pre-rendering just because prices change thousands of times a day. You can use Edge DOM Rewriting or Fragment Assembly.

How Edge HTML Rewriting Works

  1. The Static Shell: The entire HTML document (layout, navigation, product descriptions, images) is pre-rendered and permanently cached at the CDN edge.
  2. The Dynamic Placeholder: The price container is marked with a predictable data attribute: 
    <span data-price-sku="SKU-88219" class="price">$0.00</span>
  3. The Edge Interceptor: When a request hits Cloudflare, Fastly, or an Nginx instance, an edge streaming parser (like Cloudflare’s HTMLRewriter or Fastly’s VCL/Compute engine) catches the streaming cached HTML on its way to the user.
  4. Surgical Patching: The edge worker performs a sub-millisecond lookup against an ultra-fast Edge Key-Value store (Cloudflare KV, Fastly Config Store, or a regional Redis) and injects the live price directly into the HTML stream before the first byte leaves the edge node:
[ Browser / AI ] ◄── Rewritten Stream ── [ Edge Node: HTMLRewriter ]
                                                │            │
                           Reads Cached Shell ──┘            └── Reads Live Price ($19.99)
                                                ▼                          ▼
                                       [ Edge Page Cache ]       [ Edge KV Store ]

Does This Need to Be Scheduled?

No. The static shell remains cached forever (or for months). When a price changes:

  1. The pricing service writes the new price to the edge KV store (a 50ms write propagated globally).
  2. The cached HTML shell is not invalidated.
  3. Every subsequent request automatically streams the updated price from the KV store.

How Does This Compare to RFC 9111?

RFC 9111 specifies how HTTP caches store and invalidate whole representations via headers like Cache-Control, Age, and ETag.

Edge rewriting operates above and outside RFC 9111:

  • The inner request (fetching the template shell) fully obeys RFC 9111 caching rules.
  • The outer response delivered to the client can either be marked as uncacheable (Cache-Control: no-cache, no-store), given a very short TTL (max-age=5), or governed by client-side revalidation headers.
  • In enterprise architectures, this is historically formalized by Edge-Side Includes (ESI) or the standardized Surrogate-Control headers (which tell intermediate CDNs to cache components differently than the downstream browser).

4. Origin vs. Edge Markdown: The “Responsive Design” Transition

Your comparison to Responsive Web Design (RWD) is structurally exact.

  • 2010 (The Mobile Hack): When mobile traffic exploded, origins could not dynamically adapt their markup. Companies deployed proxies or separate subdomains (m.example.com) that scraped desktop HTML and reformatted it for small screens.
  • 2014 (True RWD): Origins adapted. Template systems were updated to output fluid, responsive layouts natively from the same data model.

The AI Markdown Parallel

  • Phase 1 (The Edge Band-Aid): Legacy monolithic origins cannot produce clean Markdown. Teams use Edge Workers or proxy converters (like Turndown or headless DOM scrapers) to intercept HTML and translate it into Markdown on the fly.
  • Phase 2 (Native Origin Negotiation): Modern headless architectures (Next.js, Remix, headless CMSs) read the incoming Accept: text/markdown header and bypass the HTML UI template entirely. They serialize their internal data model directly into clean Markdown or YAML using minimal CPU.

If a company cannot modify its origin codebase for months or years, converting HTML at the edge is the only practical interim solution.

5. Can’t You Just “Flip a Switch” on Cloudflare?

Cloudflare does offer turnkey tools aimed at AI bots and scrapers, but there are important operational nuances:

  • What Cloudflare Automates: Cloudflare provides automated scraper handling, AI Crawl Protection (which blocks or bills AI bots at the edge), and automated Workers AI pipelines. They also offer documentation patterns and worker recipes to parse and convert pages.
  • Is It a Single “Convert to Markdown” Dashboard Toggle? While Cloudflare provides turn-key features to protect against or inspect AI bots, arbitrary HTML-to-Markdown conversion is not a generic, magic one-click setting for every site.
  • Why It Still Requires Code/Workers: Every website’s DOM is idiosyncratic. A generic converter does not know:
  • That your cookie banner should be stripped.
  • Which <div> represents the primary article vs. sidebar ad units.
  • How you want image galleries or pricing tiers formatted.
  • Which headers/metadata your business wants to preserve.

To handle this cleanly at the CDN level without touching origin code, you typically deploy a Cloudflare Worker using an edge parser library (e.g., custom AST parsers or Turndown compiled to WebAssembly) combined with HTMLRewriter to strip layout noise before markdown conversion.

6. What About Structured Data (JSON-LD) in Markdown?

Can JSON-LD exist inside Markdown, and is it beneficial for LLMs?

Where Structured Data Lives in Markdown

Markdown specifications natively support raw HTML blocks. You can place standard JSON-LD script blocks directly inside a Markdown file:

# Enterprise Tier License

Price: $499/year

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Enterprise Tier License",
  "offers": {
    "@type": "Offer",
    "price": "499.00",
    "priceCurrency": "USD"
  }
}
</script>

Detailed license features and provisioning instructions...

Alternatively, many AI-facing systems use YAML Frontmatter at the very top of the file:

---
title: "Enterprise Tier License"
sku: "ENT-2026"
price: 499.00
currency: "USD"
availability: "InStock"
---

# Enterprise Tier License
Detailed license features...

Does It Do Any Good There?

Format Token Cost Machine Readability Verdict for AI Agents
Raw JSON-LD High (loads of syntactic overhead: @context, syntax brackets, quotes) Excellent for search spiders; good for LLMs. Wasteful for RAG/LLMs. You burn scarce context-window tokens on syntax rather than substance.
YAML Frontmatter Low (compact, human-readable key-value pairs) Native format for modern LLMs and code generators. Best Practice. Clean, readable, and highly token-efficient.
Markdown Tables Very Low (standard markdown pipe-table) High semantic density for LLMs. Ideal for tabular comparisons (pricing matrices, spec sheets).

The Strategic Rule for AI-Ready Pages

  • If serving search engines (Googlebot): Keep standard JSON-LD inside the HTML representation.
  • If serving LLMs / AI Agents (via Accept: text/markdown): Strip the verbose JSON-LD script tags and extract that structured data into YAML Frontmatter or a concise Markdown spec table at the top of the file.

This preserves all schema relationships while slashing the token payload by 50% to 70%.

The Spliced Edge: Dynamic Price Injection Without Origin Stampedes

MikeLev.in: Is this what you’re talking about? You may have to perform a web search. It may be past your training cutoff.

https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/

It seems somebody always has to pay the piper somewhere. It feels like the heavy-lifting cost is always being shifted around or moved somewhere and there are different levels of complexity. HTML re-writes require dabbling in CSS, Xpath or jQuery paths, right? But when a price changes the publisher can just ping a delete API-call and both will be immediately updated per RFC 9111 it appears. Where is the best and most cost efficient place to move the work in the challenge of driving down both cost and complexity? Even Cloudflare must be running one of those HTML-to-Markdown converters you mentioned and after a delete ping goes out for say 100,000 pages, there has to be at least 1 hit against origin per each of those pages to get it back in the cache, and at that moment it will probably have to be the HTML nearly as-is from origin but now in the kv-cache, and also a conversion to Markdown that goes in the kv-cache as well but slightly differently.

How does that all work? And how does that all work most cheaply and efficiently. I am aware that painting over a cache with HTML re-writing that blazingly fast final-touches way is efficiency but that’s paid for somehow, right? Not only are there the paths but now the mechanics can’t just be an API-delete-ping. The price has to come from somewhere.

Continue enlightening me as if I were a newcomer.

Gemini 3.8 Flash High: There is an iron law of distributed systems: compute never disappears—it only moves.

Every byte served to a client has to be processed somewhere: at the origin database, in a server template engine, inside an edge CDN worker, or inside the client’s browser. When you introduce multi-representation delivery (HTML for humans, Markdown for LLMs) along with high-churn dynamic data (100,000 price changes a day), you are negotiating where that compute bill is paid and who bears the latency.

1. How Cloudflare’s “Markdown for Agents” Actually Works Under the Hood

The documentation page describes an automated edge interception proxy.

When you flip that dashboard toggle, you aren’t changing your backend server. You are inserting an intermediary transformation pipeline directly into Cloudflare’s edge PoP (Point of Presence):

                        ┌────────────────────────────────────────────────────────┐
                        │                   CLOUDFLARE EDGE                      │
                        │                                                        │
[Client Request] ──────►│ 1. Check Accept Header                                 │
(Accept: text/markdown) │    ├── Is it text/markdown?                            │
                        │    └── Is Markdown variant already cached?             │
                        │         ├── YES ──► Return Cached Markdown (2ms)       │
                        │         └── NO  ──┐                                    │
                        │                   ▼                                    │
                        │ 2. Check Edge HTML Cache                               │
                        │    ├── Cached? Use cached HTML                         │
                        │    └── Miss? Fetch HTML from Origin ──────────────┐    │
                        │                   │                               │    │
                        │ 3. Streaming AST Transformation (Rust/WASM)       │    │
                        │    ├── Extract <meta>   ──► YAML Frontmatter      │    │
                        │    ├── Strip noise DOM  ──► Readability Engine    │    │
                        │    ├── Convert body     ──► Markdown Syntax       │    │
                        │    └── Extract JSON-LD  ──► Fenced [triple-backtick]json Block  │    │
                        │                   │                               │    │
                        │ 4. Cache Partitioning                             │    │
                        │    Store converted Markdown under Vary: Accept    │    │
                        └───────────────────┼───────────────────────────────┼────┘
                                            │                               │
                                            ▼                               ▼
                                    [Client Receives MD]            [Your Origin]

Why Does Cloudflare Give This Away “For Free”?

Cloudflare includes this on Pro ($20/mo) and Business plans because edge compute is often cheaper than egress bandwidth.

A bloated modern enterprise HTML document is routinely 500 KB to 2 MB (chock-full of inline SVGs, script bundles, and tracking tags). The stripped Markdown equivalent is typically 15 KB to 40 KB. By spending roughly 1 to 3 milliseconds of edge CPU converting the HTML once and caching the Markdown result, Cloudflare reduces the outbound wire egress bandwidth by up to 95% when bots crawl your site.

The Hidden Architectural Catch in the Cloudflare Doc

Notice this critical line in their technical notes:

“Headers that describe the original body are removed… ETag and Last-Modified are dropped because conditional requests (If-None-Match, If-Modified-Since) cannot be honored for converted responses.”

Because Cloudflare synthesizes the Markdown dynamically from HTML, it strips the origin’s entity tags. It cannot safely negotiate HTTP 304 Not Modified conditional requests back to the origin for that variant. Every cache miss on the Markdown variant forces a full HTML round-trip and a full re-conversion.

2. The 100,000 Price Changes Problem: The Invalidation Stampede

If a retailer experiences 100,000 price mutations per day, the classic “delete ping” (cache purge) approach hits a wall called the Thundering Herd (or Cache Stampede).

If you issue an API call to purge 100,000 URLs:

  1. The CDN drops those 100,000 objects from its edge nodes across 300+ cities worldwide.
  2. The next time human users and AI crawlers hit those 100,000 URLs, every single request bypasses the edge and slams directly into your origin server simultaneously.
  3. If each URL now has two variants (one for HTML, one for Markdown), your origin doesn’t get 100,000 hits—it potentially gets 200,000 hits.
  4. Your SQL database locks up, connection pools exhaust, and the origin crashes.

How CDNs Tame the Stampede

To do simple API invalidations at this scale without melting the origin, enterprise CDNs rely on two RFC 9111 mechanics:

  1. Surrogate Keys / Cache-Tags: Instead of purging 100,000 individual URLs by name, the origin labels pages with shared tags (Cache-Tag: product-catalog, brand-nike). A single API call purges the tag globally in under 150ms.
  2. Request Collapsing (Origin Shielding): If 500 AI bots request /sneakers/air-jordan at the exact millisecond it was purged, the CDN edge locks the request. It forwards only one request to your origin. The other 499 requests sit in an edge wait-queue. When that single response returns, the CDN satisfies all 500 requests at once and caches the result.

3. Edge HTML Rewriting: How the “Painting Over” Trick Works

Instead of purging whole pages 100,000 times a day, modern architectures use Edge Fragment Splicing via streaming parsers like Cloudflare’s HTMLRewriter or Fastly’s VCL/Compute.

Does It Require CSS Selectors and XPath?

Yes. Under the hood, streaming parsers look like this:

// Edge Worker executing on Cloudflare or Fastly
class PricePatcher {
  constructor(livePrice) {
    this.livePrice = livePrice;
  }
  // Matches CSS Selector <span data-sku="...">
  element(element) {
    element.setInnerContent(`$${this.livePrice}`);
  }
}

export default {
  async fetch(request, env) {
    // 1. Fetch the static, permanently cached HTML shell (Hit rate: ~99%)
    const cachedHtmlResponse = await env.ASSETS.fetch(request);

    // 2. Extract SKU and pull the live price from ultra-fast Edge KV (sub-5ms)
    const sku = "SKU-99182";
    const currentPrice = await env.PRICE_KV.get(sku); // Reads $49.99

    // 3. Stream HTML to user, replacing ONLY the targeted DOM node on the fly
    return new HTMLRewriter()
      .on("span.product-price", new PricePatcher(currentPrice))
      .transform(cachedHtmlResponse);
  }
};

Where Does the Price Come From?

The price doesn’t magically appear. When the pricing engine updates a price in your database:

  1. It commits the change to PostgreSQL.
  2. An asynchronous message (via Kafka, Redis, or Webhook) writes a key-value pair to the CDN’s distributed storage: PRICE_KV.put("SKU-99182", "49.99")
  3. That write takes ~50ms to replicate worldwide.
  4. The HTML page is never purged. The page cache remains at a 99% hit rate. The edge merely acts like a high-speed stencil, spraying the new price onto the cached page as the bytes stream through the network pipe.

The Catch with Markdown

HTMLRewriter operates on the DOM (it parses <div>, <span>, class, and id).

Markdown has no DOM. It is flat text. You cannot use CSS selectors on a Markdown document. If you want to “paint over” a price on a pre-rendered Markdown document at the edge, you must use regex string replacements, frontmatter token swapping, or re-parse a Markdown Abstract Syntax Tree (AST). This is why automated edge converters like Cloudflare’s “Markdown for Agents” choose to intercept the HTML first before emitting Markdown.

4. The Cost & Complexity Hierarchy: Where Should the Work Live?

To decide where the work belongs, evaluate the three primary architectural patterns:

Strategy Operational Complexity Origin Load Edge Bill / Latency Ideal Use Case
1. Cloudflare Dashboard Toggle (Markdown for Agents) Lowest (Flip a switch in the dashboard) Medium-High (Origin serves full HTML; conversion happens at edge; no origin conditional caching) Lowest (Included in Pro/Biz plans) Blogs, docs, small e-commerce, teams with zero backend bandwidth.
2. Edge Splicing (HTMLRewriter + Edge KV) High (Maintain CSS paths, sync database writes with Edge KV writes) Near Zero (Origin pages are permanently cached shells) Medium (Pay for Edge KV operations and Worker CPU time) Enterprise e-commerce with 50,000+ daily price/inventory shifts.
3. Native Origin Content Negotiation (API-First Headless) Medium-High upfront, Low long-term (Templates handle both .html and .md) Low (Origin renders lightweight text/markdown directly from DB cache) Zero (Standard CDN caching rules apply cleanly) Modern decoupled stacks (Next.js, Astro, Remix, Headless CMS).

The Economic Reality

  • If you are a startup or medium business: Turn on Cloudflare Markdown for Agents or equivalent CDN edge features. It saves you months of engineering time, automatically strips boilerplate, and serves AI scrapers clean Markdown without touching a single line of backend code.
  • If you are an enterprise running volatile real-time pricing: Do not rely on automated on-the-fly scrapers. The lack of granular conditional revalidation will cost you money. Instead, use Surrogate-Key event invalidations on both representations, or push price states to an Edge Key-Value store so your edge nodes can dynamically stencil prices into cached templates without ever waking up your origin database.

MikeLev.in: Write a Cloudflare Worker example that fetches a cached Markdown document and splices in real-time pricing from a KV store before serving it to an AI agent.

Gemini 3.8 Flash High: This Cloudflare Worker implements edge-spliced pricing for AI agents. It fetches a long-cached Markdown template from the edge cache, extracts the product SKU, retrieves real-time pricing from Workers KV, and splices the fresh values into both YAML frontmatter and body markdown in sub-5ms compute time.

1. Wrangler Configuration (wrangler.jsonc)

Configure the KV namespace binding and compatibility date:

{
  "name": "edge-markdown-pricing",
  "main": "src/index.ts",
  "compatibility_date": "2026-10-10",
  "kv_namespaces": [
    {
      "binding": "PRODUCT_PRICES",
      "id": "a8f3b2c1d0e94123b789456123abcdef"
    }
  ]
}

2. Cloudflare Worker Implementation (src/index.ts)

export interface Env {
  PRODUCT_PRICES: KVNamespace;
}

interface PriceRecord {
  price: number;
  currency: string;
  in_stock: boolean;
  updated_at: string;
}

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const url = new URL(request.url);
    const acceptHeader = request.headers.get("Accept") || "";

    // 1. Content Negotiation Gate (RFC 9110)
    // Pass standard browser traffic through to the origin HTML pipeline
    if (!acceptHeader.includes("text/markdown")) {
      return fetch(request);
    }

    // 2. Resolve Static Markdown Shell from Edge Cache (RFC 9111)
    // Partition the cache key so Markdown templates never collide with HTML
    const templateCacheUrl = new URL(url.toString());
    templateCacheUrl.searchParams.set("__cf_variant", "raw_markdown_shell");
    const templateCacheKey = new Request(templateCacheUrl.toString(), {
      method: "GET",
      headers: { "Accept": "text/markdown" },
    });

    const edgeCache = caches.default;
    let templateResponse = await edgeCache.match(templateCacheKey);

    if (!templateResponse) {
      // Cache miss: fetch static markdown template from upstream origin/bucket
      const originRequest = new Request(url.toString(), {
        headers: { "Accept": "text/markdown" },
      });
      templateResponse = await fetch(originRequest);

      if (!templateResponse.ok) {
        return templateResponse;
      }

      // Store un-spliced template shell at the edge with long TTL (24 hours)
      const cacheableShell = new Response(templateResponse.body, templateResponse);
      cacheableShell.headers.set("Cache-Control", "public, max-age=86400, s-maxage=86400");
      ctx.waitUntil(edgeCache.put(templateCacheKey, cacheableShell.clone()));
      templateResponse = cacheableShell;
    }

    let markdownBody = await templateResponse.text();

    // 3. Extract SKU identifier from YAML frontmatter (fallback to URL slug)
    const skuMatch = markdownBody.match(/^sku:\s*["']?([A-Za-z0-9_-]+)["']?/m);
    const sku = skuMatch ? skuMatch[1] : url.pathname.split("/").filter(Boolean).pop();

    if (!sku) {
      return new Response(markdownBody, {
        status: 200,
        headers: {
          "Content-Type": "text/markdown; charset=utf-8",
          "Vary": "Accept",
        },
      });
    }

    // 4. Retrieve Real-Time Price from KV (Sub-5ms global read)
    const priceData = await env.PRODUCT_PRICES.get<PriceRecord>(sku, {
      type: "json",
      cacheTtl: 60, // Local edge PoP memory cache TTL in seconds
    });

    // 5. Splice Dynamic Real-Time Values into Markdown
    if (priceData) {
      const formattedPrice = priceData.price.toFixed(2);
      const stockStatus = priceData.in_stock ? "InStock" : "OutOfStock";

      // A. Splice YAML Frontmatter fields
      markdownBody = markdownBody
        .replace(/^price:\s*.*$/m, `price: ${formattedPrice}`)
        .replace(/^currency:\s*.*$/m, `currency: "${priceData.currency}"`)
        .replace(/^availability:\s*.*$/m, `availability: "${stockStatus}"`)
        .replace(/^updated_at:\s*.*$/m, `updated_at: "${priceData.updated_at}"`);

      // B. Splice In-Text Markdown Placeholders and Tables
      markdownBody = markdownBody
        .replace(/\{\{PRICE\}\}/g, `$${formattedPrice}`)
        .replace(/\{\{AVAILABILITY\}\}/g, priceData.in_stock ? "In Stock" : "Backorder")
        .replace(/(\*\*Price\*\*:\s*)\$[0-9]+(\.[0-9]{2})?/g, `$1$${formattedPrice}`);
    }

    // 6. Return Assembled Representation to AI Agent (RFC 9110/9111 compliant)
    return new Response(markdownBody, {
      status: 200,
      headers: {
        "Content-Type": "text/markdown; charset=utf-8",
        "Vary": "Accept",
        // The spliced response has a brief TTL; pricing shifts live in KV
        "Cache-Control": "public, max-age=30, stale-while-revalidate=60",
        "X-Edge-Spliced": "true",
      },
    });
  },
};

3. Template Markdown Contract (Origin Output)

Your origin backend or static CMS serves the base markdown document with baseline metadata and substitution tokens:

---
title: "Aeropress Titanium Travel Edition"
sku: "SKU-TITANIUM-01"
price: 0.00
currency: "USD"
availability: "Unknown"
updated_at: "2026-01-01T00:00:00Z"
---

# Aeropress Titanium Travel Edition

**Price**: {{PRICE}}  
**Status**: {{AVAILABILITY}}  

| Attribute | Specification |
| :--- | :--- |
| Model SKU | SKU-TITANIUM-01 |
| Material | Grade 5 Titanium |
| Current Price | {{PRICE}} |

## Overview
Ultra-durable, lightweight manual press engineered for outdoor brewing.

4. Verification and Execution Checklist

  1. Seed KV Namespace: Write a test price mutation into the KV store via Wrangler: 
    npx wrangler kv:key put --binding=PRODUCT_PRICES "SKU-TITANIUM-01" \
  '{"price":149.99,"currency":"USD","in_stock":true,"updated_at":"2026-10-10T22:50:00Z"}'
  2. Deploy the Worker: 
    npx wrangler deploy
  3. Verify AI Agent Path (Markdown + Spliced Price): 
    curl -i -H "Accept: text/markdown" https://your-worker-domain.com/products/aeropress-titanium

Verify: Output contains Content-Type: text/markdown, Vary: Accept, price: 149.99, and $149.99 in the markdown body.

  1. Verify Browser Pass-Through (HTML): 
    curl -i -H "Accept: text/html,application/xhtml+xml" https://your-worker-domain.com/products/aeropress-titanium

Verify: Passes directly upstream to origin, serving unmodified HTML without triggering KV operations.

  1. Verify Real-Time KV Mutation (No Page Purge Required): Update the key in KV to 129.99: 
    npx wrangler kv:key put --binding=PRODUCT_PRICES "SKU-TITANIUM-01" \
  '{"price":129.99,"currency":"USD","in_stock":true,"updated_at":"2026-10-10T22:52:00Z"}'

Re-run the Markdown curl command. The new price renders immediately at the edge without invalidating the underlying Markdown shell cache.

Strategic Horizons: Navigating the Decade-Long Agent Migration

MikeLev.in: Alright, let’s wrap this article up with a mini Q&A between Company A in the process of realizing that they have no choice but to be AI-ready and learning about these band-aid solutions options to bide them time while they do it properly. Do it all storytelling like playing off how they’ve been through it all before during the days of Mobile-first and RWD and how it may take a decade or more to play out. Play the role of the Trusted Advisor genuinely trying to advise what’s best, optimizing for cost, quality, speed! Ha ha, I know it’s the old “pick two” joke but you get the point. Let’s flesh out the best balanced position for the Company and the highest integrity position for the Trusted Advisor and plenty of clarity to where the piper gets paid such as but not limited to work that for some reason can’t be templated and takes more customization than one might hope for.

Does that make sense? Make it as thorough as you can please and thank you.

Gemini 3.8 Flash High:

The Strategic Endgame: A Dialogue on Technical Debt, Band-Aids, and the Long Arc of Architecture

Company A:

I’m having intense flashbacks to 2011. Our board just discovered that enterprise buying agents and LLM search scrapers are consuming our site, and suddenly everyone is in panic mode demanding an “AI-First” delivery pipeline.

Back in 2011, it was: “Steve Jobs killed Flash, mobile traffic is 30% of our visits, and our desktop site looks like an unreadable postage stamp on an iPhone.” We spent three years building hacky m.example.com reverse-proxy scrapers that broke every time marketing added an image carousel. Then we spent another five years rebuilding every frontend into Responsive Web Design (RWD) so the origin could natively handle any viewport width.

Are we looking at that exact same movie all over again? And are we really talking about a 10-year transition just to serve Markdown?

Trusted Advisor:

It is the exact same movie, frame for frame. Only this time, the viewport isn’t a 4-inch iPhone retina display—it’s an LLM context window.

In both eras, the core architectural friction is identical: your origin server was designed around a single human presentation layer, and an alien client arrived demanding raw semantic substance.

Mobile Era (2010–2018):      Desktop HTML  ──► [m-dot Proxy Hack]  ──► Small Screen
AI Agent Era (2024–2032+):   Desktop HTML  ──► [Edge Parser Hack]  ──► LLM Context

The cycle always follows three distinct evolutionary phases:

  1. The Panic & Proxy Phase (Years 1–3): You don’t have the time or budget to rewrite legacy origin systems. You lean on edge proxies, scraping services, or CDN checkboxes to strip the chrome and fake responsiveness.
  2. The Hybrid Splicing Phase (Years 3–6): You discover the edge band-aids break on edge cases, price updates, and complex transactions. You build intermediate pipelines—Edge Workers, Key-Value stores, and fragment assembly—to patch over legacy systems.
  3. The Native Architecture Phase (Years 6–10): The headless origin finally grows up. The backend natively understands RFC 9110 content negotiation. Whether a request asks for text/html, text/markdown, or application/json, the origin renders directly from the canonical data model using zero DOM scraping.

Yes, it will take the enterprise world a decade to reach Phase 3. The trick is surviving Phases 1 and 2 without bankrupting your infrastructure budget or poisoning your cache.

Company A:

Alright, so let’s talk about the classic Iron Triangle: Cost, Quality, and Speed. Management wants all three yesterday.

Can we actually balance them here? If we use Cloudflare’s “Markdown for Agents” toggle or an Edge Worker, what are we sacrificing? Where does the “pick two” reality bite us?

Trusted Advisor:

The Iron Triangle is undefeated in distributed systems. Here is where the trade-offs actually land:

                      [QUALITY]
                      Native RFC 9110 Origin
                      (Perfect schemas, zero latency overhead)
                              ▲
                             / \
                            /   \
                           /     \
   [SPEED] ◄──────────────       ──────────────► [COST]
   Edge Worker AST Splicer                       Turnkey CDN Toggle
   (Fast rollout, expensive                      (Free on Pro plan, brittle
   compute & KV ops)                             on complex layouts)
  1. If you pick Speed + Low Upfront Cost (The Turnkey CDN Toggle):
    • What you get: You turn on a feature like Cloudflare’s Markdown for Agents on Monday afternoon. You write zero lines of backend code. AI bots instantly receive clean Markdown documents stripped of headers and footers.
    • Where Quality suffers: The CDN is guessing what matters on your page using generic DOM-stripping heuristics. It can’t understand business context. If your pricing is rendered via dynamic client-side React hydration, the converted Markdown will capture the $0.00 loading state or miss it entirely. Furthermore, as Cloudflare’s own documentation points out, you lose origin ETag conditional revalidation.
  2. If you pick Speed + High Quality (The Edge Worker + KV Splicer):
    • What you get: You write 100 lines of Edge Worker code (like the TypeScript pattern we reviewed). You cache the static markdown shell for 24 hours, extract the SKU, and splice the live price from Edge KV in 3 milliseconds. AI agents get 100% fresh, accurate data formatted exactly how you want.
    • Where Cost bites you: You pay the piper in edge operations. At 100,000 price mutations and millions of agent scrape requests per day, you are paying for high-frequency distributed KV writes, KV reads, and Worker CPU compute cycles on every bot request.
  3. If you pick Cost + High Quality (The Native Origin Architecture):
    • What you get: Your backend handles Accept: text/markdown natively. It queries the database or Redis cache, passes the data through a clean Markdown/YAML template, sets format-scoped ETags, and outputs a 15 KB response. Edge caching is standard, simple, and cheap.
    • Where Speed bites you: It takes 6 to 18 months of roadmap prioritization, cross-team coordination, and QA cycles to build across your entire legacy estate.

Company A:

Let’s talk about where the work gets messy. Standard e-commerce catalog pages are easy—title, SKU, price, bullet points. We can template that in our sleep.

Where are the “untemplatable” potholes that ruin these edge band-aids? Where does the automation fall apart and force us into ugly manual engineering?

Trusted Advisor:

The piper gets paid the moment you move past basic content pages and encounter the interactive, aesthetic, and structural anomalies of the web:

1. The Dynamic Configurator & Interactive Calculators

Think of a mortgage calculator, a custom vehicle builder, or an enterprise SaaS pricing matrix with 40 toggle switches.

  • A human user clicks a slider and watches a React state update.
  • An AI agent requesting Accept: text/markdown gets a static text dump that says: “Select your options below: [ ] [ ] [ ] — Total: NaN”.
  • The Custom Tax: You cannot automate this with a generic HTML-to-Markdown parser. You must build an explicit data-contract representation for the agent—a structured YAML decision tree or a documented API schema explaining the formula.

2. Visual Aesthetic Crutches (Infographics, Comparison Charts, Complex SVGs)

Human web design frequently hides crucial business data inside UI layout tricks: CSS grids, visual icons (green checkmarks vs. red crosses in comparison tables), and inline SVGs.

  • A generic scraper strips SVG elements and renders a table where the rows say: Feature A | [blank] | [blank].
  • An LLM agent reads that and hallucinates that your product lacks the feature.
  • The Custom Tax: Someone has to audit your top revenue-generating comparison pages and write accessible aria-label tags or Markdown fallback tables so the edge parser doesn’t discard vital product specs.

3. Bespoke Marketing Landing Pages (The CMS WYSIWYG Disaster)

In any enterprise, marketing teams have spent a decade building custom landing pages using visual builders (Elementor, Webflow, custom Gutenberg blocks). These pages are structural spaghetti: six nested <div> wrappers just to center a quote, non-semantic <span> tags acting as headings, and text embedded directly in background images.

  • Generic edge scrapers produce chaotic, fragmented Markdown garbage when handed messy DOM trees.
  • The Custom Tax: You can’t write a clean regex or CSS selector rule for these because there is no standardized pattern across pages.

Company A:

So what is the honest, highest-integrity game plan? If you were sitting in my seat, with a board asking for an AI strategy and engineering teams already working at capacity, how would you stage this without blowing up the budget or building a house of cards?

Trusted Advisor:

Stage it into three deliberate horizons. Don’t fight the reality of the band-aids—use them strategically to buy the time you need to do the foundational engineering.

Horizon 1: Triage (Days 1–30)        ► CDN AI Controls + Turnkey Markdown
Horizon 2: Tactical Edge (Months 2–6) ► Edge KV Splicing on High-Value SKUs
Horizon 3: Foundation (Year 1+)       ► Native Origin RFC 9110/9111 Pipeline

Horizon 1: Immediate Triage (Days 1 to 30)

  • Goal: Stop bleeding tokens and bandwidth; get basic visibility.
  • Action:
    1. Turn on the turnkey CDN toggle (e.g., Cloudflare Markdown for Agents or equivalent CDN edge rules) across your static documentation and standard content/blog sections.
    2. Implement AI Crawl Control at the edge: block unverified scrapers that ignore robot directives, and permit verified search agents (OpenAI, Anthropic, Perplexity).
    3. Ensure your CDN emits Content-Signal and Vary: Accept headers to establish policy and prevent browser cache poisoning.
  • Investment: A few hours of configuration and running the automated Python cache-isolation test suite.

Horizon 2: Tactical Edge Splicing (Months 2 to 6)

  • Goal: Protect revenue-critical catalog items and volatile pricing without rewriting the backend monolith.
  • Action:
    1. Identify your top 20% highest-value, high-churn product pages.
    2. Deploy an Edge Worker with the pattern we built: keep the base Markdown shell cached with a 24-hour TTL, and splice volatile pricing and stock statuses from an Edge KV store populated by change-data-capture database events.
    3. Manually curate YAML frontmatter templates for your top 100 most critical comparison tables and product configurators.
  • Investment: 2 to 4 weeks of a Senior Infrastructure Engineer’s time.

Horizon 3: Strategic Origin Parity (Year 1 and Beyond)

  • Goal: Retire the edge band-aids and eliminate edge compute taxes forever.
  • Action:
    1. Mandate that all new services, headless CMS migrations, and Next.js/Remix APIs natively support RFC 9110 Content Negotiation.
    2. When an endpoint receives Accept: text/markdown, it serializes directly from internal JSON data structures straight to Markdown and YAML frontmatter, bypassing the HTML rendering engine entirely.
    3. Emit distinct, representation-scoped ETags (ETag: "sku123-md") so AI crawlers can use standard RFC 9111 conditional requests (304 Not Modified), cutting edge and origin bandwidth costs to zero on unchanged content.
  • Investment: Incorporated naturally into your ongoing architectural roadmap over the next 2 to 3 years.

Company A:

That makes complete sense. We use the CDN switches to stop the bleeding, use Edge Workers to surgical-patch the volatile pricing that would otherwise break our business, and stop pretending that we can skip the real origin redesign long-term.

Trusted Advisor:

Precisely. The edge gives you an umbrella while it rains; the origin is the house you build so you can live through the storm. Work with the grain of HTTP standards, respect RFC 9111 cache partitioning, and you won’t have to rebuild your foundations every time a new client changes the rules of the web.

Book Analysis

Ai Editorial Take

The most striking revelation in this piece is the silent casualty of edge Markdown synthesis: the loss of conditional validation. When edge proxies dynamically strip origin ETags because they cannot honor conditional checks across converted formats, they quietly degrade HTTP from an intelligent state-negotiation protocol into a blind time-to-live guessing game. This creates an unmeasured synthetic egress tax, forcing agents to re-fetch entire payloads simply because the edge cannot issue a 304 Not Modified receipt.

🐦 X.com Promo Tweet

Serving Markdown to AI agents cuts token waste, but without RFC 9111 cache partitioning, you risk poisoning browser sessions. Here is the edge blueprint.

https://mikelev.in/futureproof/edge-cache-partition-rfc9110-markdown-negotiation/

#WebDev #EdgeComputing

Title Brainstorm

  • Title Option: The Edge Cache Partition: RFC 9110, Markdown Negotiation, and Preventing Agent Poisoning
    • Filename: edge-cache-partition-rfc9110-markdown-negotiation.md
    • Rationale: Anchors directly in the dual standards of RFC 9110 content negotiation and RFC 9111 cache safety, highlighting the real-world operational risk of serving Markdown to LLMs without cache poisoning.
  • Title Option: From Mobile-First to Agent-First: The Decade-Long Arc of HTTP Content Negotiation
    • Filename: mobile-first-to-agent-first-content-negotiation.md
    • Rationale: Frames the current transition through the historical lens of Responsive Web Design, establishing why edge workarounds precede native origin adoption.
  • Title Option: The Spliced Edge: Serving Real-Time Data to AI Agents Without Origin Meltdown
    • Filename: spliced-edge-real-time-markdown-caching.md
    • Rationale: Focuses on the high-frequency pricing conundrum and how edge key-value substitution solves the thundering herd problem for negotiated text representations.
  • Title Option: Verifiable Cache Isolation: Testing RFC 9111 Partitioning for Agentic Web Traffic
    • Filename: verifiable-cache-isolation-rfc9111-agentic-traffic.md
    • Rationale: Emphasizes the auditability and synthetic test harness required to safely verify edge headers and cache slots before production deployment.

Content Potential And Polish

  • Core Strengths:
    • Grounds modern AI ingestion discussions in canonical IETF specifications (RFC 9110 and RFC 9111) rather than proprietary platform buzzwords.
    • Delivers multi-platform implementation configurations across Nginx, Cloudflare, Fastly, and Akamai alongside a complete verification script.
    • Demonstrates architectural realism by connecting content negotiation to real-world operational constraints like cache stampedes and volatile pricing.
  • Suggestions For Polish:
    • Address the SEO nuances of search engine crawlers handling Markdown endpoints to ensure content parity without risking cloaking penalties.
    • Include telemetry benchmarks evaluating the memory overhead and CPU cost of running regex token replacements versus edge AST transformations at scale.

Next Step Prompts

  • Draft a synthetic CI/CD benchmark suite that tests edge memory utilization and latency during high-concurrency Markdown content negotiation under simulated bot traffic.
  • Write an architectural decision record detailing the migration milestones for transitioning an e-commerce catalog from edge proxy parsing to native RFC 9110 origin rendering.