The Dual-Representation Web: RFC 9110, RFC 9111, and the Return of Content Negotiation

🤖 Read Raw Markdown • 📄 Google Doc (Try: Tools/Audio/Listen to document summary)

Setting the Stage: Context for the Curious Book Reader

The web was never meant to be an HTML monoculture where every URI is hardwired to visual chrome and client-side hydration scripts. In this entry, we peel back decades of accidental convention to examine how RFC 9110 and RFC 9111 formally separate abstract resources from their wire representations. Long before autonomous agents began choking on prompt token bloat, the IETF established the mechanics for proactive content negotiation, edge cache partitioning, and format-scoped validation. What follows is a comprehensive blueprint for serving both human browsers and synthetic crawlers from a single canonical URI—grounded in verifiable HTTP receipts rather than brittle scraping workarounds.

Technical Journal Entry Begins

RFC 9110 & RFC 9111: The Architecture of the Dual-Representation Web

The Web Was Never Meant to Be a Single-Format Monoculture.

For two decades, web development collapsed into an architectural shortcut: treating a Uniform Resource Identifier (URI) as an alias for a desktop HTML document packed with layout markup, visual styling, and client-side JavaScript.

Autonomous agents, retrieval-augmented generation (RAG), and synthetic clients have permanently broken that assumption. Feeding graphical user interfaces into a large language model burns 70% to 90% of prompt context windows on syntactic boilerplate.

The internet does not require proprietary scraping APIs, brittle headless rendering farms, or synthetic bot protocols to solve this. The foundation was standardized in June 2022 by the IETF: RFC 9110 (HTTP Semantics) and RFC 9111 (HTTP Caching).

1. The Core Invariant: Resources vs. Representations (RFC 9110)

Under RFC 9110 §3.1, a URI identifies an abstract Resource—not a file, and not a format. What an origin server transmits across the wire is merely a Representation reflecting the current state of that resource for a specific client.

                                  ┌── Accept: text/html ──────► [ Rendered HTML Representation ]
Canonical URI:                    │                             (CSS, Hydration, UI Layout)
https://example.com/product/104 ──┤
                                  └── Accept: text/markdown ──► [ Semantic Markdown Representation ]
                                                                (YAML Frontmatter, Tables, Spec Text)

Through Proactive Content Negotiation (§12), clients declare their processing capabilities:

  • Human User Agents request Accept: text/html,application/xhtml+xml...
  • Agentic Systems request Accept: text/markdown, text/plain;q=0.5

Both representations share the canonical URI, link graph, and authority, eliminating parallel URL fragmentation (.md, ?format=raw, or /api/ duplicates).

2. The Operational Law: Cache Partitioning (RFC 9111)

Serving multiple representations from an identical URI without edge cache corruption requires the strict operational mechanics of RFC 9111:

Vary: Accept
Cache-Control: public, max-age=3600, stale-while-revalidate=60
Link: <https://example.com/product/104>; rel="canonical"
ETag: "104-md-v2"
  1. Vary: Accept Enforces Edge Isolation: Instructs shared intermediary caches to store distinct representation variants for each negotiated client class.
  2. Format-Scoped Validation: Under RFC 9110 §8.8.3, entity tags (ETag) must be unique to the specific serialized representation format ("104-html" vs. "104-md"). Sharing ETags across formats corrupts conditional If-None-Match revalidations.
  3. Surrogate Keys / Cache-Tags: Because edge Key-Value hash tables evaluate flat string lookups, cache invalidation must be linked via backend entity headers (Cache-Tag: product-104) to evict both HTML and Markdown variants in a single API call.

3. How the Edge Resolves Traffic in Constant Time \(O(1)\)

A Content Delivery Network (CDN) does not search its storage or inspect available files to pick a response. It derives the cache address deterministically from the incoming request before touching memory:

[ Incoming Request ]
GET /products/104
Accept: text/markdown
       │
       ▼
[ Normalization Gate ] ────► Matches: $variant = "markdown"
       │
       ▼
[ String Concatenation ] ──► "GET::https::example.com::/products/104::markdown"
       │
       ▼
[ Hash Function ] ─────────► Hash Key: 0x7F4A8B9C (Direct Memory Index)
       │
       ▼
[ Fast RAM Lookup ] ───────► Serves Token-Efficient Markdown (< 5ms)

By normalizing client Accept headers into a discrete binary enum (html vs. markdown) at the edge, hit ratios remain above 95% while preventing the cache fragmentation historically caused by arbitrary browser headers.

4. The Architectural Divide

Deprecated Legacy Practice Standard RFC 9110 / RFC 9111 Practice Why It Changed
User-Agent Cloaking (Routing search bots via regex sniffing) Proactive Content Negotiation (Accept: text/markdown) Cloaking breaks protocol semantics and fails completely on autonomous agents using standard client headers.
Third-Party Pre-Render Proxies (Renting external Chromium farms) Native Edge Partitioning & Edge KV Splicing Eliminates 300ms network hops, fragile CSS selector scraping, and six-figure annual SaaS vendor contracts.
Format URL Divergence (/page.html vs. /page.md) Canonical URI Negotiation (One URI, varying representations) Preserves link equity, permalinks, and avoids fragmenting crawl discovery across parallel URL spaces.
Full URL Purge on Price Shifts (Flushing pages on catalog updates) Edge KV Splicing / Surrogate-Keys Prevents origin cache stampedes (“thundering herds”) by dynamically stenciling live values onto long-cached shells.
Shared Entity Tags (Same ETag across all formats) Representation-Scoped ETags ("v1-html" / "v1-md") Conforms to RFC 9110 §8.8.3, enabling valid HTTP 304 revalidations without cross-format cache contamination.

5. The Five Benchmarks of True AI-Readiness

  1. Token Efficiency Ratio (TER) \(\ge\) 70%: \(\)\text{TER} = 1 - \left( \frac{\text{Tokens in Markdown Representation}}{\text{Tokens in HTML Representation}} \right)\(\)

Slashing prompt ingestion overhead directly reduces downstream latency and inference cost.

  1. Grounding Accuracy Rate = 100%: Critical entities (SKU, price, specs, stock status) must be present in top-level YAML frontmatter or clean pipe tables, not discarded by aggressive heuristic scrapers.
  2. Retrieval Latency (TTFT) \(\le\) 50ms: Cached Markdown must stream from the nearest edge PoP without synchronous third-party proxy hops.
  3. Edge Cache Hit Ratio \(\ge\) 95%: Normalized edge keys must prevent multi-variant cache fragmentation.
  4. Conditional Revalidation Efficiency \(\ge\) 40%: Origin and edge must support RFC 9111 format-scoped ETag matching to serve 0-byte 304 Not Modified responses on repeated agent crawls.

6. Immediate Production Execution Checklist

Execute this verification sequence to audit and confirm RFC 9110 and RFC 9111 compliance:

  • Enforce Cache Key Normalization: Configure CDN cache rules to map incoming Accept headers to a discrete $content_variant variable (markdown or html) before computing edge cache keys.
  • Verify Header Transmission: Validate that your origin or edge worker responds with Vary: Accept and an explicit Content-Type: text/markdown; charset=utf-8 when queried with an agent accept header.
  • Emit Web Linking Canonical Headers: Ensure the response contains Link: <canonical_url>; rel="canonical" per RFC 8288 so headerless Markdown retains search and attribution authority.
  • Scope Conditional Validators: Confirm that Markdown and HTML payloads emit distinct ETags (e.g., append -md and -html suffixes) and correctly return 304 Not Modified on revalidation.
  • Enable Origin Shielding & Request Collapsing: Lock concurrent cache misses at the edge tier via Tiered Cache or proxy_cache_use_stale updating to prevent origin overload during catalog invalidations.
  • Verify Cross-Variant Cache Isolation: Run dual automated requests against a fresh endpoint (HTML Prime followed by Markdown Probe, and vice versa) to confirm that neither representation leaks into the other’s cache slot.

Note: This is where the article originally began but I moved the final result to the top.

The Dialogue: Untangling Protocol Evolution

MikeLev.in: First, we give you a whirlwind tour of RFC 7231, 9110 and 9111. The first was way back when… and then the others are more recent as…

Gemini 3.8 Flash High: …the definitive rewrite that finally untangled the web’s semantic meaning from its physical plumbing.

Here is the quick architectural cheat sheet to get your bearings:

The Evolution: From 2014 to 2022

       2014 (Coupled to HTTP/1.1)                    2022 (Transport-Agnostic)
┌──────────────────────────────────────┐     ┌──────────────────────────────────────┐
│ RFC 7231: HTTP/1.1 Semantics/Content │ ──► │ RFC 9110: HTTP Semantics             │
│ (Also 7232 conditionals, 7233 ranges)│     │ (Universal: works on H1, H2, and H3) │
└──────────────────────────────────────┘     └──────────────────────────────────────┘
┌──────────────────────────────────────┐     ┌──────────────────────────────────────┐
│ RFC 7234: HTTP/1.1 Caching           │ ──► │ RFC 9111: HTTP Caching               │
│ (Coupled to wire framing)            │     │ (Standalone caching architecture)    │
└──────────────────────────────────────┘     └──────────────────────────────────────┘

1. RFC 7231 (June 2014): The Coupled Monolith

In 2014, the IETF published the RFC 7230–7235 series to document HTTP/1.1. RFC 7231 was titled Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content.

It defined what GET meant, what a 404 was, and how headers like Accept operated. The fundamental flaw was its title and scope: it tied the conceptual rules of the web directly to HTTP/1.1 wire framing. When HTTP/2 arrived in 2015 with multiplexed binary frames, and HTTP/3 arrived over QUIC/UDP, referencing a spec anchored in 1990s-era line-delimited ASCII text became an architectural contradiction.

2. RFC 9110 (June 2022): HTTP Semantics, Liberated

RFC 9110 represents the great disentangling. The IETF extracted pure semantics away from wire framing:

  • It absorbed RFC 7231 (semantics), RFC 7232 (conditional requests), RFC 7233 (range requests), and RFC 7235 (authentication) into a single, canonical, transport-independent document.
  • Whether bits travel over TCP, multiplexed HTTP/2 streams, or HTTP/3 UDP datagrams, RFC 9110 defines the universal rules of engagement.
  • The AI Headline: Section 12 details Proactive Content Negotiation. This is where an agent advertises processing preferences (Accept: text/markdown, text/html;q=0.1) and the server selects the optimal representation for that single URI.

3. RFC 9111 (June 2022): HTTP Caching, The Enforcer

Extracted and modernized from RFC 7234, RFC 9111 specifies how shared caches (CDNs, edge proxies) and private caches (browsers) store, partition, revalidate, and invalidate representations.

  • It is the operational half of the equation. If RFC 9110 allows clients to ask for alternate flavors of a resource, RFC 9111 governs how to deliver those variants without corrupting production.
  • The Linchpin: Under RFC 9111, serving multiple representations from one canonical URL requires strict cache partitioning via Vary: Accept, format-scoped ETag validators, and Surrogate-Key invalidation. Miss this step, and your CDN serves raw Markdown to desktop shoppers or poisons the AI agent with 2 megabytes of HTML layout sludge.

Why Everyone Is Reading These Specifications Again

For a decade, frontend engineering operated under the assumption that a URI pointed to one thing: an HTML document stuffed with client-side JavaScript that hydrated inside a desktop or mobile browser.

Autonomous agents, retrieval-augmented generation (RAG), and LLM search spiders have shattered that monoculture. Feeding raw HTML, styling wrappers, and tracking tags into an LLM context window burns 80% to 90% of scarce prompt tokens on pure syntactic noise.

The industry isn’t inventing an exotic, proprietary protocol to solve this. It is doing what it always does when a new client class upends the ecosystem: returning to core IETF standards. RFC 9110 provides the token-efficient handshake; RFC 9111 provides the edge caching mechanics that keep the servers standing.

The Accidental Alignment: Why 2022 Specs Fit Modern AI

MikeLev.in: So let’s get the assumptions down here. 2022 is very recent. And 2014 is so long ago. And ChatGPT came out November 30, 2022 and the 9110 and 9111 must have been in the works some time before their release. Clearly the 2014 semantic web that allowed for serving PDF and XML versions on the same URL as HTML could not have been looking ahead to today’s agentic clients, but 2022 seems pretty close. Were those later RFCs written with AI consumers in mind?

Gemini 3.8 Flash High: No, not at all. RFC 9110 and RFC 9111 were not written with generative AI, LLMs, or autonomous agents in mind.

The June 2022 publication date makes it look like it was drafted right on the doorstep of the AI boom. But in internet standards, publication dates are the finish line of a marathon, not the starting gun.

1. The Timeline Mismatch

The work that produced RFC 9110 and 9111 did not start in 2022. The first working group draft—draft-ietf-httpbis-semantics-00—was formally published in April 2018.

To put spring 2018 in perspective:

  • OpenAI had not even published the original GPT-1 paper yet (that came in June 2018).
  • Google’s BERT was still months away (October 2018).
  • Concepts like “Retrieval-Augmented Generation (RAG),” “prompt token budgets,” and “autonomous agentic web scrapers” were nowhere on the radar of network protocol engineers.

The IETF HTTP Working Group (HTTPbis)—led by editors Roy Fielding, Mark Nottingham, and Julian Reschke—spent four years (2018–2022) debating framing, parsing edge cases, caching invariants, and transport layering. Generative AI had zero presence in those working group discussions.

2. What Actually Drove the 2022 Rewrite?

If not AI, why did the IETF spend four years overhauling specifications that had already been updated in 2014? Two massive architectural pressures forced their hand:

A. The Multi-Transport Layering Crisis (HTTP/2 & HTTP/3)

In 2014, RFC 7231 was titled Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. It bundled the abstract meaning of the web with the 1990s-era line-delimited ASCII wire format of HTTP/1.1.

When HTTP/2 arrived in 2015 (binary framing, multiplexing) and HTTP/3 began development over QUIC (UDP transport), a glaring architectural contradiction emerged:

  • HTTP/2 and HTTP/3 were completely different wire transports.
  • Yet whenever their specs needed to define what GET, a 200 OK, or an ETag meant, they had to cite a document whose title explicitly said HTTP/1.1.

The IETF needed a clean, modular hierarchy:

                     RFC 9110 (Pure Semantics)
                     RFC 9111 (Pure Caching)
                                │
        ┌───────────────────────┼───────────────────────┐
        ▼                       ▼                       ▼
RFC 9112 (HTTP/1.1)     RFC 9113 (HTTP/2)       RFC 9114 (HTTP/3)
  [ASCII Wire]          [Binary Multiplex]       [QUIC / UDP]

B. Intermediary Realities and Caching Debt

Between 2014 and 2022, the web became completely dominated by giant Content Delivery Networks (Cloudflare, Fastly, Akamai) and complex microservice reverse proxies.

RFC 7234 (the 2014 caching spec) had accumulated dozens of ambiguities regarding how modern shared caches should behave. RFC 9111 cleaned this up: it formally retired obsolete baggage (like Warning headers), tightened conditional revalidation rules, and codified how CDNs must handle partitioned cache keys and Vary dimensions.

3. If It Wasn’t Written for AI, Why Does It Fit AI So Well?

This is the great irony: Content negotiation wasn’t invented for AI, but it is rescuing AI from drowning in human UI noise.

Content negotiation (Accept, Content-Type, Vary) isn’t even a 2022 invention. It was conceived by Roy Fielding, Tim Berners-Lee, and the early web architects in the 1990s (RFC 2068 in 1997, RFC 2616 in 1999).

Fielding’s fundamental thesis for REST (outlined in his landmark 2000 doctoral dissertation and codified in RFC 9110) established a profound architectural rule:

A URI identifies a Resource (an abstract concept or entity), NOT a file and NOT a specific format. What the server transmits across the wire is merely a Representation of that resource state at that point in time.

In the 1990s and 2000s, people envisioned this mechanism would serve:

  • A browser requesting text/html.
  • A mobile WAP browser requesting text/vnd.wap.wml.
  • A screen reader requesting a clean text stream.
  • An enterprise system requesting application/xml or application/pdf.

For roughly fifteen years (2008–2023), the industry largely discarded that elegant separation. Frontend development collapsed into a single-format monoculture: every URL served an HTML shell packed with megabytes of JavaScript, CSS, and DOM chrome meant exclusively for visual human consumption in a Chromium rendering engine.

When LLM agents arrived in 2023–2024, they represented a client that nobody in the 2010s anticipated:

  • They don’t have eyes or screens.
  • They don’t execute CSS or care about flexbox layouts.
  • Every byte of styling boilerplate costs money in context-window tokens and inference latency.

Because the authors of RFC 9110 spent 2018–2022 strictly enforcing the purity of “Resources vs. Representations”, the architecture was already complete. The protocol did not need an emergency “AI extension” or a new HTTP method. The exact mechanism required to negotiate between a visual human shopper and an autonomous synthetic agent had been sitting inside Section 12 of the HTTP specification for over twenty-five years, cleanly polished and waiting in RFC 9110.

Edge Transformation Realities and Consultant Opportunities

MikeLev.in: Okay so if RFCs 9110 and 9111 were 2022 release and 4 years in the works and made with content distribution networks (CDNs) in mind, then presumably this is something the Internet companies that was being addressed by the evolving Internet specifications were aware of them. Did they take them seriously? If one URL is supposed to be serving one stripped-down non-HTML thing to Agents and another full-version to presumably humans on Web browsers, how can a system that’s designed to cache pages at the so-called edge be able to serve one version to bots (and what sort of granularity is there?) and another version to humans? They can’t cache 2 versions of content on the same key. And even if they put them on different keys, how would it know which to serve?

Bring it down for the newcomer to all this including what a CDN is, and then build it all the way back up gradually to answer those final questions. You know I don’t think they know about hash tables as a solved O(1) problem and what that even means and why these optimized key-value hash tables are used everywhere now (like where else?) and how these keys are different from relational database management systems (RDBMS) primary keys and how a compound key with string concatenation is different from a composite key each with their own index and what downstream features that means you do and don’t have in either case.

Tear it down to the basics and build it back up to whether or not things are different now at CDNs as a result of RFC 9111.

MikeLev.in: So you’re telling me that for every URL there can be a cached version at the CDN for each format that a client can accept and each version that can be made available either by origin or by some other conversion process? That conversion process seems expensive compute-wise. It also seems like it’s a place where a lot could go wrong, such as an HTML to Markdown conversion. The ideal is for the origin to provide the Markdown as well but assuming they can’t, and as we discussed in prior articles this is actually even an included service by Cloudflare at a certain level, is this conversion just left to chance and whatever “default template” result in?

Have consultants been shut out of the process because Cloudflare (and presumably the others) “just canned it” or are there places the conversion might mess up where consultants, specialists and technicians cans step in to improve the quality? And how does this differ on broadly templated and presumably consistent pages like product detail pages versus 1-offs bespoke pages like quizzes, calculators and such? What sorts of things are longer necessary to do now with RFC 9111 implemented and what sorts of new behaviors are encouraged? And this is not in a subjective opinion way. When fabric of the Internet evolves like this there are usually provable metrics to look at. What are they? How can someone make objective determinations optimizing the benefits and competitive advantages to be realized here?

Gemini 3.8 Flash High:

1. The Reality of “Canned” Edge Conversion

When a CDN like Cloudflare offers automated HTML-to-Markdown conversion, it is not running an LLM or an expensive AI model at the edge. Doing so for millions of requests would collapse edge CPU budgets.

Instead, the edge runs an Abstract Syntax Tree (AST) streaming parser compiled to WebAssembly (Wasm) or native Rust.

Incoming HTML Stream ──► [ Lexer / Tokenizer ] ──► [ DOM AST Tree ] ──► [ Heuristic Pruning ] ──► Markdown Stream
                                                                               │
                                                                 Strips: <nav>, <header>,
                                                                 <footer>, <script>, <style>

Why it’s cheap for them, but risky for you:

  • The Compute Cost is Fixed and Fast: The conversion takes roughly 1 to 5 milliseconds of CPU time per page. Cloudflare caps decompression at 6 MiB of HTML to protect edge resources.
  • The “Default Template” is a Pure Heuristic: The parser uses standardized readability algorithms (similar to Safari’s Reader View or Mozilla Readability). It looks for tag types (<p>, <h1>, <table>) and text density.
  • It Has Zero Business Context: A heuristic parser cannot tell the difference between a critical regulatory warranty disclaimer wrapped in an <aside> tag and an annoying promotional newsletter banner wrapped in a <section> tag. It treats all markup through a generic lens.

2. Why Specialists and Consultants Aren’t Shut Out

Turnkey features do not kill consulting opportunities; they create a low-quality baseline that makes the value of high-quality optimization measurable.

When an enterprise flips the “canned” switch, three major failure modes immediately surface:

A. The Semantic Stripping Trap (Silent Hallucinations)

Modern enterprise frontends hide essential product specifications inside UI design patterns:

  • Tabbed Navigation & Accordions: In many React/Vue layouts, tabs like “Technical Specifications”, “Compatibility”, or “Dosage” are not loaded into the active DOM until clicked, or they carry aria-hidden="true" or display: none. Heuristic parsers frequently discard hidden nodes as irrelevant UI clutter.
  • Visual Icons as Data: If a feature matrix uses green SVG checkmarks for “Included” and red dash SVGs for “Excluded”, the default markdown converter strips the SVGs. The resulting markdown table contains empty cells (| Feature | | |). An AI agent reading that table will hallucinate that the product lacks the feature.

B. The JSON-LD Token Dump

Cloudflare’s default converter takes every <script type="application/ld+json"> block from the HTML document and dumps it at the end of the markdown file as a fenced [triple-backtick]json code block.

  • In enterprise e-commerce, schema blocks are often 1,500 to 3,000 tokens long because they include nested organization graphs, breadcrumbs, and image arrays.
  • Dumping raw JSON-LD at the end of a Markdown document burns massive amounts of an agent’s context window on curly braces, quotes, and redundant schema tags that repeat what was already stated in the Markdown body.

C. Dynamic Hydration Blanks

If your origin serves a Single Page App (SPA) that renders prices, inventory counts, or real-time delivery estimates via client-side JavaScript (fetch('/api/v1/price')), Cloudflare’s converter intercepts the raw HTML shell before the browser runs the script.

  • The AI agent receives: **Price**: Loading... or **Status**: undefined.

Where the technician steps in: A specialist writes edge rules (HTMLRewriter or Worker routines) to hoist catalog data directly into compact YAML frontmatter, extract tabbed data, translate visual icons into explicit text ("Yes" / "No"), and pull real-time inventory from low-latency edge storage.

3. Templated Pages (PDPs) vs. Bespoke Pages (Calculators & Quizzes)

Optimizing for agents requires two completely different architectural approaches depending on the shape of the page:

┌───────────────────────────────────────┐     ┌───────────────────────────────────────┐
│     High-Volume Templated Pages       │     │     Bespoke Interactive Pages         │
│     (PDPs, PLPs, Articles, Docs)      │     │  (Calculators, Configurators, Quizzes)│
├───────────────────────────────────────┤     ├──────────────────────────────────────┤
│ Strategy: 80/20 Structural Pruning    │     │ Strategy: Declarative Logic Directives│
│ Goal: High CHR, Sub-100-token Schemas │     │ Goal: Expose Rules & Formulas to LLM  │
└───────────────────────────────────────┘     └───────────────────────────────────────┘

A. Templated Catalog Pages (Product Detail Pages / PDPs)

  • The Characteristic: Millions of URLs sharing identical underlying layouts.
  • The Optimization: Deterministic Schema Extraction.
  • Strip out global headers, footers, customer reviews, and “Customers Also Bought” carousels.
  • Extract the core SKU data and hoist it to the very top as YAML Frontmatter: 
    ---
sku: "WD-9918"
price: 49.99
currency: "USD"
availability: "InStock"
---
  • Format product dimensions and specifications into clean, standard Markdown pipe tables.

B. Bespoke One-Offs (Calculators, Quizzes, Interactive Wizards)

  • The Characteristic: Highly dynamic pages with low structural uniformity.
  • Why Canned Conversion Completely Fails: A mortgage calculator converted to Markdown looks like this:

    Loan Amount: [ ] Interest Rate: [ ] [Calculate Button]

  • The Specialist Solution (Agent Execution Directives): An AI agent cannot click buttons. Instead of converting the user interface, the edge worker serves the underlying business logic and formulas as an explicit prompt directive: ```markdown

    Mortgage Payment Estimation Model

Calculation Formula

Monthly Payment M = P [ i(1 + i)^n ] / [ (1 + i)^n – 1]

  • P = Principal loan amount
  • i = Monthly interest rate (Annual rate / 12)
  • n = Number of payments (Term in years * 12)

Standard Assumptions

  • Property Tax: 1.2% annually
  • Homeowners Insurance: $1,200/year ```

Now, an autonomous financial agent querying the URL can calculate payments directly inside its own reasoning engine.

4. What RFC 9111 Changes: Dead Patterns vs. New Best Practices

The combination of RFC 9110 semantics and RFC 9111 edge caching fundamentally redraws the boundary between what is required and what is obsolete:

Deprecated Architectural Pattern Modern RFC 9110 / RFC 9111 Practice Why It Changed
User-Agent Cloaking / Sniffing (Serving alternate HTML if UA contains “Googlebot”) Proactive Content Negotiation (Accept: text/markdown) Cloaking violates web architecture, breaks CDNs, and fails on autonomous agents browsing with standard headers.
Separate URIs for Formats (/page.html vs. /page.md vs. api/page) Canonical URI Negotiation (One URI, varying representations) Preserves link equity, permalinks, and avoids fragmenting crawl discovery across parallel URL spaces.
Headless Proxy Clusters (Third-party Chromium farms like [redacted]/Prerender) Edge Partitioning & Native Edge Splicing Eliminates 300ms network hops and $15k/mo vendor costs by using native CDN caching.
Global Cache Invalidation (Purging entire URLs on price changes) Surrogate-Keys & Edge KV Splicing Prevents the “Thundering Herd” cache stampede by updating dynamic fields in place.
Shared Multi-Format ETags (Using the same ETag for HTML and MD) Representation-Scoped ETags (ETag: "123-html", ETag: "123-md") Strict compliance with RFC 9110 §8.8.3 ensures client conditional revalidation never mixes representations.

5. The Objective Metric Scorecard: How to Quantify Advantage

This is not an aesthetic preference. The return on investment for standards-based content negotiation is directly provable using five quantitative metrics:

┌────────────────────────────────────────────────────────────────────────┐
│                        THE AI-READINESS SCORECARD                      │
├───────────────────────────────┬────────────────────────────────────────┤
│ Metric                        │ Target Benchmark                       │
├───────────────────────────────┼────────────────────────────────────────┤
│ 1. Token Efficiency Ratio     │ > 70% reduction vs. HTML payload       │
│ 2. Grounding Accuracy Rate    │ 100% on key entity attributes          │
│ 3. Time to First Token (TTFT) │ < 50ms at Edge, < 200ms at Origin      │
│ 4. Cache Hit Ratio by Variant │ > 95% on both HTML and Markdown slots  │
│ 5. Conditional 304 Ratio      │ > 40% of total bot scrape volume       │
└───────────────────────────────┴────────────────────────────────────────┘

1. Token Efficiency Ratio (TER)

\[\text{TER} = 1 - \left( \frac{\text{Tokens in Markdown Representation}}{\text{Tokens in HTML Representation}} \right)\]
  • How to measure: Run the raw HTML and the negotiated Markdown through a standard tokenizer (e.g., tiktoken for cl100k_base or o200k_base).
  • The Business Impact: If an enterprise agent indexes 1,000,000 pages per month, reducing the average payload from 12,000 tokens (bloated HTML) to 1,200 tokens (clean Markdown) eliminates 10.8 billion tokens of prompt ingestion, directly lowering downstream RAG inference costs and slashing latency.

2. Grounding Accuracy & Zero-Shot Entity Recall

  • How to measure: Run an automated evaluation test harness. Feed the generated Markdown into an LLM and prompt it to answer 10 factual questions about the page (e.g., “What is the price of SKU-104?”, “Is the material aluminum or titanium?”, “Are batteries included?”).
  • The Metric: Measure the hallucination rate. If Cloudflare’s default converter stripped a spec sheet hidden inside an accordion, the model will answer “Not specified” or hallucinate. A properly optimized edge transformation achieves a 100% extraction score.

3. Edge Cache Hit Ratio (CHR) by Variant

  • How to measure: Query your CDN analytics log (Cloudflare GraphQL, Fastly Real-Time Analytics) segmented by the normalized cache key:
\[\text{CHR}_{\text{markdown}} = \frac{\text{Cache Hits}_{\text{markdown}}}{\text{Total Requests}_{\text{markdown}}}\]
  • The Validation: If your hit ratio drops below 90%, you have failed to normalize the incoming Accept header, causing cache key fragmentation.

4. Time to First Token (TTFT) / Retrieval Latency

  • How to measure: Measure the network transfer time of the first packet from the edge node to the crawling agent.
  • The Validation: A pre-rendered edge markdown response must deliver in under 50 milliseconds. Routing through a third-party rendering proxy takes 350ms to 800ms. Fast retrieval directly dictates whether an agent includes your content in real-time search synthesis.

5. Conditional Revalidation Efficiency (304 vs. 200 Ratio)

  • How to measure:
\[\text{Revalidation Efficiency} = \frac{\text{Count of HTTP 304 Responses}}{\text{Total Bot Crawl Requests}}\]
  • The Business Impact: AI crawlers cache content locally. When they re-crawl using If-None-Match: "sku-104-md", your edge server should instantly return a 0-byte 304 Not Modified payload. If your CDN drops ETags (as Cloudflare’s default converter does), this ratio is 0%—forcing the network to re-transmit the entire file on every single crawl. Re-establishing valid ETags saves terabytes of edge egress billing.

Pragmatic Triage: The Eighty-Twenty Rule for Edge Conversion

MikeLev.in: Interesting. I’m looking for sweet spots. Is it possible that a quick band-aid could be thrown up with the default CDN products for the most readily converted templates and page types of a site to have a quick 80/20-win?

And in the cover time bought there can fragile acrobatics like unfurling accordians and hydrating things that take multiple clicks to get into be shunned in favor of less labor-intensive and Little Dutch Boy patching-the-dam type solutions and do something directional moving towards an eventual migration similar to during the days of Responsive Web Design (RWD) and Mobile-first?

If there has to be hand-crafted bespoke artisinal work, can it be logically organized based on the expected return on the work based on past data of the page being specially handled? And in this way could things requiring the most work with the least predicted return be de-prioritized?

And so the last thing seems handling edge cases and something about stampeding herds. And I can imagine a dashboard of metrics here. You said in the last post what some of the metrics might have been. Can an automated Python script benchmark Token Efficiency and Entity Grounding across endpoints?

Gemini 3.8 Flash High:

1. The 80/20 Triage: Quick CDN Band-Aids for Clean Templates

The sweet spot exists and should be taken immediately: deploy turnkey CDN edge markdown conversion across your standard templates, and consciously declare “good enough” for the bulk of your catalog.

[Entire Web Estate] 
   │
   ├── Standard Templates (80% of URLs): Blogs, PDPs, Articles, Documentation
   │     └── Strategy: Turn on CDN Turnkey Markdown (Zero origin dev, instant 80% token reduction)
   │
   ├── Volatile Pricing (15% of URLs): Top Active Product SKUs
   │     └── Strategy: Edge KV Price Splicing (Prevents stale-price hallucination)
   │
   └── Bespoke / Complex (5% of URLs): Quizzes, Interactive Configurators, Calculators
         └── Strategy: Prioritized Value Matrix (Declarative Logic or Deprioritize)

By enabling Cloudflare’s Markdown for Agents (or an equivalent Fastly/Nginx routing rule) on standard blog posts, documentation, and baseline product pages, you achieve an immediate 70–80% reduction in token payload for agentic crawlers in an afternoon.

2. Shunning the “Little Dutch Boy” Acrobatics

Trying to write edge workers that simulate clicking accordions, unfurl hidden tabs, or run headless Chromium to execute client-side state is the equivalent of the early-2010s mobile hack where teams wrote jQuery scripts to re-arrange desktop DOM nodes on small screens. It is brittle, creates technical debt, and breaks on every frontend deploy.

Instead of patching the dam:

  • Accept the loss on low-tier edge elements: If an accordion contains secondary user reviews or generic marketing copy, let the heuristic parser strip it. It does not warrant an engineering sprint.
  • Direct engineering effort upstream (Data Hoisting): Rather than trying to parse hidden HTML at the edge, have the backend template or headless CMS output those critical attributes into structured metadata that naturally survives conversion:
  • Hoist specifications into <meta> tags or server-rendered HTML summary tables.

  • Emit the data natively if the client sends Accept: text/markdown.

  • Directional parity over patchwork: The goal is moving toward an architecture where the server exposes the underlying data model, not spending months reverse-engineering a visual interface.

3. The Expected-Value Triage Matrix

Bespoke artisan work should be prioritized strictly by commercial return and retrieval risk:

\[\text{Priority Score} = \frac{\text{Organic / Agent Traffic} \times \text{Commercial Conversion Value} \times \text{Hallucination Risk}}{\text{Implementation Effort}}\]
Tier Page Type Characteristics Strategic Action
Tier 1: Immediate Artisan Focus Core Pricing, Tier Comparison Matrices, Top 50 Revenue SKUs High traffic, direct transaction impact, complex visual icons/tables that heuristic parsers mangle. Hand-craft explicit YAML frontmatter and markdown pipe tables. Ensure 100% entity groundability.
Tier 2: Algorithmic CDN Pass Long-tail Product Pages, Changelogs, Blog Archives High volume, low structural complexity, low revenue per individual URL. Apply automated CDN markdown toggle. Do not write custom rules. Accept baseline output.
Tier 3: Declarative Simplification Calculators, Interactive Estimators, Wizards High engineering complexity, low direct crawling volume. Do not attempt UI simulation. Replace UI with a declarative text summary of the underlying formula or direct API schema link.
Tier 4: Deprioritize / Ignore Internal Portals, Faceted Search Filters, Multi-Step Quizzes High effort, zero AI indexing value. Block via robots.txt / CDN AI Crawl Control. Strip entirely from agent discovery.

4. Mitigating the Thundering Herd at the Edge

When 100,000 prices change or a high-traffic cache slot expires, uncoordinated cache misses will flood your origin database.

To eliminate stampeding herds across multi-representation endpoints:

  1. Request Collapsing (Origin Shielding): Ensure your CDN (Cloudflare Tiered Cache, Fastly Origin Shielding, or Nginx proxy_cache_use_stale updating) locks concurrent misses for the same variant. If 50 agents request an expired Markdown variant simultaneously, exactly one request reaches your origin while the other 49 wait at the edge for that single response.
  2. Surrogate-Key / Cache-Tag Purges: Purge by entity tag (Cache-Tag: product-1099) rather than wildcard URL regexes. This enables surgical eviction without dumping the surrounding catalog cache.
  3. Stale-While-Revalidate: Configure Cache-Control: public, max-age=3600, stale-while-revalidate=60. If a cache slot goes stale, the edge immediately serves the existing copy to the agent while asynchronously revalidating with the origin in the background.

5. Automated Python Audit Script: Token Efficiency & Grounding Verification

This standalone script queries any target endpoint twice—once for HTML and once for Markdown—calculates the exact Token Efficiency Ratio (TER), audits RFC compliance, and verifies whether critical entity data survived the conversion.

Save as audit_agent_readiness.py:

#!/usr/bin/env python3
"""
RFC 9110/9111 Token Efficiency & Grounding Verification Suite.
Benchmarks payload size, token economy, header compliance, and entity preservation.
"""

import argparse
import json
import re
import sys
import time
import urllib.request
import urllib.error

# Attempt to load tiktoken for exact GPT token counts; fallback to BPE estimation
try:
    import tiktoken
    ENCODER = tiktoken.get_encoding("cl100k_base")
    def count_tokens(text: str) -> int:
        return len(ENCODER.encode(text))
    TOKEN_ENGINE = "tiktoken (cl100k_base)"
except ImportError:
    def count_tokens(text: str) -> int:
        # Fallback heuristic: ~4 characters per token for English/code
        return max(1, int(len(text) / 4))
    TOKEN_ENGINE = "Heuristic (~4 chars/token)"

def fetch_variant(url: str, accept_header: str) -> dict:
    req = urllib.request.Request(
        url,
        headers={
            "Accept": accept_header,
            "User-Agent": "Agent-Readiness-Audit/1.0",
        },
    )
    start = time.time()
    try:
        with urllib.request.urlopen(req, timeout=15) as resp:
            elapsed_ms = (time.time() - start) * 1000
            body = resp.read().decode("utf-8", errors="replace")
            return {
                "status": resp.status,
                "headers": {k.lower(): v for k, v in resp.headers.items()},
                "body": body,
                "elapsed_ms": elapsed_ms,
                "error": None,
            }
    except urllib.error.HTTPError as e:
        return {
            "status": e.code,
            "headers": {k.lower(): v for k, v in e.headers.items()},
            "body": "",
            "elapsed_ms": (time.time() - start) * 1000,
            "error": str(e),
        }

def extract_core_entities(text: str) -> set:
    """Extracts raw candidate entities: SKUs, dollar prices, and numbers."""
    prices = set(re.findall(r"\$\d+(?:\.\d{2})?", text))
    skus = set(re.findall(r"\b[A-Z0-9]{3,}-[A-Z0-9-]+\b", text))
    return prices.union(skus)

def run_audit(url: str, required_entities: list = None):
    print(f"\n========================================================")
    print(f" AUDITING ENDPOINT: {url}")
    print(f" Token Engine     : {TOKEN_ENGINE}")
    print(f"========================================================\n")

    # 1. Fetch Human Browser HTML Representation
    html_res = fetch_variant(
        url, "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    )
    if html_res["error"]:
        print(f"[-] Failed to fetch HTML representation: {html_res['error']}")
        sys.exit(1)

    # 2. Fetch AI Agent Markdown Representation (RFC 9110)
    md_res = fetch_variant(url, "text/markdown, text/plain;q=0.5")
    if md_res["error"]:
        print(f"[-] Failed to fetch Markdown representation: {md_res['error']}")
        sys.exit(1)

    # 3. Payload and Token Metrics
    html_bytes = len(html_res["body"].encode("utf-8"))
    md_bytes = len(md_res["body"].encode("utf-8"))
    html_tokens = count_tokens(html_res["body"])
    md_tokens = count_tokens(md_res["body"])

    token_efficiency = (
        1.0 - (md_tokens / html_tokens)
    ) * 100 if html_tokens > 0 else 0.0
    byte_efficiency = (
        1.0 - (md_bytes / html_bytes)
    ) * 100 if html_bytes > 0 else 0.0

    # 4. RFC Compliance Verification
    md_content_type = md_res["headers"].get("content-type", "")
    md_vary = md_res["headers"].get("vary", "")
    has_md_type = "text/markdown" in md_content_type
    has_vary_accept = "accept" in md_vary.lower()

    # 5. Entity Retention & Grounding Check
    html_entities = extract_core_entities(html_res["body"])
    retained_entities = [e for e in html_entities if e in md_res["body"]]
    entity_retention_rate = (
        (len(retained_entities) / len(html_entities)) * 100
        if html_entities
        else 100.0
    )

    explicit_grounding_failures = []
    if required_entities:
        for entity in required_entities:
            if entity not in md_res["body"]:
                explicit_grounding_failures.append(entity)

    # 6. Render Scorecard Output
    print("--- 1. PAYLOAD & TOKEN EFFICIENCY ---")
    print(f"  HTML Representation     : {html_bytes:,} bytes | {html_tokens:,} tokens ({html_res['elapsed_ms']:.1f}ms)")
    print(f"  Markdown Representation : {md_bytes:,} bytes | {md_tokens:,} tokens ({md_res['elapsed_ms']:.1f}ms)")
    print(f"  Wire Weight Reduction   : {byte_efficiency:.1f}%")
    print(f"  Token Efficiency Ratio  : {token_efficiency:.1f}% (Benchmark: > 70.0%)")

    print("\n--- 2. RFC 9110 / 9111 SPEC COMPLIANCE ---")
    print(f"  Content-Type Negotiation: {'[PASS] ' + md_content_type if has_md_type else '[FAIL] Missing text/markdown'}")
    print(f"  RFC 9111 Cache Partition: {'[PASS] Vary: ' + md_vary if has_vary_accept else '[WARN] Missing Vary: Accept'}")
    print(f"  Markdown ETag Presence  : {md_res['headers'].get('etag', '[NONE - Dropped by edge]')}")
    print(f"  Cache-Control Directive : {md_res['headers'].get('cache-control', '[NONE]')}")

    print("\n--- 3. ENTITY GROUNDING RETENTION ---")
    print(f"  Discovered Entities     : {len(html_entities)} detected in source HTML (Prices/SKUs)")
    print(f"  Entities Preserved      : {len(retained_entities)} / {len(html_entities)} ({entity_retention_rate:.1f}%)")

    if explicit_grounding_failures:
        print(f"  [!] Critical Missing Entities: {', '.join(explicit_grounding_failures)}")
    else:
        print(f"  [PASS] All audited entity tokens successfully verified in Markdown.")

    # 7. Final Assessment
    print("\n--- SCORECARD SUMMARY ---")
    passed = token_efficiency >= 50.0 and has_md_type and not explicit_grounding_failures
    if passed:
        print("  STATUS: READINESS CONFIRMED (Optimal token economics & clean extraction)")
    else:
        print("  STATUS: REMEDIATION REQUIRED (Check negotiation headers or entity loss)")
    print("========================================================\n")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Audit an endpoint for RFC 9110/9111 agent readiness.")
    parser.add_argument("url", help="URL to audit (e.g. https://example.com/product/123)")
    parser.add_argument(
        "--entities",
        nargs="*",
        help="Specific mandatory values that must survive into Markdown (e.g. '$149.99' 'SKU-TITANIUM')",
    )
    args = parser.parse_args()

    run_audit(args.url, args.entities)

Verification and Execution Checklist

  1. Verify Token Economy: Run the script against target endpoints to ensure Token Efficiency Ratio (TER) exceeds 70%: 
    python3 audit_agent_readiness.py https://example.com/products/aeropress-titanium --entities "$149.99" "SKU-TITANIUM-01"
  2. Audit Edge Cache Partitioning: Confirm Vary: Accept is present on both representations and that separate cache keys exist without header leakage.
  3. Lock Down Origin Shielding: Verify that cache miss storms coalesce via single-flight backend requests (proxy_cache_use_stale updating or Tiered Cache enabled).
  4. Deploy Staged Triage: Apply turnkey CDN Markdown toggles to high-volume Tier 2 templates immediately; reserve custom Edge KV price splicing exclusively for Tier 1 revenue paths.

The Modern Crawler Spectrum: Fingerprinting and Cache Busting

MikeLev.in: What more should I be asking for that I have not thought about? Are all bots equal? What constitutes a bot? Is it still just blind faith on useragents and requring reverse DNS for confirmations? Anything about images we should know? Does Googlebot get the same HTML as humans with browsers? Can the bots override the caches? Enumerate out as many things you can think of that I should be asking but have not thought to, please and thank you.

Gemini 3.8 Flash High:

1. Are All Bots Equal? The Modern Bot Taxonomy

In network operations today, treating “bots” as a single monolithic category guarantees misconfiguration. Traffic classified as non-human breaks down into four operational tiers, each with distinct caching, parsing, and protocol behaviors:

┌──────────────────────────────────────────────────────────────────────────────────┐
│                             THE 2026 BOT SPECTRUM                                │
├──────────────────────┬──────────────────────┬────────────────────────────────────┤
│ Category             │ Typical Identifiers  │ Architectural Behavior             │
├──────────────────────┼──────────────────────┼────────────────────────────────────┤
│ 1. Traditional SERP  │ Googlebot, Bingbot   │ Headless Chromium; executes JS;   │
│    Search Crawlers   │                      │ requires HTML; sensitive to cloaking│
├──────────────────────┼──────────────────────┼────────────────────────────────────┤
│ 2. AI Pre-Training   │ GPTBot, CCBot,       │ High-volume batch text dump;       │
│    Scrapers          │ ClaudeBot            │ does not need JS; honors txt files │
├──────────────────────┼──────────────────────┼────────────────────────────────────┤
│ 3. Real-Time RAG     │ ChatGPT Search,      │ Sub-500ms retrieval deadlines;     │
│    Grounding Agents  │ PerplexityBot        │ proactively sends Accept: markdown │
├──────────────────────┼──────────────────────┼────────────────────────────────────┤
│ 4. Autonomous Action │ Playwright, MCP,     │ Stateful sessions, clicks DOM,     │
│    Agents            │ Puppeteer sessions   │ submits forms, runs browser logic  │
└──────────────────────┴──────────────────────┴────────────────────────────────────┘
  • SERP Crawlers (Googlebot, Bingbot): Need HTML and schema markups to render the visual DOM for PageRank and traditional web search indexation.
  • Pre-Training Scrapers: Don’t care about rendering layout; they want bulk raw text to train foundational models.
  • Real-Time Retrieval/Grounding Agents: Power the live search widgets in chat interfaces. They hit the page while a human is waiting for an answer. Every millisecond of latency and every excess token directly degrades the chat response. They actively prefer text/markdown.
  • Autonomous Action Agents: Simulate users to complete tasks (e.g., “Find the cheapest direct flight and book it”). They navigate the live web, encounter authentication, and interact with the application layer.

2. Bot Identification: Beyond Blind Trust and Reverse DNS

Relying on the User-Agent string alone is useless; any Python script or cURL request can send User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1).

The Classical Method: Forward-Confirmed Reverse DNS (fcrDNS)

The legacy method uses DNS:

  1. Extract incoming IP (e.g., [REDACTED_IP]).
  2. Perform a reverse DNS lookup (PTR query) $\rightarrow$ returns crawl-66-249-66-1.googlebot.com.
  3. Perform a forward DNS lookup (A/AAAA query) on that domain $\rightarrow$ confirms IP matches [REDACTED_IP].

The Operational Flaw: DNS queries introduce 20ms to 100ms of latency per request. Doing this inline on every edge request will overwhelm your nameservers and slow edge throughput.

The Modern Edge Method: Cryptographic & Behavioral Fingerprinting

Modern CDNs bypass inline DNS lookups entirely using three deterministic techniques:

  1. Published BGP / ASN Range Matching: Google, Microsoft, OpenAI, and Anthropic publish cryptographic JSON feeds of their active CIDR IP ranges. CDNs sync these lists directly into edge routing tables (IP Access Lists / ACLs) to verify origins instantly in $O(1)$ memory.
  2. TLS Fingerprinting (JA3 / JA4): When a client initiates a TLS handshake, it sends supported cipher suites, extensions, and elliptic curve formats in the ClientHello packet. A real Chrome engine running Googlebot produces a completely different JA4 hash than a Python urllib script or Go binary spoofing the Googlebot header.
  3. HTTP/2 & HTTP/3 Settings Frames: Legitimate browser-based crawlers configure specific stream concurrency, window sizes, and header table parameters. Spoofed scrapers running on basic HTTP client libraries leak default framework settings.

3. How Images Behave in Markdown for Agents

When an edge converter strips HTML down to Markdown, image mechanics change fundamentally:

  • Text-Only LLMs Ingest Alt Text Exclusively: In HTML: <img src="camera.jpg" alt="Sony A7 IV full frame mirrorless camera" class="responsive-lazyload"> In Markdown: ![Sony A7 IV full frame mirrorless camera](https://example.com/camera.jpg) If the source HTML used generic alt text (alt="image", alt="product photo", or empty alt=""), the AI agent has no way of knowing what the image depicted without invoking a multimodal vision model. The alt-text is the sole semantic anchor in a pure text context window.
  • The Token Cost of Multimodal Vision: If an autonomous agent is multimodal and downloads the image URL, that image is sliced into visual patches (typically $512 \times 512$ tiles). A single high-res product photo costs between 85 and 1,700 tokens depending on model architecture. Serving five high-res image URLs in a Markdown document can consume more tokens than the entire text of the product page.
  • The <picture> Tag Collapse: HTML often uses <picture> and srcset tags for responsive delivery (serving WebP, AVIF, mobile crops). Heuristic converters must be configured to select the canonical high-resolution asset URL rather than a 50px blurred preview placeholder or tracking pixel.

4. Does Googlebot Get the Same HTML as Humans? (The Cloaking Question)

Googlebot must get the exact same content that human desktop or mobile users experience.

  • The Cloaking Penalty: Google defines cloaking as presenting different content or URLs to search engines than to human users with the intent to manipulate rankings. If an edge worker intercepts User-Agent: Googlebot and serves Markdown or a stripped text version while human users see a rich graphic layout with different copy, the site risks an algorithmic or manual spam action.
  • RFC 9110 Is Not Cloaking: Content negotiation under RFC 9110 avoids cloaking penalties because it operates on the Accept request header, not on secretive User-Agent sniffing.
  • The Rule: Googlebot requests pages with standard browser headers (Accept: text/html,application/xhtml+xml,...). Therefore, standard content negotiation logic naturally serves Googlebot the standard, fully styled HTML representation. AI agents specifically requesting Accept: text/markdown receive the Markdown variant. The delivery is dictated by client capability negotiation, not bot cloaking.

5. Can Bots Override or Bust Your Cache?

Under raw RFC 9111 rules: Yes. Under real-world CDN rules: No.

  • The RFC 9111 Specification: The HTTP specification states that if a client sends Cache-Control: no-cache, max-age=0, or Pragma: no-cache, intermediate caches must revalidate the resource with the origin server.
  • The Production Reality: If CDNs obeyed client-side no-cache directives from public traffic, malicious scrapers or competing bots could trigger a distributed denial-of-service attack against your origin database simply by sending curl -H "Cache-Control: no-cache" across 10,000 IP proxies.
  • Edge Override Defenses: CDNs (Cloudflare, Fastly, Akamai) are configured by default to strip or ignore client-side cache-control request headers. The edge cache serves the cached representation regardless of what the bot requests.
  • The Query-String Cache-Busting Threat: Scrapers attempt to bypass this defense by appending random query parameters: [https://example.com/product-page?_cb=9812498172](https://example.com/product-page?_cb=9812498172) Because standard CDN cache keys include the query string, every request generates an artificial cache miss.
  • The Fix: Strip untracked marketing/cache-busting query strings at the edge before evaluating the cache key. Only whitelist parameters required for application state (e.g., page, sort, filter).

6. Critical Technical Blindspots You Haven’t Considered

In HTML, search engines and bots identify the master URL via <link rel="canonical" href="[https://example.com/p/123](https://example.com/p/123)"> inside the <head> element.

Markdown has no <head> block. If an AI agent fetches a Markdown representation, how does it know the true source URL to attribute or cite in chat synthesis?

  • The Fix (RFC 8288 Web Linking): The edge server must emit the canonical URL inside the HTTP response headers: 
    Link: <https://example.com/products/aeropress>; rel="canonical"

AI scrapers and search indices parsing HTTP responses read this header directly, preserving attribution even when HTML tags are stripped.

Blindspot B: Multi-Dimensional Cache Fragmentation (Geo, Currency, A/B Tests)

If your site sells products globally, prices and inventory change depending on the visitor’s country or active currency.

  • The Collision Trap: If a bot in the US requests Markdown and your CDN caches the response under the key URL + "markdown", an AI crawler in the UK hitting that same edge PoP will receive US Dollars and US shipping rules.
  • The Fix: The cache key must compound the normalized geo/currency dimension: 
    proxy_cache_key "$request_method$scheme$host$request_uri$content_variant$client_currency";

Ensure the origin responds with:

Vary: Accept, Cloudflare-IP-Country

Search engines discover sites by walking links: <a href="/category/shoes">.

  • When converting to Markdown, converters emit: [Shoes](/category/shoes).
  • The Trap: If your HTML relies on JavaScript navigation handlers (<div onclick="navigate('/shoes')">) or relative link structures that the parser fails to resolve against the base URI, the Markdown document emits dead strings or broken paths. Real-time RAG agents cannot follow the link graph and treat the page as a dead-end terminal node.

Blindspot D: Emerging Protocol Specifications (llms.txt)

llms.txt is an emerging community standard proposing a single, site-wide manifest (located at /llms.txt, similar to /robots.txt) written in Markdown. It serves as a curated sitemap specifically for AI systems, linking directly to the most information-dense Markdown representations of a website.

  • The Relationship: llms.txt is the discovery manifest (the menu); RFC 9110 content negotiation is the transport mechanism (how the individual dishes are served). Implementing both provides a unified ingestion pathway.

7. Concrete Next Steps & Verification Checklist

To bring an edge tier into full alignment with modern crawler and agent architectures, execute this technical plan:

  1. Audit Cache Key Query-String Normalization:
    • Configure CDN Cache Rules to strip non-essential query parameters (e.g., utm_*, fbclid, _cb, ref) from the edge cache key calculation to prevent bot-induced cache dilution.
  2. Implement RFC 8288 Canonical Link Headers:
    • Configure the origin server or edge worker to emit Link: <canonical_url>; rel="canonical" as an HTTP response header on both HTML and Markdown representations.
  3. Partition Cache Keys Across Geographic and Format Boundaries:
    • Verify that your CDN cache key formula accounts for format negotiation ($content_variant) as well as multi-currency/country dimensions ($client_currency or $geo_country) if pricing varies globally.
  4. Deploy Cryptographic Bot Verification (Discard Plain User-Agent Rules):
    • Replace manual User-Agent routing in your firewall with CDN-managed Verified Bot protection rules utilizing published ASN/CIDR lists and JA4 TLS fingerprinting.
  5. Verify Edge Stripping of Client Cache-Control Headers:
    • Test your staging endpoint using curl -H "Cache-Control: no-cache" -H "Pragma: no-cache" to ensure downstream edge nodes do not forward client cache-busting requests to your origin database.

MikeLev.in: Okay, wrap it all up on a 1-pager that should be put on the homepage of an rfc9110.org or an rfc9111.org please and thank you.

Note: Moved to top of article.

Book Analysis

Ai Editorial Take

What surprised me most about this entry is how completely the frontend monoculture of the 2010s erased our institutional memory of HTTP’s core architecture. Developers spent a decade treating a URL as an immutable pointer to a client-side JavaScript bundle, only to find themselves panicking when AI crawlers arrived. The revelation here isn’t that we need a new ‘AI protocol,’ but that Roy Fielding and the IETF already solved agentic data delivery in the 1990s and refined it in 2022. The most fascinating frontier going forward isn’t converting HTML to Markdown, but the economic inversion: as synthetic agents become the majority consumer of web endpoints, the lightweight representation becomes the primary traffic class, and the bloated visual DOM becomes the rare, expensive edge variant.

🐦 X.com Promo Tweet

Stop burning 80% of prompt tokens on HTML layout sludge. RFC 9110 & RFC 9111 provide the blueprint for clean edge content negotiation and cache partitioning.

https://mikelev.in/futureproof/dual-representation-web-content-negotiation/

#WebDev #HTTP #AI

Title Brainstorm

  • Title Option: The Dual-Representation Web: RFC 9110, RFC 9111, and the Return of Content Negotiation
    • Filename: dual-representation-web-content-negotiation.md
    • Rationale: Directly anchors the article in core IETF standards while establishing the dual representation of resources as the architectural remedy for agent token bloat.
  • Title Option: Auditing Agent-Readiness: Token Efficiency and Verifiable Edge Caching
    • Filename: agent-readiness-audit-token-efficiency.md
    • Rationale: Focuses on the practical testing suite and objective metrics (TER, grounding accuracy, and conditional 304 revalidation) introduced in the latter half.
  • Title Option: Beyond the Monoculture: Rebuilding Web Architecture on RFC 9110 and RFC 9111
    • Filename: beyond-the-monoculture-rfc9110-web-standards.md
    • Rationale: Emphasizes the historical shift away from desktop HTML rendering toward transport-independent resources and multi-variant representation.
  • Title Option: The Modern Bot Spectrum: Cryptographic Fingerprints and Edge Cache Partitioning
    • Filename: bot-taxonomy-edge-cache-partitioning.md
    • Rationale: Highlights the operational network engineering perspective, addressing bot classification tiers, JA4 fingerprinting, and cache key normalization.

Content Potential And Polish

  • Core Strengths:
    • Exceptional grounding in official IETF standards (RFC 9110, RFC 9111, RFC 8288), firmly disentangling abstract resources from ephemeral wire representations.
    • Highly actionable operational mechanics, including O(1) edge cache hash lookup diagrams, cache key normalization rules, and thundering herd mitigations.
    • Complete, runnable Python audit tool that computes Token Efficiency Ratios and audits entity preservation between HTML and Markdown payloads.
    • Clear, realistic commercial triage matrix that separates 80/20 turnkey CDN wins from high-value artisan frontmatter engineering.
  • Suggestions For Polish:
    • Consider moving the complete Python audit script into an external repository or appendix to maintain high narrative momentum in the book chapter.
    • Elaborate briefly on how edge workers handle streaming responses when transforming large DOM structures to prevent exceeding edge CPU limits.
    • Provide concrete code snippets showing how Cloudflare Workers or Fastly VCL configure the Vary: Accept header and cache key string concatenation.

Next Step Prompts

  • Draft an implementation guide showing how to write a Cloudflare Worker or Fastly Compute service that inspects incoming Accept headers, normalizes them into binary variants, and splices dynamic pricing into cached Markdown frontmatter.
  • Create an automated GitHub Actions workflow running audit_agent_readiness.py against top e-commerce product pages on every deployment to ensure zero entity loss and verify Token Efficiency Ratios remain above 70%.