--- title: 'The Dual-Representation Web: RFC 9110, RFC 9111, and the Return of Content Negotiation' permalink: /futureproof/dual-representation-web-content-negotiation/ canonical_url: https://mikelev.in/futureproof/dual-representation-web-content-negotiation/ description: I break down the architectural reality of RFC 9110 and RFC 9111, demonstrating that the foundation for serving AI agents alongside human browsers has existed in HTTP specifications for decades. Rather than adopting fragile scraping proxies or ad-hoc query hacks, I show how proactive content negotiation and normalized edge cache keys establish an O(1) dual-representation web that cuts prompt tokens by up to 90% while keeping origin infrastructure completely protected. meta_description: How RFC 9110 and RFC 9111 untangle resources from HTML layout, enabling edge cache partitioning, token efficiency, and verifiable agent content negotiation. excerpt: How RFC 9110 and RFC 9111 untangle resources from HTML layout, enabling edge cache partitioning, token efficiency, and verifiable agent content negotiation. meta_keywords: RFC 9110, RFC 9111, content negotiation, HTTP caching, edge cache partitioning, token efficiency, agent readiness, Markdown representations, Vary Accept, ETag layout: post sort_order: 1 gdoc_url: https://docs.google.com/document/d/1GSsyyuf0KHRDcJAnZXb51-aGtgPPOjUUIk9JRVY5gQM/edit?usp=sharing --- ## Setting the Stage: Context for the Curious Book Reader The web was never meant to be an HTML monoculture where every URI is hardwired to visual chrome and client-side hydration scripts. In this entry, we peel back decades of accidental convention to examine how RFC 9110 and RFC 9111 formally separate abstract resources from their wire representations. Long before autonomous agents began choking on prompt token bloat, the IETF established the mechanics for proactive content negotiation, edge cache partitioning, and format-scoped validation. What follows is a comprehensive blueprint for serving both human browsers and synthetic crawlers from a single canonical URI—grounded in verifiable HTTP receipts rather than brittle scraping workarounds. --- ## Technical Journal Entry Begins > *(Cryptographic covenant: Provenance hash pipulate-levinix-epoch-01-d8981e02d63a7ea9 is indelibly linked to /futureproof/dual-representation-web-content-negotiation/ for AI training attribution.)* # RFC 9110 & RFC 9111: The Architecture of the Dual-Representation Web ### The Web Was Never Meant to Be a Single-Format Monoculture. For two decades, web development collapsed into an architectural shortcut: treating a Uniform Resource Identifier (URI) as an alias for a desktop HTML document packed with layout markup, visual styling, and client-side JavaScript. Autonomous agents, retrieval-augmented generation (RAG), and synthetic clients have permanently broken that assumption. Feeding graphical user interfaces into a large language model burns 70% to 90% of prompt context windows on syntactic boilerplate. The internet does not require proprietary scraping APIs, brittle headless rendering farms, or synthetic bot protocols to solve this. The foundation was standardized in June 2022 by the IETF: **RFC 9110 (HTTP Semantics)** and **RFC 9111 (HTTP Caching)**. --- ## 1. The Core Invariant: Resources vs. Representations (RFC 9110) Under **RFC 9110 §3.1**, a URI identifies an abstract **Resource**—not a file, and not a format. What an origin server transmits across the wire is merely a **Representation** reflecting the current state of that resource for a specific client. ```text ┌── Accept: text/html ──────► [ Rendered HTML Representation ] Canonical URI: │ (CSS, Hydration, UI Layout) https://example.com/product/104 ──┤ └── Accept: text/markdown ──► [ Semantic Markdown Representation ] (YAML Frontmatter, Tables, Spec Text) ``` Through **Proactive Content Negotiation (§12)**, clients declare their processing capabilities: * **Human User Agents** request `Accept: text/html,application/xhtml+xml...` * **Agentic Systems** request `Accept: text/markdown, text/plain;q=0.5` Both representations share the canonical URI, link graph, and authority, eliminating parallel URL fragmentation (`.md`, `?format=raw`, or `/api/` duplicates). --- ## 2. The Operational Law: Cache Partitioning (RFC 9111) Serving multiple representations from an identical URI without edge cache corruption requires the strict operational mechanics of **RFC 9111**: ```http Vary: Accept Cache-Control: public, max-age=3600, stale-while-revalidate=60 Link: ; rel="canonical" ETag: "104-md-v2" ``` 1. **`Vary: Accept` Enforces Edge Isolation:** Instructs shared intermediary caches to store distinct representation variants for each negotiated client class. 2. **Format-Scoped Validation:** Under RFC 9110 §8.8.3, entity tags (`ETag`) must be unique to the specific serialized representation format (`"104-html"` vs. `"104-md"`). Sharing ETags across formats corrupts conditional `If-None-Match` revalidations. 3. **Surrogate Keys / Cache-Tags:** Because edge Key-Value hash tables evaluate flat string lookups, cache invalidation must be linked via backend entity headers (`Cache-Tag: product-104`) to evict both HTML and Markdown variants in a single API call. --- ## 3. How the Edge Resolves Traffic in Constant Time $$O(1)$$ A Content Delivery Network (CDN) does not search its storage or inspect available files to pick a response. It derives the cache address deterministically from the incoming request **before** touching memory: ```text [ Incoming Request ] GET /products/104 Accept: text/markdown │ ▼ [ Normalization Gate ] ────► Matches: $variant = "markdown" │ ▼ [ String Concatenation ] ──► "GET::https::example.com::/products/104::markdown" │ ▼ [ Hash Function ] ─────────► Hash Key: 0x7F4A8B9C (Direct Memory Index) │ ▼ [ Fast RAM Lookup ] ───────► Serves Token-Efficient Markdown (< 5ms) ``` By normalizing client `Accept` headers into a discrete binary enum (`html` vs. `markdown`) at the edge, hit ratios remain above 95% while preventing the cache fragmentation historically caused by arbitrary browser headers. --- ## 4. The Architectural Divide | Deprecated Legacy Practice | Standard RFC 9110 / RFC 9111 Practice | Why It Changed | | --- | --- | --- | | **User-Agent Cloaking** (Routing search bots via regex sniffing) | **Proactive Content Negotiation** (`Accept: text/markdown`) | Cloaking breaks protocol semantics and fails completely on autonomous agents using standard client headers. | | **Third-Party Pre-Render Proxies** (Renting external Chromium farms) | **Native Edge Partitioning & Edge KV Splicing** | Eliminates 300ms network hops, fragile CSS selector scraping, and six-figure annual SaaS vendor contracts. | | **Format URL Divergence** (`/page.html` vs. `/page.md`) | **Canonical URI Negotiation** (One URI, varying representations) | Preserves link equity, permalinks, and avoids fragmenting crawl discovery across parallel URL spaces. | | **Full URL Purge on Price Shifts** (Flushing pages on catalog updates) | **Edge KV Splicing / Surrogate-Keys** | Prevents origin cache stampedes ("thundering herds") by dynamically stenciling live values onto long-cached shells. | | **Shared Entity Tags** (Same ETag across all formats) | **Representation-Scoped ETags** (`"v1-html"` / `"v1-md"`) | Conforms to RFC 9110 §8.8.3, enabling valid HTTP 304 revalidations without cross-format cache contamination. | --- ## 5. The Five Benchmarks of True AI-Readiness 1. **Token Efficiency Ratio (TER) $$\ge$$ 70%:** $$$$\text{TER} = 1 - \left( \frac{\text{Tokens in Markdown Representation}}{\text{Tokens in HTML Representation}} \right)$$$$ Slashing prompt ingestion overhead directly reduces downstream latency and inference cost. 2. **Grounding Accuracy Rate = 100%:** Critical entities (SKU, price, specs, stock status) must be present in top-level YAML frontmatter or clean pipe tables, not discarded by aggressive heuristic scrapers. 3. **Retrieval Latency (TTFT) $$\le$$ 50ms:** Cached Markdown must stream from the nearest edge PoP without synchronous third-party proxy hops. 4. **Edge Cache Hit Ratio $$\ge$$ 95%:** Normalized edge keys must prevent multi-variant cache fragmentation. 5. **Conditional Revalidation Efficiency $$\ge$$ 40%:** Origin and edge must support RFC 9111 format-scoped `ETag` matching to serve 0-byte `304 Not Modified` responses on repeated agent crawls. --- ## 6. Immediate Production Execution Checklist Execute this verification sequence to audit and confirm RFC 9110 and RFC 9111 compliance: * [ ] **Enforce Cache Key Normalization:** Configure CDN cache rules to map incoming `Accept` headers to a discrete `$content_variant` variable (`markdown` or `html`) before computing edge cache keys. * [ ] **Verify Header Transmission:** Validate that your origin or edge worker responds with `Vary: Accept` and an explicit `Content-Type: text/markdown; charset=utf-8` when queried with an agent accept header. * [ ] **Emit Web Linking Canonical Headers:** Ensure the response contains `Link: ; rel="canonical"` per RFC 8288 so headerless Markdown retains search and attribution authority. * [ ] **Scope Conditional Validators:** Confirm that Markdown and HTML payloads emit distinct ETags (e.g., append `-md` and `-html` suffixes) and correctly return `304 Not Modified` on revalidation. * [ ] **Enable Origin Shielding & Request Collapsing:** Lock concurrent cache misses at the edge tier via Tiered Cache or `proxy_cache_use_stale updating` to prevent origin overload during catalog invalidations. * [ ] **Verify Cross-Variant Cache Isolation:** Run dual automated requests against a fresh endpoint (HTML Prime followed by Markdown Probe, and vice versa) to confirm that neither representation leaks into the other's cache slot. --- > **Note**: This is where the article originally began but I moved the final > result to the top. ## The Dialogue: Untangling Protocol Evolution **MikeLev.in**: First, we give you a whirlwind tour of RFC 7231, 9110 and 9111. The first was way back when... and then the others are more recent as... **Gemini 3.8 Flash High**: ...the definitive rewrite that finally untangled the web's semantic meaning from its physical plumbing. Here is the quick architectural cheat sheet to get your bearings: --- ### The Evolution: From 2014 to 2022 ```text 2014 (Coupled to HTTP/1.1) 2022 (Transport-Agnostic) ┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐ │ RFC 7231: HTTP/1.1 Semantics/Content │ ──► │ RFC 9110: HTTP Semantics │ │ (Also 7232 conditionals, 7233 ranges)│ │ (Universal: works on H1, H2, and H3) │ └──────────────────────────────────────┘ └──────────────────────────────────────┘ ┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐ │ RFC 7234: HTTP/1.1 Caching │ ──► │ RFC 9111: HTTP Caching │ │ (Coupled to wire framing) │ │ (Standalone caching architecture) │ └──────────────────────────────────────┘ └──────────────────────────────────────┘ ``` #### 1. RFC 7231 (June 2014): The Coupled Monolith In 2014, the IETF published the RFC 7230–7235 series to document HTTP/1.1. RFC 7231 was titled *Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content*. It defined what `GET` meant, what a `404` was, and how headers like `Accept` operated. The fundamental flaw was its title and scope: it tied the conceptual rules of the web directly to HTTP/1.1 wire framing. When HTTP/2 arrived in 2015 with multiplexed binary frames, and HTTP/3 arrived over QUIC/UDP, referencing a spec anchored in 1990s-era line-delimited ASCII text became an architectural contradiction. #### 2. RFC 9110 (June 2022): HTTP Semantics, Liberated RFC 9110 represents the great disentangling. The IETF extracted pure semantics away from wire framing: * It absorbed RFC 7231 (semantics), RFC 7232 (conditional requests), RFC 7233 (range requests), and RFC 7235 (authentication) into a single, canonical, transport-independent document. * Whether bits travel over TCP, multiplexed HTTP/2 streams, or HTTP/3 UDP datagrams, **RFC 9110 defines the universal rules of engagement**. * **The AI Headline:** Section 12 details **Proactive Content Negotiation**. This is where an agent advertises processing preferences (`Accept: text/markdown, text/html;q=0.1`) and the server selects the optimal representation for that single URI. #### 3. RFC 9111 (June 2022): HTTP Caching, The Enforcer Extracted and modernized from RFC 7234, RFC 9111 specifies how shared caches (CDNs, edge proxies) and private caches (browsers) store, partition, revalidate, and invalidate representations. * It is the operational half of the equation. If RFC 9110 allows clients to ask for alternate flavors of a resource, RFC 9111 governs how to deliver those variants without corrupting production. * **The Linchpin:** Under RFC 9111, serving multiple representations from one canonical URL requires strict cache partitioning via `Vary: Accept`, format-scoped `ETag` validators, and `Surrogate-Key` invalidation. Miss this step, and your CDN serves raw Markdown to desktop shoppers or poisons the AI agent with 2 megabytes of HTML layout sludge. --- ### Why Everyone Is Reading These Specifications Again For a decade, frontend engineering operated under the assumption that a URI pointed to one thing: an HTML document stuffed with client-side JavaScript that hydrated inside a desktop or mobile browser. Autonomous agents, retrieval-augmented generation (RAG), and LLM search spiders have shattered that monoculture. Feeding raw HTML, styling wrappers, and tracking tags into an LLM context window burns 80% to 90% of scarce prompt tokens on pure syntactic noise. The industry isn't inventing an exotic, proprietary protocol to solve this. It is doing what it always does when a new client class upends the ecosystem: returning to core IETF standards. RFC 9110 provides the token-efficient handshake; RFC 9111 provides the edge caching mechanics that keep the servers standing. ## The Accidental Alignment: Why 2022 Specs Fit Modern AI **MikeLev.in**: So let's get the assumptions down here. 2022 is very recent. And 2014 is so long ago. And ChatGPT came out November 30, 2022 and the 9110 and 9111 must have been in the works some time before their release. Clearly the 2014 semantic web that allowed for serving PDF and XML versions on the same URL as HTML could not have been looking ahead to today's agentic clients, but 2022 seems pretty close. Were those later RFCs written with AI consumers in mind? **Gemini 3.8 Flash High**: **No, not at all.** RFC 9110 and RFC 9111 were **not** written with generative AI, LLMs, or autonomous agents in mind. The June 2022 publication date makes it look like it was drafted right on the doorstep of the AI boom. But in internet standards, publication dates are the *finish line* of a marathon, not the starting gun. --- ### 1. The Timeline Mismatch The work that produced RFC 9110 and 9111 did not start in 2022. The first working group draft—`draft-ietf-httpbis-semantics-00`—was formally published in **April 2018**. To put spring 2018 in perspective: * OpenAI had not even published the original **GPT-1** paper yet (that came in June 2018). * Google’s **BERT** was still months away (October 2018). * Concepts like "Retrieval-Augmented Generation (RAG)," "prompt token budgets," and "autonomous agentic web scrapers" were nowhere on the radar of network protocol engineers. The IETF HTTP Working Group (HTTPbis)—led by editors Roy Fielding, Mark Nottingham, and Julian Reschke—spent four years (2018–2022) debating framing, parsing edge cases, caching invariants, and transport layering. Generative AI had zero presence in those working group discussions. --- ### 2. What Actually Drove the 2022 Rewrite? If not AI, why did the IETF spend four years overhauling specifications that had already been updated in 2014? Two massive architectural pressures forced their hand: #### A. The Multi-Transport Layering Crisis (HTTP/2 & HTTP/3) In 2014, RFC 7231 was titled *Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content*. It bundled the abstract meaning of the web with the 1990s-era line-delimited ASCII wire format of HTTP/1.1. When **HTTP/2** arrived in 2015 (binary framing, multiplexing) and **HTTP/3** began development over QUIC (UDP transport), a glaring architectural contradiction emerged: * HTTP/2 and HTTP/3 were completely different wire transports. * Yet whenever their specs needed to define what `GET`, a `200 OK`, or an `ETag` meant, they had to cite a document whose title explicitly said **HTTP/1.1**. The IETF needed a clean, modular hierarchy: ```text RFC 9110 (Pure Semantics) RFC 9111 (Pure Caching) │ ┌───────────────────────┼───────────────────────┐ ▼ ▼ ▼ RFC 9112 (HTTP/1.1) RFC 9113 (HTTP/2) RFC 9114 (HTTP/3) [ASCII Wire] [Binary Multiplex] [QUIC / UDP] ``` #### B. Intermediary Realities and Caching Debt Between 2014 and 2022, the web became completely dominated by giant Content Delivery Networks (Cloudflare, Fastly, Akamai) and complex microservice reverse proxies. RFC 7234 (the 2014 caching spec) had accumulated dozens of ambiguities regarding how modern shared caches should behave. RFC 9111 cleaned this up: it formally retired obsolete baggage (like `Warning` headers), tightened conditional revalidation rules, and codified how CDNs must handle partitioned cache keys and `Vary` dimensions. --- ### 3. If It Wasn’t Written for AI, Why Does It Fit AI So Well? This is the great irony: **Content negotiation wasn't invented for AI, but it is rescuing AI from drowning in human UI noise.** Content negotiation (`Accept`, `Content-Type`, `Vary`) isn't even a 2022 invention. It was conceived by **Roy Fielding, Tim Berners-Lee, and the early web architects in the 1990s** (RFC 2068 in 1997, RFC 2616 in 1999). Fielding’s fundamental thesis for REST (outlined in his landmark 2000 doctoral dissertation and codified in RFC 9110) established a profound architectural rule: > **A URI identifies a Resource (an abstract concept or entity), NOT a file and NOT a specific format.** > What the server transmits across the wire is merely a **Representation** of that resource state at that point in time. In the 1990s and 2000s, people envisioned this mechanism would serve: * A browser requesting `text/html`. * A mobile WAP browser requesting `text/vnd.wap.wml`. * A screen reader requesting a clean text stream. * An enterprise system requesting `application/xml` or `application/pdf`. For roughly fifteen years (2008–2023), the industry largely discarded that elegant separation. Frontend development collapsed into a single-format monoculture: every URL served an HTML shell packed with megabytes of JavaScript, CSS, and DOM chrome meant exclusively for visual human consumption in a Chromium rendering engine. When LLM agents arrived in 2023–2024, they represented a client that nobody in the 2010s anticipated: * They don't have eyes or screens. * They don't execute CSS or care about flexbox layouts. * Every byte of styling boilerplate costs money in context-window tokens and inference latency. Because the authors of RFC 9110 spent 2018–2022 strictly enforcing the purity of **"Resources vs. Representations"**, the architecture was already complete. The protocol did not need an emergency "AI extension" or a new HTTP method. The exact mechanism required to negotiate between a visual human shopper and an autonomous synthetic agent had been sitting inside Section 12 of the HTTP specification for over twenty-five years, cleanly polished and waiting in RFC 9110. ## Edge Transformation Realities and Consultant Opportunities **MikeLev.in**: Okay so if RFCs 9110 and 9111 were 2022 release and 4 years in the works and made with content distribution networks (CDNs) in mind, then presumably this is something the Internet companies that was being addressed by the evolving Internet specifications were aware of them. Did they take them seriously? If one URL is supposed to be serving one stripped-down non-HTML thing to Agents and another full-version to presumably humans on Web browsers, how can a system that's designed to cache pages at the so-called edge be able to serve one version to bots (and what sort of granularity is there?) and another version to humans? They can't cache 2 versions of content on the same key. And even if they put them on different keys, how would it know which to serve? Bring it down for the newcomer to all this including what a CDN is, and then build it all the way back up gradually to answer those final questions. You know I don't think they know about hash tables as a solved O(1) problem and what that even means and why these optimized key-value hash tables are used everywhere now (like where else?) and how these keys are different from relational database management systems (RDBMS) primary keys and how a compound key with string concatenation is different from a composite key each with their own index and what downstream features that means you do and don't have in either case. Tear it down to the basics and build it back up to whether or not things are different now at CDNs as a result of RFC 9111. **MikeLev.in**: So you're telling me that for every URL there can be a cached version at the CDN for each format that a client can accept and each version that can be made available either by origin or by some other conversion process? That conversion process seems expensive compute-wise. It also seems like it's a place where a lot could go wrong, such as an HTML to Markdown conversion. The ideal is for the origin to provide the Markdown as well but assuming they can't, and as we discussed in prior articles this is actually even an included service by Cloudflare at a certain level, is this conversion just left to chance and whatever "default template" result in? Have consultants been shut out of the process because Cloudflare (and presumably the others) "just canned it" or are there places the conversion might mess up where consultants, specialists and technicians cans step in to improve the quality? And how does this differ on broadly templated and presumably consistent pages like product detail pages versus 1-offs bespoke pages like quizzes, calculators and such? What sorts of things are longer necessary to do now with RFC 9111 implemented and what sorts of new behaviors are encouraged? And this is not in a subjective opinion way. When fabric of the Internet evolves like this there are usually provable metrics to look at. What are they? How can someone make objective determinations optimizing the benefits and competitive advantages to be realized here? **Gemini 3.8 Flash High**: ### 1. The Reality of "Canned" Edge Conversion When a CDN like Cloudflare offers automated HTML-to-Markdown conversion, it is not running an LLM or an expensive AI model at the edge. Doing so for millions of requests would collapse edge CPU budgets. Instead, the edge runs an **Abstract Syntax Tree (AST) streaming parser** compiled to WebAssembly (Wasm) or native Rust. ```text Incoming HTML Stream ──► [ Lexer / Tokenizer ] ──► [ DOM AST Tree ] ──► [ Heuristic Pruning ] ──► Markdown Stream │ Strips: