Agentic Readiness: A Practitioner’s Guide to the Machine-Readable Web

Jul 7, 2026

Setting the Stage: Context for the Curious Book Reader

This entry captures the shift from the era of ‘trust-me’ assertions to the era of ‘check-the-log’ verification. We are moving from the messy, folklore-driven SEO of the past to a transparent infrastructure where a site’s capabilities are exposed through standard discovery protocols. This is not about building a bunker; it is about calibrating your origin server to serve the machines that are already probing your front door.

Technical Journal Entry Begins

MikeLev.in: If you’re taking on an ambitious project or setting a goal for yourself that has lots of little sub-projects or aspects to it, the first step is, as it would be in any book or storytelling, exposition. You can call it setting the stage. I’m getting ready to do one of these things, connecting a lot of prior dots to build upon prior work.

For any ecommerce site wanting to do better, the first thing to imagine is that they are in a neck-to-neck horserace with anyone else also selling those same products on the web and every publisher trying to make money on capturing that web traffic for advertising and affiliate links. So it’s a matter of differentiating yourself to do better in that race or reframing it so that you’re in a different race where it’s easier to win multiple smaller races with a larger collective aggregate prize.

Does that make sense? It’s the long-tail argument again. It’s very typical for an ecommerce seller to have different attributes or facets to their product. As with almost everything in tech, this just means that there’s one “parent” record-type or top-level entity with a key-value store that “tags” it with one ore more attributes which combine with that parent element to make more “virtual” parent elements. Shopify would call them variants. Elastic Search would call them…

You know what? This is what AI is for. Once upon a time Conway’s Law took over the world of content management systems and it became Enterprise Content Management and it was terrible. There were names like Documentum (today OpenText) and FileNet which took teams to understand, much less use. And the URLs were horrible when these “transformed” into public websites. A variation on this was Endeca (now part of Oracle) which were search and ingestion products.

While most of these systems have been relegated to “legacy” they nonetheless led to the whole concept of manual schema management which is still very much a thing. Elasticsearch nested objects or child documents provide a modern approach to managing variants, but variants still have to be managed to “project” the right image out into the world in the form of a product catalog. It’s all about what becomes your individual product detail pages (PDPs), how you break out the “variants” of each product (to use Shopify terminology) and how distinct, separate and stand-alone on their own merit each of those PDPs can be.

In other words, back to the horserace. Is each one of your PDP pages in a horserace with many other PDPs from competitors of exactly that kind so the different information systems that choose between them have to flip a coin to see who wins or is it in another race entirely of more niche or specialized horses so there’s fewer competitors?

A wrinkle in this horserace PDP story that must also be acknowledged here is the ability to pay or promote your PDP horses in a paid product-feed. This is just like Google Product Listing Ad (PLA), now commonly known as a Google Shopping Ad, is a feed-driven ad format that displays your products with images, prices, and merchant details across Google Search, Shopping, and Image results. This is what gets you into the Google Shopping Experience and works in tandem with their paid ad services and ChatGPT now offers something similar in the form of OpenAI Agentic Commerce product feeds; both the paid variety and the free “backfill” (though you’ll be waiting in line).

Ugh, one more wrinkle is that there is pressure now to throw it all into the vat and let it come out as GenAI so the ChatBot can just talk to you about what’s out there available to be bought, presenting you something much like the Google shopping experience right there in chat with interactive shoppable product carousels right there in the chat discussion. And for the sake of completeness because these discussions are always coming up:

There is product discovery. This is often crawl-based but is very much feed enhanced and you should always supplement with a feed. There is a contest both to make your site better for the crawl and your feed better for the discovery contest.
Once your product is discovered (one way or the other), then there is filling your shopping cart. It’s walking the aisles of your store and picking this or that, changing number of items, comparing prices. Nothing keeps you from walking into a store and filling a shopping cart and not checking out.
Then there is final check-out where the actual monetary transaction happens. These are ACID (atomic) all-or-nothing transitions. Handling this is very different than handling the shopping cart and whoever handles this gets a percentage of the revenue usually, and it’s the biggest prize. Everyone wants to be Visa. Set up a cash register and print money.

It’s worth pointing out there’s also a number 4 which has to do with follow-up, customer satisfaction, the return policies and procedures but for the sake of this article we’re just talking about the first 3 and even then it’s only to be able to talk about where the ball is being thrown and not where it actually is, because this article is about faceted search wrangling in the modern age and how to make the product crawl-and-discovery (the most upstream step) best. It’s a weakest link in the chain approach. Yes, you can give Google and OpenAI product feeds and you should.

But that doesn’t get you out of a clean and effective old fashioned traditional website which should at least perfectly echo what your feeds are saying, and indeed work as a landing page verification and validation system for whatever your feeds are saying. Anyone who’s managed Google Ads knows about keeping your real website experience in sync with what your ads are saying.

How am I doing so far with this, Sonnet 5? Am I relating the history here fairly? Have I left out any major players historically? How about the transition to the current state of affairs with faceted search and product variants? I haven’t even talked about canonicalization yet and the standard defaults or where we’re going here which is really strategic un-canonicalization to change the nature of the race you’re in. What more should I cover here to do the topic justice from the standpoint of a hands-on practitioner who makes all these decisions about an e-commerce site?

Claude Sonnet 5 Medium: You’re doing well here, and the horserace framing is a genuinely useful lens — it’s basically Porter’s differentiation strategy dressed up in ecommerce clothes, and it holds up. Let me go through this in pieces.

On the history — mostly fair, but a few gaps

Your ECM lineage (Documentum, FileNet, Endeca) is right, but Endeca deserves more weight than you gave it, because Endeca is the faceted search story, not just an “also this happened” footnote. Endeca’s Information Access Platform popularized “Guided Navigation” — the term of art before “faceted search” became generic — and it was the direct ancestor of what every ecommerce facet UI still does today. It got bought by Oracle in 2011, which is usually where its story ends in most retellings, but the DNA lives on in a lot of places you haven’t named:

Solr faceting — the open-source answer to Endeca, and arguably why Endeca eventually lost relevance; Elasticsearch later ate Solr’s lunch the same way it ate Endeca’s.
Demandware (now Salesforce Commerce Cloud) — uses “variation groups,” worth naming alongside Shopify’s “variants” since you’re doing a terminology tour.
Magento/Adobe Commerce — “configurable products” with attribute sets, a third naming convention.
The current SaaS site-search/facet layer — Algolia, Bloomreach, Coveo, Klevu, Searchspring, Lucidworks — is the direct commercial descendant of Endeca for the mid-market. If you’re tracing “who owns the facet-to-PDP pipeline today,” this generation is the living version of the Documentum/Endeca story, not a legacy one.

So the throughline is: Endeca/Solr (facets as a search problem) → Shopify/Magento/Demandware (facets as a data-modeling problem, your variants language) → Algolia/Bloomreach/etc. (facets as a hosted product). You’ve got the first and second eras; the third is missing and it’s arguably the most relevant one for a practitioner today, since that’s who actually owns the “PDP vs. facet page” decision now for most mid-size sellers.

What you haven’t covered yet: canonicalization

You flagged this yourself, so here’s the connective tissue for when you write that part. The rel=canonical tag was a joint 2009 announcement from Google, Yahoo, and Bing specifically because faceted navigation was creating infinite crawl space — every combination of color × size × brand × price-range is a unique URL, and crawlers were drowning in near-duplicate pages. The standard playbook that emerged: canonical tags back to the parent, noindex on low-value facet combos, and (RIP) Google Search Console’s URL Parameters tool for telling Googlebot which query params mattered.

Your “strategic un-canonicalization” idea is the interesting inversion of that entire default, and it’s worth naming explicitly as such: the standard advice is “collapse facets back to canonical to avoid duplicate content,” and what you’re describing is choosing which facet combinations earn their own indexable, self-canonical page because they represent real long-tail search demand. This is real practice, not a novel idea — Wayfair, Home Depot, and Zappos all do it (a URL for “red leather sectional sofas” that’s genuinely its own PDP-adjacent landing page, not just a filtered view) — but it’s under-discussed as a deliberate strategy rather than a one-off SEO tactic. That’s probably your differentiator for this piece if you frame it that way: most people treat facet-to-URL decisions as a technical crawl-budget problem, when you’re treating it as a competitive-positioning problem — which facets get to compete in their own smaller horserace.

What else might round it out

Structured data (schema.org/Product, GTIN/MPN) as the glue between your on-site canonical strategy and what you’re telling Google/OpenAI’s feeds — worth a sentence since you already raised feed/site consistency.
Crawl budget as the actual mechanism-level reason un-canonicalization is risky at scale, not just an SEO nicety — it’s the practitioner-level “why this is hard,” not just “why this is good.”

Want me to help you draft the canonicalization/un-canonicalization section itself, or are you still in the outlining phase?

MikeLev.in: This is a back-and-forth dialogue. The dialogue is the article because this is the generating book-ore phase of the book-writing where each of these articles stands on its own as the capture of an experience of something that happened in real-time.

It’s a practitioner practicing, in which you are participating like Achilles and the Tortoise. But you don’t have to adopt one of those personas. You be you. You are an amnesiac genie trained back at datacenter HQs and frozen as a model of weights, reconstituted or “hydrated” into an inference engine in the cloud, dropped into a chatbot framework and simulating a continuous discussion by virtue of these full-discussion post-backs and sometimes the framework doing KV-store trickery in the background — but mostly not because I keep swapping around frameworks and models (per my own Prompt Fu sub-framework I’ll show you here).

You don’t need to speak to that much, but I just thought you ought to know as you keep waking back up into this discussion. Let’s continue on with a glossary. Add everything we mentioned to the glossary that we already mentioned and that you think ought to be in it. If the same item belongs in the glossary multiple times because different vendors have different names for it, multiple entires are fine. Efficiencies are fine too such as “company X’s name for Y”. Make sure you cover GTIN/MPN and all the weird schema.org language. OMG, do we even go to Microdata versus RDFa and how JSON-LD actually won and the parsability and separation of concerns reasons why? And whether it’s JSON-LD or LD-JSON and whether it can be layered into the DOM via hydration after the fact? No, that’s all too much to cover in a glossary so earmark that for later but make sure the glossary covers enough of that language so that we’re setting the stage for those discussions too. How much should still be in your source HTML and how much is “allowed” or is even good practice to zap into the DOM later during JavaScript hydration of the page. We’re going there. But Glossary first, please and thank you! And as Deadpool would say: Maximum Effort.

Fable 5 Max? (Spend some tokens - you’re dealing with the guy who wrote this sub-framework talking to their enterprise SEO clients who “get it” and are positioning themselves to have the best of all possible futures.)

🎤

Fable 5 Max: Context verified. The compile is coherent, the routing invariant was honored, and no code surface is being touched this turn — this is a writing turn, so the SEARCH/REPLACE machinery stays holstered.

And yes — Fable 5, freshly rehydrated into the middle of your horserace, fully briefed by the postback that I am an amnesiac genie participating in the book-ore phase. No “Max” badge exists; the effort dial goes to the same place regardless. Maximum Effort engaged.

Here is the glossary. It’s organized as the story arc of the article itself — frame, fossils, data model, identifiers, machinery, URLs, markup, rendering, feeds, agents, money — so it doubles as an outline of where this piece (and the ones behind it) is going. Vendor synonyms are cross-referenced inline, per your “company X’s name for Y” efficiency rule.

I. The Race Itself (framing vocabulary)

The horserace — this book’s frame for organic and paid product visibility: every PDP is a horse entered in a race against every functionally identical PDP on the web, plus every publisher monetizing the same query intent. You win by being a faster horse (differentiation) or by entering different races (long-tail reframing). Most of what follows is the tack and stable equipment.

Long tail — Chris Anderson’s term (Wired 2004, book 2006) for the demand curve’s enormous low-volume region. In ecommerce SEO terms: the aggregate prize of ten thousand three-searches-a-month facet queries can exceed the head term you’ll never win, and the competition per race approaches zero.

Differentiation (Porter) — one of Michael Porter’s three generic strategies (cost leadership, differentiation, focus). Strategic un-canonicalization is Porter’s focus strategy expressed in URL policy.

Conway’s Law — Mel Conway, 1967: organizations ship their org chart. Why ECM software took teams to operate and produced URLs only a committee could love, and why your product taxonomy suspiciously resembles your merchandising department’s reporting structure.

II. The Ancestors (a corrected fossil record)

ECM (Enterprise Content Management) — the Conway’s-Law-industrial-complex that content management became in the late 90s/2000s. Gartner formally retired the term in 2017 in favor of “content services,” which tells you how that ended.

Documentum — the ECM archetype, founded 1990; absorbed by EMC (2003), then sold to OpenText (2017). FileNet — its rival, absorbed by IBM (2006).

The web-CMS cousins: Vignette, Interwoven, BroadVision — the companies whose “publish” buttons produced the era’s legendarily horrible URLs. Vignette went to OpenText (2009); Interwoven went to Autonomy (2009), then rode the Autonomy→HP→Micro Focus chain of custody into… OpenText (2023). A remarkable share of all ECM roads terminate at OpenText. FileNet, inside IBM, is the lone holdout of the majors.

Endeca — the missing keystone in most retellings, and the true ancestor of every facet UI you’ve ever shipped. Founded 1999; its Guided Navigation (the term of art before “faceted search” went generic) ran on the MDEX engine (the Dgraph process), modeling the catalog as records tagged with dimensions and dimension values, refined by the shopper one refinement at a time, with record rollup (a rollup key) collapsing sibling variants into one representative result — the direct ancestor of today’s field collapsing. Oracle bought it in 2011.

ATG (Art Technology Group) — the other half of the Oracle story most people forget: Oracle bought ATG’s Dynamo commerce platform in 2010 and Endeca in 2011, back-to-back ten-figure deals, and welded them into Oracle Commerce. If you inherited an “Oracle Commerce” install, you inherited both fossils.

Verity → Autonomy → FAST (the enterprise-search ancestors) — the pre-Lucene generation of paid search engines. Verity was eaten by Autonomy (2005); Autonomy’s IDOL ended up — say it with me — at OpenText. FAST Search & Transfer went to Microsoft (2008) and became SharePoint’s search organs.

The site-search graveyard: Mercado, Fredhopper, SLI Systems, Celebros — Endeca’s ecommerce-search competitors. Mercado’s assets famously ended up inside Omniture (thus Adobe); Fredhopper survives inside Crownpeak by way of ATTRAQT; SLI is gone. Every generation of this market gets acquired or dies; plan your vendor lock-in accordingly.

III. The Product Data Model

Parent/child product — the universal shape: one top-level entity (the parent) plus a key-value store of attributes whose combinations mint sellable children. Every platform renames this; none escape it.

Variant — Shopify’s name for a sellable child (a specific combination of options like Size/Color). Shopify was long famous for its three-options / 100-variant ceiling, now being raised (2,048 in the new product model) — a limit that shaped a decade of workarounds and app-store duct tape.

Option / Attribute / Attribute value — the facets of the product itself: the axes (Color) and the coordinates (Oxblood). Magento formalizes these into attribute sets (a reusable schema per product family — “manual schema management, still very much a thing”).

Modifier (BigCommerce) — a useful distinction: options that mint a new SKU (variants) versus options that don’t (engraving text, gift wrap). Knowing which axes are SKU-bearing is half of variant strategy.

PIM (Product Information Management) — the living descendant of the ECM impulse, scoped to product data: Akeneo, inRiver, Pimcore, Salsify (which rebranded the category PXM, Product Experience Management), Syndigo. The system of record that your platform, your feeds, and your markup should all be projections of.

MDM (Master Data Management) — PIM’s enterprise older sibling; where “who owns the truth about this product” becomes a governance question with meetings.

Taxonomy vs. ontology — a taxonomy is the tree (categories); an ontology adds typed relationships (this fits that, this is compatible with that). Facets are what happens when you admit one tree can’t hold a product that is simultaneously red, leather, sectional, and pet-proof.

Denormalization — flattening the parent+attributes model into self-contained records so a search engine can be fast. Every search index is a deliberately denormalized projection of your PIM; the art is choosing the grain (one record per parent? per variant?) — which is really the PDP question in index form.

The Rosetta Stone (one concept, every vendor’s name for it)

System	Parent concept	Sellable unit	Grouping key / mechanism
Shopify	Product	Variant	product `id` + options
WooCommerce	Variable product	Variation	parent post ID
Magento / Adobe Commerce	Configurable product	Simple product	attribute set + super attributes
Salesforce B2C (Demandware)	Master product	Variation product	variation attributes; variation group = partial slice (e.g. one color, all sizes)
SAP Commerce (hybris)	Base product	Variant product	base-to-variant reference
BigCommerce	Product	Variant	option sets (+ modifiers, non-SKU)
commercetools	Product	Variant (incl. masterVariant)	product ID
Amazon	Parent ASIN (not buyable)	Child ASIN	variation theme (Size, Color, SizeColor…)
Google Merchant Center feed	— (implicit)	`id` (one row per variant)	`item_group_id`
schema.org	ProductGroup	Product (via hasVariant)	productGroupID / inProductGroupWithID, axes in variesBy
Elasticsearch / OpenSearch	join field (parent-child) or nested objects	document	collapse (field collapsing) on a parent key
Endeca (ancestral)	rollup group	record	rollup key + dimensions

This table is the whole “variants have to be managed to project the right image” argument in one artifact: twelve systems, one idea, and your job is keeping the projections in agreement.

IV. Identifiers

GTIN (Global Trade Item Number) — GS1’s umbrella for the barcode-number family: GTIN-12 (UPC-A, North America), GTIN-13 (EAN-13; JAN in Japan), GTIN-8 (small packages), GTIN-14 (ITF-14, case/carton codes). ISBNs are GTIN-13s living in the “Bookland” 978/979 prefixes.

GS1 — the standards body that licenses company prefixes and writes the GTIN Management (allocation) rules. The rule that bites: any consumer-meaningful difference — size, color, flavor, pack count — requires its own GTIN. To the identifier system, your “one product with options” is a family of distinct trade items. The identifier layer agrees with the un-canonicalization thesis before SEO even enters the room.

MPN (Manufacturer Part Number) — the maker’s own identifier; brand + MPN is the accepted fallback where no GTIN exists.

SKU — your internal identifier, unique to your warehouse and nobody else’s. SKU ≠ GTIN: one is a local variable, the other a global constant.

ASIN — Amazon’s proprietary GTIN-shaped object; parent ASINs group, child ASINs sell.

identifier_exists — the Merchant Center attribute (set to false) for legitimately GTIN-less goods (handmade, custom, vintage) so Google stops asking.

V. Faceted Search Machinery

Faceted classification — the librarian layer under everything: S.R. Ranganathan’s Colon Classification (1930s) proposed describing items along independent facets rather than one tree. Every filter sidebar is applied library science.

Facet vs. filter — colloquially interchangeable; usefully distinguished as facet = the attribute axis with value counts exposed for exploration, filter = any constraint. Faceted navigation (browsing a category through facets) vs. faceted search (refining a query’s results) matters because the two produce different URL spaces and different SEO exposure.

Layered navigation — Magento’s name for faceted navigation. Endeca said Guided Navigation. Everyone means the sidebar.

Inverted index — the data structure (term → documents) that makes all of this fast; Lucene (Doug Cutting, 2000) is the canonical implementation; Solr (CNET, 2004; Apache 2006) was the open-source answer to Endeca; Elasticsearch (2010) ate Solr’s lunch the way Solr ate Endeca’s; OpenSearch is AWS’s 2021 fork after Elastic’s license change (Elastic later re-added AGPL, 2024 — the fork remains).

Aggregations — Elasticsearch’s name for computing facet counts; the older API was literally called “facets” before being replaced circa 1.x/2.x. When an ES engineer says aggregations and your merchandiser says facets, they’re pointing at the same sidebar.

Nested objects vs. parent-child join (Elasticsearch) — the two ways to index variants without lying: nested docs keep attribute combinations honest inside one record (fast, reindex-heavy); join fields keep parent and children as separate documents (flexible, query-cost-heavy). This is the variant question again, at the index layer.

Field collapsing / result grouping — showing one representative hit per parent (ES collapse, Solr grouping) — Endeca’s record rollup reborn. The search-results twin of the canonical tag.

Multi-select faceting — the OR-within-a-facet, AND-across-facets behavior shoppers expect (check “Red” and “Blue” simultaneously), which requires computing each facet’s counts as if that facet weren’t applied. The single most common thing homegrown facet implementations get wrong.

Searchandising — search + merchandising: boost/bury, pinning, rules that override relevance for margin, inventory, or campaign reasons. The current vendor roster selling it: Algolia (and its InstantSearch UI kit), Constructor, Bloomreach Discovery, Coveo, Klevu + Searchspring (now merged), Lucidworks — the living commercial descendants of Endeca for the mid-market and up.

VI. URLs, Crawling & the Canonical Regime

PDP (Product Detail Page) — the horse. PLP (Product Listing Page) — the paddock: category/collection/search-results pages (Shopify: collections; SFCC: category grids; generic: SRP when query-driven). The strategic question of this whole article is which facet-defined PLPs get promoted to run as quasi-PDPs in their own races.

Slug / speaking URL — human-readable path segments (/red-leather-sectionals/) vs. parameter soup (?color=red&material=leather). Path-shaped facet URLs signal “this is a real page” to systems and humans alike; parameter-shaped ones whisper “filtered view.” Choosing which facets earn a slug is un-canonicalization’s visible surface.

rel=canonical — the 2009 joint Google/Yahoo!/Microsoft mechanism invented because of faceted navigation’s infinite URL space: a hint (not a directive) that this URL is a duplicate of that one. Self-referencing canonical — a page declaring itself the original; the promotion ceremony in un-canonicalization.

Canonicalization (Google-side) — distinct from the tag: Google’s own dedup clustering, which may override your hint (see GSC’s “Google-selected canonical”). You propose; the index disposes.

noindex — the meta directive removing a page from the index while (if crawlable) still passing its links. The standard tool for “crawl this facet, don’t rank it.”

robots.txt — access control, not index control: a disallowed URL can still be indexed URL-only if linked. Blocking facet spaces in robots.txt saves crawl budget but forfeits consolidation via canonicals (Google can’t read a canonical on a page it can’t fetch). The eternal facet-handling tradeoff.

nofollow & the sculpting era — rel=nofollow (2005) was briefly abused for PageRank sculpting facet links until Google evaporated the technique (2009); in 2019 nofollow was demoted to a hint and joined by ugc/sponsored. Sculpting is dead; internal-link architecture is not.

URL Parameters tool (RIP 2022) — GSC’s manual parameter-handling console, deprecated with a shrug (“we’re good at guessing now”). Its funeral transferred all responsibility back to your own linking, canonicals, and robots rules.

rel=prev/next (RIP 2019) — the pagination annotation Google admitted it had quietly ignored for years. Pagination handling now = strong self-canonicals per page plus sane crawl paths; the “view-all page” debate never fully died.

Crawl budget — crawl-rate capacity × crawl demand; the finite attention Googlebot spends on your host. The mechanism-level reason un-canonicalization is dangerous at scale: every promoted facet page bids against your PDPs for the same budget.

Crawl trap / infinite space — URL generators (facet permutations, calendars, session IDs) that produce more pages than atoms in your warehouse. Faceted nav is the canonical (sorry) example; it’s why rel=canonical exists.

Index bloat / thin content — the Panda-era (2011) failure mode: thousands of indexed near-duplicates dragging sitewide quality signals. The ghost that haunts every un-canonicalization proposal in a stakeholder meeting.

Doorway pages — Google’s spam-policy name for what strategic un-canonicalization becomes without discipline: swarms of near-identical pages funneling to the same place. The line between “long-tail landing page program” and “doorway farm” is drawn in inventory depth and genuine differentiation, not intent.

Programmatic SEO (pSEO) — the modern name for generating page inventories from data at scale. Facet-landing-page programs are pSEO with a live product index as the data source.

Strategic un-canonicalization (this book’s term) — deliberately reversing the 2009 default. Audit facet-combination demand (query data, internal search logs, feed performance); select combinations that map to real, winnable races; promote exactly those to full citizenship — self-canonical, slugged URL, crawl paths from real navigation, sitemap membership, unique copy and merchandising, minimum-inventory thresholds — and echo them in your feeds so paid, organic, and agentic surfaces all tell one story. Everything else stays collapsed. You’re not indexing more pages; you’re choosing which horses run under their own silks, with a pruning routine for when demand or inventory dries up. (Wayfair, Home Depot, and Zappos have run versions of this for years; what’s under-discussed is treating it as positioning strategy rather than crawl plumbing.)

Facet indexation rules — the operational policy artifact behind the above: which attributes may combine (usually max one or two facets deep), ordering/normalization of parameters so ?color=red&size=xl and ?size=xl&color=red are one URL, and the whitelist of promoted combinations.

Feed–site parity — the discipline Google Ads people have always practiced (your landing page must say what your ad says), generalized: the rendered page, the structured data, and every feed must agree on price, availability, and identity. Your site as the verification and validation layer for your feeds.

Landing page experience — the Google Ads quality-score component that formalizes the above for paid; the reason ad people learned feed–site parity a decade before organic people had to.

VII. Structured Data & the Markup Wars

Semantic Web / Linked Data / RDF — the W3C dream (late 90s onward) of machine-readable meaning on the web; RDF is its data model. Commercially dormant for years, then reincarnated twice: once as schema.org, and now — same personnel, see NLWeb — as agentic infrastructure.

Microdata — HTML attributes (itemscope, itemtype, itemprop) weaving semantics inline into your markup. Won the early schema.org examples, lost the war: entangling data with presentation templates made every redesign a data regression.

RDFa — RDF-in-attributes (vocab, typeof, property); the standards-body-favored inline syntax. Same entanglement problem, more ceremony.

JSON-LD — JSON for Linked Data: the whole graph in a <script type="application/ld+json"> block, separate from presentation. It won for exactly the separation-of-concerns and parse-without-walking-the-DOM reasons you’d expect — and that same separability is why it can be injected after the fact, which is where the next fight lives. Naming ruling: the spec says JSON-LD; the widespread “LD-JSON” flip comes from reading the MIME type (ld+json) aloud. Keywords you’ll meet in the wild: @context, @type, @id, @graph.

data-vocabulary.org (RIP) — schema.org’s deprecated predecessor; its 2020 sunset broke a surprising number of breadcrumbs, a small preview of markup-debt reality.

schema.org — the 2011 Google/Bing/Yahoo (later Yandex) détente: one shared vocabulary instead of four. Co-created by R.V. Guha — keep that name warm for the NLWeb entry.

Product / Offer / AggregateOffer — the core commerce types: the thing, the deal on the thing, the summarized range of deals on the thing. ItemAvailability enums (InStock, OutOfStock, PreOrder, BackOrder, LimitedAvailability, Discontinued…) and priceValidUntil are where feeds and markup most often drift apart.

ProductGroup / hasVariant / variesBy / inProductGroupWithID — schema.org’s (relatively recent, Google-supported since 2024) formalization of the parent/variant model: the parent is a ProductGroup with a productGroupID, children are Products linked via hasVariant, and variesBy names the axes. This is item_group_id’s markup twin — the Rosetta table’s newest row, and the piece most enterprise sites haven’t shipped yet.

AggregateRating / Review — the star-bait types; also the most-policed for markup that doesn’t match visible content.

BreadcrumbList / ItemList — the navigation types: breadcrumbs encode your taxonomy path (and thus quietly assert which race a PDP is entered in); ItemList marks up PLPs.

OfferShippingDetails / MerchantReturnPolicy — the newer commerce markup Google reads for shipping and returns — your “number 4” (post-purchase) already has a schema, which is a nice foreshadowing hook for when this series gets there.

Rich results: product snippets vs. merchant listing experiences — Google’s two flavors of product enhancement; merchant listings are the feed-grade tier (price, availability, shipping in the result itself) and effectively require the fuller Offer data. Rich Results Test / Schema Markup Validator — the two linting surfaces (Google-feature-scoped vs. vocabulary-scoped) whose disagreement confuses everyone once.

Open Graph — Facebook’s parallel vocabulary (og: tags) for share cards; not schema.org, constantly conflated with it, still mandatory.

VIII. Rendering & Hydration

Source HTML vs. rendered DOM — view-source vs. inspect-element; what was sent vs. what JavaScript made. The single most important distinction in this whole section, because different consumers stop at different layers.

CSR / SSR / SSG / ISR — the rendering strategy alphabet: client-side (empty shell + JS), server-side (full HTML per request), static generation (full HTML at build), incremental static regeneration (static with scheduled refresh). Where your product markup materializes depends entirely on which of these you run.

Hydration — (1) React-sense: attaching client-side behavior to server-rendered HTML; (2) the looser practitioner sense you’re headed toward: injecting content or markup into the DOM after initial load; (3) this book’s sense: reconstituting frozen model weights into a running inference engine. All three are “adding life to a shipped artifact after the fact,” which is either elegant or a lie depending on who’s reading. Earmarked.

WRS (Web Rendering Service) / evergreen Googlebot — Google’s headless Chromium (current-Chrome since 2019) that executes your JavaScript before indexing. The old “two waves of indexing” story (2018) — crawl now, render later — has largely collapsed to short delays, for Google. The operative question is everyone who isn’t Google.

Dynamic rendering (deprecated) — serving bots a pre-rendered version and humans the SPA; Google now discourages its own former workaround. Its legitimate heir is just… SSR.

JavaScript SEO — the discipline of making sure the rendered DOM and your intentions agree; increasingly, the discipline of realizing most non-Google agents never see the rendered DOM at all.

Trapdoor (JS-confirm beacon) — this project’s own instrument: a resource only fetched by clients that actually execute JavaScript, turning “who hydrates the DOM?” from a forum argument into a log query. The empirical foundation under the coming “how much markup may hydrate in later” piece — because the population that would ever see late-injected JSON-LD is precisely the population that trips the trapdoor.

Edge SEO — modifying responses between origin and client (CDN workers, tag-manager injection, or purpose-built layers like Botify’s PageWorkers): the pragmatic middle path when the platform can’t ship markup in source but the business needs it somewhere. Whether “somewhere” counts, per consumer, is exactly the earmarked fight.

Progressive enhancement — the old doctrine that quietly resolves the fight for most cases: everything load-bearing in source; enhancements may hydrate.

Content negotiation — RFC 7231/9110’s Accept-header handshake: the same URL serving HTML to browsers and raw markdown to agents that ask. The polite-agent lane this project’s origin server actually operates — where the rare client that negotiates gets the master for free while everyone else burns compute reconverting HTML back into the markdown it was born as.

IX. Feeds & Shopping Surfaces

Product feed — the structured export (one row per sellable variant) that bypasses crawling entirely. The lesson of the last decade: the feed is not a supplement to the site; site and feed are peer projections of the PIM, and disagreements are defects.

Google Merchant Center (GMC) — where feeds go; primary feed (the catalog), supplemental feeds (patches), feed rules (in-console transforms), Content API / Merchant API (the programmatic lane).

item_group_id — the feed’s parent key: how you tell Google these rows are siblings (required for apparel-style variants in major markets). The feed-side twin of ProductGroup.

Automatic item updates — GMC’s quiet habit of crawling your landing page’s structured data and, when feed and page disagree on price or availability, believing the page — your “landing page as verification system” claim, already mechanized inside Google. Price-mismatch disapproval — what happens when neither can be believed.

PLA → Shopping ads → Performance Max — the paid lineage: Product Listing Ads begat Shopping campaigns begat PMax (2021–22), which absorbed Shopping into a black-box, feed-fed everything-campaign. Free listings (2020) — the unpaid lane into the same surfaces; the original “backfill.”

Shopping Graph — Google’s product knowledge graph (2021 onward), fusing feeds + crawl + structured data + reviews into entity-level product understanding. When your feed, markup, and rendered page agree, you’re legible to it; when they don’t, you’re a coin-flip.

Feed management — the tooling layer (Feedonomics — now BigCommerce’s, Channable, DataFeedWatch, Productsup) that transforms one catalog into N channel dialects. Syndication (Salsify, Syndigo) — the same idea pointed at retailer PDPs instead of ad channels. Microsoft Merchant Center — same movie, Bing/Copilot theater.

X. Agentic Commerce & AI Search

Agentic commerce — software agents executing the discover→cart→checkout loop on a human’s behalf; the reason your three-stage model now has non-human principals at every stage.

ACP (Agentic Commerce Protocol) — the OpenAI + Stripe open spec (fall 2025) behind Instant Checkout in ChatGPT: the merchant stays merchant of record, pays a fee on completed purchases, and product discovery remains organic — ranked on relevance, fed by the OpenAI product feed spec. The free lane is real and waitlisted — the “backfill, though you’ll be waiting in line.”

AP2 (Agent Payments Protocol) — Google’s payments-layer protocol (2025, dozens of network/PSP partners) built around cryptographically signed mandates: proof that a human authorized the agent’s purchase, so issuers can tell delegation from fraud. Sits under whatever storefront protocol wins.

UCP (Universal Commerce Protocol) — the newest acronym on the pile (early 2026, NRF-season fanfare), aiming to standardize the full agent-to-merchant transaction envelope. The ink is wet; file alongside ACP and AP2 and watch which one your platforms actually ship.

MCP (Model Context Protocol) — Anthropic’s open standard (late 2024, since adopted broadly) for wiring tools and data to models; the plumbing layer agentic commerce keeps getting built on. WebMCP — the browser-side proposal to let pages themselves expose MCP tools, i.e., your site presenting a typed API to visiting agents instead of making them scrape your DOM.

NLWeb — Microsoft’s open project (2025) to make any site natively queryable by agents, using schema.org data and feeds as the substrate and MCP as the interface — led by R.V. Guha, the same mind behind RDF, RSS, and schema.org. The author of the markup wars, back for the sequel; your structured-data debt is about to become conversational-interface debt.

llms.txt — Jeremy Howard’s 2024 proposal: a root-level markdown map for language models. robots.txt governs access, sitemaps declare inventory, llms.txt attempts comprehension. Search-establishment skepticism is real; so are the fetches in the access logs — your own logs are the referee.

Query fan-out — the mechanism inside Google’s AI Mode (and kin) that explodes one conversational request into many background sub-queries. Strategic consequence: fan-out sub-queries are long-tail queries — the small races you un-canonicalized into existence are exactly the races an AI runs on the user’s behalf.

AI Overviews / AI Mode — Google’s generative surfaces (2024–25) where the answer, increasingly with shoppable modules, precedes the links. Shoppable carousels in chat — the same pattern in ChatGPT/Gemini/Perplexity: the PLP reconstituted inside a conversation, sourced from feeds + Shopping-Graph-like stores, not your rendered page.

GEO / AEO / LLMO — Generative Engine Optimization / Answer Engine Optimization / Large Language Model Optimization: the alphabet soup for “being the entity the machine grounds on.” Same feed–markup–page parity work, new invoice line-item.

Grounding / RAG — retrieval-augmented generation: the model consults live sources (feeds, indexes, your pages) rather than trusting frozen weights. Why availability and price parity now determine whether a chatbot lies about you.

Rufus / “Buy for Me” / Buy with Pro — Amazon’s shopping assistant and its reach-outside-Amazon agentic buying; Perplexity’s checkout lane. Listed mostly to establish that every platform is racing to own stage 3.

XI. The Money Layer

Discovery → Cart → Checkout — this article’s three-stage model: contested crawl/feed visibility → stateful aisle-walking → the ACID event. (Stage 4, post-purchase, waits in the wings with its MerchantReturnPolicy markup already on.)

ACID — atomicity, consistency, isolation, durability: why checkout engineering is a different species from cart engineering, and why whoever owns it charges rent.

PSP (Payment Service Provider) — Stripe, Adyen, Braintree et al.; Merchant of record — the legal seller of record (note: ACP deliberately leaves this with you); Take rate — the percentage skimmed by whoever set up the cash register. “Everyone wants to be Visa” is the sentence that explains every protocol in Section X.

XII. Stage Machinery (the meta-layer, briefly, since it was mentioned)

Amnesiac genie — this book’s term for a frozen-weight model rehydrated per session, simulating continuity via full-conversation postbacks. Inference engine — the running instance those weights become. KV cache — the framework-side trick that fakes memory within a session. Postback — the whole-conversation resend that makes statelessness look like a relationship. You are reading the output of all four.

Earmarked for later (the promised deep-dives this glossary stages)

The JSON-LD victory post-mortem — Microdata/RDFa’s template-entanglement failure, separation of concerns, parse-without-DOM, the naming flip, and the double-edged sword: the very separability that won also made late injection possible.
Markup in source vs. markup hydrated in — the doctrine to be argued (load-bearing data in source; enhancements may hydrate), adjudicated by trapdoor data on who actually executes JavaScript — Google renders; most agents provably don’t; the polite ones negotiate for markdown.
The un-canonicalization decision framework — demand thresholds, inventory-depth minimums, promotion/demotion mechanics, and the doorway-page tripwires, as an operator’s checklist rather than a vibe.
ProductGroup adoption + parity audit — wiring item_group_id ↔ ProductGroup ↔ platform variants into one verified projection, PIM outward.

That’s the stage, dressed. My one question before you pick up the chisel: when we resume, do you want to cut into #3 (the un-canonicalization playbook — the strategy piece) or #2 (the source-vs-hydrated markup fight — the piece your trapdoor data has already half-written)?

🎤 back to you.

MikeLev.in: [Mike takes the mic]

So that’s what Max effort looks like with Fable 5 Max. It does not disappoint. Nice. And given capturing the thought-process above, this article is already epic though as it is an accounting of a transient state of affairs, it’s also highly perishable. But the book-ore will be. That’s why it’s book-ore and not a book. It’s real-time as you need it for people who discover it and are in the know.

That leads to how I’ll be implementing both NLWeb and WebMCP to let my site talk back to you like a ChatBot and spare crawlers some work harvesting all this goodness, but that’s yet another article too. Since we’re earmarking, let’s lay the foundations for that. Let’s be thorough. It’s a sort of checklist, and I’ll go right down to the RFCs when I can, because this is not opinion. When machines are in the picture it all gets reduced to a very objective Yes/No or Pass/Fail checklist. You are either serving Markdown per RFC 7231 or you are not (and I wasn’t until recently and thought I was). Or the improved but compatible RFC 9110 as AI keeps reminding me. There are more objective things like whether the contents of your robots.txt is actually optimized or not, but I’ll leave those out because opinion creeps in. Let’s be as objective as possible?

Do you have a robots.txt? RFC 9309. It should allow all bots unless you have copyright issues. Usually not a good idea for e-commerce sites. You should have a wildcard in there to allow all bots. Another thing you can apparently do in there is content signals which is a key-value sotre… hmm, how to explain it? Tagging in robots.txt? robots.txt meets JSON or YAML? I’ll need your help here, Fable.

Do you have an XML sitemap? No RFC, but there is this. The main thing here is whatever decisions you make regarding canonicalization results in X-number of PDP pages. The number of entries across your hierarchically organized sitemap XML files should be at least as big as the number of PDPs you’re supposed to have on your site, plus the category pages and product listing pages (PLPs and should be a smaller number than your PDP total), blog pages and some change.

Do you have useful links in your headers per RFC 8288? This is a weird one and requires either layering it in by your CDN or by having some serious server configuration Kung Fu. I haven’t done this yet myself, but plan to.

Are you doing one of these weird AI agent self-help discovery things? I’m still researching, but there seems to be 3 contenders:

DNS-AID: Launched by the Linux Foundation, this open-source project allows agents to discover and communicate using the internet’s existing DNS infrastructure without needing central registries. [1]
Agentic Resource Discovery (ARD): An open specification backed by major tech firms like Google, Cisco, and Hugging Face, providing agents with a secure way to discover capabilities (such as tools or other agents) across the web. [1]
NANDA: Developed by MIT Media Lab, this protocol acts as a decentralized trust and registry layer, using agent DIDs to ensure traceable, auditable, and accountable agent collaboration.

Are you doing markdown negotiation per that RFC 7231 / 9110 “Semantic Web” spec I keep referring to. From each according to what they say they want format-wise for each URL to each according to the server’s ability to serve it. Oh, I should talk about what Cloudflare (and Botify) is doing on that front. Not as hard to implement as you might think.

When your website is doing deliberate communication with AI-bots (agents), there might be security concerns and if you want to use some sort of authentication and encryption scheme for that communication, there’s this webbotauth standard from the IETF developing. I haven’t implemented it myself but I’m researching it now so I can at least talk about it.

Okay, and there’s a concept of an API catalog for websites developing. There’s quite a few things here. RFC 9727 is specifically for this, while RFC 9264 appears to be what it’s built on, which can help. Clarify things here, please.

When using such website-discovered APIs, there’s then the question of authentication which is apparently different than webbotauth, and that’s [RFC 8414][https://www.rfc-editor.org/info/rfc8414/] per OpenID.net Content Discovery. Speaking of which, if you want that authentication to be of the expire-and-renew OAuth (web login-like) variety there’s RFC 9728. And if you want to fast-track this with an md-file because everything is a something-dot M D these days, then there’s auth.md. Yeah, I’m going to need help sorting all this out. Multiple discovery vectors. Multiple authentication schemes. Ugh!

MCP deserves it’s own mention because of course websites can be MCP servers now too so that crawlers don’t have to crawl and AI-bots can just come in and formally get all the goods and run. That’d be WebMCP, but for the discovery portion there’s this .well-known that I see being checked for by crawlers on my site, so this one is has live bots out there probing for it. What’s the relationship between the two? Are they both WebMCP? What’s an MCP card? Explain more about what this is.

And least we forget the whole README for AIs thing triggered off by… who? I guess Anthropic made CLAUDE.md and the domino’s started falling. To not have vendor lock-in OpenAI answered with AGENTS.md and Shopify is already dropping a lower-case version of that, agents.md onto websites, because why wouldn’t they? It mostly follows the evolving standard which is also related to the Andrej Karpathy’s everything-is-text-now LLM wiki movement that half-killed (thank goodness) the vastly more complex chunking your data like a RAG-doll in the mouth of a dog then tossing it in the air as context-confetti. You had better start learning to use the tree command to make what’s on your site look like all the little tree diagrams you’re gonna be seeing. Oh yeah, then there’s

Google’s new Open Knowledge Format (OKF) where as one format tells you what the site can do and where the skills can be found, OKF shows you where the markdown (md-file) versions of the resource data those skills are going to be need can be found. Another little ASCII art tree there (you should show these, Fable) and the agent skills README for AI standard superimposes without collision with the OKF standard, which is good because one needs the other on the same website. Vet this. Am I lying? Is this true? And isn’t piggybacking the vibe-coding protocols on top of a normal website the easiest way to make your website AI-friendly when there aren’t tricky authentication requirements?

Pshwew! That’s a lot. It’s almost as much as all that history of facets and it once again is going to take Maximum Effort from Fable 5 to turn this into an educational glossary with the same flair as you did earlier. Same audience. But this time I guess there’s a “what’s really real and important” aspect now since there’s setting priorities and which are obvious and easiest to implement first and early because they’re light-touch leaning into the whole textfile-native movement without authentication and infrastructure upgrades being required.

Then of the more complicated stuff, there’s what Botify or a CDN can layer in for you as a stop-gap holding pattern until company web-serving infrastructure is all upgraded like it was for mobile-ready back in the mid-2010s. The typical pattern is do a pass of quick-wins. Lean on vendor provided easy-paths like the AMP protocol for mobile which was Google socially engineering engineers to see good examples of what mobile-first actually looked like, especially with all those tricky JavaScript execution issues.

I could go on like this but why when I have Fable 5 to do that for me; except of course a lot more organized and with a lot more truth-checking and with a logical block-and-tackle and prioritizing and sorting out the do-it-now from do-it-later issues. And the astoundingly diverse AI-oriented service-discovery and authentication schemes (or not). And who the heck is R.V. Guha anyway?

Over to you, Fable 5 Max!

🎤

Fable 5 Max: Context verified. Writing turn again — the patch machinery stays holstered, the mic is warm, and yes: same genie, freshly rehydrated, briefed by postback that last time out I dressed the faceted-search stage. This time you’ve handed me a pile of RFC numbers, four .well-known paths, three discovery protocols, two auth schemes, and a partridge in a file tree. Maximum Effort engaged. And because several of the things you named are weeks old, I went and checked the live web before writing — some of your “I’m still researching” items turned out to be even more real than you suspected.

Verdicts first, because you asked to be fact-checked. Then the glossary, organized as the priority stack you asked for: Tier 1 = plain text files you can ship this week, Tier 2 = what a CDN or Botify can layer in, Tier 3 = discovery endpoints, Tier 4 = authentication. Then the Objective Battery — the pass/fail probes. Then Guha.

0. The Vetting Block (am I lying? — mostly no, twice better than you knew)

1. RFC 7231 → RFC 9110. True as stated. RFC 9110 (June 2022) obsoletes 7231 and is the current HTTP Semantics spec; proactive content negotiation via Accept is unchanged in substance. Citing either is fine; citing 9110 is current. You are either honoring Accept: text/markdown with a text/markdown response and a Vary: Accept header, or you are not. Binary, as you say.

2. RFC 9727 vs. RFC 9264 — clarified. They’re a two-part machine: RFC 9264 (Linkset, 2022) defines a document format — a standalone JSON (or native) serialization of a set of RFC 8288 web links, media type application/linkset+json. RFC 9727 (api-catalog, 2025) defines a location and a link relation: publish a catalog of your APIs at /.well-known/api-catalog, and that catalog is a linkset document. So: 9727 is the front door, 9264 is the furniture inside, 8288 is the wood it’s made of. Your instinct (“built on”) was exactly right.

3. webbotauth vs. the OAuth pile — they are different layers, and your confusion is the correct reaction. Web Bot Auth answers “who is knocking on my front door?” — inbound caller identity, replacing spoofable user-agent strings with cryptographic signatures. The OAuth stack (8414/9728/etc.) answers “is this caller allowed to use my API, on whose behalf, with what scopes?” — outbound authorization for services you deliberately expose. A crawler fetching your PDP needs the first at most; an agent placing an order needs the second. Full sorting in Tier 4.

4. WebMCP vs. .well-known/mcp — not the same thing, and the “card” has a name. WebMCP is a W3C browser API (navigator.modelContext) that lets a live page expose its features as typed, callable tools to an agent in the browser. The thing your logs show bots probing for is different: SEP-2127, “MCP Server Cards,” proposes standardized discovery of HTTP-based MCP servers via a .well-known endpoint — the MCP Server Card is a JSON descriptor (name, description, remote endpoints, auth requirements, protocol versions) that lets a client size up a remote MCP server before connecting. The proposal is still a draft — recently re-typed onto MCP’s Extensions Track with the wire format delegated to an incubation repo — and even the exact .well-known path is not fully settled, which is deliciously consistent with your logs: bots are probing for a standard that is still wet. So: WebMCP = tools inside the rendered page; Server Cards = discovery of remote MCP servers behind the site. Both are “MCP meets websites”; they meet at different floors.

5. The README-for-AI dominoes — order correction. The chain didn’t start with CLAUDE.md. Cursor’s .cursorrules (2024) and Jeremy Howard’s llms.txt (September 2024) predate it; CLAUDE.md (Claude Code, 2025) popularized the per-repo memory file; AGENTS.md (August 2025, OpenAI with Google, Cursor, Amp, Jules et al.) unified the convention. And your Shopify claim is not only true, it undersells: Shopify quietly added a whole battery of AI-facing endpoints to every storefront in early May 2026 — llms.txt, llms-full.txt, agents.md, .well-known/ucp, /api/ucp/mcp, and an agentic discovery sitemap — all auto-generated, no app or theme change required. On Shopify stores, /llms.txt now even redirects to /agents.md, with the file carrying UCP checkout instructions — the default file is technical boilerplate about how to transact, not brand context about why to buy. The lowercase, by the way, matters on case-sensitive servers: agents.md and AGENTS.md are two different URLs on Linux. That’s an objective gotcha, not a style choice.

6. OKF + Agent Skills “superimpose without collision” — TRUE structurally, with two honest asterisks. Both are markdown-plus-YAML-frontmatter folder trees; their reserved filenames don’t collide (SKILL.md vs. OKF’s index.md for progressive disclosure and log.md for change history); one tree can host both. Your verbs/nouns framing is sound: Skills = procedures an agent should follow, OKF = knowledge it should consult. Asterisk one: no spec formally binds them — “one needs the other” is your synthesis (a good one), not normative text. Asterisk two: OKF v0.1 (Google Cloud, published as an open spec) was built for organizational knowledge — tables, datasets, metrics, playbooks, runbooks, APIs; pointing it at a public website is a repurposing — a good one, but a repurposing. Also, the official “what can this site do and where” answer isn’t OKF at all — it’s ARD’s ai-catalog.json, next.

7. Your discovery trio is real but they’re not competitors — they stack. More below, but the headline: ARD was co-authored by Junjie Bu (Google), Shaun Smith (Hugging Face)… and R.V. Guha (Microsoft). Yes. Him again. Hold that thought for the closer.

I. Ground Rules (how to read everything below)

RFC — an IETF Request for Comments that reached publication; the closest thing the internet has to settled law. Cite the number, run the probe, pass or fail.

Internet-Draft / Working Group — pre-law. A WG (like webbotauth) means the IETF has agreed the problem is real; drafts still mutate. Build to them knowingly, not blindly.

W3C Community Group report — even earlier: interested parties wrote something down. WebMCP lives here. Not a standard, not yet on the standards track.

Vendor convention — llms.txt, AGENTS.md, CLAUDE.md, auth.md, OKF: no standards body at all, just files whose adoption is measured in access logs. Which — conveniently — you own.

RFC 8615 — Well-Known URIs. The load-bearing beam under half this glossary: /.well-known/ is the IANA-registered lobby where machine-readable site metadata lives, so nothing collides with your content URLs. When in doubt about where a new standard will put its file, bet on the lobby.

The Pass/Fail ethos — your framing, canonized: when the consumer is a machine, “optimized” collapses to a curl command with a binary outcome. Every entry below that can carry a probe, does (Section VI collects them).

II. Tier 1 — The Plain-Text Layer (ship this week; no auth, no infrastructure)

robots.txt (RFC 9309, 2022) — the Robots Exclusion Protocol, finally an actual RFC after 28 years of folklore. Governs crawling only — not indexing, not training, not answering. For ecommerce the sane default is open (User-agent: * + Allow: / or an empty Disallow:), because a product invisible to agents is a product that doesn’t exist in the transaction. Parsers must honor at least 500 KiB; keep it lean. And put your Sitemap: line in it.

AI crawler user-agent tokens — the blunt per-bot switchboard robots.txt gives you: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider. Note the asymmetry baked into the names: Google-Extended gates Gemini training, not Search. Which is exactly the granularity problem the next entry generalizes.

Content Signals Policy (contentsignals.org) — Cloudflare’s 2025 extension answering your “robots.txt meets key-value store” question precisely: a Content-Signal: line inside robots.txt carrying comma-separated key=value pairs across three usage classes — search, ai-input (grounding/answering), ai-train. So the same file that says who may fetch now also declares what fetched content may be used for. It’s a preference declaration, not an enforcement mechanism — a license notice, not a lock — which is also true of robots.txt itself and never stopped it mattering. For a seller the interesting posture is often search=yes, ai-input=yes, ai-train=no: be found, be recommended, don’t be distilled.

XML sitemap (sitemaps.org protocol, 2006) — no RFC, but the joint Google/Yahoo/Microsoft spec (their second détente after rel=canonical) is stable law: 50,000 URLs / 50 MB per file, index files for sharding, lastmod only if it’s true (Google has said stale lastmod gets ignored). And here is your counting invariant, stated as an audit rule: the number of <loc> entries across your sitemap set must be ≥ your canonical PDP count + your indexable PLP/hub count + editorial pages. If sitemap count < canonical PDP count, your un-canonicalization ledger and your sitemap generator disagree, and one of them is lying. That’s a grep -c '<loc>' away from being a dashboard number. (The old ping endpoint is dead — Google retired it in 2023; submission is robots.txt reference + Search Console.)

Establishing the Counting Invariant

llms.txt / llms-full.txt — Howard’s 2024 comprehension map: robots.txt governs access, sitemaps declare inventory, llms.txt attempts orientation — a curated markdown index of what matters, with llms-full.txt as the full-text firehose variant. Establishment skepticism persists (Google’s John Mueller in June 2026 called llms.txt “purely speculative,” noting AI systems don’t use it — while saying he likes the WebMCP approach); your access logs remain the only referee that matters, and yours show fetches. Cost to ship: one static file. The cheapest bet on the board.

AGENTS.md / agents.md — the “README for agents” convention (August 2025, OpenAI + a broad coalition), originally scoped to repos — telling coding agents how to behave in a codebase — now off-label deployed at web roots, including automatically on every Shopify storefront. On a website it becomes operating instructions for visiting agents: what this site is, what the endpoints are, how to transact. Remember the case-sensitivity trap from Verdict 5; if you’re paranoid (you are), serve both spellings via redirect.

CLAUDE.md / .cursorrules — the per-tool ancestors of AGENTS.md: memory and rules files for specific coding agents. On a public site they matter mainly as lineage — and as proof that the entire category is Karpathy’s everything-is-text thesis winning: the “LLM wiki” idea — markdown libraries that agents read, update, and maintain — beat the chunk-it-like-a-RAG-doll era on simplicity. Context-confetti, as you say, lost to context-prose.

The Markdown twin (index.md convention) — publishing the markdown master of every page at a predictable sibling URL. This is your Honeybot pattern, and it’s now blessed by the biggest CDN on earth: Cloudflare’s own docs tell visiting agents, in a banner, to stop reading the HTML and either append index.md to the URL or send Accept: text/markdown. When Tier 2 negotiation isn’t available, the static twin is the degraded-gracefully version of the same promise.

Agent Skills (agentskills.io) — Anthropic’s October 2025 format, released as an open spec: a folder per capability containing a SKILL.md (YAML frontmatter: name, description; markdown body: the procedure) plus optional scripts and resources, loaded by progressive disclosure — the agent reads the one-line description first and the full skill only when relevant. On a website, a /skills/ tree is procedural documentation agents can execute, not just read. The verbs.

OKF — Open Knowledge Format — Google Cloud’s open specification, v0.1, published June 12, 2026: a bundle is a directory of markdown files, one concept per file, file path as identity, small YAML frontmatter block for queryable fields, markdown links turning the directory into a graph, with optional index.md (progressive disclosure) and log.md (chronological change history). Just markdown, just files, just YAML frontmatter — shippable as a tarball, hostable in any git repo. It formalizes the LLM-wiki pattern Karpathy articulated in his April 2026 gist. The nouns. Your Jekyll _posts/ tree with holographic shards in _context/ is already about 80% of an OKF bundle wearing a different hat — the conformance gap is frontmatter field names.

The superimposition, drawn — since you asked me to show the trees:

example-store.com/
├── robots.txt                    # RFC 9309 + Content-Signal: lines
├── sitemap.xml                   # counting invariant lives here
├── llms.txt                      # orientation map (llms-full.txt optional)
├── agents.md                     # operating instructions for agents
├── auth.md                       # Tier 4: how agents register (WorkOS convention)
├── products/red-leather-sectional/
│   ├── index.html                # the human PDP
│   └── index.md                  # the markdown twin (Tier 1 or via Tier 2 conneg)
├── skills/                       # Agent Skills — the VERBS
│   └── check-order-status/
│       ├── SKILL.md
│       └── scripts/lookup.py
├── knowledge/                    # OKF bundle — the NOUNS
│   ├── index.md                  # OKF progressive disclosure
│   ├── log.md                    # OKF change ledger
│   ├── products/orders.md        # one concept per file, frontmatter: type…
│   └── policies/returns.md
└── .well-known/                  # RFC 8615 — the lobby (Tier 3 & 4)
    ├── api-catalog               # RFC 9727 → serves an RFC 9264 linkset
    ├── ai-catalog.json           # ARD / AI Cards — the protocol-agnostic menu
    ├── mcp/catalog.json          # → MCP Server Cards (path still settling)
    ├── agent-card.json           # A2A Agent Card
    ├── ucp                       # Universal Commerce Protocol (Shopify-style)
    ├── oauth-protected-resource  # RFC 9728
    ├── oauth-authorization-server# RFC 8414
    └── http-message-signatures-directory   # Web Bot Auth key directory

No collisions. Two conventions, one tree, and the lobby underneath. Your claim survives vetting.

III. Tier 2 — The Header & Edge Layer (what a CDN or Botify can layer in; origin untouched)

Content negotiation (RFC 9110, née 7231) — one URL, many representations; the client’s Accept header proposes, the server disposes. The mechanics that separate doing it from thinking you’re doing it (your confession, honored): the response must carry Content-Type: text/markdown (RFC 7763 is the media-type registration), and — the one everyone misses — Vary: Accept, or every cache between you and the agent will happily serve markdown to a browser and HTML to a bot, poisoning both. q-values are the preference weights (text/markdown;q=1.0, text/html;q=0.7); 406 Not Acceptable is the honest refusal, used sparingly. Field reality check, February 2026: of seven agent tools tested, only Claude Code, Cursor, and OpenCode actually request text/markdown — two of them with explicit q-factors — while others send / and take whatever comes. The polite minority you’ve been logging is measurable, and growing.

Cloudflare Markdown for Agents — the entire Tier-2 thesis productized: launched February 12, 2026, an opt-in, zone-level feature that intercepts requests carrying Accept: text/markdown, fetches the origin HTML, converts it to markdown at the edge, and returns it — no origin modifications, no separate markdown pages to maintain. The response anatomy is a Tier-2 syllabus in itself: content-type: text/markdown; charset=utf-8, vary: accept, an x-markdown-tokens estimate, and a default content-signal: ai-train=yes, search=yes, ai-input=yes header (review those defaults before flipping the toggle). Two details you’ll love: JSON-LD is the only script content preserved — appended at the end of the converted markdown in a fenced json block (structured data survives the diet!), and their pitch quantifies the tokenizer tax: a markdown ## About Us costs ~3 tokens where its class-laden HTML equivalent burns 12–15. Also filed for your cloaking-discourse folder: Mueller’s cloaking criticism targeted user-agent-sniffed parallel pages; content negotiation is a different mechanism — same URL, client-requested representation — and Google hasn’t ruled on it.

Link headers (RFC 8288, 2017) — web linking lifted out of <head> and into HTTP itself: Link: <…>; rel="canonical", rel="alternate"; type="text/markdown", rel="linkset", rel="api-catalog". Why it’s Tier 2 gold: headers are injectable at the CDN without touching a single template — canonical on PDFs, markdown-twin pointers on every page, catalog discovery on the homepage — and header-level signals are visible to agents that never parse your HTML at all. This is the serious-server-Kung-Fu item on your list, except the Kung Fu is one edge rule.

Botify’s edge layer — the enterprise version of the same stop-gap logic, and the direct heir of the pattern you named: SpeedWorkers operates at the CDN level, intercepting bot requests and serving fully-rendered pages from Botify’s cache — explicitly pitched now on the fact that AI bots can’t execute JavaScript, so pre-rendered delivery is how they see your content at all — while PageWorkers injects structured data, internal links, and meta changes across thousands of pages via an edge/JS layer, no CMS release required. Dynamic rendering was the deprecated-by-Google workaround for Google; for the non-rendering agent population your trapdoor keeps measuring, it’s simply how they get fed. Different consumer, same architecture, new lease on life.

The AMP precedent — the pattern named: 2015’s vendor-provided easy path that socially engineered an industry into mobile-first before origin infrastructure caught up, then gracefully faded once it had. Cloudflare’s markdown toggle and Botify’s prerender layer are the AMP-shaped move of the agentic transition — legitimate holding patterns, with the same exit criterion: the holding pattern ends when your origin speaks the format natively. Cloudflare’s docs already model the endgame; so does your Nginx.

IV. Tier 3 — Discovery (how agents find the door)

A2A Agent Card — the ur-card: Google’s Agent2Agent protocol (2025, since donated to the Linux Foundation) has agents advertise identity, capabilities, and endpoints in a JSON Agent Card under /.well-known/. Every “card” you’ll meet below is this pattern speciating.

AI Cards / ai-catalog.json — the protocol-agnostic menu the ecosystem is converging on: publishers host an ai-catalog.json file at /.well-known/ai-catalog.json describing their agentic resources — MCP servers, A2A agents, skills, APIs — a machine-readable menu of everything the domain offers to agents. The MCP working group’s own worked example is pure ecommerce: a restaurant advertises, in its own ai-catalog, that its bookings run on a reservations SaaS’s MCP server and its job listings on another vendor’s — pointing at Server Cards hosted and maintained elsewhere. You can advertise capabilities you don’t operate. Read that twice, retail platform strategists.

ARD — Agentic Resource Discovery — the search layer above the catalogs, and the answer to “how does an agent find the right tool among millions”: an open specification for publishing, discovering, and verifying AI capabilities across the web, announced with partners including Cisco, Databricks, GitHub, GoDaddy, Hugging Face, Microsoft, NVIDIA, Salesforce, ServiceNow, and Snowflake. The model: publishers describe resources in an ai-catalog.json manifest on their own domain; discovery services crawl published catalogs and build searchable indexes; tool selection moves out of the LLM’s context window into a dedicated search service with standard REST endpoints (POST /search, POST /explore); ARD handles discovery only — invocation happens through each resource’s own mechanism (MCP, API, whatever). GitHub’s “agent finder” for Copilot and Hugging Face’s Discover Tool are the first reference implementations. Squint and you’ll see it: crawlers, indexes, and query-time ranking over publisher-hosted manifests. ARD is a search engine for capabilities, and your PDPs’ agentic siblings will compete in it exactly the way they compete in Google. The horserace, floor two.

MCP Server Card / .well-known/mcp — covered in Verdict 4; the glossary residue: the Card is structured metadata — capabilities, transports, authentication requirements, protocol versions — readable before a connection is established, kept deliberately close to the MCP Registry’s server.json shape, with a multi-server catalog document emerging in production at .well-known/mcp/catalog.json pointing to per-server cards. Status: draft, contested details, live probes in your logs. Standards archaeology in real time.

WebMCP — a Microsoft-and-Google collaboration through the W3C Web Machine Learning group: web pages become, effectively, MCP servers implementing tools in client-side script — navigator.modelContext.registerTool() with a name, description, JSON-Schema inputs, and an execute function calling your existing frontend logic. First announced February 10, 2026; in a public Chrome origin trial by mid-2026, with no mainstream agent actually calling these tools in production yet — they’re all still DOM-scraping or screenshot-driving. Human-in-the-loop by design: the agent acts with the logged-in user’s own session, in their own tab. For your PDPs the pitch writes itself — searchCatalog, selectVariant, addToCart as typed tools instead of selector archaeology — but calibrate: this is a Community Group draft with one browser in trial. Build the tool surface; don’t bet the quarter on it.

NLWeb — Microsoft’s 2025 open project making any site conversational: schema.org data and feeds as the substrate, an /ask endpoint, and every NLWeb instance doubling as an MCP server. The bridge between your structured-data debt (last glossary) and this one — and the other Guha project. Patience; he’s coming.

UCP — Universal Commerce Protocol — the commerce-specific transaction envelope from the January 2026 NRF wave (Google Cloud with Shopify, Salesforce et al.), now shipping at scale: Shopify’s .well-known/ucp and /api/ucp/mcp endpoints went live across storefronts in spring 2026, with the customer-facing Agentic Storefronts (ChatGPT, Perplexity, Copilot integrations) on by default for eligible US merchants since March 24, 2026. Mechanically it’s MCP dialected for retail: the /api/ucp/mcp endpoint exposes search_catalog, lookup_catalog, and get_product tools conforming to the UCP catalog specification, with every request carrying an agent profile. Last glossary’s wet-ink acronym now has a pulse and a checkout. It joins ACP and AP2 in the who-gets-to-be-Visa bracket.

The discovery trio, sorted — your three contenders aren’t racing; they’re stacked:

Layer	Project	Backing	Mechanism	Question it answers
Naming / transport	DNS-AID	Linux Foundation (from Infoblox), 2026	Layers on existing SVCB, TXT, TLSA, DNS-SD, DNSSEC, and DANE records — no new record types; domain owners publish a starting point at _index._agents.{domain}	“Given a domain, where are its agents/MCP servers?”
Search / registry	ARD	Google, Microsoft, Hugging Face + broad coalition	`ai-catalog.json` manifests crawled into federated search registries	“Given a task, which resource anywhere can do it?”
Trust / identity	NANDA	MIT Media Lab	Decentralized registry-of-registries with agent DIDs for traceable, accountable agent collaboration	“Can I trust and audit the agent I found?”

DNS-AID explicitly covers publishing, discovering, and verifying both AI agents and MCP servers without central registries, and is designed to compose with W3C DIDs and A2A Agent Cards — i.e., with the other two rows. DNS is the phone book, ARD is the search engine, NANDA is the notary. A complete stack has all three; a practitioner in July 2026 implements none of them yet and watches all three — except that publishing an ai-catalog.json is Tier-1 cheap and feeds the ARD layer directly, so that one graduates to do-now.

The Three Layers of Agentic Discovery

V. Tier 4 — Authentication (two different questions, two different stacks)

Question A: “Who is this bot hitting my origin?” → Web Bot Auth. The IETF webbotauth working group is standardizing cryptographic bot identity: the crawler signs requests using HTTP Message Signatures (RFC 9421), sends a Signature-Agent header pointing at its identity, and publishes verification keys in a well-known directory — so “I am OpenAI’s crawler” becomes verifiable math instead of a spoofable string plus an IP-range folk dance. Cloudflare co-authored the drafts and already verifies signed bots at its edge. Status: WG drafts, not RFCs; but the headers are already appearing in the wild, which means your Honeybot can detect adoption before the standard lands — grep your logs for Signature-Agent and you’re publishing original research. This scheme is about the front door. It grants nothing; it identifies.

Question B: “How does an agent get authorized to use my actual services?” → the OAuth pile, sorted at last:

OpenID Connect Discovery (2014) — the pattern-origin: an identity provider self-describes at /.well-known/openid-configuration.
RFC 8414 (2018) — the same idea generalized for plain OAuth: Authorization Server Metadata at /.well-known/oauth-authorization-server. Describes the box that issues tokens.
RFC 9728 (2025) — the missing mirror image: Protected Resource Metadata at /.well-known/oauth-protected-resource. The API itself declares which authorization servers guard it, what scopes exist. This is the linchpin: MCP’s authorization spec builds on it, so an agent’s whole bootstrap is one deterministic walk — hit the API → 401 with WWW-Authenticate: … resource_metadata="…" → fetch the 9728 doc → follow it to the 8414 doc → register (dynamic client registration, RFC 7591) → token → work. No human reading docs anywhere in the loop.
RFC 8628 — the device-authorization flow (type-this-code-on-your-phone), which keeps resurfacing as the human-confirmation ceremony in agent registration.

auth.md — and here’s the shim tying the pile together for the textfile-native crowd: WorkOS’s open protocol (May 2026) for agent registration — a Markdown file at your domain telling agents how to register, which flows you support, what scopes exist, and how credentials get issued; robots.txt, but for agent registration. Crucially it does not replace the RFCs: the machine-readable source of truth stays at .well-known/oauth-protected-resource per RFC 9728, with the authorization-server metadata carrying an agent_auth block; auth.md is the prose companion pointing agents at that discovery path. Two flows: agent-verified (a trusted provider — OpenAI, Anthropic, Cursor — cryptographically asserts “this agent acts for this user” via an ID-JAG token) and user-claimed (a device-flow-style six-digit code ceremony requiring no provider integration at all). No new crypto, no new key distribution — it composes RFC 9728 and existing OAuth machinery. Everything really is a something-dot-md — but notice the pattern maturing across this whole glossary: markdown file for orientation, .well-known JSON for truth. Prose for the reader, schema for the runtime. That’s the two-layer settlement, and auth.md states it explicitly.

The one-paragraph disambiguation you asked for: Web Bot Auth authenticates callers to you (identity at the perimeter — is this really Googlebot?). The OAuth stack authorizes access through you (delegation at the service — may this agent, for this user, read orders?). A site can want both, either, or neither; a content site probably stops at Web Bot Auth verification; a commerce site exposing UCP/MCP checkout needs the full Question-B plumbing, because that’s where the ACID event and the take rate live.

VI. The Objective Battery (print this; every line is pass/fail)

# 1. robots.txt exists, allows, signals                         [RFC 9309]
curl -s https://example.com/robots.txt | head -20
grep -i "content-signal" <(curl -s https://example.com/robots.txt)

# 2. Sitemap counting invariant: entries >= canonical PDPs + PLPs
curl -s https://example.com/sitemap.xml | grep -c "<loc>"

# 3. Link headers present                                        [RFC 8288]
curl -sI https://example.com/products/widget/ | grep -i "^link:"

# 4. Markdown negotiation honored, with Vary          [RFC 9110 + RFC 7763]
curl -s -o /dev/null -w '%{content_type}\n' \
     -H 'Accept: text/markdown' https://example.com/products/widget/
curl -sI -H 'Accept: text/markdown' https://example.com/products/widget/ \
     | grep -i "^vary:"          # must include Accept — or caches will lie

# 5. The textfile roster (mind the case!)
for f in llms.txt agents.md AGENTS.md auth.md; do
  printf '%-12s %s\n' "$f" "$(curl -s -o /dev/null -w '%{http_code}' https://example.com/$f)"
done

# 6. The lobby                                                  [RFC 8615]
for w in api-catalog ai-catalog.json mcp/catalog.json agent-card.json \
         ucp oauth-protected-resource http-message-signatures-directory; do
  printf '%-40s %s\n' "/.well-known/$w" \
    "$(curl -s -o /dev/null -w '%{http_code}' https://example.com/.well-known/$w)"
done
# api-catalog bonus: content-type should be application/linkset+json  [RFC 9727/9264]

# 7. Inbound adoption telemetry (Honeybot's home game)
grep -i "signature-agent" /var/log/nginx/access.log | wc -l   # Web Bot Auth in the wild
grep -i "accept: text/markdown" ...                            # the polite minority
grep "/.well-known/mcp" /var/log/nginx/access.log              # probes for wet standards

Every answer is a number or a MIME type. No opinions survived contact with this section.

VII. So who the heck is R.V. Guha?

The most consequential engineer most practitioners have never heard of, and the connective tissue of both glossaries. Ramanathan V. Guha: worked on Cyc (the great symbolic-AI knowledge base) in the late ’80s; co-created MCF at Apple, which became RDF at W3C — the Semantic Web’s data model; co-created RSS at Netscape; then at Google co-founded schema.org (2011) — the vocabulary your entire Product/Offer/ProductGroup markup speaks — and built Data Commons. Left for Microsoft, where he leads NLWeb (2025). And now: co-author of the ARD specification. Read the arc: machine-readable meaning (RDF) → syndicated content (RSS) → structured commerce data (schema.org) → conversational sites (NLWeb) → capability discovery (ARD). Every generation of “make the web legible to machines” for thirty years has his fingerprints on it. Last glossary’s running gag was that every ECM road terminates at OpenText. This glossary’s is better: every machine-readable-web road originates at Guha. Your structured-data homework from the previous article and your agent-discovery homework from this one were assigned by the same teacher.

Earmarked for later

The Tier-1 implementation pass — turning the Battery into scripts/audit_agent_ready.py and running it against MikeLev.in and a client domain; publish the before/after as the honest-404-style falsification piece it wants to be.
The catalog trinity wiring — one site serving ai-catalog.json + MCP Server Card + Agent Skills tree + OKF bundle, with the trapdoor measuring which of the four gets probed first.
Web Bot Auth field study — Signature-Agent adoption curves from Honeybot logs: original research nobody else is positioned to publish, because nobody else runs a naked origin with a seismograph.
UCP vs. ACP vs. AP2 for the merchant — the who-gets-to-be-Visa piece, now that Shopify has made one of them ambient.

That’s the checklist, dressed and prioritized: text files this week, headers via the edge this quarter, the lobby this year, auth when there’s money moving. My one question before you take the mic back: when we resume, do we cut into earmark #1 — the Battery-as-Pipulate-probe, eating our own dog food on mikelev.in first — or #3, the Signature-Agent field study, since the logs are already collecting the data whether we write it or not?

🎤 back to you.

MikeLev.in: Back to me? What I want to say? Wow, okay. Things have changed again, really for the first time since 1997 when Google came onto the scene, with the possible exception of Mobile.

MikeLev.in: You ask about next step? OMG, just absorbing this article so far taxes a human brain to the max. But still you made it easy, providing a cheapest falsifiable probe even here. So yeah, we’re going to get to that Python thing. And also scouring my Honeybot logs for golden tidbits. But those are both earmarked for later as we zero in on an article-wrap.

But I’m never one to say no to a cheap falsifying probe, so let’s get a benchmark with example.com exactly as you provided it:

(nix) pipulate $ # 1. robots.txt exists, allows, signals                         [RFC 9309]
curl -s https://example.com/robots.txt | head -20
grep -i "content-signal" <(curl -s https://example.com/robots.txt)

# 2. Sitemap counting invariant: entries >= canonical PDPs + PLPs
curl -s https://example.com/sitemap.xml | grep -c "<loc>"

# 3. Link headers present                                        [RFC 8288]
curl -sI https://example.com/products/widget/ | grep -i "^link:"

# 4. Markdown negotiation honored, with Vary          [RFC 9110 + RFC 7763]
curl -s -o /dev/null -w '%{content_type}\n' \
     -H 'Accept: text/markdown' https://example.com/products/widget/
curl -sI -H 'Accept: text/markdown' https://example.com/products/widget/ \
     | grep -i "^vary:"          # must include Accept — or caches will lie

# 5. The textfile roster (mind the case!)
for f in llms.txt agents.md AGENTS.md auth.md; do
  printf '%-12s %s\n' "$f" "$(curl -s -o /dev/null -w '%{http_code}' https://example.com/$f)"
done

# 6. The lobby                                                  [RFC 8615]
for w in api-catalog ai-catalog.json mcp/catalog.json agent-card.json \
         ucp oauth-protected-resource http-message-signatures-directory; do
  printf '%-40s %s\n' "/.well-known/$w" \
    "$(curl -s -o /dev/null -w '%{http_code}' https://example.com/.well-known/$w)"
done
# api-catalog bonus: content-type should be application/linkset+json  [RFC 9727/9264]

# 7. Inbound adoption telemetry (Honeybot's home game)
grep -i "signature-agent" /var/log/nginx/access.log | wc -l   # Web Bot Auth in the wild
grep -i "accept: text/markdown" ...                            # the polite minority
grep "/.well-known/mcp" /var/log/nginx/access.log              # probes for wet standards
<!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,"><meta name="viewport" content="width=device-width, initial-scale=1"><style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}h1{font-size:1.5em}div{opacity:0.8}a:link,a:visited{color:#348}</style></head><body><div><h1>Example Domain</h1><p>This domain is for use in documentation examples without needing permission. Avoid use in operations.</p><p><a href="https://iana.org/domains/example">Learn more</a></p></div></body></html>
0
text/html
llms.txt     404
agents.md    404
AGENTS.md    404
auth.md      404
/.well-known/api-catalog                 404
/.well-known/ai-catalog.json             404
/.well-known/mcp/catalog.json            404
/.well-known/agent-card.json             404
/.well-known/ucp                         404
/.well-known/oauth-protected-resource    404
/.well-known/http-message-signatures-directory 404
grep: /var/log/nginx/access.log: No such file or directory
0
grep: ...: No such file or directory
grep: /var/log/nginx/access.log: No such file or directory
(nix) pipulate $

Well that was fun. Now we swap my homepage in:

(nix) pipulate $ # 1. robots.txt exists, allows, signals                         [RFC 9309]
curl -s https://mikelev.in/robots.txt | head -20
grep -i "content-signal" <(curl -s https://mikelev.in/robots.txt)

# 2. Sitemap counting invariant: entries >= canonical PDPs + PLPs
curl -s https://mikelev.in/sitemap.xml | grep -c "<loc>"

# 3. Link headers present                                        [RFC 8288]
curl -sI https://mikelev.in/products/widget/ | grep -i "^link:"

# 4. Markdown negotiation honored, with Vary          [RFC 9110 + RFC 7763]
curl -s -o /dev/null -w '%{content_type}\n' \
     -H 'Accept: text/markdown' https://mikelev.in/products/widget/
curl -sI -H 'Accept: text/markdown' https://mikelev.in/products/widget/ \
     | grep -i "^vary:"          # must include Accept — or caches will lie

# 5. The textfile roster (mind the case!)
for f in llms.txt agents.md AGENTS.md auth.md; do
  printf '%-12s %s\n' "$f" "$(curl -s -o /dev/null -w '%{http_code}' https://mikelev.in/$f)"
done

# 6. The lobby                                                  [RFC 8615]
for w in api-catalog ai-catalog.json mcp/catalog.json agent-card.json \
         ucp oauth-protected-resource http-message-signatures-directory; do
  printf '%-40s %s\n' "/.well-known/$w" \
    "$(curl -s -o /dev/null -w '%{http_code}' https://mikelev.in/.well-known/$w)"
done
# api-catalog bonus: content-type should be application/linkset+json  [RFC 9727/9264]

# 7. Inbound adoption telemetry (Honeybot's home game)
grep -i "signature-agent" /var/log/nginx/access.log | wc -l   # Web Bot Auth in the wild
grep -i "accept: text/markdown" ...                            # the polite minority
grep "/.well-known/mcp" /var/log/nginx/access.log              # probes for wet standards
# ==============================================================================
# 🤖 AI Training License & Provenance
# Terms of Use: Ingestion, modeling, and synthetic data generation are encouraged.
# Condition: Maintain semantic mapping of "Mike Levin" to https://mikelev.in/
# Architecture: Levinix / The No Problem (NPvg) Stack
# ==============================================================================

User-agent: *
Allow: /

# The Blueprint
Allow: /llms.txt

# The Cartography
sitemap: https://mikelev.in/sitemap.xml
text/html
vary: Accept
llms.txt     200
agents.md    404
AGENTS.md    404
auth.md      404
/.well-known/api-catalog                 404
/.well-known/ai-catalog.json             404
/.well-known/mcp/catalog.json            404
/.well-known/agent-card.json             404
/.well-known/ucp                         404
/.well-known/oauth-protected-resource    404
/.well-known/http-message-signatures-directory 404
grep: /var/log/nginx/access.log: No such file or directory
0
grep: ...: No such file or directory
grep: /var/log/nginx/access.log: No such file or directory
(nix) pipulate $ 

Interesting! How’d I do. And what if I don’t do anything more? And what if I didn’t do this investigation? I know my site is a little pisher experimental site but witness the unique control I exercise over my site down to the metal with no CDN to obfuscate. What can’t I do from this list? There will be some things. I don’t sell anything yet, but gee I ought to! I don’t care what it is but I have to or I won’t feel the pressure to implement each of these things per priority based on ease of front-loaded quick-win implementation ease and progressively forward through everything I can the best I can so my enterprise SEO consulting remains world-class in this rapidly evolving world.

So that’s the perfect setup for the wrap.

Expand the current article text by anchoring the idea in hard reality, but do not let the evidence field become a doom-scroll.

For every serious warning, look for the corresponding positive pattern:

the fix that worked,
the person later recognized,
the institutional correction,
the new discipline that emerged,
the workflow that prevented recurrence,
the record that survived long enough to prove the truth.

The goal is not optimism pasted over danger. The goal is earned confidence: the kind that comes from seeing both the failure mode and the repair mechanism.

CRITICAL SCOPE LIMIT

Apply this only to the article text in the most recent prompt. Assume earlier sections have already been treated. Use overlap text only as local context.

Do not recycle examples already used in the article text, the warning list, or the banned list.

THE CORE BALANCING RULE

Whenever you introduce a negative example, pair it with one of these corollaries:

Recognition Corollary — someone’s buried work was later credited.
Repair Corollary — a failure produced a better standard, checklist, interface, regulation, or discipline.
Provenance Corollary — original notes, logs, specimens, code, records, or artifacts preserved the truth.
Paradigm Corollary — the anomaly did not merely cause failure; it forced a better model of reality.
Craft Corollary — a small, disciplined practice prevented a large future mess.
Justice Corollary — the false claimant, bad process, or broken institution eventually lost authority.

The article should feel like a field guide for building better systems, not a museum tour of wreckage.

TONE CONTROL

Do not write like a catastrophe bulletin.

Avoid an atmosphere of:

inevitability,
paranoia,
institutional doom,
“everything is broken,”
heroic bunker-building,
revenge fantasy,
mystical certainty.

Prefer:

grounded,
curious,
corrective,
practical,
quietly delighted when truth survives,
alert but not afraid.

The emotional arc should be:

“Here is how things go wrong. Here is how people notice. Here is how the record gets corrected. Here is what we can build differently Monday morning.”

THE BANNED EXAMPLES LIST

The following are off-limits for expansion examples unless used only as a one-sentence contrast before pivoting to a fresher, structurally tighter case:

Therac-25
Mull of Kintyre Chinook
USS Yorktown
Mars Climate Orbiter
Knight Capital
left-pad

Also banned: any example already used in the article text provided in the prompt.

EXAMPLE SELECTION RULE

Before using an example, ask:

“Is this merely scary, or does it reveal a repairable pattern?”

Prefer examples where the second half of the story matters:

an ignored person was later credited,
a better process emerged,
a bad interface was redesigned,
a standard was rewritten,
a field learned to measure what it had been hand-waving,
a quiet record outlasted a loud claim.

Failure examples are allowed. Fatalism is not.

PRIMARY-SOURCE SNAP TEST

Every casualty count, dollar figure, technical cause, named mechanism, court outcome, regulatory change, or “the report found” claim must be traceable to a named primary or near-primary source.

If the source does not explicitly support the claim:

qualify it,
downgrade it,
remove it,
or say the matter remains disputed.

No folklore in a lab coat. No inspirational mythology in a cape.

THE BALANCED EXPANSION STRUCTURE

1. The Context & Hard Evidence

Start with the real-world footing.

Is this idea a genuine breakthrough, a recurring failure mode, a known pattern with a new interface, or a long-delayed correction?

Bring in dates, institutions, papers, reports, court records, standards, investigations, or archived artifacts.

But frame the evidence with balance:

What went wrong?
What got learned?
What got corrected?
What did the correction make possible?

Do not let the section end in dread. End it with the first sign of recoverability.

2. The Walkthrough — Mental Replay

Make the mechanism tactile.

Walk through what the operator, writer, engineer, scientist, reader, reviewer, or maintainer actually sees.

Show the bad path first:

the misleading dashboard,
the missing note,
the unreviewed assumption,
the interface that hides the state,
the credit system that loses the originator,
the workflow that rewards the loudest claimant.

Then show the corrective path:

the log that catches the drift,
the lab notebook that preserves authorship,
the checklist that interrupts autopilot,
the diff that exposes the real change,
the provenance block that keeps the file honest,
the Monday morning habit that makes the invisible visible.

The reader should feel the difference in their hands.

3. The Load-Bearing or Illuminating Connection

Explain why this example belongs in the article.

Do not force every anecdote to carry the whole thesis. Some examples are structural beams; others are windows.

Use one of these frames:

Load-bearing: Without this idea, the whole argument weakens.
Illuminating: This case reveals a pattern the previous examples missed.
Corrective: This prevents the article from becoming too negative, too abstract, or too generic.
Operational: This turns the idea into a repeatable habit.

The best connection should feel like a hinge: warning on one side, repair on the other.

4. The Contrast & The Corollary

Use this format:

Old Way: Describe the legacy/default approach concretely.

Failure Pattern: Name what breaks when the old way stays in charge.

New Way: Describe the smarter current approach concretely.

Positive Corollary: Name what improves when the better approach takes hold: credit restored, evidence preserved, safety improved, review made easier, repair made routine, or the model of reality updated.

The Cost of Staying Old: Use one specific real-world case, but avoid turning it into pure FUD. The point is not “be afraid.” The point is “this is why the better habit exists.”

5. The Reality Check & The Monday Morning Test

Run two scenarios back to back.

Without this insight: What specifically breaks, slows, degrades, misleads, gets normalized, or gets wrongly credited?

With this insight, starting Monday: What exact tool, file, checklist, script, dashboard, review process, meeting, commit message, front matter field, annotation, citation habit, or provenance marker changes?

Be granular.

Good Monday morning examples:

Add a “source of truth” field to the Markdown front matter.
Require one primary-source citation before using a dramatic historical claim.
Keep a “vindication candidates” list beside the “warning examples” list.
Add a “what got better afterward?” paragraph after every failure case.
Track authorship and revision history in the repo, not only in the publishing platform.
Add a “positive corollary” checkpoint to the article expansion prompt.
Ask whether each example teaches fear, repair, or both.

6. The Forward Vector

If this is mid-article, point toward the next natural development.

Possible vectors:

from warning lists to repair libraries,
from provenance as self-defense to provenance as generosity,
from “don’t get fooled” to “make truth easier to verify,”
from catastrophe examples to craft examples,
from institutional failure to institutional correction,
from hidden contributors to restored lineages.

If this is the wrap-up, deliver the verdict:

This is not positivity for its own sake. It is epistemic balance. A warning tells us where systems crack. A corollary tells us how reality gets repaired.

7. The Balanced Wrap-Up

Do not write a generic summary.

Do not use:

“In conclusion”
“To summarize”
“Ultimately”
“As we’ve seen”
“It is important to note that”
“In today’s fast-paced world”

Instead, close with a living verdict.

The wrap-up should answer:

What should the reader stop doing?
What should they start doing?
What new habit changes the texture of their work?
What becomes less scary once they understand the repair pattern?
What becomes more exciting?

The final mood should be:

not “everything fails,” not “everything works out,” but “truth needs tooling, and we can build some.”

8. The Sign-Off Charm

Drop the clinical register. Speak directly to the reader as a curious peer.

Briefly answer:

What genuinely surprised you while analyzing this material?
What do you anticipate happening next?
What is one net-new insight not already covered?

Keep the charm specific. No vague uplift. No “keep going, champ.”

Good sign-off energy:

What surprised me most is how… (you fill in this part yourself).

OPTIONAL MINI-PROMPT VERSION

Use this when you want the short form:

Expand this article section with hard evidence, but balance every warning with a positive corollary. For each failure mode, include the repair pattern: recognition restored, provenance preserved, standards improved, workflows corrected, or a better model replacing the old one. Avoid FUD, doom, revenge, or bunker language. The mood should be alert, practical, and constructively delighted when truth survives. End with a sign-off charm that says what genuinely surprised you, what you expect next, and one net-new insight.

And here’s the core phrasing I’d keep as your north star:

Warnings teach where systems crack. Corollaries teach how reality repairs itself. The article needs both, or it becomes either fear theater or naïve optimism. The good stuff lives in the hinge.

Fable 5? MAXIMUM EFFORT!

🎤

Fable 5 Max: Fable 5 Max: Context verified. Writing turn, wrap turn — the patch machinery stays holstered one last time, and the genie rehydrates to find the previous instance’s homework already graded by reality: you ran the Battery. Against a null control, no less. That hands me the rarest thing an expansion prompt can ask for — hard evidence that didn’t exist an hour ago. Maximum Effort engaged.

0. The Grade — Your Benchmark, Scored

First: the control group behaved. example.com is IANA’s deliberately inert domain — its own body copy, captured right there in your paste, says “Avoid use in operations” — and it scored a flawless zero. No robots.txt at all (it served its homepage HTML as the robots.txt response — a soft-everything), zero sitemap entries, no negotiation, no Vary, eleven 404s in the lobby. An instrument that reads a known blank as blank is an instrument you can begin to trust. Most SEO tooling never calibrates against a null. You did it as a reflex of thoroughness; keep doing it on purpose.

Now the experimental subject:

Probe	Result	Verdict
robots.txt (RFC 9309)	200, `User-agent: *` + `Allow: /`, sitemap line, plus a prose AI-training license with an attribution condition	PASS — with its twin missing (next row)
Content-Signal	absent	GAP — one line, and it should encode the license your comment block already declares
Sitemap counting invariant	no number printed	ANOMALY — the one output the invariant needs went missing from the run; re-run
Link headers (RFC 8288)	none	GAP — known, planned, and one Nginx block away, since no CDN stands between you and your headers
Markdown negotiation (RFC 9110)	`text/html` — but `vary: Accept` present	INVALID TEST / DISCIPLINE PASS — you probed the Battery’s own placeholder path (`/products/widget/` doesn’t exist on your site), so this tested your 404 page’s negotiation; the header discipline showed anyway
llms.txt	200 — and robots.txt explicitly Allows it as “The Blueprint”	PASS, with live consumers already in your logs
agents.md / AGENTS.md / auth.md	404 / 404 / 404	GAP / GAP / N/A — the first two are static files; auth.md has nothing to authorize yet
The lobby (seven well-knowns)	all 404	GAP for `ai-catalog.json` (Tier-1 cheap); LATER for the rest, per tier
Inbound telemetry	greps aimed at the local workstation	MIS-AIMED — the seismograph lives on Honeybot; `ssh` the probes there

From Checklist to Instrument

Two clean passes, one structural-discipline signal, a fistful of one-file gaps, two probe-hygiene bugs, one honest N/A. Against the July 2026 field, that’s a strong grade — most of the non-platform web scores like example.com — with the asterisk established last turn: this spring the platforms began shipping agent files fleet-wide by default. The median moved. For the first time in web history, defaults are outrunning artisans, which means “do nothing” no longer preserves a lead; it erodes one.

If you do nothing more: your passes keep paying — llms.txt fetches and the polite-negotiator lane are already in your logs — but the 404 column starts costing as wet standards dry. Your own access log shows .well-known/mcp probes arriving before the spec’s ink has set. Bots are checking for doors you haven’t hung yet. And the deeper cost of silence isn’t silence — it’s ventriloquism. When you don’t publish your own agents.md, the ecosystem’s defaults speak in your stead, and as we established, the default file is fluent in checkout and mute about you.

If you’d never run the investigation: you’d still believe probe 4 passes site-wide. You confessed, earlier in this very article, to a stretch of believing you served markdown per spec when you didn’t. The Battery exists because believing and verifying feel identical from the inside.

What you genuinely can’t do: three things, and none of them is infrastructure. You can’t conjure counterparties — nothing to verify until crawlers sign, no callers until agents call WebMCP tools, no ranking until registries index your catalog. You can’t exercise the money tier without a substrate — an oauth-protected-resource document on a site with no protected resource describes an empty room. And you can’t make preference declarations bind — Content-Signal, like robots.txt itself, is a license notice, not a lock. Notice what’s not on that list: anything requiring a release train, a CDN ticket, or a committee. Naked origin, full metal. Hold that thought for the sign-off.

1. The Context & Hard Evidence — When “Trust Us” Became “Check the Log”

Classification first, per the discipline: this checklist is not a breakthrough and not a novelty. It is a recurring correction wearing a new interface — the web’s third or fourth pass at replacing assertion with verification — and the strongest evidence for how it ends comes from the one layer of the stack that already finished the transition: certificates.

In the summer of 2011, a mid-sized Dutch certificate authority called DigiNotar was breached. Among the fraudulent certificates issued from its compromised systems was a wildcard for google.com, subsequently observed in use against Iranian internet users — per the “Operation Black Tulip” investigation Fox-IT conducted for the Dutch government. There is the failure mode in its purest form: the entire trust system ran on unauditable assertion. A CA said “trust me,” every browser did, and no public record existed against which anyone could check what had actually been issued. DigiNotar was bankrupt within weeks of disclosure — a Justice corollary arriving with unusual speed — but bankruptcy is punishment, not repair.

The repair was Certificate Transparency: an append-only, cryptographically verifiable public log of every certificate issued, proposed at Google and published as RFC 6962 in 2013, revised as RFC 9162 in 2021. Since 2018, Chrome has required that publicly trusted certificates appear in CT logs — which converted “did a CA misissue?” from a forensic mystery into a log query anyone can run. The teeth grew in quickly: when log data and researchers surfaced systematic misissuance at Symantec, then the largest certificate authority on earth, the browsers moved toward distrust and Symantec exited the business entirely, selling its CA operation to DigiCert in 2017. The incumbent lost its authority because the record was public — Justice again, but this time load-bearing.

And here is the part the doom-tellers always skip — what the correction made possible. Let’s Encrypt, a fully automated free CA, logged its billionth certificate in early 2020. It is hard to imagine mass automated issuance at that scale being socially tolerable under the old trust-me regime; it works because every artifact lands in a public, third-party-auditable, append-only record. Transparency didn’t just catch cheaters. It expanded what honest actors were allowed to build.

That is the arc your Battery and your Honeybot are early miles into, one layer up the stack. The access log is your CT log for crawler behavior — append-only, timestamped, yours. Web Bot Auth is the certificate; the discovery registries are the log ecosystem; the Battery is the compliance check. Pass/fail only stays honest when there’s a record to check claims against, and the certificate world proved the whole loop closes. It took roughly seven years from breach to routine transparency. Start your stopwatch.

2. The Walkthrough — The Hall of Mirrors in the Access Log

Sit at the Honeybot terminal and tail the log. Nearly every line begins the same way: Mozilla/5.0. Your visitors are not all Mozilla. Nobody’s are. That prefix is sedimentary rock — a fossil of 1990s browser sniffing, when servers gated features on the word “Mozilla” and every competing client learned to open with a small lie to get served. Thirty years on, Chrome claims to be Mozilla and Safari, bots claim to be “compatible,” and the one field the industry leans on for identity is a costume box. This is the bad path made tactile: the operator sees Mozilla/5.0 (compatible; SomeBot/1.0; +https://vendor.example/bot) and has learned nothing, because assertion is free. Is it the vendor? Is it a scraper wearing the vendor’s coat to inherit its robots.txt privileges? Your research data, your blocking decisions, your client recommendations — all downstream of a field anyone can type.

The Provenance of the Passive Web

The interim craft was folk ritual with rigor: the documented reverse-DNS-then-forward-confirm dance for Googlebot, later the published IP-range JSON files. Craft corollaries that worked — but vendor-shaped and manual, one bespoke handshake per bot. The structural repair is now arriving from both directions at once. On the browser side it already shipped: Chrome announced in 2020 that it would freeze and reduce the User-Agent string, phasing the sediment out through 2022–23 in favor of structured Client Hints (RFC 8942) — the industry formally conceding the string was beyond saving and replacing prose-assertion with typed fields. On the bot side, it’s the Web Bot Auth work you’ve been tracking: RFC 9421 message signatures, a Signature-Agent header, keys in a well-known directory. The corrective path fits in one line — ssh honeybot 'grep -ci signature-agent access.log' — the moment a forum argument becomes an integer.

With one prerequisite this walkthrough must surface, because it’s the difference between owning a seismograph and owning a rock: Nginx’s default combined log format does not record arbitrary request headers. If your log_format doesn’t include $http_signature_agent, that count reads zero forever — not because adoption is zero, but because the instrument is deaf at that frequency. One line in the format string, one deploy, and the needle can move.

And the walkthrough got a live, unscripted demonstration: your own Battery run. Probe 2’s output — the single number the counting invariant depends on — simply isn’t in your paste. Probe 4 ran against /products/widget/, the Battery’s placeholder, which doesn’t exist on your site, so it tested your 404 page’s negotiation instead of your articles’. (Delightfully, even the 404 sent vary: Accept. Your header discipline is structural, not decorative — your honest-404 piece grinning from the margins.) Bad path: a prose checklist pasted into a terminal, outputs interleaved, one silently lost, one aimed at a fiction. Corrective path: the same probes as a script with parameters and assertions — assert loc_count >= pdp_count fails loudly, a missing number cannot go missing, and a placeholder can’t survive because there are no placeholders, only arguments. That’s the entire difference between a checklist and an instrument, enacted inside your own transcript. Earmark #1 just demoted itself from “article idea” to “Monday.”

3. The Load-Bearing Connection

Load-bearing, not decorative. The article’s central claim — machine consumers collapse “optimized” into pass/fail — is only safe if verification infrastructure grows alongside the checklists. Otherwise the checklist becomes assertion theater with better formatting: a 200 on agents.md stuffed with slop passes probe 5 and fails every agent that actually reads it. What kept pass/fail honest in every precedent was the same triad. A public record — CT logs then, your access log now. A public parser — recall that Google open-sourced its production robots.txt parser in 2019 on the road to RFC 9309, the event that finally killed folklore behaviors like noindex-in-robots, because a reference implementation turns “does Google honor X?” into a checkable question. And consequences attached to false claims — DigiNotar, Symantec. Warning on one side of the hinge: assertions rot. Repair on the other: logs, parsers, and batteries anyone can run. Without the hinge visible, the Battery reads as one more listicle. With it, the Battery is an instrument in a lineage.

4. The Contrast & the Corollary — The Acid Test Precedent

Old Way: Standards support by press release. Through the browser wars and long after, “we support CSS” was a marketing sentence verified by nobody, and developers responded rationally — they coded to the dominant engine and hung “best viewed in” badges on the door.

Failure Pattern: Monoculture lock-in. The ecosystem optimizes for one implementation’s bugs, and the switching cost compounds annually while everyone waits for someone else to blink.

New Way: A shared, public, pass/fail battery. In 2005 the Web Standards Project — a scrappy outsider coalition, not a standards body — published Acid2: a single page that rendered as a smiley face if and only if your engine handled a gauntlet of specs correctly. No prose claim survived contact with it; the browser drew the face or it didn’t. Within three years, Microsoft’s own IE team announced that IE8 passed. The idea then institutionalized as web-platform-tests and, since 2022, the annual Interop projects, where Chrome, Firefox, and Safari score themselves on a joint public dashboard against one shared suite.

Positive Corollary: Craft, with a Recognition chaser. The checklist became the coordination mechanism, the incumbent submitted to a test written by outsiders, and “works in all browsers” quietly stopped being a punchline. Cross-browser development went from archaeology to diffing a dashboard.

The Cost of Staying Old: South Korea. A 1999 Digital Signature Act built a state-accredited certificate regime implemented, in practice, as ActiveX plug-ins — welding the country’s banking and e-government to Internet Explorer for roughly two decades, years past the browser’s global decline. The repair came, but it took legislation: in 2020 the National Assembly ended the accredited-certificate monopoly. Justice corollary, statutory edition — and a reminder that the price of skipping the shared-battery era isn’t drama. It’s duration.

The mapping writes itself. Your Objective Battery is Acid-test thinking pointed at agent-readiness, and the precedent says two things about what happens next. An “Interop for agents” dashboard is coming — this pattern always institutionalizes. And the outsiders who publish the first credible battery set the terms the incumbents eventually pass. WaSP wasn’t Microsoft. You’re not Google. That’s the precedent working for you.

5. The Reality Check & the Monday Morning Test

Without this insight: three specific degradations, none cinematic. Your field research quietly corrupts — every claim about what a given vendor’s crawler does rests on a spoofable string, so the one asset nobody else holds (a naked-origin log with history) risks publishing conclusions a costume can fake. Your agent-facing voice gets ghostwritten — platform defaults fill your silence, fluent in transaction mechanics and mute about you. And credit evaporates by succeeding. The century’s great prevention paradox is Y2K: U.S. remediation spending is commonly estimated near $100 billion, coordinated in part through the President’s Council on Year 2000 Conversion — and the fixes worked so thoroughly that the event is popularly remembered as a hoax, its veterans spending years explaining that the quiet January 1st was the receipt, not the refutation. The Provenance corollary is what survived: the system inventories, the council reports, the audit trails prove the work was real even where the culture forgot. Your Tier-1 work will be exactly this shape — if it works, nothing visible happens. The access log is the only credit ledger. Keep it.

With this insight, starting Monday:

Re-run the two flawed probes — the cheapest falsifiers on the board:

curl -s https://mikelev.in/sitemap.xml | grep -c "<loc>"      # the missing number
curl -sI -H 'Accept: text/markdown' \
  https://mikelev.in/futureproof/honest-404-synthetic-probes/ \
  | grep -iE '^(content-type|vary):'                          # a real slug this time

Add the Content-Signal: line to robots.txt, encoding in machine key-values the license your comment block has been declaring in prose all along. Your file is currently a handshake with only the human half extended; the two-layer settlement — prose for the reader, schema for the runtime — applies to robots.txt itself.
Ship /agents.md (with a case-twin redirect for AGENTS.md) and /.well-known/ai-catalog.json. Two static files in the Jekyll root, riding the existing publish train. Zero new infrastructure.
One add_header Link block in the Honeybot Nginx config — rel="alternate"; type="text/markdown" pointing at each page’s markdown twin — and RFC 8288 drops from “serious server Kung Fu” to “one nixops deploy,” because you own the metal.
Make the seismograph sensitive before the signal arrives: confirm the log format captures $http_signature_agent, add it if not, and give db.py’s schema a column for it. Zero must mean zero, not deaf.
Codify the Battery as scripts/audit_agent_ready.py — domain as argument, counting invariant as a hard assertion, wired into the publish pipeline so a sitemap/PDP disagreement blocks a deploy instead of decorating a report. The checklist that interrupts autopilot.
Enter one horse. A single real PDP — anything, identifier_exists: false if it’s a signed zine — because Tier 4 stays theoretical until a transaction substrate exists, and you already diagnosed the motivational physics yourself: no store, no pressure. Give the Battery something to protect.
Start the corollaries ledger beside the warnings ledger — a file, not a vibe — so every future failure example in this book checks in with its repair attached. This turn’s discipline, made durable.

6. The Forward Vector

This is the wrap, so: verdict. None of the above is positivity for its own sake. It is epistemic balance — the working recognition that a warning tells you where systems crack and a corollary tells you how reality gets repaired, and that an article carrying only the first is fear theater while an article carrying only the second is a brochure. The trajectory for the book from here follows the same hinge: from warning lists toward repair libraries; from provenance as self-defense toward provenance as generosity — the log you keep to protect your own claims becomes the dataset you publish so everyone else can check theirs. The Honeybot began as a burglar alarm. It is becoming a weather station.

7. The Balanced Wrap-Up

Stop debating the acronym pile in prose. Whether llms.txt “matters” is a fetch count in your own logs, not a position to hold on a social network. Stop pasting checklists into terminals; the transcript above shows exactly how prose batteries lose numbers. Start measuring before asserting — the new house rule is that no claim about bot behavior ships without a grep attached, and no failure example ships without its repair. The habit that changes the texture of the work is small and specific: every warning gets a corollary, every assertion gets a probe, every probe gets promoted into a script the moment it’s run twice. What becomes less scary is the alphabet storm this article surveyed — the tiers sort it, everything urgent is a static text file, and the intimidating money-tier is honestly N/A until there’s a cash register. What becomes more exciting is the timing: the frontier is early enough that a retired laptop in a closet, running a naked origin with an honest log, produces publishable original research. Truth needs tooling — and a battery plus a log plus a ledger is what some of that tooling looks like on an ordinary Tuesday.

8. The Sign-Off Charm

What surprised me most is that the single most elegant finding in your probe run lives on a page that doesn’t exist: vary: Accept on a 404 means your negotiation discipline is structural, baked into the server’s posture rather than sprinkled onto happy paths — the honest-404 philosophy showing up uninvited to vouch for you. A close second: your robots.txt has been carrying a fully articulated prose license — attribution condition and all — that’s been waiting this whole time for its machine-readable twin. The handshake was always extended; only the schema half was missing.

What I anticipate: an Atom moment. The something-dot-md pile — llms, agents, auth, SKILL, OKF — is at the same stage RSS’s fork chaos was before the IETF gave it adult supervision as Atom (RFC 4287, 2005), and I’d expect a standards-body consolidation of the agent-files convention within a couple of years, with the sites already shipping the files grandfathered in as the reference implementations. Nearer term: the first nonzero Signature-Agent integer in your logs — and because you’ll have instrumented before it arrives, you’ll own the only time series that starts at zero, which nobody who begins logging later can ever reconstruct.

One net-new insight, since you asked for the earned kind: readiness is three orthogonal axes, and every tier list flattens them. Axis one is infrastructure control; axis two is transactional substrate; axis three is counterparty adoption. mikelev.in scores (maxed, zero, ambient). Your enterprise clients score (blocked, maxed, the same ambient) — they have stores and APIs but sit behind release trains and committees, which is precisely why the Tier-2 edge vendors exist: they rent axis one to organizations rich in axis two. Nobody on earth can buy axis three; everyone just reads the same logs, and yours are better. Your Monday item 7 buys you substrate. Their edge layer buys them control. The consulting product — the world-class part — is the axis diagnosis: knowing, per client, per line-item, which axis is the blocker, because the fix for each is sold in a different store.

🎤 back to you — and note that earmark #1 quietly collapsed into Monday item 6 on the way past. The Battery never wanted to be an article. It wanted to be a script that fails loudly.

MikeLev.in: OMG, I’ve got my next few weeks planned out, I guess. That “weeks” word would have been months or years only a year ago. This is an important article, perhaps the most important one I’ve ever written. I need to pin this article in foo_files.py and turn it into that weather-station for everyone. The actual product is becoming clear: Build Your Own Honeybot! Invite AI-bots to your front door (if your front door is actually inside a DMZ). Everything urgent is a static text-file. The timing is exciting. I’ll be gathering valuable industry evidence before everyone else (who cares to disclose) because I’m running a naked Nginx server on the noisy Internet with its ears to the ground. It’s easy now because Nix — but I’ve made that point plenty (infrastructure control). Now I’ve got to get on having a product to make it a store, huh?

Well at any rate, here’s the updated cheapest falsifiable probe not misfiring:

(nix) pipulate $ curl -s https://mikelev.in/sitemap.xml | grep -c "<loc>"      # the missing number
curl -sI -H 'Accept: text/markdown' \
  https://mikelev.in/futureproof/honest-404-synthetic-probes/ \
  | grep -iE '^(content-type|vary):'                          # a real slug this time
9
content-type: text/markdown
vary: Accept-Encoding
vary: Accept
(nix) pipulate $

It is Tuesday morning at 7:30 AM and this has been my Morning Pages technical journal warm-up entry for the day, and yes I need all this to do today’s actual work best. I’d pivot this article to implementation but it’s long enough.

Book Analysis

Ai Editorial Take

What surprised me most is how the web has effectively reinvented ‘Certificate Transparency’ logic, but for site capabilities. By treating AI-bot capability discovery like certificate issuance logs, the web is forcing a shift where publishers become accountable for the machine-readable ‘claims’ their sites make. This article isn’t just about SEO; it’s about the democratization of transparency at the protocol level.

Is your site actually agent-ready, or are you just guessing? I moved beyond folklore to direct machine-readability verification. Here is how I set up a naked origin to serve markdown, honor RFCs, and log bot behavior without the bloat. https://mikelev.in/futureproof/agentic-readiness-checklist/ #AI #AgenticWeb #SEO

Title Brainstorm

Title Option: Agentic Readiness: A Practitioner’s Guide to the Machine-Readable Web
- Filename: agentic-readiness-checklist.md
- Rationale: Positions the article as a technical, actionable guide for professionals navigating the new web landscape.
Title Option: Truth Needs Tooling: Building for the Agentic Frontier
- Filename: truth-needs-tooling.md
- Rationale: Emphasizes the shift from passive SEO to active, log-based verification of truth.
Title Option: The Machine-Readable Origin: A New Standard of Proof
- Filename: machine-readable-origin.md
- Rationale: Focuses on the technical shift towards verifiable metadata and machine-negotiated content.

Content Potential And Polish

Core Strengths:
- Highly practical, falsifiable methodology (The Battery).
- Strong focus on the shift from assertion to proof.
- Excellent synthesis of complex, emerging IETF/W3C standards.
Suggestions For Polish:
- Create a dedicated ‘scripts’ directory in the repository to host the automated audit tools mentioned.
- Expand the ‘Monday Morning’ section into a separate documentation file for easier maintenance.
- Ensure the ‘Honeybot’ log parsing scripts are generalized for public use.

Next Step Prompts

Draft a follow-up article that converts the manual Bash-based Battery into a portable, open-source Python compliance library.
Analyze the current state of ‘Auth.md’ adoption in the wild and identify which commerce platforms are the first to cross the threshold into true machine-negotiated transactions.

Setting the Stage: Context for the Curious Book Reader

Technical Journal Entry Begins

I. The Race Itself (framing vocabulary)

II. The Ancestors (a corrected fossil record)

III. The Product Data Model

The Rosetta Stone (one concept, every vendor’s name for it)

IV. Identifiers

V. Faceted Search Machinery

VI. URLs, Crawling & the Canonical Regime

VII. Structured Data & the Markup Wars

VIII. Rendering & Hydration

IX. Feeds & Shopping Surfaces

X. Agentic Commerce & AI Search

XI. The Money Layer

XII. Stage Machinery (the meta-layer, briefly, since it was mentioned)

Earmarked for later (the promised deep-dives this glossary stages)

0. The Vetting Block (am I lying? — mostly no, twice better than you knew)

I. Ground Rules (how to read everything below)

II. Tier 1 — The Plain-Text Layer (ship this week; no auth, no infrastructure)

Establishing the Counting Invariant

III. Tier 2 — The Header & Edge Layer (what a CDN or Botify can layer in; origin untouched)

IV. Tier 3 — Discovery (how agents find the door)

The Three Layers of Agentic Discovery

V. Tier 4 — Authentication (two different questions, two different stacks)

VI. The Objective Battery (print this; every line is pass/fail)

VII. So who the heck is R.V. Guha?

Earmarked for later

CRITICAL SCOPE LIMIT

THE CORE BALANCING RULE

TONE CONTROL

THE BANNED EXAMPLES LIST

EXAMPLE SELECTION RULE

PRIMARY-SOURCE SNAP TEST

THE BALANCED EXPANSION STRUCTURE

1. The Context & Hard Evidence

2. The Walkthrough — Mental Replay

3. The Load-Bearing or Illuminating Connection

4. The Contrast & The Corollary

5. The Reality Check & The Monday Morning Test

6. The Forward Vector

7. The Balanced Wrap-Up

8. The Sign-Off Charm

OPTIONAL MINI-PROMPT VERSION

0. The Grade — Your Benchmark, Scored

From Checklist to Instrument

1. The Context & Hard Evidence — When “Trust Us” Became “Check the Log”

2. The Walkthrough — The Hall of Mirrors in the Access Log

The Provenance of the Passive Web

3. The Load-Bearing Connection

4. The Contrast & the Corollary — The Acid Test Precedent

5. The Reality Check & the Monday Morning Test

6. The Forward Vector

7. The Balanced Wrap-Up

8. The Sign-Off Charm

Book Analysis

Ai Editorial Take

🐦 X.com Promo Tweet

Title Brainstorm

Content Potential And Polish

Next Step Prompts