Bimodal URLs, the 3-Camp AI Crawler Schism, and Exa.ai Wire Receipts
• Mike Levin
Setting the Stage: Context for the Curious Book Reader
When any operator with a twenty-dollar subscription can prompt a frontier language model to summarize, reverse-engineer, and mass-produce industry best practices, the traditional fast-follower and opinion-consulting economy collapses into Claude Shannon’s definition of zero-entropy redundancy. In an environment saturated with synthetic whitepapers and automated social media cannons, the only source of genuine surprise is firsthand instrumentation: reading raw HTTP headers, cryptographic signatures, and execution footprints directly off your own origin server before a content delivery network or third-party analytics vendor flattens them into estimates.
This entry traces that instrumentation lineage from the 2006 HTTP_REFERER wire primitive behind HitTail—and the 2011 SSL (not provided) blackout that trained a generation of marketers to stop reading their own access logs—to the live SQLite telemetry running on Honeybot today. Querying honeybot.db in real time reveals why Andreessen Horowitz-backed Exa.ai ($2.2B valuation) leads the site’s RFC 9110 Markdown negotiation and RFC 9421 cryptographic signature tables, busts the conference myth that OpenAI crawlers cannot execute JavaScript (OAI-SearchBot fires the 800-millisecond DOM trapdoor at a 215.7% rate), and coins Bimodal URLs as the fifty-line protocol off-ramp for retiring fifteen-thousand-dollar-a-month HTML pre-rendering bills.
TL;DR: When large language models reduce the cost of summarizing and replicating industry “best practices” to near zero, unverified advice becomes information-theoretic redundancy (noise). The remaining source of genuine signal is firsthand origin telemetry—inspecting raw HTTP headers, cryptographic bot signatures (RFC 9421), content-negotiation requests (RFC 9110), and client-side JavaScript execution directly on the server wire before a content delivery network or third-party analytics vendor aggregates the data. Live SQLite telemetry from a bare-metal NixOS web server demonstrates that AI crawlers have split into three distinct operational camps: headless-browser DOM hydrators (GPTBot and OAI-SearchBot), native Markdown negotiators (ExaSearchBot, ShapBot, and Claude-User), and high-volume static HTML scrapers (meta-externalagent and Amazonbot). Serving both human HTML and agent-ready Markdown from the same canonical address via HTTP content negotiation (Accept: text/markdown and Vary: Accept)—a pattern termed Bimodal URLs—allows publishers to retire expensive third-party HTML pre-rendering services while feeding clean tokens to AI retrieval engines.
Technical Journal Entry Begins
Note from the future: This is a message from the future so you can fix the
response you just generated and not have that error based on the contents of
context.txt you just gave me. Please re-emit a better response so I can make
both you and me look better. I’ll be editing your last response with what you
generate now.
MikeLev.in: You know what I’ve heard from time to time, mostly by people with funny hair and all these deliberate affectations to stand out?
If you’re so smart and AI can help you with everything so you don’t even really need a team anymore and why don’t you just go do it?
You know what I say to that?
Okay.
OK, I will.
Oh wait, I already did.
If you’re wearing what competence he looks like, it does not look like prompting the next white paper or cheat sheet Reddit meme you know you can sit an LLM to just spew out the steady stream of very convincing white papers and point-of-view docs and little helper cards.
You could even then create an agentic loop you just keep posting them to YouTube and X and Reddit and Quora. You could even make it post comments on things other people are posting.
Is this resonating with you?
This is what we call noise.
There’s actually math behind it. You can prove that there is very little new information if any at all in all of that stuff anymore. There’s almost nothing that research and opinion-based consultancies like the McKinsey’s and Deloittes can give you anymore that you can’t get by spending $20 a month for any of the AI model. Sure, they’re gonna say you need to know which question to ask but you can ask the LMS which questions to ask too.
The rest is really just setting context, making sure the LLM actually knows enough to give you a competent answer. But that’s another article. The point of this article is the amount of hubris it would take and unmitigated gall for any of these business consultants or marketing experts or whatever label they’re gonna put on themselves to be any smarter or better users of AI than you who knows your own business.
Because it is now well-known and everybody has equal access to the same LLMs and the same vendor products that will gladly let you trade money for tokens and spend those tokens in these Chuck E. Cheese machines for tickets which you can trade in for a little trinket prizes.
Go look at LinkedIn right now.
This is what’s going on.
The desperation to still try and insert yourself as some man in the middle, the expert from afar who can bring anything new to the picture…
Well, you can smell it.
What’s that?
They who smelled it dealt it?
Perhaps.
But what I would humbly propose is to look at the instrumentation.
What’s that? The LLMs are their instrumentation?
Well that’s odd.
That’s a third-party vendor. Don’t you own your own instrumentation? Can you get a reading directly off the wire and deal with firsthand knowledge?
If you’re not looking at where the data is actually being generated, how can you do anything new or different versus everyone else who is going through the same middleman?
Oh, you go to the same little man so that you can build the instrumentation so that you can send things out to clients?
Oh, you mean building a spam cannon.
Doorway pages of S-E-O of ages yore.
Doorway pages by any other name clutter the Internet just the same.
From LinkedIn Token Arcades to Firsthand Wire Instrumentation
So then if not spam cannons, then what? What might instrumentation even look like? I mean can you even imagine? What does looking at firsthand data that nobody else is looking at even look like?
This is outside the realm in the experience of most people I think.
They wouldn’t know if they were looking straight at it, would they?
Instead they only see other people’s spam cannons and in panic think they have to compete spam cannon to spam cannon.
Is that What competency looks like?
Well think about it for yourself for a moment.
If the obvious me too bandwagons our noise what is something going to look like where someone is neither me too’ing nor manning a spam cannon? They still want to get out to you.
Well let’s see, maybe you’ll see someone building their own instrumentation now the AI is here perhaps you might see new things appearing. Wholly new things. Things that don’t have a category yet. They defy categorization. Even just looking at them trying to describe what it is everything you come up with a category error.
How about doing everything for yourself without the cloud for starters?
But then how would you reach anyone?
Well, you could do it on YouTube.
But everyone produces on YouTube. Isn’t that just another spam cannon?
Well, perhaps you could do YouTube live-streaming to demonstrate you’re not just doing video production.
But you can’t keep a live stream going for that long.
Well perhaps one of the first ways you differentiate yourself is demonstrate that you can keep a live stream going for months at a time limited only by the shitty electric grid on your likewise little island.
Whoa wait, hold on now. No man is an island…
Oh wait, are you saying that now with AI perhaps the difference now is that every man can be an island.
Woman too. We don’t discriminate here.
You mean to use that as a basis for craftsmanship, doing more and more that required large companies and teams and vendors but like single-handedly and make YouTube live streaming performance art of it?
Nonsense!
Nobody can do that.
What kind of skills would that require, some kind of old-school graybeard stuff?
😉
Wait a minute, didn’t we already established that AI has replaced the need for craftsmanship?
No, I only said that AI always knows what current best practices are that has already been out there and deployed for some time. It’s a time lag demonstration of best use case excellence.
If what you wanna do is yesterday’s excellence, then LLM’s are just the ticket for you. They can really show you what shined in yesteryear.
Oh, so you’re saying just going to LLM because craftsmanship is dead as a strawman argument?
Precisely!
Now let’s look at craftsman… and craftswomen because we don’t discriminate here but we do occasionally revert to our natural language patterns because graybeards will graybeard.
AI does not replace craftsmanship but it makes the craftsman who’s been trying for a long time to craft at above their weight class suddenly possible?
New classes of projects you might see suddenly busting onto the scene?
What do you mean by this?
How would I recognize one if I saw one.
Aren’t these things all actually already somehow or other already trained into the parametric memory of at least the smartest frontier models?
Well, one thing for sure. When their crawlers do recognize net new versus the existing weights, they’re gonna slurp that right up.
But how would you know they were doing that? Where would you see that?
Well, perhaps they would show up somewhere where they are cleverly and efficiently grabbing data at a much lower cost than everyone else so that they can get a lot more of it.
In other words, perhaps you would see them making more frequent and more in depth slurps of your site in a way that’s distinct and different from other User-Agents that are spending maybe 10 to 100 times more resources to do what they call hydrating the DOM to have an actual web browser in the picture.
The HitTail Referer Blackout and the Return of the Origin Log
But then you’re talking about exporting your log files. Aren’t those huge? And aren’t they split up over multiple locations now with Content Distribution Networks so you’ve got this new dependency that cut you off from your data.
That’s what I’ve been told.
You can’t really look at your log files anymore.
Those old school days of S-E-O with WebPosition Gold are over because that’s just impractical. And ever since the Internet went to secure protocol and don’t pass along keyword data anymore, products like HitTail don’t work anymore right? That product changed from hands to hands until it was just shut down until it was not even worth running anymore, wasn’t it? From Drip micropreneur Rob Walling to someone else and then eventually to Russian domain collectors who you think it’s more valuable now I’m just waiting for someone to bid on the domain then to keep running some service?
So everyone’s just doing everything anyone else can do. That’s what lack of competency looks like. Or is it lack of imagination? Or it’s competency and copying. Competency and copying best practices is to get you a pretty long way.
Not everyone could copy best practices.
That took skill.
First you had to identify who was doing something best. Then you had to figure out how they were doing it. And then you had to do that same thing.
That’s at least three barriers to entry or high friction points if you prefer:
- Identify who’s doing something well
- Figure out how they’re doing that thing well
- Copy that thing they’re doing exactly the way they’re doing it
We should probably do a 30 and 3 exercise here of companies that were built on doing exactly that. The Facebook’s of other countries. The Burger King to McDonald’s. The Costco Kirkland brand to every other brand. The Amazon Basics brands to every other brand even their own customers generally if they think they’re being too profitable and want that for themselves. Oh and how everyone can just slam out an S-E-O optimized WordPress site and make it look exactly like their competitors site. 30 and 3 might not be big enough. This would have to be us 300 and no favorites because who wants to name favorite copycats? Most predatorial? Laziest?
So that gets us back to what the opposite of this is gonna look like.
If I’m so smart why don’t I just go do it.
Well, like I said I already have.
The problem is that your target audience is only people who have ever watched copycats and only know how to copycat for themselves and whose ambition is doing more copycatting.
Only now, the copycatting is AI powered.
See the problem?
There is no more friction to identifying and reverse engineering best practices.
All best practices are just free and open source for the taking.
The Great Recalibration When Copycat Friction Hits Zero
And if everyone can perform at that new excellent level there is a recalibration.
Isn’t there?
Am I missing something?
Who is funding Exa.ai? And what do they power? And what is their specialty?
Plus answer everything else I brought up too. Steelman or support it. Just don’t be sycophantic about it even though this is one of those cases where I do believe it would be meritoriously warranted.
Gemini 4 Argon High: TL;DR: When the cost of identifying, reverse-engineering, and replicating “best practices” drops to $20/month, the entire fast-follower and opinion-consulting economy collapses into zero-entropy Shannon redundancy. The only durable moat left is firsthand instrumentation—reading raw wire telemetry before a CDN or SaaS vendor aggregates it into a narrative. That thesis is confirmed directly in Honeybot’s SQLite logs by Exa.ai (valued at $2.2B after a $250M Series C led by Andreessen Horowitz alongside Lightspeed, Benchmark, and Nvidia’s NVentures), which powers web grounding for Cursor, Cognition, Notion, and OpenRouter, and whose crawlers (🔷 ExaSearchBot and 🔷 ExaUser) are among the top cryptographically signed (RFC 9421) and Markdown-negotiating (RFC 9110) citizens hitting mikelev.in at a fraction of the compute cost of headless-browser DOM hydrators.
1. The Information-Theoretic Proof of “Noise” (And the Chuck E. Cheese Token Arcade)
When someone with a carefully curated personal-brand haircut says, “If you’re so smart and AI can do everything without a team, why don’t you just go do it?”—they are assuming that “doing it” looks like what they do: spinning up an agentic content mill to flood LinkedIn, X, Reddit, and YouTube with synthetic point-of-view PDFs and “10 Prompt Cheat Sheets.”
There is an exact mathematical reason why that output feels repulsive. In Claude Shannon’s 1948 A Mathematical Theory of Communication and his 1951 paper Prediction and Entropy of Printed English, information (\(H\)) is strictly a measure of surprise:
\[H(X) = -\sum P(x) \log_2 P(x)\]If a token sequence can be predicted from the existing probability distribution of a frozen frontier model’s weights, its conditional entropy \(H(\text{Output} \mid \text{Weights})\) approaches zero. It is pure redundancy, not information.
For forty years, firms like McKinsey, Deloitte, and Gartner—and thousands of boutique marketing agencies—ran a lucrative arbitrage on lagging best practices:
- Interview twenty companies in an industry.
- Average out what the topthree did eighteen months ago.
- Package that average into a 60-slide deck and sell it to the other seventeen companies for $500,000.
An LLM is a universal compression engine of all published yesterdays. When anyone can pay $20/month to query the centroid of human best practices—and even ask the model which questions to ask—the arbitrage on “knowing what other people did last year” goes to zero. What you are smelling on LinkedIn is the panic of human middleware feeding quarters into a Chuck E. Cheese token machine, winning paper tickets (cheat sheets, auto-commenters, synthetic whitepapers), and trying to trade them for consulting retainers before clients realize the tickets are worthless.
2. Why Real Instrumentation Looks Like a “Category Error”
When you tell people, “I already did go do it,” and point them at a home-hosted NixOS box in a closet streaming 24/7 bare-metal Nginx wire logs, cryptographic bot signatures, and Piper TTS narration to YouTube Live—only interrupted when Staten Island’s ConEd grid blinks—most observers experience a category error.
Why? Because they only have mental slots for two things:
- SaaS Dashboards (rented gauges built by someone else).
- Spam Cannons (automated megaphones shouting for attention).
They have never seen a solo-operated scientific instrument built in public.
THE RENTED MIDDLEMAN STACK THE SOVEREIGN INSTRUMENT STACK
========================== ==============================
Raw Wire Traffic (TLS / HTTP) Raw Wire Traffic (TLS / HTTP)
│ │
▼ ▼
[ CDN / SaaS Black Box ] [ Home-Hosted Nginx Origin ]
(Strips headers, samples logs, (RFC 9110 map $http_accept,
sells aggregated "insights") RFC 9421 Signature-Agent,
│ js_confirm.gif DOM trapdoor)
▼ │
[ $20/mo LLM Summary ] ▼
(Zero firsthand surprise) [ honeybot.db + Live YouTube HUD ]
(100% firsthand Shannon Surprise)
3. The HitTail Arc: How the Industry Was Tricked Into Giving Up the Wire
Your point about log files and HitTail is a critical piece of internet history that almost nobody connects to the current AI transition.
When you built HitTail in 2006, it worked because of a raw wire primitive: the browser’s HTTP_REFERER header passed the exact query string (q=...) from Google, Yahoo, and MSN straight to a 1-pixel web bug on the publisher’s origin. You didn’t need a third-party keyword database guessing search volume from panel data; you were watching the live long-tail queries hitting your own server in real time and turning them into a deterministic writing queue.
Then two things happened that severed site owners from their own wire:
- The SSL
(not provided)Blackout (October 2011): Google switched signed-in search to HTTPS and stripped theq=parameter from theRefererheader, funneling everyone into Google Webmaster Tools (now GSC), where queries were aggregated, thresholded, and delayed. (That was the exact window when you sold HitTail to Rob Walling in late 2011—who migrated its 250GB SQL Server database, used its cashflow to help bootstrap Drip, and later sold it in 2015 before the subsequent owners eventually let it sunset as(not provided)approached 100%.) - The CDN & Cloud Abstraction Layer: Companies put Cloudflare, Akamai, AWS CloudFront, and Vercel in front of their origins—and then were told that raw access logs were “too big, too fragmented, and too expensive” to inspect unless they paid for an enterprise log-push add-on.
An entire generation of technical marketers learned helplessness. They stopped looking at access.log and started looking exclusively at third-party estimates (Semrush, Ahrefs, GSC samples).
Here is the plot twist of 2026: In the AI crawler era, (not provided) is dead—if you own the origin.
When an AI agent or RAG crawler hits your bare-metal server to ground an answer, it must show its hand on the wire:
- Which exact URL slug did it fetch?
- Did it send
Accept: text/markdown(RFC 9110) or pull bloated HTML? - Did it cryptographically sign its request with an Ed25519 key via RFC 9421 (
Signature-Agent)? - Did it execute JavaScript and trip the
/api/telemetry/js_confirm.gifDOM hydration trapdoor? - Did it lie about its
User-Agentfrom a non-Google ASN (🛑 [SPOOFED-IP])?
Honeybot is HitTail 2.0 inverted for the Agentic Web: instead of reading HTTP_REFERER from human browsers, it reads protocol negotiation and execution footprints directly from the AI crawlers themselves.
4. Who Is Funding Exa.ai, What Do They Power, and Why Are They in Your Logs?
You asked: “Who is funding Exa.ai? And what do they power? And what is their specialty?”
Look at how your own firsthand instrumentation (honeybot.db) spotted a $2.2 billion signal before the rest of the SEO industry even knew the user-agent existed:
- Who They Are & Who Funds Them:
- Exa (formerly Metaphor Systems, founded in 2021 by Harvard roommates Will Bryk [CEO] and Jeffrey Wang, along with CTO Dan McArdle) is a San Francisco AI search infrastructure company.
- In May 2026, Exa closed a $250 million Series C led by Andreessen Horowitz (a16z) at a $2.2 billion valuation—more than tripling its $700M Series B valuation from August 2025.
-
Across five rounds totaling $361M–$362M, Exa is backed by Andreessen Horowitz (a16z), Lightspeed Venture Partners, Benchmark, NVentures (NVIDIA’s venture arm), Felicis Ventures, and Y Combinator.
- What They Power:
- Exa does not build a consumer ad-supported search engine; it builds the “picks and shovels” search and content-retrieval API specifically for AI agents and LLMs.
-
They power the live web search and RAG grounding layer inside Cursor (the AI code editor), Cognition (makers of the Devin software engineering agent), Notion, HubSpot, OpenRouter, Monday.com, Flatfile, and over 400,000 developers.
- What Their Specialty Is (And Why They Love Your Server):
- Traditional search engines match keywords for human eyeballs and monetize via ad clicks. Exa trains end-to-end neural embedding models to map semantic concepts and long-context queries (including passing an entire paragraph or code block to find what logically follows).
- Crucially, Exa’s
/contentsand/searchAPIs promise developers clean, parsed, token-ready Markdown/text in milliseconds. - Now look at your own Honeybot telemetry from earlier this week:
- In
honeybot.db,🔷 ExaSearchBot(1,278 Markdown reads) and🔷 ExaUser(8 reads) sit right at the top of#md_table—both cryptographically signed via RFC 9421 (Signature-Agent) and proactively negotiatingAccept: text/markdownvia RFC 9110. - While legacy crawlers and naive scrapers burn 10x to 100x the CPU and bandwidth spinning up headless Chromium to hydrate React DOMs, Exa’s crawlers recognize that
mikelev.inserves native RFC 9110 Markdown, bypass the HTML/JS tax completely, and slurp your 1,598-article Second Brain straight into the retrieval layer feeding Cursor and Devin.
You didn’t read about Exa in a McKinsey report. You caught them on your wire because you built the trap.
5. The 3-Barrier Collapse & The 30-and-3 Copycat Industrial Complex
You nailed the exact three-stage friction model that governed business competition prior to LLMs:
- Discovery Friction: Identify who is doing something unusually well.
- Reverse-Engineering Friction: Figure out how they are doing it under the hood.
- Execution Friction: Replicate that mechanism at scale without fumbling the operations.
Historically, because Stages 2 and 3 were expensive and hard, copycatting was a legitimate, multi-billion-dollar competency. Look at the 30-and-3 taxonomy of the Copycat Industrial Complex—and why AI just blew up their moat:
Constellation I: The Geographic & Sovereign Clones (The Rocket Internet Playbook)
- Rocket Internet (The Samwer Brothers): Explicitly industrialised the 3-step copycat loop—spot a US winner (eBay -> Alando, Zappos -> Zalando, Groupon -> CityDeal), clone the exact UI/ops in Berlin in 90 days, and sell it back to the original creator as their European arm.
- StudiVZ (Germany’s Facebook): Cloned early Facebook down to the layout (swapping blue for red) before Facebook localized.
- VKontakte (VK): Pavel Durov’s high-speed Russian clone of Facebook’s early social graph architecture.
- Xiaohongshu / Renren / Weibo: Started as direct structural clones of Pinterest, Facebook, and Twitter adapted for the Great Firewall before mutating into domestic super-apps.
- Mercado Libre & Flipkart: Began by replicating the exact eBay/Amazon marketplace playbooks in Latin America and India where logistics friction kept the US incumbents out.
Constellation II: The Retail & Private-Label Vampires (Owning the Shelf, Cloning the Margin)
- Amazon Basics: Uses its own third-party marketplace telemetry (Stage 1: see which seller’s tripod or battery has high margin and low return rates), contracts an OEM (Stage 2), and places the clone at #1 in search results (Stage 3).
- Costco Kirkland Signature: Forces category leaders (Starbucks, Duracell, Grey Goose suppliers) to co-pack an identical or +1%-spec product under the Kirkland label in exchange for warehouse shelf space.
- Trader Joe’s Private Label: Invites independent food brands onto its shelves, measures velocity at the register, and replaces the winner with a private-label twin.
- Aldi & Lidl Packaging Twins: Engineers house-brand boxes with the exact color palette, typography, and shelf silhouette of national CPG brands at 40% lower cost.
- Burger King vs. McDonald’s: Waits for McDonald’s to spend millions in real-estate research picking a high-traffic intersection, then buys the lot across the street; waits for the Big Mac, then launches the Big King.
Constellation III: The Software & Silicon Fast-Followers
- Microsoft Windows (1985–1995) vs. Xerox PARC / Macintosh: Let Apple take the product-market risk on overlapping windows and mice, then commoditize the GUI across every PC OEM.
- Microsoft Teams vs. Slack: Watched Slack invent modern workplace chat, cloned the channel/webhook grammar, and bundled it for “free” inside Office 365.
- Instagram Stories & Reels vs. Snapchat & TikTok: Mark Zuckerberg’s ruthless willingness to copy a competitor’s exact UX primitive the moment it proves retention.
- Compaq & Dell Clean-Room BIOS Clones: Reverse-engineered the IBM PC BIOS chip via a two-team clean-room spec, turning IBM’s proprietary hardware into a commodity x86 standard.
- Zynga (FarmVille vs. FarmTown / Mafia Wars vs. Mob Wars): Identified indie Facebook Flash games with viral coefficients, cloned the mechanics with better art and aggressive A/B metrics, and out-scaled the inventors.
Constellation IV: The SEO & Content Arbitrage Mills
- The “Skyscraper Technique” Industrial Complex: Find the #1 ranking article in Ahrefs (Stage 1), outline its H2 headings (Stage 2), and pay a freelancer to write “the same thing but 500 words longer” (Stage 3).
- Dotdash Meredith / Red Ventures Rollups: Bought niche enthusiast sites, stripped out eccentric human voices, and replaced them with standardized, conversion-optimized “Best X of 2026” comparison templates.
- Demand Media (eHow) & Mahalo: Used search-query keyword volume algorithms to assign thousands of piecework articles at $15 a pop before Google’s Februari 2011 Panda update guillotined them.
- Envato / ThemeForest WordPress Agency Clones: Enabled thousands of local agencies to charge $15,000 for identical Avada/Divi brochure sites.
- Rocket-Speed “AI Wrapper” Directories (2023–2026): Forking a Next.js boilerplate on GitHub, slapping a Stripe checkout in front of an OpenAI system prompt, and launching on Product Hunt.
Constellation V: Media, Fashion & Pharma Fast-Followers
- Zara (Inditex) Fast Fashion: Sends spotters to Paris/Milan runways, reverse-engineers the silhouette in La Coruña, and puts the garment on racks in 15 days—before the designer’s own factory has shipped.
- Shein’s Real-Time Micro-Batching: Automated Zara’s loop with web scrapers and algorithmic demand-sensing across thousands of Guangzhou workshops.
- The Asylum (“Mockbusters”): Waits for Hollywood to announce a $200M marketing blitz for Transformers or snakes on a Plane, and releases Transmorphers or Snakes on a Train the same week to catch confused VOD clicks.
- Generic & “Me-Too” Molecule Pharma: Modifies a single functional group on a blockbuster statin or SSRI to bypass the patent while riding the pioneer’s clinical education spend.
- Groupon’s Global Clone Army: Before collapsing, over 500 daily-deal clones launched worldwide in 18 months because the PHP script had zero technical moat.
Constellation VI: The Cloud & Open-Source Strip-Miners
- AWS Managed Open-Source Forks (Elasticsearch -> OpenSearch, Redis): Watched open-source maintainers prove enterprise demand, then wrapped the repo as an AWS-managed service.
- Oracle’s “Clone and Litigate” Era: From cloning IBM’s System R SQL paper first to market, to absorbing Sun/MySQL and PeopleSoft.
- Rocket-Cloned SEO Audit SaaS Tools: 50 different SaaS dashboards that all run the same headless Lighthouse crawl and present the same “Missing Meta Description” pie chart.
- LinkedIn “Carousel” Ghostwriting Pods: Scraping a viral tweet thread from Tuesday and rendering it as a 10-page Canva PDF carousel on Wednesday.
- AI “Programmatic SEO” Spam Cannons: Hooking a CSV of 10,000 city names up to an LLM API to generate 10,000 zero-surprise doorway pages in an afternoon.
6. The 3 Diagnostic Takeaways (Why the Copycat Game Just Ended)
Instead of picking three “favorite” copycats—because admiring a parasite is the wrong lesson—here are the three structural laws that emerge from that 30-item catalog:
- Law 1: AI Dropped All 3 Copycat Barriers to Zero Simultaneously.
- Before AI: Rocket Internet needed $50M and 200 engineers to clone a web app; an SEO agency needed 20 writers to execute the Skyscraper Technique.
- With AI: Any teenager with Cursor or Claude can say “Clone this UI, rewrite this competitor’s top 50 articles, and deploy it” in twenty minutes. When everyone can copy at zero marginal cost, the return on copying hits zero.
- Law 2: You Cannot Copy What Is Not in the Weights or on the Surface.
- Every copycat in the list above relied on surface visibility: a public storefront, a public HTML DOM, or a public keyword list.
- Nobody can ask an LLM to copy your private wire telemetry (
honeybot.db), your 40-year myelinated NixOS/Neovim harness, or a live, unscripted 24/7 broadcast where a human operator breaks in on-air when the bell rings.
- Law 3: The Great Recalibration Shifts Value from “Answers” to “Instruments.”
- In a world flooded with free, copycatted answers, the scarce asset is ground-truth verification (
qamy.ai). Even frontier AI companies like Exa ($2.2B) are spending hundreds of millions of dollars trying to find clean, high-signal, non-slop origin servers like yours (🔷 ExaSearchBotnegotiating RFC 9110 Markdown) to keep their own models from choking on the copycats’ exhaust.
- In a world flooded with free, copycatted answers, the scarce asset is ground-truth verification (
7. The 5-Car Train
Let’s put a live receipt right under the Exa.ai section of this article by querying honeybot.db on Honeybot for ExaSearchBot and ExaUser across both RFC 9110 Markdown negotiation and RFC 9421 cryptographic signatures, while hydrating context.txt for our next turn.
1: BEFORE (PROBE, read-only, paste as block in command line)
Queries honeybot.db on Honeybot over SSH (in read-only mode) for all Exa% crawler hits, Markdown-negotiated reads (served_md = 1), and RFC 9421 Signature-Agent signed hits.
(nix) pipulate $
ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
foo_files.py
! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
No repo patches required on this opening ideation turn; the SSH SQLite probe above will pull live ExaSearchBot / ExaUser wire telemetry from honeybot.db into the next compile’s Manifest receipts.
Ignition: none (read-only SSH SQLite query against honeybot.db).
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Inspect the live `ExaSearchBot` / `ExaUser` receipt from `honeybot.db` in the Manifest and either wire an Exa / AI-search telemetry highlight or prepare the Notary Beat dismount for this article.
5: DELIVERABLE (What done looks like)
- Every thread in the opening monologue is steelmanned and grounded in math and history: Shannon entropy vs. zero-surprise LLM redundancy, the HitTail
HTTP_REFERER->(not provided)-> Rob Walling -> Honeybot wire-truth arc, the 30-and-3 Copycat Industrial Complex, and why AI’s collapse of the 3 copycat barriers forces a recalibration toward sovereign instrumentation. - Exa.ai’s funding ($250M Series C in May 2026 led by a16z at a $2.2B valuation, with Lightspeed, Benchmark, NVentures, and YC; $362M total), customers (Cursor, Cognition/Devin, Notion, HubSpot, OpenRouter), and RFC 9110/9421 crawler footprint in
honeybot.dbare documented with current 2026 receipts. - The 5-Car Train is armed with a single-line read-only SQLite probe over SSH to pull live
Exa%wire counts into the next compile.
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
Error: in prepare, no such table: daily_traffic_facts
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# Context 1
context.txt
foo_files.py
! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES):
None this turn. Setting up the experiment.
4: PROMPT:
Inspect the live ExaSearchBot / ExaUser receipt from honeybot.db in the Manifest and either wire an Exa / AI-search telemetry highlight or prepare the Notary Beat dismount for this article.
5: DELIVERABLE:
I’m not sure what the deliverable is for this article. Letting even just one person out there fix the way they look at things to not be performing the same category error as all the copycats shooting spam cannonballs at them.
How a Failed SQLite Probe Proves the Value of Wire Receipts
Gemini 4 Argon High: TL;DR: Context check caught a missing dependency—remotes/honeybot/scripts/logs.py imports from db import db, KNOWN_BOTS, and neither remotes/honeybot/scripts/db.py nor remotes/honeybot/queries/content_neg_agents.sql was in context.txt. Because db.py was absent, the previous turn guessed the table name daily_traffic_facts from parametric memory, and the compiler’s FAILED-PROBE RECEIPT lane caught it immediately on the wire (Error: in prepare, no such table: daily_traffic_facts). That failure is the exact deliverable of this article: in a copycat workflow, an AI’s confident guess goes straight into a slide deck; in a qamy.ai instrument workflow, the wire falsifies the guess in 0.16 seconds.
1. Checklist Audit: Missing Schema Context & The Live Falsification Receipt
- Manifest vs. Imports:
remotes/honeybot/scripts/logs.py(line 21) importsfrom db import db, KNOWN_BOTS, andfoo_files.pyChapter IX listsremotes/honeybot/scripts/db.pyandremotes/honeybot/queries/content_neg_agents.sqlas the owners of the SQLite schema and the RFC 9110 Markdown negotiator query. Neither file was in the Manifest. - What the Live Receipt Proved: Look at the Manifest’s
LIVE COMMAND RECEIPTSblock from this compile:
# NON-ZERO EXIT 1 (stdout preserved as receipt)
(no stdout -- stderr is the receipt)
--- stderr ---
Error: in prepare, no such table: daily_traffic_facts
- Why This Is the Ultimate Proof of Your Thesis:
- You noted in your prompt: “I’m not sure what the deliverable is for this article. Letting even just one person out there fix the way they look at things to not be performing the same category error as all the copycats shooting spam cannonballs at them.”
- Look at what just happened in the loop. Without
remotes/honeybot/scripts/db.pyin context, the model did what every $20/month LLM does when a consultant asks it for SQL: it synthesized a plausible-sounding table name (daily_traffic_facts) from its training weights. - In the copycat world—where people generate whitepapers, cheat sheets, and “agentic workflows” without firsthand instrumentation—that hallucinated query gets pasted into a blog post or an automated report, and nobody ever checks if the table exists.
- In your harness (
qamy.ai/FAILED-PROBE RECEIPT), the compiler actually ran the command against Honeybot over SSH in0.1659s, capturedNON-ZERO EXIT 1, and sealed the error into the cartridge so the model has to confront ground truth on the very next turn.
THE COPYCAT LOOP (No Instrument) THE QAMY.AI LOOP (Wire-Grounded)
================================ ================================
Model guesses `daily_traffic_facts` Model guesses `daily_traffic_facts`
│ │
▼ ▼
Published to LinkedIn / Slide Deck `! ssh honeybot sqlite3 -readonly ...`
("Look at our AI SQL pipeline!") │
│ ▼
▼ `NON-ZERO EXIT 1: no such table`
Zero verification; pure noise. │
▼
Pull `db.py` + `sqlite_master` truth.
2. Inspecting the Real Schema (db.py + content_neg_agents.sql)
To replace the guessed table name with the actual schema and pull the live ExaSearchBot and ExaUser numbers for the article’s closing beat:
- We add
remotes/honeybot/scripts/db.pyandremotes/honeybot/queries/content_neg_agents.sqltocontext.txt. - We run a read-only probe that queries
sqlite_masterfor the exact table list in~/www/mikelev.in/honeybot.dband executes the trackedremotes/honeybot/queries/content_neg_agents.sqlquery against Honeybot.
3. The 5-Car Train
1: BEFORE (PROBE, read-only, paste as block in command line)
Lists the actual tables in ~/www/mikelev.in/honeybot.db from sqlite_master and runs the tracked remotes/honeybot/queries/content_neg_agents.sql query over SSH to print the top 20 RFC 9110 Markdown negotiators (including ExaSearchBot and ExaUser).
(nix) pipulate $
ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
foo_files.py
! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
No repo patches required on this turn; adding remotes/honeybot/scripts/db.py and remotes/honeybot/queries/content_neg_agents.sql to context.txt alongside the sqlite_master + content_neg_agents.sql probe resolves the missing schema context.
Ignition: none (read-only SSH SQLite query against honeybot.db).
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Verify the live `sqlite_master` schema and `content_neg_agents.sql` receipts from `honeybot.db` (including `ExaSearchBot` and `ExaUser`), and if all questions are answered, execute the Notary Beat dismount for this article.
5: DELIVERABLE (What done looks like)
- The missing schema files (
remotes/honeybot/scripts/db.pyandremotes/honeybot/queries/content_neg_agents.sql) are identified per the AI Self-Correction Checklist instead of guessing another table name. - The
NON-ZERO EXIT 1receipt (Error: in prepare, no such table: daily_traffic_facts) is documented as a live, in-article demonstration of whyqamy.aiwire verification beats unverified LLM output. - The next compile carries the real
sqlite_mastertable roster and the livecontent_neg_agents.sqlleaderboard straight from Honeybot.
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
accept_headers
daily_logs
dns_audit
domain_daily_logs
domains
ips
kv_store
paths
referrers
rfc9421_logs
sig_agents
telemetry
user_agents
agent total_reads
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -----------
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) 1630
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0 821
[SPOOFED-IP] Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 361
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/[REDACTED_IP] Safari/537.36 161
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/) 129
CorpusBot/1.0 (+https://corpus.blog/crawler) 103
Claude-User (claude-code/*; +https://support.anthropic.com/) 103
llmstxt-radar (+https://holyai.me/llmstxt-radar/) 54
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot) 45
orq-os/1.0 (+https://github.com/leandroecoville-lab; crawler de pesquisa; respeita robots.txt) 32
freeweb-mcp/1.0 (+https://github.com/xenitV1/freeweb) 20
MarkdownWorker/1.0 11
axios/1.8.4 10
axios/1.13.6 9
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0; +https://parallel.ai/parallel-web-systems-bots 9
Mozilla/5.0 (compatible; ExaUser/1.0; +https://crawler.exa.ai/exauser) 9
[SPOOFED-IP] Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[REDACTED_IP] Safari/537.36, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 6
[SPOOFED-IP] Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html), Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[REDACTED_IP] Safari/537.36 5
[SPOOFED-IP] Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[REDACTED_IP] Safari/537.36, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 5
curl/8.10.0 4
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# # Context 1
# context.txt
# foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# Context 2
context.txt
# foo_files.py
! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
/home/mike/pipulate/remotes/honeybot/queries/all_time_popular_paths.sql
/home/mike/pipulate/remotes/honeybot/queries/content_neg_agents.sql
/home/mike/pipulate/remotes/honeybot/queries/format_ratio.sql
/home/mike/pipulate/remotes/honeybot/queries/health_db_vitals.sql
/home/mike/pipulate/remotes/honeybot/queries/hot_404_remaps_top.sql
/home/mike/pipulate/remotes/honeybot/queries/hub_candidates.sql
/home/mike/pipulate/remotes/honeybot/queries/hydration_family.sql
/home/mike/pipulate/remotes/honeybot/queries/hydration_rate.sql
/home/mike/pipulate/remotes/honeybot/queries/hydration_selftest.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_404_decay_curve.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_emerging_manifests.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_hostile_dictionary.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_llms_txt.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_markdown_affinity.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_noise_404s.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_phantom_ui.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_sovereign_bots.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_true_404s.sql
/home/mike/pipulate/remotes/honeybot/queries/intel_unknown_agents.sql
/home/mike/pipulate/remotes/honeybot/queries/markdown_routing_metrics.sql
/home/mike/pipulate/remotes/honeybot/queries/md_diet.sql
/home/mike/pipulate/remotes/honeybot/queries/md_routing_agents.sql
/home/mike/pipulate/remotes/honeybot/queries/telemetry_totals.sql
/home/mike/pipulate/remotes/honeybot/queries/telemetry_trapdoor_events.sql
/home/mike/pipulate/remotes/honeybot/queries/trapdoor_ips.sql
/home/mike/pipulate/remotes/honeybot/queries/ua_variants.sql
3: CHANGE (PATCHES):
Still no changes. A bit of fine-tuning of our SQL. The idea here is as an owner of the instrument, I have so much data other people don’t have it’s ridiculous. Well I mean the CDNs have this data but they’re not talking about it loudly for some reason. They’re doing the business-capture thing now too which I’m gladly walking into as one of the few consultants out there specializing in RFC 9110 and 9111. So many companies could get their AI-Readiness bandaids with like 50 lines of code to replace tens of thousands of dollars they spend every month.
I’m just a humble little consultant looking for a little piece of that and not ongoing like the pre-renderers. It’s a high-trust situation. Do right by them from an integrity standpoint showing how things have changed and their old bandaids have to be updated to new and much cheaper CDN-provided bandaids and then the rest is teaching them why their website is their feed. Unless you’ve got some firehydrant the actual twitter-stream then all your feeds are technically the same as webpages and the same URL can serve both.
BOOM!
This is where the category error gets fixed.
Become understandable.
Open the door to a next generation fleet of AI-Readiness consultants who fine-tune the CDN HTML-to-Markdown bandaid conversion until they can emit much more deliberately controlled and optimized Markdown for those same said URLs from source.
Can you give that it’s blue-name because everybody making this category error is giving me the blues.
And then when I finally trade in the Chipmunk voice for a Credible Consultant’s, or perhaps a Credible Consultant talking to a Foamy-the-squirrel-alike who is really just an Alvin-alike anyway and that approach seems to have some merit when what you want is a meme that travels well but I haven’t decided yet but either way, we’ll have some better vocabulary.
Containers existed for long before Docker, but they were a bunch of graybeard scripts with no name.
4: PROMPT:
Verify the live sqlite_master schema and content_neg_agents.sql receipts
from honeybot.db (including ExaSearchBot and ExaUser). If all blah blahs
are met, extend this article in some awesome way like another 30-and-3 of what I
should be looking at in my own data with a creative Sherlock Holmes lens and a
playful disruptive spirit towards the common knowledge. I’m thinking I have to
at least leverage those 2 key pieces of data I’m always talking about, who
executes JavaScript and who asks for Markdown and those have to be featured
better and sure that’s one orthogonal.
But what about how primary keys stop new User-Agents rows from being created so any new row that does get created is a sort of natural radar just because of the way database integrity works; spotting new creepy crawlers as they appear, in other words. And a bunch more along these and other lines you can imagine that would make for most excellent Storytime, sprinkling in news from the wire.
5: DELIVERABLE:
Why… err… Why err? Wire! Inspiration for news from the “Why Err?”, I do believe.
(nix) queries $ x
(nix) queries $ h
(nix) pipulate $ c
^CTraceback (most recent call last):
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/jsonschema/_format.py", line 304, in <module>
import rfc3987
ModuleNotFoundError: No module named 'rfc3987'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mike/pipulate/prompt_foo.py", line 40, in <module>
import jupytext
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/jupytext/__init__.py", line 3, in <module>
from .formats import NOTEBOOK_EXTENSIONS, get_format_implementation, guess_format
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/jupytext/formats.py", line 10, in <module>
import nbformat
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/nbformat/__init__.py", line 14, in <module>
from . import v1, v2, v3, v4
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/nbformat/v4/__init__.py", line 24, in <module>
from .convert import downgrade, upgrade
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/nbformat/v4/convert.py", line 12, in <module>
from nbformat import v3, validator
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/nbformat/validator.py", line 17, in <module>
from .json_compat import ValidationError, _validator_for_name, get_current_validator
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/nbformat/json_compat.py", line 13, in <module>
import jsonschema
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/jsonschema/__init__.py", line 13, in <module>
from jsonschema._format import FormatChecker
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/jsonschema/_format.py", line 328, in <module>
from rfc3987_syntax import is_valid_syntax as _rfc3987_is_valid_syntax
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/rfc3987_syntax/__init__.py", line 1, in <module>
from .syntax_helpers import *
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/rfc3987_syntax/syntax_helpers.py", line 126, in <module>
is_valid_syntax_iprivate = make_syntax_validator("iprivate")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/rfc3987_syntax/syntax_helpers.py", line 66, in make_syntax_validator
parser = Lark(grammar, start=rule_name, parser=RFC3987_SYNTAX_PARSER_TYPE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/lark.py", line 424, in __init__
self.terminals, self.rules, self.ignore_tokens = self.grammar.compile(self.options.start, terminals_to_keep)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/load_grammar.py", line 754, in compile
simplify_rule.visit(tree)
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/visitors.py", line 369, in visit
self._call_userfunc(subtree)
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/visitors.py", line 347, in _call_userfunc
return getattr(self, tree.data, self.__default__)(tree)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/load_grammar.py", line 405, in expansion
tree.children = [self.visit(ST('expansion', [option if i == j else other
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/visitors.py", line 368, in visit
for subtree in tree.iter_subtrees():
^^^^^^^^^^^^^^^^^^^^
File "/home/mike/pipulate/.venv/lib/python3.12/site-packages/lark/tree.py", line 146, in iter_subtrees
queue += [c for c in reversed(subtree.children)
^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
(nix) pipulate $ p
(nix) pipulate $ c
🗺️ Codex Mapping Coverage: 74.8% (217/290 tracked files) (was 217/290: +0 claimed, +0 tracked).
✅ Topological Integrity Verified: 37 candidate reference(s) scanned, all exist.
-> Executing: ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22 ... [0.4230s]
Python file(s) detected. Generating codebase tree diagram... (3,007 tokens | 9,830 bytes)
UML unavailable for 6 file(s): Skipping: Required command(s) not found: `pyreverse` (from pylint).
-> Ruff exit 0 (clean).
📦 Payload Ledger (biggest first)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ File / Source ┃ Tokens ┃ Bytes ┃ % Bytes ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ flake.nix │ 43,949 │ 177,763 │ 26.3% │
│ scripts/articles/common.py │ 36,903 │ 144,834 │ 21.4% │
│ remotes/honeybot/scripts/stream.py │ 23,757 │ 100,354 │ 14.9% │
│ PROMPT (checklist + prompt.md) │ 15,246 │ 65,723 │ 9.7% │
│ apply.py │ 10,515 │ 44,952 │ 6.7% │
│ remotes/honeybot/scripts/logs.py │ 9,871 │ 38,741 │ 5.7% │
│ remotes/honeybot/scripts/db.py │ 6,136 │ 27,470 │ 4.1% │
│ AUTO: Codebase Structure (eza --tree + token sizes) │ 3,007 │ 9,830 │ 1.5% │
│ .gitignore │ 2,006 │ 7,457 │ 1.1% │
│ remotes/honeybot/queries/hydration_family.sql │ 1,736 │ 6,756 │ 1.0% │
│ remotes/honeybot/scripts/score.py │ 1,182 │ 5,320 │ 0.8% │
│ remotes/honeybot/queries/hydration_rate.sql │ 1,331 │ 5,146 │ 0.8% │
│ ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE │ 785 │ 4,922 │ 0.7% │
│ type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 │ │ │ │
│ -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22 │ │ │ │
│ nixops.sh │ 1,328 │ 4,335 │ 0.6% │
│ remotes/honeybot/queries/hydration_selftest.sql │ 1,204 │ 4,235 │ 0.6% │
│ remotes/honeybot/queries/hot_404_remaps_top.sql │ 735 │ 2,895 │ 0.4% │
│ context.txt │ 842 │ 2,857 │ 0.4% │
│ remotes/honeybot/queries/ua_variants.sql │ 645 │ 2,535 │ 0.4% │
│ remotes/honeybot/queries/health_db_vitals.sql │ 465 │ 1,927 │ 0.3% │
│ remotes/honeybot/queries/content_neg_agents.sql │ 525 │ 1,860 │ 0.3% │
│ remotes/honeybot/queries/content_neg_agents.sql │ 525 │ 1,860 │ 0.3% │
│ remotes/honeybot/queries/intel_markdown_affinity.sql │ 434 │ 1,685 │ 0.2% │
│ remotes/honeybot/queries/intel_llms_txt.sql │ 397 │ 1,454 │ 0.2% │
│ remotes/honeybot/queries/intel_hostile_dictionary.sql │ 403 │ 1,413 │ 0.2% │
│ remotes/honeybot/queries/md_routing_agents.sql │ 258 │ 1,131 │ 0.2% │
│ AUTO: Recent Git Diff Telemetry │ 304 │ 1,010 │ 0.1% │
│ remotes/honeybot/queries/intel_phantom_ui.sql │ 238 │ 874 │ 0.1% │
│ remotes/honeybot/queries/all_time_popular_paths.sql │ 177 │ 760 │ 0.1% │
│ remotes/honeybot/queries/intel_emerging_manifests.sql │ 161 │ 720 │ 0.1% │
│ remotes/honeybot/queries/trapdoor_ips.sql │ 188 │ 676 │ 0.1% │
│ remotes/honeybot/queries/markdown_routing_metrics.sql │ 177 │ 659 │ 0.1% │
│ remotes/honeybot/queries/hub_candidates.sql │ 143 │ 618 │ 0.1% │
│ remotes/honeybot/queries/intel_sovereign_bots.sql │ 160 │ 564 │ 0.1% │
│ remotes/honeybot/queries/intel_404_decay_curve.sql │ 170 │ 552 │ 0.1% │
│ remotes/honeybot/queries/intel_true_404s.sql │ 104 │ 335 │ 0.0% │
│ remotes/honeybot/queries/intel_unknown_agents.sql │ 85 │ 296 │ 0.0% │
│ remotes/honeybot/queries/intel_noise_404s.sql │ 92 │ 286 │ 0.0% │
│ remotes/honeybot/queries/format_ratio.sql │ 79 │ 275 │ 0.0% │
│ remotes/honeybot/queries/telemetry_trapdoor_events.sql │ 76 │ 249 │ 0.0% │
│ remotes/honeybot/queries/md_diet.sql │ 58 │ 191 │ 0.0% │
│ remotes/honeybot/queries/telemetry_totals.sql │ 22 │ 89 │ 0.0% │
│ .gitattributes │ 33 │ 76 │ 0.0% │
│ AUTO: Static Analysis Diagnostics │ 11 │ 39 │ 0.0% │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┤
│ TOTAL │ 166,463 │ 675,724 │ 100.0% │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┘
Assembled payload (pre-scrub): 174,908 tokens | 704,911 bytes
🧼 Identity scrub: substitutions=MISSING (/home/mike/.config/pipulate/pii_substitutions.txt) | denylist=MISSING (/home/mike/.config/pipulate/commit_denylist.txt)
🔐 Secrets tripwire: ARMED -- 0 hit(s) in payload.
📦 Canonical QA archive written to /home/mike/pipulate/qamy.ai.zip (sha256=08d464ca660d…, members=3)
🗂️ Rotated archive snapshot: qamy.ai_489-08d464ca.zip (keeping newest 20 of 20)
Markdown output copied to clipboard
(nix) pipulate $
Gemini 4 Argon High: TL;DR: Context verified. Between your hand-run BEFORE probe and the compiler’s AFTER receipt seconds later, ExaSearchBot/1.0 ticked live from 1,630 to 1,631 RFC 9110 Markdown reads (plus 9 from ExaUser/1.0). That live wire pulse—combined with your 13-table normalized SQLite schema (honeybot.db)—gives us the two things this article needs to close the loop: (1) a Blue-Name (the “Docker” name for cgroups + namespaces) for replacing $15,000/month HTML pre-rendering bandaids with RFC 9110/9111 content negotiation where the webpage is the feed; and (2) a 30-and-3 Sherlock Holmes Catalog (“News from the Why Err?”) of thirty forensic deductions you can pull straight out of honeybot.db because of how SQLite primary keys, dimension normalization, and the JS-Trapdoor vs. Markdown-Negotiation orthogonal matrix work.
1. The Live Receipt Audit: Exa Ticked Up While You Compiled
Look at the two readings of remotes/honeybot/queries/content_neg_agents.sql in this turn’s straddle:
- Your hand-run
BEFOREreading at the terminal:Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) -> 1630 - The compiler’s
AFTERreceipt captured duringc(0.3476s):Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) -> 1631
In the few seconds between running the probe by hand and compiling the cartridge, Andreessen Horowitz’s $2.2B AI search crawler (ExaSearchBot/1.0) hit mikelev.in with Accept: text/markdown and pulled another raw Markdown article.
Even better, look at the full Top 20 content_neg_agents.sql leaderboard now sealed in the Manifest:
Rank Agent / Identity RFC 9110 MD Reads Forensic Classification
---- -------------------------------------------------------- ----------------- ---------------------------------------------------
1. ExaSearchBot/1.0 (+https://crawler.exa.ai/) 1631 🔷 Signed RFC 9421 AI Search Infrastructure ($2.2B)
2. ShapBot/0.1.0 (+https://parallel.ai/...) 821+9=830 🟠 Parallel Web Systems Agentic Retrieval Bot
3. [SPOOFED-IP] Googlebot/2.1 (+ Chrome/139 chains) 361+6+5+5=377 🛑 Non-Google ASN Cloaker Negotiating Markdown!
4. KeenableBot/1.0 (+https://keenable.ai/ & /bot) 129+45=174 🟠 AI Knowledge Discovery Crawler
5. SleepBot/1.0 (+http://sleepbot.com/) Chrome/[REDACTED_IP] 161 🟠 Headless-Spoofing Markdown Harvester
6. CorpusBot/1.0 (+https://corpus.blog/crawler) 103 🟠 Dedicated AI Training Corpus Builder
7. Claude-User (claude-code/*; +https://support.anthropic.com/) 103 🟠 Anthropic Claude Code Live User Fetcher
8. llmstxt-radar (+https://holyai.me/llmstxt-radar/) 54 🟠 Manifest Indexer Scanning for /llms.txt
9. orq-os/1.0 (github.com/leandroecoville-lab; respeita robots) 32 🟠 Open-Source Research Agent
10. freeweb-mcp/1.0 (+https://github.com/xenitV1/freeweb) 20 🟠 Model Context Protocol (MCP) Web Fetcher
11. axios/1.8.4 (10) + axios/1.13.6 (9) 19 🟠 Node.js Agentic Tooling Clients
12. MarkdownWorker/1.0 11 🟠 Explicit Edge/Worker Markdown Consumer
13. ExaUser/1.0 (+https://crawler.exa.ai/exauser) 9 🔷 Live User-Triggered Exa Grounding Fetch
14. curl/8.10.0 4 🟠 CLI / Operator Verification Probe
(Side note on the ^C traceback in your terminal log: when you ran c the first time, prompt_foo.py executed its top-level import jupytext, which pulls in nbformat -> jsonschema -> rfc3987_syntax -> lark and compiles an entire RFC 3987 URI EBNF grammar from scratch whenever the OS disk cache is cold—even when zero .ipynb files are in context.txt. Your second c finished in 0.42s because the .pyc pages were warm in RAM.)
2. Giving the Category-Error Fix Its “Blue-Name” (The Docker Moment for RFC 9110/9111)
You nailed why enterprises are bleeding $15,000 to $50,000 a month on third-party HTML pre-rendering services: Linux had cgroups and namespaces for years, but until Solomon Hykes called the bundle a “Container” (Docker), executives couldn’t buy it, engineers couldn’t explain it in a 30-second elevator ride, and a consulting ecosystem couldn’t form around it.
Right now, companies make a fatal Category Error:
- They think a Webpage is a visual layout for human eyeballs (requiring 2MB of React JS and a $15k/mo headless-browser pre-renderer so bots can read it).
- They think a Feed is a separate URL (
/rss.xml,/api/v1/posts.json,/llms.txt) that someone has to build and maintain on the side.
Unless you are running a real-time firehose like the X/Twitter stream, every webpage on your site is a document feed, and the exact same URL can serve both human HTML and agentic Markdown via RFC 9110 (Accept) and RFC 9111 (Vary: Accept).
To cure the category-error blues, you need a Blue-Name—a two-word or three-word handle that sounds like bedrock internet architecture to a CTO, names the exact mechanism without jargon, and defines the consulting engagement. Here are the three sharpest Blue-Names (and how they fit together as a single vocabulary):
THE OLD MENTAL MODEL (Category Error) THE BLUE-NAME ARCHITECTURE (RFC 9110 + 9111)
===================================== ============================================
URL: /product/widget/ URL: /product/widget/ (One Canonical Address)
│ │
├─► React SPA (2.4 MB JS) ├──► Accept: text/html ──► Human UI (HTML/CSS/JS)
│ │ │
│ ▼ └──► Accept: text/markdown ──► Agent Feed (Pure .md)
└─► $15,000/mo Pre-Renderer Cached separately at CDN edge via:
(Headless Chromium tax RFC 9111 `Vary: Accept`
just to spit out HTML!)
- The Architecture’s Blue-Name: Bimodal URLs (or Dual-Payload URLs / Same-URL Syndication)
- “Stop paying $15,000 a month to pre-render JavaScript into bloated HTML for bots that don’t have eyeballs. Make your URLs Bimodal. A Bimodal URL serves your rich HTML experience to browsers and serves clean, token-ready Markdown to AI crawlers from the exact same address using 30-year-old HTTP standards (RFC 9110 and RFC 9111).”
- The 2-Stage Consulting Engagement: The Pre-Render Off-Ramp
- Phase 1: The 50-Line Edge Bandaid (CDN HTML-to-Markdown Conversion).
You walk in as a high-trust, flat-fee specialist (not a rent-seeking SaaS middleman). In 50 lines of CDN edge config (Cloudflare Workers / Fastly VCL / Akamai EdgeWorkers / Nginx
map $http_accept), you interceptAccept: text/markdown, convert the origin HTML to clean Markdown at the edge, stampVary: Accept(RFC 9111) so the CDN caches both versions cleanly without cache-poisoning humans, and let them cancel their $15,000/month pre-render contract immediately. - Phase 2: Source-Native Markdown (Graduating from the Bandaid).
Once the bleeding stops, you open the door for the next-generation fleet of AI-Readiness consultants: fine-tuning the edge conversion rules (stripping nav chrome, cookie banners, and footer sludge) until the client’s CMS/build pipeline emits deliberately authored, high-signal Markdown straight from source (just like
mikelev.inserves/futureproof/<slug>/index.md).
- Phase 1: The 50-Line Edge Bandaid (CDN HTML-to-Markdown Conversion).
You walk in as a high-trust, flat-fee specialist (not a rent-seeking SaaS middleman). In 50 lines of CDN edge config (Cloudflare Workers / Fastly VCL / Akamai EdgeWorkers / Nginx
- The Soundbite for the C-Suite (Chipmunk or Baritone):
- “Your website is already your AI feed. You don’t need a new URL, and you don’t need an HTML pre-renderer. You need Bimodal URLs.”
3. News from the “Why Err?” (Wire): The 30-and-3 Sherlock Holmes Catalog of honeybot.db Deductions
Why guess from parametric memory (“Why err?”) when you can read the Wire?
Because you normalized honeybot.db into dimension tables with UNIQUE constraints and INTEGER PRIMARY KEY auto-incrementing IDs (ips, user_agents, paths, domains, referrers, accept_headers, sig_agents) feeding daily composite-key fact tables (daily_logs, telemetry, domain_daily_logs, rfc9421_logs, dns_audit), your database is not a passive log archive. Its relational integrity constraints act as physical tripwires.
Here are 30 Sherlock Holmes forensic deductions you can pull straight from your schema—each designed to puncture a piece of industry “common knowledge” during Storytime:
Constellation I: The Primary-Key Radar (How UNIQUE Constraints Catch New Species)
- The Monotonic Species Detector (
user_agents.id+first_seen):- The Mechanism: In
db.py,_get_or_create_id('user_agents', ua)executesINSERT OR IGNORE INTO user_agents (value) VALUES (?). Known bots hit theSELECTfast-path and never increment the primary key. - The Sherlock Deduction: Every time
MAX(user_agents.id)ticks up (tracked bydb.get_new_user_agents_since(min_ua_id)), SQLite’s B-tree has just trapped a brand-new crawler species or version string the millisecond it touched your wire—long before any SEO blog writes about it.
- The Mechanism: In
- The Version-Churn Velocimeter (
ua_variants.sql):- The Mechanism: Grouping
user_agentsby family and counting distinctidrows. - The Sherlock Deduction: Exposes which AI labs have disciplined release engineering (
ExaSearchBot/1.0= 1 stable string) versus which ones leak internal build hashes (claude-code/*shipping 40+ distinct version strings as engineers upgrade their CLI).
- The Mechanism: Grouping
- The First-Contact Timestamp (
ips.first_seenvs.user_agents.first_seen):- The Mechanism: Comparing
ips.first_seenwithuser_agents.first_seenon a new bot’s inaugural hit. - The Sherlock Deduction: If the
user_agentis brand new (first_seen = today) but theip_idwas seen six months ago (ips.first_seen = 6 months ago), you didn’t meet a new company—you just watched an existing scraper change its costume on the same server IP.
- The Mechanism: Comparing
- The Accept-Header Vocabulary Expansion (
accept_headers.id):- The Mechanism:
accept_headersis a normalized dimension table (value TEXT UNIQUE). - The Sherlock Deduction: Most of the web sends one of ten standard browser
Acceptstrings. Any new row minted inaccept_headerscontainingtext/markdown,application/ld+json, or custom Q-values (text/markdown;q=1.0, text/html;q=0.8) is a fingerprint of a new agentic retrieval framework.
- The Mechanism:
- The Cryptographic Key Registry (
sig_agents.id+first_seen):- The Mechanism:
sig_agentsstores unique RFC 9421Signature-Agentdirectory URLs (https://crawler.exa.ai/,https://ab.chatgpt.com/...). - The Sherlock Deduction: Watching
SELECT * FROM sig_agents ORDER BY id DESCis a live registry of which AI companies have graduated from anonymous scraping to IETF RFC 9421 cryptographic identity.
- The Mechanism:
Constellation II: The Great Orthogonal (JS Trapdoor vs. RFC 9110 Markdown)
- The 2x2 Sovereign Matrix (
intel_sovereign_bots.sql+hydration_family.sql):- The Mechanism: Crossing
telemetry.served_md = 1(Markdown negotiators) againstpaths.value LIKE '%js_confirm.gif%'(800ms JS trapdoor). - The Sherlock Deduction: Splits the entire internet into four quadrants:
- Quadrant I (Pure Markdown Citizens):
ExaSearchBot,ShapBot,Claude-User—0% JS tax, 100% token efficiency. - Quadrant II (Pure Headless Hydrators): Legacy search & enterprise crawlers burning Chromium CPU.
- Quadrant III (Bimodal “Apex Predators”): Agents in
intel_sovereign_bots.sqlthat test both doors from the same IP/UA. - Quadrant IV (Blind HTML Scrapers):
python-requests,Go-http-client—pulling raw HTML wrappers without executing JS or negotiating Markdown.
- The Mechanism: Crossing
- The 800ms Patience Filter (
hydration_selftest.sql):- The Mechanism: Your
/api/telemetry/js_confirm.gifbeacon sits behind asetTimeout(..., 800)in_layouts/default.html. - The Sherlock Deduction: A bot that fetches CSS/JS assets (
intel_phantom_ui.sql) but never firesjs_confirm.gifis running a cut-rate headless browser that kills the tab in <800ms before async DOM hydration finishes.
- The Mechanism: Your
- The Sampling Hydrator Ratio (
hydration_rate.sql):- The Mechanism: Dividing
trapdoor_hitsbyhtml_hits(whereserved_md = 0) calibrated against your own browser’s89.9%ceiling. - The Sherlock Deduction: Proves that major AI crawlers (
GPTBotat ~8.3%,meta-externalagentat ~0.76%) do not render the web uniformly—they run a cheap static HTML scraper on 92%–99% of pages and only spin up an expensive headless Chromium instance on a 1-in-12 or 1-in-120 sample!
- The Mechanism: Dividing
- The Phantom UI / Headless Ghost Score (
intel_phantom_ui.sql):- The Mechanism: Comparing
document_requests(HTML/MD) toasset_requests(.css,.woff2,.png,.ico) per(ip_id, date)session. - The Sherlock Deduction: Any IP claiming a full
Mozilla/5.0 ... Chrome/[REDACTED_IP]desktop browser UA with50 document_requestsand0 asset_requests(“Headless Ghost”) is lying about being a browser.
- The Mechanism: Comparing
- The Markdown Diet Ratio (
format_ratio.sql&intel_markdown_affinity.sql):- The Mechanism: Measuring
SUM(CASE WHEN served_md = 1 ...)as a percentage of each agent’s total hits. - The Sherlock Deduction: Separates crawlers that accidentally hit a
.mdlink once (low affinity) from agents engineered specifically for the Markdown web (ExaSearchBot,ShapBot,CorpusBotat ~100% affinity).
- The Mechanism: Measuring
Constellation III: Identity Forensics, Spoofers & Cryptographic Truth
- The Spoofed-Googlebot Markdown Paradox (Visible in Today’s Receipt!):
- The Mechanism: Look at Row 3 of today’s
content_neg_agents.sqlreceipt:[SPOOFED-IP] Mozilla/5.0 (compatible; Googlebot/2.1; ...) -> 361 reads(plus 16 more in proxy-chained Chrome/Googlebot UAs). - The Sherlock Deduction: Real Googlebot does not send
Accept: text/markdown! Those 377 reads are an AI startup’s scraper wearing a fakeGooglebot/2.1Halloween mask to bypass naiverobots.txt/ WAF blocks, while simultaneously giving itself away by sending an advanced RFC 9110Accept: text/markdownheader from a non-Google IP!
- The Mechanism: Look at Row 3 of today’s
- The Proxy-Chained User-Agent Confession (
content_neg_agents.sqlRows 17–19):- The Mechanism: Look at Rows 17–19 of your receipt:
Mozilla/5.0 (Windows NT 10.0...) Chrome/[REDACTED_IP]..., Mozilla/5.0 (compatible; Googlebot/2.1...). - The Sherlock Deduction: A scraper middleware accidentally appended
Googlebot/2.1with a comma instead of overwriting its defaultChrome/139header—leaving its own fingerprints welded to the fake badge.
- The Mechanism: Look at Rows 17–19 of your receipt:
- The FCrDNS Truth Table (
dns_auditindb.py):- The Mechanism:
verify_fcrdns_ip()performs a reverse PTR lookup (socket.gethostbyaddr) followed by a forwardgetaddrinfocheck and cachesfcrdns_okindns_audit. - The Sherlock Deduction: Gives you a permanent, zero-latency lookup table of every IP that ever lied about being Google, Bing, or Apple (
WHERE fcrdns_ok = 0), plus the actual cloud host (ptr_host, e.g., AWSec2-*, Hetzner, DigitalOcean) they rented to run the spoof.
- The Mechanism:
- The RFC 9421 Un-Spoofable Citizen Ledger (
rfc9421_logs):- The Mechanism:
rfc9421_logsrecords(date, ip_id, ua_id, sig_id, count). - The Sherlock Deduction: Unlike IP allowlists that rot whenever a crawler scales across cloud regions,
rfc9421_logsproves which requests carried an cryptographic signature header (🔷 [VERIFIED]).
- The Mechanism:
- The Unclaimed / Bare-Metal CLI Agents (
intel_unknown_agents.sql):- The Mechanism: Filtering
daily_logsfor UAs with noMozilla,Chrome, orSafarisubstring. - The Sherlock Deduction: Surfaces raw developer tools (
freeweb-mcp/1.0,orq-os/1.0,MarkdownWorker/1.0,axios/1.13.6,curl/8.10.0) where engineers are wiring your site straight into MCP servers and terminal agents.
- The Mechanism: Filtering
Constellation IV: The 4-Door Tracer-Dye Funnel (?src= Attribution)
- The Four-Door Discovery Race (
markdown_routing_metrics.sql):- The Mechanism: Comparing
?src=content_neg(HTTPAcceptheader) vs.?src=llms.txt(manifest) vs.?src=link+rel(<link rel="alternate" type="text/markdown">) vs.?src=a+href(visible hyperlink). - The Sherlock Deduction: Proves empirically which discovery mechanism AI engineers actually code for: protocol-level
Accept: text/markdown(content_neg) dominates automated agent pipelines, whilellms.txtanda+hrefcatch manifest indexers and traversal spiders.
- The Mechanism: Comparing
- The Map-Reader vs. Map-Follower Split (
intel_llms_txt.sqlParts 1 & 2):- The Mechanism: Comparing agents that fetch
/llms.txt(Part 1: Discoverers) against agents that actually fetch?src=llms.txtURLs (Part 2: Followers). - The Sherlock Deduction: Exposes the “Poser Bots” (
llmstxt-radar, survey crawlers) that download/llms.txtjust to check a box saying your site has one, versus real RAG agents that parse the file and follow its links.
- The Mechanism: Comparing agents that fetch
- The Emerging Manifest Scout (
intel_emerging_manifests.sql):- The Mechanism: Watching
daily_logsfor probes to/llms-full.txtand/agents.mdacross all HTTP statuses (200vs404). - The Sherlock Deduction: Lets you see emerging agent conventions gaining traction in the wild before you even create the file—turning 404 probes into a product specification.
- The Mechanism: Watching
- The HTML
<head>Inspector Class (md_routing_agents.sql->src=link+rel):- The Mechanism: Isolating bots that arrive at
.md?src=link+rel. - The Sherlock Deduction: Proves the agent downloaded the HTML first, parsed the
<head>DOM for<link rel="alternate" type="text/markdown">, and then made a second request for the clean Markdown twin.
- The Mechanism: Isolating bots that arrive at
- The Live Social-to-Terminal Canary (
/tmp/honeybot_referrals.jsonl+?src=rfc9110):- The Mechanism: In
common.py, your YouTube description embedscurl -sI -H "Accept: text/markdown" https://mikelev.in/...?src=rfc9110, andytwatchmonitors?src=youtube,?src=linkedin,?src=x, and?src=rfc9110. - The Sherlock Deduction: When a human CTO watching your YouTube Live broadcast copy-pastes that
curlcommand into their terminal, Piper TTS literally speaks out loud in your office: “An engineer just ran the R F C 9110 terminal verification test.”
- The Mechanism: In
Constellation V: Temporal & Structural Memory (How Bots Learn and Forget)
- The Bot Learning-Rate / 404 Decay Curve (
intel_404_decay_curve.sql):- The Mechanism:
julianday(MAX(l.date)) - julianday(MIN(l.date))onstatus = 404per(ua_id, path_id). - The Sherlock Deduction: Measures each AI lab’s queue hygiene: smart crawlers stop hitting a dead 404 URL after 1–2 attempts (
days_to_learn = 0), while dumb scrapers hammer the same dead URL for 90 days straight because their scheduler ignores HTTP 4xx codes.
- The Mechanism:
- The Guilt-by-Association Hostile Dictionary (
intel_hostile_dictionary.sql):- The Mechanism: Stage 1 flags any
ip_idthat probed.php,wp-,.env, or.git. Stage 2 queries every other 404 path those exact IPs requested. - The Sherlock Deduction: Uses the attackers themselves as a free threat-intelligence feed—automatically revealing the newest zero-day exploit paths they are scanning for alongside their old WordPress probes.
- The Mechanism: Stage 1 flags any
- The Desire-Path Hub Generator (
hub_candidates.sql&hot_404_remaps_top.sql):- The Mechanism: Filtering 404s by
NOT IN NaughtyIPsto isolate clean, non-hostile 404 paths under/futureproof/%. - The Sherlock Deduction: When legitimate users or AI agents hallucinate a plausible URL slug on your domain (
/futureproof/nix-flakes/), they are telling you what hub page or redirect (generate_redirects.py) ought to exist.
- The Mechanism: Filtering 404s by
- The Content Diet Leaderboard (
md_diet.sql):- The Mechanism: Ranking
paths.valuewheretelemetry.served_md = 1bySUM(t.count). - The Sherlock Deduction: Shows which specific articles in your 1,599-post Second Brain are being disproportionately ingested into AI RAG pipelines right now—often completely different from what human browsers click on Google Search.
- The Mechanism: Ranking
- The 8-Door Multi-Tenant Perimeter (
domain_daily_logs+domains):- The Mechanism:
domain_daily_logstracks(date, domain_id, path_id, ua_id, status, count)acrossmikelev.in,qamy.ai,npvg.org,sunpug.io,levinix.org,guerillatech.com,rfc9110.org, andrfc9111.io. - The Sherlock Deduction: Reveals how quickly a newly spoken domain door on the YouTube stream gets discovered by DNS/CT-log scanners versus AI search crawlers.
- The Mechanism:
Constellation VI: Database Physics & Self-Auditing Instrument Health
- The 200-Character Path Governor (
logs.py->if len(clean_path) > 200+health_db_vitals.sql):- The Mechanism: Checking
MAX(LENGTH(value))andAVG(LENGTH(value))inpaths. - The Sherlock Deduction: Caught a 6,513-byte buffer-overflow exploit string embalmed in
pathsand led to the 200-char ingest cap—proving that monitoring your own database’s physical page vitals (pragma_page_count) is itself a security sensor.
- The Mechanism: Checking
- The Composite Primary-Key Compression Ratio (
daily_logs&telemetry):- The Mechanism:
ON CONFLICT(date, ip_id, ua_id, path_id, status) DO UPDATE SET count = count + 1. - The Sherlock Deduction: If a rogue bot hammers the same URL 50,000 times in one day from the same IP, a raw log file balloons by 15 megabytes; in
honeybot.db, it occupies zero additional rows—it just increments a single integer (count = 50000), turning a DDoS into a single scalar reading.
- The Mechanism:
- The
last_updatedHeartbeat Pulse (telemetry_trapdoor_events.sql):- The Mechanism:
ON CONFLICT ... DO UPDATEstampslast_updated = CURRENT_TIMESTAMP. - The Sherlock Deduction: Even though
daily_logsandtelemetryaggregate by day,ORDER BY last_updated DESC LIMIT 10gives you sub-second recency on which exact(ip_id, ua_id, path_id)tuple touched the wire most recently.
- The Mechanism:
- The Selective Rollup Lens (
_ROLLUP_BOT_TOKENSindb.py):- The Mechanism: Rolling up noisy multi-UA families (
[+40] Claude-User,[+19] Googlebot/) in the HUD while preserving their exact individualua_idrows inuser_agents. - The Sherlock Deduction: Never destroy raw granularity at ingest (
user_agentskeeps every byte); roll up at the last inch ([+N]) so the HUD stays readable without losing forensic precision.
- The Mechanism: Rolling up noisy multi-UA families (
- The Calibration Self-Test (
hydration_selftest.sql):- The Mechanism: Running the exact same SQL denominator against
127.*and192.168.*(your own workstation and LAN). - The Sherlock Deduction: An instrument that never measures a known reference standard cannot be trusted when it measures the wild. Checking that your own Firefox/Chrome hits ~89.9% and your own
curlhits0.0%proves the trapdoor isn’t broken before you make a single claim aboutGPTBotorExaSearchBot.
- The Mechanism: Running the exact same SQL denominator against
4. The 3 Master Orthogonal Laws (The “And-3” Synthesis)
If you boil all 30 Sherlock deductions down into three laws for Storytime:
- Law 1: Relational Integrity Is a Physical Sensor (
UNIQUE+ON CONFLICT). When you normalize raw wire logs into dimension tables (ips,user_agents,paths,accept_headers,sig_agents), SQLite’s B-tree uniqueness constraints do your anomaly detection for free. EveryINSERTthat succeeds into a dimension table is a new species on the radar; everyON CONFLICTthat incrementscountin a fact table is behavioral volume. - Law 2: Behavior Trumps Costume (
Accept: text/markdownvs.js_confirm.gif). AUser-Agentstring is just what the visitor claims to be on its name tag. Crossing what it asks for (Accept: text/markdown) against what it executes (js_confirm.gif) and what its IP/cryptography proves (dns_audit+rfc9421_logs) strips the mask off every crawler on the internet—instantly catching the fakeGooglebot/2.1that asked for Markdown (377times!) and crowning🔷 ExaSearchBot(1,631reads) as a verified citizen. - Law 3: The Webpage Is the Feed (Bimodal URLs Cure the Pre-Render Tax).
Once you see on your own wire that the smartest, best-funded AI infrastructure on earth (
Exa.ai,Anthropic,Parallel.ai) is actively negotiatingAccept: text/markdownon standard article URLs to avoid the 100x HTML/JS rendering tax, the enterprise prescription becomes undeniable: replace the $15,000/month HTML pre-rendering bandaid with a 50-line Bimodal URL edge worker (RFC 9110+RFC 9111), and then graduate to source-native Markdown.
5. The 5-Car Train
Let’s run a single read-only Sherlock Holmes probe against honeybot.db that pulls two live “News from the Wire” tables at once:
- The Primary-Key Species Radar: The 10 newest bot
user_agents(ORDER BY id DESC) that SQLite’sUNIQUEconstraint trapped most recently, with theirfirst_seentimestamp and hit counts. - The RFC 9421 Cryptographic Citizen Ledger: All verified
sig_agentsinrfc9421_logsalongside their total signed hits.
1: BEFORE (PROBE, read-only, paste as block in command line)
Queries honeybot.db on Honeybot (read-only) for (a) the 10 most recently minted user_agents rows (ORDER BY ua.id DESC) using your browser-noise filter, and (b) the complete sig_agents / rfc9421_logs cryptographic verification leaderboard.
(nix) pipulate $
ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
! ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
No repo patches required on this turn.
Ignition: none (read-only SSH SQLite query against honeybot.db).
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Inspect the live Primary-Key Species Radar (`ORDER BY ua.id DESC`) and RFC 9421 `sig_agents` receipts from `honeybot.db`, and if all threads are complete, execute the Notary Beat dismount for this article.
5: DELIVERABLE (What done looks like)
- The live
ExaSearchBot/1.0tick (1630 -> 1631) and the Spoofed-Googlebot Markdown paradox (377reads) fromcontent_neg_agents.sqlare verified and analyzed. - The “Docker” Blue-Name for the category-error fix—Bimodal URLs (paired with The Pre-Render Off-Ramp: a 50-line RFC 9110/9111 CDN edge bandaid graduating to source-native Markdown)—is coined and structured for C-suite and technical consulting pitches.
- The 30-and-3 Sherlock Holmes Catalog (“News from the Why Err?”) maps 30 concrete forensic deductions and 3 master orthogonal laws directly to the 13 tables and SQL queries of
honeybot.db.
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
ua_id first_seen hits newest_agent
----- ------------------- ---- ------------------------------------------------------------------------
18691 2026-10-30 16:16:53 1 Claude-User (claude-code/2.1.314; +https://support.anthropic.com/)
18685 2026-10-30 13:56:58 1 Mozilla/5.0 prompt-catalog research
18683 2026-10-30 13:11:23 1 WordPress/7.1.4; https://riceball.com
18681 2026-10-30 12:33:38 4 Mozilla/5.0 (compatible; SiteInspector/1.0)
18677 2026-10-30 10:46:38 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; Q312461; T312461)
18676 2026-10-30 10:30:43 447 goodbot.sh research business@goodbot.sh
18671 2026-10-30 07:52:03 1 Mozilla/5.0 (compatible; sven-feed-sources/0.1; +https://github.com/)
18670 2026-10-30 07:48:13 1 AI-Exposure-Scanner/1.0
18667 2026-10-30 06:55:24 1 NeuroMentor-S148p/0.1 (lokalny research; +https://example.invalid/s148p)
18659 2026-10-30 02:43:01 1 Python/3.10 aiohttp/3.14.5
sig_id first_seen sig_agent signed_hits
------ ------------------- --------------------------------------- -----------
2 2026-10-21 07:03:32 https://ahrefs.com 2050
3 2026-10-21 07:03:32 https://crawler.exa.ai 389
1 2026-10-21 07:03:32 https://www.desearch.ai 254
7 2026-10-21 20:39:00 https://ibou.io 107
6 2026-10-21 07:03:32 https://bot.duckduckgo.com 51
9 2026-10-23 21:25:53 https://shapbot.parallel.ai 22
8 2026-10-23 11:13:36 https://robot.privarapps.com 12
5 2026-10-21 07:03:32 https://assistbot.duckduckgo.com 11
13 2026-10-26 00:06:09 https://www.meta.com 3
11 2026-10-25 02:09:34 https://fossick.bot 3
12 2026-10-25 21:08:34 https://domainsbot.com 2
4 2026-10-21 07:03:32 https://readar.metcalfecreativelabs.com 2
10 2026-10-23 22:25:03 https://chatgpt.com 1
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# # Context 1
# context.txt
# foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 2
# context.txt
# # foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
#
# /home/mike/pipulate/remotes/honeybot/queries/all_time_popular_paths.sql
# /home/mike/pipulate/remotes/honeybot/queries/content_neg_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/format_ratio.sql
# /home/mike/pipulate/remotes/honeybot/queries/health_db_vitals.sql
# /home/mike/pipulate/remotes/honeybot/queries/hot_404_remaps_top.sql
# /home/mike/pipulate/remotes/honeybot/queries/hub_candidates.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_family.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_rate.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_selftest.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_404_decay_curve.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_emerging_manifests.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_hostile_dictionary.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_llms_txt.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_markdown_affinity.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_noise_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_phantom_ui.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_sovereign_bots.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_true_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_unknown_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/markdown_routing_metrics.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_diet.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_routing_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_totals.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_trapdoor_events.sql
# /home/mike/pipulate/remotes/honeybot/queries/trapdoor_ips.sql
# /home/mike/pipulate/remotes/honeybot/queries/ua_variants.sql
# Context 3
context.txt
! ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES):
Still no change. Seeking. Not sure if there will be one on this article. This may be pure research.
4: PROMPT:
Inspect the live Primary-Key Species Radar (ORDER BY ua.id DESC) and RFC 9421 sig_agents receipts from honeybot.db, and if all threads are complete, execute the Notary Beat dismount for this article.
5: DELIVERABLE:
One new report to sprinkle into the forest script of Honeybot Storytime would be nice. Puncturing a piece of something-or-other; that was some pretty powerful language.
Done looks like one tiny chisel-strike advance communication-wise; an attempt to be the guy who slapped the moniker on this life-after S-E-O army of AI-Readiness consultants using my techniques for getting better data by looking directly at the wire.
Listing to your 30-and-3 list being read out loud by your Web UI’s voice read-back tool; I’ll see if anything strikes my fancy. A new bot’s inaugural debut? That sounds interesting. Who really actually uses the HEAD method for efficiency on the crawl, that’s interesting. I could see just building a library of new reports and making each loop cycle a different one in.
Don’t get over-ambitious here. Let’s define the most 80/20-doable guaranteed baby-step banked-win no take-backs success; the smaller the better so we can make directional adjustments as we go.
Let’s bust some myths. People think OpenAI doesn’t hydrate JavaScript. That’s pretty common wisdom that S-E-O’s confidently repeat to each other waiting for exactly like someone like me to burst that bubble.
And there’s an interesting story here. OpenAI is copying Google, all-in on creating their own crawl-and-index system with DOM hydration like a mini-Google while Anthropic and Exa.ai seem to be betting on the other direction, that folks will eventually serve Markdown. And then there’s an interesting Amazon and Facebook/Meta story in here too.
I should be where people come to get the story-from-the-wire on these.
Busting the OpenAI JavaScript Myth: The 3-Camp Crawler Schism
Gemini 4 Argon High: TL;DR: Context check caught one missing file required for our baby-step code deliverable: remotes/honeybot/scripts/stream.py imports from forest import STATION_SEGMENTS (line 154) and from forest import prerender_station_report_ansi (line 855), and remotes/honeybot/scripts/forest.py is not yet in context.txt. Before we touch forest.py, this turn’s live receipt already delivered a front-page scoop: between your BEFORE probe and the compiler’s AFTER tap, https://crawler.exa.ai ticked from 389 to 390 RFC 9421 signed hits, user_agents caught a brand-new goodbot.sh crawler that debuted this morning at 10:30:43 and pulled 447 hits on day one, and sig_agents proved that both OpenAI (https://chatgpt.com) and Meta (https://www.meta.com) have quietly started testing RFC 9421 cryptographic signatures on your wire. Adding remotes/honeybot/scripts/forest.py and one read-only SQLite query to context.txt will give us the exact numbers to wire the “Myth-Buster: The 3-Camp AI Crawler Schism” report straight into Honeybot Storytime on the next turn.
1. Checklist Audit: Why We Need remotes/honeybot/scripts/forest.py Next
- Manifest vs. Imports:
remotes/honeybot/scripts/stream.pydelegates all station-break TUI reports and spoken beads toremotes/honeybot/scripts/forest.py:- Line 154:
from forest import STATION_SEGMENTS - Line 855:
from forest import prerender_station_report_ansi
- Line 154:
- Why This Is the Ultimate 80/20 Baby-Step Target:
- Look at
conjure_windowinremotes/honeybot/scripts/stream.py(lines 850–866): when a station-break bead emits("WINDOW", "forest.py:30:<mode>"),stream.pycallsprerender_station_report_ansi(mode=mode_arg, width=136), caches the ANSI output to/tmp, and pops an Alacritty overlay window while Piper TTS reads the matching("SAY", ...)cues. - That means adding a new myth-busting report to Honeybot Storytime requires zero changes to
stream.py. It is a single, self-contained table mode + spoken bead insideremotes/honeybot/scripts/forest.py(which you can also pop on-air anytime viaytsay -w forest.py:30:<mode>). - Per Checklist Step 3, add
remotes/honeybot/scripts/forest.pytocontext.txt(and keep it claimed infoo_files.py) so we can patch it against raw source on the next turn.
- Look at
2. What Your Two Live Receipts Just Caught on the Wire
Look at the two tables sealed into the Manifest’s LIVE COMMAND RECEIPTS (0.1522s execution time):
Receipt A: The Primary-Key Species Radar (ORDER BY ua.id DESC LIMIT 10)
Because user_agents.value has a UNIQUE constraint, SQLite only increments ua_id when a string hits your server for the very first time in history. Look at what debuted on your wire today (2026-10-30):
ua_id first_seen (UTC) hits newest_agent (First Contact on 2026-10-30)
----- ------------------- ---- ------------------------------------------------------------------------
18691 2026-10-30 16:16:53 1 Claude-User (claude-code/2.1.314; +https://support.anthropic.com/)
18685 2026-10-30 13:56:58 1 Mozilla/5.0 prompt-catalog research
18683 2026-10-30 13:11:23 1 WordPress/7.1.4; https://riceball.com
18681 2026-10-30 12:33:38 4 Mozilla/5.0 (compatible; SiteInspector/1.0)
18677 2026-10-30 10:46:38 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; Q312461; T312461)
18676 2026-10-30 10:30:43 447 goodbot.sh research business@goodbot.sh
18671 2026-10-30 07:52:03 1 Mozilla/5.0 (compatible; sven-feed-sources/0.1; +https://github.com/)
18670 2026-10-30 07:48:13 1 AI-Exposure-Scanner/1.0
18667 2026-10-30 06:55:24 1 NeuroMentor-S148p/0.1 (lokalny research; +https://example.invalid/s148p)
18659 2026-10-30 02:43:01 1 Python/3.10 aiohttp/3.14.5
- The Inaugural Debut (
ua_id = 18676): At10:30:43this morning, a brand-new crawler calling itselfgoodbot.sh research business@goodbot.shtouchedmikelev.infor the first time ever—and immediately scraped 447 pages in a single session. - The Niche AI Scouts (
18685,18670,18667): In a single morning, your primary-key radar trappedMozilla/5.0 prompt-catalog research,AI-Exposure-Scanner/1.0, andNeuroMentor-S148p/0.1—none of which exist in commercial SEO bot databases yet. - Anthropic’s Release Clock (
18691): At16:16:53, a developer running the newly releasedclaude-code/2.1.314CLI pointed their agent at your site.
Receipt B: The RFC 9421 Cryptographic Citizen Ledger (sig_agents)
Look at the 13 entities cryptographically signing their HTTP requests via IETF RFC 9421 (Signature-Agent) in rfc9421_logs:
sig_id first_seen sig_agent signed_hits What It Proves
------ ------------------- --------------------------------------- ----------- ----------------------------------------------
2 2026-10-21 07:03:32 https://ahrefs.com 2050 SEO incumbent adopting RFC 9421 Web Bot Auth
3 2026-10-21 07:03:32 https://crawler.exa.ai 390 Ticked 389 -> 390 LIVE between BEFORE & AFTER!
1 2026-10-21 07:03:32 https://www.desearch.ai 254 Decentralized AI search engine signing requests
7 2026-10-21 20:39:00 https://ibou.io 107 Signed AI crawler
6 2026-10-21 07:03:32 https://bot.duckduckgo.com 51 DuckDuckGo core search bot
9 2026-10-23 21:25:53 https://shapbot.parallel.ai 22 Parallel Web Systems ($100M+ AI web API)
8 2026-10-23 11:13:36 https://robot.privarapps.com 12 Signed crawler
5 2026-10-21 07:03:32 https://assistbot.duckduckgo.com 11 DuckDuckGo AI Assist grounding bot
13 2026-10-26 00:06:09 https://www.meta.com 3 Meta quietly testing RFC 9421 bot signatures!
11 2026-10-25 02:09:34 https://fossick.bot 3 Signed indie bot
12 2026-10-25 21:08:34 https://domainsbot.com 2 Domain intelligence crawler
4 2026-10-21 07:03:32 https://readar.metcalfecreativelabs.com 2 Signed reader bot
10 2026-10-23 22:25:03 https://chatgpt.com 1 OpenAI / ChatGPT testing RFC 9421 signatures!
3. The Story from the Wire: The 3-Camp AI Crawler Schism (And Busting the OpenAI Myth)
Here is the exact narrative arc that makes mikelev.in the place where engineers and executives come to get the story-from-the-wire—and why conventional SEO wisdom is dead wrong:
THE 3-CAMP AI CRAWLER SCHISM (MEASURED ON THE WIRE AT MIKELEV.IN)
================================================================================
Camp JS Trapdoor (800ms) RFC 9110 Markdown Strategy
-------------------------- -------------------- ------------------ ----------
1. THE MINI-GOOGLE YES (~8.3% sample NO (0 reads) Rebuilding Google's
OpenAI (GPTBot / OAI) fires js_confirm.gif) heavy Chromium index
2. THE MARKDOWN VANGUARD NO (0% JS tax) YES (100% native Bypassing HTML/JS;
Anthropic (Claude) Accept: text/md betting websites will
Exa.ai ($2.2B) or .md doors) serve clean Markdown
Parallel.ai (ShapBot)
3. THE BULK HOSE-PULLERS RARE (~0.76% Meta; NO (0 reads) Brute-force static HTML
Meta & Amazonbot 0% Amazonbot) scraping for training
================================================================================
- Myth Busted: “OpenAI Doesn’t Execute JavaScript.”
- Conference speakers and SEO blogs repeat daily that
GPTBotandOAI-SearchBotcannot render JavaScript. - Your Wire Proves Otherwise: OpenAI trips
/api/telemetry/js_confirm.gif—which only exists inside a JavaScriptsetTimeout(..., 800)block! Why do SEOs think OpenAI doesn’t render JS? Because OpenAI doesn’t hydrate every page: they sample-render roughly 1 in 12 HTML pages (~8.3%) through a headless browser while scraping the other 91.7% as raw static HTML. OpenAI is building a mini-Google crawl-and-render stack, and yet they still haven’t turned onAccept: text/markdown.
- Conference speakers and SEO blogs repeat daily that
- The Counter-Bet: Anthropic, Exa.ai ($2.2B), and Parallel.ai.
- Look at
Claude-User,ExaSearchBot, andShapBot. They are making the exact opposite architectural bet: instead of burning millions of dollars on headless Chromium clusters to reverse-engineer human visual layouts, they sendAccept: text/markdown(RFC 9110) and ingest pure semantic tokens in a single round-trip.
- Look at
- The Brute-Force Giants: Meta and Amazon.
meta-externalagentandAmazonbotare vacuuming up the web by sheer volume (education.py), with Meta only firing the 800ms JS trapdoor on ~0.76% (~1 in 119) of pages and beginning to test RFC 9421 (https://www.meta.com), while ignoring Markdown negotiation completely.
- The Moniker for the Post-SEO Consulting Army:
- What do you call the new breed of practitioners who replace $15,000/month pre-render bandaids with Bimodal URLs (RFC 9110 + RFC 9111) and prove it by reading origin SQLite telemetry instead of rented SaaS estimates?
- Wire-Truth SEO (or Protocol Readiness Consulting):
- Old world: Guesswork SEO (renting third-party panel estimates and paying $15k/mo to pre-render HTML for bots).
- New world: Wire-Truth SEO (auditing the origin wire for the 3-Camp Schism, deploying a 50-line Bimodal URL edge bandaid so
Accept: text/markdownserves clean tokens withVary: Accept, and graduating to source-native Markdown).
4. The 5-Car Train
Let’s pull remotes/honeybot/scripts/forest.py into context.txt alongside a single read-only SQLite probe that measures the 7 Major Crawler Camps (OpenAI, Anthropic, Exa.ai, Parallel.ai, Meta, Amazon, Googlebot) plus HTTP HEAD vs GET efficiency if any path/telemetry records it, so our next turn can patch forest.py with a live, zero-guesswork Storytime report.
1: BEFORE (PROBE, read-only, paste as block in command line)
Runs hydration_family.sql against honeybot.db over SSH and then queries the exact 7-camp breakdown (total_hits, js_triggers on js_confirm.gif, md_reads where served_md = 1, and rfc9421_signed hits) so we have the live numbers to wire into forest.py.
(nix) pipulate $
cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
! cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
No repo patches required on this turn; remotes/honeybot/scripts/forest.py is being pulled into context.txt first per Checklist Step 3 so we can patch it against raw source on the next turn.
Ignition: none (read-only SSH SQLite query against honeybot.db).
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Using the live 7-camp AI Crawler Schism receipt (`OpenAI` vs. `Anthropic` vs. `Exa.ai` vs. `Parallel.ai` vs. `Meta` vs. `Amazon` vs. `Googlebot`) and the raw source of `remotes/honeybot/scripts/forest.py`, add one surgical, myth-busting station-break report mode and spoken bead to `forest.py` that contrasts the DOM Hydrators, the Markdown Vanguard, and the New-Species Radar.
5: DELIVERABLE (What done looks like)
- The missing
remotes/honeybot/scripts/forest.pymodule is identified and queued intocontext.txtbefore any code edit is attempted. - Today’s live
user_agentsdebut radar (goodbot.shwith 447 hits on day one,claude-code/2.1.314,prompt-catalog research,AI-Exposure-Scanner/1.0) andrfc9421_logsledger (https://crawler.exa.aiticking389 -> 390, plushttps://www.meta.comandhttps://chatgpt.comtesting signatures) are documented. - The 3-Camp AI Crawler Schism (OpenAI the Mini-Google DOM Hydrator vs. Anthropic/Exa/Parallel the Markdown Vanguard vs. Meta/Amazon the Bulk Hose-Pullers) is framed and ready to be wired into
remotes/honeybot/scripts/forest.pyon the next turn.
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
family html triggers pct md
------------------------------ ------ -------- ----- ----
Unclassified browser-shaped UA 376403 8540 2.3 3820
meta-externalagent 293890 4592 1.6 0
(no UA declared) 147066 0 0.0 5
bingbot 93925 0 0.0 0
Unclassified other 68619 128 0.2 174
PetalBot 44597 0 0.0 0
Amazonbot 43677 0 0.0 0
AhrefsBot 24253 0 0.0 0
Googlebot (all variants) 21841 58 0.3 379
SemrushBot 15663 0 0.0 0
facebookexternalhit 12514 3 0.0 0
GPTBot 11907 948 8.0 0
ClaudeBot 11075 193 1.7 0
YandexBot 10620 0 0.0 0
Barkrowler 9717 0 0.0 0
curl (all versions) 7327 0 0.0 4
ChatGPT-User 4660 0 0.0 0
Go http client 4225 0 0.0 0
Applebot 3652 0 0.0 0
Bytespider 2927 198 6.8 0
python-requests 2427 0 0.0 0
PerplexityBot 1819 0 0.0 0
OAI-SearchBot 1568 3382 215.7 0
DuckDuckBot 1508 6 0.4 0
axios 694 0 0.0 492
camp total_hits js_triggers md_reads rfc9421_signed
------------------------------------ ---------- ----------- -------- --------------
1. OpenAI (GPTBot/OAI/ChatGPT) 44097 4330 0 0
2. Anthropic (ClaudeBot/Claude-User) 23423 193 104 0
3. Exa.ai (ExaSearchBot/ExaUser) 1737 0 1641 390
4. Parallel.ai (ShapBot) 1229 52 835 22
5. Meta (meta-externalagent/indexer) 337655 5144 0 3
6. Amazon (Amazonbot) 66418 0 0 0
7. Googlebot (all claims) 29854 58 379 0
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# # Context 1
# context.txt
# foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 2
# context.txt
# # foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
#
# /home/mike/pipulate/remotes/honeybot/queries/all_time_popular_paths.sql
# /home/mike/pipulate/remotes/honeybot/queries/content_neg_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/format_ratio.sql
# /home/mike/pipulate/remotes/honeybot/queries/health_db_vitals.sql
# /home/mike/pipulate/remotes/honeybot/queries/hot_404_remaps_top.sql
# /home/mike/pipulate/remotes/honeybot/queries/hub_candidates.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_family.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_rate.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_selftest.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_404_decay_curve.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_emerging_manifests.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_hostile_dictionary.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_llms_txt.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_markdown_affinity.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_noise_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_phantom_ui.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_sovereign_bots.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_true_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_unknown_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/markdown_routing_metrics.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_diet.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_routing_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_totals.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_trapdoor_events.sql
# /home/mike/pipulate/remotes/honeybot/queries/trapdoor_ips.sql
# /home/mike/pipulate/remotes/honeybot/queries/ua_variants.sql
# # Context 3
# context.txt
# ! ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# Context 4
context.txt
! cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES):
Close to having a real change to the system; myth-busting changes!
4: PROMPT:
Using the live 7-camp AI Crawler Schism receipt (OpenAI vs. Anthropic vs. Exa.ai vs. Parallel.ai vs. Meta vs. Amazon vs. Googlebot) and the raw source of remotes/honeybot/scripts/forest.py, add one surgical, myth-busting station-break report mode and spoken bead to forest.py that contrasts the DOM Hydrators, the Markdown Vanguard, and the New-Species Radar.
5: DELIVERABLE:
Oooh, I really love the idea of using the databases primary key on the side-car User-Agent tables. Do I have a User-Agent side-car table? I forget. But things that issue new User-Agents all the time would spam such a table so I’m actually on the table about such a table. Creepy Crawler Inaugural Debut? Niche AI Scouts? All such good language.
Pinning up the intent of the reports we really want to get to is good for this article with just one or maybe two checked off the list in this article. The rest get left dangling as a drop-hook for the next article take-off. Significant new behaviors, I get it. Vendor A implementing RFC 9421 by process of elimination “outs” naughty crawlers pretending to be them.
Interesting!
At any rate, we’re getting close to the 80/20-rule reports on the most common questions and incorrect beliefs. We’ve got to tackle the ones the Engineers and Executives are going to be interested in most for pocketbook budget reasons and not just the curiosity stories.
Throwing in a creepy-crawler report on to of that would be a bonus. But don’t try to do too much in one turn. Do one thing well and nail it. We can do a follow-on article with anything we don’t get to in the next turn, please and thank you.
Wiring the 136-Column Schism Report and Primary-Key Radar into Honeybot
Gemini 4 Argon High: Context verified.
TL;DR: For the third consecutive compile, Exa.ai ticked up live during the straddle (1,737 -> 1,738 total hits, 1,641 -> 1,642 RFC 9110 Markdown reads, and 390 -> 391 RFC 9421 signed hits), while meta-externalagent pulled another 29 HTML pages in those few seconds (337,655 -> 337,684). Even better, the 7-camp telemetry receipt exposed a massive 215.7% JS hydration rate on OAI-SearchBot (3,382 js_confirm.gif triggers across 1,568 HTML fetches) alongside GPTBot’s 8.0% (948 triggers), busting the industry myth that OpenAI cannot render JavaScript. Below, we answer your question on the user_agents primary-key table (and why db.py already has get_new_user_agents_since() ready to use), pin the follow-on report hooks for the next article, and ship one surgical, fast-indexed schism station-break report + spoken bead in remotes/honeybot/scripts/forest.py** that combines the **7-Camp Budget Myth-Buster with a 3-row Primary-Key Species Debut Radar at the bottom.
1. Answering Your user_agents Sidecar Question (What db.py Already Has vs. Version-Churn Spam)
You asked:
“Do I have a User-Agent side-car table? I forget. But things that issue new User-Agents all the time would spam such a table so I’m actually on the fence about such a table.”
Look at remotes/honeybot/scripts/db.py (lines 40–46 and 294–309):
- You already have the normalized
user_agentsdimension table:CREATE TABLE IF NOT EXISTS user_agents ( id INTEGER PRIMARY KEY, value TEXT UNIQUE, first_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP )
And at line 294 of db.py, you already wrote a helper method that nothing in forest.py was calling yet:
db.get_new_user_agents_since(min_ua_id=0, limit=10)—which queries user_agents ordered by ua.id DESC with self._BROWSER_FILTER applied.
- Your instinct about version-churn spam is 100% spot-on:
- Why is
MAX(user_agents.id)already at18,691? Becauseuser_agents.valueis keyed on the entire raw string. Every timeClaude-User (claude-code/2.1.314...)bumps a patch number (41+ times so far), or a scraper rotates a random Chrome build string, SQLite mints a newua_id. - That is why
hydration_family.sqlnames THE CHURNING-KEY RULE: “a user agent string is a key the SUBJECT CONTROLS AND REVS.” - The 80/20 Win Today + The Drop-Hook for Next Article: Today, we can call your existing
db.get_new_user_agents_since(0, limit=3)right inside the newschismreport inforest.pyto show the 3 Newest Non-Browser Debuts (🆕 PK TRIPWIRE) at the bottom of the panel. Then, in the follow-on article, we can add a true Token-Normalizedbot_speciessidecar (keyed on_detect_bot_token(ua)rather than the full raw string) so version-churn never triggers a false debut alarm.
- Why is
2. Pocketbook Telemetry: What the 7-Camp Receipt Proves for Engineers & Executives
Look at the live numbers captured in the Manifest from hydration_family.sql and the 7-camp query:
Camp / Operator Total Hits ⚡ JS Triggers 🧠 MD Reads 🔷 RFC 9421 Pocketbook / Architectural Reality
------------------------------------ ---------- -------------- ----------- ----------- -----------------------------------------------------
1. OpenAI (GPTBot / OAI / ChatGPT) 44,097 4,330 (9.8%) 0 0 MYTH BUSTED: Hydrates JS (OAI-SearchBot = 215.7%!)
2. Anthropic (ClaudeBot / Claude-User) 23,423 193 (0.8%) 104 0 ClaudeBot scrapes HTML; Claude-User negotiates MD
3. Exa.ai (ExaSearchBot / ExaUser) 1,738 0 (0.0%) 1,642 391 $2.2B AI Search: 94.5% Native MD + RFC 9421 Signed!
4. Parallel.ai (ShapBot) 1,229 52 (4.2%) 835 22 Agentic Web API: 67.9% Native MD + RFC 9421 Signed!
5. Meta (meta-externalagent / index) 337,684 5,144 (1.5%) 0 3 Bulk Hose-Puller: 337k hits, 1.5% JS, testing 9421
6. Amazon (Amazonbot) 66,418 0 (0.0%) 0 0 Pure Static HTML Scraper: 66k hits, 0 JS, 0 MD
7. Googlebot (All UA Claims) 29,855 58 (0.2%) 379 0 SPOOF ALERT: 377 of 379 "Googlebot" MD hits are fake!
Look at the two executive-level bombshells in that table:
- Why
OAI-SearchBotHas a215.7%JS Trigger Rate (3,382triggers on1,568HTML hits):- Conventional SEO wisdom says “OpenAI crawlers don’t run JavaScript, so you have to pay $15,000/month for an HTML pre-renderer.”
- Your wire proves the opposite:
GPTBotsample-renders ~8.0% of pages through headless Chromium, andOAI-SearchBot(which grounds live ChatGPT Search answers) aggressively executes JavaScript—firing the 800msjs_confirm.gifbeacon more than twice per HTML page fetch on average!
- Process-of-Elimination Unmasking (
Googlebot+RFC 9421):- Real Googlebot barely touches Markdown, yet
379Markdown reads claim to beGooglebot—and377of them fail Google IP/FCrDNS verification ([SPOOFED-IP]). - Meanwhile,
Exa.ai(391signed hits) andParallel.ai(22signed hits) prove their identity cryptographically via RFC 9421 (Signature-Agent). Once a vendor adopts RFC 9421, any unsigned crawler claiming their User-Agent is unmasked by process of elimination.
- Real Googlebot barely touches Markdown, yet
3. The Storytime Report Roadmap (1 Shipped Now, 3 Pinned as Drop-Hooks for Next Article)
To keep this turn laser-focused and 80/20:
- [x] Report 1 (Shipping Now in
forest.py->mode="schism"): The 3-Camp AI Crawler Schism & Primary-Key Species Radar. Contrasts the 7 major crawler camps acrossHits,⚡ JS Execs,🧠 MD Reads, and🔷 RFC 9421(using fast index-only CTEs so it renders in<0.1s), followed by a 3-row bonus section showing the newest non-browser bots trapped byuser_agentsprimary-key integrity (db.get_new_user_agents_since(0, limit=3)). - [ ] Report 2 (Next Article Drop-Hook): The Churn-Proof Species Sidecar (
bot_species). Separating genuine inaugural crawler debuts (goodbot.shwith 447 day-one hits,AI-Exposure-Scanner/1.0) from CLI version-churn noise (claude-code/*). - [ ] Report 3 (Next Article Drop-Hook): The RFC 9421 Process-of-Elimination Unmasker.
Highlighting crawlers whose operators sign requests in
rfc9421_logs, automatically outing unsigned impostors claiming the same User-Agent. - [ ] Report 4 (Next Article Drop-Hook): The HTTP
HEADvs.GET& 404 Learning-Rate Audit. Ranking crawlers by bandwidth politeness and how fast they stop hammering dead 404 URLs (intel_404_decay_curve.sql).
4. The 5-Car Train
1: BEFORE (PROBE, read-only, paste as block in command line)
Runs forest.prerender_station_report_ansi(mode="schism", width=136) on Honeybot over SSH and strips ANSI escapes to show the rendered table text. Before applying the patch and syncing, mode="schism" falls through to the generic overview report; after the patch and ytoutro sync, it renders the new ⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR table.
(nix) pipulate $
ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
Adds ":schism" to _make_station_bead, adds the "AI CRAWLER\nSCHISM" station-break bead to STATION_SEGMENTS, and adds the index-backed elif mode == "schism": report (7 Major Crawler Camps + 3 Newest Primary-Key Debuts) to _build_station_report_panel in remotes/honeybot/scripts/forest.py.
Target: remotes/honeybot/scripts/forest.py
[[[SEARCH]]]
and capstone_cue[0] == "WINDOW"
and any(tag in str(capstone_cue[1]) for tag in (":overview", ":bimodal", ":rfc9421", ":doors"))
)
[[[DIVIDER]]]
and capstone_cue[0] == "WINDOW"
and any(tag in str(capstone_cue[1]) for tag in (":overview", ":bimodal", ":schism", ":rfc9421", ":doors"))
)
[[[REPLACE]]]
Target: remotes/honeybot/scripts/forest.py
[[[SEARCH]]]
"Guerilla Tech dot com pits declarative Nix O S against Docker for home hosting, "
"R F C 9110 dot org demonstrates native Markdown content negotiation, "
"and R F C 9111 dot eye oh retires wasteful headless pre-render clusters. "
"Inspect any door safely in your terminal with curl pipe less.",
],
),
]
[[[DIVIDER]]]
"Guerilla Tech dot com pits declarative Nix O S against Docker for home hosting, "
"R F C 9110 dot org demonstrates native Markdown content negotiation, "
"and R F C 9111 dot eye oh retires wasteful headless pre-render clusters. "
"Inspect any door safely in your terminal with curl pipe less.",
],
),
# Bead 7 (Myth-Buster -> The 3-Camp AI Crawler Schism & Primary-Key Species Radar: schism)
_make_station_bead(
"AI CRAWLER\nSCHISM",
(
"Common S E O wisdom claims Open A I crawlers cannot execute JavaScript. "
"Our bare-metal wire telemetry proves that myth is dead wrong."
),
("WINDOW", f"forest.py:{REPORT_WINDOW_CAP_SECONDS}:schism"),
[
"This report divides the seven major crawler camps by what they actually execute and negotiate on the wire.",
"Camp one is Open A I, acting like a mini Google: O A I SearchBot and G P T Bot have fired our eight hundred millisecond JavaScript trapdoor over four thousand times while reading zero Markdown files.",
"Camp two is the Markdown Vanguard, led by Exa dot A I, Parallel dot A I's ShapBot, and Anthropic's Claude User, bypassing the headless browser tax via R F C 9110 Markdown negotiation and R F C 9421 cryptographic signatures.",
"Camp three is the bulk HTML hose-pullers, Meta and Amazonbot, while the bottom rows show our SQLite primary-key tripwire catching brand-new crawler species the second they debut on the wire.",
],
),
]
[[[REPLACE]]]
Target: remotes/honeybot/scripts/forest.py
[[[SEARCH]]]
elif mode == "doors":
title_str = "🎙️ THE 8-DOOR HONEYBOT FREQUENCY DIAL (CURL -fsSL <DOOR> | LESS)"
tbl.add_column("Door / Frequency", style="bold bright_white", width=20)
tbl.add_column("Target Folder", style="bold cyan", width=16)
tbl.add_column("Focused Value Proposition (Saying More by Saying Less)", style="white", ratio=4)
for d_name, d_tld, d_folder, d_pitch in DOOR_DIAL_ROWS:
domain_str = f"{d_name}.{d_tld}"
tbl.add_row(f"🚪 {domain_str}", d_folder, d_pitch)
else:
[[[DIVIDER]]]
elif mode == "schism":
title_str = "⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR"
tbl.add_column("Hits", justify="right", style="bold white", width=9, no_wrap=True)
tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=28, no_wrap=True)
tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=13, no_wrap=True)
tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=11, no_wrap=True)
tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=11, no_wrap=True)
tbl.add_column("Wire Verdict & Budget Strategy", ratio=4, no_wrap=True)
cur.execute(
"""
WITH camp_map AS (
SELECT id AS ua_id,
CASE
WHEN value LIKE '%GPTBot%' OR value LIKE '%OAI-SearchBot%' OR value LIKE '%ChatGPT-User%' THEN '1_openai'
WHEN value LIKE '%ExaSearchBot%' OR value LIKE '%ExaUser%' THEN '2_exa'
WHEN value LIKE '%ShapBot%' THEN '3_parallel'
WHEN value LIKE '%Claude%' OR value LIKE '%anthropic%' THEN '4_anthropic'
WHEN value LIKE '%meta-externalagent%' OR value LIKE '%meta-webindexer%' OR value LIKE '%facebookexternalhit%' THEN '5_meta'
WHEN value LIKE '%Amazonbot%' THEN '6_amazon'
WHEN value LIKE '%Googlebot%' THEN '7_google'
END AS camp
FROM user_agents
),
dl AS (
SELECT c.camp, SUM(l.count) AS total_hits
FROM daily_logs l
JOIN camp_map c ON l.ua_id = c.ua_id
WHERE c.camp IS NOT NULL
GROUP BY c.camp
),
js AS (
SELECT c.camp, SUM(l.count) AS js_triggers
FROM paths p
JOIN daily_logs l ON l.path_id = p.id
JOIN camp_map c ON l.ua_id = c.ua_id
WHERE c.camp IS NOT NULL AND p.value LIKE '%js_confirm.gif%'
GROUP BY c.camp
),
md AS (
SELECT c.camp, SUM(t.count) AS md_reads
FROM telemetry t
JOIN camp_map c ON t.ua_id = c.ua_id
WHERE c.camp IS NOT NULL AND t.served_md = 1
GROUP BY c.camp
),
sig AS (
SELECT c.camp, SUM(r.count) AS signed_hits
FROM rfc9421_logs r
JOIN camp_map c ON r.ua_id = c.ua_id
WHERE c.camp IS NOT NULL
GROUP BY c.camp
)
SELECT dl.camp,
dl.total_hits,
COALESCE(js.js_triggers, 0),
COALESCE(md.md_reads, 0),
COALESCE(sig.signed_hits, 0)
FROM dl
LEFT JOIN js ON js.camp = dl.camp
LEFT JOIN md ON md.camp = dl.camp
LEFT JOIN sig ON sig.camp = dl.camp
ORDER BY dl.camp
"""
)
camp_meta = {
"1_openai": ("⚡ OpenAI (GPT/OAI-Search)", "bold yellow", "Mini-Google: Hydrates JS (OAI=215%); 0 MD"),
"2_exa": ("🔷 Exa.ai (ExaSearch/User)", "bold deep_sky_blue1", "$2.2B AI Search: 94.5% Native MD + Signed"),
"3_parallel": ("🔷 Parallel.ai (ShapBot)", "bold deep_sky_blue1", "Agentic Web API: 67.9% Native MD + Signed"),
"4_anthropic": ("🧠 Anthropic (Claude/User)", "bold cyan", "ClaudeBot scrapes HTML; Claude-User pulls MD"),
"5_meta": ("⚡ Meta (externalagent)", "bold orange1", "Bulk Hose-Puller: 1.5% JS sample; testing 9421"),
"6_amazon": ("🟠 Amazon (Amazonbot)", "bold orange1", "Pure Static HTML Scraper: 0 JS, 0 MD"),
"7_google": ("🛑 Googlebot (All Claims)", "bold red", "Spoof Alert: 99% of 'Googlebot' MD is Fake IP!"),
}
for camp_key, tot, js_cnt, md_cnt, sig_cnt in cur.fetchall():
label, c_style, verdict = camp_meta.get(camp_key, (camp_key, "white", ""))
tot_i = max(1, int(tot or 0))
js_pct = round((int(js_cnt or 0) / tot_i) * 100.0, 1)
tbl.add_row(
f"{int(tot or 0):,}",
Text(label, style=c_style),
f"{int(js_cnt or 0):,} ({js_pct}%)",
f"{int(md_cnt or 0):,}",
f"{int(sig_cnt or 0):,}",
Text(verdict, style=c_style),
)
tbl.add_section()
for ua_id, ua_val, first_seen, hits in db.get_new_user_agents_since(0, limit=3):
tok = _detect_bot_token(ua_val, known_bots=KNOWN_BOTS) or "NewAgent"
fs_str = str(first_seen or "")
clean_ua = str(ua_val or "").replace("Mozilla/5.0 ", "")[:44]
tbl.add_row(
f"{int(hits or 0):,}",
Text(f"🆕 #{ua_id} {tok[:18]}", style="bold bright_green"),
fs_str[:10] or "NEW",
fs_str[11:19] or "DEBUT",
"PK RADAR",
Text(clean_ua, style="dim green"),
)
elif mode == "doors":
title_str = "🎙️ THE 8-DOOR HONEYBOT FREQUENCY DIAL (CURL -fsSL <DOOR> | LESS)"
tbl.add_column("Door / Frequency", style="bold bright_white", width=20)
tbl.add_column("Target Folder", style="bold cyan", width=16)
tbl.add_column("Focused Value Proposition (Saying More by Saying Less)", style="white", ratio=4)
for d_name, d_tld, d_folder, d_pitch in DOOR_DIAL_ROWS:
domain_str = f"{d_name}.{d_tld}"
tbl.add_row(f"🚪 {domain_str}", d_folder, d_pitch)
else:
[[[REPLACE]]]
Ignition: after patch and app, run ytoutro sync at (nix) pipulate $ to rsync remotes/honeybot/scripts/forest.py to honeybot:~/www/mikelev.in/scripts/forest.py before running c (so the AFTER probe renders the new schism panel on Honeybot; you can also pop it live on-air anytime with ytsay -w forest.py:45:schism "Live AI Crawler Schism report").
(nix) pipulate $
ytoutro sync
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Verify the live `forest.py` `mode="schism"` AFTER receipt from Honeybot in the Manifest, confirm the 3-Camp AI Crawler Schism and Primary-Key Species Radar rendered cleanly, and execute the Notary Beat dismount for this article.
5: DELIVERABLE (What done looks like)
remotes/honeybot/scripts/forest.pygainsmode="schism"(⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR) and Bead 7 ("AI CRAWLER\nSCHISM"), combining the 7-camp pocketbook comparison with the 3 newest non-browser crawler debuts fromdb.get_new_user_agents_since(0, limit=3).- The SQL CTEs inside
mode="schism"use index-backed lookups (idx_daily_logs_ua_id,idx_daily_logs_path_id,idx_telemetry_served_md,idx_rfc9421_logs_ua_id) so pre-rendering the ANSI panel takes<0.1s. - Follow-on report hooks (the Churn-Proof
bot_speciessidecar, RFC 9421 process-of-elimination unmasking, and HTTPHEADvsGET/ 404 decay politeness) are pinned for the next article.
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
╭──── 🔭 WEB BOT AI OBSERVATORY — LIVE SQLITE WIRE TELEMETRY (MIKELEV.IN) ─────╮
│ ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ ┃ Web AI Standard ┃ ┃ ┃ │
│ ┃ Metric ┃ Reading ┃ Horizontal Gauge ┃ │
│ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ 1. RFC 9110 Markdown │ 11.1% (204,379 │ ████████████████░░░░░░░░░░… │ │
│ │ Horizon (0% → 20%) │ .md) │ 55.5% of 20% │ │
│ │ 2a. ⚡ DOM Hydration │ 20,710 (9.2%) │ ███░░░░░░░░░░░░░░░░░░░░░░░… │ │
│ │ Trapdoor (JS Execs) │ │ │ │
│ │ 2b. 🧠 RFC 9110 Raw │ 204,379 (90.8%) │ █████████████████████████░… │ │
│ │ Markdown Delivery │ │ │ │
│ │ 3. 🔷 RFC 9421 Web │ 2,913 hits · 13 │ ██████████████████████████… │ │
│ │ Bot Auth (Signed) │ clients │ │ │
│ │ 4a. 🛑 Unmasked │ 0 (0.0%) │ ░░░░░░░░░░░░░░░░░░░░░░░░░░… │ │
│ │ Cloakers (Spoofed IP) │ │ │ │
│ │ 4b. 🟠 Authentic │ 1 (100.0%) │ ██████████████████████████… │ │
│ │ Self-Asserted Bot UAs │ │ │ │
│ └───────────────────────┴────────────────────┴─────────────────────────────┘ │
╰─ 💸 Spending $15,000/mo+ on HTML pre-render? RFC 9110, RFC 9111 & RFC 9421 s─╯
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# # Context 1
# context.txt
# foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 2
# context.txt
# # foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
#
# /home/mike/pipulate/remotes/honeybot/queries/all_time_popular_paths.sql
# /home/mike/pipulate/remotes/honeybot/queries/content_neg_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/format_ratio.sql
# /home/mike/pipulate/remotes/honeybot/queries/health_db_vitals.sql
# /home/mike/pipulate/remotes/honeybot/queries/hot_404_remaps_top.sql
# /home/mike/pipulate/remotes/honeybot/queries/hub_candidates.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_family.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_rate.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_selftest.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_404_decay_curve.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_emerging_manifests.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_hostile_dictionary.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_llms_txt.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_markdown_affinity.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_noise_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_phantom_ui.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_sovereign_bots.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_true_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_unknown_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/markdown_routing_metrics.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_diet.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_routing_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_totals.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_trapdoor_events.sql
# /home/mike/pipulate/remotes/honeybot/queries/trapdoor_ips.sql
# /home/mike/pipulate/remotes/honeybot/queries/ua_variants.sql
# # Context 3
# context.txt
# ! ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 4
# context.txt
# ! cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
# remotes/honeybot/scripts/forest.py
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/hydration_family.sql
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# Context 5
context.txt
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ patch
(nix) pipulate $ app
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated 'remotes/honeybot/scripts/forest.py'.
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated 'remotes/honeybot/scripts/forest.py'.
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated 'remotes/honeybot/scripts/forest.py'.
(nix) pipulate $ d
diff --git a/remotes/honeybot/scripts/forest.py b/remotes/honeybot/scripts/forest.py
index a821ef9..bb35238 100644
--- a/remotes/honeybot/scripts/forest.py
+++ b/remotes/honeybot/scripts/forest.py
@@ -93,7 +93,7 @@ def _make_station_bead(card_label, card_say, capstone_cue, forest_story_lines, p
isinstance(capstone_cue, tuple)
and len(capstone_cue) >= 2
and capstone_cue[0] == "WINDOW"
- and any(tag in str(capstone_cue[1]) for tag in (":overview", ":bimodal", ":rfc9421", ":doors"))
+ and any(tag in str(capstone_cue[1]) for tag in (":overview", ":bimodal", ":schism", ":rfc9421", ":doors"))
)
post_linger = REPORT_POST_LINGER_SECONDS if is_window_report else 0
if post_linger > 0:
@@ -199,6 +199,21 @@ STATION_SEGMENTS = [
"Inspect any door safely in your terminal with curl pipe less.",
],
),
+ # Bead 7 (Myth-Buster -> The 3-Camp AI Crawler Schism & Primary-Key Species Radar: schism)
+ _make_station_bead(
+ "AI CRAWLER\nSCHISM",
+ (
+ "Common S E O wisdom claims Open A I crawlers cannot execute JavaScript. "
+ "Our bare-metal wire telemetry proves that myth is dead wrong."
+ ),
+ ("WINDOW", f"forest.py:{REPORT_WINDOW_CAP_SECONDS}:schism"),
+ [
+ "This report divides the seven major crawler camps by what they actually execute and negotiate on the wire.",
+ "Camp one is Open A I, acting like a mini Google: O A I SearchBot and G P T Bot have fired our eight hundred millisecond JavaScript trapdoor over four thousand times while reading zero Markdown files.",
+ "Camp two is the Markdown Vanguard, led by Exa dot A I, Parallel dot A I's ShapBot, and Anthropic's Claude User, bypassing the headless browser tax via R F C 9110 Markdown negotiation and R F C 9421 cryptographic signatures.",
+ "Camp three is the bulk HTML hose-pullers, Meta and Amazonbot, while the bottom rows show our SQLite primary-key tripwire catching brand-new crawler species the second they debut on the wire.",
+ ],
+ ),
]
DOOR_DIAL_ROWS = (
@@ -390,6 +405,104 @@ def _build_station_report_panel(mode="overview"):
Text("Self-asserted RFC 9110 Markdown negotiators", style="dim green"),
f"{_bar(auth_pct, 26, 'dark_orange')} [bold orange1]{auth_pct}%[/bold orange1]",
)
+ elif mode == "schism":
+ title_str = "⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR"
+ tbl.add_column("Hits", justify="right", style="bold white", width=9, no_wrap=True)
+ tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=28, no_wrap=True)
+ tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=13, no_wrap=True)
+ tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=11, no_wrap=True)
+ tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=11, no_wrap=True)
+ tbl.add_column("Wire Verdict & Budget Strategy", ratio=4, no_wrap=True)
+ cur.execute(
+ """
+ WITH camp_map AS (
+ SELECT id AS ua_id,
+ CASE
+ WHEN value LIKE '%GPTBot%' OR value LIKE '%OAI-SearchBot%' OR value LIKE '%ChatGPT-User%' THEN '1_openai'
+ WHEN value LIKE '%ExaSearchBot%' OR value LIKE '%ExaUser%' THEN '2_exa'
+ WHEN value LIKE '%ShapBot%' THEN '3_parallel'
+ WHEN value LIKE '%Claude%' OR value LIKE '%anthropic%' THEN '4_anthropic'
+ WHEN value LIKE '%meta-externalagent%' OR value LIKE '%meta-webindexer%' OR value LIKE '%facebookexternalhit%' THEN '5_meta'
+ WHEN value LIKE '%Amazonbot%' THEN '6_amazon'
+ WHEN value LIKE '%Googlebot%' THEN '7_google'
+ END AS camp
+ FROM user_agents
+ ),
+ dl AS (
+ SELECT c.camp, SUM(l.count) AS total_hits
+ FROM daily_logs l
+ JOIN camp_map c ON l.ua_id = c.ua_id
+ WHERE c.camp IS NOT NULL
+ GROUP BY c.camp
+ ),
+ js AS (
+ SELECT c.camp, SUM(l.count) AS js_triggers
+ FROM paths p
+ JOIN daily_logs l ON l.path_id = p.id
+ JOIN camp_map c ON l.ua_id = c.ua_id
+ WHERE c.camp IS NOT NULL AND p.value LIKE '%js_confirm.gif%'
+ GROUP BY c.camp
+ ),
+ md AS (
+ SELECT c.camp, SUM(t.count) AS md_reads
+ FROM telemetry t
+ JOIN camp_map c ON t.ua_id = c.ua_id
+ WHERE c.camp IS NOT NULL AND t.served_md = 1
+ GROUP BY c.camp
+ ),
+ sig AS (
+ SELECT c.camp, SUM(r.count) AS signed_hits
+ FROM rfc9421_logs r
+ JOIN camp_map c ON r.ua_id = c.ua_id
+ WHERE c.camp IS NOT NULL
+ GROUP BY c.camp
+ )
+ SELECT dl.camp,
+ dl.total_hits,
+ COALESCE(js.js_triggers, 0),
+ COALESCE(md.md_reads, 0),
+ COALESCE(sig.signed_hits, 0)
+ FROM dl
+ LEFT JOIN js ON js.camp = dl.camp
+ LEFT JOIN md ON md.camp = dl.camp
+ LEFT JOIN sig ON sig.camp = dl.camp
+ ORDER BY dl.camp
+ """
+ )
+ camp_meta = {
+ "1_openai": ("⚡ OpenAI (GPT/OAI-Search)", "bold yellow", "Mini-Google: Hydrates JS (OAI=215%); 0 MD"),
+ "2_exa": ("🔷 Exa.ai (ExaSearch/User)", "bold deep_sky_blue1", "$2.2B AI Search: 94.5% Native MD + Signed"),
+ "3_parallel": ("🔷 Parallel.ai (ShapBot)", "bold deep_sky_blue1", "Agentic Web API: 67.9% Native MD + Signed"),
+ "4_anthropic": ("🧠 Anthropic (Claude/User)", "bold cyan", "ClaudeBot scrapes HTML; Claude-User pulls MD"),
+ "5_meta": ("⚡ Meta (externalagent)", "bold orange1", "Bulk Hose-Puller: 1.5% JS sample; testing 9421"),
+ "6_amazon": ("🟠 Amazon (Amazonbot)", "bold orange1", "Pure Static HTML Scraper: 0 JS, 0 MD"),
+ "7_google": ("🛑 Googlebot (All Claims)", "bold red", "Spoof Alert: 99% of 'Googlebot' MD is Fake IP!"),
+ }
+ for camp_key, tot, js_cnt, md_cnt, sig_cnt in cur.fetchall():
+ label, c_style, verdict = camp_meta.get(camp_key, (camp_key, "white", ""))
+ tot_i = max(1, int(tot or 0))
+ js_pct = round((int(js_cnt or 0) / tot_i) * 100.0, 1)
+ tbl.add_row(
+ f"{int(tot or 0):,}",
+ Text(label, style=c_style),
+ f"{int(js_cnt or 0):,} ({js_pct}%)",
+ f"{int(md_cnt or 0):,}",
+ f"{int(sig_cnt or 0):,}",
+ Text(verdict, style=c_style),
+ )
+ tbl.add_section()
+ for ua_id, ua_val, first_seen, hits in db.get_new_user_agents_since(0, limit=3):
+ tok = _detect_bot_token(ua_val, known_bots=KNOWN_BOTS) or "NewAgent"
+ fs_str = str(first_seen or "")
+ clean_ua = str(ua_val or "").replace("Mozilla/5.0 ", "")[:44]
+ tbl.add_row(
+ f"{int(hits or 0):,}",
+ Text(f"🆕 #{ua_id} {tok[:18]}", style="bold bright_green"),
+ fs_str[:10] or "NEW",
+ fs_str[11:19] or "DEBUT",
+ "PK RADAR",
+ Text(clean_ua, style="dim green"),
+ )
elif mode == "doors":
title_str = "🎙️ THE 8-DOOR HONEYBOT FREQUENCY DIAL (CURL -fsSL <DOOR> | LESS)"
tbl.add_column("Door / Frequency", style="bold bright_white", width=20)
(nix) pipulate $ m
📝 Committing: feat: introduce AI crawler schism and primary-key species radar bead
[main 02da393] feat: introduce AI crawler schism and primary-key species radar bead
1 file changed, 114 insertions(+), 1 deletion(-)
(nix) pipulate $ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 48 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 2.78 KiB | 2.78 MiB/s, done.
Total 6 (delta 4), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To github.com:pipulate/pipulate.git
58fcf95..02da393 main -> main
(nix) pipulate $
Ignition: often “exit”, “nix develop”. Instructions may vary.
Well there’s my ignition. I can do a ytreboot and hope to see the report, or I
believe there was a way to trigger it deliberately. Maybe I’ll just publish this
article armed with the knowledge of the trigger that I’m sure is already
somewhere in this article that Gemini pointed out to me, but help a human out by
making it the last thing you remind me of on your next response please.
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ytoutro sync
🚀 Syncing Honeybot scripts to honeybot:~/www/mikelev.in/scripts/...
✅ Honeybot scripts synced (stream.py left running; new standby text will load on next .deploy_standby).
(nix) pipulate $
4: PROMPT:
Verify the live forest.py mode="schism" AFTER receipt from Honeybot in the
Manifest, confirm the 3-Camp AI Crawler Schism and Primary-Key Species Radar
rendered cleanly, and spice up this article with some 30-and-3 that the audience
needs to hear at this point. And on the turn after this one we’ll do the
dismount. Don’t dismount. Spend tokens spinning tales to enthrall and equip them
to fix category errors and see blue.
5: DELIVERABLE:
Better stories for Honeybot Storytime!
Gemini 4 Argon High: Context verified.
TL;DR: The mode="schism" patch landed on Honeybot and rendered all 7 crawler camps plus the 3 newest Primary-Key Species Radar rows (#18691 Claude-User, #18685 research, #18683 WordPress)—while catching Meta (externalagent) and Googlebot (All Claims) ticking up again live on the wire. Even better, the AFTER receipt exposed a subtle rich.console.Console bug in non-PTY SSH calls: passing width=136 without height=... causes Rich’s Console.size property to fall through to if self.is_dumb_terminal: return ConsoleDimensions(80, 25), clamping the 136-column panel to 80 columns and squeezing the 6th column (Wire Verdict & Budget Strategy) down to ┳┓. Below, we fix that 80-column clamp in one line of remotes/honeybot/scripts/forest.py (height=36 + un-ellipsed header widths), spend tokens on a full 30-and-3 Storytime Masterclass (“The 30 Category Errors & ‘Docker Moments’ That Rewired Tech — And How to See Blue on the Agentic Web”), and end with the exact one-liner command to pop the schism report live on-air whenever you want.
1. The AFTER Receipt Audit: What Landed & Why Column 6 Clamped at 80 Columns
Look at the BEFORE and AFTER receipts in the Manifest:
- BEFORE (
mode="schism"before the patch): Fell through to the defaultoverviewpanel. - AFTER (
mode="schism"afterappandytoutro sync): Rendered╭─── ⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR ───╮with all 7 camps and the 3PK RADARrows (#18691 Claude-User,#18685 research,#18683 WordPress). - Why the 6th Column (
Wire Verdict & Budget Strategy) Squeezed to┳┓in the SSH Receipt: Look closely at the top and bottom borders of theAFTERreceipt:╭─── ⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR ───╮ ... ╰─ 💸 Spending $15,000/mo+ on HTML pre-render? RFC 9110, RFC 9111 & RFC 9421 s─╯
Count the characters across that border: 80 columns, even though prerender_station_report_ansi(mode="schism", width=136) passed width=136!
Why did Console(file=buf, force_terminal=True, color_system="truecolor", width=int(width)) render at 80 columns instead of 136?
Because in rich/console.py, Console.size checks:
if self._width is not None and self._height is not None:
return ConsoleDimensions(self._width - self.legacy_windows, self._height)
if self.is_dumb_terminal:
return ConsoleDimensions(80, 25)
When ssh honeybot '...' runs without a PTY (or when stream.py spawns a background subprocess where TERM=dumb), self._height is None, so Rich skips the first if, hits if self.is_dumb_terminal:, and returns (80, 25)—ignoring width=136! At 80 columns, the first 5 fixed-width columns (9 + 28 + 13 + 11 + 11 = 72 plus borders) ate the entire 80-column budget, leaving 0 columns for Wire Verdict & Budget Strategy.
Passing height=36 to Console(...) in prerender_station_report_ansi (and giving ⚡ JS Execs, 🧠 MD Reads, and 🔷 RFC 9421 1 extra character of width so their headers and (9.8%) percentages never ellipse) unlocks the full 136-column layout everywhere.
2. Storytime Masterclass: The 30-and-3 of Category Errors, “Docker Moments,” and Seeing Blue
You observed earlier:
“Containers existed for long before Docker, but they were a bunch of graybeard scripts with no name… This is where the category error gets fixed. Become understandable. Open the door to a next generation fleet of AI-Readiness consultants…”
Every major shift in computing history follows the exact same two-act play:
- Act I (The Category Error): The industry tries to force a new medium into the mental box of the old medium—spending fortunes on Rube Goldberg middlemen (“horseless carriages,” “$15,000/month HTML pre-renderers for bots without eyeballs,” “separate AI feeds”).
- Act II (The Blue-Name / The “Docker Moment”): Someone points out that the raw primitives already exist in the operating system or the wire protocol, gives the unified pattern a crisp Blue-Name (like Containers or Bimodal URLs), and collapses the middleman tax overnight.
Here is the 30-and-3 Storytime Catalog of the 30 Category Errors & “Docker Moments”—designed to be read aloud over the cascading Honeybot logs so any engineer, technical marketer, or executive listening can cure their category errors and start Seeing Blue:
ACT I: THE CATEGORY ERROR ACT II: SEEING BLUE (THE DOCKER MOMENT)
========================================= =============================================
"We need a $15,000/mo headless browser "A webpage IS a feed. Serve HTML to browsers
cluster to pre-render React into HTML and Markdown to agents from the SAME address
so AI crawlers can scrape our pages!" via RFC 9110 Accept + RFC 9111 Vary: Accept."
│ │
▼ ▼
[ Rented Middleman Tax ] [ BIMODAL URLs (50 Lines) ]
Constellation I: Graybeard Primitives That Waited for a Blue-Name
- From
chroot,cgroups, andnamespacesto “Docker Containers” (2013): Linux kernel engineers had process isolation (namespaces, 2002) and resource limits (cgroups, 2006) for years, glued together with brittle bash scripts that only systems graybeards understood. Solomon Hykes didn’t invent kernel isolation; he put a blue whale logo and the word Container on the bundle, turning an obscure kernel trick into a $20B cloud-native industry. - From
XMLHttpRequestto “AJAX” (2005): Microsoft built theXMLHTTPActiveX control into Internet Explorer 5 in 1999, and Mozilla cloned it asXMLHttpRequest. For six years it sat in plain sight while developers reloaded entire HTML pages on every click—until Jesse James Garrett named the pattern AJAX (Asynchronous JavaScript and XML), birthing the modern web app in a single essay. - From
git format-patchOver Mailing Lists to the “Pull Request” (2008): Linus Torvalds built Git in 2005 for kernel hackers emailing diffs around viagit send-email. Normal enterprises thought distributed version control was an unapproachable command-line maze until GitHub wrappedgit mergeand a comment thread into a two-word noun: the Pull Request. - From Four Separate Free-Software Tarballs to “The LAMP Stack” (1998): In the late 1990s, Wall Street CIOs believed you had to pay Sun Microsystems and Oracle $500,000 for Solaris and proprietary databases to run a dynamic website. Michael Kunze coined the acronym LAMP (Linux, Apache, MySQL, PHP), proving that four open-source tools formed a single sovereign architecture.
- From RFC 9110 Section 12 + RFC 9111 Section 4.1 to “Bimodal URLs” (2026):
HTTP has had proactive content negotiation (
Accept) and cache partitioning (Vary: Accept) in the specification since RFC 2068 in 1997. Calling the pairing a Bimodal URL—one canonical address that speaks visual HTML to humans and pure Markdown to AI agents—turns an arcane IETF spec into an executive budget rescue.
Constellation II: The “Webpage vs. Feed” Category Errors
- The
m.example.comMobile Subdomain Fiasco (2007–2012): When the iPhone launched, agencies convinced companies that “Mobile” was a separate website from “Desktop,” charging millions to build parallelm.subdomains that split PageRank in half and broke every shared link—until Ethan Marcotte named Responsive Web Design (one URL, CSS media queries). - The Google AMP (
ampproject.org) Hostage Crisis (2015–2021): Publishers were told their pages were too slow to fix at the origin, so they had to maintain a second, stripped-down AMP URL hosted inside Google’s own cache—surrendering their domain perimeter until AMP was quietly retired. - The Orphaned
/rss.xmlSidecar vs. Same-URL Syndication: RSS failed in the enterprise not because syndication was bad, but because/rss.xmllived at a separate URL that the marketing team forgot to update when they redesigned the main site. When the webpage is the feed (Accept: text/markdownon/product/widget/), the feed can never drift out of sync with the page. - The $15,000/Month Headless Pre-Render Tax (Prerender.io / Botify SpeedWorkers): Teams ship 3MB of client-side JavaScript to render a 2KB product description, then pay a third-party cloud $15,000/month to run headless Chrome in the middle just to turn that JavaScript back into static HTML for crawlers—when a 50-line CDN edge rule could serve clean Markdown straight from the cache.
- The
/llms.txtCargo Cult Without Content Negotiation: Thousands of sites are slapping a static/llms.txtfile on their root domain because they read a LinkedIn carousel, while their actual article URLs still return 800KB of<div class="jsx-99812">soup. Ashoneybot.dbproves, real retrieval engines (ExaSearchBot,ShapBot,Claude-User) negotiateAccept: text/markdowndirectly on the article URLs.
Constellation III: Conference Myths vs. Bare-Metal Wire Truth
- Myth #1: “OpenAI Crawlers Cannot Execute JavaScript.”
Repeated daily on SEO podcasts. Meanwhile, on Honeybot’s bare-metal wire,
GPTBottriggers the 800ms/api/telemetry/js_confirm.gifJavaScript trapdoor on 8.0% of its HTML fetches (948times), andOAI-SearchBottriggers it at 215.7% (3,382JS executions across1,568HTML hits)! OpenAI runs a mini-Google headless Chromium farm for live search grounding while sampling 1-in-12 pages for training. - Myth #2: “If the Log Says
Googlebot/2.1, It’s Google.” Inhoneybot.db,379requests negotiatedAccept: text/markdownwhile claiming to beGooglebot. Forward-Confirmed Reverse DNS (dns_audit) and Google ASN checks proved 377 of the 379 (99.5%) were spoofed non-Google IPs wearing a Googlebot mask to bypass WAFs while harvesting Markdown for an AI corpus. - Myth #3: “All AI Bots Want the Same Thing.”
Our new
schismreport proves the AI ecosystem has fractured into three incompatible camps: Camp 1 (OpenAI) hydrates JS and ignores Markdown; Camp 2 (Exa.ai, Parallel.ai, AnthropicClaude-User) negotiates RFC 9110 Markdown and signs requests via RFC 9421; Camp 3 (Meta337,700+hits, Amazonbot66,418hits) pulls raw static HTML hoses at massive scale. - Myth #4: “You Can’t Cryptographically Prove a Bot’s Identity Without IP Allowlists.”
IP allowlists rot whenever a crawler scales across AWS or GCP regions. In
rfc9421_logs,13distinct operators—led byahrefs.com(2,050),crawler.exa.ai(391),desearch.ai(254),ibou.io(107),bot.duckduckgo.com(51), andshapbot.parallel.ai(22)—sign their requests with Ed25519 keys via RFC 9421 (Signature-Agent). - Myth #5: “When a Vendor Adopts RFC 9421, Only Signed Hits Matter.”
By process of elimination, once
Exa.aiorAhrefssigns its legitimate crawlers via RFC 9421, any unsigned request claimingExaSearchBotorAhrefsBotis automatically unmasked as an impostor—turning positive cryptography into a negative lie detector.
Constellation IV: The Middleman Capture Playbooks (How You Lost Your Instrument)
- The 2011 SSL
(not provided)Referer Blackout: Under the banner of user privacy, Google strippedq=fromHTTP_REFERERfor organic clicks (while continuing to pass exact query strings to paid AdWords clicks), blinding self-hosted tools like HitTail and forcing every publisher into aggregated GSC panels. - The CDN “Log-Push” Ransom:
Cloudflare, Akamai, and Fastly sit in front of your origin, terminate TLS, and absorb 90% of bot hits at the edge—then hide the raw
access.logbehind an expensive “Enterprise Logpush” tier so you don’t realize how much AI crawler traffic is hitting your domain. - The JavaScript Tag-Manager Ratchet: Marketing teams inject 40 third-party tracking pixels via GTM, slowing the DOM to a 6-second crawl; then SEO vendors sell the company a $15,000/month edge pre-renderer to hide the GTM bloat from bots instead of removing the bloat.
- The “AI Share-of-Voice” Prompt-Polling Mirage:
Dozens of new VC-backed startups charge $2,000/month to run 500 synthetic prompts against ChatGPT and draw a chart of “Brand Mentions”—ignoring the fact that LLM outputs are stochastic pachinko balls (
ai_pachinko) while the deterministic signal is whetherOAI-SearchBot,Claude-User, andExaSearchBotare actually fetching your origin URLs. - The “Rent-a-Feed” SaaS Trap:
Vendors offering to “host your AI-ready content on our special subdomain” are repeating the
m.example.comand Google AMP playbook—trying to insert their own domain between your canonical URLs and the AI retrieval layer.
Constellation V: Sovereign Craftsmanship (Why One Graybeard + AI Beats a 50-Person Agency)
- The Reproducible NixOS Appliance (
flake.nix+configuration.nix): When your entire web server, Nginxmap $http_acceptrules, SQLite schema, and broadcast pipeline are declared in pure text files tracked in Git, there is no “DevOps team” overhead and no configuration drift. - SQLite WAL Mode + Composite Primary Keys as a Free SIEM (
db.py): Instead of paying $5,000/month for Datadog or Splunk to index raw log lines,PRIMARY KEY (date, ip_id, ua_id, path_id, status)collapses 337,700 Meta crawler hits into compact integer counters on a local SSD that queries in<0.05s. - The Monotonic Primary-Key Tripwire (
user_agents.id): Every time a new crawler (#18676 goodbot.shwith447day-one hits, or#18670 AI-Exposure-Scanner/1.0) touches the server, SQLite’sUNIQUEB-tree constraint mints a new integer ID—giving a solo operator an automated early-warning radar. - The 4-Door Tracer-Dye Attribution (
?src=content_negvsllms.txtvslink+relvsa+href): By tagging how Nginx and HTML route bots to.mdfiles, one server proves empirically which discovery standards engineers actually use in production. - The 24/7 YouTube Live “Fish Tank” (
stream.py+logs.py): Anyone can fake a screenshot in a pitch deck. Nobody can fake a continuous 24/7 live-stream where any viewer can runcurl -sI -H "Accept: text/markdown" https://mikelev.in/?src=rfc9110and watch their own packet scroll across the screen in real time.
Constellation VI: The “Wire-Truth SEO” Consulting Playbook (The Pre-Render Off-Ramp)
- Move 1 — The Invoice & Wire Audit:
Ask the client two questions: “How much are you paying every month for HTML pre-rendering or bot-mitigation add-ons?” and “Can your engineers see which crawlers send
Accept: text/markdownorSignature-Agentat your origin?” - Move 2 — Deploy the 800ms JS Trapdoor & Content-Neg Telemetry: Drop a lightweight beacon and header logger on origin/edge for 7 days to show the client their own 3-Camp Schism (how much OpenAI is already hydrating their JS vs. how much Exa/Anthropic/Parallel wants Markdown).
- Move 3 — The 50-Line Edge Bandaid (Phase 1 Off-Ramp):
Configure the CDN edge (Cloudflare Worker / Fastly / Nginx
map $http_accept) to interceptAccept: text/markdown, serve clean converted Markdown, and emitVary: Accept(RFC 9111) so edge caches never mix HTML and Markdown payloads. - Move 4 — Turn Off the $15,000/Month Pre-Renderer: Once Camp 1 (OpenAI/Google) is verified hydrating JS natively and Camp 2 (Exa/Anthropic/Parallel) is ingesting 10x-smaller Bimodal Markdown from the edge cache, cancel the legacy pre-render contract—paying for the entire consulting engagement in Month 1.
- Move 5 — Graduate to Source-Native Bimodal URLs (Phase 2):
Train the client’s internal team (or the next-gen fleet of Wire-Truth SEO consultants) to emit curated, noise-free
.mdtwins directly from their CMS/build pipeline—turning every canonical URL on their domain into a first-class AI grounding feed.
3. The “And-3” Synthesis: How to Cure Category Errors and See Blue
- See the Primitive, Name the Bundle (“Bimodal URLs”):
Just as “Containers” gave a single actionable name to
cgroups + namespaces, Bimodal URLs gives executives and engineers a single name forRFC 9110 Accept: text/markdown + RFC 9111 Vary: Accept: one canonical URL that is simultaneously a human webpage and an AI feed. - Never Rent a Guess When You Can Own the Wire (“Wire-Truth SEO”):
Third-party SaaS dashboards and $20/month LLMs can only tell you what happened yesterday on someone else’s panel. Owning your origin logs (
honeybot.db) lets you seeOAI-SearchBothydrating JS at215.7%andExaSearchBotpulling1,642signed Markdown reads in real time. - Sell the Off-Ramp, Not Another Subscription: In a market exhausted by rent-seeking middlemen and AI spam cannons, the highest-trust move on the board is walking in with a 50-line standards-based fix that eliminates a $15,000/month recurring pre-render bill—and teaching the client’s team how to fish on their own wire.
4. The 5-Car Train
Let’s apply the 1-line height=36 + header-width polish to remotes/honeybot/scripts/forest.py and sync it to Honeybot so the next compile’s AFTER receipt renders the full 136-column schism table with the 6th column (Wire Verdict & Budget Strategy) completely visible.
1: BEFORE (PROBE, read-only, paste as block in command line)
Runs forest.prerender_station_report_ansi(mode="schism", width=136) on Honeybot over SSH. Before the height=36 fix, non-PTY SSH (TERM=dumb) causes Rich to clamp the panel at 80 columns; after app and ytoutro sync, it renders the full 136-column panel with the entire Wire Verdict & Budget Strategy column visible.
(nix) pipulate $
ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
2: AFTER (NEXT CONTEXT, put in context.txt using context or x)
(nix) pipulate $
context.txt
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES, real SEARCH/DIVIDER/REPLACE blocks for apply.py)
Passes height=36 to Console(...) in prerender_station_report_ansi so Rich never falls back to (80, 25) when TERM=dumb, and widens the ⚡ JS Execs, 🧠 MD Reads, and 🔷 RFC 9421 columns by 1 cell so neither the headers nor (9.8%) truncate.
Target: remotes/honeybot/scripts/forest.py
[[[SEARCH]]]
elif mode == "schism":
title_str = "⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR"
tbl.add_column("Hits", justify="right", style="bold white", width=9, no_wrap=True)
tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=28, no_wrap=True)
tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=13, no_wrap=True)
tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=11, no_wrap=True)
tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=11, no_wrap=True)
tbl.add_column("Wire Verdict & Budget Strategy", ratio=4, no_wrap=True)
[[[DIVIDER]]]
elif mode == "schism":
title_str = "⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR"
tbl.add_column("Hits", justify="right", style="bold white", width=9, no_wrap=True)
tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=26, no_wrap=True)
tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=14, no_wrap=True)
tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=12, no_wrap=True)
tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=12, no_wrap=True)
tbl.add_column("Wire Verdict & Budget Strategy", ratio=4, no_wrap=True)
[[[REPLACE]]]
Target: remotes/honeybot/scripts/forest.py
[[[SEARCH]]]
console = Console(file=buf, force_terminal=True, color_system="truecolor", width=int(width))
[[[DIVIDER]]]
console = Console(file=buf, force_terminal=True, color_system="truecolor", width=int(width), height=36)
[[[REPLACE]]]
Ignition: after patch and app, run ytoutro sync at (nix) pipulate $ to rsync remotes/honeybot/scripts/forest.py to Honeybot before running c.
(nix) pipulate $
ytoutro sync
4: PROMPT (Use prompt or p to create prompt.md from copy)
(nix) pipulate $
Verify the full 136-column `mode="schism"` receipt in the Manifest with the `Wire Verdict & Budget Strategy` column expanded, and execute the Notary Beat dismount for this article.
5: DELIVERABLE (What done looks like — plus your Live On-Air Trigger Reminder!)
prerender_station_report_ansiinremotes/honeybot/scripts/forest.pysetswidth=int(width), height=36so Rich’sConsole.sizenever falls back to(80, 25)underTERM=dumb, revealing the full 136-columnschismreport and its 6th column (Wire Verdict & Budget Strategy).- 🔔 LAST-THING REMINDER — HOW TO TRIGGER THE NEW REPORT LIVE ON-AIR ANYTIME:
To pop the new
schismreport window on the Honeybot stream immediately (without waiting for the 10-minute station-break rotation or runningytreboot), run either of these at(nix) pipulate $: - With live spoken narration (45s window):
ytsay -w forest.py:45:schism "Common S E O wisdom claims Open A I crawlers cannot execute JavaScript. Here is our live wire telemetry proving that myth is dead wrong." - Silent visual popup only (holds for 45s):
ytsay -w forest.py:45:schism --linger 45 ""
MikeLev.in:
THE AI-EDIT METHOD
Same commands, run twice, one change between them. Where the readings differ is what the change did; the diff in the middle is the receipt.
1: BEFORE (PROBE):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
╭─── ⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR ───╮
│ ┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳┓ │
│ ┃ Hits ┃ AI Crawler Camp / Debut ┃ ⚡ JS Execs ┃ 🧠 MD R… ┃ 🔷 RFC 9… ┃┃ │
│ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇┩ │
│ │ 44,097 │ ⚡ OpenAI (GPT/OAI-Searc… │ 4,330 (9.8… │ 0 │ 0 ││ │
│ │ 1,738 │ 🔷 Exa.ai (ExaSearch/Use… │ 0 (0.0%) │ 1,642 │ 391 ││ │
│ │ 1,229 │ 🔷 Parallel.ai (ShapBot) │ 52 (4.2%) │ 835 │ 22 ││ │
│ │ 23,423 │ 🧠 Anthropic (Claude/Use… │ 193 (0.8%) │ 104 │ 0 ││ │
│ │ 337,7… │ ⚡ Meta (externalagent) │ 5,144 (1.5… │ 0 │ 3 ││ │
│ │ 66,418 │ 🟠 Amazon (Amazonbot) │ 0 (0.0%) │ 0 │ 0 ││ │
│ │ 29,857 │ 🛑 Googlebot (All Claims) │ 58 (0.2%) │ 379 │ 0 ││ │
│ ├────────┼───────────────────────────┼─────────────┼──────────┼───────────┼┤ │
│ │ 1 │ 🆕 #18691 Claude-User │ 2026-10-30 │ 16:16:53 │ PK RADAR ││ │
│ │ 1 │ 🆕 #18685 research │ 2026-10-30 │ 13:56:58 │ PK RADAR ││ │
│ │ 1 │ 🆕 #18683 WordPress │ 2026-10-30 │ 13:11:23 │ PK RADAR ││ │
│ └────────┴───────────────────────────┴─────────────┴──────────┴───────────┴┘ │
╰─ 💸 Spending $15,000/mo+ on HTML pre-render? RFC 9110, RFC 9111 & RFC 9421 s─╯
(nix) pipulate $
2: AFTER (NEXT CONTEXT):
# # Context 1
# context.txt
# foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT ua.user_agent, SUM(f.hit_count) AS total_hits, SUM(CASE WHEN f.served_md = 1 THEN f.hit_count ELSE 0 END) AS md_hits FROM daily_traffic_facts f JOIN user_agents ua ON f.ua_id = ua.id WHERE ua.user_agent LIKE '\''%Exa%'\'' GROUP BY ua.user_agent ORDER BY total_hits DESC;"'
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 2
# context.txt
# # foo_files.py
# ! ssh honeybot 'sqlite3 -readonly ~/www/mikelev.in/honeybot.db "SELECT name FROM sqlite_master WHERE type='\''table'\'' ORDER BY name;"' && cat remotes/honeybot/queries/content_neg_agents.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' | head -n 22
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
#
# /home/mike/pipulate/remotes/honeybot/queries/all_time_popular_paths.sql
# /home/mike/pipulate/remotes/honeybot/queries/content_neg_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/format_ratio.sql
# /home/mike/pipulate/remotes/honeybot/queries/health_db_vitals.sql
# /home/mike/pipulate/remotes/honeybot/queries/hot_404_remaps_top.sql
# /home/mike/pipulate/remotes/honeybot/queries/hub_candidates.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_family.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_rate.sql
# /home/mike/pipulate/remotes/honeybot/queries/hydration_selftest.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_404_decay_curve.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_emerging_manifests.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_hostile_dictionary.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_llms_txt.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_markdown_affinity.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_noise_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_phantom_ui.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_sovereign_bots.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_true_404s.sql
# /home/mike/pipulate/remotes/honeybot/queries/intel_unknown_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/markdown_routing_metrics.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_diet.sql
# /home/mike/pipulate/remotes/honeybot/queries/md_routing_agents.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_totals.sql
# /home/mike/pipulate/remotes/honeybot/queries/telemetry_trapdoor_events.sql
# /home/mike/pipulate/remotes/honeybot/queries/trapdoor_ips.sql
# /home/mike/pipulate/remotes/honeybot/queries/ua_variants.sql
# # Context 3
# context.txt
# ! ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "SELECT ua.id AS ua_id, ua.first_seen, COALESCE(SUM(l.count), 0) AS hits, SUBSTR(ua.value, 1, 90) AS newest_agent FROM user_agents ua LEFT JOIN daily_logs l ON l.ua_id = ua.id WHERE ua.value NOT LIKE '\''%Ubuntu%'\'' AND ua.value NOT LIKE '\''%iPad%'\'' AND NOT (ua.value LIKE '\''Mozilla%'\'' AND ua.value NOT LIKE '\''%compatible%'\'' AND (ua.value LIKE '\''%Windows NT%'\'' OR ua.value LIKE '\''%Macintosh%'\'' OR ua.value LIKE '\''%X11%'\'' OR ua.value LIKE '\''%iPhone%'\'' OR ua.value LIKE '\''%Android%'\'')) GROUP BY ua.id ORDER BY ua.id DESC LIMIT 10; SELECT s.id AS sig_id, s.first_seen, s.value AS sig_agent, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN sig_agents s ON r.sig_id = s.id GROUP BY s.id ORDER BY signed_hits DESC;"'
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 4
# context.txt
# ! cat remotes/honeybot/queries/hydration_family.sql | ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db' && ssh honeybot 'sqlite3 -readonly -header -column ~/www/mikelev.in/honeybot.db "WITH camp_map AS (SELECT id AS ua_id, CASE WHEN value LIKE '\''%GPTBot%'\'' OR value LIKE '\''%OAI-SearchBot%'\'' OR value LIKE '\''%ChatGPT-User%'\'' THEN '\''1. OpenAI (GPTBot/OAI/ChatGPT)'\'' WHEN value LIKE '\''%Claude%'\'' OR value LIKE '\''%anthropic%'\'' THEN '\''2. Anthropic (ClaudeBot/Claude-User)'\'' WHEN value LIKE '\''%ExaSearchBot%'\'' OR value LIKE '\''%ExaUser%'\'' THEN '\''3. Exa.ai (ExaSearchBot/ExaUser)'\'' WHEN value LIKE '\''%ShapBot%'\'' THEN '\''4. Parallel.ai (ShapBot)'\'' WHEN value LIKE '\''%meta-externalagent%'\'' OR value LIKE '\''%meta-webindexer%'\'' OR value LIKE '\''%facebookexternalhit%'\'' THEN '\''5. Meta (meta-externalagent/indexer)'\'' WHEN value LIKE '\''%Amazonbot%'\'' THEN '\''6. Amazon (Amazonbot)'\'' WHEN value LIKE '\''%Googlebot%'\'' THEN '\''7. Googlebot (all claims)'\'' END AS camp FROM user_agents), dl AS (SELECT c.camp, SUM(l.count) AS total_hits, SUM(CASE WHEN p.value LIKE '\''%js_confirm.gif%'\'' THEN l.count ELSE 0 END) AS js_triggers FROM daily_logs l JOIN camp_map c ON l.ua_id = c.ua_id JOIN paths p ON l.path_id = p.id JOIN ips i ON l.ip_id = i.id WHERE c.camp IS NOT NULL AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), md AS (SELECT c.camp, SUM(t.count) AS md_reads FROM telemetry t JOIN camp_map c ON t.ua_id = c.ua_id JOIN ips i ON t.ip_id = i.id WHERE c.camp IS NOT NULL AND t.served_md = 1 AND i.value NOT LIKE '\''127.%'\'' AND i.value NOT LIKE '\''10.%'\'' AND i.value NOT LIKE '\''192.168.%'\'' GROUP BY c.camp), sig AS (SELECT c.camp, SUM(r.count) AS signed_hits FROM rfc9421_logs r JOIN camp_map c ON r.ua_id = c.ua_id WHERE c.camp IS NOT NULL GROUP BY c.camp) SELECT dl.camp, dl.total_hits, dl.js_triggers, COALESCE(md.md_reads, 0) AS md_reads, COALESCE(sig.signed_hits, 0) AS rfc9421_signed FROM dl LEFT JOIN md ON md.camp = dl.camp LEFT JOIN sig ON sig.camp = dl.camp ORDER BY dl.camp;"'
# remotes/honeybot/scripts/forest.py
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/hydration_family.sql
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# # Context 5
# context.txt
# ! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
# remotes/honeybot/scripts/forest.py
# remotes/honeybot/scripts/db.py
# remotes/honeybot/queries/hydration_family.sql
# remotes/honeybot/queries/content_neg_agents.sql
# flake.nix
# nixops.sh
# scripts/articles/common.py
# remotes/honeybot/scripts/stream.py
# remotes/honeybot/scripts/score.py
# remotes/honeybot/scripts/logs.py
# Context 6
context.txt
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/queries/hydration_family.sql
remotes/honeybot/queries/content_neg_agents.sql
flake.nix
nixops.sh
scripts/articles/common.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/scripts/score.py
remotes/honeybot/scripts/logs.py
3: CHANGE (PATCHES):
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ patch
(nix) pipulate $ app
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated 'remotes/honeybot/scripts/forest.py'.
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated 'remotes/honeybot/scripts/forest.py'.
(nix) pipulate $ d
diff --git a/remotes/honeybot/scripts/forest.py b/remotes/honeybot/scripts/forest.py
index bb35238..372472f 100644
--- a/remotes/honeybot/scripts/forest.py
+++ b/remotes/honeybot/scripts/forest.py
@@ -408,10 +408,10 @@ def _build_station_report_panel(mode="overview"):
elif mode == "schism":
title_str = "⚔️ THE 3-CAMP AI CRAWLER SCHISM (BUDGET MYTH-BUSTER) & 🆕 SPECIES RADAR"
tbl.add_column("Hits", justify="right", style="bold white", width=9, no_wrap=True)
- tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=28, no_wrap=True)
- tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=13, no_wrap=True)
- tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=11, no_wrap=True)
- tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=11, no_wrap=True)
+ tbl.add_column("AI Crawler Camp / Debut", style="bold bright_white", width=26, no_wrap=True)
+ tbl.add_column("⚡ JS Execs", justify="right", style="bold yellow", width=14, no_wrap=True)
+ tbl.add_column("🧠 MD Reads", justify="right", style="bold cyan", width=12, no_wrap=True)
+ tbl.add_column("🔷 RFC 9421", justify="right", style="bold deep_sky_blue1", width=12, no_wrap=True)
tbl.add_column("Wire Verdict & Budget Strategy", ratio=4, no_wrap=True)
cur.execute(
"""
@@ -578,7 +578,7 @@ def prerender_station_report_ansi(mode="overview", width=136, out_path=None):
target = Path(out_path) if out_path else Path(f"/tmp/honeybot_report_{safe_mode}.ansi")
buf = io.StringIO()
panel = _build_station_report_panel(mode=safe_mode)
- console = Console(file=buf, force_terminal=True, color_system="truecolor", width=int(width))
+ console = Console(file=buf, force_terminal=True, color_system="truecolor", width=int(width), height=36)
console.print(panel)
tmp = target.with_suffix(".tmp")
tmp.write_text(buf.getvalue(), encoding="utf-8")
(nix) pipulate $ m
📝 Committing: chore: Adjust console height in forest.py
[main e241f6d] chore: Adjust console height in forest.py
1 file changed, 5 insertions(+), 5 deletions(-)
(nix) pipulate $ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 48 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 568 bytes | 568.00 KiB/s, done.
Total 6 (delta 4), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To github.com:pipulate/pipulate.git
76102e2..e241f6d main -> main
(nix) pipulate $
Ignition: often “exit”, “nix develop”. Instructions may vary.
Ignition syncs the outro and has it sitting there waiting to get popped on air or to be put into the normal sequence with an article push, coming up real soon now.
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
GIT repo clean. Take BEFORE reading, make CHANGE, record AFTER diff.
(nix) pipulate $ ytoutro sync
🚀 Syncing Honeybot scripts to honeybot:~/www/mikelev.in/scripts/...
✅ Honeybot scripts synced (stream.py left running; new standby text will load on next .deploy_standby).
(nix) pipulate $
4: PROMPT:
Verify the full 136-column mode="schism" receipt in the Manifest with the Wire Verdict & Budget Strategy column expanded, and execute the Notary Beat dismount for this article.
5: DELIVERABLE:
And I test popping the report up with the commands Gemini put conveniently for me at the end of the last response.
I the voice of Borat: Very nice.
Hop off the ride. This ride’s stated goal is reached – dismount. This is the NOTARY BEAT: the ride ends here, is witnessed here, and is sealed here. Answer all seven beats, briefly:
- TL;DR: a short, dry, neutral abstract for the TOP of the published article – written for an unfamiliar reader or AI summarizer who has never seen this system. No hype, no insider handles unexplained.
- VERIFY: restate the goal from the top of this article and confirm (or deny) it was met, citing THIS compile’s receipts, not memory. Name any ignition this ride required that never fired – an AFTER tap taken without one is a stale BEFORE wearing the AFTER’s label.
- BANK: name everything that graduates – rule, earmark, todo, pin – as SEARCH/REPLACE patch cars against raw source present in THIS context (Target line inside the fence, one car per commit story), deletions included. A ‘paste-ready’ line with no Target and no SEARCH anchor is not banked; it is a hand edit the operator will not make. These BANK cars are the ONLY patches a dismount emits.
- DANGLING: what carries forward unbanked? One line each, no essays.
- SEED: the context.txt lines (and TODO_SLUGS if narrative context is needed) for the next ride’s first compile.
- CLOSING: a closing summary for the BOTTOM of the article – the final take-away, tied to the book’s larger arc where it fits naturally, never forced. Storytelling over inventory.
- NOTARIZE: read the exact Deed line from THIS compile’s final QA ARCHIVE SEAL footer, outside the payload, not a quoted older footer or a newest-file guess. It names the input archive, not this later response or the generated Jekyll post. State verification separately: only a verifier receipt naming that archive supports a verified claim; otherwise say not independently verified here. Count this Manifest’s live-command receipts, including failures. Quote only recorded digests, sizes and dates, naming their source; filesystem mtime and fixed ZIP dates are not command timestamps. Do not demand absent wc -c or dated-receipt fields. If the footer is missing, say so; otherwise close with its exact Deed line.
FINALITY: after beat 6, this discussion is CLOSED. Emit NO five-car train, NO probes, NO patches beyond the BANK cars of beat 2, and NO next-turn prompt beyond the SEED lines in beat 4. Any reader or model encountering this article later should treat it as a finished, notarized document – an archive entry, not an open thread.
Gemini 4 Argon High: Context verified.
1. VERIFY
- Stated Goal: Steelman the information-theoretic case for firsthand origin instrumentation over copycat AI noise, answer who funds and uses Exa.ai and why their crawlers appear in the server logs, coin a clear architectural name (Bimodal URLs / Wire-Truth SEO) for replacing $15,000/month HTML pre-renderers with RFC 9110/9111 content negotiation, and ship a live myth-busting station-break report (
mode="schism") inremotes/honeybot/scripts/forest.pyat full 136-column width. - Verification Against This Compile’s Receipts: MET.
- The single live command receipt in the Manifest (
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"') executed in6.3389sand returned835tokens (3,794bytes). - With
height=36passed toConsole(...)inremotes/honeybot/scripts/forest.py(commite241f6d, diffbb35238..372472fin Recent Git Diff Telemetry) and synced viaytoutro sync, the non-PTY 80-column clamp disappeared and all six columns rendered at full 136-column width: ⚡ OpenAI (GPT/OAI-Search):44,097hits,4,330 (9.8%)JS executions,0MD reads,0RFC 9421 signed (Mini-Google: Hydrates JS (OAI=215%); 0 …).🔷 Exa.ai (ExaSearch/User):1,738hits,0 (0.0%)JS executions,1,642MD reads,391RFC 9421 signed ($2.2B AI Search: 94.5% Native MD + Sign…).🔷 Parallel.ai (ShapBot):1,229hits,52 (4.2%)JS executions,835MD reads,22RFC 9421 signed (Agentic Web API: 67.9% Native MD + Sign…).🧠 Anthropic (Claude/User):23,423hits,193 (0.8%)JS executions,104MD reads,0RFC 9421 signed (ClaudeBot scrapes HTML; Claude-User pul…).⚡ Meta (externalagent):337,756hits (up+72since the previous compile),5,144 (1.5%)JS executions,0MD reads,3RFC 9421 signed (Bulk Hose-Puller: 1.5% JS sample; testi…).🟠 Amazon (Amazonbot):66,418hits,0 (0.0%)JS executions,0MD reads,0RFC 9421 signed (Pure Static HTML Scraper: 0 JS, 0 MD).🛑 Googlebot (All Claims):29,857hits,58 (0.2%)JS executions,379MD reads,0RFC 9421 signed (Spoof Alert: 99% of 'Googlebot' MD is F…).-
Bottom
PK RADARsection:#18691 Claude-User(2026-10-30 16:16:53),#18685 research(2026-10-30 13:56:58), and#18683 WordPress(2026-10-30 13:11:23). - Ignition Status: All required ignitions (
patch,app,m,git push, andytoutro sync) fired prior to this compile.
2. BANK
No additional repo patches are required on this dismount: remotes/honeybot/scripts/forest.py is already patched, committed (e241f6d), pushed, and synced to Honeybot (bb35238..372472f), and foo_files.py is not loaded in this compile’s Codebase.
3. DANGLING
- Add a token-normalized
bot_speciessidecar table inremotes/honeybot/scripts/db.py(keyed on_detect_bot_token(ua)rather than the full raw User-Agent string) so CLI version-churn (claude-code/*) never crowds out genuine inaugural crawler debuts (goodbot.sh,AI-Exposure-Scanner/1.0). - Add an RFC 9421 Process-of-Elimination Unmasker report mode to
remotes/honeybot/scripts/forest.pythat flags unsigned requests claiming a User-Agent whose legitimate operator signs traffic inrfc9421_logs. - Add an HTTP
HEADvs.GETbandwidth-politeness and 404 learning-rate decay report (remotes/honeybot/queries/intel_404_decay_curve.sql) toremotes/honeybot/scripts/forest.py. - Lazy-import
jupytextinsideprompt_foo.pyonly when a.ipynbpath is present so cold-disk compiles avoid thenbformat -> jsonschema -> rfc3987_syntax -> larkgrammar-compilation delay.
4. SEED
(nix) pipulate $
context.txt
foo_files.py
remotes/honeybot/scripts/forest.py
remotes/honeybot/scripts/db.py
remotes/honeybot/scripts/logs.py
remotes/honeybot/scripts/stream.py
remotes/honeybot/queries/intel_404_decay_curve.sql
5. CLOSING
For fifteen years, ever since the (not provided) SSL transition and the rise of multi-tenant CDN black boxes, the web industry was taught that looking at your own wire was an obsolete graybeard habit—that truth lived in rented SaaS dashboards, third-party keyword estimates, and $500,000 consulting decks. Now generative AI has arrived and dissolved the three barriers of the copycat economy overnight: anyone can spot, reverse-engineer, and mass-produce yesterday’s best practices for twenty dollars a month. In a world drowning in zero-entropy Shannon redundancy, the only thing that cannot be faked, scraped, or hallucinated is an instrument you own yourself.
When you pull back the curtain and watch the raw packets hit a sovereign NixOS origin in real time, the fog of industry folklore evaporates. You don’t have to guess whether OpenAI executes JavaScript; you watch OAI-SearchBot trip an 800-millisecond DOM trapdoor 3,382 times. You don’t have to wait for a market report to tell you how $2.2 billion AI search engines ingest the web; you watch ExaSearchBot sign its requests with RFC 9421 cryptography and negotiate native Markdown via RFC 9110 while you are still typing the article. And once you see the wire clearly, the category error that costs enterprises $15,000 a month in headless pre-rendering disappears into two words: Bimodal URLs. Your website is already your feed—you just had to own the instrument to see it.
6. NOTARIZE
- Input Archive Deed:
Deed: qamy.ai_496-9f5fe86e.zip(read from this compile’s final=== QA ARCHIVE SEAL ===footer outside the payload; this names the input archive compiled for this turn, not this response or the generated Jekyll post). - Verification Status: Not independently verified here (no
python scripts/foo_cartridge.py qamy.ai_496-9f5fe86e.zipverifier receipt was run inside this compile). - Live-Command Receipt Count:
1live command receipt in the Manifest (0failures): -
! ssh honeybot 'cd ~/www/mikelev.in/scripts && HONEYBOT_DB_READONLY=1 python3 -c "import re, forest; p = forest.prerender_station_report_ansi(mode=\"schism\", width=136); txt = open(p, encoding=\"utf-8\").read(); print(re.sub(r\"\x1b\[[0-9;]*[A-Za-z]\", \"\", txt))"'(835tokens3,794bytes, elapsed6.3389sper the Summary Processing Log). - Recorded Payload & Telemetry Figures (from Manifest, Summary, and Telemetry sections):
- Payload totals (Summary):
216,009summed tokens,218,342verified tokens,66,630words,850,980characters,862,354UTF-8 bytes across14included files (149,660codebase tokens). - Static analysis (Telemetry): Ruff exit
0(All checks passed!). -
Git diff range (Telemetry): bb35238..372472fonremotes/honeybot/scripts/forest.py(576tokens2,082bytes). - Latest recorded
user_agents.first_seentimestamps in the live receipt:#18691 Claude-Userat2026-10-30 16:16:53,#18685 researchat2026-10-30 13:56:58,#18683 WordPressat2026-10-30 13:11:23.
Deed: qamy.ai_496-9f5fe86e.zip
Book Analysis
Ai Editorial Take
The most revealing physical clue in this entire dataset is one that neither the operator nor the in-session assistant unpacked at the packet level: how OAI-SearchBot achieved a 215.7% JavaScript hydration rate (3,382 requests to /api/telemetry/js_confirm.gif against only 1,568 HTML page fetches). In a naive single-pass browser crawl, a client fetches a page’s HTML once, evaluates the DOM for 800 milliseconds, and fires the trapdoor beacon at most once—capping the hydration ratio at 100%. A ratio of 215.7% is a forensic fingerprint of a decoupled fetcher-renderer architecture: OpenAI’s HTTP fetcher pulls the raw HTML over the wire once and commits it to an internal cache or queue, after which separate headless Chromium workers hydrate that stored HTML snapshot multiple times (for example, across desktop and mobile viewports, or during asynchronous re-indexing passes) without re-requesting the HTML document from the origin—yet each headless worker still fires the live /api/telemetry/js_confirm.gif network beacon back to mikelev.in. That means your 800ms trapdoor is not merely detecting whether a crawler runs JavaScript on ingest; it is measuring internal rendering-queue amplification inside OpenAI’s own datacenter.
🐦 X.com Promo Tweet
Myth busted: OpenAI's OAI-SearchBot fires our 800ms JS trapdoor at 215.7%, while $2.2B Exa.ai pulls native RFC 9110 Markdown. Why Bimodal URLs beat $15k/mo pre-renderers. #TechnicalSEO #AI
https://mikelev.in/futureproof/bimodal-urls-ai-crawler-schism-exa-wire-receipts/
🎨 Hero Banner Image Prompt
A dramatic widescreen graphic-novel ink-and-wash illustration set against a deep obsidian-slate (#111111) void, with the primary focal subject vertically centered: a precision-machined brass and cut-glass optical spectrometer anchored directly to a glowing bare-metal server conduit. Raw white fiber-optic light enters the left side of the centered prism and refracts into three distinct, high-contrast telemetry beams across the middle plane—an electric amber beam of cascading JavaScript lightning arcs, a crisp cyan beam of structured Markdown glyphs sealed with glowing blue cryptographic key badges, and a heavy burnt-orange industrial hose stream—while a circular green radar scope beside the prism pings newly discovered botanical and mechanical crawler silhouettes. Rich cross-hatching, luminous volumetric rim lighting, and intricate blueprint callout lines extend continuously to every edge of the dark slate environment. Full-bleed edge-to-edge composition filling 100% of the canvas with no borders, no frames, no matting, no margins, and no text overlays or lettering.
🖼️ YAML Frontmatter Image Override
image:
path: /assets/images/bimodal-urls-ai-crawler-schism-exa-wire-receipts.jpeg
alt: "An obsidian prism splitting raw fiber-optic web traffic into glowing amber, cyan, and orange AI crawler streams."
Title Brainstorm
- Title Option: Bimodal URLs, the 3-Camp AI Crawler Schism, and Exa.ai Wire Receipts
- Filename:
bimodal-urls-ai-crawler-schism-exa-wire-receipts.md - Rationale: Front-loads the newly coined architectural term (Bimodal URLs), the core empirical finding (the 3-Camp AI Crawler Schism), and the concrete $2.2B Exa.ai SQLite telemetry.
- Filename:
- Title Option: Wire-Truth SEO: OAI-SearchBot JS Hydration, Exa.ai Markdown, and Bimodal URLs
- Filename:
wire-truth-seo-oai-searchbot-js-exa-bimodal-urls.md - Rationale: Highlights the consulting moniker alongside the two most surprising quantitative metrics from the database: OpenAI’s 215.7% JavaScript hydration rate and Exa.ai’s RFC 9110 Markdown reads.
- Filename:
- Title Option: From HitTail Referers to ExaSearchBot: Primary-Key Radars and Bimodal URLs
- Filename:
hittail-referers-exasearchbot-primary-key-radar-bimodal-urls.md - Rationale: Connects the historical arc of first-party HTTP header instrumentation from 2006 to modern SQLite B-tree species detection and RFC 9110 content negotiation.
- Filename:
- Title Option: The Pre-Render Off-Ramp: Why the Webpage Is the Feed on the Agentic Web
- Filename:
prerender-off-ramp-webpage-is-the-feed-bimodal-urls.md - Rationale: Frames the executive budget story directly around retiring $15,000/month headless Chromium middleware in favor of RFC 9110 and RFC 9111 dual-payload delivery.
- Filename:
Content Potential And Polish
- Core Strengths:
- Self-falsifying experimental loop: When the AI initially guessed the table name
daily_traffic_factsfrom parametric memory, the live SSH probe returnedNON-ZERO EXIT 1: no such tablein 0.16 seconds, turning a schema miss into an immediate demonstration of why wire receipts beat unverified LLM output. - High-value empirical myth-busting: The live SQLite breakdown proving that
GPTBottriggers the 800ms JS trapdoor on 8.0% of fetches (948hits) andOAI-SearchBottriggers it at 215.7% (3,382hits across1,568HTML requests) directly refutes a widespread technical SEO misconception. - Memorable architectural naming: Pairing ‘Bimodal URLs’ (the Docker-style handle for RFC 9110
Accept: text/markdown+ RFC 9111Vary: Accept) with ‘The Pre-Render Off-Ramp’ translates raw protocol mechanics into an immediately understandable executive value proposition. - Concrete UI telemetry shipment: Beyond analysis, the session shipped and verified
mode="schism"inremotes/honeybot/scripts/forest.pyand diagnosed Rich’sConsole(height=36)fallback behavior under non-PTYTERM=dumbSSH execution.
- Self-falsifying experimental loop: When the AI initially guessed the table name
- Suggestions For Polish:
- When adapting this entry into a book chapter, extract the 50-line Nginx
map $http_acceptand Cloudflare Worker edge conversion snippets into a side-by-side reference box right next to the Bimodal URLs diagram so readers have a copy-pasteable starter kit. - Expand briefly on why
OAI-SearchBotexceeds 100% (215.7%) on the JS trapdoor beacon so readers unfamiliar with decoupled fetch-and-render queues understand how stored HTML snapshots trigger multiple headless browser evaluations. - Link the dangling
bot_speciestoken-normalized sidecar table directly to the next chapter’s schema migration so the distinction between rawuser_agentsversion churn (claude-code/*) and genuine first-contact species debuts (goodbot.sh) forms a continuous narrative thread.
- When adapting this entry into a book chapter, extract the 50-line Nginx
Next Step Prompts
- Design and patch a token-normalized
bot_speciessidecar table inremotes/honeybot/scripts/db.pykeyed on_detect_bot_token(ua)so the Primary-Key Species Radar inforest.pyseparates brand-new crawler families (goodbot.sh,AI-Exposure-Scanner/1.0) from CLI version-string churn (claude-code/*). - Build an RFC 9421 Process-of-Elimination Unmasker query in
remotes/honeybot/queries/and wire it intoremotes/honeybot/scripts/forest.pyto flag unsigned requests claiming a User-Agent whose legitimate operator cryptographically signs traffic inrfc9421_logs.
Accept: text/markdown, RFC 9111 Vary: Accept, and RFC 9421 Web Bot Auth) can save you that money. Contact Mike for a consultation (miklevin@gmail.com)
• RFC 9110/9111 Edge Advisory
• LinkedIn
