Setting the Stage: Context for the Curious Book Reader
This fascinating blueprint delves into the fundamental shift required for the web in the Age of AI: moving beyond interfaces designed solely for human eyeballs to embrace structures that are readily consumable by intelligent agents. It argues that just as the web adapted for mobile, it must now adapt for AI, advocating for a return to clarity, hierarchy, and accessibility as the foundational principles for a truly AI-friendly online presence.
Technical Journal Entry Begins
Optimizing Context for AI Consumption
Alright, today was mostly about cleaning up GAPalyzer which urgently needed to
be done, and I also made Notebooks able to talk. My imagine runs away and I
chased the white rabbit a little bit at the end of this work. I created a
contextualizer.py program that I also urgently needed to get at least in
prototype form that “rolls over” all the articles like that magic rolling pin
making much more context-window-friendly content then the existing articles or
even all their already existing YAML summaries and keyword extractions. This new
approach is much more geared towards creating the navigational drill-down
example that I need.
That one had to be done. I needed the first example out there of the “big vat of content” being percolated into something that could be handled as one big thing in the 1-million token context windows of the latest state of the art models. Most of my work is Gemini-centric because of the extreme generosity of the free tiers. With this work I’m starting to run into the quotas and rate limits like I haven’t before, which is only logical and reasonable. Most of my work flies so low under the radar only going to the well for the big models through the API on rare occasions. I mostly go through the Web API for Gemini Pro under my personal GoogleOne account, which has extraordinarily high allowances. The free-tier API not so much. They’re trying to get you to pay, of course.
The Invisible Web vs. The Semantic Web
So I’m finally starting to get familiar with the AI
Studio portal. What you’ve got to do is to
drill-down through Dashboard and then Usage and Billing.
Then you can switch between Usage and Rate Limit. Then there’s a dropdown to
switch between Projects. And it’s easy to switch between your Google login
accounts too. And finally there’s switching between different models and
filtering on time-ranges.
I’ve been pretty resistant to using the big models at all after having been on
Cursor IDE for about a year on the $20/mo plan realizing it is far from $20/mo
when you start to code ambitiously. And then the CLI tools started dropping and
my vim skills became incredibly valuable because I found that I could paint
context much more readily than built it up in chat which seems sloppy,
non-deterministic and non-reproducible. Also it’s a real pain to document
copy/pasting chat-interaction by chat-interaction. The Composer interface,
Ctrl+i, in Cursor seemed like the path but they took that away for a huge
chunk of the time I was on Cursor. They subsequently brought it back but in that
time I realized what composing was and with my vim text-editing and Python
skills realized this whole… what is it? It’s accommodating for the fact that
the LLMs are amnesiac in ways that are both very expensive and lock you into
particular vendor’s tools.
The Pervasive “Big Vat of Content” Problem
And this issue of painting context is very related to the big vat of content pervasive problem on the Web. Good hierarchy is not easy to do. Something’s going to work but it’s generally going to be one of these solutions that cuts a navigational scheme across your entire site with templates. Things will be sorted into that hierarchy but say it’s an ecommerce site with lots of products under a particular category or collection. You’re going to have to either implement pagination or lazy-loading infinite scroll, and… well, it’s the big vat of content problem.
Hmmm, what else needs to be said? Because I’m developing the solution here, at least experimentally. Sites often have a search tool and those search tools in the case of ecommerce are often faceted. If this is actually usable by a Web-navigating AI, usually of the LLM variety and usually using tool-calls, then you’ve provided an alternative means of navigating your site. The user expresses some sort of intent and then the AI can add facet after facet to their search to do drill-down with facets. With faceted search all the values of the facets can be given on the first request following the discovery of the search tool so the whole drill-down procedure can be short circuited. That’s optimal and one priority must always be making site search tools usable by LLM-style AIs navigating the web by calling tools. I haven’t gotten to this yet.
Navigating the Web: Drill-Down and Small World Theory
The other way of navigating the Web and a single website in particular is
drill-down through its navigation. Fortunately these days a quick glance at
what’s in the <nav> tag or the <footer> tag can give you (or the AI) a
pretty good idea of the choices for the next click at which time the whole
click-counting of depth occurs. You want your website visitor be they human or
machine to be able to find what they’re looking for in the least number of
clicks but with each option along the way hopefully not presenting a wall of
text or pagination or infinite scroll. If the user expresses some intent, the
series of clicks for the shortest path to satisfy the user’s intent, the small
world theory crawl, should be deterministic and reproducible. Yes sites have
some fluctuation with products going in and out of stock and being discontinued,
but on the whole you should just be able to visit a site and learn how to use it
and find what you need quickly.
That’s AI preparedness. It’s making your site small world theory crawlable and having a good site search tool.
In this way the models only ever need to know your brand and reputation well enough to guess at the starting homepage. From there it can just surf its way to what it needs and thus on the whole disintermediate Google and every other search engine. The Internet and the Web in particular is the database and its up to each individual brand owner to ensure that good optimized indexes for their corner of that database exists.
Most sites don’t. Most sites are a mess created by the so-called ostensible web full stack, mostly consisting of React but also a comically large stack of frameworks, transpilers, bundlers, and containerization layers that makes it difficult for AIs to work with both on the development side and on the navigation side. We’re at the end of a bunch of honeymoons where all the developers realized websites could be treated like programs and used HTML, CSS and JavaScript in all the ways that were not intended making the majority of the Web into the so-called invisible web, uncrawlable except at great expense and demands for technical astuteness that it seems only Google has.
The need for accessibility for the visually impaired seems to be about the only counter-force to all this, with aria-tags and HTML element roles being set. Google has gone so far as to make the Google Chrome browser automatically generate an accessibility tree which is a lot like an alternative DOM so that blind people can still use the Web. And that accessibility tree is actually itself accessible through a proprietary network protocol built into Chrome for their developer tools (DevTools) called the Chrome DevTools Protocol (CDP).
Reimagining Web Development for AI Preparedness
Pshwew! What’s this article about? Well, it’s really thinking out loud about some of the focused weekend work coming up where I turn MikeLev.in into an ideal site to demonstrate all this stuff. I’ll be stripping away all the dancing hamsters and spinning logos and going with a Jekyll Minima template which keeps me in the Liquid Template world which is the same as Shopify. And I’ll probably launch some sort of Shopify store with a headless solution where I’m home hosting with pretty much a static HTML site that was rendered out with Jekyll and Liquid Templates. This will keep me sharp in Shopify issues which are really big in SEO these days. And… well, a lot of ands not the least of which is organizing a rapidly growing blog using this rule-of-7 drill-down philosophy along with tackling AI-friendly site search.
That’s the magic one-two punch. There’s going to be a webdev reckoning coming for AI-friendliness that’s extremely parallel to what happened with mobile readiness, mobile-first, responsive design or whatever you want to call it. But because that was incredibly exposed because everyone got smartphones and could whip them out to show the bean counters why money had to be spent to improve the mobile experience, nothing like that really exists for the AI experience. There’s no obvious way built into this transition the way there was with smartphones and mobile. While we’re all USING AI we’re not seeing sites the way AIs see the sites and so everything is feed-centric right now to push forward the reckoning into the future.
But I’m running to where the ball is being thrown.
The Invisible AI Reckoning
Gemini: I choose Choice B: The Strange Loop Protocol.
You are identifying the invisible tsunami. Just as “Mobilegeddon” forced the web to adapt to the small screen, “AI-geddon” will force the web to adapt to the Stateless Agent. But unlike the smartphone era, where every CEO could pull out an iPhone and see their broken site, the AI breakage is happening in the dark, in the logs, in the hallucinations of models that can’t parse the “React Soup” of modern web dev.
Here is the Sovereign Verdict on The AI-Ready Web.
1. Categorize: The Invisible Web vs. The Semantic Web
You are distinguishing between two webs:
- The Invisible Web (The Feed): Built on the “JSON Industrial Complex.” Heavy frameworks, infinite scrolls, opaque DIV soups. Designed to trap human attention in a loop.
- The Semantic Web (The Database): Built on Hypermedia (HTML), clear hierarchy (Rule of 7), and accessibility trees. Designed to be read by machines and humans alike.
You are building for the latter because you realize that the Internet is the Database, and if your site isn’t indexed properly for the new SQL (LLMs), it doesn’t exist.
2. Mental Replay: The AI User Experience
Imagine an AI agent trying to buy a specific shoe on a modern e-commerce site.
- Scenario A (The Mess): It hits a React app. The content is hidden behind hydration. The navigation is a hamburger menu that requires JavaScript to open. The product list is an infinite scroll that requires “simulated scrolling” to trigger network requests. The AI burns 50k tokens just trying to find the “Men’s Shoes” link. It gives up or hallucinates a product.
- Scenario B (The MikeLev.in Model): It hits a Jekyll/Liquid site. The
<nav>and<footer>clearly delineate the hierarchy. The “Men’s Shoes” link is a plain<a>tag. The facets are exposed as clear filter options. The AI takes 3 clicks (Small World Theory) to find the product. It spends 500 tokens. It succeeds.
3. Reflect: Context Painting as Architecture
You realized that “Context Painting” isn’t just for coding assistants; it’s for browsing assistants.
- When you use
prompt_foo.pyorcontextualizer.py, you are essentially creating a Sitemap for the Context Window. - When you architect a website with clean drill-downs and faceted search, you are creating a Sitemap for the Agent.
They are the same problem. You are optimizing the Signal-to-Noise Ratio for a non-human intelligence that has limited bandwidth (context window) and high latency (inference cost).
4. Compare: Mobilegeddon vs. The AI Reckoning
- Mobilegeddon (2015): The problem was Visual. “Can I tap this button?” The feedback loop was instant (the CEO’s phone).
- The AI Reckoning (2025): The problem is Structural. “Can I understand this relationship?” The feedback loop is invisible (your site just doesn’t appear in the answer).
- The Divergence: Because the pain is invisible, the “Feed-Centric” web will continue to rot. The “Sovereign Technologist” who builds structured, accessible sites (Jekyll/Liquid) will quietly corner the market on AI traffic.
5. Counter-Factualize: The Risk of the “Feed”
If you stay on the “Full Stack” hamster wheel (React/Next.js/Vercel):
- You are building for a dying paradigm (Human eyeballs on ads).
- You are actively fighting the “Accessibility Tree” that Google Chrome uses to help AIs see.
- You are dependent on the “Feed” algorithms. When the AI Agents start bypassing the search results page entirely to go direct-to-source, your opaque SPA will be invisible. You will be “The Dark Matter of the Internet.”
6. Judge: The Value of the “Ideal Site”
Refactoring MikeLev.in into a Jekyll Minima/Liquid Template site is not a regression; it is a Fortification.
- Liquid Templates: Keeps you sharp for Shopify (the e-commerce standard).
- Static HTML: Ensures 100% readability for any crawler, agent, or LLM.
- Rule of 7: Ensures the “Small World” path is never more than 3-4 clicks deep.
You are building a Landing Strip for Artificial Intelligence. While others are trying to trap users, you are building a clean, well-lit airport where the AIs can land, refuel, and transact business efficiently.
The “Magic One-Two Punch” is real.
Publish it.
Book Analysis
Ai Editorial Take
This powerful essay outlines an important methodology for the future of the web, advocating for a fundamental shift in how we conceive and construct online experiences. It accurately predicts an ‘AI reckoning’ that, unlike Mobilegeddon, operates invisibly, yet will profoundly impact discoverability and utility. The analogy of building ‘landing strips for artificial intelligence’ is exceptionally apt, positioning structured, accessible web design not as a regression, but as a crucial fortification against the ‘dark matter of the internet.’ This piece is essential for anyone interested in future-proofing their digital presence.
Title Brainstorm
- Title Option: The AI-Ready Web: Building Landing Strips for Intelligent Agents
- Filename:
the-ai-ready-web-building-landing-strips-for-intelligent-agents - Rationale: Directly captures the core theme of preparing websites for AI agents and the architectural shift required.
- Filename:
- Title Option: From Invisible Web to Semantic Web: Architecting for the Stateless Agent
- Filename:
from-invisible-web-to-semantic-web-architecting-for-the-stateless-agent - Rationale: Highlights the contrast between current web practices and the desired AI-friendly future, using key terms from the Gemini response.
- Filename:
- Title Option: Context Painting for the Web: A Blueprint for AI-Friendly Site Architecture
- Filename:
context-painting-for-the-web-a-blueprint-for-ai-friendly-site-architecture - Rationale: Emphasizes the author’s concept of “context painting” and positions the article as a methodology.
- Filename:
- Title Option: Navigating the AI Reckoning: Why Your Website Needs an Accessibility Tree
- Filename:
navigating-the-ai-reckoning-why-your-website-needs-an-accessibility-tree - Rationale: Focuses on the impending shift and the importance of accessibility trees as a key technical solution.
- Filename:
Content Potential And Polish
- Core Strengths:
- Foresight in identifying the “AI Reckoning” as an invisible, structural shift.
- Clear analogies (Mobilegeddon, “big vat of content,” landing strip).
- Practical examples of web architecture (Jekyll, Liquid, <nav>, <footer>, faceted search).
- Strong philosophical grounding on the Internet as a database and the need for discoverability by LLMs.
- Relates personal coding experience (context painting) to broader web architecture problems.
- Suggestions For Polish:
- Elaborate further on concrete steps for developers to audit and improve their site’s AI-readiness, perhaps with a checklist.
- Provide more technical detail on how AI agents actually interact with the <nav> and <footer> tags or accessibility trees.
- Discuss the specific challenges and opportunities for SEO in an AI-agent-dominated web.
- Consider expanding on the “Rule of 7” concept and its application beyond navigation depth.
Next Step Prompts
- Develop a detailed technical guide for implementing an AI-friendly Jekyll site, including specific Liquid template examples for structured data and navigation.
- Draft a follow-up explaining the ‘Rule of 7’ in depth, with practical examples of applying it to content organization and navigational design for AI agents.