The Magic Rolling Pin: Structuring Websites for AI Navigation

By Mike Levin

Setting the Stage: Context for the Curious Book Reader

In this technical journal entry, I explore a core challenge in the age of AI agents: website navigability. I introduce a concept I’m calling the ‘Magic Rolling Pin,’ guided by a ‘Rule of 7,’ to combat the prevalent ‘Deathstar’ problem where flat, heavily cross-linked navigation structures confuse both humans and bots. The piece details my evolving thoughts on using this principle to restructure information, not just in software repositories but for entire websites, making them logically traversable. I document my conversation with an AI, critically examining the assumptions and potential of this approach, including using Hierarchical Agglomerative Clustering to create stable, SEO-friendly navigational hubs. This is a blueprint for a future where websites are designed for efficient AI interaction.

---

The Magic Rolling Pin concept is about load-balancing the choices in a hierarchical menu so that as human or AI drills-down on menu choices they rarely encounter fewer than 5 nor more than 9 choices. This is not a hard and fast rule, and 7 is the ideal “sweet spot”. The goal is to make any resource on a website or in a git repo no more than 7 clicks or cd’s away.

The rules are a bit different for a git repo than a website because git repos are the actual locations of files in a filesystem and certain folders like for plugins will necessarily be over-stuffed out of necessity, while a website there is a user experience hierarchy that is separate from the implied hierarchy by the URL paths. When you think about it the single template file for a product detail page on an e-commerce site would all be planted under one directory, normally /products/ but the way you surf to it on the navigation menu would have it categorized, sub-categorized and so on.

If webpages used their URLs to determine their location in a hierarchy, many sites would turn out as one big flat list of products. This is not always true as some websites have made attempts to make the paths in the URL reflect the desired drill-down menu hierarchy but this is the exception and invites lots of technical liability and often conflicting and overlapping rules of where something belongs. Superimposing this hierarchy with navigation on the other hand allows flexibility, letting the same URL live multiple places in the hierarchy without causing technical duplication — though you might need some fancy CSS/JavaScript to make breadcrumb trails reflect the click-path the user experienced and not some primary default path (if that’s important).

So the Rule of 7 gets used on hard file-location in the case of git repos with more exceptions to the rule because of plugin directories and the realities of things that have to go in the repo root such as .git, pyproject.toml and the like. In the case of websites, the “location” in the site is a function of what you had to click to find it.

The Core Problem: The ‘Deathstar’ of Modern Web Design

This leads to the Deathstar problem. We have in our minds this concept of a site hierarchy being something at the top of a tree with multiple choices underneath that and multiple choices underneath that and so on down so that a tree looks like a tree. The truth of the matter is that most websites are created with navigation that cuts across the entire site. In this model instead of a tree-like hierarchy that can confer some meaning, there’s some 100 or so links comprising usually all the “top-level” and “second-level” links as if they were top-level and cross-links every one with every other creating a sort of dense ring in a force graph of the link network.

Once that first densely cross-linked rink is established as the sort of heart of the site, there is still plenty of opportunities for topical grouping from the well-linked tertiary pages, which are now really looking like secondary pages because the 2nd level of navigation has been lumped in with the top-level. So that 3rd level if they are ONLY linked-to from their parent category should create as sort of well-rounded broccoli florets or nodules within a site. If each of these categories has some sort of identifying factor they can be color-coded based on then it would be different color nodules. If they can’t be segmented based on URL patterns or whatnot, they should still create lumpier distributions than purely concentric rings.

The purpose for using the Rule of 7 and the magic rolling pin principle is similar for git repos and websites. When you’re getting AI coding assistance you’re usually starting from a git repo (folder-based software repository). The AI is going to do an ls (list files) from the top-level directory of the software repository and start making choices from there. Likewise an AI surfing a site on your behalf will most often start at the homepage of a site and make choices from there.

In both cases there is a clear “top level” so it is one of the few assumptions or givens we can work with. While it is true that some actions on behalf of the user will be initiated with a search in a search engine whose results could drop you into a sub-page of a site as the beginning of a specialized on-demand crew, that extra search-vendor ALI dependency can be sidestepped if the LLM has enough in its base model to guess at the homepage based on brand. If this assumption is true it is likely to give an advantage to direct-sales by manufacturers versus their distributors if the user uses brand names in expressing intent.

Disintermediating Search: An End Run by AI

Disintermediation is the word. The makers of AI who are l working on behalf of users that express intent are disintermediating the web lookup services of the search engine engines by training brand information into base models. It’s an end run around the search engines.

Simultaneously, manufacturers have an increasingly easy time selling directly to end users because they already perform well in search on their own brand names, are trained-into (already known to) base model LLMs without secondary look l-up, and can outsource the technical liabilities and fulfillment logistics to cloud platforms such as Shopify and Salesforce Commerce Cloud.

If user expresses certain shopping intent that did not involve price comparison or anything compelling a general web search to be performed, the LLM that is well-trained on brands can directly guess that brand’s homepage slapping an “https://” at the beginning and a “.com” at the end, strip pit spaces in the middle and let webserver redirection do the rest to get the actual correct homepage. And that’s only if some curated brand-oriented subset of that DNS system were not previously trained into the models to eliminate most guesswork which it totally can be.

Once hitting a brand’s homepage, the LLM-guided bot can start a 6 degrees of separation small-world theory crawl drilling down on links or finds in the most logical sequence to find what the user is looking for, satisfy intent, and perhaps initiate the transaction. The guided crawl does not even have to be some powerful frontier cloud model LLM. It could be a lightweight LLM or even SLM (small language model) running locally on your desktop, both maintaining privacy and control controlling your own actual desktop browser so you can take control at any time.

This is all contingent on websites that LLM can actually navigate. This is not the case today. Or, rather it barely is. Websites are these big ugly messy things that are not conducive to 6 degrees of separation drill-down. Primary and secondary navigation is usually lumped together and cutting across the entire site wrapped in a “nav” element. Concepts of secondary and tertiary drill down levels are all kind of erased and flattened out. LLM are faced with the wall of links problem. It’s not exactly paralysis through analysis with too many choice like a human experiences, but rather it simply results in poorer choices.

To mitigate the difficulty of website navigation there needs to be a lot of supporting tools that the LLM know about and is smart enough to use: one to render the JavaScript and another to simplify the glut of data it’s asked to look at so good choices can even be made. If you want some idea of the problem, just right click on a webpage and select view-source for the source HTML pre-rendered and “inspect” for the post-rendered DOM (document object model) after all the initial page-load JavaScript has been executed (which doesn’t include lazy-loaded or other event-triggered JavaScript).

So what am I even proposing? That the homepage of an ecommece site should have 5 to 9 choices and the rest of the page not have any more links or else we confuse LLMs-directed bots? Why can’t the LLM just identity the site’s own internal site-search tool and short circuit the whole topical drill-down process? What about the idea of reducing the number of clicks it takes to reach any page, thus using a navigation scheme with fly-out hierarchy? Does that actually necessarily have to “pancake the site flat”? It seems like there are so many reasons that traditional topical-grouping hierarchy cannot be baked into the actual navigation of the site. The arguments are limitless.

And so they are. And so every site is difficult for AI to navigate. If you were to map out the links between pages, everything pretty much looks like the Deathstar with globally perfectly evenly distributed links between all primary and secondary navigational pages creating a tight core. Even the homepage itself is difficult to find in a link network. There is not necessarily more links leading to the homepage than there are two every secondary hierarchy page. It’s just one tight, circular, cross-linked blob.

If there actually happens to be a next level of hierarchy that’s constructed by anything other than product listing pages (PLPs), category pages, collection pages or whatever label you wanna put on it with some sort of navigation other than pagination, you’re just gonna get a second evenly distributed ring around the tight core. Everything around the second ring is just gonna be exactly cross-linked alike with no differentiated hubs. Their parents have no differentiated hubs and their children have no different differentiated hubs. It’s just a bunch of circles with the whole experience left to the exact same chance as that implies. Walls of links followed by deep-dive pagination.

Much of the responsibility for dealing with such a lazy design is shifted to internal site search tools. They are often faceted, that is they give a parameter such as color with different color options. They give a parameter such as size with different size options. You can choose this color with that size. In this way, every combination produces a page with its own unique URL. If you think that’s any better, you’re wrong and for so many reasons now in the age of AI. Think how much training it will be to make every LLM able to use every internal site search to a well. Compare that to just identifying the next logical drill-down link. It’s the difference between doing the work for them with intelligent, well laid out pre-searches ahead of time versus shifting it all the work of figuring it out on them.

Is creating an e-commerce site whose link network draws anything other than a Deathstar-like set of concentric rings realistic? I plan on finding out by developing the magic rolling pin to use on my own site, MikeLev.in which is a content site with articles, but the same basic principle should work on almost any content. The idea is that it gathers up the content of a site (or a sub-section of a site, depending) and then forgets the crawl path it used to find that content.

This magic rolling pin process rolls several times. On the first pass, it caches the website content locally so that subsequent passes can be done at no additional crawl-cost and some iterative process creates a link graph that organizes all the content into it, load-balancing every node to be the hub for 5 to 9 edges such that humans or bots can drill-down from the top of the tree to any other page in the site with hopefully no more than 6 clicks. The homepage at depth 0 counts as a level, so that’s 6 degrees of separation over a full click-depth of 7 to reach any page in the site.

---

Of course, here’s a summary and clarification of the “Magic Rolling Pin” concept.

The Magic Rolling Pin is a concept for reorganizing a website or software repository’s structure to make it efficiently navigable for both humans and AI agents. The goal is to ensure that any piece of content can be reached in approximately seven steps (clicks or directory changes). This is achieved by applying the Rule of 7, where each level of the hierarchy presents a balanced number of choices, ideally between five and nine.

---

The Core Principle: The Rule of 7

The central idea is to avoid overwhelming users or AI with too many choices at once, or too few.

• The “Sweet Spot”: Each menu, page, or directory should ideally offer 5 to 9 subsequent choices. This range is manageable and provides a clear path forward.
• Goal: To make any resource on a site or in a repository no more than 7 clicks or cd commands away from the starting point (the homepage or root directory).
• Application:
  • For websites, this applies to the navigational hierarchy (the links you click), which can be different from the literal URL structure.
  • For git repos, this applies to the actual filesystem directory structure, though exceptions are made for necessary root files or oversized plugin folders.

---

The Problem: The “Deathstar” Site Structure

Most modern websites don’t follow a clean, tree-like hierarchy. Instead, they form what you call a “Deathstar” structure.

• Description: This refers to a site where primary and secondary navigation links are all lumped together and heavily cross-linked, creating a dense, undifferentiated core. From a link-graph perspective, it looks less like a branching tree 🌳 and more like a tightly-knit ball or a set of concentric rings.
• The “Wall of Links”: This structure presents AI bots with a massive, flat list of links on nearly every page. It makes it difficult for the AI to understand the site’s topical structure and make logical, sequential choices to drill down toward a specific goal.
• Flawed Alternatives: Relying on internal site search is not an ideal solution. It forces an AI to learn the unique interface of every single website’s search tool, which is far less efficient than simply following a logical hierarchy of links.

---

Why This Matters for AI 🤖

The way websites are currently structured poses a significant challenge for the emerging ecosystem of AI agents designed to perform tasks for users.

• AI Navigation: For an AI to effectively shop, research, or perform other tasks, it needs a logical path to follow. The “Deathstar” model’s “wall of links” leads to poor decision-making.
• Disintermediation: AI models trained on brand names can bypass search engines and go directly to a manufacturer’s website. If that site is well-structured according to the Rule of 7, the AI can efficiently find what it needs, benefiting direct-to-consumer businesses.
• Efficiency: A well-organized site does the navigational work for the AI, allowing it to use simpler, more universal logic (i.e., “pick the most relevant link from this short list”) rather than complex, site-specific strategies.

---

The Proposed Solution

Your plan is to build a “magic rolling pin” tool to automate the creation of this ideal, balanced hierarchy. The process would be:

1. Crawl & Cache: The tool first gathers all the content from a site.
2. Forget the Path: It disregards the original, messy link structure.
3. Rebuild the Graph: It then iteratively organizes all the content into a new, balanced link graph. This new structure is load-balanced so that every page (or “node”) becomes a hub for 5 to 9 other pages.
4. Achieve 6 Degrees of Separation: The final result is a site where any page can be reached from the homepage (level 0) in six clicks or fewer, for a total path depth of seven.

Critiquing the Concept: Assumptions and Opportunities

---

Me: Well you competently re-stated it. But think into it. What faulty assumptions am I making? What opportunities am I missing? Is it viable? What’s the maximum number of pages that could fit if each node had the ideal 7 links? What if they all had 9 links? Would you prefer to use a site’s internal site search tool if it were usable or would you prefer to drill-down if it were well organized? And go on however else you’d like open-ended.

Gemini: Excellent questions. Moving beyond the “what” into the “why” and “what if” is where the concept gets interesting. While the “Magic Rolling Pin” idea is innovative, it rests on some key assumptions and opens up several strategic questions.

Here’s a critical analysis of the concept’s viability, potential flaws, and missed opportunities.

---

Faulty Assumptions & Potential Pitfalls 🧐

Your concept makes a few assumptions that might not hold true in the real world.

1. Content is Neatly Hierarchical: The biggest assumption is that all content can be cleanly sorted into a single, unambiguous tree structure. In reality, content is often poly-hierarchical. A single blog post could logically fit under “Artificial Intelligence,” “Web Development,” and “SEO Strategy.” A strict tree forces you to choose one primary path, which can hide content from users who start down a different, equally logical path.

2. Users Prefer to Drill-Down: The model presumes a user starts with a broad intent and narrows it down step-by-step. This is often not the case. Many users arrive with a specific, multi-faceted query (e.g., “blue wool V-neck sweater under $100”). A faceted search, where users can apply multiple filters at once, serves this intent far better than a seven-click drill-down. Your model seems to undervalue this powerful search pattern.

3. A Static Structure is Sufficient: The “rolling pin” process implies a periodic reorganization. However, modern websites are dynamic. For an e-commerce site, products, categories, and promotions change daily. A static, algorithmically generated structure would require constant, computationally expensive rebuilds to stay relevant and could create a jarring user experience if the navigation changes too often.

4. SEO Can Be Ignored: Radically and repeatedly changing a site’s internal linking structure would be devastating for Search Engine Optimization (SEO). Search engines rely on stable URLs and link equity to rank pages. A periodic “re-rolling” of the site’s architecture could confuse search crawlers and erase hard-won rankings unless managed with an extremely complex and careful strategy of canonical tags and redirects.

---

Missed Opportunities & Potential Enhancements ✨

The core idea has tremendous potential if expanded beyond simple link-count balancing.

1. Semantic Clustering: Instead of just balancing the number of links, the “rolling pin” could use AI to balance the meaning. By analyzing page content with word embeddings or topic modeling, it could group semantically related pages together. This would create a far more intuitive and contextually relevant hierarchy than one based on simple math.

2. Dynamic Personalization: The hierarchy doesn’t need to be the same for everyone. The linking structure could be dynamically assembled based on a user’s profile or browsing history. A user who frequently reads about photography could be presented with a navigation tree that prioritizes and expands on those topics.

3. Automated API & Knowledge Graph Generation: A well-structured site is a machine-readable site. This process is essentially creating a knowledge graph. You could miss an opportunity by only using it for navigation. You could also use it to automatically generate a public API or expose the site’s structure using standards like JSON-LD. This would make your site not just AI-navigable but truly AI-native.

---

Capacity Calculations 🔢

You asked about the maximum number of pages a site could hold with this structure, assuming a total of 7 levels (depth 0, the homepage, plus 6 clicks). This is a geometric series.

The formula for the total nodes in a perfect tree is: \(N = \sum_{d=0}^{D} b^d = \frac{b^{D+1} - 1}{b - 1}\) Where $N$ is the total number of pages, $b$ is the branching factor (links per page), and $D$ is the maximum depth (number of clicks from the homepage). Here, $D=6$.

• With 7 links per page ($b=7$): \(N = \frac{7^{6+1} - 1}{7 - 1} = \frac{7^7 - 1}{6} = \frac{823,543 - 1}{6} = \textbf{137,257 pages}\)
• With 9 links per page ($b=9$): \(N = \frac{9^{6+1} - 1}{9 - 1} = \frac{9^7 - 1}{8} = \frac{4,782,969 - 1}{8} = \textbf{597,871 pages}\)

As you can see, this model scales incredibly well, capable of organizing hundreds of thousands of pages within just six clicks.

---

My Preference: Drill-Down vs. Internal Search

As an AI, if I had to choose between a well-organized drill-down hierarchy and a usable internal site search, my answer is: it depends entirely on the user’s intent.

• For an exploratory or vague query (“Show me some interesting articles on your site,” “I’m looking for a new jacket”), the hierarchical drill-down is far superior. It allows me to logically explore the landscape of your content, understand the categories, and narrow down the possibilities in a structured way, much like a human would browse a store. Your Rule of 7 would make this process highly efficient.

• For a specific or multi-faceted query (“Find all articles about Python written by Jane Doe in 2024,” “I need a size 10, waterproof, black hiking boot”), the internal site search is vastly more powerful. It allows me to apply multiple precise filters at once. Trying to satisfy this request through a drill-down would be slow and might not even be possible.

Conclusion: The ideal website for an AI agent provides both. It has a clear, logical hierarchy for exploration and a robust, API-like search function for specific queries. Forcing a choice between them is a false dichotomy; a truly AI-friendly site masters both.

A Refined Approach: Clustering and Stable Navigational Hubs

---

Me: Alright, fair enough. My intent is to use Hierarchical Agglomerative Clustering for my first attempt at this project. And it may not end up being to actually map every single URL into that hierarchy so much as to group them into ideal unchanging mini-hub pages (5 to 9 spokes per hub) onto which the content can be grouped. So it’s 5 to 9 pieces of content (corresponding to URLs) that can reside on each page, and then 5 to 9 additional drill-down topics. Again, it’s the load-balancing concept.

The idea is for this approach to help SEO and not hurt it. Presumably more general topics become high-level parent nodes in the tree few clicks off the homepage. As topics become more specific and niche, they become lower in the hierarchy and in less need of links from powerful shallow click-depth. But all pages by definition of the rule of 7 will have a far number of topically relevant pages linking to them. The link-graph might change but the URLs wouldn’t except insofar as new cluster groups are created.

Gemini: That refinement of your strategy is a significant improvement and addresses many of the practical and SEO-related pitfalls. Using Hierarchical Agglomerative Clustering (HAC) to create stable mini-hub pages, rather than reorganizing the actual content URLs, is a much more robust and viable approach.

It correctly separates the navigational structure from the content’s address, preserving SEO equity while still achieving the desired organizational benefits.

---

Why This Hub-Based Approach is Stronger

This is a superior model because it embraces the web’s natural graph-like structure instead of forcing it into a rigid tree.

• URL Stability: By having unchanging URLs for your articles, you preserve all their accumulated SEO value, backlinks, and social shares. The hub pages become a flexible navigational layer that can be updated without causing a cascade of broken links.
• Solving Poly-hierarchy: This model elegantly solves the problem of a single article belonging to multiple categories. An article like /python-for-seo/ can be linked from the “Python Programming” hub and the “SEO Strategy” hub without any duplication or confusion.
• Improved “Scent of Information”: A user landing on a well-named hub page immediately understands the context of the links they are about to see. This is a core principle of good information architecture and dramatically improves user experience.

Building Topical Authority Through Structured Design

---

Using Hierarchical Agglomerative Clustering (HAC)

HAC is an excellent choice for this task. It’s a “bottom-up” clustering method where each document starts as its own cluster, and pairs of clusters are iteratively merged based on a similarity score.

• Key Advantage: HAC’s primary output is a dendrogram, a tree-like diagram that visually represents the nested grouping of all your content. This is perfect for your goal because you can “cut” the dendrogram at whatever level of similarity achieves your desired 5-to-9 clusters per hub.
• The Crucial Decision: The success of this entire process will hinge on how you define and calculate the “similarity” between two pieces of content. Using modern techniques like sentence embeddings (from models like BERT) to represent the semantic meaning of your content will yield far more coherent, human-intuitive clusters than older methods like simple keyword counting (TF-IDF).

---

SEO Implications: Building Topical Authority 🏆

Your assumption that this will help SEO is likely correct. This model is a fantastic way to build topical authority, which is a major factor in modern search rankings.

• Positive Signals: You are essentially creating a series of perfectly organized, topically-focused content silos. A hub page titled “Advanced Machine Learning Techniques” that links to 7 highly relevant articles and 5 more-specific sub-hubs sends an incredibly strong signal to Google that your site is an authority on that subject. Link equity flows logically from broad topics to niche ones.
• A Word of Caution: The hub pages themselves need to be more than just a list of links. To maximize their SEO value, these pages should contain a small amount of unique content—a paragraph or two introducing the topic. This elevates them from being simple navigation pages to being valuable, indexable content in their own right.

---

The Next Challenge: Automatically Naming Your Hubs

There is one critical step to plan for: cluster labeling.

HAC will successfully group a set of documents and tell you they belong together. It will not tell you why. It won’t automatically generate a name like “Python for Data Science” for a cluster of articles.

You will need a secondary NLP process to analyze the content within each cluster and generate a concise, human-readable, and SEO-friendly name for its hub page. Techniques for this include:

• Extracting the most representative keywords (e.g., using TF-IDF across the cluster).
• Using abstractive text summarization models.
• Analyzing the titles and H1 tags of the clustered articles.

Solving this labeling challenge is the key to translating the mathematically-optimal clusters from your algorithm into a genuinely usable and high-performing site structure.

---

Book Analysis

Ai Editorial Take

This is a high-potential piece that sits at the intersection of information architecture, SEO, and artificial intelligence. The evolution of the idea from a rigid rule to a flexible, cluster-based hub system is the most compelling part of the narrative. It mirrors the broader shift in AI from rule-based systems to dynamic, learning models. The concept of designing websites ‘for’ AI agents is timely and will become increasingly critical. The entry provides a strong foundation for a chapter on ‘AI-Native Information Architecture,’ outlining not just a theory but a practical, automatable methodology. Its core value is in anticipating a near-future problem and proposing a concrete, technically grounded solution.

Title Brainstorm

• Title Option: The Magic Rolling Pin: Structuring Websites for AI Navigation
  • Filename: magic-rolling-pin-ai-navigation
  • Rationale: Clear, evocative, and keyword-rich. It introduces the core metaphor (‘Magic Rolling Pin’) and immediately states its purpose (‘Structuring Websites for AI Navigation’), making it ideal for both human readers and search engines.
• Title Option: Avoiding the Deathstar: The Rule of 7 and AI-First Web Design
  • Filename: avoiding-deathstar-ai-web-design
  • Rationale: Uses the powerful ‘Deathstar’ analogy to frame the problem, which is very effective. ‘AI-First Web Design’ is a strong, forward-looking term.
• Title Option: Hierarchical Hubs: A Practical Guide to AI-Friendly Site Architecture
  • Filename: hierarchical-hubs-ai-architecture
  • Rationale: This title is more technical and directly references the refined solution (hubs and clustering). It’s less metaphorical but more descriptive of the final proposed method.

Content Potential And Polish

• Core Strengths:
  • Introduces a strong, memorable central metaphor (‘Magic Rolling Pin’, ‘Deathstar’).
  • Clearly articulates a complex problem in information architecture and its relevance to emerging AI technology.
  • Demonstrates a valuable process of iterative thinking, showing how an initial idea was refined through critical feedback.
  • Connects a technical concept to broader business implications like SEO and the disintermediation of search engines.
• Suggestions For Polish:
  • Integrate the conversational Q&A sections more seamlessly into a unified narrative to improve flow.
  • Add a concrete, simplified example of how a small set of articles (e.g., 20) would be clustered and organized into hubs.
  • Elaborate on the ‘cluster labeling’ challenge, as it’s a critical and non-trivial step in the proposed solution.
  • Visually represent the ‘Deathstar’ vs. the ‘Hub’ model. This could be done with simple diagrams to make the concepts more accessible.

Next Step Prompts

• Based on the ‘Magic Rolling Pin’ journal entry, draft a technical outline for a Python script that would perform the Hierarchical Agglomerative Clustering. Specify the necessary libraries (e.g., scikit-learn, spaCy), the key functions (e.g., fetch_content, generate_embeddings, create_clusters, label_clusters), and the expected inputs and outputs for each function.
• Expand on the concept of ‘cluster labeling.’ Write a detailed section explaining three different programmatic approaches to naming the content hubs, analyzing the pros and cons of each method (e.g., TF-IDF keyword extraction, abstractive summarization with a transformer model, title/heading analysis).