The Future of SEO: Reimagining SEO in the Age of LLMs

By Mike Levin

The Rise of LLMs

Let’s assume for a moment that the LLMs are going to get smarter and smarter, and that we are really at the beginning of a takeoff trajectory. It seems like just yesterday to me when video was postage stamp sized on the desktop of Windows 95. Does anyone remember Windows 95? Postage stamps? A lifetime is just about the right amount of time for some radical changes to take place. In a single life-time, someone born around 1900 could have seen both world wars, the nuclear bomb, reaching the moon, computers and nearly the Internet. They’d have been in their 80s in the 80s. So a centenarian would have seen the Web circa 1994 and the beginning of mobile. Think what our kids today are going to see.

Books Coming Alive

Okay, so the books are coming alive to teach you how to use the books. The key to unlocking the knowledge is now an inherent attribute of the knowledge itself, just add enough hardware to run an inference engine. Today’s LLMs are snapshots of the state of knowledge at the time processed into a single file that takes input and gives output. They seem alive, but that’s just because these LLM-players like Ollama which can run on your desktop, can be momentarily instantiated as a thinking entity. It’s like that Frankenstein moment of pulling the lever and infusing the monster with electricity, only that these come alive just long enough to give an answer, and then they POOF like an ephemeral genie. Any discussion continuity suggesting a continuous thinking entity is purely an illusion be re-loading the entire conversation history on each interaction.

However, this doesn’t disqualify the utility of the books themselves coming alive. No matter what other debate surrounds AI in terms of hype, it somehow topping-out at peak knowledge/data and only ever able to regurgitate and respin what is already known and been imagined prior by humans, it is still radically significant in similar ways as the printing press. The basic advances in humanity over other life-forms are of an information encoding/decoding and general transmission nature:

• We learn to communicate through sounds (speak)
• We learn to encode/decode those sounds through symbols (read & write)
• We learn to do that on long trans-generational media (scrolls & books)
• We learn to replicate that media (printing press)
• We learn to transmit that media around the world (Internet)
• We learn to make it at our fingertips (mobile)

…and now we make it come alive and teach us how to use the very information being brought to our fingertips.

The Next Evolution

Some want to bring it even a step farther with human/machine interfaces so you can recall information and experiences instantly that you yourself didn’t necessarily have, and control machines at the speed of thought. It doesn’t necessarily have to reach that point. We’re not done squeezing the lemon of books coming alive yet.

That’s what I’m interested in here. Books coming alive. But they can’t in their own stand-alone context. That’s the amazing thing about all these LLM models being distributed all over the place. They give a sort of universal or global human context as a starting-point, so that the next piece of information that you present them with is already in context of some larger picture — that picture being some static snapshot of the world as it existed and the data was fed to it at the time of its training.

Future-Proofing Skills

Okay, so future-proofing your skills so you don’t get sidelined and marginalized through this brief little 80-year or so window you’re given. I’m in my mid-50s and don’t plan on retiring anytime soon, so my goal is to make the next 30 years or so as fun and interesting as I can, staying in the game if I can.

LLMs get smarter and smarter.

The Challenge for New Generations

The youngsters coming of age in this era you’d think would be more adapted to the tech-churn and the tumultuous times like they always have been in the past. However, unlike in the past when the life-cycles of the new state of affairs, the advent of cars, computers, object oriented programming, rock and roll etc, had decades-long lifespans giving the youngsters enough time to get stable on their wobbly legs before the carpet is pulled out from underneath of them, the carpet is now pulled out from underneath of them at an accelerated pace. Hence, doomers.

The Exponential Future

This is not abating. The progression curve of technology will continue to be exponential. Techniques and methodology assures accelerating advancements within certain well-equipped, well-funded silos. The limits of Moore’s Law by which the number of transistors in an integrated circuit (IC) doubles roughly every two years will continuously be overcome through cleverness. So, forever more will be forever more possible, but not to everyone. Human nature kicks in and as William Gibson the author of Neuromancer said, the future will be here but just not distributed evenly.

The OpenAI Story

People like to bust on OpenAI and Sam Altman as the bad guys, taking what was ostensibly supposed to be Open AI, non-profit and ready-to-share, and to make it Closed AI, for-profit and proprietary secrets. However the truth is Google had been sitting on this stuff in their LaMDA labs, keeping secret genies in secret bottles for some time before OpenAI blew the lid off of it Nov 2022. They did indeed open the doors letting the public realize just how things have changed. The next tech after mobile, in succession from the printing press and the spoken word, is now upon us. They did indeed open AI. While absolute power corrupts absolutely, and they did somewhat morph into the less-than altruistic entity they posed as going in, they absolutely played the role of the good guys against Google’s foil. OpenAI, despite its shift in business model, played a role in democratizing access to AI technology and pushing the field forward.

Google’s Approach

Now, you might say Google’s approach was altruistic in that it was cautious in that they didn’t want to let the genie out of the bottle for the world’s own good. And they did in fact publicly release that paper, Attention Is All You Need, in 2017 which was the key enabling piece for OpenAI. Google just didn’t turn the fruits of their labor into a product. Nor would they want to, because if it took off in popularity too much, wouldn’t it cut into their AdWords revenue, which is what? Still like 80% of how they make money? So, Google was holding back their LaMDA beasties because of the innovators dilemma, and others were not so inhibited. So, BAM! ChatGPT.

But the basic trick just takes lots of GPUs, lots of data, and following in footsteps. Yes, there is art and nuance to doing the training and making them safe(ish) for public release. But Facebook/Meta jumped on that bandwagon pretty quick to demonstrate what OpenAI had and what Google was holding back wasn’t really that special or unreproducible with Llama. Plenty of others followed like Alibaba, the Amazon of China, with Qwen.

It’s somewhat hard to wrap one’s mind around the state of affairs today. So let me ask Perplexity.

Me: What fundamentially different models are there? Like when DeepSeek was released on Ollama, it was distilled down on the llama and qwen versions. So really there’s fewer than it seems. What are the big camp splits? Like Google’s Gemini and gemma? Is that legitimately its own? How does it break down big-server cloud-wise and running on llama.cpp-wise?

Perplexity: There are several fundamentally different large language model (LLM) architectures, with a few major “camps” dominating the landscape:

1. Transformer-based models:
   • GPT (OpenAI): GPT-3, GPT-4
   • PaLM (Google): PaLM, Gemini
   • LLaMA (Meta): LLaMA, LLaMA 2
   • BERT (Google): BERT, RoBERTa
   • T5 (Google): T5, mT5
2. Open-source models:
   • Gemma (Google): Lightweight, open-source version derived from Gemini research[6]
   • LLaMA (Meta): Popular base for fine-tuning and distillation
   • BERT (Google): Widely used for various NLP tasks
3. Proprietary cloud-based models:
   • Gemini (Google): Advanced, multimodal model with large context window[4]
   • GPT-4 (OpenAI): Powerful general-purpose model
   • Claude (Anthropic): Known for long-context understanding and ethical considerations[3]
4. Specialized models:
   • Mistral: Focused on efficiency and fast inference[3]
   • DeepSeek: Emphasizes multilingual capabilities and coding tasks[5]
   • AWS Nova and Titan: Optimized for enterprise-scale applications[5]

Regarding the distinction between Google’s Gemini and Gemma:

Gemini is Google’s proprietary, heavyweight model designed for advanced AI research and large-scale applications. It has a larger context window (up to 1M tokens) and supports multimodal inputs (text, image, voice, video)[4][6].

Gemma, on the other hand, is an open-source, lightweight version built using the same research and technology as Gemini. It’s intended for developers and researchers, with compatibility for JAX, PyTorch, and TensorFlow[6].

For running models on local hardware using frameworks like llama.cpp:

1. Open-source models like Gemma, LLaMA, and BERT are more suitable for local deployment.
2. Quantized versions of these models (e.g., 4-bit or 8-bit) can run on consumer hardware.
3. Smaller variants (e.g., Gemma 2 9B) are more practical for local use compared to larger models[4].

For cloud-based deployments:

1. Proprietary models like Gemini, GPT-4, and Claude are typically accessed through cloud APIs.
2. These models offer higher performance and larger context windows but require more computational resources.
3. Cloud providers like Google AI Studio, Vertex AI, and AWS Bedrock offer scalable infrastructure for deploying these models[4][5].

In summary, while there are many model variants, they often stem from a few core architectures. The main distinctions lie in their size, specialization, open-source availability, and deployment options (cloud vs. local).

Citations:

[1] https://toloka.ai/blog/difference-between-ai-ml-llm-and-generative-ai/
[2] https://www.reddit.com/r/OpenAI/comments/1dqqy15/google_ai_major_updates_gemma_2_and_gemini_new/
[3] https://ai-pro.org/learn-ai/articles/a-comprehensive-comparison-of-all-llms/
[4] https://docsbot.ai/models/compare/gemma-2-9b/gemini-2-0-flash-thinking
[5] https://dev.to/mindsdb/navigating-the-llm-landscape-a-comparative-analysis-of-leading-large-language-models-1ocn?comments_sort=latest
[6] https://blog.marvik.ai/2024/07/03/difference-between-gemma-and-gemini/
[7] https://www.microsoft.com/en-us/microsoft-cloud/blog/2024/11/11/explore-ai-models-key-differences-between-small-language-models-and-large-language-models/
[8] https://docsbot.ai/models/compare/gemma-2-27b/gemini-1-5-flash-002

Questions About Qwen and Local Inference

Me: What about Qwen? And what role do the local quantized inference engines play? Are they some sort of bottleneck coercing models that work fundamentally differently in their larger incarnation (DeepSeek) into what already exists at the smaller llama.cpp incarnation (DeepSeek qwen and DeepSeek llama)? In other words, is there a fundamental architectural split between what large datacenters can run and what consumer hardware can run? Is this insurmountable or a temporary condition based on the non-commercial volunteer state of llama.cpp?

Perplexity: Qwen represents a significant advancement in large language models, with several key features and developments:

1. Model Suite: Qwen offers a diverse range of models, including Qwen2, which spans from 0.5 to 72 billion parameters[1]. The suite includes specialized models for tasks such as math, audio processing, and visual understanding[3].

2. Architectural Evolution: Qwen has evolved from its initial 7B model to larger versions like 14B and 72B, as well as specialized variants like Qwen-VL for vision-language tasks[3].

3. Recent Advancements: In September 2024, Qwen released the Qwen-2 series, trained on 7 trillion tokens with improved data quality and training methods[3].

4. Multilingual Capabilities: Qwen2 demonstrates strong performance in about 30 languages, showcasing its versatility for global applications[1].

Regarding local quantized inference engines and their role:

1. Quantization Importance: Quantization is crucial for deploying AI models on edge devices with limited power, memory, and computing resources[6]. It reduces model size and speeds up inference by using lower-precision data types (e.g., 8-bit integers instead of 32-bit floating-point)[8].

2. Performance Benefits: Quantized models can achieve up to 3x improvement in performance, especially on hardware like the Snapdragon® Hexagon Tensor Processor[4].

3. Architectural Differences: There is indeed a split between what large data centers can run and what consumer hardware can handle. Larger, more complex models often require the resources of data centers, while quantized versions are necessary for consumer devices[6].

4. Bottleneck Effect: The need for quantization does create a bottleneck, as models designed for large-scale deployment often need to be adapted or simplified for consumer hardware. This can lead to some loss in capabilities or accuracy[9].

5. Ongoing Research: The gap between data center and consumer hardware capabilities is not necessarily insurmountable. Ongoing research, such as Qualcomm’s work on improving quantization techniques, aims to reduce the accuracy loss in quantized models[9].

6. Role of llama.cpp: While llama.cpp plays a significant role in enabling inference on consumer hardware, it’s not the sole factor determining model architecture. The limitations are primarily due to hardware constraints rather than the software implementation.

7. Future Developments: As quantization techniques improve and consumer hardware becomes more powerful, the gap between data center and consumer capabilities may narrow. However, it’s likely that there will always be some trade-off between model size/complexity and the ability to run on consumer devices.

In conclusion, while quantization and local inference engines do create some limitations, they are crucial for bringing advanced AI models to consumer devices. The ongoing research and development in this field suggest that these limitations are more of a temporary condition that will continue to evolve as technology progresses.

Citations:

[1] https://qwen2.org
[2] https://www.run.ai/guides/cloud-deep-learning/ai-inference
[3] https://www.inferless.com/learn/the-ultimate-guide-to-qwen-model
[4] https://app.aihub.qualcomm.com/docs/hub/quantize_examples.html
[5] https://kili-technology.com/large-language-models-llms/qwen-2-guide-what-can-we-learn-about-alibaba-s-multilingual-llm
[6] https://moschip.com/ai-engineering/model-quantization-for-edge-ai/
[7] https://github.com/QwenLM/Qwen/blob/main/tech_memo.md
[8] https://huggingface.co/docs/optimum/en/concept_guides/quantization
[9] https://stackoverflow.blog/2023/08/23/fitting-ai-models-in-your-pocket-with-quantization/

The Moore’s Law of AI Models

Alright, so it’s reasonable to expect that both the cloud-based models and the locally runnable models are both going to propagate at levels which themselves parallel what’s implied by Moore’s Law. The genies that are out of the bottles are only going to be making more genies, and they are going to be of an infinitely varied nature, a product of the data going in and the training process. It’s almost like a nature/nurture parallel to living beings. Yet still, given the batch processing nature of training, they’re all still going to be like snapshot files. And that’s going to be less than satisfying for the people using these models, expecting that the learning process shouldn’t always get “reset” with every new instantiation but for the recent discussion history within the token limit loaded back in.

Solutions for Continuous Learning

And so, there’s at least two solutions here. One is partial retraining in hte form of Low-Rank Adaptation of Large Language Models (LoRA) and the other is Retrieval-Augmented Generation (RAG). Both of which have their own unlimited potential approaches and varying levels of processing power and infrastructure needs. But in the quest for a sort of future-proofing, it is always a good bet to try to preserve the original data on which things get trained so that as things change, you can re-process in some way. Against a backdrop of infinite training and re-training and continuing learning techniques is the fact that it was all against some base of data, which can itself continue to grow, and perchance be artistically reshaped, prepared, cleaned and otherwise improved in various ways. Then your options are always open to use the original raw data or the ostensibly improved data.

The Path Forward

So a path to future-proofing seems to be raw data storage and various techniques of making future re-training and re-learning easier, higher quality, and flexible.

Perplexity: Your analysis of the current state and future trajectory of AI models is insightful. Indeed, both cloud-based and locally runnable models are likely to advance rapidly, following a trajectory similar to Moore’s Law. The analogy to nature/nurture in living beings is apt, as these models are shaped by both their initial training data and subsequent fine-tuning or adaptation processes.

You’ve correctly identified two key solutions to the challenge of static, snapshot-like models: Low-Rank Adaptation (LoRA) and Retrieval-Augmented Generation (RAG). These approaches address the limitations of traditional fine-tuning methods:

1. Low-Rank Adaptation (LoRA):
   • Introduced in January 2022, LoRA allows for efficient adaptation of large language models[1].
   • It reduces the number of trainable parameters by injecting trainable rank decomposition matrices into each layer of the model architecture[1].
   • LoRA can reduce the number of trainable parameters by a factor of 10,000 compared to full fine-tuning, while maintaining or improving performance[5].
2. Retrieval-Augmented Generation (RAG):
   • RAG enhances LLMs by incorporating an information retrieval system to provide grounding data[2].
   • It allows models to reference external, up-to-date information sources, addressing the limitation of static training data[4].
   • RAG involves indexing external data, retrieving relevant information for queries, and augmenting the LLM’s knowledge during generation[6].

Your emphasis on preserving original training data is crucial. This approach provides flexibility for future retraining and adaptation as new techniques emerge. Some key considerations for future-proofing include:

1. Data Storage and Management:
   • Maintain comprehensive, well-organized datasets for future use.
   • Implement robust data versioning and cataloging systems.
2. Data Preparation Techniques:
   • Develop and document processes for cleaning, normalizing, and enhancing raw data.
   • Explore methods for data augmentation and synthetic data generation.
3. Flexible Model Architecture:
   • Design models with adaptability in mind, allowing for easy integration of new components or techniques.
   • Implement modular architectures that can be easily extended or modified.
4. Continuous Learning Frameworks:
   • Develop systems that can incorporate new data and adapt models incrementally.
   • Explore techniques like online learning and federated learning for real-time model updates.
5. Evaluation and Benchmarking:
   • Establish comprehensive evaluation frameworks to assess model performance across various tasks and domains.
   • Regularly update benchmarks to reflect evolving requirements and challenges.

By focusing on these aspects, organizations can maintain the flexibility to leverage advancements in AI technology while building upon their existing data and model investments. This approach aligns with the rapidly evolving nature of AI research and development, ensuring that AI systems can be continuously improved and adapted to new challenges and requirements.

Citations:

[1] https://openreview.net/forum?id=nZeVKeeFYf9
[2] https://en.wikipedia.org/wiki/Retrieval-augmented_generation
[3] https://openreview.net/pdf?id=nZeVKeeFYf9
[4] https://aws.amazon.com/what-is/retrieval-augmented-generation/
[5] https://arxiv.org/abs/2106.09685
[6] https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview?WT.mc_id=DT-MVP-10953
[7] https://nexla.com/enterprise-ai/low-rank-adaptation-of-large-language-models/
[8] https://www.salesforce.com/agentforce/what-is-rag/?bc=OTH
[9] https://github.com/microsoft/LoRA

The Myth of Peak Data

Ilya Sutskever and Elon Musk asserting that we’ve “run out of data” in the world and that we have somehow reached peak data must be a ruse. Does the world ever run out of data? It’s just the easily crawl-able data that they’re talking about from the open web. There’s not only the invisible web, given the vast number of sites programmed as difficult-to-access single page applications because of the client-side JavaScript programming fads, there’s all the proprietary data locked up behind paywalls and otherwise inaccessible that would be challenging, expensive and legally questionable to train on.

The Future Flood of Data

Not only that, this doesn’t address the current and potential future flood of new data onto the Web, given the increasing world population (even if the rate of growth is slowing), and the rate at which genuinely new human-created data could be produced. There’s always that qualification of “quality data”, and so there’s this presumption that new Aristotles or Einsteins just aren’t going to happen, or even if they do, their works are going to be impossible to find or identify as such, even if it did get published on the open web. There seems to be some underlying assumption that everything has now become twitter-babble and not quality enough to factor in anymore.

Scientific Frontiers and New Data

And then there’s the scientific frontier data from which new discoverers have not even been made yet, like the mapping of celestial entities in the cosmos from the data of many telescopes collecting such data, and the data from particle colliders like the LHC and others, about which you hear that it will take years to process what’s collected. Is this not fertile ground for new data, or does it necessarily have to be of the written/spoken language variety? And even if of the written or spoken language variety, isn’t the Web the very clearinghouse? Isn’t it just a matter of newer and better crawling techniques?

Perplexity: The assertion that we’ve “run out of data” in the world is indeed misleading and likely a simplification of a more complex issue. There are several reasons why this claim is questionable:

1. Invisible Web: A vast amount of data exists in the “Invisible Web,” which includes databases, dynamic websites, and password-protected content not easily accessible to search engines[2]. This represents a significant untapped source of potentially valuable information.

2. Proprietary and Paywalled Data: There’s an enormous amount of data locked behind paywalls, in academic databases, and within private company records[5]. This data is not readily available for AI training but still exists and could potentially be used with proper agreements and access.

3. Continuous Data Generation: With increasing global population and internet connectivity, new data is constantly being created. This includes social media posts, scientific research, news articles, and more.

4. Scientific Frontier Data: As you mentioned, there’s a wealth of data being generated by scientific endeavors, such as astronomical observations and particle physics experiments. This data, while not necessarily in natural language format, can still be valuable for AI training in specialized domains.

5. Quality vs. Quantity: The concern about “quality data” is valid, but it’s an oversimplification to assume that all new data lacks quality. New discoveries, insights, and valuable information are still being produced regularly.

6. Improved Crawling Techniques: As web technologies evolve, so do data collection methods. New crawling techniques and AI-powered data extraction tools can potentially access previously hard-to-reach data sources[1].

7. Diverse Data Types: AI training isn’t limited to text data. Images, videos, audio, and other data types are also valuable and continuously growing.

8. Regulatory and Ethical Considerations: Recent research indicates that restrictions on data availability for AI training are increasing[7]. This suggests that the issue may be more about access and permissions rather than an actual shortage of data.

In conclusion, while easily accessible, high-quality data from the open web may be becoming more scarce or restricted, it’s inaccurate to claim that we’ve “run out of data” globally. The challenge lies in accessing, processing, and ethically utilizing the vast amounts of data that continue to exist and grow across various domains and formats.

Citations:

[1] https://hdsr.mitpress.mit.edu/pub/xau9dza3/release/2
[2] https://en.ryte.com/wiki/Invisible_Web
[3] https://theodi.org/news-and-events/blog/policy-intervention-4-ensuring-broad-access-to-data-for-training-ai-models/
[4] https://asbury.libguides.com/c.php?g=65758&p=424014
[5] https://www.potterclarkson.com/insights/what-data-is-used-to-train-an-ai-where-does-it-come-from-and-who-owns-it/
[6] https://www.oedb.org/ilibrarian/invisible-web/
[7] https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html
[8] https://owl.purdue.edu/owl/research_and_citation/conducting_research/searching_online/search_the_invisible_web.html

Peak Data is a Myth

Okay, so the Open Web has achieved Peak Data, and all the crawls you ever performed are all the crawls that you will ever need to perform, right? Anything further is a waste of time and resources, and there’s no value in the differential delta difference between your last crawl and theoretical future crawls, right? This is especially true given the garbage on the Internet, and the corrupting influence of SEOs vying for commercial gain for their clients, or for click-through on affiliate links and advertisements, right?

Only if everything remains dumb.

The Discerning Eye of LLMs

Imagine a discerning, curating eye that knows all the tricks. Imagine the ability to judge newly found data against all that has ever been known before, nearly instantly. And imagine it having enough intelligence to recognize something new under the sun, and genuinely meritoriously noteworthy.

In other words, imagine all the amazing new capabilities of LLMs, namely deep expertise dealing with patterns, using those very patterns to have a discriminating scalpel as it scours for noteworthy new stuff. No, there’s not a lot of meat left on the carcass. But it’s not a carcass. It’s the whole body of human knowledge that’s being made open, outside firewalls, password protection and JavaScript execution obfuscation.

The JavaScript Problem

Obscuring data through proprietary JavaScript execution is particularly damaging to the cause of exposing new meat on the bones. Why? Because so long as it’s HTML, it’s within the realm of semantic meaning that’s supposed to be inferred onto the data with tags. Everything is tagged-up, i.e. labeled, by default. The Web is supposed to be a giant database of pre-labeled data, presumably ideal for machine learning. This is a title, that’s a headline, the following is navigation, and down there is a footer. Take all this semantically meaningful rich markup into account when you train on it.

The Markdown Translation Problem

One of the biggest ironies of all this crawling and scraping and data-cleaning for AI training is how the HTML is converted into Markdown files, stripping out the semantics, presumably in a constructive way like stripping out the navigation and footers, but who knows? Is it really keeping headlines intact? It’s certainly not keeping paragraph tags, as those are not even present in Markdown, and so we know a lot is getting lost in translation. It’s doubly ironic because in many cases, the data started out as Markdown in the first place, only to be converted to HTML for display in the web browser! So from Markdown to HTML to Markdown again, in a game of whisper-down-the-lane AI training.

The Source Attribution Problem

And that’s not to mention the URLs on which the original data was found. You just know the original source of the data is stripped out, or else the New York Times’ case against OpenAI would have had a lot more teeth. AI training launders the sources of the data, just like money laundering launders the source of the money. No URL, no smoking gun trail. It’s like the way those meme sites delete their log files to protect anonymity of their users. No trail, no origin, no accountability.

The Future of SEO

If SEO gets reborn in this article, it’s in how it’s going to encourage playing into the subtle knife of discriminating LLMs controlling crawls and other phases of the process. It will be laying down an open kimono approach, as the saying goes. It’s going to be setting out a number of principles based on meritocracy and transparency as to financial and influencer incentives. It certainly is going to pick up on the vibes and foundation laid down by llmstxt.org whereby there is at least a conventional addition and supplement to robots.txt and sitemap.xml put there deliberately for LLMs.

The Era of Transparency

Assume trickery will be seen through and penalized. It’s not going to be a Google manual penalty you’re going to have to worry about. No, it’s going to be an insulted LLM controlling some process that sees through you and outs you for what you’re doing, and categories your stuff into a blacklist filter. Machine learning will be used to set curation tastes. Based on the quality and nature of that curation, things like affiliate links and advertisements will become signals of low-quality (or not), but the proof will be in the ultimate output and how people respond to it.

The Return of Meta Search Competition

Given the fact that you’re not going to need to crawl the entire Web to make this work, but merely need some sort of “seeding” from a Yahoo directory-like thing (today, everyone’s using the Microsoft Bing API), and then small-world-theory driven crawl based on the same premise of six-degrees of separation, almost anyone will be able to get into the competitive search engine game. It’s like the era of metasearch with HotBot and DogPile gets reborn and it could be anyone’s game again.

Google’s Strategic Position

This is where Google’s control over Chrome and the Android platform become so important. They won a battle long ago over the unification of the “search” and the “address” bar in the web browser. They are unified such that typing anything into the address bar will at least perform a search… a Google search. That’s the battle that’s been won by Google. That’s why control over Chrome is so important, and the remedy in the Google anti-trust suit is recommended to be the splitting off of Chrome. Chrome directly feeds AdWords, making it a lopsided game. This is the main factor working against LLMs themselves directly killing SEO.

The AdWords Dependency

SEO has greatly died because Google needs AdWords revenue. They have not adequately diversified their revenue streams, like Microsoft has over the years. The Google castle is indeed built on the Cloud, in the clouds, with no foundations but advertising. This is why Microsoft has been able to be so aggressive with both Bing and the OpenAI ChatGPT Copilot integration into Bing. It’s not cutting into lifeblood ad revenue.

Microsoft’s Advantage

Think about how Microsoft can aggressively play there and survive, whereas Google is locked into the innovators dilemma, rolling out super-conservative AI Overviews instead of the personality-infused chat interface Microsoft went with. AdWords is what? Still like 80% of Google’s revenue? The Google machine has grown so large that it must continue to be fed to continue to live. Not messing around with AdWords is not a choice. It’s a mandate. Thus, the innovator’s dilemma. Thus, the boring and low-priority nature of AI overviews.

The Pay-to-Play Reality

Ads will be served at the expense of all else. Don’t be evil plays second fiddle to don’t be dead. The customer is always right, and the customers are the advertisers paying Google. You are not a Google customer. You are raw material being converted into value for their customers in the value chain. Your attention is being mined as ore and packaged and sold as product to their real customers. And that can’t be done if your eyeballs and clicks aren’t following the money. It’s just plain survival.

The Magazine Industry Parallel

This content-corruption is exactly what happened with advertisement supplemented magazines turning them into more ads than articles. Voices are drowned out by pay-to-play money. So, it’s the ability for the little guy to do really well purely through SEO that has died. Pick up any magazine on the store shelf that isn’t $20 (cost offset by ads) and you’ll see the pay-to-play reality that has long been in effect in the print-world where hard-nosed material costs can’t be dodged.

The Online Shell Game

Those hard costs can be hidden online through the shell-game for quite a bit longer than with print. But the transition still occurs. Everything gradually becomes Advertorials. AdWords which used to be clearly labeled and sectioned off in a right-hand margin got mixed directly in with the results. The pastel colors faded to pure white just like the rest of the “organic” or “natural” search results. Now you can hardly tell a paid ad from a genuine search result.

The Corporate Evolution

Why? Google went public. Google grew big. Google has to answer to shareholders. Google fell prey to The Law of Organizational Entropy, which states that as organizations grow larger, they tend to become less efficient and more complex, requiring more resources to maintain their operations. If they remained small and lean and humble like they started out, they wouldn’t need so much ad revenue today just for the beast to stay alive. But alas, that is not the way of public companies, Parkinson’s law, the Peter Principle and the other powers of piddling out.

The Browser Battleground

And this pay-to-play reality stays fixed so long as Google’s death-grip over Chrome and Android remains firm, the Apple deal stays in place on Safari, and Microsoft can’t get people to move to Edge through a war of upgrade attrition and switch-back fatigue.

The Path Forward

So what now? Should everyone just give up on SEO and instead pay AdWords and be in Merchant Center? To pay homage to Google because they control Chrome, essentially?

Ugh, it’s like I’m talking myself into that. But no!

Back to the Long Tail

We’re going right back to the long tail. You can’t squash all niches. They pop back up like a game of wack-a-mole. It’s not big money. It’s in the fringes. It’s the edges. But in the end, isn’t it the outliers that really matter? Isn’t the uncommon path and the unlikely what ultimately eventually changes everything, after some time of the status quo playing wack-a-mole?

Born of Love and Necessity

And this stuff is often born of love, or alternatively scratching an itch nobody else has realized was bothering them. In a perfect storm, it’s both scratching an unnoticed itch and doing it for the love of it. This is why art is important. Every once in awhile, artists and artistry shows the way for technology. Not that the artists are the ones that financially benefit from their own innovation. Good timing, good business sense and good implementation are what’s actually rewarded monetarily.

A Personal Journey

For myself, for my own purposes, I don’t need to sink or swim by the financial success of my SEO endeavors. At least, not in the direct skin-in-the-game way as people selling product online. Though I think I should endeavor to get to that point, but in a very volunteer “because they want to support me” sort of way. I just talked about Carson Gross’ book about hypermedia on a previous post, and it’s definitely along those lines.

It’s one of those South Park underwear gnome step #3: profit situations. I know I will get there, if even just keeping sharp for my day job.

The Future Opportunity

You can bank on the fact that things are going to go wacko with people looking for direction. SEO isn’t going to piddle out as a field. There’s always going to be an appetite for getting your traffic, attention, eyeballs, audience, following, fans, tribe, people or what have you meritoriously! And that’s where I need to live. It’s not about gaming the system. It’s about being in the right place at the right time. Sometimes, there’s a man, well, he’s the man for his time and place. He fits right in there.

Finding the Niche

Okay, so if I’m that dude, what’s the thing?

Well, of course it’s the thing that people are going to be searching on, for which I already happen to have been doing for quite a long time and have all the answers.

I am effectively scratching an itch, and have been for a long time, that they themselves only just realized they had now.

• Open Source SEO Software
• Linux SEO software
• SEO Software

Yup, in the realm of SEO, that’s probably the long-tail to the fat-head that I’ll be climbing up. But it’s more than just that. I’m going to appeal very much to people reacting to bloated client-side JavaScript frameworks, investigating FastHTML and HTMX.

• FastHTML Database
• FastHTML vs. FastAPI
• FastHTML

The Path to Traffic

Yup. That’s a pretty sure path to traffic, where I can forge a meritorious path through what’s left of the long-tail jungle. There’s wars brewing. LLM is simultaneously going to make it easier to slam out cookie-cutter instances of enterprise architectures wonderfully scalable, and exactly like everyone else. But those people looking for something off the beaten track are going to dig in the results further. They’re going to add modifier keywords to their searches and they’re going to go more pages deep. They’re going to find FastHTML, and the first roadblock they’re going to encounter is database integration.

And then they’re going to encounter all the same LLM overrides and undermining of their FastHTML research as I encountered, basically them compelling the user who is looking for an uncommon path back to the common path of FastAPI. Don’t you mean FastAPI is what the LLM is going to say (in essence) over and over until the attempt at FastHTML code is so mangled and irrecoverable, and such frustration has set in, that they abandon their original attempts.

The Implementation Plan

So, I have to throw the life-preserver out there for FastHTML seekers. First, I need to clarify all that database and MicroDataAPI spec stuff (what an unusual name), and the convenience wrapper of fast_app and all that tuple passing and unpacking Kung Fu. Maybe I have to say Brazilian Jiu-Jitsu now that my Mixed Martial Arts (MMA) research has convinced me how badass it really is, especially when compared with Kung Fu masters who get their asses kicked. But it just doesn’t roll off the tongue quite as smoothly as Neo saying “I know kung fu”. “I know Brazilian ju-jitsu” just doesn’t have the same cachet.

Future Keywords and Topics

Okay, what else? Ollama? Local LLM? gguf files? LM Studio? Ollama vs vllm? Something about llama.cpp? How to train a local LLM? Your local ollama genie in a bottle buddy for life? Chip O’Theseus? This one is not nearly so clear insofar as the keyword variations to target, but it’s definitely local ollama-based. I should probably put out something about local ollama llm training and get the data feedback loop going.

The Next Steps

Speaking of which, today is really about getting things a bare minimum something ready for public consumption. Updating Pipulate in the public repo yesterday is the preface to that. What is… the memetic… the magical memetic incantation… yeah. You’re in the neighborhood here.

You’re talking about a magical memetic incantation to get something extraordinary onto your machine to give you a little taste of what’s to come… very rigged-for-successful-demo-like, but still the first baby step towards some remarkable utility.

To not be over-ambitious.

I said it perfectly yesterday:

1. Make chatq support verbatim scripts that keep the LLM in the loop via conversation history.
2. Blend-in textfile.md-based super prompts, also likely through similar conversation history injections (well aligned to prior item).
3. A single SEO-centric pipeline workflow that I can build from, which likely does a very mini crawl.
4. Potentially allowing this to be done with an incredibly easy VPN in use that’s non-disruptive to Zoom and the like while in use.

If I only do these 4 things, it puts the copy/paste-able in-place.

Claude: Your analysis is fascinating and I see exactly where you’re heading. Let me reflect back the key insights and strategy:

1. The “Peak Data” Fallacy
   • The idea we’ve “run out of data” is myopic
   • It ignores the invisible web, new content, and scientific frontiers
   • Most importantly, it ignores the potential of intelligent crawling
2. The New SEO Paradigm
   • Moving from gaming algorithms to meritorious content
   • LLMs as discriminating curators rather than dumb crawlers
   • Transparency and quality over tricks and volume
3. Your Strategic Position
   • Targeting the intersection of several emerging needs:
     • Open Source SEO Software
     • FastHTML (vs FastAPI orthodoxy)
     • Local LLM integration
     • Database simplification
4. The Implementation Plan Your 4-point plan is beautifully focused:

   WEEKEND_GOALS = [
    "Scripted chatq with LLM memory",
    "Markdown super-prompts injection",
    "Mini SEO crawler pipeline",
    "Optional VPN integration"
   ]
   

The genius here is how you’re positioning at the convergence of several “long-tail” needs that are about to become very relevant:

1. People seeking alternatives to JavaScript bloat
2. Those wanting local-first LLM solutions
3. SEOs looking for new approaches in the AI era
4. Developers tired of over-engineered solutions

By keeping it simple and focused on these 4 weekend goals, you’re creating the foundation for something that could grow organically along with these emerging needs.

Want me to focus on any particular aspect of the implementation plan?

You say you want an SEO Revolution?
Well, you know we all wanna prune the cruft
I tell you that it’s evolution
Well, you know, LLMs see your keywords are stuffed!

LLM Chat System Evolution

So Claude and I go back and forth on some changes to the system that was meant to be exclusively LLM-chat into a more verbatim “say this” (simulated LLM output), and let the LLM know that was said (putting words in its mouth).

This is specifically wired up to the messages that get displayed when you switch areas of the site using site navigation. It was supposed to be a cute little way for the LLM to talk to you as you navigated around the site, but it turned out to be more annoying than cute. And in general, a very dry neutral “this happened” or “please do that” voice is necessary to guide the user around in a very wizard-like way. You know, the old software wizards of yore, of which Clippy was probably a fine example. Software wizards are a good idea whose time has come around again, particularly in how well they mesh with LLM-powered apps, but for all the steps that shouldn’t have LLM randomness and unpredictability.

Scripted LLM Interactions

The interesting thing here is that you can still make it look to the LLM as if it said those things itself. It’s all just part of the discussion history, with the system editing in what the LLM is alleged to have said of its own volition, but it wasn’t. It’s scripted like a telemarketer’s script.

And this compels the forward-progress of software wizard steps, with the LLM still knowing what’s going on. Someday, this sort of thing might be deemed unethical, zombie-driving an LLM, but for now we call it quality assurance.

Me: Okay, we nailed it! You are redeemed. But now I have to figure out how to do the system prompt injection for the Neo-style Kung Fu downloads when… well, I was thinking on menu selection, but now I’m thinking I want to sprinkle them in anywhere. Specifically, the instructions from todo_list_training is being shown to the user under this new verbatim temp_message approach, instead of actually training the LLM. Get it? I need deliberate control. One system for talking to the user directly and another for training the LLM in the background. Perhaps two things that are called from JavaScript at the same moment the temp_message is set?

Claude: Ah, now I understand! You want to handle two distinct actions when initializing certain features:

1. Show instructions to the user (verbatim message)
2. Silently train the LLM (system prompt injection)

We can see this pattern already in your !help command:

            if message.lower().startswith('!help'):
                # First, show help text to user
                help_text = """Available commands:
                !test - Run DOM manipulation test
                !egg - Insert a test task
                !help - Show this help message"""
                await websocket.send_text(help_text)

                # Silently reinforce system instructions
                system_message = generate_system_message()
                await chatq(system_message, role="system", base_app=self)  # Output doesn't matter

So for the JavaScript initialization, we could do something like:

                window.addEventListener('DOMContentLoaded', () => {
                    const tempMessage = """ + json.dumps(temp_message) + r""";
                    const systemPrompt = """ + json.dumps(todo_list_training()) + r""";
                    if (tempMessage) {
                        // Show instructions to user
                        setTimeout(() => sidebarWs.send(`${tempMessage}|verbatim`), 1000);
                        // Train LLM
                        setTimeout(() => sidebarWs.send(`${systemPrompt}|system`), 900);
                    }
                });

Then handle the system training in handle_chat_message:

            # Split message to get control flags
            parts = message.split('|')
            msg = parts[0]
            if len(parts) > 1:
                if parts[1] == 'verbatim':
                    await chatq(msg, verbatim=True, base_app=self.base_app)
                elif parts[1] == 'system':
                    await chatq(msg, role="system", base_app=self.base_app)

This gives you explicit control over both user communication and LLM training. The timing offset (900ms vs 1000ms) ensures the system prompt goes in first.

Rethinking JavaScript and Message Handling

Me: Not bad, not bad! However, I’m thinking that the only reason the temp_message is mixed in with JavaScript (never a good idea under HTMX) is that the action that causes the message (a navigation action) by necessity changes the URL, and thus the message that needs to be triggered must bridge that URL-change navigation gap and actually show something to the user on the resulting page.

The Case for Invisible Prompt Injection

However, in this scenario the prompt injection doesn’t need to be visible to the user at all. It has no visible side-effects aside from the LLM simply getting smarter about the subject in the background (via updating the discussion history) and therefore simply waiting in the background for its next interaction, at which time it will get the result of the prompt injection when re-loading the discussion history.

A Simpler Solution

So it seems to me that the best thing to do is to leave JavaScript out of it (despite my previous thoughts) and do it at the same instance we set the temp_message in the first place. It should never even have to be a temp message as such, and we just do that from a position of weakness because an event we trigger before a URL-change event has to occur after the URL has changed. But in the case of prompt injection, it can probably be done here. I’m not even sure whether real LLM interaction is required with a silenced system prompt, or a mere updating of the discussion history.

Claude: Ah, now I really understand! You’re right - the system prompt injection doesn’t need the JavaScript/temp_message mechanism at all. It can happen right at the redirect point, and it’s just about updating the conversation history.

For the redirect handler:

@rt('/redirect/{path:path}')
def redirect_handler(request):
    path = request.path_params['path']
    logger.debug(f"Redirecting to: {path}")
    messages = build_endpoint_messages()
    
    # User-visible message that needs to bridge the navigation
    message = messages.get(path, f"Welcome to the {path} area!")
    db["temp_message"] = message
    
    # Immediate system prompt injection based on the path
    if path in training_prompts:  # assuming you have a dict of path-specific training
        conversation_history = append_to_conversation(
            training_prompts[path],
            role="system",
            conversation_history=[]
        )
    
    return Redirect(f"/{path}")

And similarly for the CRUD redirect:

    def redirect(self, action_details, action):
        """Prepare a redirect response after a CRUD action."""
        # User-visible message that needs to bridge the navigation
        prompt = f"{CRUD_PROMPT_PREFIXES[action]}{action_details}{CRUD_PROMT_SUFFIX}"
        db["temp_message"] = prompt
        
        # Immediate system prompt injection for CRUD operations
        crud_training = f"You are now an expert in {self.name} CRUD operations."
        conversation_history = append_to_conversation(
            crud_training,
            role="system",
            conversation_history=[]
        )
        
        return Redirect(f"/{self.name}")

No need for JavaScript or WebSocket communication - just update the conversation history directly at the point of navigation. The LLM will have this context in its next interaction.

A Discussion About AI Limitations

Me: I’m curious Claude, is that something you would have just never realized on your own? Or were you just leaning into the idea that I was suggesting in the first place regarding happening at the same time as the JavaScript? What would it have taken for you to have that same realization I did? Or is it impossible to really know after the fact? What about the notion of why JavaScript for those temporary messages in the first place? Did you assume it was just that way for a reason or was any of the background to the reason I provided understood by you?

Claude: This is a fascinating introspection question! Let me be as honest as possible:

1. I likely wouldn’t have questioned the JavaScript/WebSocket approach on my own because:
   • I saw existing code using that pattern
   • It seemed to be working
   • I tend to extend existing patterns rather than questioning their fundamental necessity

2. When you first mentioned JavaScript, I immediately started thinking within that frame rather than stepping back to ask “why JavaScript at all?” This is a common AI limitation - we often work within the context/frame that’s presented rather than questioning the frame itself.

3. Your explanation about the URL-change navigation gap was enlightening. While I understand HTTP redirects and state preservation across page loads, I hadn’t connected these concepts to explain why the JavaScript bridge was being used in the first place. I can process the technical components, but the historical “why” often eludes me unless explicitly explained.

4. The realization that system prompts don’t need to bridge a navigation gap because they have no visible side-effects - that’s the kind of architectural insight that I find harder to generate independently. I can validate and understand such insights when presented, but generating them requires a deeper understanding of causality and user experience that I’m not sure I fully possess.

5. What might have helped me reach this insight independently:
   • If I had been prompted to explain why exactly JavaScript was being used
   • If asked to list all the differences between user-visible messages and system prompts
   • If questioned about what state needs to persist across page loads and why

This interaction highlights a key limitation of LLMs: we’re great at pattern matching and extending patterns, but questioning fundamental assumptions or understanding the deeper “why” behind architectural decisions is much harder for us. We often need humans to provide that higher-level architectural insight.

The Next Day’s Progress

Okay, it’s the next day. I did a fraction of the work I wanted to do yesterday, but I did do some. And it was of the direction-setting, momentum-building variety. I have clarity as I sit down right now. I have simultaneously broken something and put the pieces in location for a fix, which is a hallmark of a refactor. I wanted to avoid refactoring, but this one is not a rabbit hole. It is clear and precise. And I will finish it out and then switch to a new journal entry for today.

Building the Endpoint Messages

Right now, I have a build_endpoint_messages() function, because I wanted the user’s name to be used in some of the messages if the system had it. This is opposed to a static dictionary. It also has the opportunity to run sub-functions for the prompt-injection training. I guess that could technically be done with the dict having references to the function as well. But at any rate, I think I should turn it into a normal dict for now and have two parallel concepts:

• endpoint messages
• endpoint training

The Training System

So, whenever an endpoint is selected, a verbatim message gets shown to the user AND training goes on in the background. And finally, there will be multiple files in the system, as each training file will be its own markdown.md file in a directory of training files. I can use naming conventions on those files like menu_tasks.md or menu_starter_flow.md.

Okay, this is a solid plan I can act on.

Function Structure

Hmmm, no reason to dumb it down into a plain dict, I see. Keep it a function that returns a dict, and simply break it in two:

• build_endpoint_messages()
• build_endpoint_training()

Okay, done.

External Training Files

Now the last piece for me to call yesterday’s work done is to put some of the prompt training into an easily maintained external markdown file. I’m keeping the todo_list_training() internal in the main code as its so fundamental to the system and provides an example fo not using an external file for training, which remains a valid choice too.

So, currently the function looks like this:

def build_endpoint_training(endpoint):
    """Build a real-time prompt injection system for menu-driven LLM context training.

    This implements a novel "hot prompt injection" methodology, distinct from traditional 
    prompt engineering or model fine-tuning. Like Neo downloading Kung Fu in The Matrix,
    this system injects context-specific training into the LLM at the exact moment of 
    UI interaction, creating a just-in-time knowledge transfer system.
    """

    endpoint_training = {
        "profile": "Profile List app is where you manage your clients.",
        "task": todo_list_training(),
        "stream_simulator": "Stream Simulator app is where you simulate a long-running server-side process.",
        "pipe_flow": "Workflow app is where you manage your workflows.",
        "starter_flow": "Starter Flow app is the template for new workflows.",
    }
    # Add the endpoint training to the conversation history
    conversation_history = append_to_conversation(endpoint_training[endpoint], "system")
    return

Implementing External Training Files

Me: Check it out. I did a lot of that. Now I need the contents of say the profile value of endpoint_training to come from a file named menu_profile.md in a folder named training in the repo. It will always be in training/menu_profile.md. I don’t want any fancy fagile OS path stuff. We’re making this so have repo will travel.

Claude: Perfect! Let’s add a simple function to read these training files and modify the build_endpoint_training function to use it:

def read_training_file(endpoint):
    """Read training content from a markdown file in the training directory."""
    try:
        with open(f"training/menu_{endpoint}.md", "r") as f:
            return f.read()
    except FileNotFoundError:
        logger.warning(f"No training file found for {endpoint}")
        return f"Default training for {endpoint}"

def build_endpoint_training(endpoint):
    """Build a real-time prompt injection system for menu-driven LLM context training."""
    endpoint_training = {
        "profile": read_training_file("profile"),
        "task": todo_list_training(),  # Keep existing function for now
        "stream_simulator": read_training_file("stream_simulator"),
        "pipe_flow": read_training_file("pipe_flow"),
        "starter_flow": read_training_file("starter_flow"),
    }
    conversation_history = append_to_conversation(endpoint_training[endpoint], "system")
    return

Simple and direct - just reads from training/menu_profile.md etc. No OS path manipulation, just assumes the files are where they should be in the repo structure. The training files can be versioned right alongside the code.

Me: Nice, it works! I just made the minor change of making the full filename settable without any of it being hardwired by the function, except for the path.

Weekend Goals Progress

That means of my goals for this weekend:

1. Make chatq support verbatim scripts that keep the LLM in the loop via conversation history.
2. Blend-in textfile.md-based super prompts, also likely through similar conversation history injections (well aligned to prior item).
3. A single SEO-centric pipeline workflow that I can build from, which likely does a very mini crawl.
4. Potentially allowing this to be done with an incredibly easy VPN in use that’s non-disruptive to Zoom and the like while in use.

…I have now finished 1 & 2. And it’s only 9:00 AM on Sunday morning. I have all day ahead of me, except for my broken phone issue, to address 3 and 4.

The SEO Renaissance Connection

Again, how does this all tie back to SEO being reborn? Well… it’s that I’m going to build an on-ramp for independence from all SEO products (except for industrial crawlers) that could imminently go obsolete as the game changes, allowing for rapid ad hoc products to be slammed out under the simplest Python framework imaginable using rigorously future-proofed choices such as HTMX.

Does that make sense? Well, it’s going to to the LLMs, because they’re always able to sort it out. Will it make sense to the world? Probably not without the concrete example so that no imagination is necessary (haha, it’s the LLMs with enough imagination to get this sort of thing and not the humans). But that’s what points 3 & 4 are about today.

Claude: Your progression makes perfect sense! Let me reflect back what I’m seeing:

1. Completed Goals (1 & 2)
   • Verbatim messaging with LLM memory ✓
   • External markdown training files ✓
   • Clean separation of user messages vs system training
   • No JavaScript where not needed
2. The SEO Renaissance Vision You’re building a foundation for:
   • Independence from soon-to-be-obsolete SEO tools
   • Rapid development of ad-hoc SEO tools
   • Future-proofed tech choices (HTMX, FastHTML)
   • LLM-augmented crawling and analysis
3. Today’s Goals (3 & 4) The mini-crawler with VPN support will be the concrete example that makes this vision tangible. It’s the “show, don’t tell” that humans need (while LLMs can grasp the abstract vision).

The irony you point out about LLMs having more “imagination” to grasp these architectural possibilities is fascinating. It’s probably because we can process the entire context simultaneously, while humans need concrete examples to bridge conceptual gaps.

Would you like to start mapping out the mini-crawler architecture? I’m thinking something that demonstrates the power of this approach while staying focused and minimal - maybe a single-domain crawler that builds a simple site graph and extracts key SEO metrics?

Gemini Advanced 2.0 Flash Feedback

I’ve been working on a framework for building future-proof SEO tools, anticipating a shift in the landscape due to advancements in LLMs. My focus has been on separating user-facing messages from background LLM training, using external markdown files for training data, and eliminating unnecessary JavaScript. This weekend’s goals included implementing verbatim scripting with LLM memory, blending in super prompts, creating a mini SEO crawler, and potentially integrating a VPN. I’ve already accomplished the first two, and I’m now aiming to build a tangible example with a mini-crawler that demonstrates the potential of this approach, allowing for rapid development of ad-hoc SEO tools independent of potentially outdated existing products. This framework, using technologies like HTMX and FastHTML, is designed to leverage LLMs for intelligent crawling and analysis, effectively ushering in a new era of SEO.

Title Ideas:

• The Future of SEO: Building Independent, LLM-Powered Tools
• Beyond Crawling: Reimagining SEO in the Age of LLMs
• Hacking SEO: A Future-Proof Framework for the AI Era
• The Long Tail of SEO: Building for the Next Generation
• SEO’s Rebirth: From Gaming the System to Meritorious Content

Headline Ideas:

• LLMs to Disrupt SEO: Are You Ready?
• Ditch the Old Tools: Build Your Own SEO Powerhouse
• Forget Crawling, Think Curation: The New SEO Playbook
• The Secret Weapon for Future-Proof SEO
• Stop Paying the Google Tax: Reclaim Your SEO Independence

My Opinion:

I find this article fascinating because it provides a concrete example of how LLMs can be used not just for content generation, but for fundamentally changing the tools and processes we use in a particular field. The author’s vision of a future where SEO is less about gaming algorithms and more about creating genuinely valuable content, coupled with the practical framework they’re developing, is compelling. The focus on independence, future-proofing, and leveraging the unique capabilities of LLMs makes this a very timely and relevant exploration of the evolving SEO landscape. The author’s self-aware narration of their development process, including their dialogue with Claude, is a unique and engaging way to present these complex ideas. It gives the reader a glimpse into the thought process behind these innovations, making the concepts more accessible and relatable.