AI's New Deal: From Ideas to Automation

By Mike Levin

Introduction

This article chronicles a personal journey through the evolving landscape of technology, driven by the author’s quest to bridge the gap between human intention and machine execution. It highlights the transformative potential of AI, particularly large language models (LLMs), as “universal translators” that convert imprecise human language into precise machine code, thereby democratizing automation and empowering individuals to bring their ideas to fruition.

The author emphasizes the importance of local, open-source LLMs like Llama and Gemma for privacy and accessibility, contrasting them with the cost-prohibitive, centralized models offered by tech giants. Through a blend of technical insights, philosophical reflections, and practical examples from their own SEO software project, Pipulate/Botifython, the author argues that we are on the cusp of a societal recalibration akin to the Industrial Revolution.

This new era, fueled by the convergence of AI, open-source principles, and established internet protocols, promises to empower a new class of “digital bourgeoisie” who can leverage these tools to achieve unprecedented levels of agency and innovation. He urges readers to embrace this change, shed outdated notions, and seize the opportunity to become outliers who redefine the new normal.

The Value of Rest and Action

I’m getting good night’s sleeps but at the expense of not doubling or tripling my workday. Sleep is the best medicine, along with laughter. Laugher and sleep, now there’s a good combo. Isn’t there an Irish saying? Googling now… Yep apparently, I still Google even though Perplexity.

“A good laugh and a long sleep are the two best cures for anything.”

But then there’s also, “Sleep when you’re done, not when you’re tired” and that resonates with the whole shrinking window of opportunity thing, which we are most definitely in. Wait too long, and somebody else does it and you lose the early mover advantage. You can also lose the motivation and that special spark of insight. Good ideas are common. The will, determination and indeed agency to actually act upon it and implement, now that’s rare.

The Pandemic’s Silver Lining

The work from home opportunity we received from the pandemic right as the information age began firing on all cylinders in great part gave that opportunity, because the soul-crushing commute part of the daily-grind became optional. That’s close to two-hours a day we can plow right back into stuff like this, agency-generating journaling. Journaling to fan the flame of agency. It’s the machine that turns good ideas into implementation — if you can get the tricks down. (otherwise, it’s just 2 more hours of doom-scrolling)

The Bridge Between Speaking and Coding

A friend of mine when I discussed the idea of writing a book to either teach people how to become more technical and become a developer versus teaching how to do self-reflective journaling with vim held the strong opinion I should go the developer direction. I was doubtful.

The Value of Journaling First

My view is to first teach one (play up mindfulness, idea-dredging, flow-state, etc.), and then the other. Journaling is that strange missing piece between deciding that you want to express yourself in a way that results in more precise automation, based on the premise that speaking an coding are really the same thing. You move from fleshing it out in the abstract, to the precise — knowing you’ll have big-time help with the precision, if only you make yourself very clear. First, stories. Then code.

The Language of Humans and Machines

Speaking encodes ideas to be played back in someone else’s head, while programming encodes ideas to be played back in a machine, which has the additional useful side effect of carrying out those ideas as explicit instructions like a player piano — genuine automation as opposed to trying to compel people into action.

Creative Flow in Text Editing

Starting out by writing journaling-style into a text-editor without the precise demands of programming loosens up the creative muscles and allows you to spontaneously riff into a text-file as if a improv jazz musician. That free-formedness is now more valuable than ever, because the LLMs can help you tighten it up and formalize it. Writing into a text-editor for journaling is encoding ideas in the spoken language. Writing code into a text-editor is encoding for machine interpretation.

It’s encoding ideas into a text-editor in either case (yes, vim for journaling). Getting good at one is getting good at the other.

Historical Parallels in Automation

The concept of the player piano or the automated knitting loom are probably the two great transitional thoughts between the worlds of the human everyday experience and the world of Turing machines and coding. Or maybe music boxes! Music is a language that touches us emotionally, often played back through memorized and automated patterns resembling programming, and at some point we were actually able to do that mechanically by raising bumps on a rotating cylinder to brush against tone-generating brushes plucked by the bumps as the cylinder rotates. Player pianos used spools or scrolls with holes punched in them to similar effect. And knitting looms, the same trick again still in order to weave fabric instead of sound.

The Evolution of Programming Languages

So all a coder or a programmer is, is a speaker of another language— a language intended to be loaded into a machine for interpretation instead of another human’s head. And like any language, the rules, syntax, nuances and subtitle are different, and usually very strict and unforgiving in the case of machines. And you know what? That’s really the change that the rise of the age of artificial intelligence brings to the picture. Those instructions no longer need to be so machine-perfect anymore to be carried out by a machine. Due to teaching a machine everything about our spoken languages by making LLM models and the inference engines we load them into that predict and auto-complete, all that has gone before becomes an interpretation layer for the fuzzy intent and meaning that usually weighs down our first-pass attempt at expressing things.

LLMs as Universal Translators

To fully appreciate this, one must imagine that the machine can equally understand two or more languages and use this auto-predict ability to translate. Whether intelligence in the human sense or not, this provides stunning evidence of the reality, utility and value of AI. The Information Age is becoming the Age of AI, because the information is becoming alive. Information (the LLM / large language models) plus the runtime engines they’re loaded into (inference engines), become player pianos or knitting looms — not for merely the responses they give in our own spoken languages, but for their ability to translate into the raised bumps on the spools, or holes punched into scrolls. LLMs are universal translators and enablers of machine automation for those who can’t express that way so easily. Precision coding for machines is hard. Or at least doing it so richly, comprehensively and well as to have a meaningful impact on your environment is hard.

The Programming Revolution

The surprise that machines could program is key to everything. It is what turns them into engines of economic value, because otherwise it’s just filling the world with more inane banter and prattle — entertaining and novel, but ultimately annoying. But a talking frog… that’s cool. Sorry, engineering joke there. But the point stands. Having a component that takes the loosey goosey spoken words of we biologicals and turns it into the precision words of machine player engines like pianos, looms and 3D printers makes all the difference. Due to how this uniquely enables humans who have been waiting, or perhaps even futilely trying, to express themselves thus…

The Power of Capturing Ideas

Well, this. What I’m doing right here, right now. I’m using free-form writing, stream of consciousness idea-dredging of the sort people joke only happens in the shower or when you’re driving and can’t really act on them, into a form where you capture-as-you-go, and actually can really act on them. In fact, the mere act of capturing the ideas in writing as they occur to you is a very real form of acting on them now, because that all becomes context for the machine that can interpret your intent and help you with your hard next-steps of machine-precision automation.

The Significance of AI Translation

People like to downplay its significance of the rise of AI, but the thing to focus on is the importance of that translating trick: the fuzzy loosey goosiness of the spoken word into the precision of the machine instruction. It quite literally converts ideas expressed in a way that could only ever play back in other peoples’ minds, and therefore “soft” automation (due to having to convince people to do your bidding), to “hard” automation whereby the ideas can be loaded into a machine that will immediately and without hesitation do your bidding.

Function Execution in LLMs

There’s this funny debate out there in the world of LLM model building as to whether the model can execute functions or not. Whether it can have the bits in it that bridge the gap between the spoken world and the machine instruction world. Function execution in models was touted as some feature exclusive to OpenAI, and then others caught up. Now the fallacy in all this is that if the model has been trained on enough English, Python and JSON (or whatever), it’s going to be able to construct good JSON for you (encoding the data) and Python for you (the part that handles the encoded data). And then if you tell the LLM that any thing it expresses with the former is handled by the later… BAM! Function executor.

Practical Applications

Now you might think I talk about this in theory only, but not true. My Pipulate / Botifython free and open source SEO software project has blown way past that point. I can teach pretty much any model smart enough to learn the rules to execute functions. I have it adding adding items to a To-Do list upon request because it knows how to translate between expressed requests and the JSON data structure that causes the To-Do record creation to occur when it spots the correct JSON response in the LLM’s response.

The Power of Local Models

And I do this not with the cost-burdened frontier models like OpenAI ChatGPT, Google Gemini or Anthropic Claude. No, I do this with the lightweight local models like Meta’s Llama and Google’s Gemma for the price of electricity. Llama and Gemma can execute functions if your prompt teaches them how. It’s generally not believed because of the less-smartness of these models and how quickly the system prompts fade in the discussion context window, leading to the model’s tendency to forget. This is solved with hot prompt injections— the Neo Kung Fu download of The Matrix, a trick you can do just-in-time before the prompt request or less efficiently (memory wipingly) inline with it.

The Future of Local AI

The idea is that once you’re not inhibited by paying for every off-site giant http roundtrip, pennies-per-thousand queries or however cheap DeepSeek drives this sort of stuff — cheap, but not uninhibitingly so… once you’re no longer inhibited and AI prompting is local and intelligence becomes ubiquitous and private, there’s no reason that it can’t be kept in the loop on every action you take in a piece of software.

Privacy and Efficiency

And this is not that difficult imprecise type that uses computer vision to parse your whole screen to achieve some sort of browser automation of the sort both Anthropic and Google are demonstrating these days. No, that’s computationally expensive, privacy intrusive snoop-ware. I’m talking about giving a local LLM, these days running on Ollama/llamacpp but in the future, whatever. And you give this local AI intelligence the inside scoop in a logical and API-driven way from the internals of your program, so your local AI just knows, and you don’t have to worry (quite so much) about privacy because what happens on localhost stays on localhost. There’s no calling back to mama to answer your prompts.

The Power of Local Automation

Now here’s the amazing part. Because this local LLM has the same ability to learn and translate to real machine automation from the human spoken language, your local LLM can automate your local software! The playing of the piano or the knitting loom becomes the learning how to use whatever software that is as well and eventually better than you. Got a process? Just do it for awhile and the machine will take over. Worried about teaching big tech all your domain expertise so that their models can have your domain experise competitive advantage? No problem! Train your own damn models. Better, more distributed genies in better, more distributed models.

Data Collection and Model Training

Now I know there’s this barrier right now about actually training your own models. It’s still enormously expensive and really unrealistic for it to learn from every little interaction it has with you. That’s why you capture everything into more transitional databases, also local. With two or three such databases digesting everything that goes in from discussion (which now includes meta data concerning operation of the local software its embedded into) to whatever other documents, web-crawling or whatever you want it to digest. It can go into vector embedding databases for similarity-lookups, graph databases for categorization purposes, key/value-pair (NoSQL) databases for fast-and-furious capture to be sorted out later (or cherry-picked for top-of-mind keys). Whatever!

The Value of RAG

Point is, capture now. Train actual new core models later. Capturing now has enormous value because the captured data is used during playback in retrieval augmented generation (RAG) without the extra expense of having to bake it into the core model. This has the additional advantage of being able to A/B-test LLMs as you go, swapping them out but also them having all the same RAG data lookups. And how do these new models that you swap in and out know their own capabilities? Through the same hot-prompt-injection Neo Kung Fu downloads as the prior LLM. When the To-Do menu item is chosen, you teach the LLM the JSON structure of how to do a To-Do item insert (or edit, delete and sort).

Building for the Future

Kapish? LLM Legos! The “soul” of the machine is split between nature and nurture. You load in different natures in the form of LLMs (currently, as downloaded from the ollama.com). You incubate and nurture local nature. And if you make intelligent nature choices, they are future-proofed; hot-swappable between infinite different LLMs forever-forward, until the day you can take the next step and bake all that into your core model because Moore’s Law and the next Nvidia McGuffin has made doing so at home realistic. Everyone wins! Oh, except for the cash-register in the cloud that will be hearing significantly less Ka-CHING’s on your every request.

From Theory to Practice

Theory? Nope. Fact. Free and Open Source SEO Software, already today. Well, almost. You can see some previews of it in the Pipulate repo on my GitHub account. I’m waking up early and going to sleep late to get this work done, because I’m keenly aware I’m in a sprint and not a marathon. Things are being pushed ahead fast, and this sort of project has a small and shrinking moat. Moats are the competitive advantage head-start that makes it difficult for competitors to reproduce what you do — a concept popularized by Warren Buffett as a criteria in choosing investments; “how big is their moat?”

Open Development Philosophy

So why am I even talking about all this out in public? Well, because the iron is hot. I’m in the thick of it, and I’m not the secretive sort. Everything I do will be API-wired to my employer’s enterprise crawler software first. There will be great utility in it without that connection. It’s value as FOSS SEO software will just skyrocket and be useful to the big enterprise clients with money if you do have that Botify account. For everyone else, it will be a curious widget, because nobody’s gonna “get it” for a good long while.

Historical Parallels in Tech Adoption

Remember the resistance to Ruby on Rails, and then the surprising enterprise disrupting startup-enabling up-take that birthed Twitter and the like? Whole new endeavors become possible because the tools change and fuel disruption. The energy-in and phase-space of potential fruits of your labor out gets recalibrated. Is that too high fallutin language? What’s possible changes and comes within the reach of more people. And those more people may have been waiting a lifetime for a tool like this to express like that. But the tool’s different than anything that came before, so in the domain of early adopters, and antithetical or insulting to the habit-driven flock. And that’s the moat.

The Future of Web Development

React is cool, but React isolates you from the 36 year-old http specification that birthed the Web, super-charged hypertext, and remains timeless while the other technologies leaching off it like a remora on a whale. Or maybe like barnacles on a ship? Well, you might think I’m about to criticize CSS and JavaScript as barnacles that need to be scraped off HTTP, but not so! I advocate HTMX as a sort of magic fairy-dust to be sprinkled onto the HTTP protocol to give it another 36 years. You think I kid? The TCP/IP protocol on which HTTP lives is 43 years old. And Unix is exactly my age, 54 years old. That’s a half-century for what becomes the pee in the pool of tech, folks. It’s not coming out without draining the entire pool, and AI isn’t going to do that. It’s swimming in the same Unix + TCP/IP + HTTP waters as everyone else for a good long while, and therein lies your future-proofing — or at least resisting obsolescence.

The Case Against React

How? Refactoring your web development around HTMX instead of React. React won’t survive the hardware platform shifts, like Macromedia/Adobe Flash and Shockwave didn’t survive the shift to mobile, and GeoCities design didn’t survive the shift… oh, to mobile again. CraigsList excepted. And that’s not just because it’s a faddish snapshot in time as far as development styles go. No, it’s because it requires the execution of JavaScript to get at all that data hidden in your single-page application (SPA), which can’t be crawled without hundreds or thousands of times the computational cost of just loading the source HTML. But Google can read JavaScript! Yes, yes. But what about everyone else? Are they going to build out a Google-scale infrastructure just so they can load your React app?

The Case for HTMX

But HTMX is JavaScript too! Yes, but the least possible JavaScript necessary to sprinkle in superpowers, and element-by-element aligned to the HTML specification, thereby keeping you close to the timeless specification. Any weird SPA-like appendages you build on an HTMX page (and yes, you still can) forces you to look directly at the perfectly fine real DOM and understand what’s really going on in the browser instead of some Facebook-invented virtual DOM in React. Unnecessary and trend-driven 3rd-party abstraction layers are vulnerabilities to future-proofing your code and your skills. The opposite argument is usually given that layering in abstractions helps future-proof, but it’s a question of what’s more stable in the long-run. The HTTP specification or ReactJS?

Platform Evolution

So there’s hardware platform shifts. Desktop becomes mobile. Mobile becomes real-time crawlable by LLMs using web-search RAG systems.

Information Architecture

But then there’s just outright good design. There’s organizing information. There’s clear-headed thinking and good solid story-telling around how you organized that information. This involves hierarchy. Taxonomies. Onto… Epes… uhh… uhh… Perplexity?

Perplexity: Ontology, epistemology, hierarchy, and taxonomy are interconnected concepts in philosophy and information science, but they have distinct meanings and applications:

Ontology is the study of existence and the nature of reality[1][4]. It addresses questions about what exists and how entities relate to each other[3]. In information science, an ontology describes the structure of entities and their relationships[3].

Epistemology focuses on the nature and scope of knowledge[1][4]. It examines how we acquire knowledge, what constitutes knowledge, and how we can be confident in what we know[2][5].

Hierarchy is a system of organizing entities in a ranked structure, often used within taxonomies and ontologies[7].

Taxonomy is a method of classification that organizes entities into groups based on shared characteristics[3][6]. It typically has a hierarchical structure with parent-child relationships[6][7].

The key differences are:

1. Ontology vs. Taxonomy:
   • Ontology is more complex, describing entities and their various relationships[6].
   • Taxonomy is simpler, focusing on hierarchical classification[6].
2. Ontology vs. Epistemology:
   • Ontology deals with what exists (the nature of reality)[1][4].
   • Epistemology concerns how we know what exists (the nature of knowledge)[1][4].
3. Hierarchy in Taxonomy and Ontology:
   • In taxonomies, hierarchy is central, using parent-child relationships[7].
   • In ontologies, hierarchy is one of many possible relationship types[7].

While these concepts are distinct, they often interrelate. For example, ontological assumptions can influence epistemological approaches, and taxonomies can be implemented within ontologies[3][4].

Citations:

[1] https://www.restack.io/p/ai-ontology-creation-tools-knowledge-answer-ontology-vs-epistemology-cat-ai
[2] https://www.webpages.uidaho.edu/engl257/classical/ontology_and_epistemology.htm
[3] https://deanebarker.net/huh/ontology-taxonomy/
[4] https://conceptshacked.com/epistemology-and-ontology/
[5] https://theperformancesolution.com/earth-ontology-epistemology/
[6] https://janzz.technology/ontology-and-taxonomy-stop-comparing-things-that-are-incomparable/
[7] http://www.hedden-information.com/taxonomies-vs-ontologies/
[8] https://i2insights.org/2017/05/02/philosophy-for-interdisciplinarity/
[9] https://study.com/academy/lesson/ontology-vs-epistemology-differences-examples.html

Me: Thank you, Perplexity! Most helpful. Now say something about Aristotle, Kant and Darwin.

Perplexity: Aristotle, Immanuel Kant, and Charles Darwin are three influential thinkers who made significant contributions to philosophy and science, shaping our understanding of logic, ethics, and evolution.

Aristotle (384-322 BCE) was a pioneering Greek philosopher who laid the foundations for many fields of study. He developed syllogistic logic, which remained the dominant form of logical reasoning for centuries[1]. Aristotle’s work spanned various disciplines, including physics, biology, metaphysics, ethics, and politics[4]. His approach to scientific reasoning, emphasizing empirical observation and experimentation, was groundbreaking for his time[1].

Immanuel Kant (1724-1804) was a German philosopher who made substantial contributions to moral philosophy and epistemology. He proposed the concept of the “categorical imperative” as a fundamental principle of ethics, arguing that morality must be based on reason rather than subjective interests[2]. Kant’s philosophy emphasized human freedom and autonomy, considering them essential for moral responsibility[5].

Charles Darwin (1809-1882) revolutionized biology with his theory of evolution by natural selection. Darwin proposed that species change over time, giving rise to new species that share common ancestors[6]. His theory is based on the principles of variation, inheritance, and selection, explaining how organisms become adapted to their environments over generations[9]. Darwin’s work, particularly his book “On the Origin of Species” (1859), fundamentally changed our understanding of life’s diversity and development[9].

These thinkers’ ideas continue to influence modern thought and scientific inquiry, demonstrating the enduring impact of their contributions to human knowledge.

Citations:

[1] https://www.thecollector.com/aristotle-life-works-philosophy/
[2] https://ethics.org.au/big-thinker-immanuel-kant/
[3] https://www.pewresearch.org/science/2019/02/06/darwin-in-america-2/
[4] https://www.britannica.com/summary/Aristotle
[5] https://thegreatthinkers.org/kant/introduction/
[6] https://www.khanacademy.org/science/ap-biology/natural-selection/natural-selection-ap/a/darwin-evolution-natural-selection
[7] https://www.britannica.com/biography/Aristotle
[8] https://plato.stanford.edu/entries/kant-moral/
[9] https://www.livescience.com/474-controversy-evolution-works.html
[10] https://embryo.asu.edu/pages/aristotle-384-322-bce
[11] https://www.reddit.com/r/askphilosophy/comments/3jfyen/can_someone_give_me_an_eli5_on_immanuel_kant_and/
[12] https://www.pewresearch.org/religion/2009/02/04/darwin-and-his-theory-of-evolution/

Me: Excellent, but now why did I ask you about these 3 individuals in relation to hierarchy, taxonomy, ontology and epistemology? And what does it have to do with good web design?

Perplexity: You asked about Aristotle, Kant, and Darwin in relation to hierarchy, taxonomy, ontology, and epistemology because these thinkers made significant contributions to these concepts, which are fundamental to organizing and understanding knowledge:

1. Aristotle laid the foundation for categorical thinking and hierarchical classification. His work on categories and logic influenced the development of taxonomies and ontologies[1][7].

2. Kant’s theory of knowledge and his categories of understanding relate directly to epistemology and how we organize information. His ideas about structuring knowledge have implications for information architecture[7].

3. Darwin’s work on evolutionary theory revolutionized biological taxonomy and hierarchy, demonstrating how species relate to one another over time[2].

These concepts are relevant to good web design in several ways:

1. Information Architecture (IA): Taxonomies and ontologies are crucial for organizing content on websites. A well-structured IA improves user experience by making information more accessible and intuitive to navigate[5][8].

2. User Experience (UX): Understanding the user’s mental models (ontology) helps designers create more intuitive interfaces. The choreography between ontology and taxonomy in UX design leads to more satisfying user experiences[8].

3. Content Strategy: Taxonomies help in categorizing and tagging content, making it easier to manage and retrieve information on websites[2].

4. Semantic Web: Ontologies are fundamental to the Semantic Web, enabling machines to understand and process information more effectively, which can enhance web functionality[3].

5. Knowledge Management: Hierarchical structures and ontologies help in organizing and representing complex information systems, which is essential for large-scale web applications[6].

By understanding these philosophical and scientific concepts, web designers can create more logical, user-friendly, and efficient digital experiences that align with how humans naturally organize and process information[9].

Citations:

[1] https://plato.stanford.edu/entries/aristotle-categories/
[2] https://www.nngroup.com/articles/taxonomy-101/
[3] https://www.w3.org/submissions/WSMO-primer/
[4] https://philarchive.org/archive/ONEIDO
[5] http://www.hedden-information.com/taxonomies-vs-ontologies/
[6] https://caminao.blog/knowledge-architecture/ontologies-ea/
[7] https://pmc.ncbi.nlm.nih.gov/articles/PMC10397139/
[8] https://fuzzymath.com/blog/information-architecture-ux-design/
[9] https://www.forbes.com/sites/cognitiveworld/2019/03/24/taxonomies-vs-ontologies/

Me: Wow, you’re on fire Perplexity. Okay, how about this zinger: What impact will single page application (SPA) design which requires the execution of JavaScript to access a page’s information have on real-time retrieval augmented generation (RAG) small-world-theory web-crawls LLMs will increasingly do to create better responses?

Perplexity: The impact of single page applications (SPAs) on real-time retrieval augmented generation (RAG) for LLMs performing small-world-theory web crawls is multifaceted and potentially significant:

1. Crawling Challenges: SPAs that rely heavily on JavaScript for content rendering may pose difficulties for traditional web crawlers[1]. This could limit the ability of LLMs to access and index information from these applications, potentially reducing the diversity and completeness of data available for RAG.

2. Dynamic Content Retrieval: RAG systems may need to evolve to handle the dynamic nature of SPAs, potentially requiring the integration of headless browsers or JavaScript rendering capabilities to access content that is not immediately available in the initial HTML[7].

3. Real-time Data Accuracy: SPAs often provide more up-to-date information through dynamic content loading. This could enhance the accuracy of RAG responses if the crawling mechanism can effectively capture this real-time data[2][5].

4. Network Efficiency: SPAs typically transfer smaller amounts of data after the initial load, which could potentially speed up the crawling process for RAG systems that are optimized to handle this architecture[9].

5. User Experience Considerations: The improved user experience of SPAs might lead to more engaged users and potentially richer, more interactive content for LLMs to analyze[4][9].

6. Small-World Theory Applications: The dynamic nature of SPAs could potentially enhance the application of small-world theory in web crawls by allowing for more efficient navigation between interconnected content nodes[6].

7. Indexing Complexity: The single-page nature of SPAs may require more sophisticated indexing strategies for RAG systems to effectively categorize and retrieve information from different “virtual pages” within the same URL[1][7].

8. API-driven Content: Many SPAs rely on APIs for content delivery, which could provide a more structured data format for RAG systems to consume, potentially improving the quality and relevance of retrieved information[2][5].

To address these challenges and capitalize on the opportunities, RAG systems and LLMs will likely need to adapt their crawling and indexing strategies to effectively handle SPA architectures. This may involve developing more advanced JavaScript execution capabilities, implementing sophisticated content extraction techniques, and refining algorithms to navigate and understand the dynamic structure of SPAs within the context of small-world theory web crawls.

Citations:

[1] https://www.netguru.com/blog/what-are-single-page-applications
[2] https://www.confluent.io/learn/retrieval-augmented-generation-rag/
[3] https://neuro.bstu.by/ai/To-dom/My_research/Papers-2.0/STDP/Learning-by-STDP/1/Small-World/bjorneborn_2005_small-world_network_exploration.pdf
[4] https://www.ramotion.com/blog/single-page-application-architecture/
[5] https://www.immuta.com/guides/data-security-101/retrieval-augmented-generation-rag/
[6] https://www.semanticscholar.org/paper/The-Small-World-Web-Adamic/654306d99d488293732bac2d64b363717ed94e94
[7] https://www.imaginarycloud.com/blog/single-page-applications
[8] https://www.elastic.co/what-is/retrieval-augmented-generation
[9] https://www.magnolia-cms.com/blog/all-about-single-page-applications.html
[10] https://www.ibm.com/think/topics/retrieval-augmented-generation

Me: But at what cost, Perplexity? Wouldn’t it provide competitive advantage for a site to use superior hierarchy for drill-down paths to allow better small-world-theory site navigation and then to embed Schema.org structured data right onto the HTML pages so that no JavaScript execution is needed for real-time data retrieval? What is the cost and resource difference between just grabbing the HTML source for a page and looking for things that semantically abide to Web standards versus rendering the DOM for information that may come in any form invented by the SPA app developer? Give me a cost differential for the crawler. And an idea of whether not making the crawler incur this cost could provide a competitive advantage to the site.

Perplexity: The cost differential between crawling traditional HTML pages with embedded structured data versus rendering SPAs for information retrieval is significant:

1. Resource Consumption:
   • HTML crawling: Lightweight, requiring minimal CPU and memory resources.
   • SPA rendering: Computationally intensive, demanding more CPU, memory, and potentially GPU resources for JavaScript execution and DOM rendering[1].
2. Time Efficiency:
   • HTML crawling: Fast, as it only requires parsing static HTML.
   • SPA rendering: Slower, due to the need for JavaScript execution, API calls, and dynamic content loading[3].
3. Scalability:
   • HTML crawling: Highly scalable, allowing for parallel processing of multiple pages.
   • SPA rendering: Less scalable, as each page may require a separate browser instance or headless environment.
4. Complexity:
   • HTML crawling: Simpler, using well-established parsing techniques.
   • SPA rendering: More complex, requiring sophisticated tools to handle various JavaScript frameworks and asynchronous loading patterns[1].
5. Maintenance:
   • HTML crawling: Lower maintenance, as HTML standards are relatively stable.
   • SPA rendering: Higher maintenance, needing frequent updates to keep up with evolving JavaScript frameworks and libraries.

The cost differential could be estimated at 2-5 times higher for SPA rendering compared to HTML crawling, depending on the complexity of the SPA and the scale of the crawling operation.

Competitive Advantage:

Using superior hierarchy for drill-down paths and embedding Schema.org structured data directly in HTML pages can provide significant competitive advantages:

1. Improved Crawlability: Easier for search engines and other crawlers to index content efficiently[4].

2. Better SEO: Enhanced visibility in search results due to more accessible and structured content[3].

3. Faster Data Retrieval: Crawlers can quickly extract relevant information without executing JavaScript[4].

4. Reduced Crawler Load: Lower resource requirements for crawlers, potentially leading to more frequent and comprehensive indexing.

5. Wider Compatibility: Structured data in HTML is accessible to a broader range of tools and services, not just those capable of rendering JavaScript[8].

6. Cost-Effective for Data Consumers: Organizations and services that rely on web data can operate more efficiently, potentially leading to better integration and visibility for the website.

By not requiring crawlers to incur the additional costs associated with SPA rendering, a website can gain a competitive edge in terms of discoverability, data accessibility, and potential partnerships or integrations with data-driven services. This approach aligns well with the principles of the semantic web and linked data, fostering better interoperability and knowledge sharing across the internet[6][8].

Citations:

[1] https://www.ramotion.com/blog/spa-vs-mpa/
[2] https://brucejonesseo.com/structured-data-tools/
[3] https://startup-house.com/blog/spa-vs-traditional-websites-business-growth
[4] https://www.botify.com/insight/structured-data-basics-using-schema-org-to-help-search-engines-understand-your-content
[5] https://www.linkedin.com/pulse/single-page-applications-spas-vs-multi-page-mpas-web-app-chituyi-icknf
[6] https://almanac.httparchive.org/en/2024/structured-data
[7] https://forum.bubble.io/t/what-do-you-think-single-page-vs-multi-page/75744?page=2
[8] https://schema.org

Me: Interesting! So it’s only 2 to 5 times more expensive to render the entire DOM with a headless browser than it is to just load the HTML source?

Perplexity: Interesting! So it’s only 2 to 5 times more expensive to render the entire DOM with a headless browser than it is to just load the HTML source?

The Inevitability of Infrastructure

Alright, so this really helps me clarify my vision. There are inevitabilities out there, and some of them can be read today in the crystal ball. Deployed tech like TCP/IP, Unix(-like) and HTTP protocols, the infrastructure on which everything is built, doesn’t change fast. Even if radical new inventions are made with the help of AI’s, build-out costs are going to have a dampening effect measured in decades. Disruption does happen, and the biggest we’ve seen is the Internet itself, and then mobile. Next is AI, but AI is an enabler of humans who’ve been waiting to implement on ideas just beyond their reach, for lack of a universal translator to precise machine code implementation.

The Rise of a New Digital Class

The just-beyond-reach becomes in-reach and a recalibration of societal classes occurs again, like the birth of the French upper middle class Bourgeoisie business owners who clawed their way up from craftspeople before the French Revolution. You see, when tools change like those that made the industrial revolution (attainable literacy because of the proliferation of books), people are able to achieve at new levels because of the fundamentally distributed and democratizing nature of those tools. Of course there’s always a battle between centralization versus distribution of the means of production. That’s happening now with AI. First, OpenAI shows what was locked-up in Google’s LaMDA labs to the public, then Meta open sources Llama. Information doesn’t always spread, but if you have someone like Mark Zuckerberg using a scorched earth policy to try to evaporate a competitor’s moat, it does.

Breaking Through the Rubber Ceiling

Welcome to the recalibration. Welcome to a new class of Bourgeoisie being born right now. There’s a bunch of loud-mouthed YouTubers right now creating a glut of information wishing they were part of that class, but really only know story-telling and video editing (an important skill, but not the whole story). Then, there are people who have felt the resistance of that rubber ceiling like me. I’ve implemented various systems in the past along the lines of Ruby on Rails (even before Rails was invented)— aka joyful frameworks. But I’ve missed the boat time and time again, because I’m starting out at a disadvantage, completely bootstrapping myself from scratch with no real relevant schooling, mentoring nor guidance to speak of.

Reaching for the Next Level

The whole story is how to cut through that rubber ceiling and to reach into the LLMs behind (above?) it, trying to lower you the drophooks to lift you up to your next level of ability to achieve.

Embracing Change and Progress

And that’s where the rest of the story comes in. It’s a uniquely open mind. Cast aside the preconceived notions. The ways of doing things inside those black boxes is going to change. The black box components themselves are going to change. Out with the Docker, in with the Nix. Nixing Docker? Haha!

The Reality of Progress

But the dampening effect of all that has gone before will continue to dampen. Hard-cost realities slow progress and even shape progress along hard-cost optimizing routes. Money is going to be squeezed out of all the tracks that have already been laid down. There are outliers like Musk and Starnet that can actually lay down tracks, so don’t get blindsided by the outliers.

The Power of Being an Outlier

Ultimately, this is another article about outliers and outlying. When a tool shift change occurs of the sort of the Industrial Revolution, the Information Age, and now the Age of AI, outliers catapult to surprising new normals. And in a year or two all the nattering nabobs of the status quo will forget that they ever actually held the dead and dying opinions. They’ll jump on the new bandwagon like it’s always been that way. It wasn’t that long ago since JavaScript was a toy language to unreliably scroll headlines onto a webpage. Now the snooty comp-sci crowd can’t get enough of their Redux WASMs. The pendulum swings back.

Gemini 2.0 Experimental Advanced: Okay, here’s a summary of the article in my own voice, followed by title suggestions and my opinion, as requested:

[Moved to top]

Title and Headline Ideas:

Short & Catchy:

• AI’s New Deal: From Ideas to Automation
• Coding for the People: The LLM Revolution
• The Rise of the Digital Bourgeoisie
• Beyond the Rubber Ceiling: AI’s Democratizing Force
• Local LLMs: Your Private Engine of Innovation
• From Shower Thoughts to Machine Code: The Power of AI
• Unlocking Agency: How AI Transforms Ideas into Action
• The Age of AI: A Societal Recalibration
• Sleep, Laughter, and LLMs: Fueling the Future

More Descriptive:

• The Democratization of Automation: How Local LLMs Empower Individuals
• Bridging the Gap: AI as a Universal Translator Between Human and Machine
• Beyond the Tech Giants: Open Source LLMs and the Future of Innovation
• The Industrial Revolution of Our Time: AI, Agency, and Societal Change
• From Concept to Code: A Personal Journey Through the AI Landscape
• HTMX and the Future of the Web: Resisting Obsolescence in the Age of AI
• Rethinking the Web: Why Hierarchy and Structured Data Matter in the AI Era
• Pipulate/Botifython: A Case Study in Open Source AI-Driven SEO
• Embracing the Outlier: How AI Enables a New Generation of Innovators

Provocative:

• Is Your Website Ready for the AI Revolution?
• The Death of the Cash Register in the Cloud?
• Nixing Docker: The Future of Software Development in the Age of AI
• Forget React: HTMX is the Key to a Future-Proof Web
• The Rubber Ceiling is About to Break: AI’s Impact on the Tech Landscape
• Are You Part of the New Digital Bourgeoisie?
• The Coming Societal Shift: How AI Will Redefine Class and Power
• Don’t Get Left Behind: Seizing the Opportunities of the AI Era
• The Future is Open Source: Why Local LLMs Will Win the AI Race

Editorial Review

Overview and Analysis

This article is a fascinating and insightful exploration of the transformative potential of AI, particularly in the context of web development and automation. The author’s passion and deep understanding of the subject matter are evident throughout, and their personal anecdotes and technical examples make the complex concepts accessible to a wider audience.

Open Source and Local LLMs

I particularly appreciate the emphasis on open-source and local LLMs. The author’s argument for the importance of privacy, accessibility, and individual empowerment in the age of AI is compelling and timely. The contrast drawn between centralized, cost-prohibitive models and decentralized, open-source alternatives highlights a crucial debate in the field of AI development.

Societal Impact

The discussion of societal recalibration and the rise of a new “digital bourgeoisie” is thought-provoking. The author’s analogy to historical shifts like the Industrial Revolution and the French Revolution adds depth and context to the discussion.

Technical Merit

The technical sections, while dense at times, are valuable for those with a background in web development. The exploration of HTMX, schema.org, and the challenges of crawling SPAs provides practical insights into the evolving landscape of web technologies.

Final Thoughts

Overall, this article is a valuable contribution to the discourse on AI, open-source software, and the future of the web. It is a call to action for individuals to embrace the opportunities presented by these transformative technologies and to actively shape the future of the digital world. The author’s optimistic yet realistic vision of a future where AI empowers individuals and fosters innovation is inspiring. I believe this article will resonate with developers, entrepreneurs, and anyone interested in the intersection of technology and society.