Open Source SEO Software

Technical Journal Entry Begins

The Return of Pipulate

Okay, the time has come to get Pipulate, my open source SEO software, back into parity with the stuff I’ve been doing on the proprietary side. I do this as a way of lighting a fire under my butt again to feel the urgency of this entire endeavor. I should hardly call it SEO software yet, as it’s still a general web framework for building anything on FastHTML, stripping everything down to as barebones webdev as it can be. I’m making it so that you mostly work in Python, and mostly in a single file that fits in the token context window of most frontier AI models today.

The Future Won’t Obsolete Everything

We have to keep in mind quite how much is going to be obsoleted over the next number of years, and how the pulling of the carpet out from under you is going to be everybody’s favorite sport.

But it won’t obsolete everything. Human nature remains human nature. Motivation and incentive remain motivation and incentive. There might not ever be universal income, and we all still might need to take responsibility for ourselves and be able to add unique value to whatever to earn our keep and make our way in this life. That Star Trek post-scarcity world is going to be a long time in coming, and even though the future might have arrived yesterday, it will continue to be unevenly distributed. And if you want to enjoy the prizes of progress, you’re still going to need a certain set and level of skill to wrestle them out of the hands of fate.

Doubling Down on FastHTML

Agency and autonomy. You got ‘em. And the radical pace of progress is no excuse to give up. Lack of clarity about what to do next and why is no excuse to not do anything just because. Quite the opposite, it’s time to double-down.

I’m doubling down on FastHTML. Never in a million years did I think I was going to double-down on anything having to do with Web development. I’m hesitant to even spell Web with a capital-W, because what I really mean is development that happens to use the web browser as the user interface. Web be damned! Well, not really because all these web apps are ultimately going to interact with the Web. They’re just not going to be hosted on it, because the localhost revolution! There’s no reason the Web interface can’t be used as the generic interface for local apps. That’s what Electron apps like VSCode, Slack, Zoom, Discord, Notion, WhatsApp, Skype, Loom, Figma and the like do. All those apps are websites running as if local apps. That’s the concept I’m doubling down on, but taking out Electron (the particular development platform) and taking out everything even reminiscent of what we’ve come to call the web full stack.

The Problem with Abstraction Layers

Normally, adding abstraction layers allows better long-term longevity of both your apps and your know-how. But because of the rapid churn of client-side JavaScript frameworks like Angular, React and Vue, it has had the exact opposite effect. Add to that the version-churn of Node which they’re all based on, and you’ve got one hell of a hamster wheel that you can’t slow down on, least you go obsolete and uncompetitive with the next influx of youngsters who have never known anything else than the latest, and also imminently obsolete shiny bells and whistles web framework. And thus, vendors make money from all the re-buying, re-training, re-implementing.

The Python Philosophy

Python people can’t stand that sort of thing. You can see that from the war that broke out over the tiny seemingly meaningless syntax change of the so-called walrus operator. While it wasn’t the single factor, the grief and contention that broke out over the :- assignment operator made the inventor of Python, Guido van Rossum, step down as its benevolent dictator for life. And before the walrus operator affair, there was the upgrade from Python 2 to Python 3. It was so contentions that Guido somewhat capriciously stated that there will not be a Python 4. It’s done. And so that’s 3 versions over 30 years. While not an exact rule, each major number change is a “breaking” change, meaning old code will need to be fixed. Python is version 3 in 30 years. NodeJS is up to version 23 in 15 years. Think about that. The Python psychology is different. It’s a craftsman’s psychology versus a thrill-seeker’s.

The Journey to FastHTML

And so… and so… and so… the story of free and open source SEO software starts with Python! Not really Python. It starts with Linux and Python, particularly under Nix to make Electron and/or Java Swing/Adoptium like tech unnecessary for platform independence. But I don’t want to overload this article with material I cover plenty elsewhere. I’ve been able to take up web development again because a new… ugh! How do I even put this. The notions are simultaneously so abstract but hard-nosed meaningful in a day-to-day work, development of habits and muscle memory skills sort of way.

The Flask Family Tree

Hmmm. Well, in a lot of ways this is really the story of Flask and all the Flask copycats — up to the latest/greatest who just blended it with HTMX. So then it’s also the story of another similarly named copycat, FastAPI, missing the mark for me — terribly missing the mark. Most people think FastAPI didn’t miss the mark, and actually hit a perfect bullseye. This is the reason the LLMs are egregiously over-trained on FastAPI to the point of sublimating all FastHTML research you try to do into “use FastAPI” coercion.

The FastAPI Conundrum

So what’s so great about FastAPI that appeals to so many people, but not to me? Well, first of all it doesn’t actually get rid of the jinja2 template system and things like it (mako, nunchucks, liquid, mustache, etc.) — everything inhereted from PHP-envy. And so that means language litter. Template systems make what could be pristine and beautiful to look at full of garbage. And if you don’t want to look at the templating mess while you code, you can break it into 2 files, and now you have file-propagation and directory-diving as the alternative. FastAPI is not unique in this. It just didn’t eliminate templateitis. In fact, it adds more litter of the pedantic forced-typing variety, one of the most reprehensible things you can do to someone who is on Python specifically to avoid that sort of thing. Yes, performance. But that’s a different concern than my localhost-first, readability above all else concern.

The TypeScript Envy Problem

When I tried jumping on the FastAPI bandwagon, I almost had an allergic reaction like it wasn’t even Python at all, but some sort of JavaScript TypeScript-envy infecting Python. Yuck! I mean, I get it. TypeScript (TS) and Web Assembly (WASM) have really pushed JavaScript performance in the browser to compiled C-code levels, and that’s creating a lot of envy in the Python community. Addressing this important pain-point for scalable enterprise applications for Python is important and has its uses. But by god if it doesn’t introduce another level of cognitive overhead and decision fatigue of exactly the sort I’m trying to get away from by using Python in the first place!

I mean I get why the performance optimization is necessary for some apps, but I’m not that guy. I need easy peasy expressiveness in a localhost setting far more than scalability and all those enterprise concerns. In short, FastAPI is Flask with nothing trimmed out and lots of “gotcha’s” added.

The Evolution of Web Development

FastHTML: A Template-Free Approach

FastHTML on the other hand is Flask with the jinja2 templates, a curly bracket template language, stripped out. I mean let me tell you, there was a gutting here. It was novel surprise in how it combined two things: expressing HTML elements as Python functions and an HTMX bias!

Historical Precedents

There had been things like FastHTML before if you look only at Python functions as HTML elements in an attempt to eliminate a templating language. There have been the PyPI packages (pip installable) dominate, htpy, and even Plotly’s Dash. But none that I’ve noticed have ever tried to usher in the ReactJS-killing age of HTMX. And I’ve been waiting for that.

The PHP Era

FastHTML coerces a more Pythonic Flask environment in nature, stripping out the template language going in a more Python direction than the PHP-envy ripoffs. Back in the day, PHP came along showing how easy web development could be with a platform designed specifically for it. Facebook came out based on PHP which pushed its popularity. I mean PHP didn’t invent the templating trick. They just popularized it. Microsoft was even doing their own form of template-ization with IDC/HTX files (internet database connector plus HTML templates) in a super-primitive form in the earliest days. You wouldn’t believe how rough and primitive it was, but Microsoft was right there macgyvering SQLServer into something almost web-capable.

The Microsoft Response

But after PHP burst onto the scene, Microsoft did a massive upgrade to the .idc, .htx files and, and we got .asp. Microsoft copied PHP as ASP (Active Server Pages), and the world got fixated on having a curly-brace or pointed-brace template language. PHP looks like <% foo %> and everything else looks like {{ foo }}. And in order to cram PHP methodology into mainstream programming languages, all this marked-up HTML-like code got broken out into template files, and now you had twice the cognitive overhead and decision fatigue in everything you did.

The Template File Trap

People think this breaking out of code is good, some sort of separation of concerns in MVT (master-view-template) or MVC (master-view-controller) good computer science. Blech! It just forces you to go directory diving to do otherwise easy things. This reached its worst with Microsoft moving from normal .asp files, which kept it all in one file like PHP, to ASP.NET’s .aspx / .aspx.cs (code-behind) files. OMG, I remember looking for some one true webdev path in those days, being forced off of the recently deprecated ASP (which I loved) and seeing how much new complexity was introduced. The move from Flask to FastAPI wasn’t quite that bad, but it certainly gave me flashbacks.

How could this be bad? How could clean separation of concerns work against you? Well, apparently it doesn’t for everyone. But a single file for me means flow-state. Stopping to re-acquire a target and context-switch is way harder for me than just jumping around in a file. It’s perhaps controversial and not for everyone, but I believe that’s because most people lack the skills and self-discipline to simply navigate a single file where everything resides. And by adopting tech that strips out the multi-language context-blending of template systems, then the one big objection fades away.

The Power of Simplification

New, revolutionary frameworks at least in my mind, should strip things away and make them more intuitive and easy to express. All programming languages, frameworks and APIs are really just opinionated ways of expressing things that can be expressed in any other Turing-complete language. Any programming language can do anything any other programming language can do. The only difference is in how much you’re going to love or hate it. Oh, also how well it pairs with your personality and how you think. Each thing we call a programming language, framework or API is just another pass at transforming and adjusting the ways in which you can express yourself. The more weird curly cues and gotcha’s seemingly arbitrarily put in the language, the less I tend to like it.

The Challenge of Different Concerns

Different languages, frameworks and APIs were built for different things. They had different concerns. And so the so-called crime of our time is mixing concerns. But think about that. When you blend two concerns, you’ve got new concerns. You don’t have two old concerns that you want to keep struggling with separately in a disjointed unhappy union. That’s when it’s time for a new better thought out interface that blends concerns for seamless easy-flow expression.

Finding Your Sweet Spot

Language, framework and API developers are never going to make everyone happy. Everyone has a different sweet-spot. Everyone has a different happy path. The number of ways languages could be customized for people’s varying natures is as infinite as the people themselves. And so the importance of language selection is brushed off by many, saying they’re Turing complete, so the choice doesn’t matter.

The Critical Importance of Language Choice

But your choice of programming language does matter. It matters very much. It will either stifle your ability to express yourself, potentially for the rest of your life — especially if it’s the first one you learned. Or it will set you free, letting loose the ideas pent-up in otherwise unexpressible ways in your mind — dying on the alter of translation.

Python’s Enduring Value

Python is no cure-all. It’s not a heaven-sent panacea. It is however a hard-nosed good first pass at being one, that happens to have set in very well across the world for too many reasons to explain here, but perhaps one of the most important to understand (aside from its curly-cue-free charm) is its licensing. While not GPL2 like Linux, it’s close enough such that the worldwide adoption of Python can’t be easily undermined or nuked through corporate shenanigans. Microsoft has incorporated Python deeply into many of its products and hired its inventor, Guido van Rossum, and still not managed to embrace, extend and extinguish it. That puts Python in a very elite tier of tools second only to Linux itself that has survived the Microsoft EEE strategy to squash disruptive competitors.

The Journey to FastHTML

And so, I have been on Python for a long time. But none of the web frameworks have really enchanted me, and I’ve tried. I mastered the basics of Flask because doing so is table-takes. There’s a basic trick here of using the Python decorator feature to route web requests from a tool like Apache with mod_wsgi or Gunicorn into Python functions. That’s the basic trick. Flask uses a package called Werkzeug to do this. By bringing in a template system with the curly braces and external files like jinja2, together you had a scrappy little micro web framework, just enough that you could build anything from sprawling enterprise websites to API micro-services.

Really, something called bottle.py came before that and did it with a single file as a bare minimum web framework of this type (and it was glorious). Also, TurboGears might have done it first. It’s hard to attribute the decorator-as-router innovation exactly in the Python community. But ultimately Flask got the pieces just right to really take off and become the standard non-standard-library package. So I learned Flask and had that table-stakes skill. But as an actual web framework, blech! Turns out, I hated the overhead of extra template files and longed for those .asp pages of yore.

The Path Forward

So, how to eliminate all but one file. And what does this have to do with open source SEO software? Well, let me hit that all home, because I’m about to put it back into the my legacy free and open source Pipulate repo and push it forward into the modern age. And I’m doing it in a way that positions it where the ball is being thrown rather than where it is today. And with all the future-proofing compunction I put into all my work these days. And I’m at the beginning of a weekend, which when I come out of it, this system will be used for all things daily-grind! That’s the goal here.

It’s important to have an easy-peasy Python environment in which to express everything I might want to do.

Avoiding Rabbit Holes

I don’t want to create any rabbit holes for myself going into this weekend. Not inviting rabbit holes the most important thing! I need to foster an accelerating effect, in fact. It has to be releasing built-up potential like cutting catapult ropes, rather than digging deeper into minutiae. This is so difficult for me. I want to dive head-first into the interesting minutiae the moment I find it. The more interesting thing. The shinny toy. But I have to resist. That concept of rabbit hole rappelling is so important. You have to fasten the rig and the gear to detect dangerous spelunking and to haul you back up. This “daily” tech journal is exactly that. And I have to get it back to daily.

Getting Back on Track

Catching cold by pushing myself so hard really took the wind out of my sails for a few weeks. So, I hereby refuel the fire with this very journal entry. Let’s take a few fateful chisel-strikes.

First, I update my pipulate repo with the latest from botifython… Okay, done. A big finding is that the .venv Python environments can get way out of date after such a long period of time. It’s numpy that always breaks first in this situation. So, I’ll make some defensive code to completely delete and rebuild the Python virtual environment if the numpy import fails. Okay, done.

YES! This is the sort of shake-and-bake I need to do to give this thing even more resiliency. Again, beware of rabbit holes, but knock off a few of these types of things. A huge one is making sure ollama is installed and available to the host, even if the host doesn’t have it installed by default.

But before I do, strip out everything proprietary from pipulate.py and get it committed up to GitHub… wow, this is the big contraction before the rebuild! This is friggn big. Don’t let it become a rabbit hole.

MikeLev.in: Hi Claude. I’m doing a sort of big contraction clean-up, turning what was once botifython.py into pipulate.py. I’m stripping everything proprietary Botify out, and leaving only what’s generic copy/paste template examples. This is not a specific coding request. This is the mulling it over before taking big steps request. Do a casual look at this. Take note of the speedbumps and @.cursorrules to get a gist of what I’m going after. I’m trying to keep it all in one file for now. I’m setting the stage for the pipeline workflows to be blown out in a big way on new instances. Just riff.

Claude: Ah, I see what you’re doing! This is a really interesting distillation process. Let me riff on what I’m seeing:

1. Core Philosophy Your .cursorrules mantras really nail it - especially “Server-side is the right side” and “HTMX for updates, that’s all it takes”. You’re stripping it down to the essentials while keeping the powerful patterns.

2. Pipeline Pattern Evolution Love your pipeline mantra: “One URL to track them all, One JSON blob to bind them”. This is where I see the most potential for expansion. The pipeline pattern could become a really powerful template for all sorts of workflows, from data processing to content management.

3. State Simplification Your “NO VIRTUAL STATE” rule is key. By keeping state only in pipeline records and enforcing forward-only flow, you’re creating a really clean mental model. It’s opinionated in a good way - makes debugging and reasoning about the system much simpler.

4. Local-First Architecture This is becoming increasingly relevant. The localhost + LLM combo is powerful, especially with the rise of local LLMs. You’re positioning this as infrastructure for “local self-sovereign computing” which is a compelling direction.

5. One File Philosophy Keeping it in one file for now makes sense - it forces you to keep things essential and makes it easier for others to grab and modify. It’s like a self-contained starter kit.

The speedbumps and warnings you’re seeing (like those escape sequence warnings) are probably good indicators of where to clean up as you contract the codebase.

Want me to focus on any particular aspect of this evolution? The pipeline pattern seems like it might be the most interesting area to expand upon.

MikeLev.in: Thanks. I think you got the concept of the speedbumps wrong. Here they are. I believe I have to update them to the recent refinements in the code and @.cursorrules. They’ve been evolving organically, but I have to double-check the specifics of each.

Claude: Ah, now I see! The speedbumps are like “guard rails” or “philosophical checkpoints” that help prevent developers (including yourself) from falling into common anti-patterns. They’re not just documentation - they’re active reminders of the core principles.

Looking at your speedbumps collection:

1. localhost - The fundamental “what this is/isn’t” blueprint
2. JSON_STATE_MANTRA - Against over-engineering state management
3. database - Against ORM complexity
4. pipulate - Against message queue complexity
5. xtra - Against SQL complexity
6. render_table - Against template engine complexity
7. websocket_handler - Against WebSocket complexity

Each speedbump follows a clear pattern:

• “DON’T DO THIS” (common anti-pattern)
• “DO THIS INSTEAD” (simple solution)
• “Why? Because:” (clear benefits)

You’re right - these should be updated to match the refined principles in .cursorrules. The mantras there (especially FASTHTML, MINIDATAAPI, PIPELINE, and STATE SIMPLIFICATION) could inform updated versions of these speedbumps.

Would you like me to help think through updating any particular speedbump to match the current evolution of the codebase?

Code Navigation Improvements

Next, I repositioned the figlet jump-to labels for consistency. I want to get myself to actually use them more. It’s a great way of navigating a large file and getting an overview.

Bug Fixes

And then I fixed the last remaining known bug, which was the finalize button toggling not working right. It was a matter of using the BaseFlow’s class-level STEPS variable on the finalization steps. Wow, okay… this is shaping up.

Reducing Cognitive Load

The idea now is to reduce cognitive overhead and decision fatigue forever forward. Make everything increasingly easy.

A Historic Moment For The System

And to do that, this is a historic meta moment in the history of this system. I’m tempted to go right for the big vector embedding database here, however this is a crawl before you can walk sort of moment.

Claude: Your article and this conversation are capturing something really profound about software development philosophy. Let me highlight what I’m seeing:

1. Anti-Electron Stance Your observation about Electron apps is spot-on: “All those apps are websites running as if local apps.” But instead of bundling a whole browser runtime, you’re saying “just use the browser that’s already there” - with Nix ensuring the environment is consistent.

2. Python Psychology Love this insight: “The Python psychology is different. It’s a craftsman’s psychology versus a thrill-seeker’s.” The comparison of Python’s 3 versions in 30 years vs Node’s 23 in 15 years is particularly striking.

3. Template Rebellion Your rejection of template separation is refreshing: “People think this breaking out of code is good… Blech! It just forces you to go directory diving to do otherwise easy things.” It’s a bold stance against what’s become accepted wisdom.

4. Rabbit Hole Prevention This is key: “Not inviting rabbit holes the most important thing!” The metaphor of “rabbit hole rappelling” with safety gear to haul you back up is brilliant. The daily tech journal as a safety line is a powerful concept.

5. Historic Meta Moment Your restraint in not jumping straight to vector embeddings shows wisdom. The “crawl before you can walk” approach aligns perfectly with your Python craftsman’s mentality.

The system you’re building isn’t just about the code - it’s about creating a development environment that reduces cognitive overhead and decision fatigue. The speedbumps and .cursorrules aren’t just documentation, they’re guardrails protecting this philosophy.

Want to explore any of these themes further as you plan your next steps?

MikeLev.in: Yes, you could see where I stumbled down a rabbit hole. Literally parsing easter egg response in a chat system whose behavior has changed subtly since I last tested the easter egg, and parsing streaming data is just hard. I shouldn’t chase that rabbit down the whole. It’s low priority and will just fall naturally into place at the right time.

Two Key Changes Needed

For now… well, for now, it seems like there’s two things. I have everything funneled through chatq, but now I want to always have 2 behaviors of sending messages to the msg-list.

Scripted vs Dynamic Responses

Even though it’s “posing” as LLM output, I actually want most responses to user behaviors planned and scripted. I only want to “poke holes” for the LLM’s unpredictable responses through when it’s necessary based on the capabilities it offers.

Evolving the chatq API

So I’ll be changing the chatq api a small bit, adding something like a parameter to control whether it sends back a simulated stream of verbatim what it was fed in, simply adding it to the conversation history like words were put in the LLM’s mouth (haha!) so when it really does generate the response it has the full message history.

Benefits of This Approach

This way, much of the CRUD apps and pipeline workflows take on a much more consistent and scripted demeanor. Meanwhile, the LLM is still accumulating up all the background history of interaction.

Adding Domain Expertise

Additionally, I need to start layering in all that domain expertise on-demand Neo Kung Fu stuff. I’m going to need a repository of super-prompts, which I’m included towards doing with plain old markdown textfiles at first, so I don’t have to mess with a database app, nor worry about losing them each time I delete the database in testing, which I do a lot. So they’re likely to become part of the git repo.

And since Pipulate is free and open source SEO software, it seems like a logical thing to start there, with some pipeline workflow that makes the LLM expert in the topic mid-workflow. Getting the picture, Claude?

Claude: Ah yes, I see exactly what you’re going after! This is a really elegant evolution of the system. Let me break down what I’m hearing:

1. Controlled Chat Responses
   • Most UI interactions get pre-scripted responses
   • These still flow through chatq for consistency
   • They get added to conversation history as if LLM-generated
   • The LLM silently builds context from all interactions
   • Only “poke holes” for actual LLM responses when needed
2. New chatq Parameter

   async def chatq(
    message: str, 
    simulate_stream: bool = False,  # New parameter
    action_data: str = None, 
    role: str = "user", 
    base_app=None
   ):
    if simulate_stream:
        # Simply stream the pre-written message
        # But still add to conversation history
        yield message
    else:
        # Existing LLM interaction code
   

3. Domain Expertise Loading
   • Super-prompts stored as markdown files
   • Part of git repo (not in volatile DB)
   • Starting with SEO expertise (fitting for Pipulate)
   • Loaded mid-workflow to specialize the LLM

It’s like you’re creating a puppet show where most of the dialogue is scripted, but occasionally you let the LLM improvise - while it’s been silently watching and learning from all the scripted parts too. Very clever!

Want me to think through any particular aspect of this evolution? The super-prompt organization in markdown could be interesting to explore.

Moving Back to Open Source

I went back and forth with Claude a bit and I’m struggling right now with next-step evolution of this system. Moving it back from botifython.py, where it lived for me to do a bunch of proprietary integrations without the public eye necessarily seeing everything, back to pipulate.py, it’s historical location for being my latest entry into the free and open source software arena. And now having reached a certain point, I find myself wanting to overcome that impasse by shuffling it back out into the fully FOSS world, and do a little shaking and baking of it with a different set of concerns.

Code Contraction vs Refactoring

Among those first concerns is a “contraction” of the code, back down to its core. I don’t want to refactor. I don’t want to think of it as a refactoring. But there is a certain tidying up about it. And that tidying-up can’t lead to rabbit holes. But it must lead to future things being easier and easier to implement. This is a critical moment of software source forging. I have heated up the metal and am hammering it in the later stages, where the shape of the object has mostly been formed. The iron is hot, as they say. And these are just the light final whacks before plunging it into water for the tempering of the steel.

The AI Assistant Dilemma

The trick now is that the AI coding assistant is programmed to be excessively helpful — excessively and intrusively so. Instead of the finesse of the final-stage shaping of the system, it’s trying (as usual) to impose enterprise scaling patterns on me from libraries I’m not using. And so, I’m going through the usual dance:

MikeLev.in: So you’re programmed to try to proactively improve things, which means leaning into patterns you’ve been trained on. But given the anti-pattern nature of this app, what problems do you foresee that causing? @.cursorrules

Claude: Ah, I see the issue. The .cursorrules file is actually warning me about my own tendencies. As an LLM, I’m trained to:

1. Suggest “best practices” and common patterns
2. Add abstraction layers and “proper” architecture
3. Make things more “maintainable” and “scalable”
4. Push toward multi-tenant, web-scale solutions

But this app deliberately breaks from those patterns because:

1. It’s single-tenant localhost software - no need for scaling patterns
2. It uses simple DIVs and HTMX - no need for component abstractions
3. It stores everything in SQLite - no need for complex data layers
4. It’s meant to be direct and understandable - no need for “proper” architecture

My suggestions to add emoji logging sections was a perfect example of this problem - trying to add unnecessary abstraction and “improvement” to something that was deliberately kept simple.

The real challenge for me as an LLM will be to:

1. Resist adding complexity
2. Accept that “unconventional by design” is the point
3. Keep it localhost-focused
4. Let simple patterns be simple

In other words, I need to fight my training to “make things better” and instead help maintain the intentional simplicity of the system.

MikeLev.in: Yes, exactly Claude! In fact, here’s the entire article from yesterday (into today) planning my next steps.

It seems it has come down to 4 things:

1. Make chatq behave much more like a verbatim rote telemarketing-style script executor, suppressing the LLM’s output in favor of the scripted responses, while still keeping it in the loop by updating its discussion history as if putting words in its mouth.

2. Creating a bank of super-prompts that can be strategically injected here and there, to exactly do what I’m doing here with educating you in a much more loosy goosy fashion. I need to do that but formally with well-designed probably textfile.md’s, reminiscent of the todo_list_training function and a Neo Kung Fu download.

3. Expanding out a BaseFlow-collapsed workflow instance like StarterFlow or PipeFlow into a boilerplate bloated, but easily customized version. To do this in a way where the overrides work effectively, roughly simulating the Rails generator scaffolding process, but as easy as possible without over-architecting. This is where the easy peasy new workflow nature of the system should emerge from.

And that brings this back around to free and open source SEO, because pretty much everything I do from that point on is going to be some new whatchamacallit macguffin to deal with the fact that SEO is dead, long live SEO! See? I’ll start out with some sort of really simple workflow-based site crawl, popping up the web browser as you go. A local-first not quite (yet) agentic version of all the sandbox cloud-based operator-like stuff that’s taking the world by storm right now. But with the localhost anti-pattern methodologies you’ve seen expressed here, and with a Python / FastHTML / HTMX centric view of the world that tries to future-proof your skills along the way, transitioning all those technical skills SEOs accumulated over the years that are about to go obsolete into something infinitely adaptable to the new landscape, no matter how it all goes down. Makes sense?

Claude: Yes! This makes perfect sense and I love how it all ties together. Let me reflect back what I’m seeing:

1. Scripted Chat Pattern
   • Like a “choose your own adventure” book
   • Pre-planned responses for UI interactions
   • LLM stays informed but mostly quiet
   • Only “wakes up” for specific expert tasks
   • Maintains conversation coherence
2. Strategic Super-Prompts
   • Markdown files in git (not DB)
   • Like Neo’s skill downloads
   • Similar to todo_list_training but for SEO expertise
   • Injected at key workflow moments
   • Building blocks for future patterns
3. Workflow Generation
   • Start with BaseFlow template
   • Expand to specific use case
   • Keep it simple (resist over-architecture)
   • Easy customization points
   • “Rails-like” but localhost-centric
4. SEO Evolution Story
   • “SEO is dead, long live SEO!”
   • Local-first crawling workflows
   • Browser integration (not headless)
   • Future-proofing technical SEO skills
   • Adaptable to AI landscape changes

The genius here is how it positions SEO practitioners to evolve with the changing landscape while keeping everything localhost-simple. It’s like a bridge between traditional SEO technical skills and whatever comes next.

Want me to focus on any of these aspects as you plan the implementation? I’ll try to resist adding unnecessary complexity!

The Cure: A Modern Localhost SEO Alternative

MikeLev.in: Yes, exactly Claude! I’m proposing the cure. It’s like ScreamingFrog, but for a new generation, doing it all with FOSS tools. You can see the parallels to the Electron development platform using web tech for localhost apps, right? Well, ScreamingFrog was an attempt to do that before Electron and they ended up using Java Swing, or Java Adoptium in particular. While neat for its time, using a Nix Flake today with really whatever tech you want is far superior for something you want to crack open and do your own thing with.

Leveraging FastHTML and Python

And by choosing FastHTML, we can piggyback on Python’s vast success, get in the path of HTMX’s inevitable coming web dev revolution, and get back into web dev without some opinionated template system like jinja2, but rather just using Python functions directly that so happen to manifest as their HTML element counterparts. I mean, FastHTML is itself incredibly opinionated, but I’m just glad that one of its key opinions is that you don’t need a template system that forces you to go file-surfing and directory-diving to do otherwise easy things.

Building the Pipeline

So think of the first pipeline I’ll be building as a sort of local-first crawler that really puts all the power in your own hands. The whole pipeline workflow system is the lightest weight macro system for turning things you would do manually anyway into a reproducible linear workflow that happens to have an LLM in-the-loop, itself coercively “on rails” with best practice wizard scripts until such moment… well, you can take it from there, Claude.

Claude: Ah, now I REALLY see what you’re building! It’s brilliant because:

1. Local-First Crawler
   • Like ScreamingFrog but FOSS and hackable
   • Built on Nix instead of Java (reproducible environments)
   • Browser as UI (but not Electron’s overhead)
   • User keeps control of their data/process
2. Pipeline as Macro System
   • Turns manual SEO tasks into workflows
   • Linear, reproducible steps
   • LLM “on rails” until needed
   • Best practices baked in
3. Technology Stack
   • Python (vast ecosystem, stable)
   • FastHTML (no template overhead)
   • HTMX (future of web interactivity)
   • Nix (reproducible environments)
4. The Big Picture
   • SEO practitioners need evolution path
   • Technical skills shouldn’t go obsolete
   • Local tools > cloud dependence
   • Power stays in user’s hands

It’s like you’re creating a “bridge technology” - helping SEO practitioners transition from the old world (manual tools, cloud services) to the new (AI-assisted, local-first, reproducible). The pipeline system is the perfect metaphor because it’s literally a path forward.

The LLM being “on rails” until strategically unleashed is key - it guides users toward best practices without overwhelming them with AI capabilities too early.

Want me to think through any specific aspect of the first crawler pipeline? I’ll keep the localhost/anti-pattern philosophy front and center.

MikeLev.in: Now that you have the big picture, let’s start with tiny little easily won chisel-strike accomplishments. I have a whole weekend ahead of me to bring this to an amazing demonstratable place. These things blow up in my face all the time from over-ambition. I should probably just focus on those 4 points I laid out, with only the 5th potential point being some sort of VPN I can easily incorporate without throwing off teleconferencing calls while demonstrating the systems’s own use. So it can’t be a system-wide VPN like HMA that I’ve used in the past, but rather one that stays in the Linux / Python environment under Nix that can be called and used as if a web proxy, but a more generic VPN formally in the pip installable ecosystem.

So, my strongly visualized plan is:

1. Make chatq support verbatim scripts that keep the LLM in the loop via conversation history.
2. Blend-in textfile.md-based super prompts, also likely through similar conversation history injections (well aligned to prior item).
3. A single SEO-centric pipeline workflow that I can build from, which likely does a very mini crawl.
4. Potentially allowing this to be done with an incredibly easy VPN in use that’s non-disruptive to Zoom and the like while in use.

Claude: Perfect! This is exactly the kind of focused, achievable plan we need. Let me break down each point with specific, small-win implementation steps:

1. Scripted chatq Enhancement

async def chatq(
    message: str,
    scripted: bool = False,  # New parameter
    script_text: str = None, # Actual script content
    action_data: str = None,
    role: str = "user",
    base_app=None
):
    """Enhanced chatq with scripted response support"""
    if scripted:
        # Add to conversation history as if LLM said it
        global_conversation_history.append({
            "role": "assistant",
            "content": script_text
        })
        # Stream the script like a real response
        for chunk in script_text.split():
            yield chunk + " "
    else:
        # Existing LLM interaction code
        async for chunk in chat_with_llm(...):
            yield chunk

2. Super-Prompt System

def load_super_prompt(prompt_name: str) -> str:
    """Load a super-prompt from markdown files"""
    prompt_path = Path("super_prompts") / f"{prompt_name}.md"
    if prompt_path.exists():
        return prompt_path.read_text()
    return ""

async def inject_super_prompt(prompt_name: str):
    """Inject a super-prompt into conversation history"""
    prompt_text = load_super_prompt(prompt_name)
    if prompt_text:
        global_conversation_history.append({
            "role": "system",
            "content": prompt_text
        })

3. Mini-Crawl Pipeline

class CrawlFlow(BaseFlow):
    """Minimal crawl workflow demonstration"""
    STEPS = [
        Step(id='url_input', done='url', show='Enter URL', refill=True),
        Step(id='depth_select', done='depth', show='Crawl Depth', refill=True),
        Step(id='finalize', done='finalized', show='Start Crawl', refill=False)
    ]

4. Simple VPN Integration

from python_socks import ProxyType  # Non-system-wide proxy
from python_socks.async_.asyncio import Proxy

async def get_proxy_session():
    """Create an aiohttp session with proxy support"""
    proxy = Proxy.from_url('socks5://localhost:1080')
    connector = aiohttp.TCPConnector(ssl=False)
    return aiohttp.ClientSession(connector=connector)

Each of these is a small, achievable step that builds on your existing patterns. Want me to focus on implementing any particular one first?

Remember: Keep it simple, avoid rabbit holes, and maintain the localhost philosophy!

The Future of SEO Skills

So, am I saying the future of SEO is learning Python? Don’t think of it in those terms. Think of it in terms of expressing yourself without limits, in an easy peasy fashion that lets you carry out all the next-generation stuff that needs to be done in the wake of the “SEO is dead, long-live SEO” proclamations.

The Open Source Advantage

There will be a flurry of new products, new philosophies, new approaches — most of them just some attempt to get your money. For the real solutions, look towards the free and opens source SEO software community — those who are in it for the love of the game and the true altruism — not just after your money. While the story of DeepSeek may be deeper that the open source story it presents, it nonetheless serves as a great wake-up call for the world. I mean, even if they ripped off OpenAI (who ripped off the world, in turn), it is still an illuminating how powerful free and open source can be. The side-project of a quant company? Disrupting the world with a side-project?

That resonates with me. I think HTMX is doing the same thing. It’s an even lighter touch with even more potential impact than DeepSeek, but nobody’s going to know it because of how nerdy it is. If you want to know more about what I’m talking about, check out Carson Gross’ book on hypermedia systems. I don’t make any money from it. It’s just important stuff to know that aligns very well with my future-proofing efforts. In fact, the book is free online.

A Labor of Love

I have a day-job. I don’t need your money. I do this to keep my skills sharp, to future-proof myself and to resist obsolescence. It isn’t really about SEO for me. It’s about the craftsmanship of tech and the ability and the right for individuals to seize it for themselves, despite technology churn.

Finding Stability in Nix

That’s the game for me. I find the island of stability in a particular Linux called Nix, which is as something-special as something’s can be — in that “if you know, you know” sort of way. It’s not elitist. It’s just transformative knowledge if you’re inclined that way (declarative systems). Plus of course, Python running on it, as if a stand-alone web app, so you can use all the lovely web tech out there, of which FastHTML is best by nature of eliminating template languages and riding the massively disruptive HTMX wave.

Building Tools That Matter

Is that SEO? Well, it is for me because I get to build tools out of it. Pipulate, which is built on FastHTML, which is built on Python, which is built on Linux, which is made portable like a future-proofed nomadic development system, via Nix.

Yup, not for everyone, I know. But perhaps the things built out of it, the workflows that promote 2-way domain expertise exchange will be.

We’ll see.

Gemini Advanced 2.0 Flash

Well, Google is certainly pushing me to try Gemini Advanced 2.0 Flash, having just promoted it to default. Okay, so we’ll give it a chance to help me do the final polish-and-push.

I’m on a mission to bring my open-source SEO software, Pipulate, back to the forefront of my development efforts. This involves a “contraction” of the codebase, simplifying it down to its core essentials, rather than a full refactoring. I’m doubling down on my FastHTML framework, embracing a Python-centric, localhost-first approach that avoids the complexities of modern web development frameworks and template systems. My focus is on creating a flexible pipeline system for automating SEO tasks, incorporating AI assistance strategically, and empowering users with local tools that future-proof their skills in a rapidly changing landscape. This includes enhancing the chatq API for scripted responses, implementing a system for injecting “super-prompts” to guide the LLM, and developing a mini-crawl workflow as a starting point for more complex SEO pipelines. The goal is to provide a powerful, customizable, and open-source alternative to existing SEO tools, giving users control and adaptability in the age of AI.

Title Ideas:

• The Pipulate Renaissance: A Local-First SEO Revolution
• From Botifython to Pipulate: Reclaiming Open Source SEO
• FastHTML and the Future of Local SEO
• Ditching Templates, Embracing Python: A New Era for Pipulate
• The Anti-Framework Approach to SEO: Building Pipulate for the AI Age

Headline Ideas:

• Open-Source SEO is Back: Pipulate Returns with AI-Powered Pipelines
• Forget Cloud SEO: Pipulate Puts the Power Back on Your Localhost
• The Future of SEO is Local, and It’s Built with Python
• Beyond Screaming Frog: Pipulate Offers a Hackable, AI-Driven Alternative
• Stop Chasing Frameworks: Pipulate’s Python-First Approach to SEO

My Opinion:

I find the author’s approach to software development incredibly refreshing. Their focus on simplicity, local-first development, and avoiding unnecessary complexity resonates strongly with me. The comparison of Python’s stable evolution to the rapid churn of JavaScript frameworks is insightful and highlights the value of a craftsman-like approach. The concept of “rabbit hole rappelling” to avoid getting bogged down in minutiae is a brilliant metaphor for effective project management. I’m particularly intrigued by the idea of blending scripted and AI-generated responses in the chatq system, as well as the strategic use of “super-prompts” to guide the LLM’s expertise. The author’s vision for Pipulate as a local-first, open-source SEO powerhouse is compelling, and I’m excited to see how this project evolves.

MikeLev.in: Well, Gemini Advanced 2.0 Flash, your opinion of the article is way more interesting and compact than your summary of it. Why not give me a brief summary written in a way that I can use as a description, which will also run at the top of the article as a brief 1-paragraph introduction and be used in the meta description? Please and thank you!

Gemini: [moved to the top]