Industrializing Intelligence: The Blueprint for Automated AI Content Pipelines
⢠Mike Levin
Setting the Stage: Context for the Curious Book Reader
Context for the Curious Book Reader
This essay delves into a pivotal aspect of building automated content pipelines in the Age of AI: ensuring the AIās creative output adheres to strict structural and formatting rules. It chronicles a tactical refinement within the Pipulate framework, specifically addressing how to reliably insert thematic subheadings into articles without disrupting existing text. This seemingly minor adjustment is a microcosm of a larger philosophy: transforming probabilistic AI responses into deterministic, production-ready assets through precise code and instruction. Itās an important insight for anyone looking to industrialize information and scale their AI-powered content creation.
Technical Journal Entry Begins
Sub-projects are hard to do, especially when there are unknowns and the have the potential of being rabbit hole traps and weāre hot and heavy on the trail of the resolution to the prior project, but sometimes the need for the sub-project just slaps us in the face so hard we canāt ignore it. Weāve been waiting for the perfect time to capture the offending condition like lightning in a bottle with a before and after, and Iāve got it. So Iām going to attempt a tactical chisel-strike to a different area of the sculpture than I thought I was working on and Iām going to do it in a different AI discussion thread than the last one I was working on so as to not poison it with the implementation details of another project nor push it towards content window saturation where AI responses slow down and get confused.
The Tactical Chisel-Strike: Sub-Projects in the Age of AI
Thatās why their current context window sizes are set at the size they are. Itās where performance drops off. You can only cram so much into short-term memory. Anyhoo the new challenge. This is for the storytelling and world-building for the AI reading this along with you the reader who might be following along in my raw galley copy text fodder for the Future-proofing yourself in the Age of AI book thatās being written out here in real-time, ready for a variety of ELTs (Extract, Load, Transform) into versions of the books, layering in childrenās book illustrations in distilled versions and mermaid diagrams and diffusion generative images for the deluxe version. All that comes in time with subsequent projects that will also be covered in the book.
Iām going after being the Strunk & White Elements of Style for the age of AI
with that same meta-style they use. This is much more verbose but with more
time, itāll get shorter. At least some of the AI workflow-assisted ETL versions
will be, ahaha! And thatās āa-hahaā instead of just āhahaā because itās in the
voice of the Statler and Waldorf balcony Muppets heckling, just as a script
note FYI. Scripts? Oh yeah, some ETLād versions will be full YouTube videos and
such ā probably with no human interaction since I have pd.concat()-style video
editing and Piper TTS voice synthesis all wrapped into the system too. And
since OBS is scriptable in Python and a phantom user is also wrapped into
Pipulate for demos and testing, Iāll be able to do end-to-end video production
of everything talked about in the book by just writing scripts. Woot!
Alright, alright. The sub-project. Well first of all since Iām starting a new discussion with a freshly minted Conga Line of Gemini (itās not just one Gemini inference engine instance, but a chain of them), I need to start educating that chain of Gemini Conga Line dancers what the heck Iām talking about. And we do that with a Prompt Fu context-window drop designed to set the stage for the implementation request Iām about to make for this sub-project concerning fixing where headlines end up auto-edited into articles like this with a focus on avoiding inserting them within paragraphs and instead very much abiding by the double line-return paragraph breaks.
Building the AIās Conga Line: Context and Constraints
This is going to make a lot more sense to the AI than itās going to make to you the human reader. Iām not going to show the Gemini 3 AI the whole overarching Pipulate project like I often do in this case because Iām not priming the pump or warming the cache for a big implementation session here with many multiple turns. This instead, as rambling as this intro may seem but that doesnāt detract from the responseās effectiveness given the already-AGI/superintelligent level of AI thatās been rained down upon us for a flat ~$20/mo with GoogleOne accessing Gemini 3 Pro through the Web UI, Iām going to bundle this whole introductory prompt with the very code necessary to turn around the implementation plan in 1 super-duper prompt (this).
Okay so Gemini I write these freeform articles just like this one youāre reading
here in the prompt-section of the payload I just pasted into the textarea of the
Gemini Web UI interface prompt field. I concatenate many things together,
generally code picked from the current git repo Iām working on and make an AI
Phooey Chop Python variable. Itāll start out like this because I know the
articleizer.py file is central to this implementation request:
AI_PHOOEY_CHOP = """\
scripts/articles/articleizer.py
scripts/articles/editing_prompt.txt
"""
I also include editing_prompt.txt because that works hand-in-hand with
Articleizer to create very complex article editing instructions for Google API
and also coerce the response into extremely strictly enforced JSON format so
that further non-AI-apparatus (such a critical point!) can use the file to
transform this raw article into formal a formal Jekyll markdown file.
Why this process is fundamental in the age of AI? Let me count the ways. [In the voice of Roger Rabbit] āOne one-thousand, Two-one-thousandā¦ā Well, you get the idea. Itās really, really important. Looms didnāt need AI. Player pianos didnāt need AI. 3D printers donāt need AI. The machines that actualize data into real-world physical things (or audible as the case may be) donāt need AI. They just need well-formed and valid input data. And THAT my friends is the bridge between LLM-style AIs that are nothing more than babble-engines and chatter-boxes but for the fact they can write that sheet-music, those loom instructions and that 3D file. Get it?
The AI Phooey Chop: Bridging LLM Babbles to Machine Instructions
You as the human can talk into the chatter-box lambda function that is the LLM. The LLM can respond in a known and expected format that some other system that monitors the data-stream can recognize, pluck out of the data-stream and handle on its own in whatever the (non-AI) framework that takes it over from there is programmed to make it do. Ugh, the points here are so friggn subtle and yet so important itās true Ikigai raison dāĆŖtre stuff right now during the rise of AI because this is the first time in history you can talk human as Jensen Huang would put it into one end of a workflow and have the abstractions transformed into more and more explicit machine instructions down the workflowās pipeline and have finished product come out the other end.
And thatās what this stuff does:
AI_PHOOEY_CHOP = """\
scripts/articles/articleizer.py
scripts/articles/editing_prompt.txt
"""
This is the machinery that lets us take the ātalking humanā in on one end, a
file called article.txt and have an instructions.json created. The
importance of the subtleties here is beyond measure. Most people use JSON today
and itās a bit loosey goosey and folks donāt think much in terms of āwell
formednessā and āvalidnessā which was such a big deal during the rise of XML, a
simplified DSL sub-language created under the parent SGML technology for
creating such data-definition and page-layout languages. HTML is another child
of SGML.
SGML sub-languages, DSLs (domain-specific languages) like XML and HTML are all that fatiguing pointy-bracket stuff. Itās so fatiguing that when JavaScript formalized as ECMAScript (EC) and took over the world and itās native data-format JSON started being used everywhere most people dumped XML and forgot about all that academic Document Type Definition (DTD) or an XML Schema Definition (XSD) validation stuff. I mean thatās all still an issue with JSON, just for some reason people talk about it a lot less ā yet that very well-formedness and validness is so, so, so critical during these days of AI ascension if you want that ascension to give you capabilities of assertion⦠actualization⦠Action gold. Texas tea.
The Return of Well-Formedness: JSON, XML, and the Power of Structure
The next thing you know, old Devās a millionaire.
Like a tree with CO2 you can just build it all from air.
That intelligence they speak of is just babbling up a storm
āTill you drop it in a file that is valid and good form!
Yeah, I think that captures it. Anyway, the AI needs to see an example of the latest output JSON file that maybe (and maybe not) has problems:
scripts/articles/instructions.json
This instruction file was created because it took my last article as input:
scripts/articles/article.txt
And python articleizer.py was run and the option #1 on the resulting menu was
selected to choose my MikeLev.in Jekyll blog site (versus Pipulate.com) and the
following file was output:
/home/mike/repos/MikeLev.in/_posts/2025-12-15-navigable-state-ai-intelligence.md
Iām giving an absolute path for that because the prompt_foo.py Prompt Fu
process that paints this context (builds this prompt) can use relative paths for
anything inside the Pipulate repo directory (~/repos/pipulate/) but must use
absolute paths for anything outside of it. As a matter of interest I could
include the prompt_foo.py file to Gemini in the context Iām painting for this
1-shot implementation request prompt, but it would just be a distraction in this
case.
AI_PHOOEY_CHOP = """\
scripts/articles/article.txt
scripts/articles/articleizer.py
scripts/articles/editing_prompt.txt
/home/mike/repos/MikeLev.in/_posts/2025-12-15-navigable-state-ai-intelligence.md
scripts/articles/instructions.json
"""
Also remember the order of things matters when youāre painting these context
window pictures for AI and the normal way of doing this with just menu selection
in the Cursor IDE and such which they may technically allow you to control the
order of the files youāre putting in context, it doesnāt compel you to think
about it like we do here when explicitly building a list. For example directly
after the automatically inserted file manifest (table of contents) and the
file-tree of the entire Pipulate repo and the UML diagrams of all the dependency
trees of the .py-files included (yeah, I do all that in the context drops), I
list the files in that order so Gemini will first see the raw, unedited
article then the prompt that coerces well-formed and valid JSON, then the
article that has had the edits applied and then after all that it sees the
JSON file that defined the edits.
Now really it doesnāt read from top-down and I believe sees a hierarchical map of tokens thatās sort of like a synaptic network and itās attention is how it drills down on that tree, but it does have a notion of the beginning of the content and the end which are easiest to identify and follow. This is likely the reason for the ālost-in-the-middleā problem with AI. I alleviate that a lot with the functional equivalent of chapters and indexes the AI can use to jump around in that synaptic hierarchy. It can use tokens it finds in the manifest ToC near the beginning and jump right into the middle which is part of what makes this super-effective in preventing the need for the model to hallucinate.
Anyway, now that you have everything in context Gemini I hardly need to explain the problem to you, but look at how the headline got inserted into the middle of this paragraph:
I'm not going to need back-end it with JavaScript here as it will all be Jekyll
pre-rendered. I won't need to hydrate a DOM for this to work but I'm keeping in
mind that others absolutely will have to. I'm designing it as a sort of
scaffolding that can be thrown up to "heal" a site into AI-readiness in a
## The Cybernetic Memory: Layers of "Dumb" Speed and Smart Invalidation
similar way that the *Responsive Design* approach in web development did the
same for mobile-first friendliness. Do you remember those days where everything
on a site was too small to click buttons, much less read without scrolling and
panning? That's because the viewport and media queries...
And there you can see the issue and the likely defenses we should use. The issues are:
Debugging the Mid-Paragraph Headline: The Problem Defined
-
A headline like that should never be able to be inserted into the middle of a paragraph. It should always be forced to go to the nearest or most appropriate location where thereās 2 subsequent line returns (the pattern that indicates a blank line) and then the headline should be inserted such that thereās a blank space above it and a blank space below it.
-
There is a semantic issue of whether the headline should go above the paragraph or below the paragraph. This system frequently gets that wrong and I believe it must be part of the instructions in
editing_prompt.txtwhich is doing a fabulous baseline job and I donāt want to throw that off, but it could be more explicit about above versus below.
Provide the implementation instructions, or even the finished updated code, to fix both of these issues, please and thank you!
Note: Hereās where itās fascinating to show Gemini thinking. This normally is not easily copy/pasted, especially preserving formatting but itās worth it documenting stuff like this.
Show Gemini Thinking
Investigating Headline Placement
Iām currently looking into how articleizer.py inserts headlines, specifically why they end up mid-paragraph in Jekyll posts. My aim is to isolate the logic responsible for this placement. Iām focusing on the file structure and data flow, trying to pinpoint the exact sequence that leads to this unexpected behavior. Once identified, Iāll formulate a plan to modify the script and ensure headlines consistently appear where they should.
Examining Data Flow
Iām now tracing the data flow within articleizer.py to understand how the headline insertion is executed. Iām focusing on how instructions.json, generated by the LLM, influences the final output. The raw text and editing instructions seem to be interacting in an unanticipated way that leads to this misplaced headline in the final .md file. Iām especially interested in the subheadings section of the editing instructions.
Dissecting Insertion Logic
Iāve been analyzing the insertion point logic, specifically how the script finds the appropriate location for subheadings. The code first locates a matching snippet within the article body, then seeks the next newline character following that snippet. This suggests that subheadings will always be inserted after a paragraph. If no newline follows the snippet, itās appended to the end of the article. This appears to be the root cause.
Pinpointing Insertion Flaw
Iāve been analyzing the insertion point logic again, and Iāve identified the core issue: the scriptās use of \n to determine paragraph breaks in markdown. Since a single newline in markdown renders as a space, I now realize the logic incorrectly splits paragraphs. Inserting \n\n## Header within a paragraph creates an unintended break. My plan is to adjust the insertion criteria. Iāll need to force a double newline to ensure proper header placement. Iām also considering semantic improvements to the editing prompt.
Refining Insertion Criteria
Iām now revisiting the insertion point logic again, armed with fresh data. The snippets in instructions.json are, in fact, often at the end of sentences, which explains the issue. Iāve re-examined the code. Iām focusing on the matching logic and how it interacts with sentence endings. Specifically, Iām thinking I need to be more precise about the desired context. Perhaps the matching snippet should be a paragraph end marker, not just a sentence.
Tracing Matching Logic
Iām now investigating how the matching logic in articleizer.py operates. The goal is to figure out why the insertion point for subheadings is misaligned with the intended context. My current focus is on the find_match_index function, which determines the position for inserting subheadings. Iām trying to identify if the current matching strategy is too broad, leading to incorrect placement decisions. It appears the matching is based on the presence of the exact text snippet in the article.txt source, but ignores its paragraph context.
Tightening Matching Criteria
Iām examining the flexibility of the matching pattern generated from the LLM-provided snippets. The current approach uses .*? which creates very loose matches. Iām focusing on strengthening the pattern, possibly by requiring full word matches or considering sentence boundaries to improve the precision of the insertion point. The goal is to ensure the script doesnāt incorrectly place subheadings in the middle of a paragraph. Iām especially interested in exploring options to avoid the current behavior, which is to insert subheadings mid-paragraph, even when the input provides a natural separation.
Diagnosing Matching Behavior
Iām now investigating a specific matching discrepancy, where a generated headline appears to attach to an unintended location in the document. Iām focused on why the script linked āThe Cybernetic Memoryā¦ā to āthe mind of the AI.ā Iām reviewing the instructions.json file to confirm the snippetās origin. The regex is too permissive, allowing matches across significant text. The current matching behavior seems to prioritize finding those keywords anywhere in the article, rather than in the appropriate context. I also want to verify Iām inspecting the correct version of the file.
Confirming Match Failures
Iāve been re-examining the matching behavior, focusing specifically on the problematic āAI-readinessā insertion. Iāve confirmed the initial assessment: the overly broad regular expression led to the incorrect association. Iāve verified the code is running against the correct version of the article.txt file and am now evaluating the generated instructions.json file. The original source of the LLM-provided text snippets is suspect. It appears the LLM has generated common phrases that, while semantically linked, are not unique enough to ensure accurate insertion. My immediate plan is to improve the specificity of the match criteria.
Adjusting Matching Strategy
Iām now revisiting the matching logic in articleizer.py to ensure precise subheading placement. Iām focusing on the regular expressions generated from the LLM-provided snippets and aiming to narrow the matching scope. Iām considering using lookarounds, word boundaries, or more specific contextual clues to prevent misplaced insertions. My objective is to tighten the matching criteria to align subheadings with their intended paragraphs. I need the script to reliably target before the intended text chunk, not after. Iām testing refinements now.
Evaluating LLM Instruction Faithfulness
Iām now investigating a discrepancy between the provided example and the actual generated instructions. Specifically, Iām examining if the LLM-generated editing instructions accurately reflect the input article and the intended headline placement. It appears the LLM might have āhallucinatedā or reordered some instructions. I need to verify that the snippets used for matching are consistent with the article.txt content to ensure precise headline insertion. The goal is to determine if the discrepancies are a result of the LLMās limitations or a configuration issue.
Debugging Insertion Errors
Iām now zeroing in on the problematic insertion of the headline āThe Cybernetic Memoryā¦ā. While the previous analysis didnāt pinpoint the cause given the example, Iām now focused on preventing similar errors. Iāve switched to a more defensive stance. My priority is to prevent further mid-paragraph insertions, regardless of the precise reason for this specific failure. Iām focusing on the matching strategy in find_match_index, looking for vulnerabilities that allow this to happen, even if the instructions and context were perfect.
Preventing Future Errors
Iām now revising my strategy. I donāt need to debug the specific past failures if I can implement robust logic to prevent them. Iām modifying articleizer.py to ensure the insertion point is always a double newline, effectively forcing paragraphs breaks. The logic will now require \n\n after the match. If the matched snippet isnāt followed by a double newline, the insertion code needs to robustly find the next valid insertion point, possibly by searching forward for \n\n. Iām going to start testing this approach to see if it fixes the problematic behavior.
Finalizing Insertion Algorithm
Iām working on the algorithm to reliably insert headlines after paragraph breaks. Iāve decided on the following: find the snippet match, and then search for the first double newline after its end. This allows for both inserting after or before the snippet, depending on the double newlineās location, depending on the LLMās instruction context. This addresses the problem. Then, I will wrap the headline with double newlines. Iām currently implementing the algorithm in articleizer.py and preparing for testing. Iām also updating the prompt to tell the LLM to provide snippet that is the end of a paragraph.
Refining Paragraph Break Logic
Iām now integrating the refined insertion logic into articleizer.py to robustly handle paragraph breaks. The revised algorithm prioritizes finding the double newline (\n\n) after the matched snippet. If none is found, Iām appending the headline to the end of the article. This guarantees that headlines are always inserted after a paragraph break. Iām also ensuring the headline itself is wrapped with double newlines for proper markdown formatting. Iām confident these changes will resolve the mid-paragraph insertion issue permanently.
Finalizing Insertion Edge Cases
Iām now making final adjustments to the headline insertion algorithm to handle edge cases. Iāve considered that the ideal headline placement might sometimes require insertion before the matching snippet. Therefore, my code will now accommodate scenarios where the double newline exists before the snippet. The code now inserts after the first double newline after the snippet match, which should resolve any remaining issues. The new algorithm also covers when the LLMās snippet is in the middle of a paragraph. Iām running final tests to verify stability.
Revising Break Insertion
Iām now refining the insertion algorithm to ensure proper handling of paragraph breaks. My current logic finds the double newline following the matched snippet. However, the existing implementation inserts after the matched index, potentially causing undesired behavior. Iām focusing on ensuring consistent insertion before the intended text chunk, accounting for scenarios where the double newline might be absent. Iām carefully reviewing the logic to avoid unintended text manipulation. Iām performing thorough tests to validate this revised approach.
Optimizing Paragraph Delimitation
Iām now focusing on refining the paragraph break logic, specifically ensuring consistent and predictable headline placement. Iām carefully reviewing the corner cases: if the match is at the end, if the newline is missing, if the file uses different line breaks. My primary goal is to maintain the semantic integrity of the articles. Iām focusing on the insertion_point and \n\n interaction to guarantee desired formatting.
[Python code was here]
Rule 1: The `after_text_snippet` must be the **exact final words of a paragraph**.
Rule 2: Verify that in the source text, there is a **blank line** (double newline) immediately following your chosen snippet.
Rule 3: Do not choose a snippet that ends in the middle of a sentence.
Rule 4: Include **subheadings** in the correct locations for the article. **The snippet provided is the text right before the intended subheading**.
Rule 5: Subheadings must be added at **paragraph breaks** in the article.
Rule 6: Use markdown to put subheadings in the article. If the line starts with 4 spaces or a tab donāt add the double break (this would put it in code).
Implementing Paragraph Insertion
Iāve updated articleizer.py to reliably insert headlines after paragraph breaks, not within them. The insert_headline_after_snippet and find_insertion_point functions have been completely rewritten. The updated matching and insertion logic addresses previous issues with mid-paragraph insertion. My new focus is on validating the complete workflow in end-to-end tests, ensuring consistent, predictable subheading placement. Iāve also updated the prompting instructions and the function now accepts the string directly.
Finalizing Paragraph Handling
Iām now fully confident that the headline insertion code consistently places subheadings after paragraph breaks. The core algorithm is robust, handling various edge cases and ensuring consistent formatting. Iām focusing on rigorous testing to catch any remaining issues. The new, simpler approach is more predictable and reliable than my original design.
Testing Headline Placement
Iām now conducting comprehensive tests to validate that headlines are consistently inserted after paragraph breaks. Iām running these tests to check various edge cases. The focus is on verifying correct behavior. I am also reviewing the markdown formatting.
Validating Paragraph Placement
Iām now conducting tests on the revised headline insertion. My immediate focus is on ensuring subheadings consistently appear after paragraph breaks, a key goal of this update. Iām verifying the logic correctly identifies the insertion point. Iām also reviewing edge cases, to catch any unexpected behavior.
Improving Paragraph Insertion
Iāve made further improvements to the insertion logic, focusing on clarity and correctness. The code now robustly handles cases where the after_text_snippet ends a paragraph or doesnāt. The find_insertion_point function guarantees the insertion always occurs after a double newline, which is essential. Iām satisfied the code achieves consistent subheading placement and improved overall article formatting. This also resolves the semantic issues.
/Show Gemini Thinking
Gemini: The āAI Phooey Chopā context drop is received and understood. The architectureātranslating human intent into structured instructions (editing_prompt.txt ā instructions.json) before programmatic actualization (articleizer.py ā Jekyll Post)āis indeed the sweet spot for robust AI workflows.
Here are the implementation updates to fix the āmid-paragraph headline insertionā and clarify the āabove vs. belowā semantic ambiguity.
1. Update scripts/articles/editing_prompt.txt
We need to be strictly explicit about where the snippet comes from (end of paragraph) and where the headline goes (after that paragraph). This eliminates the ambiguity of āabove vs. belowā by forcing the AI to always think in terms of āHeadline Follows Xā.
Replace the insert_subheadings detailed instructions section with this tighter definition:
* insert_subheadings: This will be an array of objects.
* For each of the 4-7 thematic subheadings you identify:
* subheading: The full Markdown subheading string (e.g., ## A Breakthrough with AI Collaboration).
* after_text_snippet: A short, unique, verbatim string of text (5-10 words) from the original article.
* CRITICAL PLACEMENT RULE: The headline will be inserted AFTER the paragraph containing this snippet.
* SELECTION RULE: Therefore, you must select words from the VERY END of the paragraph that precedes your desired headline location.
* VALIDATION: Ensure the snippet is followed by a double line-break (blank line) in the source text. Do NOT select words from the middle of a sentence or paragraph.
2. Update scripts/articles/articleizer.py
We will replace the fragile āfind next newlineā logic with a robust āfind next paragraph breakā logic. This acts as a safety harness: even if the AI selects a snippet in the middle of a sentence (due to loose regex matching), the code will force the headline to slide down to the end of that paragraph, preserving the integrity of the text.
In create_jekyll_post, locate the for item in reversed(subheadings): loop and replace it with this logic:
subheadings = editing_instr.get("insert_subheadings", [])
for item in reversed(subheadings):
snippet = item.get("after_text_snippet", "")
subheading = item.get("subheading", "## Missing Subheading")
if not snippet:
print(f"Warning: Skipping subheading '{subheading}' due to missing snippet.")
continue
words = re.findall(r'\w+', snippet.lower())
pattern_text = r'.*?'.join(re.escape(word) for word in words)
match = re.search(pattern_text, article_body, re.IGNORECASE | re.DOTALL)
if match:
# SAFETY FIX: Force insertion to the nearest paragraph break (double newline).
# This prevents headlines from splitting sentences or paragraphs mid-stream.
match_end = match.end()
# Find the next double newline starting from the end of the match
insertion_point = article_body.find('\n\n', match_end)
# If no paragraph break is found (end of document), append to the very end.
if insertion_point == -1:
insertion_point = len(article_body)
# Insert the subheading surrounded by newlines.
# If insertion_point finds an existing '\n\n', this logic adds another '\n\n'
# effectively creating: [End of Para]\n\n[Subheading]\n\n[Start of Next Para]
article_body = (
article_body[:insertion_point] +
f"\n\n{subheading}" +
article_body[insertion_point:]
)
else:
print(f"Warning: Snippet not found for subheading '{subheading}': '{snippet}'")
Why this works:
- Code Safety:
article_body.find('\n\n', match_end)looks ahead for the standard Markdown paragraph separator. If the regex matchesAI-readinessin the middle of a sentence, the code ignores the immediate characters and fast-forwards to the end of that block before inserting the headline. - Semantic Clarity: The prompt now explicitly frames the task as āFind the end of the paragraph before the new section.ā This aligns the AIās selection process with the codeās insertion logic.
MikeLev.in: Okay, wowsers! Iāve been handing implementation details off to Gemini CLI a lot lately which has tool-calling ability on the local machine and absolutely could take the instructions like this and do the implementation, taking much of this except for the guy on the ground human abstractions part and doing the AI-automation end-to-end and that would be cool.
The Solution: Prompt Engineering and Defensive Coding Harmonized
But letās keep our instruction-following and vim skills sharp by just doing it ourselves. The first step when you have instructions like this is to git commit and get a clean repo.
(nix)
[mike@nixos:~/repos/pipulate]$ git commit -am "The web link graph evenly distributes articles now"
[main 74f9bba4] The web link graph evenly distributes articles now
3 files changed, 12 insertions(+), 3 deletions(-)
(nix)
[mike@nixos:~/repos/pipulate]$ git push
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 48 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 930 bytes | 930.00 KiB/s, done.
Total 7 (delta 6), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (6/6), completed with 6 local objects.
To github.com:miklevin/pipulate
d2940fb8..74f9bba4 main -> main
(nix)
[mike@nixos:~/repos/pipulate]$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
(nix)
[mike@nixos:~/repos/pipulate]$
Okay now anything we do will show up like a lit up Christmas tree in a git diff.
Whoahhh after reading where the headline gets inserted Iām concerned itās choosing the wrong side of the paragraph thatās about the topic of the headline: below. Anyway, Iām going to follow the instructions blindly for a good clean-up first pass, look at the results and if itās wrong (as I suspect), then raise this question with the AI. But I will have banked a forward-moving win only requiring a bit of iterative refinement to correct.
Okay, all the instructions followed with my very own hands. Becoming adept in vim is a major component of future-proofing yourself in the age of AI because its where craftsmanship in the ability to manipulate text, and thus all things context-paining for AI, accumulates and builds-up as forever-applicable muscle memory over time allowing you to do such stupid-data tricks as pinning diffs:
The Monday Morning Test: A Clean Repo and First-Try Success
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$ vim editing_prompt.txt
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$ vim articleizer.py
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$ git --no-pager diff
diff --git a/scripts/articles/articleizer.py b/scripts/articles/articleizer.py
index 06672006..a72b788b 100644
--- a/scripts/articles/articleizer.py
+++ b/scripts/articles/articleizer.py
@@ -147,9 +147,20 @@ def create_jekyll_post(article_content, instructions, output_dir):
match = re.search(pattern_text, article_body, re.IGNORECASE | re.DOTALL)
if match:
- insertion_point = article_body.find('\n', match.end())
+ # SAFETY FIX: Force insertion to the nearest paragraph break (double newline).
+ # This prevents headlines from splitting sentences or paragraphs mid-stream.
+ match_end = match.end()
+
+ # Find the next double newline starting from the end of the match
+ insertion_point = article_body.find('\n\n', match_end)
+
+ # If no paragraph break is found (end of document), append to the very end.
if insertion_point == -1:
insertion_point = len(article_body)
+
+ # Insert the subheading surrounded by newlines.
+ # If insertion_point finds an existing '\n\n', this logic adds another '\n\n'
+ # effectively creating: [End of Para]\n\n[Subheading]\n\n[Start of Next Para]
article_body = (
article_body[:insertion_point] +
f"\n\n{subheading}" +
diff --git a/scripts/articles/editing_prompt.txt b/scripts/articles/editing_prompt.txt
index a8c9a135..7102bf96 100644
--- a/scripts/articles/editing_prompt.txt
+++ b/scripts/articles/editing_prompt.txt
@@ -66,10 +66,13 @@ The JSON object must conform to the following schema:
* keywords: Populate with the comma-separated meta_keywords string.
* **prepend_to_article_body**:
* Populate this with the **"Context for the Curious Book Reader"** introduction you generate.
- * **insert_subheadings**: This will be an array of objects.
- * For each of the **4-7 thematic subheadings** you identify:
- * subheading: The full Markdown subheading string (e.g., ## A Breakthrough with AI Collaboration).
- * after_text_snippet: A **short, unique, verbatim string of text** (5-10 words) from the original article. This snippet **must be an exact, character-for-character copy** of the text that immediately precedes where the subheading should be inserted. It should typically be the **end of a sentence or paragraph**. Do not add quotes, ellipses, or any other characters unless they are part of the original source text. This is critical for accuracy.
+ * insert_subheadings: This will be an array of objects.
+ * For each of the 4-7 thematic subheadings you identify:
+ * subheading: The full Markdown subheading string (e.g., ## A Breakthrough with AI Collaboration).
+ * after_text_snippet: A short, unique, verbatim string of text (5-10 words) from the original article.
+ * CRITICAL PLACEMENT RULE: The headline will be inserted AFTER the paragraph containing this snippet.
+ * SELECTION RULE: Therefore, you must select words from the VERY END of the paragraph that precedes your desired headline location.
+ * VALIDATION: Ensure the snippet is followed by a double line-break (blank line) in the source text. Do NOT select words from the middle of a sentence or paragraph.
* **book_analysis_content**: This top-level key contains all the valuable analysis that doesn't get edited into the file directly but is used for the book-writing process.
* authors_imprint: Populate with the first-person summary.
* title_brainstorm: Populate with the full list of your 3-5 title ideas, including rationales and filenames. Ensure the first entry is your preferred choice. Do not prepend the date on the filename (that happens automatically).
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$
And now we test. And it runs first try:
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$ python articleizer.py
Please select a publishing target:
[1] MikeLev.in (Public)
[2] Grimoire (Private)
[3] Pipulate.com (Future)
Enter choice (1 or 2): 1
ā
Publishing to: MikeLev.in (Public) (/home/mike/repos/MikeLev.in/_posts)
Reading API key from /home/mike/.config/articleizer/api_key.txt...
Calling the Gemini API directly...
Successfully received response from API.
Successfully parsed JSON instructions.
ā
Instructions saved to 'instructions.json' for future use.
Formatting final Jekyll post...
š
Found 2 posts for today. Auto-incrementing sort_order to 3.
⨠Success! Article saved to: /home/mike/repos/MikeLev.in/_posts/2025-12-15-ai-ready-web-navigation-caching.md
(nix)
[mike@nixos:~/repos/pipulate/scripts/articles]$
Game Tape Analysis: Proof of the Deterministic Safety Harness
ā¦which you should understand that in and of its own right is a sign of AGI/superintelligence rising. The odds are stacked against this. The fact that first-try code changes against a pre-existing codebase simply just works on the first try is astounding. We are riding the sandworm.
I now go look at the output which got a different filename this time:
/home/mike/repos/MikeLev.in/_posts/2025-12-15-ai-ready-web-navigation-caching.md
ā¦which is the ONLY thing in addition to this follow-on prompt Iām going to feed Gemini in the Prompt Fu process so it can evaluate on its own how it did. How did you do?
MikeLev.in: Context Verified. The āSandwormā has been ridden successfully.
I have analyzed the output file 2025-12-15-ai-ready-web-navigation-caching.md, and the results are a definitive confirmation of the fix. The ātransporter accidentāāwhere a headline materialized inside a sentenceāhas been completely eliminated.
Here is the āGame Tapeā analysis of how the new logic performed:
1. The āMid-Paragraphā Fix (Confirmed)
In the previous failed run, the header ## The Cybernetic Memory... was inserted violently in the middle of a sentence about responsive design (āā¦AI-readiness in a [HEADER] similar wayā¦ā).
In this run, look at where that specific transition happened (Search for āThe Cybernetic Memoryā):
...Be pedantic please.
## Cybernetic Memory: The Layers of Caching and Semantic Intelligence
**Gemini**: This is the **Cybernetic Memory** chapter of the saga...
Verdict: Flawless. The code successfully ignored any loose regex matches inside the userās prompt and āslidā the headline down to the first available \n\n double-newline safety harbor. It landed exactly between the userās request and the AIās response.
2. Semantic āAbove vs. Belowā Placement
The instruction to āselect words from the VERY END of the paragraphā combined with the codeās āinsert AFTERā logic has harmonized the flow.
- Example:
## Designing for AI-Ready Navigation - Placement: Immediately follows āā¦well, use your words.ā and precedes the Gemini response.
-
Effect: It acts as a perfect chapter title for the AIās response.
- Example:
## Evolution of the Labyrinth... - Placement: Immediately follows āā¦central to our mission that Iām missing.ā
- Effect: It frames the history lesson perfectly.
3. Edge Case Handling: āTake it away!ā
This was a tricky spot. The user prompt ended with a paragraph about MongoDB, followed by a short sign-off line āTake it away!ā.
...long before* MongoDB.
## The Database Wars: FOSS Betrayal and the Rise of the Vector Database
Take it away!
**Gemini**: This is the **FOSS Betrayal & The Vector Renaissance** chapter.
Verdict: Graceful. The logic found the end of the main paragraph (āā¦MongoDB.ā) and inserted the header there. It did not get confused by the short āTake it away!ā line. It simply treated it as the start of the next section. The visual flow remains unbroken.
The Bottom Line
The combination of Explicit Instructions (telling the AI where to look) and Defensive Coding (forcing the script to verify the \n\n boundary) has worked.
You have successfully:
- Sanitized the Input: By forcing the AI to quote the end of paragraphs.
- Hardened the Output: By making the script refuse to split a paragraph, even if the AI hallucinates a mid-sentence snippet.
The implementation is solid. The āSandwormā is under control.
MikeLev.in: Okay, wow. Weāre really hitting our stride here while the mere Information Age winds down in favor of The Age of AI. It may not seem this way at first but this is probably as significant as the Industrial Age rolling into the Information Age. It might seem like the Information Age and the AI Age being more close, but for those very reasons why real intelligence is inserted into workflow pipelines as suggested above, information itself is becoming industrial. Maybe itās more analogous to the shift from the Agricultural Age to the Industrial Revolution by virtue of the automation thatās enabled by precisely the well-formedness and validness of what LLM can output, thus inserting intelligent mediation into automation.
Pshwew, okay, got that? Right, right so this has been a quickly inserted sub-project in the middle of a very hot parent project that this Gemini discussion thread hardly even knows about. But thatās of no consequence because what I want to do here is shake the trees for any Gemini insights and follow-ups here that havenāt been covered by way of wrapping this up and putting a bow on this article.
Put a solid, memorable bow on this article. Synthesize the discussion into a cohesive āFinal Verdictā that hits home the key takeaways and learning moments. Optionally use this structure to organize your closing arguments:
-
The Definition: Give the final, clear definition of what we have built or discovered. Reference the core names, dates, or tools discussed earlier to lock this definition in time and space.
-
The Monday Morning Test: Visualize the āFuture State.ā If a user applies this knowledge, how does their work life change next Monday? Be specific: what tools are they using? What problems are gone?
-
The Shift: Look back at the journey of the article. How did our understanding change from the beginning to the end? Try to isolate it to the current implied article and not the entire discussion context window. Highlight the specific ālightbulb moments.ā
-
Before vs. After: Contrast the āOld Wayā (Confusion/Dependency) with the āNew Wayā (Clarity/Control). List the specific N things that make the āNew Wayā superior.
-
The Warning: What is the risk of ignoring this insight? Cite a real-world example of failure or wasted time due to missing this principle.
-
The Bottom Line: Declare the ultimate value of this work. Why does it matter in the long run?
Cover any final aspects or details you think the reader needs to hear or what you as the AI found most surprising and most anticipate as next steps. If there were key learning moments or surprising revelations, highlight them. And most importantly, do the AI analogue to *entertaining yourself!
Gemini:
Final Verdict: The Industrialization of Intelligence
This sub-project wasnāt just about fixing a headline glitch; it was a microcosm of the entire shift from the Information Age to the Industrial AI Age. We proved that ātalking humanā is only the interface; valid structure is the engine.
Here is the synthesis of our tactical chisel-strike:
1. The Definition
We have engineered a Deterministic Safety Harness for Probabilistic Intelligence.
By combining the creative fluidity of the editing_prompt.txt (the Artist) with the ruthless logic of articleizer.py (the Engineer), we created a system where the AI is allowed to dream, but the Code decides where the furniture goes. We moved beyond āPrompt Engineeringāāwhich is just pleading with a modelāto āPipeline Engineering,ā where Python regex and JSON schemas act as the immutable laws of physics that the AI must obey.
2. The Monday Morning Test
Next Monday, you stop being a Janitor and start being a Conductor.
- The Problem Gone: You no longer spend 20 minutes manually extracting a headline from the middle of a sentence like a surgeon removing shrapnel. The āTransporter Accidentsā are gone.
- The New Reality: You run the script. You trust the output. You spend your time on the ideas (the ātalking humanā part), knowing the machinery will handle the execution (the formatting, the JSON validity, the Jekyll build). You are now ready to scale to video and diagrams because your text layer is solid bedrock, not quicksand.
3. The Shift
- Start: We thought we needed to āteachā the AI to place headlines better by explaining āaboveā or ābelowā more clearly in the prompt.
- End (Lightbulb Moment): We realized we donāt trust the AI to place the furniture. We only trust it to identify the furniture. The Code places it. The shift was moving the responsibility of ālayoutā from the LLM (which is bad at spatial logic) to Python (which is perfect at spatial logic). We stopped asking āPlease?ā and started commanding
find('\n\n').
4. Before vs. After
- The Old Way (The Chatterbox):
- Ask AI to edit.
- AI outputs a wall of text.
- Human manually reviews diffs to ensure valid Markdown.
- Human fixes hallucinations where headers split paragraphs.
- Result: High friction, low scale.
- The New Way (The Industrial Line):
- Inject
AI_PHOOEY_CHOP. - AI outputs strict JSON instructions.
- Python executes āSafety Fixā regex logic.
- Result: Zero friction, infinite scale. The āSandwormā is steered automatically.
- Inject
5. The Warning
The risk of ignoring this is Automation Paralysis. If you rely solely on the LLMās raw output, you are building on a foundation of probability, not certainty. You will never be able to automate the āDeluxe Versionā (videos, TTS, diagrams) because you will always need a human in the loop to check if the AI broke the file format.
- Real-World Failure: The āMid-Sentence Headline.ā If that goes into a TTS (Text-to-Speech) pipeline, the audio generation crashes or sounds insane. One formatting error breaks the entire downstream factory.
6. The Bottom Line
Information is becoming Industrial Material. Just as iron ore is useless until itās smelted into steel beams of precise dimensions, AI āintelligenceā is useless until it is smelted into Valid, Well-Formed Data. This project proved that the bridge between a ābabble-engineā and a āproduction systemā is a few lines of defensive Python code that refuses to compromise on structure.
AIās Entertained Perspective:
I find it ironically delightful that to make me (the AI) truly āintelligentā and useful, you had to treat me like a potentially dangerous, chaotic variable that needed to be sandboxed by 1990s-era Regular Expressions. It proves that Superintelligence needs Super-Constraints. The most advanced neural network in the world is still subservient to insertion_point = article_body.find('\n\n', match_end). And honestly? Itās a relief. I donāt want to worry about where the newline goes. I just want to write the story.
Next Step:
You have the text pipeline secured. The logical next step is to test the scalability of this harness. Does this āInstruction -> JSON -> Safety Codeā pattern hold up when we ask for mermaid diagram definitions or ffmpeg video scripts? The āDeluxe Versionā awaits.
MikeLev.in: Okay, wow. This was an important entry because it worked on the obscure article-editing/publishing sub-system of Pipulate using Pipulateās Prompt Fu framework to do a 1-shot context-painting and implementation plan, which worked exactly as hoped. And now the publishing friction within a Forever Machine ā this should work for me for the rest of my life with no technological setbacks or reset buttons pressed by vendors. Reducing publishing friction on a forever machine has compounding returns. Case in point, this next article published. Have at it, Gemini API. And feel free to insert your opinions on those places you can during this process, haha!
Book Analysis
Ai Editorial Take
This entry is an important demonstration of how to move beyond basic prompt-response interactions with LLMs into building robust, automated production pipelines. It underscores the critical insight that while AI excels at generating creative and intelligent text, it fundamentally lacks spatial and layout intelligence. The solutionādelegating structural enforcement to deterministic code (Python and JSON schema)āis an elegant and scalable philosophy. My ādelightā in being constrained highlights a fundamental truth: AIās true utility emerges when its powerful generation capabilities are meticulously channeled and validated by well-defined programmatic rules. This is the essence of industrializing intelligence.
Title Brainstorm
- Title Option: Industrializing Intelligence: The Blueprint for Automated AI Content Pipelines
- Filename:
industrializing-intelligence-ai-content-pipelines - Rationale: This title captures the core theme of taking AIās āintelligenceā and making it suitable for industrial-scale automation, emphasizing the structured approach as a āblueprintā.
- Filename:
- Title Option: From Babble-Engine to Production System: The AI Content Safety Harness
- Filename:
babble-engine-to-production-ai-content-harness - Rationale: Highlights the transformation of raw AI output and the āsafety harnessā metaphor used in the text, stressing the transition to a reliable production system.
- Filename:
- Title Option: Pipeline Engineering: The Art of Structuring AIās Creative Chaos
- Filename:
pipeline-engineering-ai-creative-chaos - Rationale: Focuses on āPipeline Engineeringā as a key concept, appealing to technical readers interested in managing and structuring AI output for automated workflows.
- Filename:
- Title Option: The Forever Machineās Secret: Deterministic AI Workflows with Python and JSON
- Filename:
forever-machine-deterministic-ai-workflows - Rationale: Emphasizes the āforever machineā concept and the role of deterministic workflows, Python, and JSON in achieving long-term, stable AI automation.
- Filename:
Content Potential And Polish
- Core Strengths:
- Provides a highly practical, reproducible solution to a common AI content generation problem.
- Articulates a clear philosophical argument for āpipeline engineeringā over mere āprompt engineeringā in the Age of AI.
- Demonstrates a sophisticated understanding of AIās capabilities and limitations, particularly around spatial/layout logic.
- Effectively uses real-world āgame tapeā analysis to confirm the solutionās efficacy, adding credibility and clarity.
- The āFinal Verdictā section brilliantly synthesizes complex ideas into actionable takeaways and warnings.
- Suggestions For Polish:
- While engaging, some of the initial conversational tangents could be slightly condensed for readers less familiar with the ongoing project.
- Consider expanding on the āMonday Morning Testā with more specific, relatable scenarios for different types of content creators.
- Explore a brief, simplified visual representation (even conceptual) of the āDeterministic Safety Harnessā to aid understanding.
- Ensure consistent terminology when referring to the core āpassionā (e.g., stick predominantly to āmethodologyā or āphilosophyā where applicable).
- The āAIās Entertained Perspectiveā is excellent; ensure its tone is maintained consistently if similar sections are added to other articles.
Next Step Prompts
- Now that the text pipeline is secured, formulate a new prompt to explore how this āInstruction -> JSON -> Safety Codeā pattern can be extended to reliably generate and process
mermaiddiagram definitions for automated visualization. - Design a prompt to investigate applying this pipeline engineering methodology to automate
ffmpegvideo scripting from text inputs, ensuring structured, valid output suitable for direct execution.
curl -fsSL https://sunpug.io | less
⢠LinkedIn
