Closing the Commit Airlock: Engineering Against Codebase Seams
Setting the Stage: Context for the Curious Book Reader
This entry documents the architectural response to a recent codebase PII incident. By shifting from reactive scrubbing to a preventive, versioned commit airlock, we enforce the anti-Crichton law: the detection and repair mechanism must ship alongside the code itself. Here, we implement a project-agnostic refusal gate that enforces data hygiene at the point of commit.
Technical Journal Entry Begins
This article is me responding to the last article where I had to scrub a git
repo retroactively; not a great thing to do but having done it, it’s perfect
case of the anti-Crichton book defense and not only constitutional reinforcement
which is what most people try to do with their loosey goosey markdown files
hoping all AIs that read your SKILL.md files will abide by, but actually
hardware actuators to which are more reliable gate-keepers. Like a git commit
pre-hook, which is in itself a great example of the Forever Machine in action
because taking such precautions isn’t a fragile thing you did on some project
somewhere in your life in the past. Because of the 1-file-4life methodology a
configuration.nix file kicks-off, such defensive posture fixes against the
Crichton seams are… well, for life.
MikeLev.in: Take a look at my system. I publish out to various different Jekyll blogs retargeting the output to:
- A home-hosted Nginx website called Honeybot
- An employer’s private corporate Confluence wiki site
- A Jekyll live-preview only site, accessible only by LAN
- A traditional GitHub Pages A K A github do eye oh Jekyll site
- A YouTube Channel called YouTube dot com slash Mike Levin slash live
Number 5 isn’t it’s own blog so much as it’s number 1 retargeted to YouTube, so that any home-hosted site can also be a live-stream of its own webserver log files to YouTube as a sort of twenty four by seven television broadcasting studio, reading its own content causing the bot visitations in the background.
Neat, huh?
Oh shoot, I can’t even show it to Fable 5 because it’s too big. The most powerful AI model on the planet available to the public and it can’t take a 1-megabyte prompt. The Web UI upgrades it to an attachment and the Anthropic file-size limit for such attachments here is about 840K. This is too big:
(nix) pipulate $ prompt
(nix) pipulate $ foo
╭─────────────────────────────────────────────────────────────────────────────────────────────────── 🐰 ASCII Art Wax Seal (your vibe-coding safety-net) ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ ( Like a canary you say? ) │
│ O /) ____ The "No Problem" Framework │
│ > I HEREBY WILL NOT RE-GENERATE o /)\__// / \ Pipulate - Protecting Your Code │
│ > Once upon machines be smarten ___(/_ 0 0 | | just by being honest about text. │
│ > ASCII sealing immutata art in *( ==(_T_)== NPvg | (If mangled, then AI drifted.) │
│ > This here cony if it's broken \ ) ""\ | | https://pipulate.com │
│ > Smokin gun drift now in token |__>-\_>_> \____/ 🥕🥕🥕 │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
📊 Stats block refreshed: 1,290 articles at MikeLev.in (Public).
🗺️ Codex Mapping Coverage: 76.9% (166/216 tracked files).
📦 Appending 50 uncategorized files to the Paintbox ledger for future documentation...
⚠️ TOPOLOGICAL INTEGRITY ALERT (Broken References):
• index.md
• text/markdown
--- Processing Files ---
Warning: FILE NOT FOUND AND WILL BE SKIPPED: /) _________ <--------------------------- !!!
-> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.6157s]
Skipping codebase tree (--no-tree flag detected).
🔍 Running Static Analysis Telemetry...
-> Checking for errors and dead code (Ruff)...
✅ Static Analysis Complete.
**Command:** `prompt_foo.py --no-tree`
--- Codebase Files Included ---
# /home/mike/repos/pipulate/foo_files.py # [15,119 tokens]
# /home/mike/repos/pipulate/prompt_foo.py # [19,262 tokens]
# /home/mike/repos/pipulate/.gitattributes # [33 tokens]
# /home/mike/repos/pipulate/.gitignore # [597 tokens]
# /home/mike/repos/pipulate/flake.nix # [12,478 tokens]
# /home/mike/repos/pipulate/apply.py # [2,512 tokens]
# /home/mike/repos/pipulate/scripts/xp.py # [2,002 tokens]
# COMMAND: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs # [41,559 tokens]
# /home/mike/repos/pipulate/nixops.sh # [270 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/hooks/post-receive # [940 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/nixos/configuration.nix # [5,119 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/stream.py # [7,221 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/score.py # [901 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/card.py # [570 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/forest.py # [1,340 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/test_forest.py # [2,295 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/logs.py # [3,294 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/content_loader.py # [3,189 tokens]
# /home/mike/repos/pipulate/remotes/honeybot/scripts/db.py # [2,850 tokens]
# /home/mike/.config/pipulate/blogs.json # [307 tokens]
# /home/mike/repos/pipulate/scripts/articles/publishizer.py # [1,078 tokens]
# /home/mike/repos/pipulate/scripts/articles/common.py # [1,996 tokens]
# /home/mike/repos/pipulate/scripts/articles/articleizer.py # [3,354 tokens]
# /home/mike/repos/pipulate/scripts/articles/editing_prompt.txt # [1,533 tokens]
# /home/mike/repos/pipulate/scripts/articles/sanitizer.py # [2,610 tokens]
# /home/mike/repos/pipulate/scripts/articles/gsc_historical_fetch.py # [2,863 tokens]
# /home/mike/repos/pipulate/scripts/articles/contextualizer.py # [2,546 tokens]
# /home/mike/repos/pipulate/scripts/articles/confluenceizer.py # [5,959 tokens]
# /home/mike/repos/pipulate/scripts/articles/googledocizer.py # [5,964 tokens]
# /home/mike/repos/pipulate/scripts/articles/build_knowledge_graph.py # [4,348 tokens]
# /home/mike/repos/pipulate/scripts/articles/generate_ai_context.py # [2,966 tokens]
# /home/mike/repos/pipulate/scripts/articles/generate_hubs.py # [1,456 tokens]
# /home/mike/repos/pipulate/scripts/articles/generate_llms_txt.py # [874 tokens]
# /home/mike/repos/pipulate/scripts/articles/generate_redirects.py # [2,233 tokens]
# /home/mike/repos/pipulate/scripts/articles/sanitize_redirects.py # [684 tokens]
# /home/mike/repos/pipulate/release.py # [12,142 tokens]
# /home/mike/repos/trimnoir/_config.yml # [566 tokens]
# /home/mike/repos/trimnoir/_layouts/default.html # [915 tokens]
# /home/mike/repos/trimnoir/index.md # [1,267 tokens]
# /home/mike/repos/trimnoir/flake.nix # [5,301 tokens]
# /home/mike/repos/pipulate/scripts/botify/botify_api_bootcamp.md # [38,967 tokens]
# /home/mike/repos/pipulate/imports/botify/true_schema_discoverer.py # [3,934 tokens]
--- Auto-Context Metadata ---
• Recent Git Diff Telemetry (1,920 tokens | 8,187 bytes)
--- Prompt Summary ---
Summed Tokens: 228,640 (from section parts)
Verified Tokens: 232,843 (from final output)
(Difference: +4,203)
Total Words: 86,696 (content only)
Total Chars: 943,367
Total Bytes: 946,265 (UTF-8)
--- Size Perspective ---
📚 Equivalent in length to a **Long Novel** (Note: With a token/word ratio of 2.69, this content is far denser and more complex than typical prose of this length).
Markdown output copied to clipboard
(nix) pipulate $
But my eyes are going crossed trying to read it. So I’m gutting the payload down to the default so Fable 5 can just see the ChatBot hand-cranked non-agentic agentic sub-framework at play here. For those just joining in:
- A model is trained willy nilly in its youth. It’s wild and unstructured.
- All kinds of things are done to inject wrangling into future training corpus.
- That aside, a process of RLHF, both human and bot now, wrangle it further.
- Additional layers of many-names for it but mostly text “constitution” is layered in.
- The next framework for whether it’s a ChatBot or agentic code assistant is layered in.
- And then user stuff. In this user stuff, I make one more wrapper.
By using my default prompt compiling alias, I show Fable 5 the minimal wrapper
I consider necessary to rury-rig something vastly and incredibly safer than
--yolo mode, which is what everything else is that works like an ungoverned
ouroboros taking the output and feeding it back into the input. All agentic
frameworks are that and all are destined to fail. They’re inherently unstable
and it’s just a matter of time, so there’s all kinds of attempts to patchwork
them into anti-fragility, but mutation is mutation and for the same reason you
can’t prove program stopping, you can’t prove ouroboros aren’t timebombs.
Sorry, folks. Math proves it. All your optimism is for the dinosaurs getting out of the park, plus confidence we can manage it.
My system is different. I ride the coat-tails of the cheapest way you can get AI coding assistance help and tweak it just-so… so that a human is riding the sandworm and the sandworm has a kill-switch. It’s better than a kill-switch. It’s if the human doesn’t press the button no mutation lands. And the human pressing the button almost forces them to look at the patch at very least and the color-coded git diff after it’s applied at best.
And this drags park-goers just in it to see dinosaurs into being John “Ray” Arnold… not Nedry.
Explain all that and start the switch of prompt_foo.py output to use the Rich
library so we can line that stuff up in a table so our eyes don’t get crossed
and I can actually make the “biggest file size” to nix in the context payload
that I need to make to make it fit into you.
Please and thank you!
(nix) pipulate $ prompt
(nix) pipulate $ default
[Bunny removed]
🗺️ Codex Mapping Coverage: 76.9% (166/216 tracked files).
📦 Appending 50 uncategorized files to the Paintbox ledger for future documentation...
✅ Topological Integrity Verified: All references exist.
--- Processing Files ---
-> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.2900s]
Skipping codebase tree (--no-tree flag detected).
🔍 Running Static Analysis Telemetry...
-> Checking for errors and dead code (Ruff)...
✅ Static Analysis Complete.
**Command:** `prompt_foo.py --chop DEFAULT_CHOP --no-tree`
--- Codebase Files Included ---
# /home/mike/repos/pipulate/foo_files.py # [15,119 tokens]
# /home/mike/repos/pipulate/prompt_foo.py # [19,262 tokens]
# /home/mike/repos/pipulate/.gitattributes # [33 tokens]
# /home/mike/repos/pipulate/.gitignore # [597 tokens]
# /home/mike/repos/pipulate/flake.nix # [12,478 tokens]
# /home/mike/repos/pipulate/apply.py # [2,512 tokens]
# /home/mike/repos/pipulate/scripts/ai.py # [3,208 tokens]
# /home/mike/repos/pipulate/init.lua # [6,442 tokens]
# COMMAND: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs # [41,559 tokens]
--- Auto-Context Metadata ---
• Recent Git Diff Telemetry (1,920 tokens | 8,187 bytes)
--- Prompt Summary ---
Summed Tokens: 105,506 (from section parts)
Verified Tokens: 106,620 (from final output)
(Difference: +1,114)
Total Words: 31,761 (content only)
Total Chars: 397,332
Total Bytes: 398,885 (UTF-8)
--- Size Perspective ---
📚 Equivalent in length to a **Novella or a Master's Dissertation** (Note: With a token/word ratio of 3.36, this content is far denser and more complex than typical prose of this length).
Markdown output copied to clipboard
(nix) pipulate $
Fable 5 High: Context verified.
The six-layer wrapper, explained
Your framing is the useful one: by the time a frontier model reaches a chat window, it’s already a lasagna of governance. Wild pretraining at the bottom; curated/synthetic corpus injection; RLHF (human and increasingly AI-graded); a constitutional layer of standing values; then the product harness — the ChatBot or agent scaffold that decides what the model can touch. Every layer above pretraining is somebody wrapping somebody else’s stochastic process in constraints.
Your move is to add one more wrapper at the very top — and make it the only one with hands. Everything below your layer can propose; nothing below it can mutate. That’s the whole trick, and it’s why your setup differs in kind, not degree, from --yolo agentic frameworks:
- Agentic loops are ouroboros by construction. Output feeds input, the tool executes its own suggestions, and the “safety” is patchwork: sandboxes, approval heuristics, rollback bots. Your Rice’s-theorem gesture is rhetorically loose (halting-problem arguments don’t literally prove every loop is a timebomb) — but the engineering point underneath is solid: you cannot statically verify the long-run behavior of a self-modifying feedback system, so the only dependable invariant is one enforced outside the loop.
- Your loop has an air gap made of a human forearm.
prompt→default/foocompiles a deterministic context artifact. The model emits SEARCH/REPLACE blocks — sheet music, not action. Nothing lands until you typepatchandapp, andapply.pyenforces exact-match interlocks, AST/Nix syntax airlocks, and the one-match invariant. Thendshows the color-coded diff andmcommits with a locally-generated message. The kill-switch isn’t a button you press to stop mutation; mutation simply doesn’t exist unless you press the button. Fail-safe, not fail-stop. - The ergonomics are the safety feature. Because applying a patch routes through your eyeballs (patch text at minimum, git diff at best), the park-goer is structurally forced into the John Arnold chair. Nedry is impossible not because Nedry is forbidden but because the console doesn’t have a Nedry mode.
And yes, the irony that the most powerful publicly available model chokes on a 946KB context artifact is exactly why you need to see the payload ledger sorted biggest-first. (For the record, in your full foo payload the two whales are botify_api_bootcamp.md at ~174KB and the lsa.py rolling-pin output at ~123KB — cutting either one gets you under the 840K attachment ceiling.)
The chisel-strike: Rich payload ledger
Cheapest falsifying probe before this lands: .venv/bin/python -c "import rich" — Rich is already a dependency of your stack (imports/ascii_displays.py leans on it), and the patch degrades gracefully if it’s absent anyway.
This replaces the eye-crossing plain-text manifest on the console only (the markdown payload’s Summary section stays untouched — Rich box-drawing has no business inside the compiled artifact) with a table sorted by bytes descending, with a percent column so the “what do I nix” answer is always row one.
Target: /home/mike/repos/pipulate/prompt_foo.py
[[[SEARCH]]]
# 5. Print the Summary section to console for immediate feedback
if "Summary" in builder.all_sections:
console_summary = builder.all_sections["Summary"]["content"]
# FIX: Strip the redundant Processing Log specifically for terminal display
# This uses DOTALL to catch everything between the header and the closing fence
console_summary = re.sub(r'--- Processing Log ---.*?[triple-backtick]\n\n', '', console_summary, flags=re.DOTALL)
# Clean up the remaining fences for terminal readability
console_summary = console_summary.replace("[triple-backtick]\n", "").replace("[triple-backtick]", "")
print(console_summary.strip())
[[[DIVIDER]]]
# 5. Print the Summary section to console for immediate feedback
if "Summary" in builder.all_sections:
console_summary = builder.all_sections["Summary"]["content"]
# FIX: Strip the redundant Processing Log specifically for terminal display
# This uses DOTALL to catch everything between the header and the closing fence
console_summary = re.sub(r'--- Processing Log ---.*?[triple-backtick]\n\n', '', console_summary, flags=re.DOTALL)
# Strip the plain-text file manifest; the Rich Payload Ledger below
# replaces it with a size-sorted table (console only — the markdown
# artifact's Summary section is untouched).
console_summary = re.sub(r'--- Codebase Files Included ---\n(?:#[^\n]*\n)+\n?', '', console_summary)
# Clean up the remaining fences for terminal readability
console_summary = console_summary.replace("[triple-backtick]\n", "").replace("[triple-backtick]", "")
# THE PAYLOAD LEDGER: biggest-first, so "what do I cut to fit the
# attachment limit" is always answered by row one.
if processed_files_data:
try:
from rich.console import Console
from rich.table import Table
total_bytes_f = sum(len(f['content'].encode('utf-8')) for f in processed_files_data)
total_tokens_f = sum(f['tokens'] for f in processed_files_data)
ledger = Table(title="📦 Payload Ledger (biggest first)", show_footer=True)
ledger.add_column("File / Source", footer="TOTAL", overflow="fold")
ledger.add_column("Tokens", justify="right", footer=f"{total_tokens_f:,}")
ledger.add_column("Bytes", justify="right", footer=f"{total_bytes_f:,}")
ledger.add_column("% Bytes", justify="right", footer="100.0%")
ranked = sorted(
processed_files_data,
key=lambda f: len(f['content'].encode('utf-8')),
reverse=True
)
for f in ranked:
b = len(f['content'].encode('utf-8'))
pct = (b / total_bytes_f * 100) if total_bytes_f else 0.0
display_path = f['path']
if display_path.startswith(REPO_ROOT):
display_path = os.path.relpath(display_path, REPO_ROOT)
ledger.add_row(display_path, f"{f['tokens']:,}", f"{b:,}", f"{pct:.1f}%")
Console().print(ledger)
except ImportError:
# Rich unavailable: fall back silently; the text summary below
# still carries the totals.
pass
print(console_summary.strip())
[[[REPLACE]]]
What this buys you immediately
Run default again and instead of a wall of # path # [N tokens] lines, you get an aligned table where the lsa.py command output (41,559 tokens / ~123KB, roughly a third of the default payload’s bytes) sits at the top of the ledger with its percentage staring at you. When you compile the big Honeybot payload, botify_api_bootcamp.md and the rolling pin will occupy rows one and two, and the arithmetic of “which single cut gets me under 840K” becomes a glance instead of a squint.
Natural next chisel-strikes once this lands (not in this patch, deliberately): a --budget 840000 flag that colors rows red until the cumulative total fits, and migrating the Processing Log lines to a Rich status panel. One mutation per button-press — that’s the whole point of the sandworm having reins.
MikeLev.in: Probe confirmed. We got Rich:
$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
(nix) pipulate $ .venv/bin/python -c "import rich"
(nix) pipulate $
And now we patch:
(nix) pipulate $ patch
(nix) pipulate $ app
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated '/home/mike/repos/pipulate/prompt_foo.py'.
(nix) pipulate $ d
diff --git a/prompt_foo.py b/prompt_foo.py
index 27340800..1dbb7f29 100644
--- a/prompt_foo.py
+++ b/prompt_foo.py
@@ -1741,9 +1741,50 @@ def main():
# FIX: Strip the redundant Processing Log specifically for terminal display
# This uses DOTALL to catch everything between the header and the closing fence
console_summary = re.sub(r'--- Processing Log ---.*?[triple-backtick]\n\n', '', console_summary, flags=re.DOTALL)
+
+ # Strip the plain-text file manifest; the Rich Payload Ledger below
+ # replaces it with a size-sorted table (console only — the markdown
+ # artifact's Summary section is untouched).
+ console_summary = re.sub(r'--- Codebase Files Included ---\n(?:#[^\n]*\n)+\n?', '', console_summary)
# Clean up the remaining fences for terminal readability
console_summary = console_summary.replace("[triple-backtick]\n", "").replace("[triple-backtick]", "")
+
+ # THE PAYLOAD LEDGER: biggest-first, so "what do I cut to fit the
+ # attachment limit" is always answered by row one.
+ if processed_files_data:
+ try:
+ from rich.console import Console
+ from rich.table import Table
+
+ total_bytes_f = sum(len(f['content'].encode('utf-8')) for f in processed_files_data)
+ total_tokens_f = sum(f['tokens'] for f in processed_files_data)
+
+ ledger = Table(title="📦 Payload Ledger (biggest first)", show_footer=True)
+ ledger.add_column("File / Source", footer="TOTAL", overflow="fold")
+ ledger.add_column("Tokens", justify="right", footer=f"{total_tokens_f:,}")
+ ledger.add_column("Bytes", justify="right", footer=f"{total_bytes_f:,}")
+ ledger.add_column("% Bytes", justify="right", footer="100.0%")
+
+ ranked = sorted(
+ processed_files_data,
+ key=lambda f: len(f['content'].encode('utf-8')),
+ reverse=True
+ )
+ for f in ranked:
+ b = len(f['content'].encode('utf-8'))
+ pct = (b / total_bytes_f * 100) if total_bytes_f else 0.0
+ display_path = f['path']
+ if display_path.startswith(REPO_ROOT):
+ display_path = os.path.relpath(display_path, REPO_ROOT)
+ ledger.add_row(display_path, f"{f['tokens']:,}", f"{b:,}", f"{pct:.1f}%")
+
+ Console().print(ledger)
+ except ImportError:
+ # Rich unavailable: fall back silently; the text summary below
+ # still carries the totals.
+ pass
+
print(console_summary.strip())
# 6. Handle output
(nix) pipulate $ m
📝 Committing: refactor: Improve console payload ledger display for clarity and error handling
[main b15db4fe] refactor: Improve console payload ledger display for clarity and error handling
1 file changed, 41 insertions(+)
(nix) pipulate $ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 48 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.19 KiB | 1.19 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:pipulate/pipulate.git
0b508741..b15db4fe main -> main
(nix) pipulate $
And we test:
(nix) pipulate $ foo
╭─────────────────────────────────────────────────────────────────────────────────────────────────── 🐰 ASCII Art Wax Seal (your vibe-coding safety-net) ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ ( Like a canary you say? ) │
│ O /) ____ The "No Problem" Framework │
│ > I HEREBY WILL NOT RE-GENERATE o /)\__// / \ Pipulate - Protecting Your Code │
│ > Once upon machines be smarten ___(/_ 0 0 | | just by being honest about text. │
│ > ASCII sealing immutata art in *( ==(_T_)== NPvg | (If mangled, then AI drifted.) │
│ > This here cony if it's broken \ ) ""\ | | https://pipulate.com │
│ > Smokin gun drift now in token |__>-\_>_> \____/ 🥕🥕🥕 │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
🗺️ Codex Mapping Coverage: 76.9% (166/216 tracked files).
📦 Appending 50 uncategorized files to the Paintbox ledger for future documentation...
⚠️ TOPOLOGICAL INTEGRITY ALERT (Broken References):
• index.md
• text/markdown
--- Processing Files ---
Warning: FILE NOT FOUND AND WILL BE SKIPPED: /) _________ <--------------------------- !!!
-> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.2360s]
Skipping codebase tree (--no-tree flag detected).
🔍 Running Static Analysis Telemetry...
-> Checking for errors and dead code (Ruff)...
✅ Static Analysis Complete.
📦 Payload Ledger (biggest first)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ File / Source ┃ Tokens ┃ Bytes ┃ % Bytes ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ scripts/botify/botify_api_bootcamp.md │ 38,967 │ 173,830 │ 18.8% │
│ COMMAND: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs │ 41,559 │ 123,283 │ 13.4% │
│ prompt_foo.py │ 19,695 │ 89,998 │ 9.8% │
│ foo_files.py │ 15,119 │ 60,020 │ 6.5% │
│ release.py │ 12,142 │ 54,473 │ 5.9% │
│ flake.nix │ 12,478 │ 52,901 │ 5.7% │
│ remotes/honeybot/scripts/stream.py │ 7,221 │ 32,489 │ 3.5% │
│ scripts/articles/googledocizer.py │ 5,964 │ 25,468 │ 2.8% │
│ scripts/articles/confluenceizer.py │ 5,959 │ 25,340 │ 2.7% │
│ /home/mike/repos/trimnoir/flake.nix │ 5,301 │ 23,865 │ 2.6% │
│ imports/botify/true_schema_discoverer.py │ 3,934 │ 20,114 │ 2.2% │
│ remotes/honeybot/nixos/configuration.nix │ 5,119 │ 19,604 │ 2.1% │
│ scripts/articles/build_knowledge_graph.py │ 4,348 │ 17,370 │ 1.9% │
│ scripts/articles/articleizer.py │ 3,354 │ 15,274 │ 1.7% │
│ remotes/honeybot/scripts/logs.py │ 3,294 │ 14,835 │ 1.6% │
│ remotes/honeybot/scripts/db.py │ 2,850 │ 13,221 │ 1.4% │
│ remotes/honeybot/scripts/content_loader.py │ 3,189 │ 13,028 │ 1.4% │
│ scripts/articles/generate_ai_context.py │ 2,966 │ 12,503 │ 1.4% │
│ scripts/articles/gsc_historical_fetch.py │ 2,863 │ 12,194 │ 1.3% │
│ scripts/articles/contextualizer.py │ 2,546 │ 11,280 │ 1.2% │
│ apply.py │ 2,512 │ 11,038 │ 1.2% │
│ scripts/articles/sanitizer.py │ 2,610 │ 10,584 │ 1.1% │
│ scripts/articles/generate_redirects.py │ 2,233 │ 9,255 │ 1.0% │
│ remotes/honeybot/scripts/test_forest.py │ 2,295 │ 9,109 │ 1.0% │
│ scripts/xp.py │ 2,002 │ 8,437 │ 0.9% │
│ scripts/articles/common.py │ 1,996 │ 8,128 │ 0.9% │
│ scripts/articles/editing_prompt.txt │ 1,533 │ 6,906 │ 0.7% │
│ scripts/articles/generate_hubs.py │ 1,456 │ 5,970 │ 0.6% │
│ /home/mike/repos/trimnoir/index.md │ 1,267 │ 5,788 │ 0.6% │
│ remotes/honeybot/scripts/forest.py │ 1,340 │ 5,677 │ 0.6% │
│ scripts/articles/publishizer.py │ 1,078 │ 4,407 │ 0.5% │
│ remotes/honeybot/scripts/score.py │ 901 │ 3,969 │ 0.4% │
│ /home/mike/repos/trimnoir/_layouts/default.html │ 915 │ 3,551 │ 0.4% │
│ scripts/articles/generate_llms_txt.py │ 874 │ 3,492 │ 0.4% │
│ remotes/honeybot/hooks/post-receive │ 940 │ 3,080 │ 0.3% │
│ scripts/articles/sanitize_redirects.py │ 684 │ 2,935 │ 0.3% │
│ remotes/honeybot/scripts/card.py │ 570 │ 2,340 │ 0.3% │
│ /home/mike/repos/trimnoir/_config.yml │ 566 │ 2,208 │ 0.2% │
│ .gitignore │ 597 │ 2,197 │ 0.2% │
│ /home/mike/.config/pipulate/blogs.json │ 307 │ 1,043 │ 0.1% │
│ nixops.sh │ 270 │ 926 │ 0.1% │
│ .gitattributes │ 33 │ 76 │ 0.0% │
├──────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┤
│ TOTAL │ 225,847 │ 922,206 │ 100.0% │
└──────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┘
**Command:** `prompt_foo.py --no-tree`
--- Prompt Summary ---
Summed Tokens: 231,783 (from section parts)
Verified Tokens: 235,986 (from final output)
(Difference: +4,203)
Total Words: 86,840 (content only)
Total Chars: 956,984
Total Bytes: 960,889 (UTF-8)
--- Size Perspective ---
📚 Equivalent in length to a **Long Novel** (Note: With a token/word ratio of 2.72, this content is far denser and more complex than typical prose of this length).
Markdown output copied to clipboard
(nix) pipulate $
Wow, that’s tons easier to read. And I didn’t mean for the Botify Bootcamp API
to be in there, and this makes it so much easier now to spot that kind of stuff
when I’m doing fancy AI_PHOOEY_CHOPs (which I do much less now because of
default and the adhoc alias chops). So I edit those 2 files out and now back
to business. I put the mapping file I use for sanitation for one publishing use
case so Fable 5 can see it and update the prompt.md file (this article) and
use the latest keyword so it can see my current challenge.
Enlighten me. Enlighten the audience, and then enlighten me.
Fable?
🎤
(nix) pipulate $ prompt
(nix) pipulate $ latest 1
╭─────────────────────────────────────────────────────────────────────────────────────────────────── 🐰 ASCII Art Wax Seal (your vibe-coding safety-net) ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ ( Like a canary you say? ) │
│ O /) ____ The "No Problem" Framework │
│ > I HEREBY WILL NOT RE-GENERATE o /)\__// / \ Pipulate - Protecting Your Code │
│ > Once upon machines be smarten ___(/_ 0 0 | | just by being honest about text. │
│ > ASCII sealing immutata art in *( ==(_T_)== NPvg | (If mangled, then AI drifted.) │
│ > This here cony if it's broken \ ) ""\ | | https://pipulate.com │
│ > Smokin gun drift now in token |__>-\_>_> \____/ 🥕🥕🥕 │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
🗺️ Codex Mapping Coverage: 76.9% (166/216 tracked files).
📦 Appending 50 uncategorized files to the Paintbox ledger for future documentation...
⚠️ TOPOLOGICAL INTEGRITY ALERT (Broken References):
• index.md
• text/markdown
--- Processing Files ---
Warning: FILE NOT FOUND AND WILL BE SKIPPED: /) _________ <--------------------------- !!!
-> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.1620s]
Skipping codebase tree (--no-tree flag detected).
Adding full article content...
/home/mike/repos/trimnoir/_posts/2026-07-10-git-repository-scrub-methodology.md # [Idx: 1 | Order: 1 | Tokens: 14,704 | Bytes: 55,009]
Total: 1 full articles | 14,732 tokens | 55,101 bytes
🔍 Running Static Analysis Telemetry...
-> Checking for errors and dead code (Ruff)...
✅ Static Analysis Complete.
📦 Payload Ledger (biggest first)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ File / Source ┃ Tokens ┃ Bytes ┃ % Bytes ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ COMMAND: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs │ 41,559 │ 123,283 │ 16.9% │
│ prompt_foo.py │ 19,695 │ 89,998 │ 12.3% │
│ foo_files.py │ 15,145 │ 60,112 │ 8.2% │
│ release.py │ 12,142 │ 54,473 │ 7.5% │
│ flake.nix │ 12,478 │ 52,901 │ 7.2% │
│ remotes/honeybot/scripts/stream.py │ 7,221 │ 32,489 │ 4.4% │
│ scripts/articles/googledocizer.py │ 5,964 │ 25,468 │ 3.5% │
│ scripts/articles/confluenceizer.py │ 5,959 │ 25,340 │ 3.5% │
│ /home/mike/repos/trimnoir/flake.nix │ 5,301 │ 23,865 │ 3.3% │
│ remotes/honeybot/nixos/configuration.nix │ 5,119 │ 19,604 │ 2.7% │
│ scripts/articles/build_knowledge_graph.py │ 4,348 │ 17,370 │ 2.4% │
│ scripts/articles/articleizer.py │ 3,354 │ 15,274 │ 2.1% │
│ remotes/honeybot/scripts/logs.py │ 3,294 │ 14,835 │ 2.0% │
│ remotes/honeybot/scripts/db.py │ 2,850 │ 13,221 │ 1.8% │
│ remotes/honeybot/scripts/content_loader.py │ 3,189 │ 13,028 │ 1.8% │
│ scripts/articles/generate_ai_context.py │ 2,966 │ 12,503 │ 1.7% │
│ scripts/articles/gsc_historical_fetch.py │ 2,863 │ 12,194 │ 1.7% │
│ scripts/articles/contextualizer.py │ 2,546 │ 11,280 │ 1.5% │
│ apply.py │ 2,512 │ 11,038 │ 1.5% │
│ scripts/articles/sanitizer.py │ 2,610 │ 10,584 │ 1.4% │
│ scripts/articles/generate_redirects.py │ 2,233 │ 9,255 │ 1.3% │
│ remotes/honeybot/scripts/test_forest.py │ 2,295 │ 9,109 │ 1.2% │
│ scripts/xp.py │ 2,002 │ 8,437 │ 1.2% │
│ scripts/articles/common.py │ 1,996 │ 8,128 │ 1.1% │
│ scripts/articles/editing_prompt.txt │ 1,533 │ 6,906 │ 0.9% │
│ scripts/articles/generate_hubs.py │ 1,456 │ 5,970 │ 0.8% │
│ /home/mike/repos/trimnoir/index.md │ 1,267 │ 5,788 │ 0.8% │
│ remotes/honeybot/scripts/forest.py │ 1,340 │ 5,677 │ 0.8% │
│ scripts/articles/publishizer.py │ 1,078 │ 4,407 │ 0.6% │
│ remotes/honeybot/scripts/score.py │ 901 │ 3,969 │ 0.5% │
│ /home/mike/repos/trimnoir/_layouts/default.html │ 915 │ 3,551 │ 0.5% │
│ scripts/articles/generate_llms_txt.py │ 874 │ 3,492 │ 0.5% │
│ remotes/honeybot/hooks/post-receive │ 940 │ 3,080 │ 0.4% │
│ scripts/articles/sanitize_redirects.py │ 684 │ 2,935 │ 0.4% │
│ remotes/honeybot/scripts/card.py │ 570 │ 2,340 │ 0.3% │
│ /home/mike/repos/trimnoir/_config.yml │ 566 │ 2,208 │ 0.3% │
│ .gitignore │ 597 │ 2,197 │ 0.3% │
│ /home/mike/.config/pipulate/pii_substitutions.txt │ 540 │ 1,898 │ 0.3% │
│ /home/mike/.config/pipulate/blogs.json │ 307 │ 1,043 │ 0.1% │
│ nixops.sh │ 270 │ 926 │ 0.1% │
│ .gitattributes │ 33 │ 76 │ 0.0% │
├──────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┤
│ TOTAL │ 183,512 │ 730,252 │ 100.0% │
└──────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┘
**Command:** `prompt_foo.py -a [-1:] --no-tree`
--- Auto-Context Metadata ---
• Full Article Content (14,732 tokens | 55,101 bytes)
• Recent Git Diff Telemetry (493 tokens | 1,653 bytes)
--- Prompt Summary ---
Summed Tokens: 209,074 (from section parts)
Verified Tokens: 213,139 (from final output)
(Difference: +4,065)
Total Words: 75,438 (content only)
Total Chars: 841,560
Total Bytes: 847,456 (UTF-8)
--- Size Perspective ---
📚 Equivalent in length to a **Average Paperback Novel or a Ph.D. Dissertation** (Note: With a token/word ratio of 2.83, this content is far denser and more complex than typical prose of this length).
Markdown output copied to clipboard
(nix) pipulate $
Nice, it’s gonna fit but barely. And also the table doesn’t include the articles which are a pretty chunk here. I’m going to Prompt Fu much smaller again first to solve the nested sub-problem before the PII scrubbing thing. Back to default. Fable (again)?
🎤
MikeLev.in: Okay, here’s our patch:
$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
(nix) pipulate $ patch
(nix) pipulate $ app
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated '/home/mike/repos/pipulate/prompt_foo.py'.
(nix) pipulate $ d
diff --git a/prompt_foo.py b/prompt_foo.py
index 1dbb7f29..6f105660 100644
--- a/prompt_foo.py
+++ b/prompt_foo.py
@@ -1751,14 +1751,33 @@ def main():
console_summary = console_summary.replace("[triple-backtick]\n", "").replace("[triple-backtick]", "")
# THE PAYLOAD LEDGER: biggest-first, so "what do I cut to fit the
- # attachment limit" is always answered by row one.
- if processed_files_data:
+ # attachment limit" is always answered by row one. Auto-context
+ # sections (full articles, narrative lists, tree, UML, shards, git
+ # diff telemetry) are folded in as AUTO rows so the table gauges the
+ # whole payload, not just the Codebase section.
+ ledger_rows = []
+ for f in processed_files_data:
+ display_path = f['path']
+ if display_path.startswith(REPO_ROOT):
+ display_path = os.path.relpath(display_path, REPO_ROOT)
+ ledger_rows.append({
+ 'label': display_path,
+ 'tokens': f['tokens'],
+ 'bytes': len(f['content'].encode('utf-8'))
+ })
+ for ac_title, ac_data in builder.auto_context.items():
+ ledger_rows.append({
+ 'label': f"AUTO: {ac_title}",
+ 'tokens': ac_data.get('tokens', 0),
+ 'bytes': len(ac_data.get('content', '').encode('utf-8'))
+ })
+ if ledger_rows:
try:
from rich.console import Console
from rich.table import Table
- total_bytes_f = sum(len(f['content'].encode('utf-8')) for f in processed_files_data)
- total_tokens_f = sum(f['tokens'] for f in processed_files_data)
+ total_bytes_f = sum(r['bytes'] for r in ledger_rows)
+ total_tokens_f = sum(r['tokens'] for r in ledger_rows)
ledger = Table(title="📦 Payload Ledger (biggest first)", show_footer=True)
ledger.add_column("File / Source", footer="TOTAL", overflow="fold")
@@ -1766,18 +1785,9 @@ def main():
ledger.add_column("Bytes", justify="right", footer=f"{total_bytes_f:,}")
ledger.add_column("% Bytes", justify="right", footer="100.0%")
- ranked = sorted(
- processed_files_data,
- key=lambda f: len(f['content'].encode('utf-8')),
- reverse=True
- )
- for f in ranked:
- b = len(f['content'].encode('utf-8'))
- pct = (b / total_bytes_f * 100) if total_bytes_f else 0.0
- display_path = f['path']
- if display_path.startswith(REPO_ROOT):
- display_path = os.path.relpath(display_path, REPO_ROOT)
- ledger.add_row(display_path, f"{f['tokens']:,}", f"{b:,}", f"{pct:.1f}%")
+ for r in sorted(ledger_rows, key=lambda r: r['bytes'], reverse=True):
+ pct = (r['bytes'] / total_bytes_f * 100) if total_bytes_f else 0.0
+ ledger.add_row(r['label'], f"{r['tokens']:,}", f"{r['bytes']:,}", f"{pct:.1f}%")
Console().print(ledger)
except ImportError:
(nix) pipulate $ m
📝 Committing: `chore: Improve payload ledger display with auto-context integration`
[main f396b131] `chore: Improve payload ledger display with auto-context integration`
1 file changed, 26 insertions(+), 16 deletions(-)
(nix) pipulate $ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 48 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 847 bytes | 847.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:pipulate/pipulate.git
d867ce98..f396b131 main -> main
(nix) pipulate $
Still not vibe-coding. Let’s see what this did. And if did what I think, then back to the PII challenge please. You’ll see what I want to do in the article. Explain it to the audience and then explain the fix to me.
(nix) pipulate $ latest 1
[Bunny was here]
🗺️ Codex Mapping Coverage: 76.9% (166/216 tracked files).
📦 Appending 50 uncategorized files to the Paintbox ledger for future documentation...
⚠️ TOPOLOGICAL INTEGRITY ALERT (Broken References):
• index.md
• text/markdown
--- Processing Files ---
Warning: FILE NOT FOUND AND WILL BE SKIPPED: /) _________ <--------------------------- !!!
-> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.1399s]
Skipping codebase tree (--no-tree flag detected).
Adding full article content...
/home/mike/repos/trimnoir/_posts/2026-07-10-git-repository-scrub-methodology.md # [Idx: 1 | Order: 1 | Tokens: 14,704 | Bytes: 55,009]
Total: 1 full articles | 14,732 tokens | 55,101 bytes
🔍 Running Static Analysis Telemetry...
-> Checking for errors and dead code (Ruff)...
✅ Static Analysis Complete.
📦 Payload Ledger (biggest first)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ File / Source ┃ Tokens ┃ Bytes ┃ % Bytes ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ COMMAND: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs │ 41,559 │ 123,283 │ 15.7% │
│ prompt_foo.py │ 19,823 │ 90,492 │ 11.5% │
│ foo_files.py │ 15,145 │ 60,112 │ 7.6% │
│ AUTO: Full Article Content │ 14,732 │ 55,101 │ 7.0% │
│ release.py │ 12,142 │ 54,473 │ 6.9% │
│ flake.nix │ 12,478 │ 52,901 │ 6.7% │
│ remotes/honeybot/scripts/stream.py │ 7,221 │ 32,489 │ 4.1% │
│ scripts/articles/googledocizer.py │ 5,964 │ 25,468 │ 3.2% │
│ scripts/articles/confluenceizer.py │ 5,959 │ 25,340 │ 3.2% │
│ /home/mike/repos/trimnoir/flake.nix │ 5,301 │ 23,865 │ 3.0% │
│ remotes/honeybot/nixos/configuration.nix │ 5,119 │ 19,604 │ 2.5% │
│ scripts/articles/build_knowledge_graph.py │ 4,348 │ 17,370 │ 2.2% │
│ scripts/articles/articleizer.py │ 3,354 │ 15,274 │ 1.9% │
│ remotes/honeybot/scripts/logs.py │ 3,294 │ 14,835 │ 1.9% │
│ remotes/honeybot/scripts/db.py │ 2,850 │ 13,221 │ 1.7% │
│ remotes/honeybot/scripts/content_loader.py │ 3,189 │ 13,028 │ 1.7% │
│ scripts/articles/generate_ai_context.py │ 2,966 │ 12,503 │ 1.6% │
│ scripts/articles/gsc_historical_fetch.py │ 2,863 │ 12,194 │ 1.6% │
│ scripts/articles/contextualizer.py │ 2,546 │ 11,280 │ 1.4% │
│ apply.py │ 2,512 │ 11,038 │ 1.4% │
│ scripts/articles/sanitizer.py │ 2,610 │ 10,584 │ 1.3% │
│ scripts/articles/generate_redirects.py │ 2,233 │ 9,255 │ 1.2% │
│ remotes/honeybot/scripts/test_forest.py │ 2,295 │ 9,109 │ 1.2% │
│ scripts/xp.py │ 2,002 │ 8,437 │ 1.1% │
│ scripts/articles/common.py │ 1,996 │ 8,128 │ 1.0% │
│ scripts/articles/editing_prompt.txt │ 1,533 │ 6,906 │ 0.9% │
│ scripts/articles/generate_hubs.py │ 1,456 │ 5,970 │ 0.8% │
│ /home/mike/repos/trimnoir/index.md │ 1,267 │ 5,788 │ 0.7% │
│ remotes/honeybot/scripts/forest.py │ 1,340 │ 5,677 │ 0.7% │
│ scripts/articles/publishizer.py │ 1,078 │ 4,407 │ 0.6% │
│ remotes/honeybot/scripts/score.py │ 901 │ 3,969 │ 0.5% │
│ /home/mike/repos/trimnoir/_layouts/default.html │ 915 │ 3,551 │ 0.5% │
│ scripts/articles/generate_llms_txt.py │ 874 │ 3,492 │ 0.4% │
│ remotes/honeybot/hooks/post-receive │ 940 │ 3,080 │ 0.4% │
│ scripts/articles/sanitize_redirects.py │ 684 │ 2,935 │ 0.4% │
│ remotes/honeybot/scripts/card.py │ 570 │ 2,340 │ 0.3% │
│ /home/mike/repos/trimnoir/_config.yml │ 566 │ 2,208 │ 0.3% │
│ .gitignore │ 597 │ 2,197 │ 0.3% │
│ /home/mike/.config/pipulate/pii_substitutions.txt │ 540 │ 1,898 │ 0.2% │
│ /home/mike/.config/pipulate/blogs.json │ 307 │ 1,043 │ 0.1% │
│ nixops.sh │ 270 │ 926 │ 0.1% │
│ .gitattributes │ 33 │ 76 │ 0.0% │
├──────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┤
│ TOTAL │ 198,372 │ 785,847 │ 100.0% │
└──────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┘
**Command:** `prompt_foo.py -a [-1:] --no-tree`
--- Auto-Context Metadata ---
• Full Article Content (14,732 tokens | 55,101 bytes)
--- Prompt Summary ---
Summed Tokens: 211,358 (from section parts)
Verified Tokens: 215,423 (from final output)
(Difference: +4,065)
Total Words: 75,484 (content only)
Total Chars: 851,801
Total Bytes: 859,965 (UTF-8)
--- Size Perspective ---
📚 Equivalent in length to a **Average Paperback Novel or a Ph.D. Dissertation** (Note: With a token/word ratio of 2.85, this content is far denser and more complex than typical prose of this length).
Markdown output copied to clipboard
(nix) pipulate $
Wow, neat! It’s a bit over what I think Anthropic will take as an attachment, but let’s give it a try.
Fable?
🎤
Fable 5 High: Context verified.
The ledger verdict
The AUTO fold-in did exactly what it should: AUTO: Full Article Content now sits at row four with 55KB / 7.0%, and the table’s TOTAL (785,847 bytes) finally gauges the real payload instead of just the Codebase section. The remaining ~74KB gap between the table’s TOTAL and the final 859,965-byte artifact is wrapper overhead — manifest, checklist, summary, recapture, the prompt itself — which is roughly constant per compile. A future chisel-strike could add a synthetic WRAPPER: everything else row so the footer equals the true output byte-for-byte, but the question the table exists to answer (“what do I cut?”) is now answered at a glance. Not urgent.
For the audience: what the PII challenge actually is
Yesterday’s article documents a full surgical incident: a client’s domain name leaked into the public Pipulate repo — once as scratch recon pasted into foo_files.py’s ADHOC slot, and once baked into example data in gap_analyzer_sauce.py, where it quietly rode along in every commit that ever contained that file. The cleanup required git-filter-repo rewriting 7,783 commits, a force-push, and the discovery that GitHub still serves the old commit objects by SHA afterward — the “ghost links” that make “still resolves” and “still canonical” two different truths.
Here’s the structural insight: Mike’s publishing system already has a sanitization lane for prose. sanitizer.py scrubs article.txt on its way out the door, applying the role-substitution table in ~/.config/pipulate/pii_substitutions.txt ([Redacted] → Account Director, client Slack URLs → [internal reference], and so on). confluenceizer.py and googledocizer.py apply the same table when projecting to corporate targets. Prose is guarded.
But code has no such gate. The article lane transforms content at a chokepoint (write_post → sanitizer → articleizer). The repo lane has a chokepoint too — the commit — but nothing standing at it. That’s the asymmetry that let a client name recur (“again,” in Mike’s own word). And this is the anti-Crichton law the codebase preaches to itself: the failure mode and its repair mechanism must ship together. The repair (filter-repo methodology, now documented and tool-installed in flake.nix) shipped yesterday. The detector ships now.
One design decision matters and it’s counterintuitive: do not reuse pii_substitutions.txt as the denylist. That file is a rewrite table — its patterns are tuned for substitution and include things like bare timestamps and generic role words that would false-positive constantly as commit blockers. Transform-on-the-way-out and refuse-on-the-way-in are different functions deserving different files. Same directory convention, though — and critically, outside the repo, because a denylist of client names is itself the most radioactive file in the system. It must never be trackable.
The Asymmetry of Repository Safety
Second decision: scan the whole staged index, not just added lines. The gap_analyzer_sauce.py lesson is that pollution’s worst form is latent — already sitting in a file while unrelated edits commit around it for months. A whole-index scan means one hit anywhere blocks the very next commit, surfacing the problem while it’s a 30-second working-tree fix instead of a 7,783-commit rewrite. And case-insensitive matching closes the “capitalization variants” residual the scrub post-mortem flagged as unproven.
The fix
Two pieces: a versioned hook script, and a three-line idempotent installer in flake.nix’s miscSetupLogic — the exact same pattern already used there for the nbstripout filter, so every nix develop re-arms it. No denylist file means silent no-op, so magic-cookie end-users are untouched.
Cheapest falsifying probe: apply.py already runs nix-instantiate --parse on the flake edit; the hook itself gets a live canary test below.
Target: /home/mike/repos/pipulate/scripts/git_hooks/pre-commit
[[[WRITE_FILE]]]
#!/usr/bin/env bash
# THE COMMIT AIRLOCK — staged-content client-identity denylist guard.
#
# Installed into .git/hooks/pre-commit by flake.nix on every shell entry
# (same idempotent pattern as the nbstripout filter config). Patterns live
# OUTSIDE the repo so the denylist itself can never leak into history:
#
# ~/.config/pipulate/commit_denylist.txt
#
# Format: one extended-regex pattern per line; blank lines and '#' comments
# ignored. Matching is case-insensitive, so a single 'clientname' line
# catches ClientName, CLIENTNAME, clientname.com — closing the
# capitalization-variant gap from the git-filter-repo post-mortem.
#
# Scope: scans the ENTIRE staged index (git grep --cached), not just added
# lines. One hit anywhere blocks the commit, so latent pollution surfaces
# on the very next commit instead of riding along for months and forcing
# another 7,783-commit history rewrite.
#
# No denylist file, or an empty one, means this hook is a silent no-op —
# fresh magic-cookie installs are completely unaffected.
#
# NOTE: this is deliberately a DIFFERENT file from pii_substitutions.txt.
# That table is a rewrite ruleset for the article/wiki lanes (transform on
# the way out); this is a refusal ruleset for the repo lane (block on the
# way in). Its substitution patterns would false-positive as blockers.
## Implementing the Commit Airlock
DENYLIST="$HOME/.config/pipulate/commit_denylist.txt"
[ -f "$DENYLIST" ] || exit 0
PATTERNS_FILE="$(mktemp)"
trap 'rm -f "$PATTERNS_FILE"' EXIT
grep -v '^[[:space:]]*#' "$DENYLIST" | grep -v '^[[:space:]]*$' > "$PATTERNS_FILE" || true
[ -s "$PATTERNS_FILE" ] || exit 0
HITS="$(git grep --cached -I -n -i -E -f "$PATTERNS_FILE" 2>/dev/null || true)"
if [ -n "$HITS" ]; then
echo "🛑 COMMIT BLOCKED: denylisted identifier found in staged content."
echo "$HITS" | head -20
HIT_COUNT="$(printf '%s\n' "$HITS" | wc -l)"
if [ "$HIT_COUNT" -gt 20 ]; then
echo " ... and $((HIT_COUNT - 20)) more hit(s)."
fi
echo ""
echo " Patterns: $DENYLIST"
echo " Scrub the string(s) above before committing. To bypass ONCE,"
echo " deliberately and with eyes open: git commit --no-verify"
exit 1
fi
## Validation via Canary Testing
exit 0
[[[END_WRITE_FILE]]]
Target: /home/mike/repos/pipulate/flake.nix
[[[SEARCH]]]
# Set up nbstripout git filter
if [ ! -f .gitattributes ]; then
echo "*.ipynb filter=nbstripout" > .gitattributes
fi
git config --local filter.nbstripout.clean "nbstripout"
[[[DIVIDER]]]
# Set up nbstripout git filter
if [ ! -f .gitattributes ]; then
echo "*.ipynb filter=nbstripout" > .gitattributes
fi
git config --local filter.nbstripout.clean "nbstripout"
# THE COMMIT AIRLOCK: install the client-identity denylist guard.
# Hook logic is versioned in scripts/git_hooks/pre-commit; patterns
# live OUTSIDE the repo (~/.config/pipulate/commit_denylist.txt) so
# the denylist itself can never leak. No denylist file = no-op.
if [ -d .git ] && [ -f "$PIPULATE_ROOT/scripts/git_hooks/pre-commit" ]; then
mkdir -p .git/hooks
cp "$PIPULATE_ROOT/scripts/git_hooks/pre-commit" .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
fi
[[[REPLACE]]]
Arming and proving it
The flake installs the hook on next shell entry; to arm and test it right now without re-entering:
mkdir -p .git/hooks && cp scripts/git_hooks/pre-commit .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit
echo 'zzz-canary-zzz' >> ~/.config/pipulate/commit_denylist.txt
echo 'zzz-canary-zzz' > canary.txt && git add canary.txt
git commit -m "canary test" # expect 🛑 COMMIT BLOCKED
git reset HEAD canary.txt && rm canary.txt
sed -i '/zzz-canary-zzz/d' ~/.config/pipulate/commit_denylist.txt
Then seed the real denylist by hand — client domains, client names, anything from the pii_substitutions.txt left-hand side that’s a hard identifier rather than a substitution nuance. That curation stays human, private, and off-repo, which is exactly where it belongs.
Note what this composes into: your m alias runs git commit -am, which stages and then fires the hook — so the airlock sits inside your existing muscle memory with zero new keystrokes. The Nedry console still doesn’t exist; now it also can’t be smuggled in one staged file at a time.
MikeLev.in: Okay, first our patches:
$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
(nix) pipulate $ patch
(nix) pipulate $ app
✅ WHOLE-FILE WRITE: CREATED '/home/mike/repos/pipulate/scripts/git_hooks/pre-commit'.
(nix) pipulate $ d
(nix) pipulate $ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
scripts/git_hooks/
nothing added to commit but untracked files present (use "git add" to track)
(nix) pipulate $ git add /home/mike/repos/pipulate/scripts/git_hooks/pre-commit
(nix) pipulate $ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: scripts/git_hooks/pre-commit
(nix) pipulate $ m
📝 Committing: refactor: Enhance pre-commit hook for denylist checks
[main 511125c2] refactor: Enhance pre-commit hook for denylist checks
1 file changed, 54 insertions(+)
create mode 100644 scripts/git_hooks/pre-commit
(nix) pipulate $ patch
(nix) pipulate $ app
✅ DETERMINISTIC PATCH APPLIED: Successfully mutated '/home/mike/repos/pipulate/flake.nix'.
(nix) pipulate $ d
diff --git a/flake.nix b/flake.nix
index 6cc339e1..437e60e4 100644
--- a/flake.nix
+++ b/flake.nix
@@ -530,6 +530,15 @@ runScript = pkgs.writeShellScriptBin "run-script" ''
echo "*.ipynb filter=nbstripout" > .gitattributes
fi
git config --local filter.nbstripout.clean "nbstripout"
+ # THE COMMIT AIRLOCK: install the client-identity denylist guard.
+ # Hook logic is versioned in scripts/git_hooks/pre-commit; patterns
+ # live OUTSIDE the repo (~/.config/pipulate/commit_denylist.txt) so
+ # the denylist itself can never leak. No denylist file = no-op.
+ if [ -d .git ] && [ -f "$PIPULATE_ROOT/scripts/git_hooks/pre-commit" ]; then
+ mkdir -p .git/hooks
+ cp "$PIPULATE_ROOT/scripts/git_hooks/pre-commit" .git/hooks/pre-commit
+ chmod +x .git/hooks/pre-commit
+ fi
# Set EFFECTIVE_OS for browser automation scripts
if [[ "$(uname -s)" == "Darwin" ]]; then export EFFECTIVE_OS="darwin"; else export EFFECTIVE_OS="linux"; fi
# ── THE WORKSHOP HOOK (team-sync substrate) ──────────────────
(nix) pipulate $ m
📝 Committing: chore: Install pre-commit hook for commit denylist guard
[main 139be189] chore: Install pre-commit hook for commit denylist guard
1 file changed, 9 insertions(+)
(nix) pipulate $ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 48 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 2.15 KiB | 2.15 MiB/s, done.
Total 8 (delta 4), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
To github.com:pipulate/pipulate.git
f396b131..139be189 main -> main
(nix) pipulate $
Now our flake rebuild:
(sys) pipulate $ ndq
warning: updating lock file '/home/mike/repos/pipulate/flake.lock':
• Added input 'flake-utils':
'github:numtide/flake-utils/11707dc2f618dd54ca8739b309ec4fc024de578b?narHash=sha256-l0KFg5HjrsfsO/JpG%2Br7fRrqm12kzFHyUHqHCVpMMbI%3D' (2024-11-13)
• Added input 'flake-utils/systems':
'github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e?narHash=sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768%3D' (2023-04-09)
• Added input 'nixpkgs':
'github:NixOS/nixpkgs/0bb7ec54c8483066ec9d7720e780a5caa71f8612?narHash=sha256-iffAls3iaNTyJC2faYcUXSI%2BGp02cDjYl%2BMygxKl2GI%3D' (2026-07-08)
(nix) pipulate $
Ugh, that remains me. I have to document not pinning the flake so it’s actually roll forward (and not actually pinned like I say — must be explicit if I’m using this anti-Crichton story because that’s exactly such a seam!)
Okay and we probe and prove:
$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
(nix) pipulate $ mkdir -p .git/hooks && cp scripts/git_hooks/pre-commit .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit
echo 'zzz-canary-zzz' >> ~/.config/pipulate/commit_denylist.txt
echo 'zzz-canary-zzz' > canary.txt && git add canary.txt
git commit -m "canary test" # expect 🛑 COMMIT BLOCKED
git reset HEAD canary.txt && rm canary.txt
sed -i '/zzz-canary-zzz/d' ~/.config/pipulate/commit_denylist.txt
🛑 COMMIT BLOCKED: denylisted identifier found in staged content.
canary.txt:1:zzz-canary-zzz
Patterns: /home/mike/.config/pipulate/commit_denylist.txt
Scrub the string(s) above before committing. To bypass ONCE,
deliberately and with eyes open: git commit --no-verify
(nix) pipulate $
This is good. Wrap up the article. Give me the little instruction thingie I need now on how to use this, what it means, whether it’s active on all Jekyll blog outputs or just he public channel, that sort of thing.
Fable 5 High: Context verified.
Wrapping the article: what actually shipped today
Three chisel-strikes, each one turn, each landed through the same airlock they exist to celebrate:
- The Payload Ledger.
prompt_foo.pynow renders a Rich table, sorted biggest-first, so “what do I cut to fit the 840K attachment ceiling” is answered by row one instead of by squinting at a wall of# path # [tokens]lines. It immediately earned its keep: you spotted the Botify Bootcamp file (174KB, 18.8%) riding along uninvited. - The AUTO fold-in. The ledger now includes auto-context rows (full articles, git diff telemetry, narrative lists), so the TOTAL gauges the actual payload, not just the Codebase section.
AUTO: Full Article Contentat 7.0% is exactly the kind of chunk that was previously invisible. - The Commit Airlock. A pre-commit hook, versioned in
scripts/git_hooks/pre-commit, installed idempotently byflake.nixon every shell entry, reading a denylist that lives outside the repo. The canary test blocked on the first try. The failure mode (yesterday’s 7,783-commitgit-filter-reporewrite) and its detector now ship in the same news cycle — the anti-Crichton law, obeyed.
And the meta-point held throughout: three mutations, three human button-presses, three color-coded diffs reviewed, zero ouroboros. The sandworm proposed; the rider decided.
How to use the Commit Airlock
The denylist file: ~/.config/pipulate/commit_denylist.txt
- One extended-regex pattern per line. Blank lines and
#comments are ignored. - Matching is case-insensitive, so a single lowercase
clientnamecatchesClientName,CLIENTNAME, andclientname.com— the capitalization-variant gap from the scrub post-mortem is closed. - Seed it with hard identifiers: client domains, client names, the left-hand-side patterns from
pii_substitutions.txtthat are identifiers rather than substitution nuances. Do not copy the whole substitution file over — timestamps and generic role words would false-positive constantly. - If the file doesn’t exist or is empty, the hook is a silent no-op. Fresh magic-cookie installs are completely unaffected.
What it scans: the entire staged index (git grep --cached), not just added lines. Latent pollution already sitting in a file — the gap_analyzer_sauce.py failure mode — surfaces on the very next commit touching anything, while it’s still a 30-second fix.
When it fires: any commit in a repo where the hook is installed, including your m alias (git commit -am stages first, then the hook fires — zero new keystrokes). Deliberate one-time bypass: git commit --no-verify.
Re-arming: flake.nix copies the hook into .git/hooks/pre-commit on every nix develop of the pipulate repo. If GitHub’s magic-cookie transformation ever recreates .git, the next shell entry re-installs it.
The coverage question — and the honest answer
Right now the airlock guards the pipulate repo only. This is a git-hook, so it protects wherever it’s installed, and only the pipulate flake installs it. Your other repos each have their own .git and are currently unguarded at the commit chokepoint:
- trimnoir (Honeybot/public) — no hook. Mitigated because
sanitizer.pyruns the full public-lane scrub (private fences, inline spans, PII table) before prose ever lands in_posts/. But hand-edits to_posts/, includes, or config bypass that lane. Thepublishalias commits this repo blind. - grimoire (private lane) — no hook, and the private lane deliberately preserves names. Lowest risk since it never leaves LAN.
- botifyml — no hook. Its Confluence projection applies PII substitution at output, but the git repo itself accepts anything.
gobotcommits it blind. - Pipulate.com — no hook, and
release.pycommits into it automatically.
The distinction to keep in your head: the sanitizer lanes are transform-on-the-way-out; the airlock is refuse-on-the-way-in. Prose has both defenses on the public channel. Code has the airlock only where the flake installed it — which happens to be exactly where both historical leaks occurred, so the highest-value target is covered.
Operationalizing Across the Fleet
The natural next chisel-strike, when you want it: add the same three-line installer block to trimnoir/flake.nix’s commonHook (and optionally the others). The hook script is repo-agnostic — it reads $HOME/.config/pipulate/commit_denylist.txt regardless of which repo it’s installed in — so it’s a pure copy of the pattern, one denylist guarding every gate. Since publish and release.py both commit to those repos without a human reading the diff, they’re the two places where an automated refusal gate matters most: they’re the closest thing your system has to an ouroboros, and this puts a human-authored invariant in front of each one.
One last loose thread you flagged yourself: the flake.lock regeneration during ndq reminded you that the environment rolls forward on nixos-unstable rather than being truly pinned. That’s a seam worth an explicit paragraph in the book — a system that preaches reproducibility should be honest about which inputs float. Todo line earned.
MikeLev.in: Todo-line added (which in great part is doing the to-do item itself, because now it technically is documented).
(nix) pipulate $ d
diff --git a/foo_files.py b/foo_files.py
index 2025de19..c2f0339c 100644
--- a/foo_files.py
+++ b/foo_files.py
@@ -882,6 +882,7 @@ scripts/xp.py # [1,981 tokens | 8,377 bytes]
"""
# #todo #to-do
+# - Document that the flake.lock file remains unpinned for roll-forward on Nix unstable branch.
# - Do the work of purging PicoCSS "Card" references so I can unpin FastHTML in requirements.in
# - Add "What's dangling that should be carried forward" language in the `prompt_foo.py` instructions
# - Swap in uv for pip for a faster install
(nix) pipulate
Book Analysis
Ai Editorial Take
What strikes me most is the ‘metadata radioactivity’ concern. By insisting the denylist stay outside the repo, you’ve solved the ‘who watches the watchmen’ problem. The most surprising insight here is that you’ve effectively turned your git workflow into a compiler that rejects ‘illegal syntax’ (PII) before it can be linked into your history.
🐦 X.com Promo Tweet
Prevent PII leaks before they hit your git history. I just implemented a versioned, idempotent pre-commit airlock that uses a private denylist to stop credential drift in its tracks. See the logic here: https://mikelev.in/futureproof/closing-the-commit-airlock/ #GitSecurity #Nix #DevSecOps
Title Brainstorm
- Title Option: Closing the Commit Airlock: Engineering Against Codebase Seams
- Filename:
closing-the-commit-airlock.md - Rationale: Direct, technical, and emphasizes the ‘airlock’ metaphor established in the text.
- Filename:
- Title Option: Beyond Filter-Repo: Preventive Invariants for Repository Hygiene
- Filename:
preventive-invariants-repository-hygiene.md - Rationale: Positions the article as the logical successor to a traumatic git-scrubbing event.
- Filename:
- Title Option: The Anti-Crichton Law: Shipping the Detector with the Code
- Filename:
anti-crichton-law-shipped-detection.md - Rationale: Focuses on the philosophical core of the methodology—coupling the repair to the breach.
- Filename:
Content Potential And Polish
- Core Strengths:
- Clear distinction between prose sanitization (outbound) and code airlocking (inbound).
- Practical integration of Nix flakes for idempotent deployment.
- Compelling narrative shift from incident recovery to systemic prevention.
- Suggestions For Polish:
- The distinction between the different repo-types (trimnoir, grimoire, etc.) is vital; move this into a small table or list for clarity.
- Add a visual flow-chart snippet to the next edit showing the ‘Airlock’ placement relative to the commit cycle.
Next Step Prompts
- Draft a follow-up article detailing how to share the same commit_denylist.txt across different machines using a secure, non-git mechanism.
- Design a formal RFC for a ‘–budget’ flag in the
prompt_foo.pyutility that prevents commits from exceeding the 840K threshold.