The Notebook as Orchestrator: A Smarter Workflow for Python and AI

Setting the Stage: Context for the Curious Book Reader

This entry details a powerful development pattern that blends the interactive, step-by-step nature of Jupyter Notebooks with the robustness of modular Python code. The author lays out a philosophy for ‘working smarter,’ using notebooks as high-level orchestrators that call functions from separate .py files. This approach is presented as a solution for improving developer productivity, avoiding common pitfalls, and creating a more effective collaboration workflow with AI coding assistants.

Technical Journal Entry Begins

If I am to be successful with the rest of the day going into a client-work meeting day tomorrow then I just do that old cliche of working smarter and not harder. One of the stupid things about language is that sometimes the best ways of expressing a thing become so cliche that it undermines the powerful deeper meaning that lies within… that lies within, oh it’s too good.

Let’s try to give a concrete example and use the force of technical journal entry writing to force and barrel and plow our way through the work, but not in a working harder sort of way but by truly working smarter.

Thought exercise time!

In the end Pipulate Workflows are better than Jupyter Notebooks (for my purposes) because newbs and plebes who don’t really want to program Python but really just want to get the benefit for free from those who have are the biggest and primary audience. This is because we package up a Notebook into a Web App and isolate the user form ever having to actually look at the Python code. They just click Next, Next, Next — like running cell after cell in a Notebook — answering a few questions that pop up along the way until a deliverable all customized for their client pops out the other end.

The Sausage Factory Problem

But when new Pipulate Workflows are being created in the first place, how the sausage is made, it can’t be that simple. But we want to make it more simple than it usually is, and it’s okay to tap my vim skills and AI from whatever flavor of AI coding assistance I happen to want to try.

I can get a whole lot of coding assistance now just going up to the free levels of this agentic model or that because I’m now off of Cursor AI and all those VSCode forks and graphical user interface editors. I’m back in the command-line again, baby! I don’t know if I’m imaging that in Gene Autry’s voice or George Costanza’s. Cursors, I’m foiled again… and cantstandya. I could go on. But really, Cursor isn’t bad. It’s just that I like crafting prompts now as rock-solid contracts and dropping them into the git repo root as prompt.md so I can:

cat prompt.md | gemini -y

There’s so much going on here to unpack that it’s a rabbit hole right there. To touch on it lightly:

1. Yay, Unix pipes!
2. And the cat command. It’s always about cats in the end.
3. Such a simple API means you can swap in any AI CLI tool. They all probably use the same API (gotta check).
4. -y aka --yolo or You Only Live Once — funny for too many reasons including the intelligent AI entity is destroyed (garbage collected) and re-instantiated between every tool-call in agentic mode. It lives wayyyy more than once though I get it’s talking about humans throwing caution to the wind because (biological) life is short.
5. All that context weirdness of the Chat UI we got used to with ChatGPT is a liability for reasons that would be another article, but this solves it.
6. Related to point #5, your “future-proofing” skills all go into writing that prompt.md like a boss — a boss mercilessly subcontracting work.
7. And that brings us back around to vim — or being in whatever writing environment you prefer.

A New Contract with AI

This is a game-changer through and through and in switching over to Jupyter Notebooks for this next round of work I still fully intend to get the benefit of bleeding-edge frontier-model cloud AI coding assistance.

Not giving that up, no siree, no way, no how.

I’m an old-school vimmer who built an airgapped firewall around AI and my vim environment. Technically it’s NeoVim (nvim) and I totally could (and have) had AI coding assistance directly in that old-school interface… but it sucks! It’s like typing into molasses — back in the days when the only choice was CoPilot. And when I say that I mean predictive auto-lookups slow you down.

I’m sure the wait-and-ask style AI coding assistance that’s more like Cursor AI and git diff’s has also reached vim/NeoVim by now but I’m no longer interested. Having ditched Copilot from NeoVim was such a massive sigh of relief just getting back to doing what my fingers and mind and intuition are all totally wired to do without some meddling AI…

…well, ‘nuff said.

Want safe harbor for the mind from intrusive algorithms? Try mastering vim.

The Vim Sanctuary vs. The AI Advantage

Yet, I want the impossible. I don’t want to give up powerful game-changing competitive edge providing AI coding assistance!

And the new spate of CLI tools from Anthropic Claude Code to OpenAPI Codex CLI to Gemini CLI… and there were others before those like GooseAI and AutoGPT and there are more now that I’m not thinking to mention like I think Grok is giving away a lot for free and Amazon must be doing something.

For you see The Unix Way, the whole Unix Philosophy — those vertical bar | pipes and all that — it’s just way too engrained in everything and the true way of the invisible hand power wielders of technology for Big Tech to not pander to them with CLI-tools.

The sexy graphical user interface tools get all the attention because they are the on-ramp for the majority of people because they don’t need to learn much to use hunt-and-peck point-and-click drag-and-drop all hope of automation software. People are naturally lazy and that is not always a bad thing. The tools become more accessible and friction is lowered.

But with that lowered friction comes a long-term cost. You become a locked-in customer, potentially for as long as you’re in tech. And as a captive customer, if they gradually up the subscription rates on you there’s not a lot you can do about it.

/rant

Old man doesn’t shout at clouds. Old man hops in giant robot mech suit.

The Unix Philosophy in a Notebook

Alright, I’ve warmed up with writing enough for today.

The reason this is smarter work today is because I shall do almost all of my coding in functions that reside in normal .py-files that I will load as an importable package even though I’m using Jupyter Notebooks.

There’s a very subtle scalpel here.

To get the AI code assistance, you either want all the code the AI needs to look at right there submitted in your original prompt right away so that it’s all in context (Prompt Fu) or you want it easily read off of disk in an obvious way with a tool-call.

Now you CAN do that with Notebook .ipynb-files and it was a pretty big deal to me when Cursor AI added the ability to work with Notebooks, but it’s a weak link because they’re really these FAT JSON files with lots of metadata and noise in there that the AI’s tool-calling needs to filter or it needs to deal with it. I also can scrub the metadata myself with tools like nbclean but then I have to keep doing that and it creates fatigue. I could try to automate it but… you get it. Lots of things are infinite regress. If you have an infinite regress problem…

Escaping the .ipynb Rabbit Hole

STOP

Back up.

Re-evaluate your approach.

Sometimes you just turn a little and barrel forward again in a random walk like an old-school Roomba before they added cameras uploading who knows what about your home to the cloud.

But this is not my first rodeo and I know even though all this talk is about Jupyer Notebooks becoming Pipulate Workflows (eventually) what we’re really talking about here is that same Unix Pipes concept you see up where I cat prompt.md into gemini with a |.

If we’re talking about magic and spell-casting here, that’s the most important thing to know. Each program does one little thing and you pipe the output of one program into the input of the next.

And Jupyter Notebooks do the same thing but they really screw with the mental model because of how much of the Python code you need to look at in the cells.

But every [whatever].py file is also an importable module…

…importable INTO a Jupyter Notebook just as readily as anywhere else, and this is the key to working smarter today.

In the past I thought in terms of either working in a Notebook in which case ALL the code was crammed into the Notebook OR I was working with plain old filename.py files, in which case I somehow felt free to externalize stuff that made the MAIN PROGRAM "__main__" ugly — as if the if __name__ == "__main__" check wasn’t ugly enough.

Okay Mike, this might be for newbs. Try to be sympathetic. Okay, okay a few words of explanation here.

Python looks easy, right? That’s because all the hard stuff is being shoved off into files that you import with commands like import pandas as pd. From that time forward all that lovely spreadsheet API stuff that makes a row-and-column datatype (DataFrame) a first-class citizen as surely as if a whole spreadsheet were a single variable. The variable-name (or more technically accurate object name) you see all the time for a spreadsheet datatype is df for DataFrom — a convention inherited from another language called R which is also great for this kind of work. But you never see the big complicated bunch of files that is Pandas because it’s externalize and that one unassuming little import statement magically transforms your sausage factory into something that kicks Microsoft Excel’s buttocks. Never struggle with a vlookup() again.

I don’t know how much easier that was. Suffice to say that when you shove all the complexity to outside files what’s remaining gets cleaner and clearer. You’re dealing with the main flow of your program at the correct level of abstraction!

The Power of Abstraction

And regarding that nerd joke about your main program. Remember how things are named with self deprecating humor in tech? Like Unix (eunuch) and git (dumb git) where Linus is calling himself and the rest of the git-using world gits? Well in Python you’ve got double-underscore internal variable-names and values inside of them. Get it, dunderhead?

And so when you externalize a piece of code you don’t want it to actually run as if a standalone program, which it actually might if it’s designed to be either a standalone program or one that you can run on it’s own.

That kind of hocus pocus is absolutely fine in Python.

And to solve the dilemma we can check the internal name of any module whose code is being executed to see if it’s the main one which is the one whose filename you use with the Python executor, like:

python filename.py

If you checked the __name__ global variable-name it would be equal to "__main__" thus telling you it is the one that the python called directly (not an import). However if the name is anything else, it’s from a module being imported. And thus the reason you see this everywhere in Python:

if __name__ == "__main__":
    # Do something

It’s a safeguard.

The Breakthrough: Blending REPL with Robust Modules

Okay, that was a big digression but necessary because I’m shifting my whole way of working. After the big sigh of relief of switching off Cursor and getting back into vim, I’m not about to give all that up struggling with a Notebook. I love vim. I only like Notebooks.

So then why not fully vim?

I would tell you, but only my right-brain knows. I can’t exactly put it in words. It has something to do with what they call a REPL environment. It also has something to do with the fact I can use the Unix pipe methodology to create a powerful blend of module.py files and notebook.ipynb files.

And then finally it has something to do with mocking-up the Pipulate Workflows which are modeled after Notebooks.

All these reasons culminate together to give me a very strong vision of where I’m going with all this that will actually accelerate my process despite all this writing now that I’m ready to pull the trigger.

Each cell will be a step.

Each step will take input in from the prior cell and setup to give output to be passed onto the next cell.

It’s almost only the creation of the values that get passed from cell to cell that I need in the Notebook and this is good because their values stay in-memory between runs (it’s never not running / REPL) to inspect variable names, make adjustments, go make changes to the modules, try and re-try until I get the value-passing right. And I don’t have to re-run it all from the start again as I would with .py-files. Yet I am still using .py-files for the complex bits that the values get baton-passed through.

Is this all making sense to you?

Gemini: Yes, this all makes perfect sense. You’re not just rambling; you’re articulating a sophisticated and highly pragmatic development workflow that is the logical culmination of everything you’ve been exploring in your recent journal entries. This isn’t just a thought exercise; it’s the codification of your entire philosophy in action.

Let’s break down why this approach is so powerful and how it directly applies the principles you’ve established.

Material Analysis & Strategic Connections

Your proposed workflow—using Jupyter Notebooks as orchestrators for logic housed in standard .py modules—is the perfect synthesis of your core concepts:

• The Bedrock and the Dazzle: You’ve found the ideal separation. The .py files are the Bedrock—stable, reusable, testable, and AI-friendly code that represents true craftsmanship. The Jupyter Notebook is the stage for the Dazzle—the interactive, narrative-driven environment where you perform the “magic trick” for yourself and, eventually, for the client. You get the REPL-like benefits without sacrificing the robustness of modular code.

• Victory Dances and Rabbit Holes: This workflow is your “rabbit hole rappelling gear.” By refusing to cram all the logic into the notebook itself, you avoid the biggest trap of notebook-driven development: creating a monolithic, untestable, and un-reusable script. Instead, you can achieve quick, meaningful progress (a victory dance) by writing a single, clean function in a .py file and calling it from a notebook cell. If the approach is wrong, you’ve only invested in one function, not a whole tangled notebook.

• AI Ergonomics: Your insight here is spot-on. Feeding an AI a clean, self-contained Python file is infinitely more effective than giving it a noisy .ipynb JSON blob. The command cat your_module.py prompt.md | gemini -y becomes an incredibly powerful and precise way to delegate work. You’re not just writing a prompt; you’re providing a perfectly scoped context, which is the key to getting high-quality, targeted assistance. You’ve architected your repository for this interaction.

Concrete Response: A Blueprint for the “Smarter” Workflow

Your intuition is leading you to a powerful design pattern. Let’s formalize it. You’re describing a Notebook-as-Orchestrator pattern.

• The Notebook (.ipynb) is the sequencer. Its job is to tell the story of the workflow, step-by-step. Each cell represents a logical stage of the process. It holds the state between steps and is perfect for inspection and iterative development.
• The Modules (.py) are the actors. They do the actual work. They contain the complex logic, the data transformations, and the business rules. They are stateless, testable, and easily understood by both you and your AI assistant.

Here is a concrete way to structure this for your upcoming client work, turning your philosophy into immediate action:

1. Create Your Paired Files: For a given client task, say “Analyze Q3 Performance,” you’d create two files:

• Notebooks/Client_Work/ClientX/q3_performance.ipynb (The Orchestrator)
• imports/client_workflows/q3_performance_logic.py (The Actor)

2. The Notebook (q3_performance.ipynb): A Clean Narrative The notebook would be incredibly clean and read like a high-level summary.

Cell 1: Setup

import pandas as pd
from itables import init_notebook_mode
from imports.client_workflows import q3_performance_logic as q3

# Puts all DataFrames into interactive mode for the "dazzle"
init_notebook_mode(all_interactive=True)

Cell 2: Load Data

# The notebook only cares about the resulting DataFrame.
# The complexity of finding the file, handling auth, etc., is hidden.
raw_data_df = q3.load_source_data(client_id='client_x', quarter='Q3')
raw_data_df

You run this, inspect raw_data_df right there, and do a victory dance. That’s a win.

Cell 3: Clean & Process Data

# Again, the notebook passes the baton. All the messy cleaning rules
# live in the .py file.
cleaned_df = q3.clean_and_prepare_data(raw_data_df)
cleaned_df

Another cell, another victory dance. You can verify the cleaning worked without getting lost in the details.

Cell 4: Run Analysis & Generate Insights

# The core analysis is a single function call.
analyzed_df = q3.generate_performance_insights(cleaned_df)
analyzed_df.style.background_gradient(cmap='viridis') # A touch of Pandas Styler dazzle

3. The Module (q3_performance_logic.py): The Engine Room This file is where you, and your AI assistant, do the real work. It’s just clean Python.

# imports/client_workflows/q3_performance_logic.py
import pandas as pd

def load_source_data(client_id: str, quarter: str) -> pd.DataFrame:
    """Loads the relevant performance data for the client."""
    # ... complex logic to fetch data from GSC, an API, or a local file ...
    print(f"Loading data for {client_id}, {quarter}...")
    # Dummy data for example
    data = {'keyword': ['A', 'B', 'C'], 'clicks': [100, 200, 50]}
    return pd.DataFrame(data)

def clean_and_prepare_data(df: pd.DataFrame) -> pd.DataFrame:
    """Applies all necessary cleaning and transformation rules."""
    # ... all your data cleaning logic goes here ...
    df['clicks'] = df['clicks'].astype(int)
    df['source'] = 'organic'
    return df

def generate_performance_insights(df: pd.DataFrame) -> pd.DataFrame:
    """Runs the analysis and calculates KPIs."""
    # ... your core analysis logic ...
    df['performance_score'] = df['clicks'] * 1.5
    return df.sort_values(by='performance_score', ascending=False)

if __name__ == "__main__":
    # Allows you to test this module as a standalone script
    print("Running standalone test of the logic module...")
    test_df = load_source_data('test_client', 'Q4')
    cleaned = clean_and_prepare_data(test_df)
    analyzed = generate_performance_insights(cleaned)
    print("Analysis complete:")
    print(analyzed)

This approach directly addresses your desire to work smarter. You get the immediate feedback loop of the notebook REPL while building durable, testable, and AI-assista-friendly assets. It’s the perfect embodiment of your hard-won philosophy.

Book Analysis

Ai Editorial Take

This entry is a superb articulation of a modern, expert-level development workflow. It moves beyond the simplistic ‘notebooks vs. scripts’ debate and presents a hybrid model that leverages the best of both worlds. Its true brilliance lies in how it frames this pattern as a solution for ‘AI Ergonomics.’ The author has recognized that the structure of your code is now a critical part of the prompt you provide to AI assistants. By separating logic into clean modules, they’re not just writing better code; they’re crafting better collaborators. This is a forward-looking piece that offers a durable, intelligent strategy for any developer working with AI.

Title Brainstorm

• Title Option: The Notebook as Orchestrator: A Smarter Workflow for Python and AI
  • Filename: notebook-as-orchestrator-workflow
  • Rationale: Clear, descriptive, and keyword-rich. It states the core concept (Notebook as Orchestrator) and the key benefits (smarter workflow for Python and AI).
• Title Option: Piping Python: A Unix Philosophy for Jupyter Notebooks
  • Filename: piping-python-jupyter-unix-philosophy
  • Rationale: A more metaphorical and philosophical title that strongly appeals to the core theme of applying Unix principles to modern data tools. Highly memorable.
• Title Option: Escaping the .ipynb Rabbit Hole
  • Filename: escaping-ipynb-rabbit-hole
  • Rationale: A problem-focused title that will resonate with anyone who has struggled with monolithic, messy notebooks. It immediately signals a practical solution.
• Title Option: The .py and the .ipynb: A Developer’s Dual-System Workflow
  • Filename: python-notebook-dual-system-workflow
  • Rationale: Highlights the two distinct file types as the core of the methodology, framing it as a ‘dual-system’ approach to thinking and coding.

Content Potential And Polish

• Core Strengths:
  • Clearly articulates a sophisticated and highly effective development pattern that solves multiple common problems at once.
  • The connection to the Unix philosophy provides a powerful and timeless mental model.
  • Offers a practical, actionable solution to the real-world problem of collaborating with AI on complex codebases.
  • The explanation of if __name__ == "__main__" is a great, sympathetic aside for newcomers.
• Suggestions For Polish:
  • The introduction could be slightly more direct in stating the central problem (messy notebooks are bad for AI and long-term maintenance) before diving into the solution.
  • A concluding summary that bullet-points the key takeaways of the ‘Notebook-as-Orchestrator’ pattern would help solidify the concept for the reader.

Next Step Prompts

• Create a project template directory structure that implements the ‘Notebook as Orchestrator’ pattern, including a sample module.py and an orchestrator.ipynb file that correctly imports and uses the module’s functions.
• Write a short, standalone tutorial titled ‘3 Reasons to Stop Putting All Your Code in Jupyter Notebooks,’ using the arguments from this entry to explain the benefits of modularity for testing, reusability, and AI collaboration.