Simplifying State: A Workflow Refactoring Journey

In this article, I describe my journey to refactor a multi-step workflow system, aiming for simpler state management and a better user experience. With the help of the Anthropic AI assistant in Cursor, I tackled redundant state tracking, optimized data handling for 'revert' actions, and made the system more extensible. We introduced a base class for workflows, refined how step completion and reversion are handled, and used HTMX strategically for dynamic UI updates. The result? A cleaner, more manageable, and user-friendly workflow system.

By Mike Levin

Wednesday, January 15, 2025

Introduction

This article explores how to simplify a complex, multi-step workflow system, like those found in online forms and applications, to make it more user-friendly and easier to manage. Through a collaborative effort between the author and an AI assistant, the article describes a journey of refactoring – a process of improving the underlying code – to create a cleaner, more organized system. Key improvements include streamlining how user progress is tracked, making it easier for users to go back and change previous entries (reverts), and employing a particularly elegant solution to handle these “go back” actions without overcomplicating the system. The result is a workflow that’s easier to use, understand, and maintain.

The Problem

Me: How can I get rid of the whole virtual state thing and have one source of truth, and chain reacting always goes up to the last thing that has data and it just repopulates by default, instead of going one past it and then struggling to get data back for a form refill?

The Solution

Claude: Ah, now I get what you’re saying! Instead of messing with virtual states, we could simplify this whole thing by always working backwards from the last completed step. Here’s how we could modify the finalize handler:

async def handle_finalize(self, steps: list, app_name: str) -> Card:
    """Handles finalize step display and manages pipeline state.
    
    Simplified to work backwards from last completed step - no virtual state needed.
    """
    pipeline_id = db.get("pipeline_id", "unknown")
    state = self.read_state(pipeline_id)
    
    # Find last completed step by working backwards
    last_completed = None
    for step in reversed(steps[:-1]):  # Exclude finalize step
        if step.field in state.get(step.id, {}):
            last_completed = step
            break
            
    logger.debug(f"Last completed step: {last_completed}")
    
    # If finalized, show locked view
    finalize_step = steps[-1]
    if finalize_step.field in state.get(finalize_step.id, {}):
        return Card(
            H3("All Cards Complete"),
            P("Pipeline is finalized. Use Unfinalize to make changes."),
            Form(
                Button("Unfinalize", type="submit", style="background-color: #f66;"),
                hx_post=f"/{app_name}/unfinalize",
                hx_target=f"#{app_name}-container",
                hx_swap="outerHTML"
            ),
            style="color: green;",
            id="finalize"
        )
    
    # If all steps complete, show finalize button
    if last_completed and last_completed.id == steps[-2].id:
        return self.render_finalize_button(app_name)
        
    # Otherwise show progress message
    return Card(
        P("Complete all steps before finalizing."),
        style="color: #666;",
        id="finalize"
    )

Key changes:

No more virtual state - just read directly from pipeline state
Work backwards to find last completed step
Chain reaction naturally flows from last completed step
Simpler logic for determining what to show

Would you like me to show the corresponding changes needed for the step handling code as well? That’s where we’d implement the “chain reaction” part.

Me: But how much refactoring would that be? But if not now, when? And if not at all, at what cost? If you don’t just bite the bullet today, you might be doomed to suffer with it forever?

Claude: You’re absolutely right. Technical debt is like a loan shark - the longer you wait, the more painful it gets. Let’s rip off this band-aid and do it right:

class PipelineManager:
    """Core pipeline state manager with single source of truth."""
    
    def read_state(self, pipeline_id: str) -> dict:
        """Read pipeline state with no virtual nonsense."""
        records = db.query(f"SELECT * FROM pipeline WHERE url = '{pipeline_id}'")
        if not records:
            return {}
        return json.loads(records[0].data)

    def write_step_data(self, pipeline_id: str, step_id: str, data: dict):
        """Write step data and cascade updates forward."""
        state = self.read_state(pipeline_id)
        
        # Find step index in full pipeline
        step_idx = next(i for i, step in enumerate(self.steps) if step.id == step_id)
        
        # Update this step
        state[step_id] = data
        
        # Clear all following steps to maintain consistency
        for step in self.steps[step_idx + 1:]:
            if step.id in state:
                del state[step.id]
        
        self.save_state(pipeline_id, state)

    async def handle_step(self, step: Step, app_name: str) -> Union[Form, Card]:
        """Handle step display based on pipeline position."""
        pipeline_id = db.get("pipeline_id", "unknown")
        state = self.read_state(pipeline_id)
        
        # Find last completed step
        last_completed = None
        for s in reversed(self.steps):
            if s.id in state and s.field in state[s.id]:
                last_completed = s
                break
        
        # If this step comes after last completed, show input
        if not last_completed or self.steps.index(step) == self.steps.index(last_completed) + 1:
            return self.render_step_input(step, app_name)
            
        # If this step is completed, show completion
        if step.id in state and step.field in state[step.id]:
            return self.render_step_completion(step, state[step.id][step.field])
            
        # Otherwise show placeholder
        return self.render_step_placeholder(step)

    async def handle_jump(self, target_step: str):
        """Jump to step and clear forward to maintain consistency."""
        pipeline_id = db.get("pipeline_id", "unknown")
        state = self.read_state(pipeline_id)
        
        # Find target index
        target_idx = next(i for i, step in enumerate(self.steps) 
                         if step.id == target_step)
        
        # Clear everything after target
        for step in self.steps[target_idx + 1:]:
            if step.id in state:
                del state[step.id]
                
        self.save_state(pipeline_id, state)

The key changes:

Eliminated all virtual state code
Single source of truth in pipeline state
Consistent forward clearing on any state change
Natural progression based on last completed step
Simplified jump logic that maintains consistency

Yes, it’s a bigger refactor. But it:

Removes the virtual state complexity
Makes the code more predictable
Eliminates state sync issues
Makes future changes easier

The short-term pain of refactoring is worth avoiding the long-term pain of maintaining two states. Sometimes you have to pay the technical debt to stop the interest from killing you.

Want me to show the rest of the changes needed to complete this refactor? There are probably a few more handlers that need updating to match this pattern.

Do we still use:

    self.STEPS = [
        Step(id='step_01', refill=False, field='name',      label='Name'),
        Step(id='step_02', refill=False, field='quest',     label='Quest'),
        Step(id='step_03', refill=False, field='color',     label='Color'),
        Step(id='step_04', refill=None,  field='finalized', label='Final')
    ]

…or is it:

    self.STEPS = [
        Step(id='step_01', refill=False, done='name',      show='Name'),
        Step(id='step_02', refill=False, done='quest',     show='Quest'),
        Step(id='step_03', refill=False, done='color',     show='Color'),
        Step(id='step_04', refill=None,  done='finalized', show='Final')
    ]

…because they’re all fields and labels, dumbarse, and done indicates the card’s done when this is acquired and show indicates we show the user?

Claude: Hah! Yes, field and label are terrible names that don’t convey intent. Your done and show are much clearer - they actually tell you what they’re for!

Let’s fix that Step class:

@dataclass
class Step:
    """A pipeline step with clear intent in property names."""
    id: str                # Unique step identifier 
    done: str             # Field that marks step as complete when acquired
    show: str             # Text shown to user in UI
    refill: bool = False  # Whether step can be refilled after completion

And update the steps definition:

self.STEPS = [
    Step(id='step_01', done='name',      show='Name'),
    Step(id='step_02', done='quest',     show='Quest'), 
    Step(id='step_03', done='color',     show='Color'),
    Step(id='step_04', done='finalized', show='Final', refill=None)
]

Now when we check if a step is done, it’s super clear:

if step.done in state[step.id]:  # Much clearer than step.field!

And when we show the UI:

Label(step.show)  # Obviously the display text vs step.label

Want me to update all the handler code to use these better property names? It’ll make the intent way clearer throughout.

Current Status

And that brings us up to now. I am getting ready to do an effective and cleansing refactoring, but with minimal breakage. Or more accurately, causing breakage that will be maximally easy to fix.

Implementation Plan

The perfecting of StarterFlow
The rapid re-implementation of TenCardFlow
The rapid re-implementation of LinkGraphFlow

…and time allowing, the rapid re-implementation of BridgeFlow.

Claude: Looking at your journal entry and the code discussion, I see you’re at a critical refactoring point. The key insights that emerge:

Pipeline State Simplification

@dataclass
class Step:
 """A pipeline step with clear intent."""
 id: str                # Unique identifier 
 done: str             # Completion marker field
 show: str             # User-facing text
 refill: bool = False  # Refill capability flag

Clear Forward Pattern

def write_step_data(self, pipeline_id: str, step_id: str, data: dict):
 """Write step data and clear forward for consistency."""
 state = self.read_state(pipeline_id)
 step_idx = next(i for i, step in enumerate(self.steps) 
                if step.id == step_id)
    
 state[step_id] = data
 # Clear forward to maintain consistency
 for step in self.steps[step_idx + 1:]:
     if step.id in state:
         del state[step.id]

The refactoring priority should be:

Get StarterFlow perfect as the reference implementation
Use it as template for TenCardFlow and LinkGraphFlow
Tackle BridgeFlow last as it’s likely most complex

Want me to help map out the StarterFlow refactor first?

Choosing the Right Data Structure

Why Namedtuples Over Dataclasses

Yes, but I hate dataclasses. I like namedtuples. We don’t want pedantic mypy data typing where it will only add noise and clutter, and the namedtuple labels provide precise clarity. Look again at:

self.STEPS = [
    Step(id='step_01', done='name',      show='Name'),
    Step(id='step_02', done='quest',     show='Quest'), 
    Step(id='step_03', done='color',     show='Color'),
    Step(id='step_04', done='finalized', show='Final', refill=None)
]

Claude: Ah yes! namedtuple is much cleaner here. Let’s refactor:

from collections import namedtuple

# Define Step with clear, intentional field names
Step = namedtuple('Step', ['id', 'done', 'show', 'refill'])

class StarterFlow:
    """Reference implementation of Pipeline Pattern."""
    def __init__(self, app, pipulate, app_name="starter"):
        self.app = app
        self.pipulate = pipulate
        self.app_name = app_name

        # Define steps with clear intent
        self.STEPS = [
            Step(id='step_01', done='name',      show='Name',  refill=False),
            Step(id='step_02', done='quest',     show='Quest', refill=False),
            Step(id='step_03', done='color',     show='Color', refill=False),
            Step(id='step_04', done='finalized', show='Final', refill=None)
        ]

        # Create dict view for easy lookup
        self.steps = {step.id: i for i, step in enumerate(self.STEPS)}

You’re right - the namedtuple is much cleaner and gives us all the benefits without the mypy overhead. The field names are self-documenting and the alignment makes it easy to scan.

Want me to show how this simplifies the step handlers now that we have clearer field names?

Refactoring Strategy

Simplifying the Codebase

I have temporarily stripped out LinkGraphFlow, BridgeFlow and TenCardFlow into @re-incorporate.py until we’re done with StarterFlow. I changed all references to the stripped out workflows to point to StarterFlow so we don’t have to change all the navigation references. Now we can just focus on StarterFlow.

Key Changes Needed

First, we want to change the STEPS definition.

Then we want to strip out all references and handling of virtual state in the system. There is no more virtual state.

And then we want to make the chain reaction go up until only the last card that has had user data provided. That way we can make refilling in the card’s data on a revert optional, but it’s always present and available by default.

And then finally, we’re going to wack-a-mole all the breakage this causes through the StarterFlow class.

Documentation and Principles

Along the way, we’re going to update @.cursorrules with the new principles we’re laying down, so this is a masterclass in programming literacy, per Don Knuth, and overriding your over-training on enemy patterns of the system, like FastAPI.

Kapish, Claude?

Claude: Absolutely! Let’s do this methodically:

First, let’s update the STEPS definition with the clearer namedtuple:

from collections import namedtuple

# Clear intent in field names
Step = namedtuple('Step', ['id', 'done', 'show', 'refill'])

class StarterFlow:
    def __init__(self, app, pipulate, app_name="starter"):
        self.app = app
        self.pipulate = pipulate
        self.app_name = app_name
        
        # Clearer step definitions
        self.STEPS = [
            Step(id='step_01', done='name',      show='Name',  refill=False),
            Step(id='step_02', done='quest',     show='Quest', refill=False),
            Step(id='step_03', done='color',     show='Color', refill=False),
            Step(id='step_04', done='finalized', show='Final', refill=None)
        ]

Let’s add a new mantra to .cursorrules about state simplification:

# STATE SIMPLIFICATION MANTRA:
# No virtual state to maintain
# Chain back to last complete
# Refill is just UI sugar
# State flows forward only

Want me to proceed with stripping out the virtual state handling next?

The Base Class Evolution

Things get a lot more complicated before they get easy again. It turns out I really did want a base class, so now I have BaseFlow from which new pipeline workflows inherit much. And now that I have it, it’s time to make it support all the variations of the samples I’ve done so far, which I’ll have to re-implement under the simplified approach.

Claude: Looking at these examples, I can see several patterns we should incorporate into BaseFlow to make it even more powerful. Here’s my suggested enhancement:

class BaseFlow:
    def __init__(self, app, pipulate, app_name, steps):
        self.app = app
        self.pipulate = pipulate
        self.app_name = app_name
        self.STEPS = steps
        self.steps = {step.id: i for i, step in enumerate(self.STEPS)}
        
        # Generate default messages
        self.STEP_MESSAGES = {
            "new": "Enter an ID to begin.",
            "finalize": {
                "ready": "All steps complete. Ready to finalize workflow.",
                "complete": "Workflow finalized. Use Unfinalize to make changes."
            }
        }
        
        # Add messages for each step
        for step in self.STEPS:
            if step.done != 'finalized':  # Skip finalize step
                self.STEP_MESSAGES[step.id] = {
                    "input": f"Step {step.id}: Please enter {step.show}",
                    "complete": f"{step.show} complete: <>. Continue to next step."
                }

        # Auto-register all routes
        routes = [
            (f"/{app_name}", self.landing),
            (f"/{app_name}/init", self.init, ["POST"]),
            (f"/{app_name}/unfinalize", self.unfinalize, ["POST"]),
            (f"/{app_name}/jump_to_step", self.jump_to_step, ["POST"])
        ]

        # Add step routes automatically from STEPS
        for step in self.STEPS:
            routes.extend([
                (f"/{app_name}/{step.id}", getattr(self, step.id)),
                (f"/{app_name}/{step.id}_submit", getattr(self, f"{step.id}_submit"), ["POST"])
            ])

        # Register all routes
        for path, handler, *methods in routes:
            method_list = methods[0] if methods else ["GET"]
            self.app.route(path, methods=method_list)(handler)

    async def handle_step(self, request, step_id, next_step_id=None):
        """Generic step handler following the Step Display Pattern."""
        step_index = self.steps[step_id]
        step = self.STEPS[step_index]
        pipeline_id = db.get("pipeline_id", "unknown")
        step_data = self.pipulate.get_step_data(pipeline_id, step_id, {})
        user_val = step_data.get(step.done, "")

        # If locked, always chain to next step
        finalize_data = self.pipulate.get_step_data(pipeline_id, "finalize", {})
        locked = ("finalized" in finalize_data)
        if locked:
            return Div(
                Card(P(f"🔒 {step.show}: <{user_val}>")),
                self.pipulate.chain_reaction(next_step_id, self.app_name),
                id=step_id
            )

        # If user_val exists => show revert and chain to next step
        if user_val:
            return Div(
                self.pipulate.revert_control(
                    step_id=step_id,
                    app_name=self.app_name,
                    message=f"{step.show}: {user_val}",
                    steps=self.STEPS
                ),
                self.pipulate.chain_reaction(next_step_id, self.app_name),
                id=step_id
            )

        # Show input message for new step
        await chat.simulated_stream(self.STEP_MESSAGES[step_id]["input"])

        # Return basic input form
        return Form(
            self.wrap_with_inline_button(
                Input(type="text", name=step.done),
                "Continue"
            ),
            hx_post=f"/{self.app_name}/{step_id}_submit",
            hx_target=f"#{step_id}"
        )

    async def init(self, request):
        """Standard init handler that sets up pipeline state."""
        form = await request.form()
        pipeline_id = form.get("pipeline_id", "untitled")
        db["pipeline_id"] = pipeline_id

        # Initialize pipeline
        state = self.pipulate.initialize_if_missing(pipeline_id, {
            "app_name": self.app_name
        })

        if state is None:
            return Card(
                H3("ID Already In Use"),
                P(f"The ID '{pipeline_id}' is already being used. Please try a different ID."),
                style="background-color: var(--pico-del-color);"
            )

        # Generate placeholders for all steps
        placeholders = self.pipulate.generate_step_placeholders(self.STEPS, self.app_name)

        return Div(
            *placeholders,
            id=f"{self.app_name}-container"
        )

Key improvements:

Auto-route registration like TenCardFlow
Generic step handler that implements the Step Display Pattern
Standard init handler with ID collision checking
Better message formatting with step labels
Locked state handling with chain reaction
Revert controls built into the base handler

This gives derived classes a lot more functionality out of the box while still allowing them to override specific behaviors when needed. They can focus on defining their steps and customizing messages rather than reimplementing common patterns.

Refactoring for Cleaner Separation of Concerns

[I moved stuff that wasn’t pipeline state management related from out of Pipulate and into BaseFlow. Cleaner separation of concerns. And it’s now time to fix those revert buttons.

The Revert Button Challenge

Me: Alright. All the transplanting done. Let’s get those Revert buttons working. They do make http connections. Endpoints are wired up. But it chain reacts right past them. And this is one of the stickiest areas. Because… well you tell me.

Claude: Ah yes, the Revert buttons are tricky because they break the normal forward-flow pattern. The key challenges are:

When you revert to step N, you need to:
- Clear state for steps > N
- Stop the chain reaction at step N
- Show the input form for step N
- Preserve completed states for steps < N
The chain reaction system is designed to automatically progress forward, which conflicts with reverting.

Here’s how we can fix it in BaseFlow:

async def handle_revert(self, request):
    """Handle revert action by clearing steps after the reverted step."""
    form = await request.form()
    step_id = form.get("step_id")
    pipeline_id = db.get("pipeline_id", "unknown")
    
    if not step_id:
        return P("Error: No step specified", style="color: red;")
        
    # Clear forward steps in state
    await self.pipulate.clear_steps_from(pipeline_id, step_id, self.STEPS)
    
    # Get the step we're reverting to
    step_index = self.steps[step_id]
    step = self.STEPS[step_index]
    
    # Show input message for the step we're reverting to
    await chat.simulated_stream(self.STEP_MESSAGES[step_id]["input"])
    
    # Return just the input form for this step
    # Important: No chain reaction trigger!
    return Form(
        self.wrap_with_inline_button(
            Input(type="text", name=step.done),
            "Continue"
        ),
        hx_post=f"/{self.app_name}/{step_id}_submit",
        hx_target=f"#{step_id}"
    )

The key is that when reverting, we don’t want to trigger the chain reaction - we want to stop at the reverted step and wait for new input. This breaks the normal pattern but provides better UX for corrections.

Let me know if you want me to show how to modify the chain reaction system to be more revert-aware!

Understanding Revert Approaches

Me: Alright, so you get it! Excellent. So there’s clearly the easy way to revert, which is clearing state after a certain point in the JSON blob and just the natural chain reaction will stop at the right field where there is no data. But then there will be no refill data. And what about the fields where we want to refill it with what was there.

Exploring Implementation Options

Enumerate the different possible approaches. And there are more than one. And as you go, perform over-engineering and exploding complexity evaluations. We’re looking for a sweet spot. Maybe something unlikely, but ridiculously easy to implement in a “you get it for free” kind of way.

Claude: Let me enumerate the approaches, from simplest to most complex:

Pure State Clear (Simplest)
```
# Just clear state and let chain reaction handle it
await clear_steps_from(step_id)
return generate_placeholders()
```
- Pros: Dead simple, works with existing code
- Cons: Loses refill data, might feel jarring
- Complexity: 1/10

State Clear with Refill Flag (Simple+)

# Store cleared values in temp state if refill=True
async def clear_steps_from(self, step_id, preserve_refill=True):
 state = self.read_state()
 for step in self.STEPS[step_index:]:
     if preserve_refill and step.refill:
         state[f"_refill_{step.id}"] = state[step.id]
     del state[step.id]

Pros: Preserves refill data with minimal changes
Cons: Clutters state with temp data
Complexity: 3/10

Shadow State (Medium)

# Maintain parallel state object for reverts
class PipelineState:
 def __init__(self):
     self.active = {}  # Current state
     self.shadow = {}  # Previous values

Pros: Clean separation of concerns
Cons: Doubles state management complexity
Complexity: 6/10

Full History (Complex)

# Keep full history of all values
class StateManager:
 def __init__(self):
     self.history = []  # [{step_id: value, timestamp: ts}, ...]

Pros: Could add undo/redo
Cons: Major overkill, performance implications
Complexity: 8/10

The Sneaky Solution (Elegant?)

# Don't clear state at all - just hide it in UI
async def handle_revert(self, request):
 step_id = request.form.get("step_id")
 # Don't clear state, just return the input form
 return Form(
     Input(
         type="text",
         name=step.done,
         value=self.get_step_data(step_id) if step.refill else ""
     ),
     hx_post=f"/{self.app_name}/{step_id}_submit"
 )

Pros:
- Surprisingly simple
- Natural refill behavior
- No state gymnastics
- Previous values still there if needed
Cons:
- Feels almost too easy
- State doesn’t match UI (but does this matter?)
Complexity: 2/10

The “Sneaky Solution” is interesting because it:

Leverages existing step.refill flag
Keeps all data available
Uses UI state rather than data state to control flow
Gets complex features “for free”

It’s like discovering that instead of building a complex state management system, we could just let the UI be the source of truth for “current step” while keeping all data available.

What do you think? Is this too clever, or is it the kind of elegant simplicity we’re looking for?

Implementation Challenges

Me: Well of course I like your last one. But I challenge you. Is implementation really that easy? The card is going to have to be redrawn, going from its non-editable state back to its editable state. It sounds like you’re proposing not using the chain reaction? How would that work? We have HTMX at our disposal, so I’m not counting it out. Additionally, the HTML structure of these chain reacting forms is nested. Think about what a single HTMX action could do. Upon clicking a Revert button…

Nested Complexity

Claude: Ah, now I see the full complexity! You’re right - the HTML structure is deeply nested with chain reactions, and we need to consider:

The actual DOM structure shows each step nested inside the previous one
There are parallel placeholder divs at the bottom for chain reactions
When you revert to step N, you need to:
- Replace step N’s content with the input form
- Remove steps > N from the DOM
- Keep steps < N intact
- Maintain the chain reaction structure

Here’s a revised approach using HTMX more strategically:

async def handle_revert(self, request):
    """Handle revert action by rebuilding from the reverted step."""
    form = await request.form()
    step_id = form.get("step_id")
    pipeline_id = db.get("pipeline_id", "unknown")
    
    step_index = self.steps[step_id]
    step = self.STEPS[step_index]
    
    # Get the current value if it should be refilled
    current_value = ""
    if step.refill:
        state = self.pipulate.read_state(pipeline_id)
        step_data = state.get(step_id, {})
        current_value = step_data.get(step.done, "")

    # Return a structure that:
    # 1. Shows completed steps up to N-1
    # 2. Shows input form for step N
    # 3. Adds placeholder divs for steps > N
    return Div(
        # Previous completed steps remain unchanged
        *[self.render_completed_step(s) for s in self.STEPS[:step_index]],
        
        # Current step becomes input form
        Card(
            H3(f"Enter {step.show}"),
            Form(
                Input(
                    type="text",
                    name=step.done,
                    value=current_value,
                    required=True,
                    autofocus=True
                ),
                hx_post=f"/{self.app_name}/{step_id}_submit",
                hx_target=f"#{step_id}"
            )
        ),
        
        # Add fresh placeholder divs for chain reaction
        *[Div(
            id=s.id,
            hx_get=f"/{self.app_name}/{s.id}",
            hx_trigger=f"stepComplete-{prev.id} from:{prev.id}",
            hx_swap="outerHTML"
        ) for prev, s in zip(self.STEPS[step_index:], self.STEPS[step_index+1:])],
        
        id=f"{self.app_name}-container"
    )

This approach:

Keeps the chain reaction mechanism
Preserves refill values
Correctly rebuilds the DOM structure
Maintains the proper nesting and triggers

The key insight is that we need to rebuild the entire container, but we can do it in a way that preserves both state and structure. What do you think?

The Chain Reaction Challenge

Understanding the Core Issues

Me: So what you have to answer is why won’t chain reaction cause all cards to render to the last empty one or the end? What actually stops it on the card we want to revert to? That’s half the challenge. Merely stopping a chain reaction whose behavior is inherent in the cards themselves. Notice the hx_trigger="load"’s How are you going to interrupt that? Not so easy, is it? Cards load as cards load, right up until empty or done and there’s not much you can do about it without exploding complexity. Or is there?

The Data Preservation Problem

And then the other half of it is even if you can get the chain reaction to stop on the same card as the ID in the Revert button you press, how are you sitting on top of the refill data you need to use? If you’ve taken the easy chain reacting solution and cleared forward from N, you’re not. If you offset N, your chain reaction isn’t going to stop on the correct card.

Riddle me that, Claude.

Claude: Ah, now I REALLY see the challenge! The chain reaction is baked into the cards themselves through those load triggers, and the data availability creates a catch-22:

If we clear state from step N:
- Chain reaction stops naturally at step N (no data)
- BUT we lost our refill data
If we clear state from step N+1:
- We preserve refill data for step N
- BUT chain reaction continues past N because step N still has data
If we try to manipulate the triggers:
- We’d need to modify every card’s HTML
- Complex and fragile

But wait… what if we:

Keep ALL the data
Add a “revert_target” to the pipeline state
Modify handle_step to check this target

async def handle_revert(self, request):
    form = await request.form()
    step_id = form.get("step_id")
    pipeline_id = db.get("pipeline_id", "unknown")
    
    # Don't clear state! Just set revert target
    state = self.pipulate.read_state(pipeline_id)
    state["_revert_target"] = step_id
    self.pipulate.write_state(pipeline_id, state)
    
    # Return same container structure as init()
    placeholders = self.generate_step_placeholders(self.STEPS, self.app_name)
    return Div(*placeholders, id=f"{self.app_name}-container")

async def handle_step(self, request):
    """Generic step handler that checks for revert target."""
    step_id = request.url.path.split('/')[-1]
    pipeline_id = db.get("pipeline_id", "unknown")
    state = self.pipulate.read_state(pipeline_id)
    
    # If this step is the revert target, show input form
    if state.get("_revert_target") == step_id:
        step = self.STEPS[self.steps[step_id]]
        return Card(
            H3(f"Enter {step.show}"),
            Form(
                Input(
                    type="text",
                    name=step.done,
                    value=state.get(step_id, {}).get(step.done, ""),
                    required=True,
                    autofocus=True
                ),
                hx_post=f"/{self.app_name}/{step_id}_submit",
                hx_target=f"#{step_id}"
            )
        )
    
    # Normal step handling continues...

This lets us:

Use the natural chain reaction
Keep all data intact
Intercept at exactly the right step
Access refill data directly
Avoid complex DOM surgery

The chain reaction still happens, but handle_step catches it at exactly the right moment and shows the input form instead of the completed card. What do you think? Too clever or just clever enough?

Understanding the State Management Approach

Me: Put it to me in other words. Break it down Barney style. Why isn’t this a virtual state solution that doubles state management and explodes complexity?

Claude: Okay, Barney style:

Normal Flow (No Revert)
- Step 1 done: “Mike” ✓
- Step 2 done: “email” ✓
- Step 3: empty
- Chain reaction stops at 3 (no data)
With Revert to Step 2
- We add ONE piece of info: _revert_target: "step_02"
- ALL data stays exactly as is
- When chain reaction hits step 2:
  - Handler sees: “Oh, you’re the revert target!”
  - Shows input form instead of completed card
  - With bonus: Old value is right there for refill

It’s not doubling state because:

We’re not copying anything
We’re not tracking history
Just ONE extra field that says “show input form here”
Everything else uses existing mechanisms

It’s like putting a “STOP” sign on a specific step rather than rebuilding the whole road.

Is that clearer? The beauty is we’re not fighting the chain reaction - we’re just teaching one step to behave differently when it sees its name on the “revert list”.

Understanding the Revert Button Behavior

Me: So describe the Revert button you would click and what value it would contain relative to the card it’s being displayed on. Do you click Step 2’s Revert button to edit Step 2?

Claude: Ah, now I see a potential gotcha in my thinking…

When you see:

Step 1: "Mike" [Revert]
Step 2: "email" [Revert]
Step 3: "phone" [Revert]

If you click Step 2’s Revert button, you actually want to edit Step 2, not Step 3. But in my solution, by the time the chain reaction hits Step 2, it would already have rendered Step 1 and be working on Step 2’s display.

So we’d need to offset the target. If you click “Revert” on Step 2’s card, we’d need to set:

state["_revert_target"] = "step_01"  # One step back

But that feels wrong and fragile. It’s the kind of “off by one” complexity that leads to bugs.

Was I thinking about this backwards? Should the revert button on Step 2 actually revert to Step 2 and I need to rethink the whole approach?

Challenging the Fragility Assumption

No, I believe you have the right solution. And there will be a tiny bit to it that has that hackish feeling, but I challenge that it’s fragile and bug-prone because you’ve got explicit self.STEPS to work with. Offset is offset in a rather fixed way. If you offset knowing your STEPS data, it’s the same every day! And the way I propose, it’s the same way you said, but button values are one step ahead!

The HTMX Timing Challenge

Now the insidious thing, the fact that’s defiant, is htmx is done triggered by client. By the time that it is, your idea that’s great, involving that offset is one card too late! Now before you go trash it, give those AI brains a flex and consider the server when serving step_xx.

The Solution Hint

Now it might need to do a wee bit more than does now, but methinks that self.STEP is the key to just how.

Claude: AHA! Now I get it!

When serving step_xx, we can look at the state’s _revert_target and know:

If this step is one AFTER the revert target (using STEPS)
That THIS is where we need to show the input form
And we have the previous step’s data for refill right there!

So:

async def handle_step(self, request):
    step_id = request.url.path.split('/')[-1]
    pipeline_id = db.get("pipeline_id", "unknown")
    state = self.pipulate.read_state(pipeline_id)
    
    # NEW BIT: Check if previous step is revert target
    step_index = self.steps[step_id]
    if step_index > 0:  # Not first step
        prev_step = self.STEPS[step_index - 1]
        if state.get("_revert_target") == prev_step.id:
            # Show input form with previous step's data
            prev_data = state.get(prev_step.id, {}).get(prev_step.done, "")
            return Card(
                H3(f"Enter {prev_step.show}"),
                Form(
                    Input(type="text", name=prev_step.done, 
                          value=prev_data if prev_step.refill else "",
                          required=True, autofocus=True),
                    hx_post=f"/{self.app_name}/{prev_step.id}_submit",
                    hx_target=f"#{prev_step.id}"
                )
            )
    
    # Rest of normal handle_step logic...

The revert button shows the step it’s on, but sets the target to itself, and the server catches it one step later! Clean, deterministic, and works with the chain reaction instead of fighting it.

Is this what you were driving at? 🎯

Understanding Chain Reaction Control

The And-Not Condition

Me: Elaborate again. Break it down Barney style. Honestly, I was thinking an and-not condition here,

        # If completed, show with revert and chain
        if user_val:
            return Div(
                self.pipulate.revert_control(
                    step_id=step_id,
                    app_name=self.app_name,
                    message=f"{step.show}: {user_val}",
                    steps=self.STEPS
                ),
                Div(id=next_step_id, hx_get=f"/{self.app_name}/{next_step_id}", hx_trigger="load")
            )

…so it slips past chain reaction and ends up on the fallover condition sitting on data. The trick is to set the and not condition. Or would that be an end knot condition?

Claude: AHA! Even simpler! In handle_step, we already have this logic fork:

        # If completed, show with revert and chain
        if user_val:
            return Div(...)

        # Show input form
        return Div(
            Card(
                H3(f"Enter {step.show}"),
                Form(...)
            ),
            Div(id=next_step_id),
            id=step_id
        )

So we can just add our revert check to the first condition:

        # If completed AND NOT revert target, show with revert and chain
        if user_val and state.get("_revert_target") != step_id:
            return Div(...)

        # Show input form (either empty OR revert target)
        return Div(
            Card(
                H3(f"Enter {step.show}"),
                Form(
                    Input(
                        type="text",
                        name=step.done,
                        value=user_val if step.refill else "",  # Refill data right here!
                        required=True,
                        autofocus=True
                    )
                )
            ),
            Div(id=next_step_id),
            id=step_id
        )

Beautiful! The chain reaction naturally:

Shows completed cards until it hits revert target
Falls through to input form on target (with refill data!)
Stops chain reaction there (no completed card)

No offset needed. No complexity. Just one extra AND NOT condition. 🎯

The Breakthrough

Me: That was it Claude, we did it! It was worth all this refactoring and the making of the base class. This is such simpler code and is going to be so much more manageable over time. And that always clears forward thing I was so tied to because of data integrity, I’m going to make a per-workflow parameter, because I can already anticipate users objecting to it in some situations where those later fields were hard to fill out and aren’t really dependencies on the prior. And maybe with per-field granularity, but all that can be worked out later. The important thing is that almost anything can be overwritten by the workflow.

Future Extensibility

I also like that it will be logical to put a growing library of input field types like textarea in BaseFlow, and that will make individual workflow instances like the ExampleFlow I was planning less important.

Claude: Yes, this is a great outcome! Let me summarize the key wins:

Simpler State Management
- No complex state tracking
- Just one _revert_target field
- Natural use of existing chain reaction
Flexible Data Handling
- Clear-forward becomes optional per workflow
- Refill data naturally available
- Field dependencies can be workflow-specific
Extensible Base Class
- Common UI components can live in BaseFlow
- Input field types (textarea, select, etc.) as reusable methods
- Workflows just need to define steps and custom validation
Clean Separation of Concerns
- Pipulate handles state
- BaseFlow handles UI/UX
- Individual workflows handle business logic

The “you get it for free” solution we found for revert is exactly the kind of elegant simplicity you want in a framework - it works with the existing patterns rather than fighting them. And because it’s all built on a solid foundation, adding new features (like per-field clear rules) becomes much more manageable.

This is a great example of how sometimes making code more flexible actually makes it simpler, not more complex.

Me: I’m glad it worked out so well (it did, by the way) balancing both StarterFlow and MadLibsFlow. Wait until you see LinkGraphFlow. Think that’ll work in this new BaseFlow approach of ours?

Claude: Looking at the LinkGraphFlow code, it’s significantly more complex than both StarterFlow and MadLibsFlow. It has:

API integrations with Botify
File download handling
Complex state management for exports
Polling mechanisms
Multiple data fields per step
Custom routing for deletions
Specialized step completion logic

While our new BaseFlow approach provides a good foundation for simple linear flows, LinkGraphFlow might need some adaptations:

We might need to extend the Step namedtuple to handle multiple fields and API states
The clear_steps_from logic might need to account for in-progress API operations
We may need hooks for pre/post step processing to handle API calls
The suggestion system might need to work with complex data structures
The revert control might need special handling for cleanup of downloaded files

Would you like me to propose some specific adaptations to make LinkGraphFlow work with our new BaseFlow system? Or should we keep LinkGraphFlow as a separate implementation for now?

Me: Yes. But see why I was so opposed to Base classes for so long? It’s mostly going to be stuff like this. We can not only override base classes, but we can wholesale ignore it when we have to, if it would be more hindrance than help. My hope is it will provide a sort of scaffolding at least.

Claude: Absolutely! This is a perfect example of why inheritance and base classes need to be approached carefully. The classic “is-a” relationship can break down quickly when real-world complexity hits.

In this case:

StarterFlow and MadLibsFlow are truly simple linear flows where BaseFlow provides genuine value
LinkGraphFlow is more of a state machine / orchestration system that happens to have some flow-like properties

We could:

Keep LinkGraphFlow separate, using BaseFlow’s patterns as inspiration where helpful
Cherry-pick useful BaseFlow methods without full inheritance
Create a new base class specifically for API-integrated flows (though that might be premature)

This is why patterns like composition over inheritance and “favor small interfaces” exist. Real systems rarely fit perfectly into clean hierarchies.

Would you like to explore how we might selectively adopt some BaseFlow patterns into LinkGraphFlow without forcing a full inheritance relationship? Or should we keep it entirely separate for now?

Me: It’s still completely linear. It just has fancier menus and side effects.

Claude: Ah, you’re right! I was getting distracted by the complexity of the individual steps and missing the forest for the trees. At its core, LinkGraphFlow is still:

Linear progression through steps
Each step builds on previous steps
Clear forward on submit
Revert controls for going back
Final step for completion

The main differences are:

Richer side effects (API calls, file downloads)
More complex data structures per step
Additional routes for polling/deletion
Specialized step completion rendering

We could actually adapt it to use BaseFlow while keeping its specialized functionality. The core flow mechanics would be handled by BaseFlow while LinkGraphFlow adds its API integration and file handling.

Would you like to see how we could refactor LinkGraphFlow to leverage BaseFlow without losing any of its current capabilities? It might actually help simplify some of the boilerplate code.

Me: The goal here is to have a hammer and make everything look like a nail. It’s an intentional simplifcation. Everything is Unix pipes. Everything is a Juptyer Notebook with top-down cells.

Claude: Ah NOW I get it! This is a profound insight about intentional simplification. By treating everything as a linear flow:

Each step is like a cell in a notebook
Output from one step feeds into the next
You can always “restart kernel and run all cells”
You can modify and re-run from any point
The complexity is contained within each step, not in the flow structure

So LinkGraphFlow isn’t really different - it’s just cells that happen to:

Make API calls
Download files
Handle polling
Manage side effects

But the fundamental pattern is identical to StarterFlow and MadLibsFlow:

Step 1 -> Step 2 -> Step 3 -> ... -> Finalize

Would you like to see how we could adapt LinkGraphFlow to use BaseFlow while keeping this “everything is a notebook cell” mentality? The API complexity stays inside each step, but the flow mechanics become standardized.

Gemini 1.5’s Summary

Gemini 1.5: This article describes a process of refactoring a multi-step workflow system to simplify state management and improve user experience. The author and Claude, an AI assistant, collaborate to eliminate redundant state tracking, optimize data handling for “revert” actions, and enhance the system’s extensibility. They achieve this by introducing a base class for workflows, refining the handling of step completion and reversion, and strategically using HTMX for dynamic UI updates. The result is a cleaner, more manageable, and user-friendly workflow system.

Title and Headline Ideas

Taming the Chain Reaction: A Workflow Refactoring Journey
Simplifying State: A Quest for Elegant Workflow Management
Refactoring Revelations: When Less is Truly More
From Chaos to Clarity: A Workflow System’s Transformation
The “Sneaky Solution”: A Clever Trick for Effortless Reverts

Opinion

This article provides a fascinating glimpse into the collaborative process of refactoring code for improved clarity and efficiency. The conversational style effectively conveys the challenges and “aha!” moments encountered along the way. The author and Claude demonstrate a thoughtful approach to problem-solving, carefully considering various options and their implications before arriving at an elegant solution. The final result is a testament to the power of collaborative design and the pursuit of simplicity in code.