Understanding Pipulate Workflows: Getting Started
This document dives into the technical specifics of building automated processes, called “workflows,” within a software framework known as Pipulate. Modern web applications often need ways to guide users through multiple steps, and updating the user interface smoothly between these steps can be complex. The approach discussed here uses a technology called HTMX, which helps manage these dynamic updates efficiently, often reducing the need for extensive custom JavaScript.
The core focus is on a specific pattern called the “HTMX chain reaction,” which allows workflow steps to automatically load one after another. The text also explains a technique called “placeholder steps,” which are used to structure the workflow initially with minimal functionality. These guides and patterns are particularly relevant because the author is using them to instruct an AI coding assistant (like Claude) on how to modify the workflow code correctly, as the underlying mechanisms require very precise implementation.
Implementation Guides For AIs
This is an accelerator. I have individual little examples of modules that go into Workflows, but I don’t have the finished Workflows themselves. The idea is I should be able to slam them together from parts. It’s a matter of lowering the friction on something that’s full of anti-patterns and off the beaten track thinking. LLMs resist.
Consequently, they have to be more or less cornered into correct implementations with a full and complete understanding of what’s being asked… and why! Yes, I know it’s hard to believe that giving an LLM context for why you’re doing a thing is helpful, but when you go off the beaten track, you must illuminate the way. LLMs will buck and resist uncharted territory, so make the charts.
I don’t really expect anyone to read this like an article. It’s more of an “I’ll
just put this here” situation. I was tempted to make a new docs
directory for
it, but I put it in the Pipulate repo under the same training
folder that’s
used for real-time prompt-injection for the local LLM as the apps are running.
Example Use of The Guide
I’m going to start this out with how I actually used the implementation guide to steer the LLM (Claude 3.7 Sonnet in Cursor AI) through 2 consecutive perfect edits doing something that was nearly impossible before this guide.
Me: That’s friggin amazing! You did a perfect step 2 and step 3 blank placeholder insertion, right down to the proper use of the revert buttons and the maintenance of the proper chain reaction effects. Do a detailed breakdown of why you think it went off so well. Include what things that surprised you most, and what things you most had to self-correct to get it right on the 1st try.
Claude: The implementation went well because the widget_implementation_guide.md provided clear patterns to follow with detailed explanations. Here’s why it was successful:
Key Success Factors
- Clear Step-by-Step Patterns
- The guide showed explicit code patterns for each component
- The ASCII diagrams illustrated the chain reaction flow visually
- The placeholder step pattern was precisely documented
- Critical Element Preservation
- I understood that the chain reaction Div with
hx_trigger="load"
is non-negotiable - Recognized the importance of
pip.revert_control
for proper navigation - Maintained exact pattern matching for state management
- I understood that the chain reaction Div with
- Systematic Approach
- Added steps to the step list first
- Ensured handler methods followed the exact same pattern
- Preserved all parameter names and context variables
What Surprised Me
- The Importance of Empty Divs
- The input form has
Div(id=next_step_id)
without thehx_trigger="load"
attribute - Only the completed state includes the full chain reaction, preventing premature progression
- The input form has
- State Persistence Strategy
- Using a fixed placeholder value “completed” rather than capturing form data
- The pattern
await pip.update_step_state(pipeline_id, step_id, placeholder_value, steps)
being preferred over direct state modification
- Locked State Handling
- The separate code path for finalized workflows with the 🔒 emoji
- The continuation of chain reactions even in locked state
Self-Correction Points
- Revert Control Implementation
- Initially considered using a custom revert button construction
- Corrected to use
pip.revert_control
for consistency and proper functioning
- Chain Reaction Timing
- Was careful to ensure the chain reaction div was only active when needed
- Verified that
hx_trigger="load"
appears only in submission responses and finalized views
- Message Queue Usage
- Made sure to include the appropriate message queue notifications before form rendering
- Used the step-specific messages from
self.step_messages[step_id]["input"]
- Step Indices Management
- Recognized the pattern of looking up step information using
self.steps_indices[step_id]
- Verified
next_step_id
calculation using the step index pattern
- Recognized the pattern of looking up step information using
The most valuable aspect of the implementation guide was how it documented not just what to do, but why each pattern exists. Understanding that the chain reaction is the core mechanism for automatic progression made it clear why certain elements couldn’t be modified.
The “WET” approach (Write Everything Twice) actually made implementation easier - by copying and adapting the existing pattern rather than trying to be clever with abstractions, I ensured that all the critical elements remained intact.
Guide and Primer On Pipulate Workflow Patterns
This serves as a guide and primer on the fundamental patterns used for building and modifying workflows within the Pipulate framework, particularly focusing on step insertion and maintaining the crucial HTMX chain reaction mechanism. It doesn’t introduce new requirements for the codebase but rather documents how the codebase works and how developers (or AI assistants) should interact with it when adding or modifying workflow steps.
# PIPULATE WORKFLOW PROGRESSION GUIDE
## 1. EMPTY WORKFLOW (LANDING → FINALIZE)
┌─────────────┐ ┌───────────┐
│ landing │ │ finalize │
│ (method) │ ---------------> │ (method) │
└─────────────┘ └───────────┘
Key Code Connection:
# In landing method return statement
return Div(
Card(...),
# Chain reaction initiator
Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"),
id="landing"
)
## 2. INSERTING STEP_01 (LANDING → STEP_01 → FINALIZE)
┌─────────────┐ ┌──────────────┐ ┌───────────┐
│ landing │ │ step_01 │ │ finalize │
│ (method) │ -----> │ (placeholder)│ -----> │ (method) │
└─────────────┘ └──────────────┘ └───────────┘
Changes Required:
1. Add step_01 to steps list in __init__
2. Create step_01 and step_01_submit methods
3. Modify landing chain reaction to point to step_01
# Updated landing return statement
return Div(
Card(...),
# Chain reaction now points to step_01
Div(id="step_01", hx_get=f"/{app_name}/step_01", hx_trigger="load"),
id="landing"
)
# step_01 method
return Div(
Card(...),
# Chain reaction to finalize
Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"),
id="step_01"
)
## 3. ADDING STEP_02 (LANDING → STEP_01 → STEP_02 → FINALIZE)
┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────────┐
│ landing │ │ step_01 │ │ step_02 │ │ finalize │
│ (method) │ -----> │ (placeholder)│ -----> │ (placeholder)│ -----> │ (method) │
└─────────────┘ └──────────────┘ └──────────────┘ └───────────┘
Changes Required:
1. Add step_02 to steps list in __init__
2. Create step_02 and step_02_submit methods
3. Modify step_01 chain reaction to point to step_02
# Updated step_01 return statement
return Div(
Card(...),
# Chain reaction now points to step_02
Div(id="step_02", hx_get=f"/{app_name}/step_02", hx_trigger="load"),
id="step_01"
)
# step_02 method
return Div(
Card(...),
# Chain reaction to finalize
Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"),
id="step_02"
)
## 4. COMPLETE WORKFLOW WITH STEP_03 (LANDING → STEP_01 → STEP_02 → STEP_03 → FINALIZE)
┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────────┐
│ landing │ │ step_01 │ │ step_02 │ │ step_03 │ │ finalize │
│ (method) │ -----> │ (placeholder)│ -----> │ (placeholder)│ -----> │ (placeholder)│ -----> │ (method) │
└─────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └───────────┘
Changes Required:
1. Add step_03 to steps list in __init__
2. Create step_03 and step_03_submit methods
3. Modify step_02 chain reaction to point to step_03
# Updated step_02 return statement
return Div(
Card(...),
# Chain reaction now points to step_03
Div(id="step_03", hx_get=f"/{app_name}/step_03", hx_trigger="load"),
id="step_02"
)
# step_03 method
return Div(
Card(...),
# Chain reaction to finalize
Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"),
id="step_03"
)
## 5. DYNAMIC CHAIN REACTION (GENERALIZED PATTERN)
For any step_XX in a workflow:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ step_prev │ │ step_XX │ │ step_next │
│ (method) │ -----> │ (your step) │ -----> │ (method) │
└──────────────┘ └──────────────┘ └──────────────┘
Key Pattern Code:
```python
# Determine the next step dynamically
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
# Return with proper chain reaction
return Div(
Card(...),
# Dynamic chain reaction
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
```
## 6. CRITICAL CHAIN REACTION ELEMENTS
┌───────────────────────────────────────────────────────────────────┐
│ CHAIN REACTION COMPONENT │
├───────────────────────────────────────────────────────────────────┤
│ Div(id=next_step_id, │
│ hx_get=f"/{app_name}/{next_step_id}", │
│ hx_trigger="load") │
└───────────────────────────────────────────────────────────────────┘
│
│ Must preserve these three attributes
▼
┌───────────────────────────────────────────────────────────────────┐
│ id=next_step_id : Container to be replaced with next step │
│ hx_get=... : URL to fetch next step content │
│ hx_trigger="load" : Automatically triggers when this div appears │
└───────────────────────────────────────────────────────────────────┘
## 7. STEP METHOD STRUCTURE AND CONNECTIONS
┌── Shared step_id = "step_XX"
│
│ ┌─ Common code pattern
▼ ▼
┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
│ step_XX (GET) │ │ step_XX_submit (POST) │
├────────────────────────────────────────┤ ├────────────────────────────────────────┤
│ 1. Get step metadata │ │ 1. Get step metadata │
│ 2. Get pipeline state │──┐ │ 2. Process form submission │
│ 3. Display form with submit button │ │ │ 3. Validate input │
│ 4. Include revert button │ │ │ 4. Save state │
│ 5. Setup chain reaction to next step │ │ │ 5. Update progress message │
└────────────────────────────────────────┘ │ │ 6. Setup chain reaction to next step │
│ └────────────────────────────────────────┘
│ │
│ │
▼ ▼
┌────────────────────────────────────────────┐
│ hx_post │
│ Form(... │
│ hx_post=f"/{app_name}/{step_id}_submit" │
│ ) │
└────────────────────────────────────────────┘
## 8. SPLICING IN NEW STEPS (BETWEEN STEP_01 AND STEP_02)
Before:
step_01 → step_02
After:
step_01 → new_step → step_02
Required Changes:
1. Add new_step to steps list between step_01 and step_02
2. Update step_indices in __init__
3. Create new_step and new_step_submit methods
4. Modify step_01's chain reaction to point to new_step
5. Set new_step's chain reaction to point to step_02
```
This guide illustrates the key progression patterns in Pipulate workflows, focusing on:
1. How steps connect through chain reactions
2. The critical HTMX attributes needed for automatic progression
3. The standardized method structure for both GET and POST handlers
4. How to insert steps at different positions (beginning, middle, end)
5. The dynamic determination of the next step ID
When changing placeholders to functional widgets, preserve these essential elements while adding your specific form elements and processing logic.
---
# Guide to Adding Placeholder Steps in Pipulate Workflows
## Analysis of the Placeholder Step Process
When adding a placeholder step to a Pipulate workflow, I followed a precise, minimalist approach that maintained all core workflow patterns while removing unnecessary UI elements. Here's an analysis of what made this approach successful:
### Key Components Preserved
1. **Chain Reaction Pattern**: Maintained the critical `Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load")` structure that automatically advances the workflow.
2. **Revert Controls**: Kept the widget_container pattern which ensures proper revert button generation.
3. **Step State Management**: Preserved state persistence using `pip.read_state()` and `pip.write_state()`.
4. **Finalization Handling**: Maintained support for both normal and finalized workflow states.
5. **Position Awareness**:
- For steps in the middle: Connection to both previous and next steps
- For the last step: Special handling of the finalize step
- For the first step: Connection to the landing page
### Special Considerations by Position
#### First Step (after Landing)
- Must handle initial state setup
- Links directly from the landing page
- Usually sets a foundation for later steps
#### Middle Steps
- Must handle data from previous steps
- Must properly chain to next steps
- Must support reverting from any point
#### Last Step (before Finalize)
- Must properly connect to the finalize step
- Often serves as a summary or confirmation
- Must handle the transition to locked state
## Guide to Inserting Placeholder Steps in Pipulate Workflows
### 1. Add the Step Definition
```python
Step(
id='step_XX', # Use appropriate numbering
done='placeholder', # Field to store completion state
show='Placeholder Step', # User-facing name
refill=True, # Allow refilling on revert
),
```
### 2. Add Example Suggestion
Add a simple placeholder suggestion to the `get_suggestion` method:
```python
'step_XX': """Placeholder step - no user content needed.
This step serves as a placeholder for future widget types."""
```
### 3. Create the GET Handler
The get handler should:
- Handle normal, completed, and finalized states
- Include only a simple Proceed button
- Maintain chain reaction pattern
```python
async def step_XX(self, request):
"""Handles GET request for the placeholder step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
placeholder_value = step_data.get(step.done, "placeholder_value")
# Finalized state
finalize_data = pip.get_step_data(pipeline_id, "finalize", {})
if "finalized" in finalize_data and placeholder_value:
return Div(
Card(
H3(f"🔒 {step.show}"),
P("Placeholder step completed")
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load")
)
# Completed state
if placeholder_value and placeholder_value != "placeholder_value" and state.get("_revert_target") != step_id:
content_container = pip.widget_container(
step_id=step_id,
app_name=app_name,
message=f"{step.show} Completed",
widget=P("Placeholder step completed"),
steps=steps
)
return Div(
content_container,
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load")
)
# Input state - simple Proceed button
await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True)
return Div(
Card(
H3(f"{pip.fmt(step_id)}: {step.show}"),
P("This is a placeholder step. No input is required."),
Form(
Div(
Button("Proceed", type="submit", cls="primary"),
style="margin-top: 1vh; text-align: right;"
),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}"
)
),
Div(id=next_step_id),
id=step_id
)
```
### 4. Create the Submit Handler
The submit handler should:
- Skip data collection but maintain state persistence
- Use a fixed value for the completion state
- Maintain chain reaction to the next step
```python
async def step_XX_submit(self, request):
"""Process the submission for placeholder step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
pipeline_id = db.get("pipeline_id", "unknown")
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
# Use fixed value instead of form data
placeholder_value = "completed"
# Save to state
state = pip.read_state(pipeline_id)
if step_id not in state:
state[step_id] = {}
state[step_id][step.done] = placeholder_value
pip.write_state(pipeline_id, state)
# Simple confirmation widget
placeholder_widget = P("Placeholder step completed")
# Standard widget container (includes revert controls)
content_container = pip.widget_container(
step_id=step_id,
app_name=app_name,
message=f"{step.show} Completed",
widget=placeholder_widget,
steps=steps
)
# Chain reaction to next step
response_content = Div(
content_container,
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
await self.message_queue.add(pip, f"{step.show} complete.", verbatim=True)
return HTMLResponse(to_xml(response_content))
```
## Critical Patterns to Maintain
1. **HTMX Chain Reaction**: The `Div(id=next_step_id, hx_get=...)` pattern is essential for workflow progression.
2. **ID Consistency**: All elements must use the appropriate `step_id` for targeting.
3. **State Management**: Always read and update state properly to maintain workflow integrity.
4. **Widget Container**: Always use `pip.widget_container()` to ensure revert controls are generated.
5. **Next Step Determination**: Always calculate `next_step_id` based on position to handle the finalize step correctly.
## Sample Prompts for AI Assistance
1. **Adding a placeholder in the middle:**
```
Please add a new placeholder step named "step_04" between the current step_03 and step_04 in the workflow class WorkflowName. The placeholder should have a minimal UI with just a Proceed button, maintaining all standard workflow patterns including the chain reaction, revert controls, and state management. Don't implement any specific widget functionality yet.
```
2. **Adding a placeholder at the end:**
```
Add a final placeholder step "step_06" to the workflow before the finalize step. It should maintain all workflow patterns but have a minimal UI with only a Proceed button. Make sure it properly connects to the finalize step.
```
3. **Extending an existing workflow:**
```
Extend the ExistingWorkflow class by adding a placeholder step "step_05" after the last current step. The placeholder should follow the same structural patterns as other steps but contain only a Proceed button instead of any input widgets. Ensure proper chain reaction connections.
```
4. **Converting complex step to placeholder:**
```
Convert step_03 in MyWorkflow from its current complex implementation to a simple placeholder step with just a Proceed button. Maintain all existing step IDs, connections, and workflow patterns, but remove the complex UI components.
```
5. **Incrementally building a workflow:**
```
I'm building a workflow incrementally. First, please add a placeholder step "step_02" after my existing step_01. Keep all standard workflow patterns intact, but implement only a minimal UI with a Proceed button. Later, I'll replace this with actual functionality.
```
By following this guide, you can reliably add placeholder steps to Pipulate workflows that maintain all the necessary structural patterns while providing a clean slate for future widget implementation.
---
## Cursor Rules: htmx-chain-reactions.mdc
**placeholder-step-pattern** - A new rule that covers:
- When and why to use placeholder steps
- Step-by-step implementation pattern
- Critical elements that must be preserved (chain reactions, revert buttons, etc.)
- Placement considerations based on position (first, middle, or last step)
- Examples and upgrade instructions
# chain-reaction-pattern
## Overview
The chain reaction pattern is the core mechanism for step progression in Pipulate workflows. It enables automatic advancement through workflow steps without requiring manual user intervention for each transition.
## Standard Implementation
```python
return Div(
Card(...), # Current step content
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), # Next step loader
id=step_id
)
```
## Key Elements
- The inner `Div` with `id=next_step_id` serves as a container for the next step
- The `hx_get` attribute requests the next step from the server
- `hx_trigger="load"` makes this happen automatically when current step renders
- The outer `Div` with `id=step_id` is replaced when the current step completes
## Breaking the Chain (Cautionary Pattern)
The `no-chain-reaction` class should only be used in two specific scenarios:
1. **Terminal responses** - For endpoints requiring explicit user action:
```python
return Div(
result_card,
download_button,
cls="terminal-response no-chain-reaction",
id=step_id
)
```
2. **Polling states** - For continuous status checking without progression:
```python
return Div(
result_card,
progress_indicator,
cls="polling-status no-chain-reaction",
hx_get=f"/{app_name}/check_status",
hx_trigger="load, every 2s",
hx_target=f"#{step_id}",
id=step_id
)
```
## Best Practices
- Always maintain the chain reaction pattern unless absolutely necessary
- When breaking the chain, provide clear UI indicators of what's happening
- Resume the chain reaction as soon as the exceptional condition is complete
- Document any use of `no-chain-reaction` with explicit comments explaining why
## Example in Botify Export
See [50_botify_export.py](mdc:pipulate/plugins/50_botify_export.py) lines 1515-1540 for implementation examples of both patterns.
## Cursor Rules: wet-workflows.mdc
**wet-workflows** - Enhanced the existing rule with:
- Detailed step progression patterns for both GET and POST handlers
- Step insertion guidance
- Special case handling for different workflow scenarios
- Clear reference to the placeholder-step pattern
# Pipulate Workflow Development Guide
## Core Structure
Every workflow must follow this basic structure:
```python
class WorkflowName:
APP_NAME = "unique_name" # Unique identifier
DISPLAY_NAME = "User Name" # UI display name
ENDPOINT_MESSAGE = "..." # User guidance
TRAINING_PROMPT = "name.md" # Training file
PRESERVE_REFILL = True/False # State preservation
```
## Required Files
- Main workflow file in `pipulate/plugins/` with format `XX_name.py`
- Training prompt file referenced in `TRAINING_PROMPT`
- Plugin numbering indicates load order and dependencies
## Step Definition
Steps use the namedtuple pattern:
```python
Step = namedtuple('Step', ['id', 'done', 'show', 'refill', 'transform'])
steps = [
Step(
id='step_01', # Format: step_XX
done='field_name', # State field name
show='Display Name', # UI display text
refill=True/False, # Preserve on revert
transform=lambda x: x # Optional transform
)
]
```
## Required Methods
Every workflow must implement:
1. `__init__`: Setup routes and steps
2. `landing`: Initial page display
3. `init`: Workflow initialization
4. `finalize/unfinalize`: State management
5. `handle_revert`: Step reversal
6. Per-step handlers: `step_XX` and `step_XX_submit`
## UI Components
- Use FastHTML components consistently
- Follow HTMX patterns for dynamic updates
- Maintain chain reaction for step progression
- Example structure:
```python
return Div(
Card(
H4(f"{pip.fmt(step_id)}: {title}"),
Form(...),
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
```
## Step Progression Pattern
The standard step progression pattern has these key elements:
1. **GET Handler Pattern**:
```python
async def step_XX(self, request):
"""Handles GET request for Step XX."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
# Form with required fields and revert button
return Div(
Card(...),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
```
2. **POST Handler Pattern**:
```python
async def step_XX_submit(self, request):
"""Process the submission for Step XX."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
pipeline_id = db.get("pipeline_id", "unknown")
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
# Process form data
form = await request.form()
value = form.get("field_name", "")
# Validate
if not value:
return P("Error: Field is required", style=pip.get_style("error"))
# Store in state
state = pip.read_state(pipeline_id)
state[step.done] = value
pip.write_state(pipeline_id, state)
# Update progress and message
await pip.update_step_state(pipeline_id, step_id, value, steps)
await self.message_queue.add(pip, f"{step.show} complete: {value}", verbatim=True)
# Return response with chain reaction
return Div(
Card(...),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
```
## Adding or Inserting Steps
To add a new step or insert one between existing steps:
1. **Add to the steps list** in the correct position
2. **Update step indices** in the steps_indices dictionary in __init__
3. **Create both handler methods** following the patterns above
4. **Verify chain reactions** in surrounding steps
5. **Add suggestion text** in the get_suggestion method
For placeholder steps, see the [placeholder-step-pattern](mdc:.cursor/rules/placeholder-step-pattern.mdc) rule.
## State Management
- Use Pipulate helpers for state:
- `pip.read_state(pipeline_id)`
- `pip.get_step_data(pipeline_id, step_id, {})`
- `pip.update_step_state(pipeline_id, step_id, val, steps)`
- Handle finalization properly
- Clear state appropriately on revert
## Message Queue Usage
- Use ordered message delivery:
```python
await self.message_queue.add(
pip,
message,
verbatim=True,
spaces_before=0,
spaces_after=1
)
```
## Error Handling
- Validate all inputs
- Use consistent error presentation
- Provide clear user feedback
- Handle edge cases gracefully
- Example:
```python
try:
# Operation
except Exception as e:
return P(f"Error: {str(e)}", style=pip.get_style("error"))
```
## Advanced Features
- Use transform functions for step data flow
- Implement custom state preservation
- Use advanced revert controls when needed
- Track progress for long operations
- Handle file operations safely
## Documentation
- Provide clear docstrings
- Document step dependencies
- Explain state management
- Detail UI/UX considerations
- Document error handling
## Testing Considerations
- Test step progression
- Verify state management
- Check revert functionality
- Validate error handling
- Test chain reactions
## Special Cases and Edge Case Handling
1. **Terminal steps** may break the chain reaction pattern
2. **Polling operations** require special HTMX handling
3. **File downloads** use no-chain-reaction with manual continuation
4. **Error recoveries** should guide users on how to proceed
Remember: Workflows are intentionally WET (Write Everything Twice) to allow maximum customization while maintaining consistent patterns.
Reference Examples:
- [10_connect_with_botify.py](mdc:pipulate/plugins/10_connect_with_botify.py): Simple authentication workflow
- [20_hello_workflow.py](mdc:pipulate/plugins/20_hello_workflow.py): Basic workflow template
- [50_botify_export.py](mdc:pipulate/plugins/50_botify_export.py): Complex multi-step workflow
- [60_widget_examples.py](mdc:pipulate/plugins/60_widget_examples.py): Visualizing data in workflow
These rules will make it easier to consistently add and modify steps in Pipulate workflows, ensuring proper chain reactions, state management, and UI patterns are preserved.
The key insight from our implementation was recognizing the importance of maintaining the HTMX chain reaction pattern and revert functionality, while creating a minimal step that can be expanded later with specific widget functionality.
Let me create a Cursor rule that documents the pattern for placeholder steps in Pipulate workflows:
# placeholder-step-pattern
## Overview
The placeholder step pattern is used to build the structure of a workflow with minimal functionality before implementing full step logic. It's ideal for scaffolding workflows and maintaining correct progression patterns.
## Key Components of a Placeholder Step
1. **Step Definition**
```python
Step(
id='step_XX', # Unique identifier for this step
done='placeholder', # Simple state field name
show='Step Name', # User-friendly display name
refill=True, # Whether input should be preserved on revert
)
```
2. **GET Handler**
```python
async def step_XX(self, request):
"""Handles GET request for placeholder Step XX."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
placeholder_value = step_data.get(step.done, "")
# Check workflow finalization
finalize_data = pip.get_step_data(pipeline_id, "finalize", {})
if "finalized" in finalize_data and placeholder_value:
return Div(
Card(
H3(f"🔒 {step.show}"),
P("Placeholder step completed")
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
# Check completion status
if placeholder_value and state.get("_revert_target") != step_id:
return Div(
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
else:
# Show minimal UI with just a Proceed button
await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("This is a placeholder step. Click Proceed to continue to the next step."),
Form(
Button("Proceed", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}"
)
),
Div(id=next_step_id), # Note: No hx_trigger="load" here to prevent auto-progression
id=step_id
)
```
3. **POST Handler**
```python
async def step_XX_submit(self, request):
"""Process the submission for placeholder Step XX."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_XX"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
# Set a fixed completion value
placeholder_value = "completed"
# Update state and notify user
await pip.update_step_state(pipeline_id, step_id, placeholder_value, steps)
await self.message_queue.add(pip, f"{step.show} complete.", verbatim=True)
# Return with revert control and chain reaction to next step
return Div(
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
```
## Workflow Step Progression at a Glance
```
Empty Workflow Workflow With Multiple Steps
┌───────────────┐ ┌───────────────┐
│ Landing Page │ │ Landing Page │
└───────┬───────┘ └───────┬───────┘
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Form UI │ │ Form UI │
│ (pipeline_id) │ │ (pipeline_id) │
└───────┬───────┘ └───────┬───────┘
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Init Response │ │ Init Response │
│ with step_01 │ │ with step_01 │
│ div(hx_trigger│ │ div(hx_trigger│
│ ="load") │ │ ="load") │
└───────┬───────┘ └───────┬───────┘
│ │
│ Chain Reaction │ Chain Reaction
▼ ▼
┌───────────────┐ ┌───────────────┐
│ step_01 │ │ step_01 │
│ GET UI │ │ GET UI │
└───────┬───────┘ └───────┬───────┘
│ │
│ User Clicks "Proceed" │ User Clicks "Proceed"
▼ ▼
┌───────────────┐ ┌───────────────┐
│ step_01 │ │ step_01 │
│ Submit UI │ │ Submit UI │
│ revert_ctrl │ │ revert_ctrl │
│ next_step div │ │ next_step div │
│(hx_trigger= │ │(hx_trigger= │
│ "load") │ │ "load") │
└───────┬───────┘ └───────┬───────┘
│ │
│ Chain Reaction │ Chain Reaction
▼ ▼
┌───────────────┐ ┌───────────────┐
│ finalize │ │ step_02 │
│ GET │ │ GET │
└───────────────┘ └───────┬───────┘
│
│ More steps...
▼
┌───────────────┐
│ finalize │
│ GET │
└───────────────┘
```
## Critical Elements for Step Progression
- **Chain Reaction Pattern**: Using `hx_trigger="load"` to automatically load the next step
- **Proper Step Connections**: Each step must properly link to the next via the `next_step_id`
- **Finalization Flow**: All paths must eventually lead to the finalize step
- **Revert Controls**: Using `pip.revert_control()` consistently after step completion
## Benefits of Placeholder Steps
- Quickly build workflow structure with minimal implementation effort
- Test progression logic before implementing complex functionality
- Maintain consistent state management and navigation between steps
- Ensure proper revert and finalize capabilities from the beginning
## Common Pitfalls
- Missing `hx_trigger="load"` in the chain reaction div
- Forgetting to use `pip.revert_control()` for completed steps
- Not handling the finalization state properly
- Incorrect step indices in the steps_indices dictionary
## Implementation Strategy
When building workflows, consider starting with placeholder steps for all planned functionality, then incrementally replace each with full implementations while preserving the progression pattern.
This implementation plan focuses on how to apply the principles outlined in the guide when making changes, ensuring architectural integrity and adherence to existing patterns.
1. Core Patterns Recap (From Article & Codebase)
Before detailing changes, let’s recap the essential patterns documented in the article and reflected in the codebase (plugins/
files, server.py
, .cursorrules/
):
- Workflow Structure:
- Workflows are Python classes within the
plugins/
directory. - They define metadata (
APP_NAME
,DISPLAY_NAME
,ENDPOINT_MESSAGE
,TRAINING_PROMPT
). - Steps are defined sequentially in the
__init__
method using theStep = namedtuple('Step', ['id', 'done', 'show', 'refill', 'transform'])
structure. - Each step requires corresponding GET (
step_XX
) and POST (step_XX_submit
) handler methods. - Core methods (
landing
,init
,finalize
,unfinalize
,handle_revert
, etc.) provide the workflow engine and should generally not be modified significantly.
- Workflows are Python classes within the
- HTMX Chain Reaction:
- This is the core mechanism for automatic step progression.
- It relies on returning a specific
Div
structure from step handlers:# In step_XX or step_XX_submit return statement return Div( Card(...), # Current step's content or completion view # CRITICAL: This inner Div triggers the loading of the next step Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id # Outer Div ID matches the current step )
- The
hx_trigger="load"
on the innerDiv
causes HTMX to immediately fetch the content for thenext_step_id
as soon as the current step’s content is swapped into the DOM. - Breaking this chain (using
no-chain-reaction
class) should only be done for specific cases like terminal responses or polling, as documented inhtmx-chain-reactions.mdc
.
- State Management:
- Workflow state is stored per
pipeline_id
in thepipeline
table (managed viaDictLikeDB
accessed throughself.pipeline
andself.pipulate.table
). - The
pipulate
instance (self.pipulate
orpip
) provides helper methods for safe state interaction:pip.read_state(pipeline_id)
pip.get_step_data(pipeline_id, step_id, default={})
await pip.update_step_state(pipeline_id, step_id, value, steps)
(Recommended for saving primary step value)pip.write_state(pipeline_id, state)
(Use carefully for bulk updates)await pip.clear_steps_from(pipeline_id, step_id, steps)
(Used inhandle_revert
)await pip.finalize_workflow(pipeline_id)
/await pip.unfinalize_workflow(pipeline_id)
- Workflow state is stored per
- Placeholder Steps:
- As detailed in
placeholder-step-pattern.mdc
and the article, these are minimal steps used for scaffolding. - They implement the basic GET/POST handlers and maintain the chain reaction but collect no user data (usually just a “Proceed” button).
- They use a simple
done='placeholder'
state field.
- As detailed in
2. Required Changes (When Adding/Inserting Workflow Steps)
Based on the article’s guidance and existing patterns, adding or inserting a new step (let’s call it step_new
) involves these specific changes:
- Modify
__init__
:- Define the
Step
: Add a newStep(...)
definition forstep_new
into thesteps
list at the correct sequential position.steps = [ # ... existing steps before ... Step(id='step_new', done='new_field', show='New Step Label', refill=True, transform=None), # ... existing steps after ... ]
- Update Indices: Regenerate
self.steps_indices
after defining the completesteps
list to ensure correct mapping.# Add this *after* the steps list definition self.steps_indices = {step.id: i for i, step in enumerate(steps)}
- Register Routes: Ensure the route registration loop runs after the
steps
list (including the new step) is finalized. This is already the standard structure, but verify.
- Define the
- Modify Adjacent Steps’ Chain Reactions:
- Preceding Step (
step_prev
): Update theDiv
returned bystep_prev
’s GET and POST handlers. Itsid
should now bestep_new
, andhx_get
should point to/{app_name}/step_new
.# In step_prev or step_prev_submit return statement return Div( Card(...), # step_prev content # Update this Div: Div(id="step_new", hx_get=f"/{app_name}/step_new", hx_trigger="load"), id="step_prev" )
- Subsequent Step (
step_next
): No direct change needed here, but the new step must correctly point to it.
- Preceding Step (
- Create New Step Handlers:
async def step_new(self, request):
(GET Handler)- Follow the standard pattern: get
pip
,db
,steps
,app_name
,step_id
,step_index
,step
,pipeline_id
,state
,step_data
. - Calculate
next_step_id
correctly (pointing tostep_next
or'finalize'
). - Check for finalized state and return locked view if necessary.
- Check for completion (
step.done
instep_data
) and_revert_target
to decide whether to show the completed view (usingpip.widget_container
orpip.revert_control
) or the input form. - If showing the form, get
display_value
(usingstep.refill
andget_suggestion
if needed). - Add input prompt message using
self.message_queue.add(...)
. - Return the
Div
containing theCard
with the form/widget, the critical chain reactionDiv
pointing tonext_step_id
, and the outerid=step_id
.
- Follow the standard pattern: get
async def step_new_submit(self, request):
(POST Handler)- Follow the standard pattern: get context variables.
- Process
await request.form()
to getuser_val
. - Perform validation using
pip.validate_step_input
and potentially custom checks. Return error component on failure. - Perform any necessary processing on
user_val
to getprocessed_val
. - Save state using
await pip.update_step_state(pipeline_id, step_id, processed_val, steps)
. - Add completion message using
self.message_queue.add(...)
. - Check if finalize is needed (
pip.check_finalize_needed(...)
) and add appropriate message. - Return the navigation component using
pip.create_step_navigation(...)
which includes the revert control and the chain reactionDiv
for the next step.
- Update
step_messages
(Optional but Recommended):- Add entries for
step_new
inself.step_messages
for clear UI feedback.
- Add entries for
- Update
get_suggestion
(Optional):- If the new step’s input can be derived from the previous step, update
get_suggestion
accordingly, potentially using thetransform
lambda defined in theStep
.
- If the new step’s input can be derived from the previous step, update
3. Integration Strategy (Following Patterns)
Integrating new steps requires strict adherence to the established patterns documented in the article and codebase rules:
- WET Workflow Convention: Embrace the explicitness. Copy, paste, and modify the standard GET/POST handler structure for new steps. Avoid creating complex abstractions within a single workflow’s step logic. Refer to
wet-workflows.mdc
. - HTMX Chain Reaction: This is paramount. Double-check the
id
,hx_get
, andhx_trigger="load"
attributes in the returnedDiv
s of all modified and new handlers. Ensure theid
of the inner div matches thenext_step_id
, and thehx_get
path is correct. Refer tohtmx-chain-reactions.mdc
. - Pipulate Helpers: Consistently use
self.pipulate
(orpip
) methods for state (read_state
,get_step_data
,update_step_state
), UI generation (revert_control
,widget_container
,create_step_navigation
,fmt
), and validation (validate_step_input
). This ensures integration with the core engine. - Placeholder Pattern: For incremental development or complex steps, first implement a placeholder step following
placeholder-step-pattern.mdc
. This ensures the structure and connections are correct before adding complex logic or UI. - Message Queue: Use
self.message_queue.add(pip, ...)
for all user-facing status updates in the chat sidebar to ensure messages appear in the correct order, especially during asynchronous operations. - Backward Compatibility: Adhering to these patterns inherently maintains compatibility. The workflow engine relies on these structures. Deviations (like breaking the chain reaction incorrectly) are the primary risk to existing functionality. Modifying the content of steps is generally safe if the core patterns are preserved.
4. Implementation Plan (Guide for Adding a Step)
This plan outlines the recommended process for adding a new workflow step (step_new
) between step_prev
and step_next
.
- Backup: Create a copy of the workflow file (e.g.,
xx_my_flow (backup).py
) before starting. - Define Step: Open the workflow file (e.g.,
plugins/NN_my_flow.py
). Add theStep(...)
definition forstep_new
in thesteps
list within__init__
at the correct sequence. - Update Indices: Immediately after the
steps
list, ensureself.steps_indices = {step.id: i for i, step in enumerate(steps)}
is present to recalculate indices. - Update Preceding Step (
step_prev
):- Locate the
return Div(...)
statement in bothasync def step_prev(...)
andasync def step_prev_submit(...)
. - Find the inner chain reaction
Div
:Div(id=step_next.id, hx_get=f"/{app_name}/{step_next.id}", hx_trigger="load")
. - Change its
id
andhx_get
path to point tostep_new
:Div(id="step_new", hx_get=f"/{app_name}/step_new", hx_trigger="load")
.
- Locate the
- Create New GET Handler (
step_new
):- Copy the structure from an existing GET handler (like
step_prev
). - Update
step_id = "step_new"
. - Ensure
next_step_id
is calculated correctly (should point tostep_next.id
or'finalize'
). - Implement the logic for finalized, completed, and input states.
- Crucially: Ensure the returned
Div
has the outerid="step_new"
and the inner chain reactionDiv
points correctly tonext_step_id
. - Checkpoint: Run the workflow. Verify that after completing
step_prev
,step_new
’s input form loads automatically. Check the HTML source for correct IDs andhx_get
attributes.
- Copy the structure from an existing GET handler (like
- Create New POST Handler (
step_new_submit
):- Copy the structure from an existing POST handler.
- Update
step_id = "step_new"
. - Implement form processing, validation, state saving (
update_step_state
), and user messages (message_queue.add
). - Return the standard navigation component using
pip.create_step_navigation(...)
. This function correctly includes the chain reactionDiv
pointing tonext_step_id
. - Checkpoint: Run the workflow. Submit the
step_new
form. Verify the data is saved correctly (check logs/DB state). Verify the UI updates to the completed view forstep_new
and thatstep_next
loads automatically.
- Update Messages/Suggestions (Optional):
- Add entries to
self.step_messages
forstep_new
. - Update
get_suggestion
if applicable.
- Add entries to
- Testing:
- Test the full workflow sequence.
- Test reverting to
step_new
. - Test reverting from
step_new
back tostep_prev
. - Test finalizing the workflow with
step_new
included. - Test unfinalizing and reverting again.
- Test with invalid input in
step_new
.
Potential Challenges & Risks
- Broken Chain Reaction: Forgetting
hx_trigger="load"
or having incorrectid
/hx_get
values. Mitigation: Carefully check the structure of returnedDiv
s in logs or browser dev tools. - State Errors: Incorrectly saving data (e.g., wrong key in
step.done
) or failing to clear state on revert. Mitigation: Usepip.update_step_state
for primary values. Log state before/after updates for debugging. Test revert thoroughly. - Index Errors: Forgetting to update
steps_indices
after modifying thesteps
list. Mitigation: Always place the index recalculation after the finalsteps
list definition. - Syntax Errors: Preventing server restart. Mitigation: Rely on the built-in
check_syntax
and manually runpython plugins/NN_my_flow.py
if needed. - UI Glitches: Issues with client-side rendering (Mermaid, Prism, custom JS). Mitigation: Use unique IDs, employ
HX-Trigger
headers, use browser dev tools for JS debugging.
By carefully following the patterns established in the codebase and documented in the article and rules, you can reliably extend Pipulate workflows with new steps and widgets while maintaining the integrity of the system. Remember to leverage the placeholder pattern for complex additions.
AI Analysis
Title/Headline Ideas & Filenames:
- Title: Mastering Pipulate Workflows: The HTMX Chain Reaction and Placeholder Pattern Guide
Filename:
pipulate-workflow-htmx-chain-reaction-placeholder-guide.md
- Title: Guiding LLMs: Precise Patterns for Building Pipulate Workflows with HTMX
Filename:
guiding-llms-pipulate-workflow-htmx-patterns.md
- Title: Reliable Step Insertion in Pipulate: Understanding Workflow Progression and Chain Reactions
Filename:
reliable-pipulate-step-insertion-workflow-progression-chain-reactions.md
- Title: Pipulate Workflow Patterns: HTMX Chain Reactions & Placeholder Steps Explained
Filename:
pipulate-workflow-patterns-htmx-chain-reaction-placeholder-steps-explained.md
- Title: Deep Dive: Implementing Pipulate’s HTMX Chain Reaction for Step Progression
Filename:
pipulate-htmx-chain-reaction-step-progression-implementation.md
Strengths:
- Detailed Specificity: Provides in-depth technical explanations, code examples, and diagrams for the core patterns (chain reaction, placeholders).
- Practical Focus: Directly addresses a real-world challenge (guiding LLMs for specific coding tasks) and provides concrete examples (Claude interaction, sample prompts).
- Rationale Included: Explains why certain patterns (like the chain reaction div structure or WET approach) are necessary, enhancing understanding.
- Clear Structure (within sections): The guides themselves (e.g., “Guide to Adding Placeholder Steps,” the Cursor rules) are logically structured with numbered steps or clear headings.
Weaknesses:
- High Barrier to Entry: Assumes significant prior knowledge of Pipulate, HTMX, Python (async), and the specific project context, making it difficult for outsiders.
- Disjointed Format: The mix of narrative, dialogue with AI, code blocks, ASCII diagrams, and formal rule definitions can feel fragmented rather than a cohesive whole.
- Internal Dependencies: References external
.mdc
files which are not included, limiting its self-contained usefulness outside its original context. - Repetitive Code: While justified by the WET principle and the need for explicitness (especially for LLMs), the repetition of similar code structures across different steps might obscure underlying principles for a human reader seeking abstraction (in a final, polished version).
AI Opinion:
This document appears to be a highly valuable technical resource for its intended audience – likely the author, collaborators on the Pipulate project, or as training material for an AI coding assistant. Its strength lies in the detailed, explicit documentation of non-obvious but critical implementation patterns within the Pipulate framework, particularly the HTMX chain reaction. While clarity is high for someone already familiar with the context, it would be largely impenetrable to a newcomer due to jargon and assumed knowledge. Its usefulness seems high for the specific task of ensuring consistent and correct implementation of Pipulate workflows, especially when automating or assisting development with LLMs. It serves well as a technical log or pattern repository.