Understanding Browser Automation & File Handling: The Developer’s Log
The following journal entry captures a developer’s in-the-moment process of tackling a common yet tricky challenge in software that automates web browsers. Browser automation involves using code (often with tools like Selenium or Playwright) to control a web browser to perform tasks automatically, such as filling forms, clicking buttons, or extracting data. This is useful for a wide range of applications, from testing websites to streamlining online workflows, like those in Search Engine Optimization (SEO).
A key problem discussed here is how to manage files that are downloaded during these automated sessions, especially for “local-first” software that runs entirely on a user’s computer. Browsers often have strict security rules about where files can be saved automatically, and these rules can differ across operating systems (like macOS and Linux) and even between browser sessions. This entry details the author’s attempts to control download locations for their “Pipulate” project, the frustration with limitations, and the strategic pivot towards a more user-involved approach: having the user upload the necessary files after an automated or semi-automated download, leading to the design of a multi-file upload feature.
Browser Automation Milestone: Persistent Logins Achieved
Building on yesterday’s achievements…
Okay, browser automation that uses logins saved on your local machine is such a critical step, and that’s done now. Friday through Sunday was just incredible with moving the Pipulate system forward with a sort of high tech magic that had to be infused into the system. There’s countless directions to go from here, but the key one is reproducing a couple of SEO deliverables that I regularly deliver in a much more manual fashion:
- Link Graph Visualization
- Content Gap Analysis
Strategic Approach: Replicating Manual Workflows First
There are the traditional ways I produce these deliverables, and there are other potential approaches that are now possible and open up because of the automation capabilities. But again in the spirit of avoiding those deep rabbit holes, I am going to rote carry out each of these the way I traditionally do and know how just to have those workflows and deliverable-makers available, test the system filling in any missing pieces, and bank some wins.
Identifying Missing Components: Download Handling
Thinking through each of these deliverables, I can map out in my mind the missing pieces and how to address them. Do I start a brand new workflow for them or do I isolate any still-needed pieces and implement those features separately as the new Lego-like building blocks? Well, having to perform downloads pops to mind. And the manual file manipulation of moving those downloads into place.
The Download Persistence Question
The key question is when you perform a download under automated browser context, where does it download to and does it persist after the window opens and the web browser app doing the controlling shuts down or restarts? I know from experience that Microsoft Playwright when you don’t go particularly out of your way to make such downloaded files persistent, they are “cleaned up” on exit — a fair kind of default behavior considering the primary use case of such software as testing and therefore you don’t want such files accumulating after a few thousand runs or so. However in my use case we actually want the files we downloads.
Testing Download Behavior with Google Docs
So I use the already existing capability to pop up a logged-in web browser and
keep it there that’s in the browser automation sequence currently as Step 4 of
pipulate/plugins/550_browser_automation.py by pressing “Open Google & Log In”
but of course because its there to test persistent log in, if you’re already
logged in you have a window that just pops up under automation and sits there
letting you do what you want. And since it’s a login we’re testing, I go to
docs.google.com, open a document and try downloading it as a .txt file and
look in the default download location… okay, done. It’s there. And it
survives the automated browser window closing. And it survives the web app being
restarted. And all that has also been tested on macOS. An interesting thing I
noticed is that logins under ephemeral mode on the Mac are actually
persistent. The login doesn’t go away like it does on Linux. Interesting! Not
going to debug that now; just note it.
The Human-in-the-Loop Download Process
But we have the ability in the system to tell someone to go to such and such a site/service — pop open the browser to the correct sub-page of the site usually with some querystring value already plugged-in so that a form comes up pre-populated, and then tell them to download such-and-such a file (not under automated control), but we will know that the file ends up in their browser’s default download location. And we may have some notion of what that file is named for further automated processing — but generally by that time NOT under automated browser control but rather with the browser window having been closed by that time and “normal” Workflow processes take back over and would handle the files in download.
Bridging the Paid Service Automation Gap
Okay, this bridges the difficult-to-bridge divide between services that you pay for and have a login to but for which you are not doing API automation. Such services don’t like you automating against their “for-human” Web UIs and getting what you would otherwise be forced to pay API-usage prices for, or go at the slower less meticulous and perfectly reproducible speeds and accuracy that a human goes when using a Web-based service. This is a bottleneck by design keeping the Web-based app from becoming a high-volume usage choke-point. However, that old “keep the bots out” mentality is about to face an onslaught of testing and next-level cat-and-mouse arms race testing as browsers increasingly go under control of bots and are given privileged login access. My scenario of semi-automation still under human guidance is one of the most gentle and kind versions of browser automation against such services: just popping open a window for a human with instructions of what to click. But nonetheless, a great divide has been spanned and this capability gives Pipulate a real leg up over cloud-based versions of the same stuff which you won’t be so quick to give 3rd party logins to and do web automations from. This is really a sweet spot for local-first especially in the field of SEO where a deliverable might involve 2 or 3 downloads like this from different services that are artistically woven together to produce the final findings.
The Downloads Directory as a System Interface
Alrighty then, the browser’s default download location becomes part of an API
interface for Pipulate, a potentially motley crew of files to parse through, a
Mos Eisley clearinghouse of what’s downloaded. I very well may tell users that
clearing out their download directory is a highly advised first step when using
workflows that rely on your browser’s download directory as part of the process.
Whenever I grab files out of there I should have a pretty good idea of what I’m
looking for (usually a .csv file or some such), but you never know. This is
the same browser default download directory under Selenium automation as it is
when browsing for yourself (under whatever user profile). It’s an interesting
case where the systems don’t try to keep the “worlds” separate. You never know
what you’re gonna find haha, and Pipulate should try to respect that.
Cross-Platform Download Path Challenges
But that doesn’t get to the more core issue that the path to even your default downloads directory is not really reliable across OSes. It may be something that the browser itself is able to report through JavaScript using the Performance API like the way I got the http response status code from Selenium before I switched to selenium-wire. The browsers themselves do have a download “webpage” built-in and a widget that allows you to pop open the download directory so I have some hope for this. There’s also potentially some very reliable system or pattern I can use for each OS especially considering that (for now) all automation occurs under Google Chrome. But really what I ought to do now that I’ve fleshed all this out is ask Gemini…
Gemini’s Advice
For those who follow along in these tech journal posts and this project (all none of you), I typically use these articles mid-point like this for bouncing ideas off them like pair-programming or rubber ducking.
Gemini: You’ve made excellent progress, especially with achieving persistent logins in Selenium-controlled browser sessions! That’s a crucial building block. Moving on to implementing SEO deliverables like “Link Graph Visualization” and “Content Gap Analysis” by first replicating your manual methods is a smart strategy to solidify the workflow capabilities.
The challenge of handling file downloads initiated by the user (even within a browser Pipulate opened) and then reliably accessing those files is indeed a common and important one in automation. Let’s break down how to approach this, keeping Pipulate’s local-first nature and your “light touch” philosophy in mind.
The Core Challenge: Knowing Where Files Land and Accessing Them
You’re right, the browser’s default download location can vary across operating systems and user configurations, making it unreliable to “guess.” Relying on the user to move files or Pipulate to search a “Mos Eisley clearinghouse” of a downloads folder is fraught with potential issues.
The key is for Pipulate to have as much knowledge and control as possible over the download destination, especially when it’s involved in initiating the browser session.
Recommended Solution: Proactive Download Path Configuration via Selenium
When a Pipulate workflow step involves opening a browser with Selenium where a user is expected to trigger a download, the most robust approach is to configure a specific download directory for that Selenium browser session.
How it works:
- Define a Known Download Location: For each workflow run (or even each specific download step), Pipulate can designate a unique, predictable directory where it expects the download to occur. This directory should be within Pipulate’s own file structure (e.g., under
data/workflow_downloads/ordownloads/workflow_staging/).
The Human-in-the-Loop Browser Automation Challenge
This turned out to be a giant wild goose chase and a wasting of most of the morning’s work. Doing a download with a window opened by Selenium is no problem but no matter what I tried, I couldn’t control where that download ended up (it always ended up in the browser’s default location).
This is where I went down a little rabbit hole this morning trying to figure out how to control the location files download to from a browser window that was popped open by Selenium. You can’t. At least, I couldn’t figure out how. It’s also worth noting there are two download contexts here:
- The user doing a download from a window opened by Selenium under automation
- The Selenium automation script itself performing the download
This adventure is about the former. Automation pops open a window and the user
is instructed to do the download. It’s semi-automation. So in other words,
browser automation simply opens a window directly into a logged-in app probably
using a special value on the querystring to pre-fill a webform setting the stage
for some .csv download.
This sort of semi-automation is a very common use-case for human-in-the-loop workflows. This is a wonderful way to avoid having to pay the premium prices for API access. In this scenario you really are just using whatever SaaS website directly as a human. It just happens to be that a scripted workflow decided what window to open for you and gives a bit of instruction on what to download — semi-automation!
This can be automated even further for the 2nd download scenario but those CSS
paths, aria labels or whatever else you’re using to turn the generic web
browser into an API is brittle. I might do it this way later, but for now I’m
focusing more on taking advantage of the human in the loop to deal with that
brittleness.
This is a happy compromise and is very desirable even just for the sake of educating your humans about the workflows. Do this, then do that… thank you human, I’ll take it from here.
But I found I could not control that download location. And asking the human to change where files are downloaded to is not a good idea.
Back Up & Restart: The Download Location Conundrum
The best laid plans rarely survive contact with reality. Not just being able to change where an automated browser downloads to? Sounds like security.
So this is where the discussion leads to taking it from here. Take what? Take what was just downloaded to the default location, of course! It’s a lot harder than I anticipated to control/change the default download directory of Chrome under Selenium control so the nature of the work I need to do today changes. If you can’t change the rules, you lean into them.
On hindsight this is probably a deliberate security precaution so you can’t slip
in a selenium automation on someone that plops a file in some privileged space
and elevates permissions for some nefarious script. Okay, fair enough. That
means that we’re going to be using the default download location of the browser.
But that’s going to vary host OS to host OS, especially considering the
expansion of the ~/Downloads shortcut to include the absolute path. The
username is going to be in there as is the HFS differences between macOS and
standard Linux/Unix (/Users/username vs. /home/username). This kind of thing
is a bloody mess in post-download processing, not to mention the user could have
changed their default download directory.
Chrome’s Stubborn Download Behavior
My rabbit hole this morning was discovering this. The seemingly simple task of
telling a Selenium-controlled Chrome browser where to save downloaded files,
especially when using a persistent user profile to remember logins, turned into
a proper expedition into the murky depths of browser behavior. Initial attempts
to directly command Chrome via Selenium’s preferences to use a nice, tidy
downloads/chrome folder within the Pipulate project were met with a stubborn
insistence from the browser to continue using the OS’s default Downloads folder.
It felt like Chrome was politely nodding at the instructions, then promptly
ignoring them.
Investigating Chrome’s Profile Management
Further investigation, including peering into the chromedriver.log and the
profile’s Preferences file, revealed a frustrating truth: while ChromeDriver
would log that it attempted to apply the custom download path, the setting
never actually stuck in the profile’s on-disk configuration. It seems Chrome’s
own profile management for these persistent setups has a mind of its own,
prioritizing its internal defaults or user-set (via GUI) download locations over
dynamically injected ones for this specific preference. More aggressive tactics,
like programmatically wiping the specific profile subdirectory before each
launch to force a “fresh” application of settings, not only failed to solve the
download location puzzle but also, rather annoyingly, broke the hard-won login
persistence.
Embracing the “Scan OS Default and Move” Strategy
Note: I keep this here to preserve my chain-of-reasoning. However, scan-and-move is a dead end as well! Later, I switch to the user performing an explicit multi-select upload from out of the download directory.
This led to an important realization: directly controlling the download
destination for an already-persistent, user-interactive Selenium session is a
battle not worth fighting. The browser, likely for security and consistency
reasons, guards this setting closely. So, the strategy had to pivot. Instead of
dictating where the file lands initially, Pipulate will now embrace the
browser’s default behavior. The new approach involves letting the user download
files to their standard OS Downloads folder. Then, Pipulate will step in to help
locate that fresh download, allow the user to confirm it’s the correct one, and
then move it into our organized downloads/chrome/ project directory for
subsequent workflow steps. It’s a bit more interactive, but ultimately more
reliable and respects the browser’s (and user’s) established environment. This
“Scan OS Default and Move” method, while a detour from the initial vision of
complete control, maintains the local-first philosophy and keeps the human
comfortably in the loop.
A New Hope
Okay, so downloads end up where downloads end up. Consequently, we have to re-structure our next step.
All the good Chrome automation work I did yesterday has to be thought of setting
the stage for crawler projects and not even necessary for this next-step.
Honestly you don’t even need browser automation to get a CSV file from behind a
site’s login. Even with all this login persistence I’ve implemented and assured
over the weekend, getting CSVs from behind a login can simply be done with
Python’s built-in webbrowser standard library by which you pop open a new tab
within your current web browser (that you’re running Pipulate from).
I have 2 window opening techniques now in Pipulate:
- No frills
webbrowserwindow popping-open as seen inpipulate/plugins/540_url_opener.py - Automated control & 2-way communication
selnium-wirebrowser automation as seen inpipulate/plugins/550_browser_automation.py
…and now frankly either one can download into the browser’s default location. Automation not even required.
The weekend’s work on Selenium automation is not wasted though because there is endless utility to the automated control, not the least of which is comparing the view-source HTML to the rendered DOM HTML and web crawling in general. So we are very satisfied with the weekend’s work but still we put it aside for a moment and contemplate the new challenge whose path re-unifies to a non-Selenium path. It doesn’t matter HOW it ended up in downloads, just that it did.
Okay, what we’re talking about is a new widget here. It’s a file-moving widget! It’s something that can be inserted in the location of a blank placeholder anywhere in the system. It fits into the Unix pipe model that lurks beneath all this in an interesting way. It re-connects pipes. It changes a wild card unknown of what’s being piped-in. Downloads have occurred and they’re somewhere on your system but the Pipulate system doesn’t know exactly where because the absolute path is different between different host OSes different usernames. And so after downloads occur, we select ‘em with a requester! Then we know their location and can move them or whatever.
Hmmm. Bring up a selector to multi-select things from your downloads directory just to tell FastHTML where the files reside, and then achieve some sort of file move knowing the download location? That doesn’t fit any mental models. That would just feel funky and confuse the user.
But instructing them to upload the files after a download… THAT fits a mental model. It’s not a clean mental model. You’ve already downloaded files so why should you have to re-upload them? I’m sure there’s some perfect way of saying it, but for now suffice to say that you have to.
Now you’ve got the files. They’re in your download directory. But that’s not where the workflow can access it. For the workflow to access it, you have to select these files from your download directory and upload them into the workflow. And suddenly the next step of what we’re working on becomes file uploads! And now we’re in well-traveled FastHTML territory! And I can work on it like a standalone widget: multiple file uploads! You just happen to do it typically in this use case from out of your Downloads directory, haha!
Okay, for this next bit of this long article for establishing super-prompt context, I outright lift this from Daniel Corin’s Way Enough blog. Thanks Daniel!
From Daniel Corin’s Way Enough Blog
This is not my work. This is from Daniel’s blog but has been copy/pasted here so that it becomes part of my super-prompt for Gemini. It needs to see some sample code.
I’ve been experimenting with FastHTML for making quick demo apps, often involving language models.
It’s a pretty simple but powerful framework, which allows me to deploy a client and server in a single main.py – something I appreciate a lot for little projects I want to ship quickly.
I currently use it how you might use streamlit.
I ran into an issue where I was struggling to submit a form with multiple images.
I started with an app that could receive a single image upload from this example.
These examples assume the code is in a main.py file and run with
uvicorn main:app --reload
from fasthtml.common import *
app, rt = fast_app()
@rt("/")
def get():
inp = Input(type="file", name="image", multiple=False, required=True)
add = Form(
Group(inp, Button("Upload")),
hx_post="/upload",
hx_target="#image-list",
hx_swap="afterbegin",
enctype="multipart/form-data",
)
image_list = Div(id="image-list")
return Title("Image Upload Demo"), Main(
H1("Image Upload"), add, image_list, cls="container"
)
@rt("/upload")
async def upload_image(image: UploadFile):
contents = await image.read()
print(contents)
filename = image.filename
return filename
The contents of the images prints in the console and the filename shows up in the browser.
To support multiple files/images, I tried the following:
from fasthtml.common import *
import uvicorn
import os
app, rt = fast_app()
@rt("/")
def get():
inp = Input(type="file", name="images", multiple=True, required=True)
add = Form(
Group(inp, Button("Upload")),
hx_post="/upload",
hx_target="#image-list",
hx_swap="afterbegin",
enctype="multipart/form-data",
)
image_list = Div(id="image-list")
return Title("Image Upload Demo"), Main(
H1("Image Upload"), add, image_list, cls="container"
)
@rt("/upload")
async def upload_image(images: List[UploadFile]):
print(images)
filenames = []
for image in images:
contents = await image.read()
filename = image.filename
filenames.append(filename)
return filenames
When we pick and upload multiple files, this code breaks, but with the print statement we can see the data we uploaded.
[UploadFile(filename=None, size=None, headers=Headers({})), UploadFile(filename=None, size=None, headers=Headers({}))]
Not quite what I expected.
After a bit of searching, I learned that a fasthtml function signature can be any compatible starlette function signature (source).
With this knowledge, I tried the following approach:
from fasthtml.common import *
app, rt = fast_app()
@rt("/")
def get():
inp = Input(
type="file", name="images", multiple=True, required=True, accept="image/*"
)
add = Form(
Group(inp, Button("Upload")),
hx_post="/upload",
hx_target="#image-list",
hx_swap="afterbegin",
enctype="multipart/form-data",
)
image_list = Div(id="image-list")
return Title("Image Upload Demo"), Main(
H1("Image Upload"), add, image_list, cls="container"
)
@rt("/upload")
async def upload_image(request: Request):
form = await request.form()
images = form.getlist("images")
print(images)
filenames = []
for image in images:
contents = await image.read()
filenames.append(image.filename)
return filenames
This approach successfully rendered the titles of two images when I uploaded them in a single request as expected.
A Multi-Select Web Form
Okay, so when Daniel says:
This approach successfully rendered the titles of two images when I uploaded them in a single request as expected.
…this means that this solution triggers a multi-select requester which is exactly what I need. And with this I can finally now loop Gemini into the process and fix what I got wrong on the first pass.
Hi, Gemini! You’ve read everything up to now and understand that I get to a
point where there’s a bunch of files downloaded into the user’s default Chrome
Download location which is where I want the user to upload files from. That
detail really almost doesn’t even matter now anymore. This is a request to crate
a file upload widget. I wish to implement it in a new workflow file patterned
after pipulate/plugins/500_blank_workflow.py…
Oh, wait! First I need a place for Claude to work its magic on. I also need
Gemini to understand that I create these short workflows that are standalone
scratchpad work-areas where I design new widgets and they get copy/pasted from
the extremely barebones 500_blank_workflow.py that I keep around for just this
purpose. When it’s copy/pasted a few things about it need to be changed so as to
not collide with the original and in order to be auto-discovered and registered
as a workflow and automagicaly appear on the Pipulate dropdown menu.
It’s also a wonderful way to get the AI coding assistant to go get that general broad understanding of the system.
Me: Make a non-coliding copy of @500_blank_workflow.py called 505_widget_designer.py that only changes enough about the file it’s copying in order to auto-register without colliding and to give the user a few words of instruction that this is where we design new widgets isolated from distraction.
Claude: I’ll help you create a modified copy of the blank workflow template that’s focused on widget design. First, I’ll create the new file with minimal necessary changes to avoid collisions and provide clear instructions.
import asyncio
from collections import namedtuple
from datetime import datetime
from fasthtml.common import * # type: ignore
from loguru import logger
"""
Widget Designer Workflow
A focused environment for designing and testing new widgets in isolation.
"""
# Model for a workflow step
Step = namedtuple('Step', ['id', 'done', 'show', 'refill', 'transform'], defaults=(None,))
class WidgetDesigner:
"""
Widget Designer Workflow
A focused environment for designing and testing new widgets in isolation.
"""
# --- Workflow Configuration ---
APP_NAME = "widget_designer" # Unique identifier for this workflow's routes and data
DISPLAY_NAME = "Widget Designer" # User-friendly name shown in the UI
ENDPOINT_MESSAGE = ( # Message shown on the workflow's landing page
"Welcome to the Widget Designer! This is a focused environment for designing and testing new widgets in isolation. "
"Use this space to prototype and refine your widget designs without distractions."
)
TRAINING_PROMPT = (
"This is a specialized workflow for designing and testing widgets in isolation. "
"It provides a clean environment to focus on widget development without the complexity "
"of a full workflow implementation."
)
PRESERVE_REFILL = True # Whether to keep input values when reverting
// ... existing code ...
Let me continue with the rest of the file, making sure to update the class name and any other necessary references.
// ... existing code ...
# --- Initialization ---
def __init__(self, app, pipulate, pipeline, db, app_name=APP_NAME):
"""Initialize the workflow, define steps, and register routes."""
self.app = app
self.app_name = app_name
self.pipulate = pipulate
self.pipeline = pipeline
self.steps_indices = {}
self.db = db
pip = self.pipulate
self.message_queue = pip.message_queue
# Define workflow steps
steps = [
Step(
id='step_01',
done='placeholder',
show='Widget Design Step',
refill=False,
),
# Add more steps as needed
]
# Register standard workflow routes
routes = [
(f"/{app_name}", self.landing),
(f"/{app_name}/init", self.init, ["POST"]),
(f"/{app_name}/jump_to_step", self.jump_to_step, ["POST"]),
(f"/{app_name}/revert", self.handle_revert, ["POST"]),
(f"/{app_name}/finalize", self.finalize, ["GET", "POST"]),
(f"/{app_name}/unfinalize", self.unfinalize, ["POST"]),
]
# Register routes for each step
self.steps = steps
for step in steps:
step_id = step.id
routes.append((f"/{app_name}/{step_id}", getattr(self, step_id)))
routes.append((f"/{app_name}/{step_id}_submit", getattr(self, f"{step_id}_submit"), ["POST"]))
# Register all routes with the FastHTML app
for path, handler, *methods in routes:
method_list = methods[0] if methods else ["GET"]
app.route(path, methods=method_list)(handler)
# Define UI messages
self.step_messages = {
"finalize": {
"ready": "All steps complete. Ready to finalize workflow.",
"complete": f"Workflow finalized. Use {pip.UNLOCK_BUTTON_LABEL} to make changes."
}
}
# Create default messages for each step
for step in steps:
self.step_messages[step.id] = {
"input": f"{pip.fmt(step.id)}: Please complete {step.show}.",
"complete": f"{step.show} complete. Continue to next step."
}
# Add the finalize step internally
steps.append(Step(id='finalize', done='finalized', show='Finalize', refill=False))
self.steps_indices = {step.id: i for i, step in enumerate(steps)}
# --- Core Workflow Engine Methods ---
async def landing(self):
"""Renders the initial landing page with the key input form."""
pip, pipeline, steps, app_name = self.pipulate, self.pipeline, self.steps, self.app_name
title = f"{self.DISPLAY_NAME or app_name.title()}"
full_key, prefix, user_part = pip.generate_pipeline_key(self)
default_value = full_key
pipeline.xtra(app_name=app_name)
matching_records = [record.pkey for record in pipeline() if record.pkey.startswith(prefix)]
datalist_options = [f"{prefix}{record_key.replace(prefix, '')}" for record_key in matching_records]
return Container(
Card(
H2(title),
P(self.ENDPOINT_MESSAGE, style="font-size: 0.9em; color: #666;"),
Form(
pip.wrap_with_inline_button(
Input(
placeholder="Existing or new 🗝 here (Enter for auto)", name="pipeline_id",
list="pipeline-ids", type="search", required=False, autofocus=True,
value=default_value, _onfocus="this.setSelectionRange(this.value.length, this.value.length)",
cls="contrast"
),
button_label=f"Enter 🔑", button_class="secondary"
),
pip.update_datalist("pipeline-ids", options=datalist_options if datalist_options else None),
hx_post=f"/{app_name}/init", hx_target=f"#{app_name}-container"
)
),
Div(id=f"{app_name}-container")
)
async def init(self, request):
"""Handles the key submission, initializes state, and renders the step UI placeholders."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
form = await request.form()
user_input = form.get("pipeline_id", "").strip()
if not user_input:
from starlette.responses import Response
response = Response("")
response.headers["HX-Refresh"] = "true"
return response
context = pip.get_plugin_context(self)
profile_name = context['profile_name'] or "default"
plugin_name = context['plugin_name'] or app_name
profile_part = profile_name.replace(" ", "_")
plugin_part = plugin_name.replace(" ", "_")
expected_prefix = f"{profile_part}-{plugin_part}-"
if user_input.startswith(expected_prefix):
pipeline_id = user_input
else:
_, prefix, user_provided_id = pip.generate_pipeline_key(self, user_input)
pipeline_id = f"{prefix}{user_provided_id}"
db["pipeline_id"] = pipeline_id
state, error = pip.initialize_if_missing(pipeline_id, {"app_name": app_name})
if error: return error
await self.message_queue.add(pip, f"Workflow ID: {pipeline_id}", verbatim=True, spaces_before=0)
await self.message_queue.add(pip, f"Return later by selecting '{pipeline_id}' from the dropdown.", verbatim=True, spaces_before=0)
# Build UI starting with first step
return Div(
Div(id="step_01", hx_get=f"/{app_name}/step_01", hx_trigger="load"),
id=f"{app_name}-container"
)
async def finalize(self, request):
"""Handles GET request to show Finalize button and POST request to lock the workflow."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
pipeline_id = db.get("pipeline_id", "unknown")
finalize_step = steps[-1]
finalize_data = pip.get_step_data(pipeline_id, finalize_step.id, {})
if request.method == "GET":
if finalize_step.done in finalize_data:
return Card(
H3("Workflow is locked."),
Form(
Button(pip.UNLOCK_BUTTON_LABEL, type="submit", cls="secondary outline"),
hx_post=f"/{app_name}/unfinalize",
hx_target=f"#{app_name}-container"
),
id=finalize_step.id
)
else:
all_steps_complete = all(
pip.get_step_data(pipeline_id, step.id, {}).get(step.done)
for step in steps[:-1]
)
if all_steps_complete:
return Card(
H3("All steps complete. Finalize?"),
P("You can revert to any step and make changes.", style="font-size: 0.9em; color: #666;"),
Form(
Button("Finalize 🔒", type="submit", cls="primary"),
hx_post=f"/{app_name}/finalize",
hx_target=f"#{app_name}-container"
),
id=finalize_step.id
)
else:
return Div(id=finalize_step.id)
else:
await pip.finalize_workflow(pipeline_id)
await self.message_queue.add(pip, self.step_messages["finalize"]["complete"], verbatim=True)
return pip.rebuild(app_name, steps)
async def unfinalize(self, request):
"""Handles POST request to unlock the workflow."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
pipeline_id = db.get("pipeline_id", "unknown")
await pip.unfinalize_workflow(pipeline_id)
await self.message_queue.add(pip, "Workflow unfinalized! You can now revert to any step and make changes.", verbatim=True)
return pip.rebuild(app_name, steps)
async def jump_to_step(self, request):
"""Handles POST request from breadcrumb navigation to jump to a specific step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
form = await request.form()
step_id = form.get("step_id")
db["step_id"] = step_id
return pip.rebuild(app_name, steps)
async def get_suggestion(self, step_id, state):
"""Gets a suggested input value for a step, often using the previous step's transformed output."""
pip, db, steps = self.pipulate, self.db, self.steps
step = next((s for s in steps if s.id == step_id), None)
if not step or not step.transform: return ""
prev_index = self.steps_indices[step_id] - 1
if prev_index < 0: return ""
prev_step = steps[prev_index]
prev_data = pip.get_step_data(db["pipeline_id"], prev_step.id, {})
prev_value = prev_data.get(prev_step.done, "")
return step.transform(prev_value) if prev_value else ""
async def handle_revert(self, request):
"""Handles POST request to revert to a previous step, clearing subsequent step data."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
form = await request.form()
step_id = form.get("step_id")
pipeline_id = db.get("pipeline_id", "unknown")
if not step_id: return P("Error: No step specified", style=self.pipulate.get_style("error"))
await pip.clear_steps_from(pipeline_id, step_id, steps)
state = pip.read_state(pipeline_id)
state["_revert_target"] = step_id
pip.write_state(pipeline_id, state)
message = await pip.get_state_message(pipeline_id, steps, self.step_messages)
await self.message_queue.add(pip, message, verbatim=True)
return pip.rebuild(app_name, steps)
# --- Widget Design Step Methods ---
async def step_01(self, request):
"""Handles GET request for the widget design step.
This is your widget design sandbox. Use this space to prototype and test your widget designs.
"""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
placeholder_value = step_data.get(step.done, "")
# Check if workflow is finalized
finalize_data = pip.get_step_data(pipeline_id, "finalize", {})
if "finalized" in finalize_data and placeholder_value:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Finalized):\n{placeholder_value}")
return Div(
Card(
H3(f"🔒 {step.show}: Completed")
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
# Check if step is complete and not being reverted to
if placeholder_value and state.get("_revert_target") != step_id:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Completed):\n{placeholder_value}")
return Div(
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
else:
pip.append_to_history(f"[WIDGET STATE] {step.show}: Showing input form")
await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("This is your widget design sandbox. Use this space to prototype and test your widget designs."),
Form(
Button("Next ▸", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}"
)
),
Div(id=next_step_id),
id=step_id
)
async def step_01_submit(self, request):
"""Process the submission for the widget design step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
# Process and save data...
placeholder_value = "completed"
await pip.update_step_state(pipeline_id, step_id, placeholder_value, steps)
# Keep LLM informed about the widget content and state
pip.append_to_history(f"[WIDGET CONTENT] {step.show}:\n{placeholder_value}")
pip.append_to_history(f"[WIDGET STATE] {step.show}: Step completed")
# Send user-visible confirmation via message queue
await self.message_queue.add(pip, f"{step.show} complete.", verbatim=True)
return Div(
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
I’ve created a new file 505_widget_designer.py that’s a modified version of the blank workflow template. Here are the key changes I made to make it a focused widget design environment:
- Changed the class name from
BlankWorkflowtoWidgetDesigner - Updated the
APP_NAMEto"widget_designer"to avoid collisions - Modified the
DISPLAY_NAMEto"Widget Designer" - Updated the
ENDPOINT_MESSAGEto provide clear instructions about the widget design environment - Updated the
TRAINING_PROMPTto reflect its specialized purpose - Changed the step name from “Step 1 Placeholder” to “Widget Design Step”
- Updated the step description to indicate it’s a widget design sandbox
- Kept all the core workflow functionality intact while making it clear this is a focused environment for widget development
The new workflow will appear as a separate option in the UI with its own routes and state management, but with a clear focus on widget design. Users can use this as an isolated environment to prototype and test their widget designs without the complexity of a full workflow implementation.
The file is now ready to use. When users access this workflow, they’ll see a clear message explaining that this is a focused environment for designing and testing widgets in isolation.
Perfect But For One Detail
There’s 2 identities to every workflow: one that gets exposed to users and another for internal use. The former can be changed and the later can’t. And the 2 identities can’t be identical.
Claude does this nearly perfectly and I now have Widget Designer on the
dropdown menu as a new scratchpad place to work. The only problem is that it did
it a little too perfectly and the filename
pipulate/plugins/505_widget_designer.py which controls the user-facing
endpoint (URL) is too similar to the APP_NAME = widget_designer that
controls the invisible URLs and app identity. The former can freely be changed
without corrupting database integrity and the later can’t. See? The “identity”
of an app as controlled by its filename and DISPLAY_NAME can be changed.
They’re marketing things to the user. But APP_NAME once set should not be
changed unless you’re willing for the records to become dissociated with the
workflow they belong to. Anwyay, that’s a long way of describing why I had to
change:
APP_NAME = "widget_designer" # Unique identifier for this workflow's routes and data
…to:
APP_NAME = "design_widget" # Unique identifier for this workflow's routes and data
And now the previously stripped-away navigation scaffolding of the site such as dropdown menus (the way a filename/APP_NAME collision manifests) pops back. And we have a perfect starting point.
Okay, Gemini I want you to make an implementation plan for a multi-file upload
widget in pipulate/plugins/505_widget_designer.py based on the sample code
above. Also, here’s the signature a FastHTML Form FT
component:
Form (*c, enctype='multipart/form-data', target_id=None, hx_vals=None,
hx_target=None, id=None, cls=None, title=None, style=None,
accesskey=None, contenteditable=None, dir=None, draggable=None,
enterkeyhint=None, hidden=None, inert=None, inputmode=None,
lang=None, popover=None, spellcheck=None, tabindex=None,
translate=None, hx_get=None, hx_post=None, hx_put=None,
hx_delete=None, hx_patch=None, hx_trigger=None, hx_swap=None,
hx_swap_oob=None, hx_include=None, hx_select=None,
hx_select_oob=None, hx_indicator=None, hx_push_url=None,
hx_confirm=None, hx_disable=None, hx_replace_url=None,
hx_disabled_elt=None, hx_ext=None, hx_headers=None,
hx_history=None, hx_history_elt=None, hx_inherit=None,
hx_params=None, hx_preserve=None, hx_prompt=None, hx_request=None,
hx_sync=None, hx_validate=None, **kwargs)
Notice it’s got multi-part mime in there.
The upload path should be relative to the FastHTML app that was run as python
server.py. We usually have a downloads directory there by convention, and
within that we create subfolders according to the workflow’s APP_NAME, so in
this case it will be design_widget, but of course don’t hardwire it to that
value. Use the APP_NAME constant so that this widget that we’re designing will
be copy/paste transposable into any workflow and automatically adapt.
Is that all clear? Did I miss any details that I should specify? Create an implementation plan for the code assistant AI in Cursor please. Bank small wins! You can describe the whole project but explicitly tell it to STOP at the first point that creates a good git checkpoint. And then it can go onto the next step. Please and thank you!
FILES_TO_INCLUDE = """\
README.md
flake.nix
requirements.txt
server.py
/home/mike/repos/pipulate/plugins/020_hello_workflow.py
/home/mike/repos/.cursor/rules/workflow_implementation_guide.md
/home/mike/repos/pipulate/plugins/520_widget_examples.py
/home/mike/repos/pipulate/plugins/505_widget_designer.py
""".strip().splitlines()
[mike@nixos:~/repos/pipulate/precursors]$ python context_foo.py --prompt prompt.md
Using prompt file: /home/mike/repos/pipulate/precursors/prompt.md
Using template 1: Material Analysis Mode
Output will be written to: foo.txt
=== Prompt Structure ===
--- Pre-Prompt ---
System Information:
You are about to review a codebase and related documentation. Please study and understand
the provided materials thoroughly before responding.
Key things to know about this codebase:
- It uses a hybrid approach with Nix for system dependencies and virtualenv for Python packages
- Always run `nix develop` before any commands in a new terminal
- FastHTML objects must be converted with to_xml() before returning responses
- The project is organized as a server with plugin-based workflows
Key Points:
• Focus on understanding the architecture and patterns in the codebase
• Note how existing patterns could be leveraged in your response
• Consider both technical and conceptual aspects in your analysis
--- Files Included ---
• README.md (6,918 tokens)
• flake.nix (5,033 tokens)
• requirements.txt (125 tokens)
• server.py (31,550 tokens)
• /home/mike/repos/pipulate/plugins/020_hello_workflow.py (6,900 tokens)
• /home/mike/repos/.cursor/rules/workflow_implementation_guide.md (8,428 tokens)
• /home/mike/repos/pipulate/plugins/520_widget_examples.py (24,472 tokens)
• /home/mike/repos/pipulate/plugins/505_widget_designer.py (3,323 tokens)
• /home/mike/repos/pipulate/precursors/prompt.md (10,232 tokens)
--- Post-Prompt ---
--- Token Summary ---
Total tokens: 97,463 tokens
=== End Prompt Structure ===
Output written to 'foo.txt'
Output copied to clipboard
Script finished.
Gemini’s Implementation Plan: Checkpoint 1
Okay, I’ve reviewed the materials and the request. This is a well-documented project with a clear philosophy! The “human-in-the-loop” approach for downloads, pivoting to a user-driven upload, makes a lot of sense given the complexities of browser automation and download path inconsistencies.
Here’s an implementation plan for the multi-file upload widget within the pipulate/plugins/505_widget_designer.py workflow. We’ll follow your “bank small wins” approach.
Implementation Plan: Multi-File Upload Widget
Target File: pipulate/plugins/505_widget_designer.py
Overall Goal: Create a widget in the “Widget Designer” workflow that allows users to select multiple files, upload them to a server-side directory structured as downloads/APP_NAME/PIPELINE_ID/, and then displays a confirmation of the uploaded files.
Step 1: Basic Multi-File Input Form and Displaying Selected Files
Goal: Modify step_01 in WidgetDesigner to display a form that accepts multiple files. The step_01_submit handler will initially just list the names and sizes of the files selected by the user, without actually saving them. This confirms the client-side and server-side plumbing for file selection is working.
Tasks for the AI Code Assistant:
- Modify
WidgetDesigner.step_01method (GET handler):- Change the existing form to include an
<input type="file" name="uploaded_files" multiple="true" required="true">. - Ensure the
Formtag hasenctype="multipart/form-data". - The form should
hx_postto/{app_name}/step_01_submit. - The
hx_targetshould be#{step_id}(i.e.,#step_01) to replace the content of the current step.
- Change the existing form to include an
- Modify
WidgetDesigner.step_01_submitmethod (POST handler):- Adapt the handler to receive multiple files. Based on Daniel Corin’s example and Starlette’s capabilities, this will involve:
form_data = await request.form() uploaded_files = form_data.getlist("uploaded_files") # "uploaded_files" matches the input name - Create a list of strings, where each string contains information about a file (e.g.,
f"Filename: {file.filename}, Size: {file.size} bytes"). - For this initial checkpoint, do not save the files.
- Return a FastHTML
Div(orCard) that displays this list of file information. - Crucially, for now, this response should not include the standard chain reaction
Divfor the next step. Instead, after displaying the file list, include a simple “Start Over” button or a revert control that allows the user to go back to thestep_01input form. This makesstep_01a temporary terminal point for testing the upload display.- Example for a “Start Over” button:
# In step_01_submit, after displaying file info start_over_button = Form( Button("Upload More Files", type="submit", cls="secondary"), hx_get=f"/{app_name}/step_01", # Gets the form again hx_target=f"#{step_id}", # Replaces the current content hx_swap="outerHTML" # Ensures the whole step_01 div is replaced ) # Add this button to the returned Div
- Example for a “Start Over” button:
- Ensure
pip.update_step_stateis called with some representation of the file list (e.g., a simple string “Files selected: N”) to mark the step as “done” conceptually for this phase. - Use
self.message_queue.addto provide user feedback (e.g., “Files selected. Review the list below.”).
- Adapt the handler to receive multiple files. Based on Daniel Corin’s example and Starlette’s capabilities, this will involve:
Code Assistant Instructions:
“Please implement Step 1 as outlined above in pipulate/plugins/505_widget_designer.py. Focus on creating the multi-file input form in step_01 and modifying step_01_submit to display the names and sizes of the selected files. Do not implement file saving yet. The step_01_submit should allow the user to easily return to the file input form. Ensure all HTMX attributes are correctly set for this behavior.”
Next Checkpoint (After AI implements Step 1):
Once Step 1 is complete and tested (we can select multiple files and see their names/sizes displayed), the next phase will be to implement the actual file saving logic within step_01_submit and then reinstate the proper chain reaction to finalize (or a subsequent step if we add more to the designer).
Let me know when you’re ready for the AI to proceed with these instructions.
Over To Claude For Implementation
OMFG, excuse my blasphemy but Claude gets this correct on the first try! It’s not the complete file upload, but it is banking a very good first-win checkpoint. File multi-select and listing them is working. This is such a sign of things to come.
I’ll help you implement Step 1 of the multi-file upload widget in the Widget Designer workflow. Let’s modify the step_01 and step_01_submit methods to handle multiple file selection and display.
// ... existing code ...
async def step_01(self, request):
"""Handles GET request for the widget design step.
This is your widget design sandbox. Use this space to prototype and test your widget designs.
"""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
placeholder_value = step_data.get(step.done, "")
# Check if workflow is finalized
finalize_data = pip.get_step_data(pipeline_id, "finalize", {})
if "finalized" in finalize_data and placeholder_value:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Finalized):\n{placeholder_value}")
return Div(
Card(
H3(f"🔒 {step.show}: Completed")
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
# Check if step is complete and not being reverted to
if placeholder_value and state.get("_revert_target") != step_id:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Completed):\n{placeholder_value}")
return Div(
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
else:
pip.append_to_history(f"[WIDGET STATE] {step.show}: Showing input form")
await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("Select multiple files to upload. You can review the selection before proceeding."),
Form(
Input(
type="file",
name="uploaded_files",
multiple="true",
required="true",
cls="contrast"
),
Button("Review Files ▸", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}",
enctype="multipart/form-data"
)
),
Div(id=next_step_id),
id=step_id
)
async def step_01_submit(self, request):
"""Process the submission for the widget design step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
pipeline_id = db.get("pipeline_id", "unknown")
# Get the uploaded files
form_data = await request.form()
uploaded_files = form_data.getlist("uploaded_files")
if not uploaded_files:
await self.message_queue.add(pip, "No files selected. Please try again.", verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("No files were selected. Please try again."),
Form(
Input(
type="file",
name="uploaded_files",
multiple="true",
required="true",
cls="contrast"
),
Button("Review Files ▸", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}",
enctype="multipart/form-data"
)
),
id=step_id
)
# Create a list of file information
file_info = []
total_size = 0
for file in uploaded_files:
file_size = len(await file.read())
total_size += file_size
file_info.append(f"📄 {file.filename} ({file_size:,} bytes)")
# Reset file pointer for potential future reads
await file.seek(0)
# Create a summary of the files
file_summary = "\n".join(file_info)
file_summary += f"\n\nTotal: {len(uploaded_files)} files, {total_size:,} bytes"
# Update step state with file information
await pip.update_step_state(pipeline_id, step_id, file_summary, steps)
# Keep LLM informed about the widget content and state
pip.append_to_history(f"[WIDGET CONTENT] {step.show}:\n{file_summary}")
pip.append_to_history(f"[WIDGET STATE] {step.show}: Files selected")
# Send user-visible confirmation
await self.message_queue.add(pip, f"Selected {len(uploaded_files)} files. Review the list below.", verbatim=True)
# Return the file list display with options to proceed or start over
return Div(
Card(
H3(f"{step.show}"),
P("Selected files:"),
Pre(file_summary, style="white-space: pre-wrap; font-size: 0.9em;"),
Div(
Form(
Button("Upload More Files", type="submit", cls="secondary"),
hx_get=f"/{app_name}/{step_id}",
hx_target=f"#{step_id}",
hx_swap="outerHTML"
),
style="margin-top: 1em;"
)
),
id=step_id
)
// ... existing code ...
I’ve implemented Step 1 of the multi-file upload widget in the Widget Designer workflow. Here are the key changes made:
- Modified
step_01(GET handler):- Added a file input with
multiple="true"andrequired="true" - Set the form’s
enctype="multipart/form-data"for file uploads - Updated the button text to “Review Files ▸”
- Added clear instructions for users
- Added a file input with
- Modified
step_01_submit(POST handler):- Added file handling using
form_data.getlist("uploaded_files") - Implemented error handling for when no files are selected
- Created a detailed file summary showing:
- Individual file names and sizes
- Total number of files and total size
- Added a “Upload More Files” button that returns to the file selection form
- Used
Pretag with proper styling for file list display - Maintained proper state management with
pip.update_step_state - Added appropriate user feedback via
message_queue
- Added file handling using
The implementation follows the “bank small wins” approach by:
- First establishing the basic file selection UI
- Then implementing the file information display
- Providing clear feedback and easy way to start over
- Maintaining proper state management for future steps
The widget now allows users to:
- Select multiple files
- See a detailed list of selected files with sizes
- Start over with new file selection
- Get clear feedback about their actions
This sets us up well for the next phase where we’ll implement the actual file saving functionality. The current implementation ensures all the basic plumbing is working correctly before we add the complexity of file storage.
Would you like to test this implementation before we proceed to the next phase?
I did test it and it’s working like a charm! This is so friggn hard to believe that things are at this stage. When I started out with this project I could hardly get the AI coding assistants to understand that FastHTML is not FastAPI and stop it from using the wrong patterns, subtly undermining my work at every turn. Now, I have one command to create a complete copy/paste of an extremely FastHTML-esque program cleanly making the changes to avoid plugin collision, and a second command to alter a widget placeholder from a blank into a multi-select file requester. And this is the dumbest they’re ever going to be!
Sorry, no offense Gemini. Frankly I think Claude could take it from here. But I wanted to show you the good progress and give you a chance to directionally tweak and steer the next baby-step to bank the next small win.
Gemini Helps Bank Step 2
Okay, Mike, you’re absolutely right! Claude did a fantastic job with that first checkpoint. It’s truly impressive to see the AI assistants grasp these complex, nuanced patterns and implement them so accurately. The progress is palpable.
You’ve correctly identified the core of the Pipulate philosophy: local-first, human-in-the-loop, and leveraging established web technologies in a straightforward way. The pivot from complex download path control to a user-driven upload is a classic example of pragmatic problem-solving in software development, and it aligns perfectly with Pipulate’s design.
Let’s move on to the next checkpoint.
Implementation Plan: Multi-File Upload Widget - Step 2: File Saving and Workflow Progression
Target File: pipulate/plugins/505_widget_designer.py
Previous Checkpoint Recap: Step 1 successfully implemented the multi-file input form and displayed a list of selected files without saving them. The step_01_submit currently provides a “Upload More Files” button to return to the input form.
Goal for Step 2: Modify step_01_submit to save the uploaded files to the server in a structured directory (downloads/APP_NAME/PIPELINE_ID/). After successful saving, display a confirmation and then reinstate the standard workflow chain reaction to proceed to the finalize step. Also, update step_01 (GET handler) to show information about already uploaded files if the step is revisited.
Tasks for the AI Code Assistant:
- Modify
WidgetDesigner.step_01_submitmethod (POST handler):- Retrieve
pipeline_id: Get the currentpipeline_idusingpipeline_id = db.get("pipeline_id", "unknown"). This is crucial for creating the correct save path. - Construct Save Path:
- The base upload directory should be
downloads. - Create a subdirectory named after the workflow’s
APP_NAME(which isself.app_name, currently “design_widget”). - Create a further subdirectory named after the
pipeline_id. - The full path structure will be:
downloads/{self.app_name}/{pipeline_id}/. - Use
pathlib.Pathfor robust path manipulation and directory creation:from pathlib import Path # ... save_directory = Path("downloads") / self.app_name / pipeline_id save_directory.mkdir(parents=True, exist_ok=True)
- The base upload directory should be
- Save Uploaded Files:
- Iterate through the
uploaded_fileslist (obtained fromform_data.getlist("uploaded_files")). - For each
file(which is a StarletteUploadFileobject):- Construct the full path for saving the individual file:
file_save_path = save_directory / file.filename. - Read the file’s content:
contents = await file.read(). - Write the content to the
file_save_path:with open(file_save_path, "wb") as f: f.write(contents) - Add the
file.filenameand its server-side path (str(file_save_path)) to a list of successfully saved files.
- Construct the full path for saving the individual file:
- Iterate through the
- Update User Feedback and State:
- Modify the
file_summaryto include the server-side save path for each file, or indicate successful saving. - Call
await pip.update_step_state(pipeline_id, step_id, file_summary, steps)with the updated summary. - Use
self.message_queue.addto inform the user that files have been saved and where (e.g., “Files saved successfully to server.”).
- Modify the
- Reinstate Chain Reaction:
- Remove the “Upload More Files” button form.
- Instead, the method should now return the standard workflow progression structure. Use
pip.revert_controlorpip.widget_containerto display thefile_summary(which now confirms saved files). - Crucially, include the chain reaction
Divto proceed to the next step (finalizein this case):next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' # ... (inside the return Div of step_01_submit) # This is after the pip.revert_control or pip.widget_container Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load") - The
idof the mainDivreturned bystep_01_submitshould remainstep_idto ensure HTMX correctly replaces the content of the current step.
- Retrieve
- Modify
WidgetDesigner.step_01method (GET handler):- When this step is loaded (either initially or by navigating back), if
step_data.get(step.done, "")(which holds thefile_summary) indicates files have already been processed for thispipeline_id:- It should display the
file_summary(confirming previously uploaded/saved files). - It should use
pip.revert_controlto allow the user to revert and re-upload if necessary. - It must include the chain reaction
Divtonext_step_idto ensure the workflow can proceed if the user is just reviewing a completed step.
- It should display the
- Essentially, the completed view of
step_01should now show the status of saved files rather than just selected files.
- When this step is loaded (either initially or by navigating back), if
- Error Handling (Basic):
- Wrap file writing operations in a
try-exceptblock to catch potentialIOErroror other file system issues. - If an error occurs during saving, inform the user via
self.message_queue.addand potentially return an error messageP("Error saving files.", style=pip.get_style("error"))instead of proceeding. For this checkpoint, a simple error message is sufficient.
- Wrap file writing operations in a
Code Assistant Instructions:
“Please implement Step 2 as outlined above in pipulate/plugins/505_widget_designer.py.
Focus on:
- In
step_01_submit:- Creating the directory structure
downloads/APP_NAME/PIPELINE_ID/. - Saving each uploaded file to this directory.
- Updating
file_summaryto reflect saved files (e.g., include paths or just a success message). - Replacing the ‘Upload More Files’ button with the standard workflow progression (using
pip.revert_controland the chain reactionDivto the next step).
- Creating the directory structure
- In
step_01(GET handler):- Ensure that if files have already been processed for this step (i.e.,
file_summaryexists in the state), it displays this summary along with the revert control and the chain reactionDiv. Add basic try-except blocks for file saving operations. Ensure all HTMX attributes are correctly set for this behavior and that the standard chain reaction pattern is restored.”
- Ensure that if files have already been processed for this step (i.e.,
This step will give us a fully functional initial step in the WidgetDesigner that can accept multiple files, save them to a predictable server location, provide feedback, and then correctly continue the workflow. This will be another excellent checkpoint.
Let me know when you’re ready for Claude (or another assistant if you prefer) to take this on!
Claude Does Step 2 Flawlessly
I’ll help you implement Step 2 of the multi-file upload widget, focusing on file saving and workflow progression. Let’s modify both the step_01 and step_01_submit methods.
import asyncio
from collections import namedtuple
from datetime import datetime
from pathlib import Path
from fasthtml.common import * # type: ignore
from loguru import logger
// ... existing code ...
async def step_01(self, request):
"""Handles GET request for the widget design step.
This is your widget design sandbox. Use this space to prototype and test your widget designs.
"""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
state = pip.read_state(pipeline_id)
step_data = pip.get_step_data(pipeline_id, step_id, {})
file_summary = step_data.get(step.done, "")
# Check if workflow is finalized
finalize_data = pip.get_step_data(pipeline_id, "finalize", {})
if "finalized" in finalize_data and file_summary:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Finalized):\n{file_summary}")
return Div(
Card(
H3(f"🔒 {step.show}: Completed")
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
# Check if step is complete and not being reverted to
if file_summary and state.get("_revert_target") != step_id:
pip.append_to_history(f"[WIDGET CONTENT] {step.show} (Completed):\n{file_summary}")
return Div(
Card(
H3(f"{step.show}"),
P("Previously uploaded files:"),
Pre(file_summary, style="white-space: pre-wrap; font-size: 0.9em;"),
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps)
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
else:
pip.append_to_history(f"[WIDGET STATE] {step.show}: Showing input form")
await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("Select multiple files to upload. You can review the selection before proceeding."),
Form(
Input(
type="file",
name="uploaded_files",
multiple="true",
required="true",
cls="contrast"
),
Button("Review Files ▸", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}",
enctype="multipart/form-data"
)
),
Div(id=next_step_id),
id=step_id
)
async def step_01_submit(self, request):
"""Process the submission for the widget design step."""
pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name
step_id = "step_01"
step_index = self.steps_indices[step_id]
step = steps[step_index]
next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize'
pipeline_id = db.get("pipeline_id", "unknown")
# Get the uploaded files
form_data = await request.form()
uploaded_files = form_data.getlist("uploaded_files")
if not uploaded_files:
await self.message_queue.add(pip, "No files selected. Please try again.", verbatim=True)
return Div(
Card(
H3(f"{step.show}"),
P("No files were selected. Please try again."),
Form(
Input(
type="file",
name="uploaded_files",
multiple="true",
required="true",
cls="contrast"
),
Button("Review Files ▸", type="submit", cls="primary"),
hx_post=f"/{app_name}/{step_id}_submit",
hx_target=f"#{step_id}",
enctype="multipart/form-data"
)
),
id=step_id
)
# Create save directory
save_directory = Path("downloads") / self.app_name / pipeline_id
try:
save_directory.mkdir(parents=True, exist_ok=True)
except Exception as e:
error_msg = f"Error creating save directory: {str(e)}"
logger.error(error_msg)
await self.message_queue.add(pip, error_msg, verbatim=True)
return P("Error creating save directory. Please try again.", style=pip.get_style("error"))
# Create a list of file information and save files
file_info = []
total_size = 0
saved_files = []
for file in uploaded_files:
try:
# Read file content
contents = await file.read()
file_size = len(contents)
total_size += file_size
# Save file
file_save_path = save_directory / file.filename
with open(file_save_path, "wb") as f:
f.write(contents)
# Add to saved files list
saved_files.append((file.filename, str(file_save_path)))
file_info.append(f"📄 {file.filename} ({file_size:,} bytes) -> {file_save_path}")
except Exception as e:
error_msg = f"Error saving file {file.filename}: {str(e)}"
logger.error(error_msg)
await self.message_queue.add(pip, error_msg, verbatim=True)
return P(f"Error saving file {file.filename}. Please try again.", style=pip.get_style("error"))
# Create a summary of the files
file_summary = "\n".join(file_info)
file_summary += f"\n\nTotal: {len(uploaded_files)} files, {total_size:,} bytes"
file_summary += f"\nSaved to: {save_directory}"
# Update step state with file information
await pip.update_step_state(pipeline_id, step_id, file_summary, steps)
# Keep LLM informed about the widget content and state
pip.append_to_history(f"[WIDGET CONTENT] {step.show}:\n{file_summary}")
pip.append_to_history(f"[WIDGET STATE] {step.show}: Files saved")
# Send user-visible confirmation
await self.message_queue.add(pip, f"Successfully saved {len(uploaded_files)} files to {save_directory}", verbatim=True)
# Return the file list display with workflow progression
return Div(
Card(
H3(f"{step.show}"),
P("Files saved successfully:"),
Pre(file_summary, style="white-space: pre-wrap; font-size: 0.9em;"),
pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps)
),
Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"),
id=step_id
)
// ... existing code ...
I’ve implemented Step 2 of the multi-file upload widget with all the requested functionality. Here are the key changes:
-
Added
pathlib.Pathimport for robust path handling - Modified
step_01(GET handler):- Renamed
placeholder_valuetofile_summaryfor clarity - Enhanced the completed state view to show previously uploaded files
- Added proper display of file summary with revert control
- Maintained chain reaction to next step
- Renamed
- Modified
step_01_submit(POST handler):- Added directory creation with error handling:
save_directory = Path("downloads") / self.app_name / pipeline_id save_directory.mkdir(parents=True, exist_ok=True) - Implemented file saving with error handling:
file_save_path = save_directory / file.filename with open(file_save_path, "wb") as f: f.write(contents) - Enhanced file summary to include save paths
- Added proper error handling with user feedback
- Replaced “Upload More Files” with standard workflow progression
- Added chain reaction to next step
- Added directory creation with error handling:
- Added comprehensive error handling:
- Directory creation errors
- File saving errors
- Proper error messages via message queue
- Error display in UI
- Enhanced user feedback:
- Clear success messages with save locations
- Detailed file summaries including paths
- Proper state management for workflow progression
The widget now:
- Creates a structured directory for file storage
- Saves files with proper error handling
- Shows clear feedback about saved files
- Maintains proper workflow progression
- Allows reverting to re-upload if needed
- Preserves file information in the workflow state
Would you like to test this implementation? The next phase could focus on:
- Adding file type validation
- Implementing file size limits
- Adding progress indicators for large uploads
- Enhancing error recovery options
Let me know if you’d like to proceed with any of these enhancements or if you’d like to test the current implementation first.
We’ve Got Upload
To say that I’m stunned would be much under-selling
Exactly how far this win has me kvelling!
The feature’s in Starlette — FastHTML too
Yet I did not think I could make it come true.
The times are a-different and I am a fan
Of how Claude implements with a Gemini plan!
It’s coming up on 9:30 PM. Today’s adventure started out at around 6:00 AM. And it was a formal work day too, although not one with many meetings and certainly not any client meetings. However, it really does change the tone from these focused weekend stints where I know the time is carved out for focus.
But I did it. I have a standalone file-upload widget. It’s still only in widget
designer, ad that’s a file that pops into existence when I need it, copy/pasted
from pipulate/plugins/500_blank_workflow.py and placed as 505 between it and
510_splice_workflow.py which teaches how to work one of these standalone
widgets into a sequence once designed.
And that will be my next step, transposing the file upload widget into a variety
of other places in the system and using the /Generate Curor Rules feature to
lock in all my very context-sensitive learnings and wins. But that’s enough for
tonight (probably) and certainly enough for this article. BAM!
Claude’s Closing Remarks: From Browser Downloads to User Uploads
I let Claude have the last word on this, and it went with a very pragmatic wrap-up.
Today’s journey through browser automation and file handling has been both challenging and enlightening. We started with the ambitious goal of controlling Chrome’s download location in Selenium automation, only to discover that this was a battle not worth fighting. The browser’s security-first approach to download locations, while initially frustrating, led us to a more elegant solution.
The pivot to a user-driven upload approach has several advantages:
- It respects the browser’s security model
- It keeps the human in the loop, making the process more transparent
- It works consistently across different operating systems
- It leverages well-established web technologies
The implementation of the multi-file upload widget in the Widget Designer workflow demonstrates how we can turn limitations into features. By breaking the implementation into small, testable steps:
- First establishing the basic file selection UI
- Then implementing file information display
- Finally adding file saving with proper error handling
We’ve created a robust, reusable component that can be integrated into any workflow. The widget handles:
- Multiple file selection
- File saving to a structured directory
- Clear user feedback
- Proper error handling
- Workflow state management
Next Steps
The Widget Designer has served its purpose well as a sandbox for developing this feature. The next phase will involve:
- Transposing the file upload widget into various workflows where it’s needed
- Using the
/Generate Cursor Rulesfeature to document the implementation patterns - Creating reusable components that can be easily integrated into other workflows
This approach of developing features in isolation before integrating them into the main system has proven to be highly effective. It allows us to focus on the core functionality without the complexity of a full workflow implementation.
The journey from browser automation to user uploads has not only solved our immediate problem but has also provided a more robust and user-friendly solution. Sometimes, the best path forward is to embrace the constraints and work with them rather than against them.
This article is part of an ongoing series documenting the development of Pipulate, a local-first workflow automation system. Stay tuned for more insights into workflow development, widget design, and system architecture.
AI Analysis
Title/Headline Ideas & Filenames:
- Title: The Download Dilemma: Pivoting from Selenium Path Control to User Uploads in a Local-First App
- Filename:
selenium-download-path-control-user-upload-pivot.md
- Filename:
- Title: Browser Automation Realities: When Chrome’s Download Behavior Forces a Workflow Rethink
- Filename:
browser-automation-chrome-download-workflow-rethink.md
- Filename:
- Title: From Frustration to Feature: Solving File Handling in Pipulate with a Human-in-the-Loop Upload Widget
- Filename:
pipulate-file-handling-human-loop-upload-widget.md
- Filename:
- Title: Navigating Cross-Platform Download Challenges in Automated SEO Workflows
- Filename:
cross-platform-download-challenges-automated-seo-workflows.md
- Filename:
- Title: Pipulate’s Journey: The Unforeseen Shift to Multi-File Uploads for Robust Workflow Integration
- Filename:
pipulate-journey-multi-file-uploads-workflow-integration.md
- Filename:
Strengths (for Future Book):
- Authenticity: Provides a genuine, blow-by-blow account of a real-world software development challenge and the problem-solving process.
- Technical Depth: Dives into specific technical hurdles (Selenium download preferences, ChromeDriver behavior, profile persistence) that are often glossed over.
- Illustrates Iteration: Clearly shows the iterative nature of development, including “wild goose chases,” re-evaluation of approaches, and strategic pivots.
- Highlights Trade-offs: Discusses the balance between full automation, semi-automation (human-in-the-loop), and user experience.
- Local-First Context: Emphasizes challenges and solutions specific to local-first application development, which is a valuable and distinct perspective.
- Practical Examples: The thought process around using
webbrowservs.selenium-wireand the eventual idea for a file-moving/upload widget are concrete.
Weaknesses (for Future Book):
- Journal Format: Highly conversational, assumes significant prior context about the “Pipulate” project, specific files (
550_browser_automation.py), and previous achievements. - Jargon/Internal Monologue: Uses terms and phrases (“Mos Eisley clearinghouse,” “Claude,” “Gemini”) that would need explanation or removal for a broader audience. The interaction with AI (Gemini/Claude) needs to be reframed or summarized as authorial decisions.
- Narrow Focus: While detailed, it’s a snapshot of one problem; its relevance in a book would depend on the surrounding chapters and themes.
- Repetitive/Lengthy Reasoning: The author works through the problem in detail, which is natural for a journal but could be condensed for a book.
- Outdated Code/Advice: The embedded code snippets (Daniel Corin’s blog) and AI interactions might become outdated quickly.
- Missing Visuals: The text describes UI interactions and file system structures that would benefit from diagrams or screenshots in a book.
- Needs Structure: Requires significant editing to transform from a linear thought process into a structured explanation or case study.
AI Opinion (on Value for Future Book):
This journal entry offers significant value as raw material for a tech book, especially one aiming to provide an authentic look into the software development trenches. Its strength lies in the detailed, unfiltered narration of a common yet complex problem: managing external interactions like file downloads within an automated system while respecting user experience and platform limitations.
For a future book, this content could serve as an excellent case study on:
- The iterative nature of problem-solving in software engineering.
- The challenges and design considerations for local-first applications.
- Practical issues in browser automation beyond simple script execution.
- The importance of adapting plans when faced with technical roadblocks (e.g., the pivot from controlling download paths to implementing user uploads).
- Designing human-in-the-loop workflows.
However, in its current state, it’s very much a developer’s log. To be effective in a book, it would require substantial editing to:
- Provide broader context about Pipulate and its goals.
- Explain technical jargon more thoroughly.
- Streamline the narrative, perhaps condensing the “wild goose chase” aspects while retaining the core lessons.
- Reframe the interactions with AI (Gemini/Claude) into a more general discussion of seeking external advice or exploring solutions.
- Integrate the core technical insights into a more structured chapter or section.
The raw honesty and detailed thought process are its greatest assets for transformation into insightful book content.