From CDN to Local: Managing JavaScript Dependencies in Pipulate Workflows

This morning, I tackled lingering issues with my Pipulate widget examples, realizing I needed a better way to manage JavaScript dependencies locally, so I built a download script and pulled in `marked.js`. Following that, leveraging extensive context fed to AI assistants Gemini and Claude, I researched and confirmed Prism.js as the right library, then planned its integration to add a crucial new workflow step for displaying syntax-highlighted code with a copy button, setting the stage for the next phase of focused development before tackling client work.

By Mike Levin

Sunday, April 27, 2025

Getting Started with Pipulate: Enhancing Workflows with Code Visualization

This article delves into the practical development process of enhancing a specific web application called “Pipulate.” Pipulate utilizes a “workflow” system, essentially sequences of automated steps, and the author is focused on improving the visual components, known as “widgets,” within these workflows. The main challenge discussed is adding the capability to display computer code (specifically JavaScript) within a workflow step, complete with syntax highlighting (colors to make the code readable) and an easy way to copy it.

The text provides a real-time glimpse into the developer’s thought process, showing how they identify problems with existing widgets, decide to manage external code libraries (like those for Markdown formatting or code highlighting) locally rather than relying on external servers, and even create custom tools for this purpose. It also highlights the increasing role of Artificial Intelligence (AI) tools like Gemini and Claude in modern software development, showcasing how the author uses them for research, code generation, and refining implementation plans. This offers a window into solving specific technical challenges in web development, managing project dependencies, and collaborating with AI coding assistants.

Early Morning Development Planning

How are we going to use this precious focus day? It’s coming up on 6:00 AM on Sunday. I had a good 7 hours or so of sleep. Yesterday, I finished the widget template work. I’ve got solid widget templates — although they are of the 2-step combined variety which may not be copy/paste ready for the most common use case. And the rich grids do look like ASCII text. I mean they look like ASCII text grids. They’re rendering properly as if they were console text, and not HTML. I have the mermaid diagrams working correctly, but not generic markdown. And the plain text widget isn’t showing it’s contents after a submit. And when the workflow is in locked mode, none of the widgets are showing. And even in unlocked mode, the widget previews should not show the original input AND rendered output which is a bit confusing and space-wasting. So, is the widget workflow example really finished?

AI-Assisted Workflow Development Strategy

Okay, so for all these things really the only one we don’t want to do in 60_widget_exmples.py is un-bundling the combined steps, and I think perhaps only because that would be too much of a refactor. The immediate work is preparing it for AI-assisted copy/paste that can do the unbundling into a more stretched-out linear workflow when necessary. At this point in history, and with the Workflows WET as they are, it is unlikely that these things will be put together by anything other than AI from now forward. Wet Workflows I do believe may be a perfect sweet spot for AI. Make a very novel system that provides all sorts of competitive advantages with sufficient contrarian anti-pattern thinking so as to create quite a moat for humans — but provide plenty of template examples to help AIs immediately “get it” (overcoming the contrarian anti-pattern moat), and provide effective assistance.

Integrating MarkedJS for Markdown Support

I don’t think I actually need to add Markdown as a step here, but I should make the Mermaid step easily switch-able to markdown. Okay, the AI assistant wanted to use the markedjs library, so let’s get that local! The CDN location is: https://cdn.jsdelivr.net/npm/marked/marked.min.js

Creating a JavaScript Library Download Script

Alright, so of course we could do “making it local” manually with the browser, visiting that URL in the browser and right-clicking or Ctrl+s‘ing it and using the requester to save it into location. But I make enough normally CDN hosted or NPM-installed JavaScript libraries local in Pipulate that it’s time to make a Python download helper script, both to get the new ones and to refresh the existing list. I want to program it as mostly straight forward initially as I can so the project itself doesn’t become a rabbit hole. But I think it is an accelerator because there’s going to be more and more of these JavaScript library based visualizers (and eventually interactive widgets through AnyWidget) that I want to at least get the ball rolling.

Project Structure Considerations

Okay, at first pass this seems like just something to go into pipulate/precursors. That location is intended for Jupyter Notebooks and executable Python scripts that are Workflow precursors. In other words, things on their way to becoming Pipulate Workflows — and not helper scripts in general, but I have no helper script location and don’t want spreading this kind of code over multiple locations. If it’s not core or a plugin, it goes in precursors for now — even helper scripts that may never become Workflows. But who knows? If it’s Python and it takes parameters, maybe it should be a Workflow. The only reason not at the moment is because of how such scripts are tied into core. It’s on the instantiation of the main app webserver object:

app, rt, (store, Store), (profiles, Profile), (pipeline, Pipeline) = fast_app(
    DB_FILENAME,
    exts='ws',
    live=True,
    default_hdrs=False,
    hdrs=(
        Meta(charset='utf-8'),
        Link(rel='stylesheet', href='/static/pico.css'),
        Script(src='/static/htmx.js'),
        Script(src='/static/fasthtml.js'),
        Script(src='/static/surreal.js'),
        Script(src='/static/script.js'),
        Script(src='/static/Sortable.js'),
        Script(src='/static/mermaid.min.js'),
        Script(src='https://cdn.jsdelivr.net/npm/marked/marked.min.js'),
        Script(src='/static/widget-scripts.js'),
        create_chat_scripts('.sortable'),
        Script(type='module')
    ),
    store={
        "key": str,
        "value": str,
        "pk": "key"
    },
    profile={
        "id": int,
        "name": str,
        "real_name": str,
        "address": str,
        "code": str,
        "active": bool,
        "priority": int,
        "pk": "id"
    },
    pipeline={
        "pkey": str,
        "app_name": str,
        "data": str,
        "created": str,
        "updated": str,
        "pk": "pkey"
    }
)

Understanding Pipulate’s Core Architecture

Wow, so much of the Pipulate system can be gleaned right there! There’s the FastHTML pattern that pops out a rt object to take the place of app.route, there’s the server-side cookie system (store), the profile-switcher and the pipline database objects — schema and all! There’s the activating of websockets, live-reloading developer mode, the turning-off of all FastHTML’s default headers and most importantly to this discussion, the turning-on of my own. For anyone actually following along, this can all be found in server.py (latest) or precisely here (as of time of writing) in GitHub.

Future Plans for JavaScript Library Management

So to avoid rabbit holes, I’m just going to make the script install the latest version from the CDN into pipulate/static, but we can see how in the future I’ll likely maintain a registry of the external JavaScript libraries that Pipulate is able to use in Workflows and automatically include them. But we’re going for simple initially.

Requesting a JavaScript Download Script

Me: Hey Gemini, write me a simple Python script that fetches https://cdn.jsdelivr.net/npm/marked/marked.min.js and saves it into /home/mike/repos/pipulate/static

Gemini 2.5 Experimental:

Gemini handily complies.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Downloads a JavaScript file from a specified URL and saves it locally.
"""

import requests
from pathlib import Path
import sys

# --- Configuration ---
# URL of the JavaScript file to download
JS_URL = "https://cdn.jsdelivr.net/npm/marked/marked.min.js"

# Target directory to save the file
# IMPORTANT: Adjust this path if it's different on your system
TARGET_DIR = Path("/home/mike/repos/pipulate/static")

# --- Main Script Logic ---
def download_js_library(url: str, save_dir: Path):
    """
    Downloads a file from the given URL and saves it into the specified directory.

    Args:
        url (str): The URL of the file to download.
        save_dir (Path): The directory where the file should be saved.
    """
    try:
        # Ensure the target directory exists, create if it doesn't
        save_dir.mkdir(parents=True, exist_ok=True)
        print(f"Target directory: {save_dir}")

        # Extract the filename from the URL
        filename = Path(url).name
        if not filename:
            print(f"Error: Could not determine filename from URL: {url}", file=sys.stderr)
            sys.exit(1)

        save_path = save_dir / filename
        print(f"Attempting to download '{filename}' from {url}...")

        # Perform the GET request to download the file
        response = requests.get(url, stream=True, timeout=30) # Added timeout
        response.raise_for_status()  # Raise an exception for bad status codes (4xx or 5xx)

        # Write the content to the local file
        with open(save_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)

        print(f"Successfully downloaded and saved to: {save_path}")

    except requests.exceptions.RequestException as e:
        print(f"Error downloading file: {e}", file=sys.stderr)
        sys.exit(1)
    except OSError as e:
        print(f"Error saving file to {save_path}: {e}", file=sys.stderr)
        sys.exit(1)
    except Exception as e:
        print(f"An unexpected error occurred: {e}", file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    # --- Installation Note ---
    # Ensure you have the 'requests' library installed:
    # pip install requests
    # --- ---

    print("Starting JavaScript library download...")
    download_js_library(JS_URL, TARGET_DIR)
    print("Download process finished.")

…I run it:

[mike@nixos:~/repos/pipulate/precursors]$ python install_static_library.py 
Starting JavaScript library download...
Target directory: /home/mike/repos/pipulate/static
Attempting to download 'marked.min.js' from https://cdn.jsdelivr.net/npm/marked/marked.min.js...
Successfully downloaded and saved to: /home/mike/repos/pipulate/static/marked.min.js
Download process finished.

…and now I change this line:

Script(src='https://cdn.jsdelivr.net/npm/marked/marked.min.js'),

…to this:

Script(src='/static/marked.min.js'),

…and finally, I commit it to the repo:

[mike@nixos:~/repos/pipulate]$ git add precursors/install_static_library.py static/marked.min.js

[mike@nixos:~/repos/pipulate]$ git commit -am "Added static library installer and the markedjs library"
[main 16faf23] Added static library installer and the markedjs library
 3 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 precursors/install_static_library.py
 create mode 100644 static/marked.min.js

[mike@nixos:~/repos/pipulate]$ git push

And that is how we include a new static JavaScript library in Pipulate. Now it’s time to move onto the 60_widget_exmples.py refinements. But I need to do so with a plan so all of today doesn’t disappear. I have to leave enough time for some client work that I bumped to Monday because of all of this. Ideally, I’m making workflows that directly help with the client work. Generally, yes but in this case no. And so, what I really need to do is blast through this difficult focus-requiring work that changes everything forever-forward by virtue of giving me an example everyone at the office can sink their teeth into while still having enough energy left over to do the client work.

Prioritizing the Mermaid Diagram Step for Maximum Impact

Hmm. Okay, to that end making the Mermaid Diagram step able to be swapped out easily with other JavaScript visualizers is actually the most important step, because the changes-everything Workflow I want to distribute has a piece of JavaScript code as the end deliverable. It’s not the execution of JavaScript in the widget as Step 3 actually demonstrates in the widget examples, but rather the actual displaying of preferably syntax color-coded JavaScript, and preferably with a simple click-to-copy link to get the JavaScript code into your OS copy-buffer for easy pasting elsewhere.

Leveraging the Two-Step Approach for Parameter Collection

Oh, and finally the input for the JavaScript visualization is some web form parameters! So it actually can take advantage of the 2-step approach that AI collapsed into a single-step for the widget examples! Wow so toggling back and forth between “edit mode” and “display mode” for a particular step (i.e. hitting the “revert” button for that step) will show you the interface to collect input-parameters for that visualization step! Wow, this is potentially quite powerful in the way it shortens and makes clearer potentially long multi-step workflows where there’s a lot of collecting of arguments for input parameters. To change the final parameters that control the displaying of the copy/paste-able JavaScript, you just hit revert staying on the same step, change some numbers without having to look at the JavaScript (which is actually programmatically generated on the back-end as the final widget output display), submit and voilà!

Identifying the Right JavaScript Visualization Library

Okay, so I need to identify what JavaScript library is going to do the visualization and the copy/paste button (preferably they come together as a package). Generally I turn one of these articles into a single super-prompt for implementation, but there’s no reason I can’t do multiple super-prompt submissions to Gemini or Grok along the way. Let’s put the context together for our context foo…

FILES_TO_INCLUDE = """\
README.md
flake.nix
server.py
plugins/60_widget_examples.py
.cursorrules
/home/mike/repos/MikeLev.in/_posts/2025-04-26-htmx-widgets-ai-implementation.md
/home/mike/repos/MikeLev.in/_posts/2025-04-26-llm-architecture-ai-collaboration-coding-workflows.md
/home/mike/repos/MikeLev.in/_posts/2025-04-26-fasthtml-htmx-javascript-rendering.md
""".splitlines()[:-1]  # Remove the last empty line

This is the first time I’ve included 3 articles in the context foo. And the way I feed this current article name on the command-line, that makes it FOUR articles establishing background story-telling context, in addition to the various files from the repo that give the code context. At this time, I’m forced to give this particular article a name, moving it out of my ad hoc writing system and into the Jekyll blogging system, however I don’t really know how I’m going to name the file and such. You know how sometimes you’re too in the weeds to see the big picture? So that’s where I do an AI analysis of the article, usually at the end to help with final details. But there’s no reason I can’t do it now… [Inserted at bottom]

Articulating The Prompt

Okay, so now with that AI analysis of this article in-hand, I can articulate the next prompt. Hey Gemini! I know you’re going to be seeing a whole lot of stuff as you scan this, including the article analysis below that makes it seem like a finished article. But it’s not! This is where the actual prompt is.

Priorities

I discuss a lot of clean up in 60_widget_exmples.py in the ways suggested in this article, but it’s all too much to mix into one prompt. Let’s set priorities.

LOW: Rich Library rendering. This can be deferred
LOW: Using <pre> tags for step 1 output widget rendering
LOW: Showing rendered widgets on locked workflows
LOW: Removal of double-display of original input rendered output of widgets
HIGH: Selection of JavaScript library for JavaScript code rendering
HIGH: Figuring out how the copy button will let copying raw JavaScript
HIGH: Adding a step that mimics the Mermaid step but for JavaScript display

The Prompt

So in other words, I need the Mermaid Diagram step 4 to be able to be swapped out easily with a as-yet unidentified JavaScript library that lets you actually display syntax color-coded JavaScript (and other languages?) with an accompanying “copy” button to let you copy the raw JavaScript (sans color-coding syntax) into your OS copy buffer. It can be added as a Step 5 to keep things simple, keeping the Mermaid Diagram step in location for reference.

Once the JavaScript library is identified, it will be easy to incorporate it into the local system given the work already done, so please focus on actually doing the research to identify the correct JavaScript library for the task of displaying color-coded JavaScript to the user in the widget and making it easy for the user to copy that JavaScript into their copy-buffer.

The Precise Prompt

Identify the best JavaScript library and way to do the copy function, and give me an implementation plan focusing on the Workflow Step 5, please and thank you!

Context Foo

Here is the abbreviated output of the context foo prompt. What actually happens is that 72K token prompt with all the XML-wrapping is plopped into my OS copy-buffer so that I can paste it into the prompt box of the Web versions of the frontier AI models. As of this writing, only Gemini and Grok can handle a submit of this size. On the ChatGPT-front, I’d have to go down to 4o-mini, and so only Gemini and Grok’s responses are included below.

[mike@nixos:~/repos/pipulate/precursors]$ python context_foo.py --article-mode --article-path /home/mike/repos/MikeLev.in/_posts/2025-04-27-local-javascript-download-script.md
Using template 1: Article Analysis Mode

=== Prompt Structure ===

--- Pre-Prompt ---
System Information:

You are about to review a codebase in preparation for implementing changes requested in an article.
Please study and understand the codebase thoroughly, as you will need this context
to provide meaningful implementation suggestions based on the article's requirements.

Key things to know about this codebase:
- It uses a hybrid approach with Nix for system dependencies and virtualenv for Python packages
- Always run `nix develop` before any commands in a new terminal
- FastHTML objects must be converted with to_xml() before returning responses
- The project is organized as a server with plugin-based workflows

Key Points:
  • Focus on understanding how the codebase currently implements related functionality
  • Note any existing patterns that could be leveraged for the requested changes
  • Consider how the requested changes would integrate with the current architecture

--- Files Included ---
• README.md (6,580 tokens)
• flake.nix (2,000 tokens)
• server.py (25,295 tokens)
• plugins/60_widget_examples.py (11,298 tokens)
• .cursorrules (688 tokens)
• /home/mike/repos/MikeLev.in/_posts/2025-04-26-htmx-widgets-ai-implementation.md (12,717 tokens)
• /home/mike/repos/MikeLev.in/_posts/2025-04-26-llm-architecture-ai-collaboration-coding-workflows.md (8,839 tokens)
• /home/mike/repos/MikeLev.in/_posts/2025-04-27-local-javascript-download-script.md (4,195 tokens)

--- Post-Prompt ---
Introduction:

Now that you understand the codebase, please review the article's requirements and provide
specific implementation suggestions. Focus on how to modify the codebase to meet these requirements
while maintaining its architectural integrity and existing patterns.

Implementation Areas:
  • Required Changes
  • Integration Strategy
  • Implementation Plan

Focus Areas:
  • Practical implementation of the article's requirements
  • Maintenance of codebase integrity and patterns
  • Clear, actionable implementation steps

--- Token Summary ---
Total tokens: 72,104 tokens (≈$2.16 at GPT-4 rates)
Maximum allowed: 2,087,152 tokens (≈$62.61 at GPT-4 rates)
Remaining: 2,015,048 tokens (≈$60.45 at GPT-4 rates)

=== End Prompt Structure ===

--- Clipboard Instructions ---
Complete XML content successfully copied to clipboard using pyperclip.
You can now paste it.

Script finished.

Gemini 2.5 Experimental Responds

Gemini: Okay, I’ve processed the context, including the new article discussing the local JavaScript download script and the clear priority on adding a code visualization step.

Here’s the research and implementation plan focused on adding Step 5: Code Visualization Widget to plugins/60_widget_examples.py.

Research: JavaScript Library for Code Highlighting & Copying

Several excellent libraries exist for syntax highlighting. We need one that’s relatively easy to integrate with dynamic content loading (HTMX) and provides or facilitates a copy-to-clipboard feature.

Prism.js:
- Pros: Lightweight, extensible, good language support, semantic class names. Has official plugins for line numbers, copy button, etc. Works well with dynamically loaded content via Prism.highlightAll() or Prism.highlightElement().
- Cons: Requires separate CSS theme file. Plugins might add complexity.
- Copying: Has an official Copy to Clipboard Button plugin.
Highlight.js:
- Pros: Automatic language detection, wide language support, simple API (hljs.highlightAll(), hljs.highlightElement()).
- Cons: Can be larger than Prism depending on languages included. Automatic detection can sometimes be wrong.
- Copying: No official built-in copy button, but easy to implement manually using the library’s structure and the browser’s Clipboard API (navigator.clipboard.writeText). Many third-party integrations exist.
CodeMirror:
- Pros: Full-featured code editor library, excellent highlighting, many addons.
- Cons: Significant overkill for just displaying code. Much larger and more complex integration.
- Copying: Trivial, as it’s an editor.

Recommendation:

Prism.js seems like the best fit. It’s designed for highlighting static code blocks, is lightweight, has good performance with dynamic content, and importantly, has an official, well-documented Copy to Clipboard Button plugin. This reduces the amount of custom JavaScript we need to write for the copy functionality.

Goal: Add a step that takes code (e.g., JavaScript) as input, displays it with syntax highlighting using Prism.js, and provides a button to copy the raw code.

Area 1: Required Changes

plugins/60_widget_examples.py:
- __init__:
  - Add a new Step definition for step_05 before the ‘finalize’ step:
    Step( id='step_05', done='code_content', show='Code Visualization', refill=True, ),
  - Update self.steps_indices to reflect the new step.
  - Add route registrations for /widgets/step_05 (GET) and /widgets/step_05_submit (POST).
  - Add default messages for step_05 to self.step_messages.
- step_05(self, request) (GET Handler):
  - Implement logic similar to step_04: check finalization, check if step is complete (using step_data.get('code_content')), check for revert target.
  - If finalized: Show locked card.
  - If complete and not reverting: Create the Prism widget container (similar to step_04_submit but using existing code_content) and return it with the next step trigger (finalize). Use pip.widget_container. Crucially, this widget also needs the HX-Trigger to activate Prism highlighting and the copy button after it’s rendered from saved state.
  - If incomplete or reverting: Show an input form (Textarea) asking for code input. Pre-populate with example JavaScript or the previous value if refill=True. Provide context about the expected input.
- step_05_submit(self, request) (POST Handler):
  - Get code_content from the form data.
  - Validate the input (ensure it’s not empty).
  - Save the code_content to the workflow state using pip.update_step_state.
  - Generate a unique widget_id (e.g., prism-widget-{pipeline_id}-{step_id}).
  - Create the HTML structure for Prism.js:
    # Important: Use the correct language class (e.g., 'language-javascript') # The 'data-prismjs-copy' attribute is used by the Prism copy button plugin prism_widget_content = Pre( Code(user_val, cls="language-javascript"), # Use user_val (raw code) # Attributes for the copy button plugin data_prismjs_copy="Copy raw code", data_prismjs_copy_error="Press Ctrl+C to copy", data_prismjs_copy_success="Code copied!", # Add any other necessary attributes or wrappers cls="line-numbers" # Optional: Add line numbers plugin class ) # Container with unique ID widget_container_div = Div(prism_widget_content, id=widget_id)
  - Use pip.widget_container passing the widget_container_div as the widget argument.
  - Create the HTMLResponse containing the rendered widget_container and the next step trigger (finalize).
  - Add HX-Trigger header: This is critical to tell the client-side JS to initialize Prism after this new content is swapped in.
    response.headers["HX-Trigger"] = json.dumps({ "initializePrism": { # We might not even need a targetId if Prism.highlightAll() works, # but passing it is safer for targeted highlighting. "targetId": widget_id } })
  - Return the HTMLResponse.
server.py:
- fast_app call / hdrs tuple:
  - Add the Prism.js core library JS file. Use the local copy strategy:
    - Run python precursors/install_static_library.py (modified to download Prism files) or manually download prism.js to static/.
    - Add Script(src='/static/prism.js') to hdrs.
  - Add the Prism.js CSS theme file:
    - Download a theme (e.g., prism-okaidia.css) to static/.
    - Add Link(rel='stylesheet', href='/static/prism-okaidia.css') to hdrs.
  - Add the Prism.js Copy to Clipboard Button plugin JS file:
    - Download prism-copy-to-clipboard.min.js to static/.
    - Add Script(src='/static/prism-copy-to-clipboard.min.js') to hdrs.
  - Add the Prism.js Line Numbers plugin JS file (optional but recommended):
    - Download prism-line-numbers.min.js to static/.
    - Add Script(src='/static/prism-line-numbers.min.js') to hdrs.
  - Add the Prism.js Line Numbers plugin CSS file (required if using the plugin):
    - Download prism-line-numbers.css to static/.
    - Add Link(rel='stylesheet', href='/static/prism-line-numbers.css') to hdrs.
- precursors/install_static_library.py (Optional but Recommended):
  - Modify this script to accept URL and target directory as arguments or manage a list of libraries to download, including Prism core, theme, and plugins.

static/widget-scripts.js (or static/script.js):

Add Event Listener for Prism Initialization: Add a listener for the initializePrism trigger sent from step_05_submit.

  document.body.addEventListener('initializePrism', function(evt) {
      console.log('initializePrism trigger received:', evt.detail);
      const targetId = evt.detail.targetId;
      const targetElement = targetId ? document.getElementById(targetId) : document;

      if (targetElement && typeof Prism !== 'undefined') {
          // Use setTimeout to ensure rendering occurs after potential DOM settling
          setTimeout(() => {
              console.log('Highlighting Prism elements within:', targetElement);
              // Highlight elements within the specific target or globally if no targetId
              Prism.highlightAllUnder(targetElement);
              // If using highlightAll, it might be simpler: Prism.highlightAll();
              // Note: Prism plugins (like copy button, line numbers) often
              // initialize automatically when highlightAll/highlightAllUnder runs.
              // Double-check Prism plugin docs if they need explicit calls.
          }, 100); // Small delay might be needed
      } else {
          console.error('Prism library or target element not found for initialization.');
      }
  });

Copy Functionality: The Prism “Copy to Clipboard” plugin should handle this automatically when Prism.highlightAllUnder or Prism.highlightAll is called, provided the HTML structure includes the data-prismjs-copy attributes as shown above. No extra JS should be needed for the basic copy function.

Area 2: Integration Strategy

Integration with Existing Functionality:
- Step 5 slots naturally into the existing workflow before ‘finalize’.
- It reuses the established patterns for GET/POST handlers, state management (pip.update_step_state), UI rendering (pip.widget_container), and client-side interaction (HX-Trigger).
Patterns and Conventions:
- Follow the Step namedtuple pattern.
- Use pipulate helper methods consistently.
- Leverage HTMX for UI updates and triggers.
- Use widget_container for consistent display.
- Use HX-Trigger + client-side listener for activating JS on dynamic content.
- Load JS libraries/CSS globally via hdrs in server.py, using local static files.
- Use to_xml() correctly for HTMLResponse.
Backward Compatibility:
- Adding a new optional step to an existing workflow plugin is backward compatible. Old saved states won’t have step_05 data, but the workflow will proceed correctly.
- Adding Prism JS/CSS files to hdrs is additive and shouldn’t affect other parts of the application unless there are unlikely CSS class collisions.
- Adding the new JS event listener is additive.

Area 3: Implementation Plan

Install Prism:
- Use the modified install_static_library.py or manually download:
  - prism.js
  - A Prism CSS theme (e.g., prism-okaidia.css)
  - prism-copy-to-clipboard.min.js
  - prism-line-numbers.min.js (optional)
  - prism-line-numbers.css (optional)
- Save them to the static/ directory.
- Update the hdrs tuple in server.py to include these new files.
Implement Step 5 Server-Side (60_widget_examples.py):
- Add the Step definition.
- Implement the step_05 GET handler (input form logic).
- Implement the step_05_submit POST handler (validation, state saving, creating Prism HTML structure, returning HTMLResponse with widget_container and HX-Trigger).
Implement Client-Side Listener (static/widget-scripts.js):
- Add the initializePrism event listener using document.body.addEventListener.
- Include console.log statements for debugging.
- Implement the call to Prism.highlightAllUnder(targetElement) within the listener (inside a setTimeout if needed).
Test and Refine:
- Run the workflow. Submit code in Step 5.
- Verify: Code is displayed with syntax highlighting. The copy button appears and functions correctly. Line numbers appear (if enabled). Check the browser console for errors.
- Test reverting to Step 5 – ensure the highlighting and copy button are re-initialized correctly via the GET handler’s HX-Trigger.
- Test the finalized view for Step 5.

Key Milestones:

M1: Prism.js library and plugins successfully downloaded locally and included via hdrs.
M2: Step 5 server-side logic (step_05, step_05_submit) implemented, returning the correct HTML structure and HX-Trigger.
M3: Client-side initializePrism listener implemented and successfully triggering Prism.highlightAllUnder.
M4: Code submitted in Step 5 displays with syntax highlighting and a functional copy-to-clipboard button.

Potential Challenges:

Prism Initialization Timing: Getting Prism.highlightAllUnder to run at the exact right moment after HTMX swap might require adjusting the setTimeout delay or potentially using htmx:afterSwap event if the custom trigger proves unreliable.
CSS Conflicts: The Prism theme CSS could potentially conflict with Pico.css or custom styles.
Copy Button Plugin: Ensure the copy button plugin is correctly included and activated by the highlightAllUnder call. Its interaction depends on the HTML structure (data-prismjs-copy attributes).
Large Code Blocks: Performance of highlighting very large code blocks on the client side could be a concern, though unlikely for typical use cases.

This plan focuses specifically on the high-priority task of adding the code visualization widget using Prism.js and its copy-to-clipboard plugin, leveraging the patterns established by the Mermaid widget integration.

Grok 3 Responds

Grok 3: Below is a detailed response to your query, providing specific implementation suggestions for adding new widget types to Pipulate workflows, as outlined in the article within the provided context. The response is structured according to the requested XML format, focusing on practical changes while maintaining the architectural integrity of the Pipulate codebase.

Final Solution (not)

Required Changes

What specific code changes are needed to implement the article’s requirements?
- New Workflow Plugin (plugins/60_widget_examples.py):
  - Create a new file plugins/60_widget_examples.py using existing templates (e.g., 20_hello_workflow.py).
  - Define a WidgetExamples class with:
    - APP_NAME = "widgets"
    - DISPLAY_NAME = "Widget Examples"
    - ENDPOINT_MESSAGE = "This workflow demonstrates different types of widgets: a simple HTMX widget, a Rich table widget, a JavaScript execution widget, and a Markdown renderer widget."
    - TRAINING_PROMPT = "Demonstrates various widget types in Pipulate Workflows"
  - Implement four steps within the __init__ method:
    - Step 1: Simple HTMX Widget - Basic input form and static HTML display.
    - Step 2: Rich Table Widget - Input JSON data, generate an HTML table using the Rich library.
    - Step 3: JavaScript Execution Widget - Input JavaScript code, execute it client-side post-HTMX injection.
    - Step 4: Markdown Renderer Widget - Input Markdown text, render it using marked.js.
  - Define corresponding step_XX (GET) and step_XX_submit (POST) methods for each step.
- Step Implementations:
  - Step 1 (Simple HTMX):
    - step_01: Render a form with a Textarea for user input.
    - step_01_submit: Validate input, store it in state using pipulate.update_step_state, return a Div with the input content and a next-step trigger (hx-get).
  - Step 2 (Rich Table Widget):
    - step_02: Render a form with a Textarea for JSON input.
    - step_02_submit:
      - Validate JSON input.
      - Use the Rich library (rich.table.Table, rich.console.Console) to generate an HTML table.
      - Wrap the HTML in a Div with NotStr and _raw=True to preserve raw HTML.
      - Use pipulate.widget_container to display below the revert control.
      - Return an HTMLResponse with to_xml() conversion.
  - Step 3 (JavaScript Execution Widget):
    - step_03: Render a form with a Textarea for JavaScript code.
    - step_03_submit:
      - Validate input.
      - Store in state.
      - Create a Div with a unique ID for the widget content.
      - Use pipulate.widget_container to encapsulate the widget.
      - Return an HTMLResponse with an HX-Trigger header to execute the JavaScript post-injection.
  - Step 4 (Markdown Renderer Widget):
    - step_04: Render a form with a Textarea for Markdown input.
    - step_04_submit:
      - Validate input.
      - Store in state.
      - Create a Div with a unique ID and the Markdown content.
      - Use pipulate.widget_container for display.
      - Return an HTMLResponse with an HX-Trigger header to trigger marked.js rendering.
- Server Modifications (server.py):
  - Update the hdrs tuple in the fast_app call to include:
    - Script(src='/static/marked.min.js') (already local per context).
  - Ensure widget_container in the Pipulate class handles raw HTML correctly by preserving NotStr content with _raw=True.
- Static JavaScript (static/script.js):
  - Add event listeners for HX-Trigger events:
    - "runJavaScript": Execute user-provided JavaScript on a specific DOM element.
    - "renderMarkdown": Use marked.parse() to render Markdown content into a target element.
Which existing components need to be modified or extended?
- Pipulate Class (server.py):
  - Extend widget_container to ensure it correctly renders raw HTML from Step 2 without escaping (already supports this with proper use of NotStr and _raw=True).
- server.py:
  - Modify the fast_app call’s hdrs tuple to include marked.min.js (already updated per context).
- static/script.js:
  - Extend to include new global functions or event listeners for HX-Trigger events triggered by Steps 3 and 4.
What new components or patterns need to be introduced?
- New Workflow Plugin: 60_widget_examples.py as a standalone plugin.
- Rich Library Integration: Import and use rich for HTML table generation in Step 2.
- JavaScript Execution Pattern: A pattern using HX-Trigger headers to execute JavaScript post-HTMX injection, ensuring reliable timing and scope.
- Client-Side Rendering Pattern: Use HX-Trigger with marked.js for Markdown rendering, introducing a reusable approach for JS library initialization.

Integration Strategy

How should these changes be integrated with existing functionality?
- Use the existing plugin architecture by placing 60_widget_examples.py in the plugins directory for automatic discovery.
- Leverage Pipulate helper methods (read_state, update_step_state, widget_container, revert_control) for state management and UI consistency.
- Integrate HTMX for dynamic updates, maintaining the chain of hx-get triggers for step progression.
- Use HX-Trigger headers for client-side JavaScript actions, aligning with the pattern seen in 90_roadmap.py for Mermaid.js.
What existing patterns or conventions should be followed?
- Local-First Design: All processing occurs locally, with JavaScript libraries stored in static.
- Server-Side State: Store widget data in the pipeline table as JSON blobs via Pipulate.
- FastHTML Rendering: Convert all FastHTML objects to strings with to_xml() for HTTP responses.
- HTMX Updates: Use hx-post, hx-get, hx-target, and hx-swap for UI interactions.
- Plugin-Based Architecture: Keep the core minimal, extending functionality via plugins.
How can we ensure backward compatibility?
- Adding 60_widget_examples.py as a new plugin does not affect existing workflows.
- Including marked.min.js in hdrs is additive and won’t disrupt current functionality unless naming conflicts occur (unlikely with standard libraries).
- Updates to static/script.js should be additive (new functions/listeners), preserving existing behavior.

Implementation Plan

What’s the recommended order for implementing these changes?
1. Setup:
  - Create plugins/60_widget_examples.py from a template (e.g., 20_hello_workflow.py).
  - Define the four steps in __init__.
2. Step 1 (Simple HTMX):
  - Implement step_01 and step_01_submit.
  - Test basic form submission and state persistence.
3. Step 2 (Rich Table):
  - Add rich to requirements.txt.
  - Implement step_02 and step_02_submit with Rich table generation.
  - Test HTML rendering within widget_container.
4. Step 4 (Markdown Renderer):
  - Verify marked.min.js in hdrs (already done).
  - Add renderMarkdown listener to static/script.js.
  - Implement step_04 and step_04_submit with HX-Trigger.
  - Test Markdown rendering.
5. Step 3 (JavaScript Execution):
  - Add runJavaScript listener to static/script.js.
  - Implement step_03 and step_03_submit with HX-Trigger.
  - Test JavaScript execution.
6. Finalize and Test:
  - Test the full workflow, including revert and finalize states.
What are the key milestones or checkpoints?
- Milestone 1: Step 1 submits and displays content correctly.
- Milestone 2: Step 2 renders a Rich table as HTML, not text.
- Milestone 3: Step 4 renders Markdown using marked.js.
- Milestone 4: Step 3 executes JavaScript post-injection.
- Milestone 5: Entire workflow completes with all widgets functioning.
What potential challenges or risks need to be addressed?
- JavaScript Execution Timing: Ensure HX-Trigger events fire after the DOM is updated. Use timeouts or htmx:afterSwap if needed.
- Raw HTML Handling: Verify Rich table HTML isn’t escaped; use NotStr and _raw=True correctly with to_xml().
- Library Loading: Confirm marked.min.js loads correctly from static and is available for Step 4.
- Scope Issues: JavaScript in Step 3 must target the correct widget container; use unique IDs to avoid conflicts.

This solution provides a practical, actionable plan to implement the four widget types in Pipulate, leveraging its existing architecture while addressing the article’s requirements. The focus on HTMX, FastHTML conventions, and local-first principles ensures the changes enhance the framework without compromising its integrity.

Claude 3.7 Sonnet Analyzes Gemini and Grok’s Responses

Wow, did Grok miss the mark. Granted, the context was enormous. But I explicitly say that the prompt will be found at the bottom of the last article included. Gemini grokked that and Grok didn’t. Handing it over to Claude for an analysis of the AI-generated implementation plans.

Me: Analyze the responses from Gemini and Grok on this topic. Tell me what Gemini saw that Grok didn’t and make qualitative assessments about their respective responses, especially from your perspective as the code assistant that will be helping with implementation.

Claude: Both Gemini and Grok missed the mark on the primary request. You asked specifically for research on the best JavaScript library for code highlighting with copy functionality, focused on implementing a new Step 5 similar to the Mermaid diagram step.

Gemini’s response was more on target:

Correctly identified that you needed a code syntax highlighting library with copy functionality
Researched and compared options (Prism.js, Highlight.js, CodeMirror)
Recommended Prism.js with detailed implementation steps
Provided a clear plan for adding Step 5 to the widget examples

Grok misunderstood completely:

Provided a general implementation plan for the entire 60_widget_examples.py file
Focused on implementing Steps 1-4 which already exist
Failed to address your request for a new Step 5
Didn’t research code highlighting libraries at all

For implementation, Gemini’s recommendation of Prism.js is solid. It’s lightweight, has good language support, and includes a copy-to-clipboard plugin. The implementation plan is thorough, covering server-side code, client-side JavaScript, and the required static files.

So, What Now Claude?

Claude 3.7 Sonnet in Cursor sees what I mean about Grok’s reply and now also has more than enough of what it needs to get today’s work underway. I give it an open-ended prompt.

Me: Indeed! So Claude, please continue this article as you see fit. My goal is to get you into context, condition, shape and readiness (or whatever) to actually do the implementation we’re discussing. Talk to the nice people…

Claude: I’ve analyzed your needs, and based on Gemini’s research (which was on-target), Prism.js is indeed the best choice for code syntax highlighting with copy functionality in your Pipulate workflow system.

Let’s modify the existing download script to fetch Prism.js and its required components:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Downloads JavaScript libraries from specified URLs and saves them locally.
"""

import requests
from pathlib import Path
import sys
from typing import List, Dict, Union, Optional

# Library definitions - URLs and file types
LIBRARIES = {
    "prism-core": {
        "url": "https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/prism.min.js",
        "type": "js"
    },
    "prism-theme": {
        "url": "https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/themes/prism-okaidia.min.css",
        "type": "css"
    },
    "prism-copy": {
        "url": "https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/plugins/copy-to-clipboard/prism-copy-to-clipboard.min.js",
        "type": "js"
    },
    "prism-line-numbers": {
        "url": "https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/plugins/line-numbers/prism-line-numbers.min.js",
        "type": "js"
    },
    "prism-line-numbers-css": {
        "url": "https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/plugins/line-numbers/prism-line-numbers.min.css",
        "type": "css"
    }
}

# Target directory to save the files
TARGET_DIR = Path("/home/mike/repos/pipulate/static")

def download_library(name: str, lib_info: Dict[str, str], save_dir: Path) -> Optional[Path]:
    """
    Downloads a file from the given URL and saves it into the specified directory.
    
    Args:
        name: The name of the library
        lib_info: Dictionary containing url and type
        save_dir: The directory where the file should be saved
    
    Returns:
        Path to the saved file or None if download failed
    """
    try:
        url = lib_info["url"]
        
        # Ensure the target directory exists, create if it doesn't
        save_dir.mkdir(parents=True, exist_ok=True)
        
        # Extract the filename from the URL
        filename = Path(url).name
        if not filename:
            print(f"Error: Could not determine filename from URL: {url}", file=sys.stderr)
            return None
        
        save_path = save_dir / filename
        print(f"Downloading '{name}' from {url}...")
        
        # Perform the GET request to download the file
        response = requests.get(url, stream=True, timeout=30)
        response.raise_for_status()
        
        # Write the content to the local file
        with open(save_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)
        
        print(f"Successfully saved to: {save_path}")
        return save_path
    
    except Exception as e:
        print(f"Error downloading {name}: {e}", file=sys.stderr)
        return None

def download_all_libraries(libraries: Dict[str, Dict[str, str]], save_dir: Path) -> List[Path]:
    """
    Downloads all libraries defined in the libraries dictionary.
    
    Args:
        libraries: Dictionary of library definitions
        save_dir: Directory to save files to
        
    Returns:
        List of paths to successfully downloaded files
    """
    successful_downloads = []
    
    for name, lib_info in libraries.items():
        result = download_library(name, lib_info, save_dir)
        if result:
            successful_downloads.append(result)
    
    return successful_downloads

if __name__ == "__main__":
    print("Starting JavaScript library download...")
    successful = download_all_libraries(LIBRARIES, TARGET_DIR)
    print(f"Download process finished. {len(successful)}/{len(LIBRARIES)} libraries downloaded successfully.")

Now, we need to update the server.py file to include these new libraries in the hdrs tuple. Let’s modify line 2005:

app, rt, (store, Store), (profiles, Profile), (pipeline, Pipeline) = fast_app(
    DB_FILENAME,
    exts='ws',
    live=True,
    default_hdrs=False,
    hdrs=(
        Meta(charset='utf-8'),
        Link(rel='stylesheet', href='/static/pico.css'),
        Link(rel='stylesheet', href='/static/prism-okaidia.min.css'),
        Link(rel='stylesheet', href='/static/prism-line-numbers.min.css'),
        Script(src='/static/htmx.js'),
        Script(src='/static/fasthtml.js'),
        Script(src='/static/surreal.js'),
        Script(src='/static/script.js'),
        Script(src='/static/Sortable.js'),
        Script(src='/static/mermaid.min.js'),
        Script(src='/static/marked.min.js'),
        Script(src='/static/prism.min.js'),
        Script(src='/static/prism-copy-to-clipboard.min.js'),
        Script(src='/static/prism-line-numbers.min.js'),
        Script(src='/static/widget-scripts.js'),
        create_chat_scripts('.sortable'),
        Script(type='module')
    ),
    store={
        "key": str,
        "value": str,
        "pk": "key"
    },
    # ... rest of the code remains unchanged

Next, we need to update static/widget-scripts.js to add an event listener for Prism initialization:

// Add this to your widget-scripts.js file

// Initialize Prism.js for dynamically added content
document.body.addEventListener('initializePrism', function(evt) {
    console.log('Prism initialization triggered');
    const targetId = evt.detail?.targetId;
    const targetElement = targetId ? document.getElementById(targetId) : document;
    
    if (targetElement && typeof Prism !== 'undefined') {
        // Use setTimeout to ensure DOM is fully updated
        setTimeout(() => {
            console.log('Highlighting code in:', targetId || 'document');
            Prism.highlightAllUnder(targetElement);
        }, 100);
    } else {
        console.error('Prism library not loaded or target element not found');
    }
});

Finally, let’s implement Step 5 in the plugins/60_widget_examples.py file:

First, add the new step to the __init__ method right before the finalize step
Implement the GET and POST handlers for the new step

The complete implementation for a syntax highlighting step with copy functionality would follow the same pattern as the Mermaid step, but customize it for code highlighting with Prism.js.

This implementation covers all the high-priority items you identified:

Selected Prism.js as the optimal JavaScript library for code highlighting
Added copy button functionality through Prism’s plugin
Created a plan for adding a new Step 5 that mimics the Mermaid step but for JavaScript display

Ready, Set, Go!

Me: 8:00 AM sharp! And so my day begins.

AI Analysis

Title/Headline Ideas:
- Integrating Prism.js for Code Highlighting in Pipulate Workflows via Local Scripts & AI
- Developer Log: From Pipulate Widget Bugs to AI-Assisted Prism.js Implementation
- Pipulate Enhancement: Adding Local JS Dependencies and a Code Visualization Widget
- AI Collaboration Deep Dive: Solving Pipulate Workflow Challenges with Gemini & Claude
- Building a Code-Highlighting Widget in Pipulate with Prism.js and Custom Tooling
Strengths:
- Transparency: Shows the raw, unfiltered thought process of development, including self-correction, planning, and interaction with AI.
- Practical Examples: Includes concrete code snippets (Python download script, hdrs configuration, JavaScript listener structure) and command-line interactions.
- AI Integration Showcase: Provides a tangible example of how developers leverage large language models (Gemini, Grok, Claude) for research and implementation planning within a complex project.
- Problem-Solving Detail: Documents the specific steps taken to address technical hurdles (like local JS management and library selection).
Weaknesses:
- High Context Dependency: Extremely difficult for readers unfamiliar with Pipulate, its architecture (FastHTML, workflows, plugins), and the author’s specific setup to fully grasp without significant prior knowledge.
- Meandering Narrative: The journal format naturally jumps between different sub-problems (widget cleanup, download script, library research) which can make the primary goal (code highlighting widget) less immediately obvious.
- Jargon and Implicit Knowledge: Relies heavily on project-specific terms and assumes understanding of concepts like “WET workflows” or the “contrarian anti-pattern moat.”
- Potential for Noise: The inclusion of multiple AI responses, including one that missed the mark (Grok), adds length and detail that might distract from the core technical implementation path.
AI Opinion: This article serves as a valuable, detailed developer log documenting a specific problem-solving process within the Pipulate project. Its strength lies in its transparency regarding the iterative nature of development, dependency management strategies (local JS files), and the practical application of AI assistants (Gemini, Claude) for research and planning. While its high specificity and reliance on prior context limit its utility as a general tutorial, it offers significant value to the author or others deeply involved in the project as a record of decisions and implementation steps. The comparison between different AI responses also provides an interesting meta-commentary on the current state of AI coding assistance.