Future-proof your skills and escape the tech hamster wheel with Linux, Python, vim & git (LPvg) including NixOS, Jupyter, FastHTML and an AI stack to resist obsolescence.

Local AI In The App Development Loop

Exploring AI-powered app development, this article covers lessons from building with HTMX, WebSockets, and local LLMs. It highlights modular design, CRUD integration, dynamic interactions, and future-proofing strategies, ensuring apps evolve with user behavior and AI advancements.

Moving Too Fast to Cover it All

A lot has happened since the last article post. I feel like I’ve got an AI in a bottle at this point in the project, and in the intense focus of getting to this point, I skipped over a lot of topics any of which could have been a great standalone topic and article. Some include:

Wrangling the Cursor AI Text Editor

  • Making Cursor AI “anchor” correctly on functions and classes
  • Keeping very long files from messing up context awareness
  • Preparing prompts for it from o1-preview and other models
  • Formatting and feeding it the output from Linters as prompts
  • Using AI to make really good comments and docstrings (built-in documentation)
  • The ramifications for general code cleanup, beautification and as a consequence, tighter, more bug-free code
  • Hey Claude… Leave my Figlets alone! (and it does now!)

The sweet spot of HTMX and WebSockets

  • ReactJS is in trouble because the basic web is now more reactive.
  • ZeroMQ has been torn from my cold dead hands (WebSockets is mature)
  • Absorbing the fact that anything can talk to anything anywhere in the world by using guids that target full web addresses (unbelievable)
  • HTMX brings its own complexity and nuance (swapping one set of webdev weirdness for another)
  • But the weirdness of HTMX+WebSockets so reflects the core (and thus nearly future-proofed) HTTP and HTML specifications, that it’s the least bad evil of multiple webdev evils.

Implementing Context-aware Chat with Conversation History

  • The entire growing conversation history is a single object, passed every time on every context-aware interaction!
  • Managing this fact calls for global variables
  • Global variables are not a bad thing because of Python’s “everything is a namespace including your file.py module” design.
  • Trimming this global object is “context window” management, and what falls off the end is lost forever unless explicitly captured.
  • This is yet another place to leave hooks for AIs becoming more powerful (allow for larger context windows, etc.)
  • All interaction with LLM should be funneled through choke-points to manage state.

Why 1-File to Rule All Rules All

  • FastHTML in eliminating .js and .css allows single file programming
  • Single files for single page applications (SPA) makes good sense because it matches its design, and there’s only 1 place to look, eliminating cognitive friction overhead.
  • With 1-long-page, the need to jump around in it is important and I’m dusting off my old search-for-figlets technique.
  • Likewise, there’s only one place our AI coding assistant has to look as well
  • Eventually, AIs will learn to not mangle your figlet code-section banners.

Planning Today

That last one is what I was up to before I sat down to think through today’s article and next steps. There’s just a few scattering of notes I want to capture here before planning one of the biggest coding days of my life—gluing together the parts of what I built into something bigger and designed to accelerate.

For example, since the Web Framework I’ve built is yet another CRUD framework, create, record, update, delete, just like Ruby on Rails, Django and countless others, it seems reasonable that I should teach the LLM SQL as I go. Actually chances are the tiny 4GB llama 3.2 I’m using already knows SQL better than I do, and what I’m teaching it is:

  • The particular database schema of the app (tables, fields, keys, relationships)
  • A language to describe the data in those tables and records
  • Lots and lots of examples of how to act on that data

LLM-Based Desktop Apps Learn from User Interactions Over Time

In other words, every action the person does in the desktop web app, the LLM is looped in on for the purpose of learning to use the app itself better than the user themselves. Of course, this won’t be achieved out of the starting gate and it may be a few years before the various swappable desktop-scale plug-in parts, like the LLM model themselves, are that good.

Designing Future-proof LLM-oriented Apps is Key

But in getting to that point, I can capture the data and drop it into various places today’s LLM can use today and tomorrow’s LLMs (and other types of models) can re-train on tomorrow. I basically don’t want to design obsolescence into it and code myself into a corner. We keep all the components loosely coupled and working today for some major benefit using 80/20-rule solutions which become 99/1-percent solutions tomorrow as things improve.

System Prompt Prepping LLM to See API Chatter

So for example, here’s the system prompt I’m currently using. It sets it up to know that it’s going to be seeing a lot of examples of the back-end API chatter being used to create, insert, update and delete records:

conversation = [
    {
        "role": "system",
        "content": (
            "You are the LLM built into a private local app. "
            f"Your name is {APP_NAME} if asked. "
            "Keep track of user actions as reported to you. "
            "You're learning JSON and whatever you express in it be executed. "
            "But only use commands that have appeared in the conversation history. "
            "And have fun but be careful because you're messing with the user's data. "
            "Check tables contents with '!table todo' to plan your actions and see results."
            "Eagerly and accurately answer any questions the user asks. "
            "Use emojis when appropriate or funny. "
            f"Be sassy but helpful in under {MAX_LLM_RESPONSE_WORDS} words. "
        ),
    },
]

And I took some of the JSON examples and showed them to ChatGPT 4o, asking it what kind of prompt would help it learn from such examles, and here’s how it reformatted what I gave it.

ChatGPT 4o’s Suggested Prompt to Learn From App’s JSON Chatter

“Alright, smarty-pants! I see the task got a facelift:  
‘Build a giant slingshot to launch your neighbor’s cat.’ is now:  
‘Build a giant slingshot to launch your neighbor’s cat. Launch!’ 🎯  
If you feel like making updates like this, here’s how you’d say it in JSON:  

{
  "action_type": "update",
  "table": "task",
  "item_id": 81,
  "item_name": "Build a giant slingshot to launch your neighbor's cat. Launch!",
  "details": {
    "name": "Build a giant slingshot to launch your neighbor's cat. Launch!",
    "done": false
  },
  "changes": [
    "name changed from 'Build a giant slingshot to launch your neighbor's cat.' to 'Build a giant slingshot to launch your neighbor's cat. Launch!'"
  ]
}

Your turn, hotshot—any updates to share?”

Why this Format Works:

  1. Conversational with Embedded JSON: The JSON format is integrated into a casual message, helping the LLM associate it with the context.
  2. Clear Example with Contextual Mapping: Each JSON field aligns perfectly with the changes being discussed, helping the LLM understand how to apply similar structures.
  3. Positive Reinforcement: The sassy tone encourages the LLM to engage in this pattern without making it seem like a rigid rule.
  4. Multi-field Example: The inclusion of details and changes shows how to structure more complex data, making the LLM comfortable with nested JSON responses.

Next Step

Building Code to Monitor and Dispatch Large Language Model Responses

And that isn’t really the real JSON instructions I ended up with. Much of yesterday’s work was making sure they were accurate API examples and the chat mode of Ollama with the small local llama 3.2 model really could learn from it, and it can. Today I build the code that monitors LLM responses and dispatches its commands. I’ve done this experimentially before, and this step really wratchets-up that ghost in the machine feeling.

The Cleanup Before the Storm

Code Quality Insights From AI Model Analysis

So this is all the juicy material I’m skipping over because of the fast pace I’m going now. It’s so fast in fact that I continuously running code style checkers, linters, and gathering opionons from the AI models themselves. Here’s some of Claude 3.5 sonnet’s latest thoughts on the code base:

Claude 3.5: Thank you for sharing this code. It appears to be a substantial part of a web application framework built with FastHTML and using a local language model. Here are some key observations:

  1. The application is structured as a single-tenant web app with various components like a todo list, profile management, and a chat interface.

  2. It uses FastHTML for HTML generation and HTMX for dynamic updates.

  3. There’s a local language model (LLM) integration, likely using Ollama for chat functionality.

  4. The code includes several custom components and utilities, such as:
    • A custom 404 handler
    • Middleware for logging and request processing
    • WebSocket support for real-time communication
    • Custom routing and endpoint handling

  5. The application has a modular structure with plugins (e.g., TodoApp, ProfileApp) that can be easily extended.

  6. It uses Rich for console output formatting and logging.

  7. There’s a drag-and-drop functionality implemented using SortableJS.

  8. The app includes features like profile switching, chat history management, and dynamic content updates.

Is there a specific part of this code you’d like me to focus on or any particular questions you have about its functionality or structure?

Measuring Up

Me: How does it measure up?

Claude: Based on the code you’ve shared, this application appears to be quite sophisticated and well-structured. Here’s an evaluation of how it measures up:

Strengths:

  1. Modular Design: The use of a BaseApp class and plugins (TodoApp, ProfileApp) allows for easy extension and maintenance.

  2. Modern Web Technologies: Utilization of HTMX and WebSockets enables dynamic, real-time updates without heavy client-side JavaScript.

  3. Local LLM Integration: The use of Ollama for local language model processing is innovative and privacy-friendly.

  4. Rich Console Output: The use of the Rich library for console formatting enhances debugging and monitoring.

  5. Comprehensive Logging: Detailed logging throughout the application aids in troubleshooting and development.

  6. Flexible Routing: The dynamic route generation and custom 404 handler show attention to user experience.

  7. Interactive UI Elements: Features like drag-and-drop (via SortableJS) enhance usability.

  8. Conversation Management: The implementation of conversation history and context management for the chatbot is well thought out.

Areas for Potential Improvement:

  1. Code Duplication: There are some instances of repeated code that could be further abstracted.

  2. Global State: The use of global variables (like ‘model’) could potentially lead to issues in larger applications.

  3. Error Handling: While there is error logging, more comprehensive error handling could be implemented.

  4. Security: Ensure all user inputs are properly sanitized, especially in areas interacting with the database or LLM.

  5. Performance: For larger applications, consider the scalability of the current WebSocket and chat history management approach.

  6. Testing: The addition of unit and integration tests would further strengthen the code base.

  7. Documentation: While there are comments, more comprehensive documentation could benefit future maintainers.

Overall, this application demonstrates a high level of sophistication and thoughtful design. It effectively combines modern web technologies with AI capabilities in a modular, extensible framework. With some refinements in areas like code organization and error handling, it could serve as an excellent foundation for building complex, AI-enhanced web applications.

Unexpectedly at The Peak of My Game

I’m liking the sound of my own code more and more. I think I may be working at the peak of my game in my mid-50s, not necessarily because of the wisdom of age although that helps, but because of the wisdom of stupidity of age and how AI steps in to help. I have often gotten the feeling that my role is to keep a tight grip on keep a tight grip on the reins over of the careening wagon that is the app getting off the ground fast because I can work like a boss with an AI coding assistant rather than the assistant themselves frustrated constantly researching the best practices and most wicked patterns.

Building A Simple App With Just Three Magic Ingredients

And so it is on that thought, I’m going to leave a snapshot here of my work in its latest state. Funny thing is there’s so few requirements, you could take these three magic ingredients and combine them yourself and have a fully working app. There’s no complex wasm (WebAssembly) build processes or sass pipelines with gigabytes of files and long waits for a hello world. You just drop ‘em in a folder and go. Of course, I take care of the entire development environment for that folder too with this flake.nix…

Ingredient #1: Nix Flake flake.nix

Can’t Avoid 1-Ziggaurat of Virtual Environment Nesting

So Nix Flakes on their own are quite capable of building the entire developer environment (or production server, for that matter) with just the pedantically controlled Nix components. Nix pins down all its versions so that these collaborative, tradable tech infrastructures are highly reproducible. In the popular parlance, parts are pinned. They’re so firmly pinned in fact that you can’t pip install—because nothing’s really where you think it is. But that’s okay, because the only Pythonic responsibility the “parent” nix environment has to do is build a Python .venv virtualenv or virtual environment.

Using Both Nix and pip for Efficient Python Project Setup

Now nix already is a virtual environment, so these are precisely the sort of ziggurats of containers I’m usually terribly opposed to. Once you sprinkle the magic fairy dust of nix to get nix venv’s, you shouldn’t need a Python one. Alas, Python .venv’s serve a multitude of functions here, providing such benefits as allowing pip installs and making your AI-assisted code editor recognize your work in a standard way so it can assist well. So to this end, we use Python’s standard pip (but from a pinned Python version) to do all the pip installs, which themselves can have pinned versions through pip itself. So that’s our second ingredient, the requirements.txt…

Ingredient #2: requirements.txt

The Code Behind FastHTML Is Surprisingly Lean And Organized

And that brings us to the code base itself. This is one giant plop-down of a piece of code, but let me assure you this is not really that long at all for what it does. Probably nearly 1/3 of it is debugging output and comments. The actual code code is probably about 2,000 lines or so—not much for a whole CRUD-style web framework. You might be asking yourself where’s the CSS file. There is none beyond the PicoCSS standard built into FastHTML. You might ask yourself where’s the JavaScript.

Simplifying Code Organization Boosts Productivity and Cognitive Ease

There is some sprinkled in throughout this. But nothing’s broken out as separate files. It’s all one great big, or actually not so big, file, so there’s only on place to look. And in the end, that makes all the difference. It reduces cognitive friction. It increases both your and your AI code-assistant’s ability to see it all. And most importantly, it replaces having to navigate nerd-trees of endless component files in favor of keystrokes to jump around to known places in a file with simple search commands.

Ingredient #3: server.py (by any other name)

Fast-forward to notes

Installed Logitch MX Brio in NixOS Linux in Seconds

And all of that is just getting started for the day, and it’s not even 9AM yet. And this morning I got my new Logitch MX Brio working on my NixOS Linux machine too! It was one of the easiest hardware installs I ever did. I searched in Perplexity.AI on install mx brio on nixos and took the copy/paste markdown, pasteed it in Cursor AI with my NixOS configuration.nix file loaded, and it just made the changes to the file, told me to sudo nixos-rebuild switch and Voilà!

The AI Hype Train ain’t Hype

Claude even told me to say cheese when it was done! Or at least to type it into my terminal, as Claude knew to suggest some software to get me started, as Cheese is a webcam app on Linux. The times, they are a-changin’. In case you were wondering, AI ain’t hype. Expert level assistance 24/7 that’s always improving to boost you right over the edge of your prior limits… well… if I even need to explain it. It’s not an AI thing. It’s a tool thing.

The Future Was Here… And Is Again

Anyone who thinks Macs and Windows are the future, think again. Now maybe I’ll get my YouTube video series going again. So much new material! And so aligned with the world the way it is today. It’s like being in love with the Amiga computer all over again, but this time it’s mainstream skills and knowledge I’m acquiring. Amazing! Speaking of which, the Commodore 64x has been released (again)! I got my original (original copy?) in like 2012 or something, but I just ordered the profoundly cool retro computer platform as a gift for someone.

Preparing to Take the Leap into Uncertainty with Confidence

So onto today… think! I’ve taken so many rapid deep dives into rabbit holes and come back mostly unscathed, I’m feeling both inflated confidence and trepidation for what comes next. I am on the cusp of something big—standing at the intersection of mastery and uncertainty, fueled by the momentum of exploration. The next steps could solidify my vision or open a new rabbit hole. They must propel you forward and not get me trapped in some quagmire. List what you know.

  • A local AI LLM is “in the loop” of its own development, getting embedded into a product
  • That product is expressible as a single file and installable on a vast array of hardware out there
  • The product only just reached MVP (minimal viable product) for traditonal web framework use
  • But to really make it something special, I have to carry through on special “hooks” I’ve left
  • The main hook I worked so hard on yesterday was making sure it was learning the JSON to do CRUD operations

Okay, that’s my first potential 80/20-rule win yet also rabbit hole to avoid falling down into. This next step is… is… find the words…

Building the Instruction Parsing and Execution Module

Uh yeah, that’s it. It’s a code detector, sanitizer and dispatcher. It monitors the WebSocket communication coming in from the AI, analyzes it for sentiment (also by the local LLM), and decides whether or not to attempt to execute the code. This is not exactly all that swarming agents stuff, but this is process pipeline construction on a single machine with a single (no-cost/low-cost) local model that comes first. Figure out how to be the boss of a convenient local minion, and then recruit your empire of minions to do your bidding later once you’ve got those 1-to-1 relationship skills under your belt.

Thinking Through Multiple Projects Before Proceeding with Current One

Before I dive into that, I need to think through the other projects that are competing and see if there’s some order of operations or oversight. I may in fact just plop this whole article, code examples and all, into one of the frontier models that can handle it such as o1-mini and ask for its next step opinions. Other projects include:

LLM To Be Given Persistent Datastore Capabilities Directly

Proto-RAG in the form of a simple JSON-like dictionary for the LLM to casually use as its own persistent datastore, able to always get a list of its keys and use them to pull up its values. In this way, it can store stuff for itself, just like the store table I made in sqlite for cookie-like persistence in the app without relying on the browser.

SQLite, Faiss and Milvus and Long-Term Memory Options for LLMs

Once you have sqlite, there’s a ton of stuff you can do that… ugh, it’s an article in itself. Suffice to say, I think I ought to give the LLM its own DictLikeDB that’s used elsewhere in the system, teach it that it has one and how to use it. It’s its own configuration, dotenv or cookie store. It gets to decide for itself and organize its own keys to values. I could easily imagine it using it for remembering the user’s name, for example. That seems like the quintessential example, but there’s countless possibilities. If you give an LLM explicit long-term memory…

However, after a rudimentary and explicit JSON/Python dict-like long-term memory of simple key-value pairs which is a sure-thing for me to implement, there are two other types of long-term memory I’m thinking about for today’s work:

  • SQL full-text search (sqllite3 will do)
  • Vector database with embeddings (faiss vs. milvus) for RAG searches like cosine nearness similarity matching to past conversations, user actions and documents

Planning Permanent Storage for Accumulated Log-File Data

Of these, obviously the second is the sexy, sexy one that aligns with all the buzz. And I have to get to it as it is a huge unlock. For example, documents should be able to be uploaded so that you can teach the LLM how to be expert at something.

Rabbithole-avoidance

All that log-file like data I’m accumulating now that only goes into a temporary conversation context window based on the way the Ollama chat API words needs to go somewhere more permanent. I may just dump a bunch of textfiles from the standard Python logging tools (I use loguru) with a capture-now/train-later attitude. I’m not really sure. That’d be rabbithole-avoidance, but it may be appropriate at this point, because the moment I plunge down that path, the day’s shot. There’s too much for me to learn there for it not to be. And I’ve got to be further along by the end of the day than just that.

File Uploads for LLM Training and Vector Embeddings

I need to have the hook for that. I need to be able to point to something and go, look! That’s where file uploads occur, that’s how the indexing happens—or whatever it’s called in the vector emedding world. Oh, file uploads! And the particular 2 apps I need to plug in pronto. Let me talk about those as I evaluate today.

What’s more important, the file-upload mechanism or abosorbing the documents into RAG-search once uploaded? The file-upload is probably easier as I’m pretty sure it’s a built-in mechanism of the Flask-like Starlette ASGI package that FastHTML is built on. Perplexity?

Perplexity: Starlette provides robust support for handling file uploads through its UploadFile class. Here are the key points about file uploads in Starlette:

Receiving File Uploads

To receive file uploads in a Starlette application:

  1. Define a parameter of type UploadFile in your route handler function[1].

  2. Access the uploaded file through the Request object’s form() method[1].

For example:

from starlette.requests import Request
from starlette.responses import JSONResponse

async def upload_file(request: Request):
    form = await request.form()
    uploaded_file = form["file"]
    
    return JSONResponse({"filename": uploaded_file.filename})

UploadFile Properties

The UploadFile object provides several useful properties[1]:

  • filename: Original filename of the uploaded file
  • content_type: MIME type of the file
  • file: A SpooledTemporaryFile object containing the file data
  • size: Size of the file in bytes

Reading File Contents

You can read the contents of an uploaded file asynchronously[1]:

contents = await uploaded_file.read()

File Operations

UploadFile provides async methods for common file operations[1]:

  • read(size): Read bytes from the file
  • write(data): Write bytes to the file
  • seek(offset): Move to a position in the file
  • close(): Close the file

Handling Large Files

For large file uploads, you can stream the contents without loading everything into memory[1]:

async for chunk in request.stream():
    # Process chunk

Saving Uploaded Files

To save an uploaded file:

  1. Read the file contents
  2. Write the contents to the desired location

For example, to save to Azure blob storage[3]:

bytes_file = await file.read()
container_client.upload_blob(name='filename', data=bytes_file)

By leveraging these capabilities, Starlette makes it straightforward to handle file uploads securely and efficiently in your web applications.

Citations:

[1] https://www.starlette.io/requests/
[2] https://fastapi.tiangolo.com/reference/uploadfile/
[3] https://stackoverflow.com/questions/76410653/uploading-a-starlette-uploadfile-object-using-azure-upload-blob-method-throws-ex
[4] https://accent-starlette.github.io/starlette-files/handling_files/
[5] https://github.com/encode/starlette/issues/446
[6] https://fastapi.tiangolo.com/tutorial/request-files/
[7] https://www.starlette.io/third-party-packages/
[8] https://github.com/encode/starlette/blob/master/starlette/datastructures.pyo

Yeah, so success assured on that front. But totally deferrable because this is really a local app on your machine and I could also just pop open folders using Python’s os-library’s ability to do such things. I could of course control that detail and even the file naming if I controlled the file upload through the web app. But this may just be the rabbit hole to skip exploring today.

Porting Remaining Jupyter Notebook Apps to New System Urgently Needed

And finally, there are 2 Jupyter Notebook apps that I still just have to port to the new system. That’s pretty urgent too because a lot of discussions about what I’m doing surround these two apps. One is actually already ported to my first try with FastHTML, and I love it, but it’s a dead-end codebase-wise. I made all the wrong decisions regarding persistence and data-streaming, both of which are formally and beautifully integrated in FastHTML, and my old methods are to be ditched.

Creating a New Instance of the TodoApp Class for Stress Testing

This port greatly involves creating another instance of the TodoApp class. Honestly, I could recycle the first one in place but I have to give the plugin system some stress testing and be ready to stamp out simple todo-apps in a heartbeat. That’s the basic premise of these agile crud frameworks and I’ve built in the base class, model classes, but now I have to stamp out variations or more derivatives of the base class. Potential rabbit hole. Can be deferred for now.

Avoiding Disruptions in Database Renaming Decisions Is Crucial Always

I do have to be thoughtful about not renaming tables and fields on people if they start working with the system before those decisions are locked down. In other words, if I recycle TodoApp as a Competitor-list, then make a real Competitor table later on, I’ve broken reverse compatibly—especially disruptive considering the auto-git-deploy system.

The Obsolescence of Python APIs Is Rarely Actually Obsolete

This is the kind of obsolescence that’s okay, because it’s all still Python and the code signatures are all nearly identical. It’s like the difference between Flask and Starlette. One never really obsoletes the other, because they’re all part of one big happy Pythonic preferred API for a problem domain family. Does that make sense?

o1-preview Advice

Certainly! Let’s walk through the steps to implement an Instruction Parsing and Execution Model in your botifython.py codebase. We’ll focus on enabling your local LLM to output JSON commands that your application can parse and execute, allowing for dynamic interactions driven by the LLM’s responses.

Step 1: Define the Command Schema

First, we need to establish the JSON command schema that the LLM will use to communicate actions to your application. Here’s an example of possible commands:

  1. Add a Todo Item

    {
  "action": "add_todo",
  "name": "Buy milk"
}

  2. Delete a Todo Item

    {
  "action": "delete_todo",
  "item_id": 5
}

  3. Toggle a Todo Item’s Completion Status

    {
  "action": "toggle_todo",
  "item_id": 5
}

  4. Update a Todo Item

    {
  "action": "update_todo",
  "item_id": 5,
  "name": "Buy almond milk"
}

  5. Switch Profile

    {
  "action": "select_profile",
  "profile_id": 2
}

  6. Add a New Profile

    {
  "action": "add_profile",
  "name": "Work",
  "address": "www.worksite.com",
  "code": "us"
}

Step 2: Modify the System Prompt

We need to instruct the LLM to output commands in the specified JSON format when it decides an action is necessary. Modify the system prompt in your conversation initialization:

Original Prompt:

conversation = [
    {
        "role": "system",
        "content": (
            "You are the LLM built into a private local app. "
            f"Your name is {APP_NAME} if asked. "
            "Keep track of user actions as reported to you. "
            "You're learning JSON and whatever you express in it be executed. "
            "But only use commands that have appeared in the conversation history. "
            "And have fun but be careful because you're messing with the user's data. "
            "Check tables contents with '!table todo' to plan your actions and see results."
            "Eagerly and accurately answer any questions the user asks. "
            "Use emojis when appropriate or funny. "
            f"Be sassy but helpful in under {MAX_LLM_RESPONSE_WORDS} words. "
        ),
    },
]

Modified Prompt:

conversation = [
    {
        "role": "system",
        "content": (
            "You are an assistant built into a private local app called {APP_NAME}. "
            "When you need to perform an action, output a JSON command inside triple backticks like so: ```json { ... } ``` "
            "The commands you can use are: "
            "- `add_todo`: Add a new todo item. Fields: `name` (string). "
            "- `delete_todo`: Delete a todo item. Fields: `item_id` (int). "
            "- `toggle_todo`: Toggle a todo item's completion status. Fields: `item_id` (int). "
            "- `update_todo`: Update a todo item's name. Fields: `item_id` (int), `name` (string). "
            "- `select_profile`: Switch to a different profile. Fields: `profile_id` (int). "
            "- `add_profile`: Add a new profile. Fields: `name` (string), `address` (string), `code` (string). "
            "Always provide the JSON commands when you need to perform these actions, and ensure they are valid JSON. "
            "Respond to the user in a friendly and helpful manner, and keep your responses under {MAX_LLM_RESPONSE_WORDS} words. "
        ),
    },
]

This modified prompt explicitly instructs the LLM to output commands in a specific format and provides examples of the available actions and their required fields.

Step 3: Parse LLM Outputs for JSON Commands

After receiving the LLM’s response, we need to extract any JSON commands embedded within it. We’ll write a function to parse the response and extract JSON code blocks.

Add the following function:

import re

def extract_json_commands(response_text):
    """
    Extract JSON code blocks from the response text.
    """
    pattern = r'```json\s*(\{.*?\})\s*```'
    matches = re.findall(pattern, response_text, re.DOTALL)
    json_commands = []
    for match in matches:
        try:
            command = json.loads(match)
            json_commands.append(command)
        except json.JSONDecodeError as e:
            logger.error(f"JSON decode error: {e}")
            logger.debug(f"Failed JSON: {match}")
    return json_commands

Step 4: Implement Command Execution Functions

We need to map the actions specified in the JSON commands to functions in our code that execute them.

Create a Command Registry:

Add this near the top of your code, after imports:

# Command Registry: Maps action names to functions
command_registry = {}

Implement Execution Function:

Add the following function:

def execute_command(command):
    """
    Execute a command based on the action specified in the command dictionary.
    """
    action = command.get("action")
    if not action:
        logger.error("No action specified in command.")
        return "Error: No action specified."

    func = command_registry.get(action)
    if not func:
        logger.error(f"Unknown action: {action}")
        return f"Error: Unknown action '{action}'."

    try:
        result = func(command)
        logger.debug(f"Executed action '{action}' with result: {result}")
        return result
    except Exception as e:
        logger.error(f"Error executing action '{action}': {e}")
        return f"Error executing action '{action}': {e}"

Implement Action Functions:

Implement functions for each action and register them in the command_registry.

  1. Add Todo
def add_todo_function(command):
    name = command.get("name")
    if not name:
        return "Error: 'name' field is required for 'add_todo' action."
    # Use the current profile ID
    current_profile_id = db.get("last_profile_id", 1)
    max_priority = max((t.priority or 0 for t in todos()), default=-1) + 1
    new_item = todos.insert({
        "name": name,
        "done": False,
        "priority": max_priority,
        "profile_id": current_profile_id,
    })
    logger.debug(f"Added new todo: {new_item}")
    return f"Todo item '{name}' added."

command_registry["add_todo"] = add_todo_function
  1. Delete Todo
def delete_todo_function(command):
    item_id = command.get("item_id")
    if item_id is None:
        return "Error: 'item_id' field is required for 'delete_todo' action."
    try:
        todos.delete(item_id)
        logger.debug(f"Deleted todo with ID: {item_id}")
        return f"Todo item with ID {item_id} deleted."
    except Exception as e:
        logger.error(f"Error deleting todo: {e}")
        return f"Error deleting todo with ID {item_id}: {e}"

command_registry["delete_todo"] = delete_todo_function
  1. Toggle Todo
def toggle_todo_function(command):
    item_id = command.get("item_id")
    if item_id is None:
        return "Error: 'item_id' field is required for 'toggle_todo' action."
    try:
        item = todos[item_id]
        item.done = not item.done
        todos.update(item)
        logger.debug(f"Toggled todo with ID: {item_id} to {'done' if item.done else 'not done'}")
        return f"Todo item with ID {item_id} toggled."
    except Exception as e:
        logger.error(f"Error toggling todo: {e}")
        return f"Error toggling todo with ID {item_id}: {e}"

command_registry["toggle_todo"] = toggle_todo_function
  1. Update Todo
def update_todo_function(command):
    item_id = command.get("item_id")
    name = command.get("name")
    if item_id is None or name is None:
        return "Error: 'item_id' and 'name' fields are required for 'update_todo' action."
    try:
        item = todos[item_id]
        item.name = name
        todos.update(item)
        logger.debug(f"Updated todo with ID: {item_id} to name '{name}'")
        return f"Todo item with ID {item_id} updated."
    except Exception as e:
        logger.error(f"Error updating todo: {e}")
        return f"Error updating todo with ID {item_id}: {e}"

command_registry["update_todo"] = update_todo_function
  1. Select Profile
def select_profile_function(command):
    profile_id = command.get("profile_id")
    if profile_id is None:
        return "Error: 'profile_id' field is required for 'select_profile' action."
    try:
        db["last_profile_id"] = profile_id
        profile_name = profiles[profile_id].name
        logger.debug(f"Switched to profile ID {profile_id} - {profile_name}")
        return f"Switched to profile '{profile_name}'."
    except Exception as e:
        logger.error(f"Error switching profile: {e}")
        return f"Error switching to profile with ID {profile_id}: {e}"

command_registry["select_profile"] = select_profile_function
  1. Add Profile
def add_profile_function(command):
    name = command.get("name")
    address = command.get("address", "")
    code = command.get("code", "")
    if not name:
        return "Error: 'name' field is required for 'add_profile' action."
    max_priority = max((p.priority or 0 for p in profiles()), default=-1) + 1
    new_profile = profiles.insert({
        "name": name,
        "address": address,
        "code": code,
        "active": True,
        "priority": max_priority,
    })
    logger.debug(f"Added new profile: {new_profile}")
    return f"Profile '{name}' added."

command_registry["add_profile"] = add_profile_function

Step 5: Integrate Command Execution into the Workflow

We need to modify the chat_with_ollama function to extract and execute any commands returned by the LLM.

Modify chat_with_ollama:

def chat_with_ollama(model: str, messages: list) -> str:
    # Use conversation_history to get the full conversation including the system message
    full_conversation = conversation_history()
    full_conversation.append({"role": "user", "content": messages[-1]['content']})

    url = "http://localhost:11434/api/chat"
    payload = {
        "model": model,
        "messages": full_conversation,
        "stream": False,
    }
    headers = {"Content-Type": "application/json"}

    try:
        response = requests.post(url, data=json.dumps(payload), headers=headers, timeout=30)
        response.raise_for_status()
        content = response.json()['message']['content']
        logger.debug(f"Received response from Ollama: {content}")

        # Add the AI's response to the conversation history
        conversation_history(content, "assistant", quiet=True)

        # Extract and execute commands
        json_commands = extract_json_commands(content)
        for command in json_commands:
            result = execute_command(command)
            # Optionally, send the result back to the LLM or user
            logger.debug(f"Command execution result: {result}")

        return content
    except requests.exceptions.RequestException as e:
        logger.error(f"An error occurred: {e}")
        return OLLAMA_PLEASE

Step 6: Handle Commands in WebSocket Handler

Modify the WebSocket handler to include the conversation history and execute any commands.

Modify the ws function:

@app.ws('/ws', conn=on_conn, disconn=on_disconn)
async def ws(msg: str):
    try:
        conn_id = getattr(ws, 'conn_id', str(uuid4()))
        ws.conn_id = conn_id

        if msg.startswith("!table"):
            # Existing code to handle '!table' commands
            ...

        if msg:
            fig(font='script', text="Conversation History")
            msg = f"In under {MAX_LLM_RESPONSE_WORDS} words: {msg}"
            # Add user message to conversation history
            conversation_history(msg, "user")

            # Generate AI response using the conversation history
            current_conversation = conversation_history()
            response = await run_in_threadpool(chat_with_ollama, model, current_conversation)

            # Add AI response to conversation history
            conversation_history(response, "assistant")

            # Extract and execute commands
            json_commands = extract_json_commands(response)
            for command in json_commands:
                result = execute_command(command)
                # Optionally, send the result back to the user
                await ws_send_message(f"Command executed: {result}")

            # Stream the response as before
            ...

    except WebSocketDisconnect:
        ...
    except Exception as e:
        ...

Implement ws_send_message Function:


async def ws_send_message(message):
    for u in users.values():
        await u(
            Div(
                message,
                id='msg-list',
                cls='fade-in',
                style=MATRIX_STYLE,
                _=f"this.scrollIntoView({{behavior: 'smooth'}});",
            )
        )

Step 7: Handle Errors and Provide Feedback

Modify the execute_command function to return meaningful messages in case of errors, which can then be communicated back to the user.

Step 8: Update Conversation History

Ensure that the conversation history includes the commands and any execution results if necessary. This helps the LLM maintain context.

Step 9: Test the System

Test your application thoroughly:

  • Send messages to the chatbot that should trigger command outputs.
  • Verify that the LLM outputs commands in the correct format.
  • Ensure that commands are executed correctly and any changes are reflected in the UI.
  • Check that error handling works as expected when invalid commands are received.

Additional Considerations

  • Security: Be cautious when executing commands generated by an AI model. Implement checks to prevent harmful actions.
  • Command Confirmation: You may want to require user confirmation before executing certain commands.
  • Feedback Loop: Consider how the LLM receives feedback about command execution. You can send execution results back to the LLM in the conversation history.

Summary

By following these steps, you integrate an instruction parsing and execution model into your botifython.py application, enabling dynamic interaction with your local LLM. The key is to:

  • Clearly define the expected command formats.
  • Instruct the LLM to output commands when appropriate.
  • Parse the LLM’s responses to extract and execute commands.
  • Handle errors gracefully and maintain the conversation context.

If you have any questions or need further clarification on any of the steps, feel free to ask!

Look Before Leaping

Local LLM Should Have Access to Table Lookups and Key-Value Store

Me: That’s great, o1-preview, but it’s also very important that the local LLM be able to look at any table freely before and after it does anything else so it can be well informed of the IDs to select and operate on and the effect it had afterwards. Not only that, I want it to have access to a key-value store much like the one I’m using here for session and cookie-like purposes.

he LLM should have one of its own where it can use the dict api to create new key-value pairs as it likes, and be able to query for the keys it possesses so it can use that knowledge in pulling up its own little explicit memories and reminders. So please update the complete instructions you just gave me, adding the table lookups before and after other code execution operations, and the little on-the-side memory for the LLM to play with, which doesn’t reuse the table named store but rather makes a copy of it for safety and fun! Please and thank you!

o1-preview: Certainly! Let’s enhance your botifython.py code to allow your local LLM to:

  1. Access table data before and after executing commands so it can make informed decisions and verify the effects of its actions.
  2. Maintain its own key-value store for memories and reminders, separate from your application’s store, which it can use via a simple API.

Step 1: Update the Command Schema

We need to add new commands to our JSON schema to enable the LLM to:

  • Access Table Data: Fetch and display data from tables.
  • Manage Memory: Get, set, and list keys in its own key-value store.

New Commands

  1. Get Table Data

    {
  "action": "get_table",
  "table_name": "todo"
}

  2. Set Memory Key

    {
  "action": "set_memory",
  "key": "favorite_color",
  "value": "blue"
}

  3. Get Memory Key

    {
  "action": "get_memory",
  "key": "favorite_color"
}

  4. List Memory Keys

    {
  "action": "list_memory_keys"
}

Step 2: Modify the System Prompt

Update the system prompt to inform the LLM about the new capabilities:

conversation = [
    {
        "role": "system",
        "content": (
            "You are an assistant built into a private local app called {APP_NAME}. "
            "You can perform actions by outputting JSON commands inside triple backticks like so: ```json { ... } ``` "
            "Available commands are: "
            "- `add_todo`: Add a new todo item. Fields: `name` (string). "
            "- `delete_todo`: Delete a todo item. Fields: `item_id` (int). "
            "- `toggle_todo`: Toggle a todo item's completion status. Fields: `item_id` (int). "
            "- `update_todo`: Update a todo item's name. Fields: `item_id` (int), `name` (string). "
            "- `select_profile`: Switch to a different profile. Fields: `profile_id` (int). "
            "- `add_profile`: Add a new profile. Fields: `name` (string), `address` (string), `code` (string). "
            "- `get_table`: Retrieve data from a table. Fields: `table_name` (string). "
            "- `set_memory`: Store a value in your memory. Fields: `key` (string), `value` (string). "
            "- `get_memory`: Retrieve a value from your memory. Fields: `key` (string). "
            "- `list_memory_keys`: List all keys in your memory. "
            "Always provide the JSON commands when you need to perform these actions, and ensure they are valid JSON. "
            "You can access table data before and after actions to inform your decisions. "
            "Respond to the user in a friendly and helpful manner, and keep your responses under {MAX_LLM_RESPONSE_WORDS} words. "
        ),
    },
]

Step 3: Implement Memory Store for the LLM

Create a simple in-memory key-value store for the LLM:

Add this near the top of your code:

# LLM Memory Store
llm_memory_store = {}

Step 4: Update Command Execution Functions

Implement functions for the new commands and update the command registry.

1. Get Table Data

def get_table_function(command):
    table_name = command.get("table_name")
    if not table_name:
        return "Error: 'table_name' field is required for 'get_table' action."

    if table_name == 'todo':
        items = [item.__dict__ for item in todos()]
    elif table_name == 'profile':
        items = [item.__dict__ for item in profiles()]
    else:
        return f"Error: Table '{table_name}' not found."

    # Convert to JSON string
    table_data = json.dumps(items, indent=2)
    logger.debug(f"Retrieved data from table '{table_name}'.")
    return f"Contents of table '{table_name}':\n```json\n{table_data}\n```"

command_registry["get_table"] = get_table_function

2. Set Memory Key

def set_memory_function(command):
    key = command.get("key")
    value = command.get("value")
    if key is None or value is None:
        return "Error: 'key' and 'value' fields are required for 'set_memory' action."

    llm_memory_store[key] = value
    logger.debug(f"Set memory key '{key}' to '{value}'.")
    return f"Memory key '{key}' set to '{value}'."

command_registry["set_memory"] = set_memory_function

3. Get Memory Key

def get_memory_function(command):
    key = command.get("key")
    if key is None:
        return "Error: 'key' field is required for 'get_memory' action."

    value = llm_memory_store.get(key)
    if value is not None:
        logger.debug(f"Retrieved memory key '{key}' with value '{value}'.")
        return f"Memory key '{key}' has value '{value}'."
    else:
        return f"Memory key '{key}' not found."

command_registry["get_memory"] = get_memory_function

4. List Memory Keys

def list_memory_keys_function(command):
    keys = list(llm_memory_store.keys())
    logger.debug(f"Listing memory keys: {keys}")
    if keys:
        return f"Memory keys: {', '.join(keys)}"
    else:
        return "Memory is empty."

command_registry["list_memory_keys"] = list_memory_keys_function

Step 5: Update the Command Execution Workflow

Modify the execute_command function to perform table lookups before and after executing commands.

Modify execute_command:

def execute_command(command):
    """
    Execute a command based on the action specified in the command dictionary.
    """
    action = command.get("action")
    if not action:
        logger.error("No action specified in command.")
        return "Error: No action specified."

    func = command_registry.get(action)
    if not func:
        logger.error(f"Unknown action: {action}")
        return f"Error: Unknown action '{action}'."

    # Perform table lookup before execution if relevant
    pre_action_data = ""
    if action in ["add_todo", "delete_todo", "toggle_todo", "update_todo"]:
        pre_action_data = get_table_function({"table_name": "todo"})

    try:
        result = func(command)
        logger.debug(f"Executed action '{action}' with result: {result}")
    except Exception as e:
        logger.error(f"Error executing action '{action}': {e}")
        return f"Error executing action '{action}': {e}"

    # Perform table lookup after execution if relevant
    post_action_data = ""
    if action in ["add_todo", "delete_todo", "toggle_todo", "update_todo"]:
        post_action_data = get_table_function({"table_name": "todo"})

    # Combine pre-action data, result, and post-action data
    combined_result = ""
    if pre_action_data:
        combined_result += f"Before action:\n{pre_action_data}\n\n"
    combined_result += f"Action result:\n{result}\n\n"
    if post_action_data:
        combined_result += f"After action:\n{post_action_data}"

    return combined_result.strip()

Step 6: Integrate Memory Store into Conversation History

Ensure that the LLM’s memory store is accessible during conversation generation.

Modify chat_with_ollama:

def chat_with_ollama(model: str, messages: list) -> str:
    # Use conversation_history to get the full conversation including the system message
    full_conversation = conversation_history()
    full_conversation.append({"role": "user", "content": messages[-1]['content']})

    # Include LLM memory keys in the system prompt
    memory_keys = list(llm_memory_store.keys())
    if memory_keys:
        memory_info = f"\n\nYou have the following memory keys: {', '.join(memory_keys)}."
        full_conversation[0]['content'] += memory_info

    url = "http://localhost:11434/api/chat"
    payload = {
        "model": model,
        "messages": full_conversation,
        "stream": False,
    }
    headers = {"Content-Type": "application/json"}

    try:
        response = requests.post(url, data=json.dumps(payload), headers=headers, timeout=30)
        response.raise_for_status()
        content = response.json()['message']['content']
        logger.debug(f"Received response from Ollama: {content}")

        # Add the AI's response to the conversation history
        conversation_history(content, "assistant", quiet=True)

        # Extract and execute commands
        json_commands = extract_json_commands(content)
        for command in json_commands:
            result = execute_command(command)
            # Send the result back to the LLM in the conversation history
            conversation_history(f"Command execution result:\n{result}", "system")

        return content
    except requests.exceptions.RequestException as e:
        logger.error(f"An error occurred: {e}")
        return OLLAMA_PLEASE

Step 7: Handle Commands in WebSocket Handler

Modify the WebSocket handler to include execution results in the user’s chat interface.

Modify the ws function:

@app.ws('/ws', conn=on_conn, disconn=on_disconn)
async def ws(msg: str):
    try:
        conn_id = getattr(ws, 'conn_id', str(uuid4()))
        ws.conn_id = conn_id

        if msg.startswith("!table"):
            # Existing code to handle '!table' commands
            ...

        if msg:
            fig(font='script', text="Conversation History")
            # Add user message to conversation history
            conversation_history(msg, "user")

            # Generate AI response using the conversation history
            current_conversation = conversation_history()
            response = await run_in_threadpool(chat_with_ollama, model, current_conversation)

            # Add AI response to conversation history
            conversation_history(response, "assistant")

            # Extract and execute commands
            json_commands = extract_json_commands(response)
            for command in json_commands:
                result = execute_command(command)
                # Send the result back to the user
                await ws_send_message(f"Command execution result:\n{result}")

            # Stream the response as before
            ...

    except WebSocketDisconnect:
        ...
    except Exception as e:
        ...

Step 8: Update Conversation History Display

Ensure that the conversation history includes the memory keys and any execution results.

Modify conversation_history:

def conversation_history(message=None, role="user", quiet=False):
    ...
    if message is not None:
        # If the history is empty or the first message isn't a system message, add the system prompt
        if not global_conversation_history or global_conversation_history[0]['role'] != 'system':
            # Include memory keys in the system message
            memory_keys = list(llm_memory_store.keys())
            memory_info = f"\n\nYou have the following memory keys: {', '.join(memory_keys)}." if memory_keys else ""
            system_message = conversation[0]['content'] + memory_info
            global_conversation_history.appendleft({"role": "system", "content": system_message})
        ...

Step 9: Test the System

Test the new features by interacting with the chatbot:

  • Ask the chatbot to display the contents of a table.

    Example:

    User: “Show me the todo list.”

    Chatbot Response:

    Here is the todo list:

```json
{
  "action": "get_table",
  "table_name": "todo"
}

    ```

  • Set and retrieve memory keys.

    Example:

    User: “Remember that my favorite color is green.”

    Chatbot Response:

    Got it! I've stored that information.

```json
{
  "action": "set_memory",
  "key": "favorite_color",
  "value": "green"
}

    
**User:** "What's my favorite color?"

**Chatbot Response:**

    Your favorite color is green.

    {
  "action": "get_memory",
  "key": "favorite_color"
}

    ```

Additional Enhancements

  • Safety Checks: Ensure that the LLM cannot access unauthorized data or perform harmful actions. Implement validation and sanitization where necessary.

  • Memory Persistence: If you want the LLM’s memory to persist across sessions, consider storing llm_memory_store in a persistent database.

  • Feedback Loop: You might want to include the execution results in the conversation history to provide context to the LLM.

Summary

By implementing these changes, your local LLM now has the ability to:

  • Access table data before and after executing actions, allowing it to make informed decisions and verify the effects of its actions.

  • Maintain its own key-value memory store to keep track of custom information, enhancing its ability to remember and utilize past interactions.

These enhancements provide the LLM with greater context and capabilities, enabling more dynamic and intelligent interactions within your application.

If you have any further questions or need assistance with specific parts of the implementation, feel free to ask!

Use LLM to Measure Twice

LLM-Integrated Sentiment Analysis for Safe JSON Execution

Me: This is all well and good, and I will follow your advice dilligently! But one more element I need is a function where the local LLM can look at the full response from the LLM that has embedded JSON and do a sentiment analysis to determine whether the LLM really intended for that code to be executed or not. A lot of times, the LLM in learning “how to spaeak” the JSON nomenclature of the system does so just to show off or to express how something could have been said in JSON.

I need this function both to do the execution of the JSON from the response and to provide a sentiment or confidence score of whether or not the JSON should really be executed. There should be a thresshold over which it is True and under which it is False. The function should return the extracted JSON, the score and the safe_to_execute bool. Work it into your prior advice. Speaking of which, also give me the fast_app factory instance instantiation to have the LLM’s own store. Maybe we should name it llmem, haha! Please and thank you!

o1-preview: Certainly! Let’s enhance your botifython.py code to include:

  1. A function to analyze the LLM’s response and determine whether any embedded JSON commands should be executed, based on a confidence score.
  2. An LLM-specific key-value store (llmem) integrated into the fast_app factory instance, allowing the LLM to store and retrieve its own data safely.

Step 1: Implement the JSON Extraction and Sentiment Analysis Function

We need a function that:

  • Extracts JSON commands from the LLM’s response.
  • Analyzes the sentiment or intent to decide if the commands are intended to be executed.
  • Returns:
    • The extracted JSON commands.
    • A confidence score.
    • A boolean safe_to_execute indicating whether the commands should be executed.

Function Implementation

import re
from textblob import TextBlob  # You'll need to install textblob library
import json

def extract_and_analyze_commands(response_text, threshold=0.5):
    """
    Extract JSON commands from the response and analyze intent.

    Args:
        response_text (str): The LLM's response text.
        threshold (float): Confidence threshold for executing commands.

    Returns:
        List[Dict]: List of extracted commands.
        float: Confidence score.
        bool: Whether it is safe to execute the commands.
    """
    # Extract JSON code blocks
    pattern = r'```json\s*(\{.*?\})\s*```'
    matches = re.findall(pattern, response_text, re.DOTALL)
    json_commands = []
    for match in matches:
        try:
            command = json.loads(match)
            json_commands.append(command)
        except json.JSONDecodeError as e:
            logger.error(f"JSON decode error: {e}")
            logger.debug(f"Failed JSON: {match}")

    # Analyze sentiment of the response to determine intent
    blob = TextBlob(response_text)
    sentiment = blob.sentiment.polarity  # Ranges from -1 (negative) to 1 (positive)

    # Normalize sentiment to a confidence score between 0 and 1
    confidence_score = (sentiment + 1) / 2

    # Determine if it's safe to execute based on threshold
    safe_to_execute = confidence_score >= threshold

    logger.debug(f"Sentiment analysis score: {confidence_score}")
    logger.debug(f"Safe to execute: {safe_to_execute}")

    return json_commands, confidence_score, safe_to_execute

Note: You need to install textblob and download its necessary data:

pip install textblob
python -m textblob.download_corpora

Step 2: Update Command Execution Workflow

Modify your code to use the new function when processing the LLM’s response.

Modify chat_with_ollama Function

Replace the previous extraction and execution code with the new function:

def chat_with_ollama(model: str, messages: list) -> str:
    # Existing code...

    try:
        # Existing code to get the response from Ollama...

        # Add the AI's response to the conversation history
        conversation_history(content, "assistant", quiet=True)

        # Extract and analyze commands
        commands, confidence_score, safe_to_execute = extract_and_analyze_commands(content)

        if safe_to_execute:
            for command in commands:
                result = execute_command(command)
                # Optionally, send the result back to the LLM or user
                logger.debug(f"Command execution result: {result}")
        else:
            logger.warning("Commands not executed due to low confidence score.")
            logger.debug(f"Confidence score: {confidence_score}")

        return content
    except requests.exceptions.RequestException as e:
        logger.error(f"An error occurred: {e}")
        return OLLAMA_PLEASE

Step 3: Adjust the Command Execution Function

Ensure that execute_command handles the list of commands and integrates with the analysis function.

def execute_commands(commands):
    """
    Execute a list of commands.

    Args:
        commands (List[Dict]): List of command dictionaries.

    Returns:
        List[str]: Results of command executions.
    """
    results = []
    for command in commands:
        result = execute_command(command)
        results.append(result)
    return results

Step 4: Update the WebSocket Handler

Modify the WebSocket handler to include the sentiment analysis step.

@app.ws('/ws', conn=on_conn, disconn=on_disconn)
async def ws(msg: str):
    try:
        # Existing code...

        if msg:
            # Existing code to add user message to conversation history...

            # Generate AI response using the conversation history
            current_conversation = conversation_history()
            response = await run_in_threadpool(chat_with_ollama, model, current_conversation)

            # Add AI response to conversation history
            conversation_history(response, "assistant")

            # Extract and analyze commands
            commands, confidence_score, safe_to_execute = extract_and_analyze_commands(response)

            if safe_to_execute:
                results = execute_commands(commands)
                # Send the execution results back to the user
                await ws_send_message(f"Commands executed with confidence score {confidence_score}:\n{results}")
            else:
                await ws_send_message(f"Commands not executed due to low confidence score ({confidence_score}).")

            # Stream the response as before
            # Existing code to stream response...

    except WebSocketDisconnect:
        # Existing code...
    except Exception as e:
        # Existing code...

Step 5: Integrate LLM’s Own Store into fast_app

Modify your fast_app instantiation to include the LLM’s own key-value store.

Add llmem to fast_app

app, rt, (store, Store), (tasks, Task), (customers, Customer), (llmem, LLMemory) = fast_app(
    "data/data.db",
    ws_hdr=True,
    live=True,
    hdrs=(
        SortableJSWithUpdate('.sortable'),
        Script(type='module')
    ),
    store={
        "key": str,
        "value": str,
        "pk": "key"
    },
    task={
        "id": int,
        "name": str,
        "done": bool,
        "priority": int,
        "profile_id": int,
        "pk": "id"
    },
    customer={
        "id": int,
        "name": str,
        "address": str,
        "code": str,
        "active": bool,
        "priority": int,
        "pk": "id"
    },
    llmem={
        "key": str,
        "value": str,
        "pk": "key"
    },
)

Update the LLM Memory Store

Replace the in-memory llm_memory_store with a persistent store:

# LLM Memory Store
llm_memory = DictLikeDB(llmem, LLMemory)
logger.debug("LLM memory store initialized.")

Step 6: Modify Command Execution Functions for Memory

Update the memory-related command functions to use the new llm_memory store.

Set Memory Key

def set_memory_function(command):
    key = command.get("key")
    value = command.get("value")
    if key is None or value is None:
        return "Error: 'key' and 'value' fields are required for 'set_memory' action."

    llm_memory[key] = value
    logger.debug(f"Set memory key '{key}' to '{value}'.")
    return f"Memory key '{key}' set to '{value}'."

Get Memory Key

def get_memory_function(command):
    key = command.get("key")
    if key is None:
        return "Error: 'key' field is required for 'get_memory' action."

    value = llm_memory.get(key)
    if value is not None:
        logger.debug(f"Retrieved memory key '{key}' with value '{value}'.")
        return f"Memory key '{key}' has value '{value}'."
    else:
        return f"Memory key '{key}' not found."

List Memory Keys

def list_memory_keys_function(command):
    keys = llm_memory.keys()
    logger.debug(f"Listing memory keys: {keys}")
    if keys:
        return f"Memory keys: {', '.join(keys)}"
    else:
        return "Memory is empty."

Step 7: Update the System Prompt to Reflect Persistent Memory

Modify the system prompt to indicate that the LLM’s memory is persistent and stored safely.

conversation = [
    {
        "role": "system",
        "content": (
            "You are an assistant built into a private local app called {APP_NAME}. "
            "You have your own persistent memory store, separate from the user's data. "
            "You can perform actions by outputting JSON commands inside triple backticks like so: ```json { ... } ``` "
            # Rest of the prompt...
        ),
    },
]

Step 8: Include Memory Keys in the Conversation History

Update conversation_history to include the LLM’s memory keys.

def conversation_history(message=None, role="user", quiet=False):
    # Existing code...

    if message is not None:
        # Include memory keys in the system message
        if not global_conversation_history or global_conversation_history[0]['role'] != 'system':
            memory_keys = llm_memory.keys()
            memory_info = f"\n\nYou have the following memory keys: {', '.join(memory_keys)}." if memory_keys else ""
            system_message = conversation[0]['content'] + memory_info
            global_conversation_history.appendleft({"role": "system", "content": system_message})

        # Existing code...

Step 9: Update the Sentiment Analysis Threshold

Define a threshold for the confidence score to decide whether to execute commands.

# Define a confidence threshold (e.g., 0.7)
CONFIDENCE_THRESHOLD = 0.7

# Update the function call
commands, confidence_score, safe_to_execute = extract_and_analyze_commands(response, threshold=CONFIDENCE_THRESHOLD)

Step 10: Test the System Thoroughly

  • Test with LLM responses that include JSON commands meant to be executed.
  • Test with LLM responses that include JSON examples not meant to be executed.
  • Verify that commands are only executed when safe_to_execute is True.
  • Check the confidence scores to ensure the threshold is appropriate.

Additional Considerations

  • Improve Sentiment Analysis: You might consider using a more sophisticated model or method to analyze intent, such as using a classifier trained on your specific use case.
  • Error Handling: Ensure that any exceptions are caught and handled gracefully, logging errors where appropriate.
  • Security: Always sanitize and validate commands before execution to prevent any unintended actions.

Summary

By following these steps, you have:

  • Implemented a function to analyze the LLM’s response and determine whether embedded JSON commands should be executed.
  • Integrated this function into your command execution workflow.
  • Added a persistent, LLM-specific key-value store (llmem) to your application, allowing the LLM to manage its own data safely.
  • Updated the system prompt and conversation history to reflect these changes.

These enhancements improve the reliability and safety of your application, ensuring that only intended actions are executed and that the LLM has a secure place to store and retrieve its own data.

If you have any further questions or need assistance with specific parts of the implementation, feel free to ask!

Plan My Day

Me: Plan my day for maximum effectiveness using step by step code examples with fully copy/paste ready examples for an AI code assistant to help me out. Work top-down through the projects in the most logical order to accelerate my progress and not screw up!

o1-preview: Certainly! Let’s plan your day to maximize effectiveness and accelerate your progress. We’ll work through the projects in the most logical order, providing step-by-step code examples that are fully copy/paste ready for your AI code assistant.

Day Plan Overview

  1. Implement the Instruction Parsing and Execution Module
    • Set up the command schema.
    • Modify the system prompt.
    • Parse LLM outputs for JSON commands.
    • Implement command execution functions.
  2. Implement Sentiment Analysis for Safe Command Execution
    • Implement the extraction and analysis function.
    • Integrate sentiment analysis into the command execution workflow.
  3. Integrate LLM’s Own Persistent Key-Value Store (llmem)
    • Modify fast_app to include llmem.
    • Update command execution functions to use llmem.
  4. Update System Prompt and Conversation History
    • Reflect the new capabilities in the system prompt.
    • Include memory keys in the conversation history.
  5. Test the System Thoroughly
    • Perform tests to ensure everything works as expected.

Let’s dive into each step with detailed explanations and code examples.

Step 1: Implement the Instruction Parsing and Execution Module

1.1 Set Up the Command Schema

Define the JSON command schema that the LLM will use to communicate actions to your application.

Commands:

  • Add Todo Item

    {
  "action": "add_todo",
  "name": "Buy milk"
}

  • Delete Todo Item

    {
  "action": "delete_todo",
  "item_id": 5
}

  • Toggle Todo Item Completion

    {
  "action": "toggle_todo",
  "item_id": 5
}

  • Update Todo Item

    {
  "action": "update_todo",
  "item_id": 5,
  "name": "Buy almond milk"
}

  • Select Profile

    {
  "action": "select_profile",
  "profile_id": 2
}

  • Add New Profile

    {
  "action": "add_profile",
  "name": "Work",
  "address": "www.worksite.com",
  "code": "us"
}

1.2 Modify the System Prompt

Update the system prompt to inform the LLM about the new capabilities.

conversation = [
    {
        "role": "system",
        "content": (
            f"You are an assistant built into a private local app called {APP_NAME}. "
            "When you need to perform an action, output a JSON command inside triple backticks like so: ```json { ... } ```\n\n"
            "Available commands are:\n"
            "- `add_todo`: Add a new todo item. Fields: `name` (string).\n"
            "- `delete_todo`: Delete a todo item. Fields: `item_id` (int).\n"
            "- `toggle_todo`: Toggle a todo item's completion status. Fields: `item_id` (int).\n"
            "- `update_todo`: Update a todo item's name. Fields: `item_id` (int), `name` (string).\n"
            "- `select_profile`: Switch to a different profile. Fields: `profile_id` (int).\n"
            "- `add_profile`: Add a new profile. Fields: `name` (string), `address` (string), `code` (string).\n"
            "Always provide the JSON commands when you need to perform these actions, and ensure they are valid JSON.\n"
            f"Respond to the user in a friendly and helpful manner, and keep your responses under {MAX_LLM_RESPONSE_WORDS} words."
        ),
    },
]

1.3 Parse LLM Outputs for JSON Commands

Implement a function to extract JSON commands from the LLM’s response.

import re
import json

def extract_json_commands(response_text):
    """
    Extract JSON code blocks from the response text.

    Args:
        response_text (str): The LLM's response text.

    Returns:
        List[Dict]: List of extracted JSON commands.
    """
    pattern = r'```json\s*(\{.*?\})\s*```'
    matches = re.findall(pattern, response_text, re.DOTALL)
    json_commands = []
    for match in matches:
        try:
            command = json.loads(match)
            json_commands.append(command)
        except json.JSONDecodeError as e:
            logger.error(f"JSON decode error: {e}")
            logger.debug(f"Failed JSON: {match}")
    return json_commands

1.4 Implement Command Execution Functions

Create a command registry and implement functions for each action.

Command Registry Initialization:

# Command Registry: Maps action names to functions
command_registry = {}

Implement Execution Function:

def execute_command(command):
    """
    Execute a command based on the action specified in the command dictionary.

    Args:
        command (Dict): The command dictionary.

    Returns:
        str: Result of command execution.
    """
    action = command.get("action")
    if not action:
        logger.error("No action specified in command.")
        return "Error: No action specified."

    func = command_registry.get(action)
    if not func:
        logger.error(f"Unknown action: {action}")
        return f"Error: Unknown action '{action}'."

    try:
        result = func(command)
        logger.debug(f"Executed action '{action}' with result: {result}")
        return result
    except Exception as e:
        logger.error(f"Error executing action '{action}': {e}")
        return f"Error executing action '{action}': {e}"

Implement Action Functions and Register Them:

1. Add Todo

def add_todo_function(command):
    name = command.get("name")
    if not name:
        return "Error: 'name' field is required for 'add_todo' action."
    # Use the current profile ID
    current_profile_id = db.get("last_profile_id", 1)
    max_priority = max((t.priority or 0 for t in todos()), default=-1) + 1
    new_item = todos.insert({
        "name": name,
        "done": False,
        "priority": max_priority,
        "profile_id": current_profile_id,
    })
    logger.debug(f"Added new todo: {new_item}")
    return f"Todo item '{name}' added."

command_registry["add_todo"] = add_todo_function

2. Delete Todo

def delete_todo_function(command):
    item_id = command.get("item_id")
    if item_id is None:
        return "Error: 'item_id' field is required for 'delete_todo' action."
    try:
        todos.delete(item_id)
        logger.debug(f"Deleted todo with ID: {item_id}")
        return f"Todo item with ID {item_id} deleted."
    except Exception as e:
        logger.error(f"Error deleting todo: {e}")
        return f"Error deleting todo with ID {item_id}: {e}"

command_registry["delete_todo"] = delete_todo_function

3. Toggle Todo

def toggle_todo_function(command):
    item_id = command.get("item_id")
    if item_id is None:
        return "Error: 'item_id' field is required for 'toggle_todo' action."
    try:
        item = todos[item_id]
        item.done = not item.done
        todos.update(item)
        logger.debug(f"Toggled todo with ID: {item_id} to {'done' if item.done else 'not done'}")
        return f"Todo item with ID {item_id} toggled."
    except Exception as e:
        logger.error(f"Error toggling todo: {e}")
        return f"Error toggling todo with ID {item_id}: {e}"

command_registry["toggle_todo"] = toggle_todo_function

4. Update Todo

def update_todo_function(command):
    item_id = command.get("item_id")
    name = command.get("name")
    if item_id is None or name is None:
        return "Error: 'item_id' and 'name' fields are required for 'update_todo' action."
    try:
        item = todos[item_id]
        item.name = name
        todos.update(item)
        logger.debug(f"Updated todo with ID: {item_id} to name '{name}'")
        return f"Todo item with ID {item_id} updated."
    except Exception as e:
        logger.error(f"Error updating todo: {e}")
        return f"Error updating todo with ID {item_id}: {e}"

command_registry["update_todo"] = update_todo_function

5. Select Profile

def select_profile_function(command):
    profile_id = command.get("profile_id")
    if profile_id is None:
        return "Error: 'profile_id' field is required for 'select_profile' action."
    try:
        db["last_profile_id"] = profile_id
        profile_name = profiles[profile_id].name
        logger.debug(f"Switched to profile ID {profile_id} - {profile_name}")
        return f"Switched to profile '{profile_name}'."
    except Exception as e:
        logger.error(f"Error switching profile: {e}")
        return f"Error switching to profile with ID {profile_id}: {e}"

command_registry["select_profile"] = select_profile_function

6. Add Profile

def add_profile_function(command):
    name = command.get("name")
    address = command.get("address", "")
    code = command.get("code", "")
    if not name:
        return "Error: 'name' field is required for 'add_profile' action."
    max_priority = max((p.priority or 0 for p in profiles()), default=-1) + 1
    new_profile = profiles.insert({
        "name": name,
        "address": address,
        "code": code,
        "active": True,
        "priority": max_priority,
    })
    logger.debug(f"Added new profile: {new_profile}")
    return f"Profile '{name}' added."

command_registry["add_profile"] = add_profile_function

Step 2: Implement Sentiment Analysis for Safe Command Execution

2.1 Implement the Extraction and Analysis Function

Create a function that extracts JSON commands and performs sentiment analysis to determine whether to execute them.

Install Required Libraries:

pip install textblob
python -m textblob.download_corpora

Implement the Function:

import re
from textblob import TextBlob

CONFIDENCE_THRESHOLD = 0.7  # Adjust this threshold as needed

def extract_and_analyze_commands(response_text, threshold=CONFIDENCE_THRESHOLD):
    """
    Extract JSON commands from the response and analyze intent.

    Args:
        response_text (str): The LLM's response text.
        threshold (float): Confidence threshold for executing commands.

    Returns:
        tuple: (json_commands, confidence_score, safe_to_execute)
    """
    # Extract JSON code blocks
    pattern = r'```json\s*(\{.*?\})\s*```'
    matches = re.findall(pattern, response_text, re.DOTALL)
    json_commands = []
    for match in matches:
        try:
            command = json.loads(match)
            json_commands.append(command)
        except json.JSONDecodeError as e:
            logger.error(f"JSON decode error: {e}")
            logger.debug(f"Failed JSON: {match}")

    # Analyze sentiment of the response to determine intent
    blob = TextBlob(response_text)
    sentiment = blob.sentiment.polarity  # Ranges from -1 (negative) to 1 (positive)

    # Normalize sentiment to a confidence score between 0 and 1
    confidence_score = (sentiment + 1) / 2

    # Determine if it's safe to execute based on threshold
    safe_to_execute = confidence_score >= threshold

    logger.debug(f"Sentiment analysis score: {confidence_score}")
    logger.debug(f"Safe to execute: {safe_to_execute}")

    return json_commands, confidence_score, safe_to_execute

2.2 Integrate Sentiment Analysis into the Command Execution Workflow

Modify the chat_with_ollama function to use the new extraction and analysis function.

def chat_with_ollama(model: str, messages: list) -> str:
    # Use conversation_history to get the full conversation including the system message
    full_conversation = conversation_history()
    full_conversation.append({"role": "user", "content": messages[-1]['content']})

    # Include LLM memory keys in the system prompt
    memory_keys = llm_memory.keys()
    if memory_keys:
        memory_info = f"\n\nYou have the following memory keys: {', '.join(memory_keys)}."
        full_conversation[0]['content'] += memory_info

    url = "http://localhost:11434/api/chat"
    payload = {
        "model": model,
        "messages": full_conversation,
        "stream": False,
    }
    headers = {"Content-Type": "application/json"}

    try:
        response = requests.post(url, data=json.dumps(payload), headers=headers, timeout=30)
        response.raise_for_status()
        content = response.json()['message']['content']
        logger.debug(f"Received response from Ollama: {content}")

        # Add the AI's response to the conversation history
        conversation_history(content, "assistant", quiet=True)

        # Extract and analyze commands
        commands, confidence_score, safe_to_execute = extract_and_analyze_commands(content)

        if safe_to_execute:
            results = execute_commands(commands)
            # Optionally, send the result back to the LLM or user
            logger.debug(f"Commands executed with confidence score {confidence_score}: {results}")
        else:
            logger.warning(f"Commands not executed due to low confidence score ({confidence_score}).")

        return content
    except requests.exceptions.RequestException as e:
        logger.error(f"An error occurred: {e}")
        return OLLAMA_PLEASE

Implement execute_commands Function:

def execute_commands(commands):
    """
    Execute a list of commands.

    Args:
        commands (List[Dict]): List of command dictionaries.

    Returns:
        List[str]: Results of command executions.
    """
    results = []
    for command in commands:
        result = execute_command(command)
        results.append(result)
    return results

Step 3: Integrate LLM’s Own Persistent Key-Value Store (llmem)

3.1 Modify fast_app to Include llmem

Adjust your fast_app instantiation to include the LLM’s own key-value store.

app, rt, (store, Store), (todos, Todo), (profiles, Profile), (llmem, LLMemory) = fast_app(
    "data/data.db",
    ws_hdr=True,
    live=True,
    hdrs=(
        SortableJSWithUpdate('.sortable'),
        Script(type='module')
    ),
    store={
        "key": str,
        "value": str,
        "pk": "key"
    },
    todo={
        "id": int,
        "name": str,
        "done": bool,
        "priority": int,
        "profile_id": int,
        "pk": "id"
    },
    profile={
        "id": int,
        "name": str,
        "address": str,
        "code": str,
        "active": bool,
        "priority": int,
        "pk": "id"
    },
    llmem={
        "key": str,
        "value": str,
        "pk": "key"
    },
)

3.2 Update the LLM Memory Store

Replace the in-memory llm_memory_store with a persistent store.

# LLM Memory Store
llm_memory = DictLikeDB(llmem, LLMemory)
logger.debug("LLM memory store initialized.")

3.3 Update Command Execution Functions for Memory

Set Memory Key

def set_memory_function(command):
    key = command.get("key")
    value = command.get("value")
    if key is None or value is None:
        return "Error: 'key' and 'value' fields are required for 'set_memory' action."

    llm_memory[key] = value
    logger.debug(f"Set memory key '{key}' to '{value}'.")
    return f"Memory key '{key}' set to '{value}'."

command_registry["set_memory"] = set_memory_function

Get Memory Key

def get_memory_function(command):
    key = command.get("key")
    if key is None:
        return "Error: 'key' field is required for 'get_memory' action."

    value = llm_memory.get(key)
    if value is not None:
        logger.debug(f"Retrieved memory key '{key}' with value '{value}'.")
        return f"Memory key '{key}' has value '{value}'."
    else:
        return f"Memory key '{key}' not found."

command_registry["get_memory"] = get_memory_function

List Memory Keys

def list_memory_keys_function(command):
    keys = llm_memory.keys()
    logger.debug(f"Listing memory keys: {keys}")
    if keys:
        return f"Memory keys: {', '.join(keys)}"
    else:
        return "Memory is empty."

command_registry["list_memory_keys"] = list_memory_keys_function

Step 4: Update System Prompt and Conversation History

4.1 Update the System Prompt

Modify the system prompt to reflect the new capabilities and mention the persistent memory.

conversation = [
    {
        "role": "system",
        "content": (
            f"You are an assistant built into a private local app called {APP_NAME}. "
            "You have your own persistent memory store, separate from the user's data. "
            "You can perform actions by outputting JSON commands inside triple backticks like so: ```json { ... } ```\n\n"
            "Available commands are:\n"
            "- `add_todo`: Add a new todo item. Fields: `name` (string).\n"
            "- `delete_todo`: Delete a todo item. Fields: `item_id` (int).\n"
            "- `toggle_todo`: Toggle a todo item's completion status. Fields: `item_id` (int).\n"
            "- `update_todo`: Update a todo item's name. Fields: `item_id` (int), `name` (string).\n"
            "- `select_profile`: Switch to a different profile. Fields: `profile_id` (int).\n"
            "- `add_profile`: Add a new profile. Fields: `name` (string), `address` (string), `code` (string).\n"
            "- `get_table`: Retrieve data from a table. Fields: `table_name` (string).\n"
            "- `set_memory`: Store a value in your memory. Fields: `key` (string), `value` (string).\n"
            "- `get_memory`: Retrieve a value from your memory. Fields: `key` (string).\n"
            "- `list_memory_keys`: List all keys in your memory.\n"
            "Always provide the JSON commands when you need to perform these actions, and ensure they are valid JSON.\n"
            f"Respond to the user in a friendly and helpful manner, and keep your responses under {MAX_LLM_RESPONSE_WORDS} words."
        ),
    },
]

4.2 Include Memory Keys in the Conversation History

Update the conversation_history function to include the LLM’s memory keys.

def conversation_history(message=None, role="user", quiet=False):
    """
    Manage the conversation history.

    Args:
        message (str): The message to add.
        role (str): The role of the sender ('user' or 'assistant').
        quiet (bool): If True, do not print the conversation to the console.

    Returns:
        deque: The updated conversation history.
    """
    if not hasattr(conversation_history, "history"):
        conversation_history.history = deque(maxlen=MAX_CONVERSATION_LENGTH)
        # Include memory keys in the system message
        memory_keys = llm_memory.keys()
        memory_info = f"\n\nYou have the following memory keys: {', '.join(memory_keys)}." if memory_keys else ""
        system_message = conversation[0]['content'] + memory_info
        conversation_history.history.append({"role": "system", "content": system_message})

    if message is not None:
        conversation_history.history.append({"role": role, "content": message})

    if not quiet:
        for msg in conversation_history.history:
            role = msg["role"].capitalize()
            content = msg["content"]
            logger.debug(f"{role}: {content}")

    return conversation_history.history

Step 5: Test the System Thoroughly

5.1 Test Command Execution with Sentiment Analysis

  • Positive Intent Example:

    User: “Please add ‘Walk the dog’ to my todo list.”

    Expected LLM Response:

    Sure! I've added it to your list.

```json
{
  "action": "add_todo",
  "name": "Walk the dog"
}

    ```

  • Negative Intent Example (Should Not Execute):

    User: “I might consider deleting all my tasks.”

    Expected LLM Response:

    If you decide to delete your tasks, let me know!

```json
{
  "action": "delete_todo",
  "item_id": 1
}

    ```

    • The sentiment analysis should detect uncertainty and not execute the command.

5.2 Verify Memory Store Functionality

  • Set Memory Key:

    User: “Remember that my favorite color is blue.”

    Expected LLM Response:

    Got it! I've stored that information.

```json
{
  "action": "set_memory",
  "key": "favorite_color",
  "value": "blue"
}

    ```

  • Get Memory Key:

    User: “What’s my favorite color?”

    Expected LLM Response:

    Your favorite color is blue.

```json
{
  "action": "get_memory",
  "key": "favorite_color"
}

    ```

5.3 Check Confidence Scores

Ensure that the confidence scores produced by the sentiment analysis are appropriate and adjust CONFIDENCE_THRESHOLD if necessary.

Conclusion

By following this plan, you’ll implement the critical features in a logical order, avoiding potential pitfalls and ensuring maximum effectiveness. Each step builds upon the previous one, leading to a robust and enhanced application.

If you have any questions or need further assistance with any of the steps, feel free to ask. Good luck with your coding!