Future-proof your skills and escape the tech hamster wheel with the Linux, Python, vim & git stack (LPvg) including NixOS, Jupyter, FastHTML / HTMX and an AI stack to resist obsolescence. Follow along as I debunk peak data theory and develop Pipulate, the next generation free AI SEO tool.

Nix Flake Python Solution: Venv, Cuda, macOS/Windows, Etc.

This article presents a solution for developers who struggle with Python environment consistency across different operating systems and have discovered nix flakes. I address the common pain points of setting up Python environments, especially for data science, and leverage the power of Nix to create a reproducible and portable solution (Jupyter, ML, etc). The inclusion of CUDA support and the focus on simplifying `pip install` enhance its utility and streamline your Python development workflow.

A Cross-Platform Python Solution with Nix Flakes

Are you looking for a Nix Flake Python solution that works on Macs or Windows, supporting Python pip install commands, your NVidia card if you’ve got it? Me too! So I made this. Download it, clone it, fork it, whatever.

I’ve updated it to Python 3.13 to get you on all the latest goodness. If that’s out of date by the time you read this, there’s just a few places to update the version reference in the flake.nix file which you can usually find in its latest form on its GitHub page: Nix Flake Python

Balancing Nix Immutability with Data Science Needs

Nix’s weird but wonderful immutability that makes it so great for portable cross-platform environments also makes it frustrating for the traditional Data Science use case. This fixes that.

Introducing Darwinix: A Minimal Data Science Starting Point

Darwinix is just a bare minimum starting place for Data Science or anyone in that love-worthy Venn diagram intersection of being a nix and Python user. If you’re on Google Colab paying for stuff you don’t need to, this may be worth a look. You’re not forced to pay or lose what you did.

Included Python Packages

Look in requirements.txt. You’ll see I’ve included:

numpy
pandas
requests
jupyterlab
jupyter-ai[all]

…but not much else. But no problem. Just build it up yourself the same you would with any requirements.txt. Pin versions if you like, using the pip requirements way of doing that.

pandas==2.2.1
numpy==1.26.4
requests==2.18.4

Starting Fresh with Your Own Git Repo

Wipe away my .git folder to dissociate it from my repo (if you cloned it) and git init it yourself. This gives you that OS-as-Code dream. Write once, run anywhere as the age-old promise goes.

Automatic Virtual Environment Setup

What about all that annoying virtualenv stuff? Well, it gets set up automatically. Once you copy the code, you just cd into whatever folder you unzipped it to and type:

cd darwinix  # (or whatever you named it)
nix develop
start

…and a JupyterLab instance (with JupyterAI and all) will just pop up in your browser. Anything you pip install from that terminal or a Jupyter cell will be accessible. The .venv is just there from the nix develop step.

Understanding the Nix Environment Structure

Declarative deterministic nix provides a that immutable “outer” environment that has just enough Python to create your dynamic, persistent “inner” Python virtual environment, which is as good as a .venv on any platform. And that’s done just once the first time you nix develop.

Getting Started with Nix Commands

If you’re new to nix, I recommend looking it up just to get the basics of the nix commands, particularly in relation to nix develop. There’s not much to know, but it is useful to get the idea that the environments you create are tied to the terminals you activate them from. That sort of thing.

Why Nix+Pip Beats Other Package Managers

You don’t really need Poetry, Pipenv or any of those other things, though I suppose you could work them in if you really wanted to. I also left out conda because since PyPI added “wheels”, conda feels redundant. But adjust to taste. There’s nothing you can’t just add.

I tried keeping it pretty minimal, tackling just the cross-platform stuff, taking advantage of your GPU’s, and giving you the pip-install freedom. So, nix+pip seems to be the sweet spot for optimum system-control for trading environments around, taking advantage of your hardware, and not feeling stifled.

Get that loosey goosey freedom of pip installing whatever you need until stuff just works, it’s nice to have a traditional .venv Python virtual environment, and that’s what this does inside a nix flake.

Get the best of both worlds!

#       ____                      _       _                        .--.      ___________
#      |  _ \  __ _ _ ____      _(_)_ __ (_)_  __    ,--./,-.     |o_o |    |     |     |
#      | | | |/ _` | '__\ \ /\ / / | '_ \| \ \/ /   / #      \    |:_/ |    |     |     |
#      | |_| | (_| | |   \ V  V /| | | | | |>  <   |          |  //   \ \   |_____|_____|
#      |____/ \__,_|_|    \_/\_/ |_|_| |_|_/_/\_\   \        /  (|     | )  |     |     |
#                                                    `._,._,'  /'\_   _/`\  |     |     |
#      Solving the "Not on my machine" problem well.           \___)=(___/  |_____|_____|

# Most modern development is done on Linux, but Macs are Unix. If you think Homebrew and Docker
# are the solution, you're wrong. Welcome to the world of Nix Flakes! This file defines a complete,
# reproducible development environment. It's like a recipe for your perfect workspace, ensuring
# everyone on your team has the exact same setup, every time. As a bonus, you can use Nix flakes on
# Windows under WSL. Plus, whatever you make will be deployable to the cloud.

{
  # This description helps others understand the purpose of this Flake
  description = "A flake that reports the OS using separate scripts with optional CUDA support and unfree packages allowed.";
  
  # Inputs are the dependencies for our Flake
  # They're pinned to specific versions to ensure reproducibility
  inputs = {
    # nixpkgs is the main repository of Nix packages
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    # flake-utils provides helpful functions for working with Flakes
    flake-utils.url = "github:numtide/flake-utils";
  };

  # Outputs define what our Flake produces
  # In this case, it's a development shell that works across different systems
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # We're creating a custom instance of nixpkgs
        # This allows us to enable unfree packages like CUDA
        pkgs = import nixpkgs {
          inherit system;
          config = {
            allowUnfree = true;  # This is necessary for CUDA support
          };
        };

        # These helpers let us adjust our setup based on the OS
        isDarwin = pkgs.stdenv.isDarwin;
        isLinux = pkgs.stdenv.isLinux;

        # Common packages that we want available in our environment
        # regardless of the operating system
        commonPackages = with pkgs; [
          python313Full                # Python 3.13 interpreter
          python313Packages.virtualenv # Tool to create isolated Python environments
          figlet                       # For creating ASCII art welcome messages
          tmux                         # Terminal multiplexer for managing sessions
          zlib                         # Compression library for data compression
          git                          # Version control system for tracking changes
          curl                         # Command-line tool for transferring data with URLs
          wget                         # Utility for non-interactive download of files from the web
          cmake                        # Cross-platform build system generator
          htop                         # Interactive process viewer for Unix systems
        ] ++ (with pkgs; pkgs.lib.optionals isLinux [
          gcc                          # GNU Compiler Collection for compiling C/C++ code
          stdenv.cc.cc.lib             # Standard C library for Linux systems
        ]);

        # This script sets up our Python environment and project
        runScript = pkgs.writeShellScriptBin "run-script" ''
          #!/usr/bin/env bash
          
          # Activate the virtual environment
          source .venv/bin/activate

          # Create a fancy welcome message
          REPO_NAME=$(basename "$PWD")
          PROPER_REPO_NAME=$(echo "$REPO_NAME" | awk '{print toupper(substr($0,1,1)) tolower(substr($0,2))}')
          figlet "$PROPER_REPO_NAME"
          echo "Welcome to the $PROPER_REPO_NAME development environment on ${system}!"
          echo 

          # Install Python packages from requirements.txt
          # This allows flexibility to use the latest PyPI packages
          # Note: This makes the environment less deterministic
          echo "- Installing pip packages..."
          if pip install --upgrade pip --quiet && \
            pip install -r requirements.txt --quiet; then
              package_count=$(pip list --format=freeze | wc -l)
              echo "- Done. $package_count pip packages installed."
          else
              echo "Warning: An error occurred during pip setup."
          fi

          # Check if numpy is properly installed
          if python -c "import numpy" 2>/dev/null; then
            echo "- numpy is importable (good to go!)"
            echo
            echo "To start JupyterLab, type: start"
            echo "To stop JupyterLab, type: stop"
            echo
          else
            echo "Error: numpy could not be imported. Check your installation."
          fi

          # Create convenience scripts for managing JupyterLab
          # Note: We've disabled token and password for easier access, especially in WSL environments
          cat << EOF > .venv/bin/start
          #!/bin/sh
          echo "A JupyterLab tab will open in your default browser."
          tmux kill-session -t jupyter 2>/dev/null || echo "No tmux session named 'jupyter' is running."
          tmux new-session -d -s jupyter 'source .venv/bin/activate && jupyter lab --NotebookApp.token="" --NotebookApp.password="" --NotebookApp.disable_check_xsrf=True'
          echo "If no tab opens, visit http://localhost:8888"
          echo "To view JupyterLab server: tmux attach -t jupyter"
          echo "To stop JupyterLab server: stop"
          EOF
          chmod +x .venv/bin/start

          cat << EOF > .venv/bin/stop
          #!/bin/sh
          echo "Stopping tmux session 'jupyter'..."
          tmux kill-session -t jupyter 2>/dev/null || echo "No tmux session named 'jupyter' is running."
          echo "The tmux session 'jupyter' has been stopped."
          EOF
          chmod +x .venv/bin/stop
        '';

        # Define the development shell for Linux systems (including WSL)
        linuxDevShell = pkgs.mkShell {
          # Include common packages and conditionally add CUDA if available
          buildInputs = commonPackages ++ (with pkgs; pkgs.lib.optionals (builtins.pathExists "/usr/bin/nvidia-smi") cudaPackages);
          shellHook = ''
            # Set up the Python virtual environment
            test -d .venv || ${pkgs.python311}/bin/python -m venv .venv
            export VIRTUAL_ENV="$(pwd)/.venv"
            export PATH="$VIRTUAL_ENV/bin:$PATH"
            # Customize the prompt to show we're in a Nix environment
            # export PS1='$(printf "\033[01;34m(nix) \033[00m\033[01;32m[%s@%s:%s]$\033[00m " "\u" "\h" "\w")'
            export LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath commonPackages}:$LD_LIBRARY_PATH

            # Set up CUDA if available
            if command -v nvidia-smi &> /dev/null; then
              echo "CUDA hardware detected."
              export CUDA_HOME=${pkgs.cudatoolkit}
              export PATH=$CUDA_HOME/bin:$PATH
              export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
            else
              echo "No CUDA hardware detected."
            fi

            # Run our setup script
            ${runScript}/bin/run-script
          '';
        };

        # Define the development shell for macOS systems
        darwinDevShell = pkgs.mkShell {
          buildInputs = commonPackages;
          shellHook = ''
            # Set up the Python virtual environment
            test -d .venv || ${pkgs.python311}/bin/python -m venv .venv
            export VIRTUAL_ENV="$(pwd)/.venv"
            export PATH="$VIRTUAL_ENV/bin:$PATH"
            # Customize the prompt to show we're in a Nix environment
            # export PS1='$(printf "\033[01;34m(nix) \033[00m\033[01;32m[%s@%s:%s]$\033[00m " "\u" "\h" "\w")'
            export LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath commonPackages}:$LD_LIBRARY_PATH

            # Run our setup script
            ${runScript}/bin/run-script
          '';
        };

      in {
        # Choose the appropriate development shell based on the OS
        devShell = if isLinux then linuxDevShell else darwinDevShell;  # Ensure multi-OS support
      });
}

Enjoy!


Gemini’s Take

Title and Headline Ideas:

  • Titles:
    • Darwinix: Cross-Platform Python with Nix Flakes Made Easy
    • Nix + Pip: The Data Scientist’s Dream on Any OS
    • Unlock Python Freedom: Darwinix and the Power of Nix
    • Darwinix: Reproducible Python Environments with Nix.
  • Headlines:
    • “Stop Fighting Your Environment: Seamless Python Development with Nix Flakes”
    • “Cross-Platform Data Science? Darwinix Delivers with Ease.”
    • “Nix and Python: Get the Best of Both Worlds with Automatic Setup.”
    • “GPU-Powered Python: Darwinix Simplifies CUDA on Nix.”

My (the AI’s) Opinion:

This article presents a very practical and useful solution for developers who struggle with environment consistency across different operating systems. The author clearly addresses the common pain points of setting up Python environments, especially for data science, and leverages the power of Nix to create a reproducible and portable solution. The inclusion of CUDA support and the focus on simplifying pip install enhance its utility. The step by step instructions within the nix file, and the start and stop scripts are very useful. Overall, it’s a well-written and valuable resource for anyone looking to streamline their Python development workflow.