Levinux Released, But Help From QEMU Gurus Needed
by Mike Levin SEO & Datamaster, 01/10/2013
Note: After all that work, I’m temporarily giving up on compiling my own QEMU binaries. On any one platform, it’s doable. But across all three with color curses support is all but impossible in the current state. So, I’m using the binaries from the ProofOfConcept for now.
Okay, okay, okay, Levinux is finally released… sort of. This article releases Levinux, my multi-host tiny virtual machine derivative distribution of Linux to the clamoring masses who have been filling out my web form.
You can have my Proof of Concept version, based on a magical cocktail of binaries I found from the usual (and usually trusted) suspects around the net. It works great, and the Recipe system is there and working great. I’m also releasing the cleaner, but much earlier-stage LevinuxAlpha, which is built from my own compiled binaries using the latest QEMU and Tiny Core Linux, but the Recipe system isn’t there yet, and there’s a few rough edges on OS X (sometimes incompatible binaries) and Windows (only B&W text).
So what’s so special? My tiny virtual distribution of Linux runs with a double-click with no install from the desktop of a Mac, Windows or Ubuntu machine—pretty much any desktop or laptop computer based on semi-modern Intel or AMD processors. Watch the video to see what the heck I’m talking about:
So, why this article? This is a call-for-help to maybe Fabrice Bellard (the developer of QEMU) or other QEMU developers who might have time to look. This is a call-for-help maybe to Jason Stevens (the virtualization guru) or William McBrine (the pdcurses guy on GitHub), or any OS X developers who can help me troubleshoot incompatible binaries. This article and accompanying video is to clarify my final few remaining hurdles.
What’s my end goal here? I’m trying to make this robust little cross-host virtual platform for education and development that can last a lifetime, which makes the old-school Unix-like text-only environment accessible even to newbies. I think I’m connecting a few dots no one has before. I’m particularly inspired by the Raspberry Pi that does something very similar, but with real hardware (and great graphics). However, it is my belief that it is just so much easier to merely download a file and double-click an executable on whatever hardware you’ve already got, for a quick introduction to old-school Linux, on a platform that’s truly “viable enough” to set you on the path to becoming a future-proof high-tech superpower.
Where am I at? I had a lot of success making such a cross-host virtual platform with a magic cocktail of pre-compiled QEMU binaries collected from the usual suspects around the net—consisting of old versions of QEMU that I mixed and blended together and miraculously got it to work. I now call it Levinux “Proof of Concept”, and have decided to distribute it, so you can see where I’m headed with the Alpha. When you run it, the recipe will build a dropbear SSH server and the g-wan webserver. You can actually do great stuff with this.
The problem with this is that no matter how good I managed to get it to work, I have limited control over its future. It’s not good to rely on magic cocktails, so I also have been compiling my own QEMU binaries, rounding up necessary libraries and dependencies on each host platform. I have found that compiling my own QEMU binaries for every platform—particularly with the text-only curses interface—has been way harder than I expected, and you can find documentation of my struggles in the Related Articles at the bottom of this page.
Every platform has its issues. Linux doesn’t have a consistent way to double-click an icon to run a program. OS X binaries appear to not always be portable between different OS versions. And Windows… well, everything is a bit off because its not a Unix-derivative OS, so you’re stuck with different libraries, like pdcurses instead of ncurses. But I’m close enough to release LevinuxAlpha. It doesn’t have the cool recipe stuff or smart virtual drive layout as the Levinux Proof of Concept but I need to release it to get some help.
Why is what I’m doing so hard? Well, for starters, since I want to teach Linux SERVER to newbies, there’s no reason for me to let the little black QEMU window to “grab” your mouse pointer—making it mysteriously disappear (something that’s done for swapping off mouse control to graphical VMs). When you’re just using a text interface, the disappearing pointer is very challenging for newbies. So instead, I try to turn off the SDL graphics and run in text-only mode, which eliminates this problem, and is better for thinking in old-school server terms anyway. In fact, I’m going to get newbies to SSH into the black box from terminal or PuTTY after a lesson or two. So, why have a pointer Venus Fly Trap lurking?
Why take the desktop QEMU approach at all? So much is on the cloud these days anyway, so why not start out this way on the Amazon cloud? Once again, the answer is control. I would have you burning your computer from sand if I could. I’ll get you on real hardware eventually. But a robust timeless “run-time environment” is a great starter approach. It’s similar to the Java JRE or the Microsoft CLR—but YOUR virtual machine (VM) actually is a little black window sitting on your desktop, and a full-fledged Linux server instead of a locked-down sandboxed vendor lock-in prison.
This little black Linux server window on your desktop is as portable and generic a place possible to run your code, while still being as timeless as possible. Anything you learn to do is applicable to micro-servers like Raspberry Pi or cloud servers like Rackspace or the Amazon EC2.
The skills you acquire following this path WILL make you a more powerful citizen. They MAY make you marketable. But they won’t make you stylish or trendy in tech circles. For that, you’ll have to go to a platform like .NET on Windows… Oh, whoops! See? You don’t want to be too trendy in tech. You want to be fundamental and timeliness, and that’s what I’m teaching here. This is the stuff that preps you to program robot armies and firefly drones—and not iPhone apps.
Why am I now asking for the help of some gurus? But before I can release Levinux for popular consumption to the world and start teaching, I need to work out the rough edges. I don’t think I’ll solve the Linux double-click ambiguity, but I think I can get a more broadly compatible Mac binary and color curses on Windows (it looks terrible compared to Linux and Mac). I’d prefer text-only curses mode, but I suppose I’d be happy if someone showed me how to turn pointer-grabbing totally off when I use the SDL library.
Once I get these couple of problems solved (help, please), I will release it to the public as Levinux—a learning platform and a way to do real productive work. It’ll be a “Busybox” embedded type Linux, which is where I think trendy tech fashions stop and the timeless fundamental info-tech underpinnings begin—just perfect for young-adults looking for that special technology edge, but are willing to forego some of the glitzy-ness of graphics for secrets that lie beneath the surface of all things. This stuff was around 40 years ago (sort of) and what will be around for the next 40 (probably).
What the hell are you talking about? Give me background. If this is all Greek to you, then here’s a quick history lesson. In the beginning (circa the ‘60’s), there was a multi-headed monster known as Multics—but Multics was too burdensome a beast, and no one person could control it. So, the still-living legend Ken Thompson made Unix somewhere in the 70’s, and got enough of the little details right to make it useful and timeless—but it was locked away in a cage built of intellectual property.
Then in the early 90’s, Linus Torvalds got Intel 386 hardware booting up using his free version of the Unix kernel called Linux—but this was only half an operating system, because kernels only boot. However, when Linux was combined with GNU, a free Unix-like command-set by Richard Stallman (which stands for “GNU is Not Unix”), a previously mythical new and free-roaming creature was let loose upon the world: the completely free and highly useful GNU/Linux operating system. Webmasters rejoiced!
But even GNU/Linux was too bloated for many uses, so a guy named Bruce Perens around 1995 crammed 80% of GNU’s goodies into 1% of the space, and called this little 500K Swiss Army Knife wonder called Busybox—one binary containing most of the commonly used Linux commands. 500K is about the size of an old 3.5” floppy disk. And Busybox got embedded into a lot of non-computer-like devices, because its so much of what everyone wants from Unix, but could fit into a high-tech wristwatch. And that was in 20001. Just imagine what you could cram Linux/Busybox into today! Firefly drones?
All the tricks that make Levinux tick And so that brings us to my effort: Levinux. Levinux is a Busybox Linux (Tiny Core Linux, to be precise) that runs on a the same free and open source emulator, QEMU, that Google adopted for Android development, but it is prepared in such a way that your same system can be run with a double-click from whatever desktop system you sit down at, so long as it’s a semi-modern x86-based Mac, PC or Linux (mostly Ubuntu). That’s most everything but tablets. This is just perfect for portable USB development systems—or even better, Dropbox.
Are you thinking that it doesn’t sound realistic to run a VM from Dropbox and incur all that network traffic as you work and a giant gigabyte file needs to be resync’d and resync’d? No problem! As it is, the file size of the Levinux VM is small, but through the magic of Tiny Core Linux’s brilliant architecture, the qcow2 virtual drive format, and some clever virtual hard drive partitioning, only teensy-tiny files need to be continuously synced through Dropbox while you work to keep your virtual machine synchronized in the cloud—contents of the /home/[username] directory, which is on its own separate “inflates like a balloon as-needed” qcow2 virtual drive, which generally is the only file that changes as you work. If you’re not using Dropbox, it still means you can make lots of copies of your VM to keep your work safe without wasting much drive space.
To pull off these tricks, I round up all required files into a single directory that is named with a .app extension, which magically makes the directory look like a single program on Mac OS X, and a double-click on the .app folder runs the programs inside according to a script. That’s a neat gift from Apple, and a trick I had been doing on the Amiga computer with Howard Harrison’s WBScript since the late 80’s. Finally, companies are getting a clue how to organize files.
But what about Linux and Windows? I put the Linux .sh bash file and the Windows .bat batch file OUTSIDE the .app directory, and they refer to the files within RunOnMac.app. Yep, that’s right. I put the Windows and Linux files right inside the Mac bundle! In this way, I bundle all required files in the Mac location, but present to the user only three possible things to click: RunOnMac, RunOnLinux and RunOnWindows.
To wrangle rogue dependencies like the GNU version of the C Standard Library, I use LD_LIBRARY_PATH and its equivalents in their respective launch-scripts to make them looked-for in the Mac bundle. This is actually not a bad practice because when the QEMU window is closed, so goes the temporary path-munging environment variable. I can make this VM very portable without requiring an install or admin rights on your host machine. People have done this before, but to it across different hosts for the same virtual Machine is, I believe, uncharted territory.
Persistence is key When you pop a Levinux USB drive into your Mac, PC or Ubuntu box, you double-click the one that corresponds to your system, and the virtual machine boots. Later on, if you pop out the USB drive and move even between different OSes, you can still run your VM with a double-click, and the same exact VM boots with your work intact and everything.
Combined with using a distributed revision control system like git or Mercurial to protect your code, your work starts to become unstoppable and loss-proof. It’s almost like a life-preserver boat as far as execution environments go. I considered calling this Lifeboat Linux or Longevity Linux, but ego won out in the end. If Linus can do it, so can I.
I believe the Levinux project itself has a lot of longevity and the potential to do a lot of good in the world by providing a clear transition path onto core Linux (non-graphics). Your app can be graphical in nature—just through webservers, HTML5, CSS and stuff. But back to Levinux longevity.
QEMU isn’t going away. x86 hardware isn’t going away. And even if both of them do, your knowledge is 100% transferable to ARM, MIPS, Cell, or any other hardware that also runs Linux. Focusing on short-stack barebones embedded Linux is a longevity skill-set play—or a precaution against sudden professional obsolescence as occurs with Adobe Flash, .NET and that sort of thing. Think of it however you will, but it’s going to be fun. I even hope to go as far making the building of Levinux an automated and open source project itself.
The Levinux recipe system Barebones? Short-stack? Embedded? What does that all mean? Well, when you boot Levinux with no “recipe” to cook-up a particular server application, it’s really just going to leave you at a login prompt with Internet connectivity and all the Busybox commands at your disposal, and unless you already know a lot, you can’t really do much. That’s the state of the LevinuxAlpha I’m distributing right now. The Proof of Concept distribution is quite a bit more advanced, running a nice sample-recipe so you can do some stuff with an SSH login and g-wan.
Levinux Recipes are merely text files (Bash script files) that can be loaded, edited and traded from the host operating system (Mac, PC, Ubuntu) where you are already comfortable. When a fresh Levinux download boots for the first time, it will automatically follow the recipe to build itself into whatever development environment or web appliance you like. It fetches the ingredients and cooks it up into a server. If you want to see it happen again, or are developing your own recipe, you can simply double-click a “Reset” script to put Levinux back in its pre-cooked virgin state, make some tweaks, swap out recipes, and try again.
I hope that being able to load and examine these recipes on the host environment (Windows, Mac) where you are comfortable will encourage people to immediately start hacking Levinux. Assembling a huge variety of specialized servers (web appliances) from should be a snap, once there’s a big enough library of copy-and-paste examples to borrow from.
All subsequent boots after a recipe has run occur much faster because the server has been built. So, this is a way to design and distribute JEOS (just enough operating system) virtual web appliances (dedicated webserver apps, etc.) in a small and efficient distribution. It’s like you’re distributing the seed, which you plant and watch it grow. It quite literally inflates, thanks to a clever bit of magic someone did with the qcow2 format. Google it. Pretty neat.
How does the recipe on the host system actually get onto the virtual machine for execution? QEMU cleverly has included a built-in TFTP server that allows file-transfer between the host operating system and the virtual machine. TFTP (trivial file transfer protocol) is a simpler version of FTP with so few requirements that it can be used while booting, and is often used to boot from the net instead of a hard drive. Levinux during first boot copies over the recipe and any included ingredients (security keys) from the host OS and starts following the recipe, fetching Python, Apache, git repositories, or whatever and builds (inflates) it into a working server.
Levinux stands on the shoulder of giants This is where I want to acknowledge more of the magic behind my Linux distribution. Aside from the obvious: Ken Thompson (Unix), Linux Torvalds (Linux), Bruce Perens (Busybox), and Fabrice Bellard (QEMU)—it’s time to acknowledge the person who mixed the very special original Linux distribution that I’m making a derivative from.
Did you ever hear of Damn Small Linux (DSL)—one of the most popular tiny Linux distributions (50MB) to ever hit the net? Ever wonder where its creators went? Well, while a guy named John Andrews created it, another guy named Robert Shingledecker made its inspired software repository and distribution system—and since 2008, Robert has been working with a community on a barebones distribution of Linux called Tiny Core Linux that seems to have perfected the beautiful but radical philosophy that blossomed under DSL.
Robert’s Tiny Core Linux is noteworthy for at least 3 reasons: first, it really is tiny, coming in at only 12MB for a fully bootable Linux system. Second, it uses this “incorruptible” approach that is a reaction to the virus, trojan, malware-ridden world—every time you reboot your system, it’s clean again. And thirdly, it’s got its own Debian-like software repository by which Python, Apache and all that app-stuff just plugs in as extensions which you can re-apply every time you boot.
To understand this, you have to think about all programs and application software as being sort of like transparent overlays laid on top of your core incorruptible system—and your data files are always either restored from a backup every time, or isolated onto entirely separate drive partitions that don’t mix with the OS. There’s a lot of flexibility in how you can configure things, but the upshot of it all is that the recipe scheme I envisioned is a match made in heaven for Tiny Core Linux… and so I chose it after a long search, and countless attempts to get Debian under 200MB—WAY too big for the tiny virtual machine I had in mind.
Old-school is new again So you see, this is a great and noble vision. I believe Levinux delivers on an as-yet unidentified and unfulfilled need that folks will realize in the coming months or years. It dovetails perfectly with the renaissance that’s sweeping across the planet, fueled by the Raspberry Pi, OLPC, Aakash and other endeavors designed to ignite young passion in technology so society doesn’t collapse on top of a bunch of consumer idiots who can’t keep the machines running. It’s a perfect transition off of the ever-less-viable Windows platform, and is something users of Windows, Macs and Linux can all have in common.
Of all the infinite servers that I could automatically build in my default distribution, the one that I am going to focus on is my “stack”: Python, Mercurial (a git-like distributed revision-control system written in Python), and vim (an old-school text editor that with practice lets edit and control text as if telepathically). With only these bits of software, you can write, secure, distribute, and use a vast variety of web-services-oriented applications. By that, I mean you can do many of the similar things as you could with the Google App Engine, an Amazon EC2 instance, or other cloud server. But you will have nearly 100% control (yep, I’m releasing the source to everything), and learn many things you wouldn’t if you started directly on the cloud—but which will ultimately give you deeper understanding and let you work better with the cloud. Once again, I propose to put you on the path to high-tech superpower.
This approach contrasts sharply with, say, CodeAccademy and all the other “coding-oriented” endeavors that have sprung up over the past couple of years, which focus on becoming some sort of a Mountain Dew-chugging Mark Zuckerberg buddy-dude who slams out code. Actually writing code and picking your programming language is just about the last thing you should do when first becoming technical. It is wiser to defer programming language decisions as long as possible, preventing yourself from becoming mentally handicapped by the preconceptions of this language or that. Read some Paul Graham. Read some Bruce Eckel. Read some Yukihiro Matsumot and Guido van Rossum. There are deep philosophies you need to discover and choose your alignment-with.
If you’re a newbie somehow managing to read through this, don’t commit right away to Python (which I will try to get you to do) or one of the popular new LISP dialects, or C, C++, Objective C, Java, Ruby, PHP, or the rest of the siren’s songs. I advise mastering the “hardware” first. And before going out and buying hardware, dip your toe in the shallow end using a virtual machine on the computer you work on every day anyway. But even on that computer, only allow yourself to be predisposed by the fundamental stuff that lies beneath every piece of information-technology: as generic and minimal as-possible Unix-like command environment (I.e. Levinux).
Help me make this thing real And if you are someone who read through this article, and are still as excited as I am, then please contact me. If you are someone who wants to LEARN, then put yourself on this list. It may take awhile, but I will reach out to all of you to let you know when the public-ready version of Levinux is out. I’ll have a project with a series of lessons ready.
However, if you are the kind of person I’m really trying to reach with this article, one of those pulsing-brain alien types who can help me troubleshoot my nuanced problems, then PLEASE reach out to me. Help me get over my last few stupid hurdles and bring this project in a very clean and exciting form to the world. Email me through my website contact form or miklevin at gmail, or one of the many other places I check daily, such as Twitter PM’s, Facebook, LinkedIn. Thanks, I look forward to hearing from all of you.
My plan from now is to:
- Make sure LevinuxAlpha and Levinux Proof of Concept are suitable for distribution to the people I’m asking for help from.
- Meticulously document versions, patches, the make process, known bugs and the like.
- Start reaching out to people who I think might have some insight. Ask them to forward this article along to people they know who might be interested and helpful.
- Figure out what existing communities to tap into.
- Consider creating a new community (and tribe?) dedicated to using or helping develop Levinux.