Isolating Dependencies for QEMU on OS X for My Tiny Linux Distribution

by Mike Levin SEO & Datamaster, 11/21/2012

Note: If you’re looking for a Linux distro that’s really designed to run with QEMU, download Levinux. It’s really only text-based Linux Server, but it will run with a double-click from your Mac, Windows or Linux desktop with no install.

I am fighting off a funk. Maybe its the holidays. Maybe its the accumulating effects of my previous post. Things are in flux, and my responsibilities are stuck in the past. I need to forge forward. I do have one piece of real work I have to do today, which is suggesting some topics for a monthly SEO strategy meeting coming up this Tuesday. Beyond that, the day is wide open, and I have to use it to keep my sanity. So, I’m taking another crack at moving my unique Linux distribution, Levinux, forward.

The precise reasons for choosing the work I choose to keep my sanity is a bit unclear, but my unconscious mind is pushing me relentlessly in this direction. It is the one thing that my unconscious mind consistently keeps corraling me back to: a tiny linux virtual machine that runs anywhere. People reading this might imagine me going off into some sort of fugue state in which my-body-knows-better than my mind, and I just sort of mechanically figure out what needs to be done next for maximum effect.

My own Linux distribution, Levinux, has been an on-and-off project for over a year now. It’s exactly the type of thing that I feel is a linchpin cornerstone groundwork for unlimited future awesomeness. It’s much like how the move to replatform myself onto Linux itself and learn Python several years ago saved my ass. But to get there, I jumped through so many hoops, and I have a very clear way to eliminate that hoop-jumping for other people, while simultaneously creating a nice momentum-preserving code execution platform for myself, and starting to address the huge societal issue of the new generation going technologically dumb due to consumerism and disposable electronics.

My attempt to create Levinux was born when I realized that the free and open source PC emulator named QEMU ran on every x86 platform, including Windows, Macs and Linux, and that the exact same boot-image could run on any of these platforms. It’s the great equalizer of standard plumbing execution platforms. By that, I mean there is great value in learning and mastering the old-school Unix/Linux-like environment that underlies everything these days, from Web/Internet development, to iPhones and Android. Whenever you’ve got something with a computer inside, chances are it’s running some variation on this, and there’s no ideal way to get started and actually preserve your work over long stretches of time (I’m talking decades). A runs-anywhere core bare-bones tiny linux virtual machine is exactly it.

However, the trick with QEMU is that its something of an octopus piece of software, with tendrils reaching out to various software libraries as dependencies on the “host” machine. It also uses special hardware capabilities as optimizations and accelerators, different on each host OS. Getting QEMU running on any one platform is not too tricky, but getting the exact same behavior on every platform is—as I’m learning—a deep learning process and an art-form. What’s more, I have some very particular criteria for the behavior once running—namely, I want to disable “graphics mode” and force the thing to run in old-school text mode, which is surprisingly trickier than one would think.

I had great early successes with Levinux based an “pre-compiled” binaries. As free and open source software, QEMU is mainly distributed as uncompiled source code, and the organization that maintains the code doesn’t really make a point of providing ready-to-run versions of the software. People around the planet take it upon themselves to compile QEMU and distribute the binary on their own, which is riddled with issues and problems. None-the-less, I cobbled together a magical cocktail of QEMU binaries and their dependencies to make a tiny portable Linux that would run with a double-click no matter what host OS you were on. But a magic cocktail was not suitable for distribution, or to form the bais of my grand master plan. For that, I needed a little more sanity and control over what constituted Levinux than a serrindipitous cocktail of software found all over the net. If things are working correctly, I want to know why and have a good plan to keep advancing it.

And so began my journey of compiling my own QEMU binaries. I’m not really a C programmer, nor was I a big compiler-of-Linux-software person. I tend to shy away from software dependency-hell, because its not a logical use of my time. Remote package manager software (RPM), like the Debian repository that automatically resolves dependencies pooling the knowledge of the entire Linux distro community is a much more sensible way to go, if you’re not a C programmer or into the joys of mixing your own linux distro’s. But the Levinux challenge kept steering me down this rather scary route. If I wanted to have the same up-to-date version of QEMU on each platform, have it run in non-graphics mode, and truly understand, isolate, and make-portable the dependencies, I had to compile QEMU for each platform.

Compiling QEMU equivalently to support text-only mode is easier said than done. There is first the series of standard dependencies that are required for compiling QEMU—mostly amounting to the GNU C standard library. On Windows there is also the added difficulty of it not being a Unix-like environment at its core, the way Mac OS X is, and the common route to get it that way, CygWin, burdens the resulting binary with such a heavy dependency that any hope of a lightweight portable binary Linux is out of the question. On Windows, you have to discover MinGW, which stands for Minimum GNU for Windows, which allows you to compile binaries with as few dependencies as possible—namely, a file that’s been distributed with Windows since XP for Visual Basic.

After getting to the point where I could merely compile QEMU on every platform, I had to deal with the fact that trying to enable text-only mode required its whole own set of libraries, and once again, it was different on Windows than it was on OS X and Linux. OS X and Linux went fairly easily compared to Windows. Linux went most easily of all, in great part due to the easy way of installing dependencies through the RPM Debian repository apt-get system. Missing C standard library? Just fire off a command: sudo apt-get install glibc. Need the GNU toolchain? sudo apt-get install build-essential. OS X was a bit harder. I had to choose an RPM system, and after an unsuccessful attempt at MacPorts (I got tired of waiting 1/2 hour for small programs), and switched to Homebrew, it went more smoothly. One command installs Homebrew: ruby -e “$(curl -fsSkL raw.github.com/mxcl/homebrew/go)”. Oops, something called pkg-config is required, so one command: brew install pkg-config. And finally the C standard library, one command: brew install glib. And that’s it. The mysterious text-only ncurses libraries seemed to just take care of itself on both Linux and OS X. Not so on Windows, which was something of a nightmare in comparison.

On Windows, I tried compiling native (on a Windows machine), then cross-compiling from Linux, and then back to native. To do this, I had to discover MinGW (the GNU toolchain), figure out the MinGW Shell, install Python and then set the path in the MinGW shell to Python: export PATH=”.:/usr/local/bin:/mingw/bin:/bin:/c/python27:/c/python27/dlls:$PATH”. Next, I followed the instructions at http://wiki.qemu.org/Hosts/W32 and manually downloaded GLib Run-time, GLib Development, gettext-runtime, unzipped the files, and manually drag-copy the files into location in the MinGW file structure, which in itself was a big trick to figure out. And finally, I needed to find a version of the text-only library, called ncurses in all other situations, but called pdcurses on the PC, from http://sourceforge.net/projects/pdcurses/files/pdcurses/ but I had to use version 3.3 from 2007 in order to not get a screen full of question marks when I tried running the compiled binary. Those files had to be drag-copied into place too. Oh wait! It wasn’t just as easy as figuring out a working version of pdcurses and putting the files in place. Bart van Andel from the MinGW mailing list told me how to edit curses.h which ended up online here http://lists.gnu.org/archive/html/mingw-cross-env-list/2012-10/msg00006.html. Is that all? No, there is one more impossible-to-know dependency that had to be resolved, involving intl.dll which is documented here http://virtuallyfun.superglobalmegacorp.com/?p=1622 thanks to some old-schooler that calls himself neozeed. Gotcha, gotcha, gotcha, gotcha, oops! It’s only black & white. So be it. Compiling QEMU in text-only mode on Windows is like solving the Rubik’s Cube.

Pshwew! Okay, that brings me to where I am right now. I’ve actually got Levinux booting with a double-click from Linux, Windows and OS X. There’s a whole other story here, which I’ll document separately having to do with script files, directory arrangement and the Mac OS X application bundle package trick—whatever you want to call it. Apple’s not consistent in its terminology for this wonderful bit of magic. But what I have to do now is isolate library dependencies for each platform, and move those dependencies into the directory of the QEMU executable, and hope that the executable knows to look for its dependencies in the folder its being run from first. If not, I’ll have to use wrapper scripts (which I’m using anyway) to change where the executable should look for libraries. I know that trick is doable in Linux and OS X. I already figured out the dependencies for Windows, as MinGW installed QEMU somewhere strange, and I had to copy over all the libraries to get it to run. Then finally, I’ll have to do some testing, probably using fresh installs of Windows and Ubuntu in VMWare Fusion on the Mac. Since the Mac isn’t so easy to run in virtual mode, I’ll have to boot from a USB stick to get a fresh Mac OS environment.

Let’s tackle each one of these challenges. We need to know dependency requiremnts. That’s sort of a big deal. On Linux, the trick is the ldd command. After a bit of research, I see that /usr/bin/otool -L is the equivalent on OS X. Here is it’s output:

MyMac$ /usr/bin/otool -L qemu-system-i386 qemu-system-i386: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.12.0) /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0) /usr/local/lib/libgthread-2.0.0.dylib (compatibility version 3201.0.0, current version 3201.4.0) /usr/local/lib/libglib-2.0.0.dylib (compatibility version 3201.0.0, current version 3201.4.0) /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0) /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0) /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libsasl2.2.dylib (compatibility version 3.0.0, current version 3.15.0) /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 19.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 45.0.0) /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.11.0) /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1187.34.0)

Hmmmm. Very interesting. On a Mac, /usr/local does not exist by default. Therefore, ANYTHING located in /usr/local as a dependency is a Homebrew thing.

So, only these 3 libraries need to be moved into location for the Mac distribution:

/usr/local/lib/libgthread-2.0.0.dylib (compatibility version 3201.0.0, current version 3201.4.0) /usr/local/lib/libglib-2.0.0.dylib (compatibility version 3201.0.0, current version 3201.4.0) /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)

…and the mystery of an external ncurses library not being required on the Mac is solved by this line:

/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

Ooookay, so I cd into /usr/lib/local, and fire off these commands:

cp libgthread-2.0.0.dylib ~/Dropbox/ cp libglib-2.0.0.dylib ~/Dropbox/

…and then I cd into /usr/local/opt/gettext/lib/ and fire off:

cp libintl.8.dylib ~/Dropbox/

Okay, then I open my dropbox folder and drag-copy these three files into the folder that contains my Mac QEMU binary… oh wait! It looks like there’s a specific share folder in RunOnMac.app/share/qemu. I’m going to but the lib files in there. It looks like this is where QEMU looks to retrieve everything shared. Anyway, how to test now on Mac? I’m tempted to uninstall Homebrew or even those particular files, but this is my compile environment now, and I don’t want to set it up again. Okay, so just rename /usr/local to something else. Done. Broke. It’s very explicitly telling me:

dyld: Library not loaded: /usr/local/lib/libgthread-2.0.0.dylib Referenced from: /Users/miklevin/Dropbox/Levinux/RunOnMac.app/mac/bin/qemu-system-i386 Reason: image not found Trace/BPT trap: 5

And so I try the LD_LIBRARY_PATH trick to “wrap” the executable, with the command, which for OS X is actually LD_LIBRARY_PATH:

do script “DYLD_LIBRARY_PATH=” & myPosixFolder & “RunOnMac.app/mac/share && export DYLD_LIBRARY_PATH && resize -s 25 80 && “ & myPosixFolder & “RunOnMac.app/mac/bin/qemu-system-i386 -nographic -curses -kernel “ & myPosixFolder & “RunOnMac.app/vmlinuz -initrd “ & myPosixFolder & “RunOnMac.app/core.gz -hda “& myPosixFolder & “RunOnMac.app/linux-0.2.img -append \“quiet\””

And voila! I have isolated the dependency for QEMU for OS X. I have also done it for Windows. Now, only Linux remains—which should be easy due to the ldd program and applicability of the same LD_LIBRARY_PATH trick. But it is 5:45 before the day of Thanksgiving. So I will give thanks I got this far. Success assured. This truly was food for the soul today, and helped me fight the funk.