One of the handy features in Mac OS X's universal binary support is the ability to easily compile a binary for multiple architectures with one command. This simplifies the process of producing a binary for multiple devices, avoiding the need to run the build multiple times and manually stitch together the results.
For example, one could build a mixed armv7, armv7s, and armv6 binary -- with thumb enabled only for armv6 -- with the following command:
gcc -arch armv7 -arch armv7s -arch armv6 -Xarch_armv6 -mthumb
In reality, the compiler is still being run multiple times; an arch-aware driver handles automatically running the compilers, writing the build results to temporary files, and then stitching them together with lipo(1). In the case of clang, the code to do this actually lives in clang itself. However, in the case of gcc, Apple implemented a front-end driver that sits in front of the actual gcc executables, called driverdriver.c. A similar driver is also supplied for as(1); it handles calling out to the correct assembler, but only supports a single -arch flag.
In implementing FatELF on Haiku, I needed the functionality provided by these drivers. Unfortunately, they were written somewhat non-portably for Mac OS X, and in the case of the gcc driverdriver.c, relied on internal GCC APIs that have been removed in later versions of GCC.
To solve this, I went ahead and wrote new front-end drivers that work with modern gcc(1) and as(1) versions. I implemented them as part of the Haiku fatelf-tools project, but they're meant to be portable to multiple hosts, and should be straight-forward to hoist out for other purposes. These could be used, for example, to support building universal binaries using more recent gcc releases on Mac OS X, where the traditional driverdriver.c cannot otherwise be used (due to reliance on GCC APIs that have now been removed).
If you want to review the initial implementation, the drivers can be found in my haiku-buildtools repository: fatelf-gcc.c, fatelf-as.c.
With these tools, I can build multi-arch Haiku FatELF binaries using the standard GCC toolchain:
landonf@lambda:fatelf> ./fatelf-gcc -arch i386 -arch x86_64 -c foo.c -o foo.o landonf@lambda:fatelf> ./fatelf-info foo.o foo.o: FatELF format version 1 2 records. Binary at index #0: OSABI 0 (sysv: UNIX System V) version 0, 32 bits Littleendian byteorder Machine 3 (i386: Intel 80386) Offset 4096 Size 547 Target name: 'i386:32bits:le:sysv:osabiver0' or 'record0' Binary at index #1: OSABI 0 (sysv: UNIX System V) version 0, 64 bits Littleendian byteorder Machine 62 (x86_64: AMD x86-64) Offset 8192 Size 799 Target name: 'x86_64:64bits:le:sysv:osabiver0' or 'record1'
One of the features that I think has been critical to Apple's ability to leverage improvements in underlying CPUs -- as well as weather migrations to completely new architectures -- has been their Universal Binary Format.
The concept is fairly simple, and dates back to NeXT, where they targeted a plethora of available platforms. Put succinctly, a universal binary is composed of:
For such a seemingly simple idea, the value is huge: universal binaries simplify the user experience for application users and developers alike.
From the user's perspective, they aren't forced to choose between x86-32 and x86-64 binaries depending on what version of the OS they happened to install. 32-bit and 64-bit Macs can use the same installation media, and web sites containing application downloads don't need to list two (or more) different downloads for each version. When new iPhones are released with updated ARM revisions, older binaries that are forwards-compatible can keep running, and newer binaries can be selected when supported by the hardware, maximizing the user (and developer's) ability to seamlessly take advantage of improvements in hardware revisions.
At runtime, the OS can avoid loading a duplicate set of libraries for binaries that are forwards-compatible -- for instance, ARMv6+hardfloat binaries can be linked against ARMv7 libraries on the iPhone -- and are. When hosting plugins, the hosting application can automatically 'downgrade' to the lowest common denominator ABI by relaunching itself, thus allowing for gradual transitions between old ABIs and new ABIs -- this was done by Apple in System Preferences when the entire OS switched over to x86-64, allowing System Preferences to switch back to i386 inthe case where the user still had legacy plugins installed.
From a developer's perspective, the universal binary ecosystem makes it trivial to support building their application for multiple architectures. While testing still needs to be done on the native system, a binary can be built and linked for all supported architectures simply by supplying the appropriate -arch flags to the compiler, using the same set of libraries and headers. Supporting a new target from your existing desktop system is as easy as downloading an updated SDK and including the target architecture in your build. Binaries that are required for the build (such as a protobuf compiler) can be compiled universal and included in the build system for any supported build host.
Even on the App Store, where Apple controls distribution and in theory could automatically supply different versions of an application for different devices, Apple has continued to leverage universal binaries, to Apple's advantage. The simplicity of being able to synchronize a single binary with multiple phones remains, but in addition, Apple has been able to rapidly deploy new ARM hardware features and see immediate developer adoption, because the cost and implementation costs are so low. By comparison, Linux, which traditionally has not supported fat binaries, has had to weigh the advantages of adopting new hardware features with the cost of having to build and deploy a completely new graph of the OS and all available software for that OS.
Apple is largely unique in their use of the Mach-O binary format and fat binaries, but there has been an effort to bring fat binaries to Linux, and other operating systems that use the (much more popular) ELF binary format. Ryan Gordon -- who has been a driving force in porting numerous games to Mac OS X and Linux (including Quake III Arena) -- set out in 2009 to define the FatELF format, which provides support for fat binaries on ELF-using platforms. Ryan implemented a full suite of command line tools for assembling fat binaries, Linux kernel support, and the necessary patches for binutils, gdb, glibc, etc.
Using Ryan's specifications and tools, I went ahead and implemented initial FatELF support for the Haiku kernel, boot loader, and runtime loader. With this work -- assuming the patches are integrated upstream -- Haiku will be able to use universal binaries for the kernel, kernel modules, shared libraries and executables.
The next step will be to extend these additions to the developer tools, with the goal of making it seamless to produce FatELF binaries using the same approach that Apple has used for Mac OS X and iOS. Once completed, I can begin to integrate the new toolchain into the build system, and start on the goal of building a fully universal 32-bit/64-bit Haiku installation.
If you're curious, you can find my FatELF-enabled branch here.
Haiku Alpha 4 was recently released, to the excitement of everyone. Or, at least those of us that maintain a strong sense of nostalgia for pervasive multithreading and yellow window tabs.
If you feel the urge to satiate your nostalgia by hacking on Haiku under VMware, you'll find that the current alpha does not fully support the VMware SVGA display, or mouse integration. One solution would be to use VirtualBox, which has first-class Haiku support mainlined into their repository. Unfortunately, at least from my experience, VirtualBox does not have first-class Mac OS X support, and tends to crash my machine.
Fortunately, however, there are actually VMware mouse and display drivers in the Haiku repository; you can either build and install these into your existing installation, or you can build a custom installation image including the drivers -- it only takes a few minutes to build the full OS.
To build a custom image, follow the directions provided in the Building Haiku documentation to configure your checkout and build the standard cross toolchain. Once that's done, you'll need to create a build/jam/UserBuildConfig file and add the following lines:
AddDriversToHaikuImage graphics : vmware ; AddFilesToHaikuImage system add-ons input_server filters : vmware_mouse ; AddFilesToHaikuImage system add-ons accelerants : vmware.accelerant ;
If you build a normal image as documented in Compiling Haiku for x86, the in-kernel and userspace VMware drivers will be included. Note, however, that without full guest additions, enhanced guest features will not be available: window resizing will not automatically adjust the guest resolution, copy+paste will not be functional, shared folders are not supported, etc.
Some months ago, I picked up a an Aeon Labs Z-Stick Series 2. The Z-Stick is USB-equipped Z-Wave controller, using CP210x chipset to provide access to the Z-Wave module's UART over USB. The official driver from Silicon Labs simply didn't work at all, timing out on any read or write to the device. So I decided to take a crack at writing my own driver.
The driver works great for my purposes -- I can turn my lights on and off without ever leaving my desk. Awesome, right?
The bad news? That's where I'm going to stop, at least for now. Between poor documentation, ambiguously defined behavior, and more layers of indirection than I can reasonably decipher in a spare afternoon, I've had my fill of IOKit for a while. Of the 1.2k lines of code in the driver so far, maybe 40 LoC actually has anything to do with talking to the CP210x; the rest is all IOKit-required boilerplate.
In my experience, third-party USB serial drivers for Mac OS X tend to be extremely buggy. Most of the drivers I've tested panic during use, don't work at all, or fail under relatively standard usage such as unplugging the device while running. I previously assumed that the entire blame for this rested on the driver's authors. Then I tried to write a USB serial driver for Mac OS X, and gained a whole new appreciation for anyone embarking on this journey.
If I were to revisit this in the future, I'd like to spend some time abstracting out the driver into a generic re-usable USB serial layer that I could use to implement additional drivers. Most USB serial devices have nearly identical interfaces -- a bulk input pipe, a bulk output pipe, and actual control requests that vary between devices, but tend to provide similar functionality -- a good fit for centralizing all the complex IOKit state management and boilerplate in one place. This is what Haiku does, to pretty good effect: compare the roughly 138 lines of Haiku's implementation with my IOKit monster
If you'd like to use the driver -- or help with the remaining issues -- you can find the source available at the mac-cp210x project page. The additional work will be solving lifecycle issues around asynchronous device control messages and unplug/driver unload behavior, enabling the existing code to allow manual control over RTS/DTR lines, and implementing state updates on changes to the incoming control lines.
Otherwise, I'll probably revisit the code whenever spending a few weekends with IOKit sounds like fun again.