Landon Fuller

This was originally sent to the OpenJDK Porters Development List, and reproduced here so I can shamelessly call for volunteers. If you've a desire to work on Mac OS X Java Integration, your help is always appreciated. Send me an e-mail, or join the OpenJDK porters group!

AWT Native Toolkit

I've spent a bit of time working on a native AWT toolkit. My initial tasks have been:

• Running the Cocoa event loop on the 'main' (first) thread
• Getting a window on the screen
• Drawing to the window

The first two items are done -- It's possible to display a basic AWT window (CPanelPeer, CWindowPeer), but not draw to it.

In pursuit of drawing, I've begun work on implementing a MacGraphicsDevice and a MacGraphicsEnvironment -- my current plan is to leverage the existing java2d OpenGL pipeline, if possible. I'm still getting my head around the best approach to this, especially in relation to resolution independence on Leopard.

If you'd like to check out the work, it's available via the development mercurial repository:

http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/

The code is currently located in:

j2se/src/solaris/classes/sun/awt/mac/
j2se/src/solaris/native/sun/awt/mac/

I will probably need to move the Mac-specific code out of the j2se/src/solaris directory. One of my goals is to retain support for both the X11 and the CG/Cocoa AWT toolkits.

The build only AWT, you can cd to j2se/make/sun/awt and run:

make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=5 PARALLEL_BUILD_JOBS=5 \
ALT_HOTSPOT_CLIENT_PATH=/usr/local/soylatte16-i386-1.0/jre/lib/i386/client/ \
ALT_HOTSPOT_SERVER_PATH=/usr/local/soylatte16-i386-1.0/jre/lib/i386/server/

This will output a working JVM in j2se/build/bsd-i586. For the initial build, you make need to run make from top-level j2se/make directory.

Merge to BSD Repository

I've completed the merge of the Mac OS X port to the BSD Java project. With this work done, I've split Mac OS X development into two branches:

• Development - Integration with Mac OS X-specific technology (native AWT, etc.)
• Stable - A baseline Mac OS X port without additional native integration. Synchronized with the BSD repository.

If you'd like information on accessing the Mercurial repositories, check out the SoyLatte Project Page

General Release

After several weeks of using SoyLatte for my own development, and the testing of an exceptional group of early adopters, it's time to issue a 1.0 release. This release provides:

• A stable JVM, usable for real-world development
• The benefits of Java 6 performance enhancements
• X11 Swing (Nothing is perfect!)

Download Information

To download SoyLatte 1.0, please see the SoyLatte Project Page. If you are interested in contributing, please consider joining the OpenJDK Porter's Mailing List to say hello!

Changes since DP3

The only change in the 1.0 release is the acceptance of the -XstartOnFirstThread option. Nearly all outstanding changes have been merged upstream into the BSD Java project, and I have split development into two branches -- a stable branch, and an unstable branch. It's my intention (and I have begun work on) integrating Mac OS X-specific functionality within the unstable development branch.

Java Posse

The exceptionally nice guys from the Java Posse podcast called me up for an interview about Java 6 on Mac OS X. You can hear me sounding like a goof here.

I've also been awarded the first-ever Java Posse award. Free beer! Cheers to that!

The BSD Java Project deserves considerable credit as well. The Mac OS X Java port is dependent on the work of amazing people like Greg Lewis (FreeBSD) and Kurt Miller (OpenBSD), and I'll happily buy *them* a beer.

As the podcast states, if you're interested in working on native graphics for Mac OS X, sound, or anything else you think needs doing, check out the project page and join in!

OpenJDK Porters Group

I'm very happy to report that the OpenJDK Project's Porters Group has been established!

While the OpenJDK code base covers some mainstream CPU architectures and operating systems, there are many more platforms that are attractive targets for porting efforts for Java. This group exists to bundle and aid such efforts under the OpenJDK umbrella, and integrate them in the OpenJDK community through porting projects.

The BSD porting projects, led by Kurt Miller and Greg Lewis, have expressed interest in joining OpenJDK, as well as the Mac OS X porting project led by Landon Fuller.

This is fantastic -- many thanks to the OpenJDK developers for putting this together. I hope to move development work over to OpenJDK in the near(ish?) future.

Now with more Soy

Thanks to all for testing Release 2! I'm pleased to announce Developer Preview Release 3, which includes compatibility and feature improvements based on the latest round of community suggestions. I hope you don't mind the rapid release cycle -- I figured that it's important to iterate quickly with the initial testing releases.

New Features & Compatibility Fixes

• Support for Thread.setPriority() (maps to Mach thread priorities)
• Implemented support for gathering thread CPU time statistics via ThreadMXBean (Reported by Ralph Martin)
• JVM now ignores (and prints a warning) any Apple-specific -Xdock: JVM options (Reported by Fabrizio Giudici)
• Added fall-back support for the .jnilib library extension, if a .dylib can not be found -- Apple JVM compatability (Reported by Werner Randelshofer and Fabrizio Giudici)

Binaries, source, build, and contribution instructions are all available from the new SoyLatte Project Page -- my blog posts were starting to get a little long!

Community Process

I'd like to thank everyone who has downloaded, tested, and reported bugs. The result is "Developer Preview Release 2" -- Thanks!

If you haven't already, consider downloading a copy and running your software. Let me know how it works out for you!

Changes since Release 1

Features

• Support for 32-bit Mac OS X 10.4 Tiger.
• SoyLatte binary release.

Bug fixes

• Fix handling of setrlimit(RLIMIT_NOFILE, ...RLIM_INFINITY) on Leopard. Ensures that Java is able to raise the process file limit to the maximum allowed. (Reported by Tore Halset).
• Added official cacerts store from the BSD port, fixing SSL's x509 certificate validation (Reported by Tore Halset)
• Corrected the dylib path for loading the debugger library. Fixes java debugger (Fix suggested by Curtis Stanford, issue reported by Alexander Babaev)
• Add a missing LDFLAGS setting to the Darwin build configuration. Now correctly links JDK binaries, removing the requirement for setting DYLD_LIBRARY_PATH (Reported by Fred Jean and Nick Shanny)
• Key Leopard-specific uc_mcontext defines to MAC_OS_X_VERSION_MAX_ALLOWED >= 10.5. Fixes 64-bit builds on Tiger. (Reported by Ed Carrel).
• Added support for building against the 10.4 (and other) SDK using the DARWIN_SDK build setting
• Use DYLD_FALLBACK_LIBRARY_PATH for Java library loading. Avoids conflicts between Java libraries, and libraries enclosed in system frameworks. Fixes java2d binding against libGL.dylib

Fetching and Building from Source

Building

The build has been simplified. To build the 32-bit JVM:

cd control/make
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1

The following additional make options are available:

• BSD_OVERRIDE_ARCH: Set to amd64 to build the 64-bit JVM
• DARWIN_SDK: Set to target a specific Mac OS X SDK.

To build with extra debugging code and symbols, use the "debug_build" target.

To target Tiger, set the MACOSX_DEPLOYMENT_TARGET environmental variable to "10.4", and pass DARWIN_SDK=/Developer/SDKs/MacOSX10.4u.sdk as a make flag. The Tiger SDK currently must be build on Leopard, due to bugs in the 10.4 compiler's -mstackrealign code generation. This is on the long-term to-be-fixed list.

Fetching the source

Sources are available as a downloadable archive, or from a mercurial repository. To download the source code, you must be a licensee in good standing under the Java Research License. To ensure compliance, downloading the source requires authentication:

Username: 'jrl' and Password: 'I am a Licensee in good standing'

Download: jdk6_devpreview_r2.tar.gz (sig)

The development repository is also available via mercurial: http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/. The release tag is jdk6_devpreview_r2.

By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Binary Access

I am making available the SoyLatte binary release. (Red Hat already claimed IcedTea, and I drank a lot of double soy lattes while working on this. Plus, I think it's funny).

SoyLatte is based on the BSD Port of Sun's Java 6 JDK, and is made available under the Java Research License. JDK and Java are trademarks of Sun. Like RedHat, I want to make it exceptionally clear that while SoyLatte is a port of Java, it is not Sun's Java, JDK, or OpenJDK. Unlike IcedTea, SoyLatte is made available under the JRL. Please see below for a licensing discussion.

By downloading these binaries, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Downloads

Binaries are available for Mac OS X 10.4 and 10.5 (32-bit), and Mac OS X 10.5-only (64-bit). The soylatte directory can be placed anywhere on your system -- I chose /usr/local/soylatte16-amd64. Like other Java platforms, setting the JAVA_HOME and PATH environmental variables to point at these locations will work as expected.

32-bit JDK for Mac OS X 10.4 and 10.5: soylatte16-i386-r2.tar.gz (sig)

64-bit JDK for Mac OS X 10.5: soylatte16-amd64-r2.tar.gz (sig)

To ensure compliance, downloading the source requires authentication:

Username: 'jrl' and Password: 'I am a Licensee in good standing'

Feature TODO List

I'll start looking at all of these eventually, but feel free to jump in:

• Sound support. If you know CoreAudio, your help is much appreciated!
• Integration with the Mac OS X certificate store (x509)
• Support for setting mach thread priority
• Native aqua Swing (as opposed to X11). This is a big one, and not likely to be done anytime soon. The effort needs to be led by someone familiar with Cocoa, JNI, and Swing

JSR 223 JavaScript Support

Sun has pulled Rhino from the JRL and OpenJDK (GPLv2) sources, due to concerns regarding licensing. As a work-around, there is a BSD-licensed javascript JSR223 engine implementation, using Rhino, available via https://scripting.dev.java.net/:

We have built and tested with Rhino version 1.6 release 5. There is a Rhino based JavaScript engine bundled in JDK 6 (http://jdk6.dev.java.net). The JDK 6 bundled version is based on Rhino 1.6 release 2. Unlike JDK 6 bundled engine, all Rhino features (optimizer, E4X) are enabled in this version.

This seems to work as a drop-in replacement for Sun's sun.org.mozilla code. Here's JavaScript Hello World using Rhino 1.6r5 and the Scripting package:

java -cp ~/Downloads/jsr223/javascript/build/js-engine.jar:/tmp/rhino1_6R7/js.jar:. EvalScript
Found engine factory: com.sun.phobos.script.javascript.RhinoScriptEngineFactory
Found engine factory: com.sun.phobos.script.javascript.EmbeddedRhinoScriptEngineFactory
Hello, World

I am looking into the feasibility of integrating this solution.

Licensing

The Mac OS X work is based heavily on the BSD Java port, which is licensed under the JRL. The BSDs develop Java under the JRL; FreeBSD has negotiated a license with Sun to distribute FreeBSD Java binaries based on the JRL sources.

As the Mac port stabilizes, I am merging my work upstream into the BSD port, and in turn, it is a goal of the FreeBSD Java project to merge their work into OpenJDK. I've signed a Sun Contributor Agreement in preparation for this, and an OpenJDK Porters group has been proposed: http://thread.gmane.org/gmane.comp.java.openjdk.general/630

While the JRL makes this initial port possible, OpenJDK's GPLv2+CE licensing makes development and distribution far simpler. I hope to contribute this work to OpenJDK as soon as is feasible.

Submitting Bug Reports

There are bugs, and you're likely to find one. The most useful bug report is one that includes a simple, compilable reproduction case -- that gives me what I need to track down the really tricky bugs.

In submitting a bug, please include the following information:

• An e-mail address I can use to contact you if I have any further questions
• If at all possible, a simple, compilable test case that reproduces the issue
• The java hotspot error log (hs_err_pid.log), if the JVM crashed.

Bug reports may be submitted to landonf (at) macports (dot) org.

Native widgets via the Carbon SWT Implementation

The native Carbon SWT implementation appears to be working with the 32-bit Preview Release Mac OS JDK, without X11. This is the first step towards running SWT applications like Eclipse.

Non-X11 SwingWT Demo

The SwingWT project provides an implementation of most of the Swing API, using SWT to provide native graphics. While it's not a production solution, I thought it'd be fun to show a quick Hello, World demo with native Swing widgets, using SwingWT.

NOTE: You probably want to check out the latest release

Announcing Developer Preview Release 1

I'm pleased to announce the first Developer Preview Release of the open-source port of Java 6 to Mac OS X. This release includes support for 32-bit and 64-bit Intel machines running Mac OS X Leopard (10.5).

Port Status

(Nearly) everything up to and including Swing (X11) is functional. Sound is not currently supported.

While I've spent some time testing this release with my own projects, this preview release should be considered beta quality. The project is very much in need of additional community testing -- If you're hankering for Java 6, please give this a try! See below for information on submitting bug reports.

This work represents quite a few of my weekend and evening hours -- I hope you find it useful! Additionally, this port wouldn't be possible without building on the amazing FreeBSD, OpenBSD, and NetBSD porting work.

Tiger Support

While bugs in Tiger's compiler currently prevent the building of the 32-bit JDK, it should be possible to build the JDK on Leopard using MACOSX_DEPLOYMENT_TARGET and the MacOSX 10.4u SDK.

I plan on working on Tiger support over Thanksgiving. If you get this working before I get around to it, let me know!

Supporting PowerPC

Implementing PowerPC hotspot/interpreter support would require an incredible amount of work, and is not something I have planned. Most impressively, Gary Benson of Red Hat has implemented initial PowerPC interpreter support for OpenJDK. It's possible that this could be used to implement Mac OS X/PowerPC at some future date. I still have PowerPC Macs, so I'll look into this eventually.

Fetching the Sources

Sources are available as a downloadable archive, or from a mercurial repository. To download the source code, you must be a licensee in good standing under the Java Research License. To ensure compliance, downloading the source requires authentication:

Username: 'jrl' and Password: 'I am a Licensee in good standing'

Download: jdk6_devpreview_r1.tar.gz

The development repository is also available via mercurial: http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/. The release tag is jdk6_devpreview_r1.

By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Building the JDK

Building the full JDK will take quite some time -- up to an hour on my Mac Pro. You can tune the number of parallel jobs with the HOTSPOT_BUILD_JOBS and PARALLEL_BUILD_JOBS make tunables (eg, PARALLEL_BUILD_JOBS=4), which will speed things up considerably on multi-core machines.

There are two builds available for the 64-bit and 32-bit VMs: debug, and release. The debug JRE is significantly slower than the release version -- for day to day work, I suggest using the release build.

OpenMotif is a required build dependency. I've installed it using MacPorts. If you install it using Fink, or by hand, be sure to update the ALT_MOTIF_DIR build setting accordingly.

When the build is complete, a full JDK image will be available in javasrc_1_6_jrl_darwin/control/build/<build style>/j2sdk-image. You can copy/rename the j2sdk-image directory as appropriate (ie, /usr/local/darwin-jdk16-32bit). To use the new JDK, set the JAVA_HOME environmental variable and (optionally) add the j2sdk-image directory to the start of your path.

Lastly, I expect the 64-bit JVM to be faster than the 32-bit JVM, but I've yet to do benchmarks.

64-bit Release Version:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-amd64/lib/amd64/server \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1 BSD_OVERRIDE_ARCH=amd64

The JDK will be built in control/build/bsd-amd64/j2sdk-image

64-bit Debug Version:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-amd64-debug/lib/amd64/server \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1 BSD_OVERRIDE_ARCH=amd64 debug_build

The JDK will be built in control/build/bsd-amd64-debug/j2sdk-image

32-bit Release Version:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-i586/lib/i386/client \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1

The JDK will be built in control/build/bsd-i586/j2sdk-image

32-bit Debug Version:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-i586-debug/lib/i386/client \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1 debug_build

The JDK will be built in control/build/bsd-i586-debug/j2sdk-image

Submitting Bug Reports

There are bugs, and you're likely to find one. The most useful bug report is one that includes a simple, compilable reproduction case -- that gives me what I need to track down the really tricky bugs.

In submitting a bug, please include the following information:

• An e-mail address I can use to contact you if I have any further questions
• If at all possible, a simple, compilable test case that reproduces the issue
• The java hotspot error log (hs_err_pid.log)

Bug reports may be submitted to landonf (at) macports (dot) org.

Puzzle Pirates - A More Entertaining Hello World

Status Update

I've spent some more time working on porting JDK 1.6 -- adding support for Tiger (10.4), fixing stack alignment issues for both amd64 and i386, fixing build system issues, and more. On Leopard, the 32-bit JDK now builds to completion. As evidenced by the above "Hello, World", X11 Swing is working on the 32-bit JVM.

Work is still progressing on the 16-byte stack alignment issue. In the meantime, I've enabled the gcc -mstackrealign option to force stack alignment in compiled JVM code. This ensures that calls back into the JVM are properly aligned:

On the Intel x86, the -mstackrealign option will generate an alternate
prologue and epilogue that realigns the runtime stack. This supports mixing
legacy codes that keep a 4-byte aligned stack with modern codes that keep a
16-byte stack for SSE compatibility. The alternate prologue and epilogue are
slower and bigger than the regular ones, and the alternate prologue requires
an extra scratch register

To replace this temporary work-around, I'm currently working on fixing the call stack alignment in the HotSpot Interpreter. Currently most calls are properly aligned, with the JVM running until it hits a JNI call. The JNI code generation assumes that arguments can be pushed directly to the stack prior to calling the stub generator, making alignment more difficult. I have not started work on alignment in the c1 or c2 compilers. Due to -mstackrealign compiler bugs in Tiger, the JVM must be built on Leopard -- however, it should be possible to build a JDK to run on Tiger.

For 64-bit Leopard systems, I've fixed up some latent alignment issues in HotSpot (Leopard is very picky about checking alignment in dyld), and the code appears to work fine with a few caveats. I've only spent a few hours on this today, but 64-bit Java should be the most stable and easiest to port. There is a known issue in the AMD64 JVM that I have not had time to track down, occaisionally triggering the following assert:

 javasrc_1_6_jrl/hotspot/src/cpu/amd64/vm/frame_amd64.cpp:118
  assert((intptr_t) sp() <= (intptr_t) result,
           "result must >= than stack pointer");

As a temporary work-around, the x86_64 JVM appears to run fine (if slowly) in interpreter mode (-Xint).

Future Direction

As a FreeBSD-Java committer, I will be merging low-risk changes upstream to the FreeBSD Java Project (Code). Potentially destabilizing changes to Hotspot interpreter / compiler will not be merged until they've seen adequate review by the FreeBSD and OpenJDK teams.

There are tentative plans to merge the BSD Java changes into the OpenJDK project. This isn't something I'm directly involved with, but I have signed the Sun Contributor Agreement in preparation.

I am also very interested in external contribution -- this is a part time, after-work project for me! See below for code access.

In progress work:

• i386 JIT 16-byte intepreter and c1/c2 compiler stack alignment
• Outstanding AMD64 bug(s)
• Audio support (HAE implementation needs to be written using CoreAudio)
• Support for setting java thread priority (should be simple, just requires a implementation using the mach thread API).

Code Access

I am making the JRL-licensed source code available via a Mercurial repository. Since the OpenJDK project is moving to Mercurial, I thought I'd get on the bandwagon -- it is working out well. The repository is available at:

http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/

To download the source code, you must be a licensee in good standing under the Java Research License. To ensure compliance, the repository requires authentication -- the username is "jrl", and the password is 'I am a Licensee in good standing'. By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Currently, all FreeBSD JDK16 work is under the JRL. In the future, the work may be relicensed, with a copyright grant, for OpenJDK under the GPLv2.

This not something I like to do, but I request that all submissions assign copyright to myself, such that I have the ability to relicense/grant copyright for submission to the upstream projects. If anyone has a better idea of how to handle this, I'm very interested.

To build the x86_32 JDK:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-i586-debug/lib/i386/client \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1 debug_build

To build the x86_64 (amd64) JDK:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-amd64-debug/lib/amd64/server \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=4 PARALLEL_BUILD_JOBS=4 BSD_OVERRIDE_ARCH=amd64 JAVA_JVM_FLAGS=-Xint JAVAC_FLAGS=-Xint debug_build

Remove the "-Xint" flags to disable interpreter mode (and trigger the known issue in frame_amd64.cpp).

Note that the debug build is considerably slower than a production build, and interpreted mode is even slower!

If you're interested in assisting with IA32 stack alignment issues, the current patch is available here: patch-stack-align-darwin-v2.

Looking for a Job?

Lastly, a plug for my group at Three Rings. We're looking to hire Java Engineers (not just billing!) and FreeBSD Developer/Admins. The group rocks (I run it!), so feel free to drop me a line if you're interested.

NOTE: You probably want to check out the latest updates

Introduction

I've long wondered what it would take to get the FreeBSD Java Port running on OS X, so this weekend I spent a couple days getting Java 1.6 running on my x86 Leopard machine.

Weekend is over, and I can report partial success -- hotspot compiles, the jre mostly bootstraps, and Hello World runs. Anything complex appears to trigger stack alignment issues (Apple's i386 API requires a 16-byte aligned stack)

landonf@max:javasrc_1_6_jrl/control/build/bsd-i586> ./bin/java -version
java version "1.6.0_03-p3"
Java(TM) SE Runtime Environment (build 1.6.0_03-p3-landonf_04_nov_2007_00_06-b00)
Java HotSpot(TM) Server VM (build 1.6.0_03-p3-landonf_04_nov_2007_01_30-b00, mixed mode)

landonf@max:javasrc_1_6_jrl/control/build/bsd-i586> ./bin/java Hello
Hello World!

What's missing?

Missing pieces:

• i386 JIT 16-byte stack alignment issues
• Audio support (HAE implementation needs to be written using CoreAudio)
• Support for setting java thread priority (should be simple, just requires a implementation using the mach thread API).
• Browser plugin
• PowerPC or AMD64 support (PowerPC would require a considerable amount of work)

Additionally, there's no Aqua support -- Swing would require X11.

Source? Compiling?

The patch is based on the BSD 1.6 JDK patchset, which means you need to download Sun's JDK source, the FreeBSD patchset, AND the Darwin patches. Instructions on downloading the Sun source, and the FreeBSD patchset, are available from the FreeBSD JDK site

The Darwin patchset should be applied on top of the BSD patchset. By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License ("License") of the Java(tm) 2 SDK, and that your access, use and distribution of code and information you may obtain at this site is subject to the License. Download patch-jdk16-apple-alpha1.

Compilation requires OpenMotif, Nawk, and possiblity additional software -- I installed all the dependencies using MacPorts. To build, cd to control/make and run the following (tweaked appropriately):

env DYLD_BIND_AT_LAUNCH=1 \
DYLD_LIBRARY_PATH=<javasrcpath>/control/build/bsd-i586/lib/i386/client \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
DONT_BUILD_DEPLOY="YES" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=4 PARALLEL_BUILD_JOBS=4 

The build does not complete (yet), but a bootstrap JVM is built (and can be run from) ./control/build/bsd-i586/bin/java

Four months ago, Chris Evans, of Google's Security Team, released an advisory regarding a heap overflow in Sun's Java ICC (image) profile parsing code. (CVE-2007-2788). The release was cordinated with Sun, and an updated Java release (JDK 1.5.0_11-b03) was made available for Window, Solaris, and Linux.

Apple's Java runtime has not yet been updated, so I've gone ahead and written a run-time patch for my own use. If you'd like to use the patch too, you can download the source, or a pre-built binary. You'll need to install Application Enhancer to use the patch. Alternatively, you could simply disable Java in your browser to close the most likely vector.

The issue is due to an integer overflow that occurs when validating that an ICC header tag does not exceed the total length of the heap allocated profile data buffer; The comparison will overflow if the header declares an too-large tag size (See section 7.1 of the ICC.1:2004-10 specification for more information on the header tag format).

For a proof of concept, I've uploaded my regression pages here -- fair warning -- this link will crash an unpatched browser.

Update

Google Group's file hosting was giving Safari users trouble, so I'm now hosting the run-time patch locally.

NOTE: This post has been archived for historical purposes. The toolchain has advanced considerably, and Apple is planning to release their own SDK. I plan to hold out on further development, toolchain or otherwise, until it's released.

Please see iPhone Toolchain Project for up-to-date instructions.

To facilitate my own iPhone development, I've committed three new ports for the iphone-binutils project to MacPorts. Once installed, you're ready to compile Hello, World. Prior to installation, you'll need to acquire a copy of the iPhone root disk image ("Heavenly") and install its contents in /opt/local/arm-apple-darwin/heavenly. The image is required to provide the necessary libraries for linking cross-compiled iPhone binaries, and can't be re-distributed.

Extracting and Installing the iPhone Libraries

To start, download and decompress iPhone1,1_1.0_1A543a_Restore.ipsw:

 user@host> curl -O http://appldnld.apple.com.edgesuite.net/content.info.apple.com/iPhone/061-3538.20070629.B7vXa/iPhone1,1_1.0_1A543a_Restore.ipsw
 user@host> unzip iPhone1,1_1.0_1A543a_Restore.ipsw \*.dmg

This will extract two disk images: "694-5259-38.dmg" and "694-5262-39.dmg". The encrypted "694-5262-39.dmg" disk image contains the iPhone root. The decryption key for this image is stored in plain text within the "asr" binary on the corresponding "694-5259-38.dmg". To retrieve the key, run the following:

 user@host> strings 694-5259-38.dmg| grep "^[0-9a-fA-F]*$" | awk '{ if (length($1) == 72) print; }'

This should output a 72 character hex string, which you'll use as the decryption key.

In order to perform the decryption, you'll need modified 'vfdecrypt' -- a command utility for decrypting Mac OS X disk images. The source is available here. To compile, simply type "make" in the vfdecrypt-iphone directory. The provided version of vfdecrypt was slightly modified to support direct input of the private AES and SHA1 HMAC keys -- these are normally wrapped with a user-supplied passphrase (via 3DES-EDE), which is not available. vfdecrypt was written by Ralf-Philipp Weinmann, Jacob Appelbaum, and Christian Fromme.

Once you've build vfdecrypt, use it to decrypt the disk image:

 user@host> ~/vfdecrypt-iphone/vfdecrypt -i 694-5262-39.dmg -k <hex key> -o heavenly.dmg 

Now, mount the disk image and copy the contents to /opt/local/arm-apple-darwin/heavenly:

 user@host> open heavenly.dmg
 user@host> sudo mkdir -p /opt/local/arm-apple-darwin/heavenly
 user@host> (cd /Volumes/Heavenly1A543a.UserBundle && tar cf - .) | (cd /opt/local/arm-apple-darwin/heavenly && sudo tar xvf -)

Installing the Toolchain

To install the toolchain:

 sudo port install arm-apple-darwin-runtime

You should now be able to compile standard Unix software:

 CC=arm-apple-darwin-cc CPP=llvm-cpp ./configure --host=arm-apple-darwin

For more information on compiling your first GUI application, check out the UIKit Hello World

Today Apple released Security Update 2007-004. The update includes quite a few important fixes, and also includes a fix for a bug I stumbled across during the Month of Apple Bugs, while regression testing the patch for the Quicktime RTSP URL Handling Buffer Overflow Vulnerability.

While testing the RTSP fix, I had experimented with providing long HTTP URLs to the QuickTime Plugin, and caused a crash. At the time, I mistakenly assumed that the two bugs were the same -- it wasn't until after the RTSP issue was fixed that I looked more closely and submitted the issue to Apple.

In the end, Apple did all the hard work in tracking down the bug to Libinfo. They were reasonably communicative on status, and provided the opportunity to regression test their fix prior to release.

Introduction

At Three Rings, we use Bacula as a disk-based backup system. We originally implemented off-site backups by rsync'ing snapshots to an out-of-state server, but as the size of our backups headed towards terabytes, we started looking for a solution that didn't require installing a remote multi-terabyte RAID.

Amazon's Simple Storage Service (S3) provides access to Amazon's "highly scalable, reliable, fast, inexpensive data storage infrastructure" -- In brief, S3 provides a limitless amount of internet-accessible storage, accessed by a simple REST (or SOAP) API.

As a part of our (Three Rings) next project, Whirled, I implemented a relatively complete Java library for S3's REST API. Having the code on hand, it made sense to leverage both the code, and S3, for our off-site backups.

Streaming Data to (and from) S3

To back up multiple terabytes of data remotely, I had three requirements:

• The data must be streamed directly to S3. (ie, no local copies)
• S3 objects can not be larger than 5 gigabytes, so any data must be broken up into smaller blocks.
• Data structures must be future-proof; it must be possible to evolve the data format.

The result is S3Pipe, a small utility which implements piping of streams to and from S3:

landonf:s3lib> echo "hello, world" | s3pipe ... upload --stream helloworld

landonf:s3lib> s3pipe ... download --stream helloworld
hello, world

In other words, S3Pipe supports reading data from stdin and writing it to S3, as well as reading data from S3 and writing it to stdout. The implementation:

• Breaks input data into 5 megabyte blocks. An input thread places each block into a queue, and an output thread uploads them to S3, preventing starvation of the writer thread. The size of the block queue is limited -- once filled, the input thread will not read data until space becomes available.
• Stores streams as future-proof versioned data structures, with support for arbitrary metadata.
• Validates block checksums on both upload and download.
• Retries block uploads and downloads should a transient failure occur.

I hope that others will assist in incrementally improving the feature set; built-in public-key encryption & signing, simple HMAC support, even more robust transient/network failure recovery.

Download & Use

Both the S3 client library and S3Pipe are available under the BSD license, as a 1.0-beta1 release. This is new software, and I strongly recommend testing any backups -- that said, the code has nearly 100% unit test coverage, has been tested locally, and is in active production use. The primary reason for the beta designator is that I still haven't written documentation!

You can download the library source here (pgp signature). To build s3lib and s3pipe, you will need the Java JDK (1.5+) and Ant (1.7+, available via MacPorts). Inside the source directory, run:

ant dist

This will create dist/s3lib.jar and dist/s3pipe.jar. S3Pipe is a stand-alone jar -- it can be copied anywhere once built. To execute S3Pipe (and print command usage), run:

java -jar dist/s3pipe.jar

S3Pipe requires you to provide your AWS id and key in a seperate properties file. See 'test.properties.example' as an example.

If you'd like javadocs for s3lib, run:

ant javadoc

However, be aware that the API is still subject to change.

S3Pipe in action: S3 Dump and Restore

All of our servers at Three Rings run FreeBSD, and so I decided to use dump(8), with its' built-in support for UFS2 file system snapshots, to dump file system snapshots from our primary Bacula server to Amazon S3.

The dump command makes use of "-P", or "pipe command". When provided a -P option, dump(8) will execute the given sh script, and pipe its output to the child's stdin. The following command line is a big one: it dumps a snapshot of the /export file system, piping the output through gzip for compression, gpg for symmetric encryption, throttle to rate limit the output to 20 Mbps, and S3Pipe, to write the stream to S3.

/sbin/dump -a -P "gzip -c | gpg --symmetric --batch --passphrase-file key.txt | /usr/local/bin/throttle -m 20 \
| java -jar s3pipe.jar --keyfile s3secrets.properties --bucket example.backup.bucket --stream bacula-/export upload" -0Lu /export

Restoration is, of course, almost the same thing in reverse.

java -jar s3pipe.jar --keyfile s3secrets.properties --bucket example.backup.bucket --stream bacula-/export download |\
gpg --decrypt --batch --passphrase-file key.txt | gzip -cd | restore

I run the Engineering Infrastructure Team at Three Rings. We're an engineering-focused IT group, and we're responsible for everything from FreeBSD kernel development to development of our billing transaction/gateway system. We maintain several open source projects of our own, as well as donating time to external projects -- I implemented Bacula's network encryption (TLS) support for our use here at Three Rings.

We're a small company, and I have a strong personal investment in our success; I'd like to hire a lead web engineer who will bring solid engineering practices to our web sites and applications.

For more information, and if you'd like to apply, the job ad is posted here. Also, please feel free to send me any questions.

Howdy All. I've updated the plugin for 10.4.9/Security Update 2007-003. This update removes a number of patches for issues fixed by the Software Update:

• [MOAB-04-01-2007] - iLife iPhoto Photocast XML title Format String Vulnerability
• [MOAB-06-01-2007] - PDF Document Catalog Denial of Service Vulnerability
• [MOAB-20-01-2007] - Apple iChat aim:// URL Handler Format String Vulnerability
• Disk Image "Warn before opening" Support

The following patches remain:

• [MOAB-03-01-2007] - Apple Quicktime HREFTrack Cross-Zone Scripting vulnerability
• [MOAB-24-01-2007] - Apple Software Update Catalog Filename Format String Vulnerability (Security Update 2007-003 removes the document bindings in Software Update, but the actual bug still remains. This is not a critical issue.)
• [MOAB-26-01-2007] - Apple Installer Package Filename Format String Vulnerability
• [MOAB-27-01-2007] - Telestream Flip4Mac WMV Parsing Memory Corruption Vulnerability

You can download the binary release, or the source. GPG signatures are available for the binary and the source.

Today's patch includes fixes for three different issues:

• Apple Software Update Catalog Filename Format String Vulnerability
• Apple Installer Package Filename Format String Vulnerability
• Telestream Flip4Mac WMV Parsing Memory Corruption Vulnerability

Thanks to my friends William Carrel and Rosyna of Unsanity for their work on tracking down the bugs and assistance in implementing the patches.

You can download the source, or a pre-built binary. As always, you'll need Application Enhancer to use the patches.

Update 9:00 1/29/06: Modified the patch (to 27-1) to match the latest 10.4.8 version of the PowerPC Software Update.app, 2.0.5 (Intel is 2.0.4).

This release also removes a few patches:

• The patch for the QuickTime RTSP buffer overflow has been removed, so make sure you've installed the latest update via Software Update.
• The patch for Panic's Transmit has been removed. You should upgrade to the latest Transmit release.

Flip4Mac WMV Vulnerability

The ever-handy Flip4Mac WMV QuickTime components are susceptable to a buffer overflow due to an integer overflow when comparing two buffer lengths as signed values. The comparison results in a stack-based buffer overflow that is potentially remotely exploitable by a malicious WMV movie. Both Mail.app and Safari will inline WMV movies (I accidentally crashed Mail.app when I tried to attach a proof-of-concept to an e-mail I'd drafted).

This patch is more complex; nearly all symbols are stripped in Flip4Mac, so finding the vulnerable code requires locating a public symbol and then computing the offset to the vulnerable code, for both PowerPC and Intel. Due to the relatively short length of the function in question, I decided to completely re-implement it, using proper unsigned comparisons. You can peruse the implementation in Google Code's web interface.

Software Update Catalog Filename Format String Vulnerability

Software Update falls prey to a common format string vulnerability, passing the user-supplied file name to an alert panel. However, I'm not aware of any means of delivering a maliciously named file without requiring direct user interaction. The error dialog in question is only displayed when a file without the "sucatalog" extension is passed to Software Update (eg, "swutmp") -- the patch checks file extension, and escaps any '%' characters if the extension is not "sucatalog".

Installer Package Filename Format String Vulnerability

Installer is also susceptable to the same issue -- when an error is encountered while opening a file, Installer will pass the user-supplied file name to an alert panel. A zipped installer package will be automatically opened by Safari if "Open Safe Files" is enabled. The patch works by "context patching" NSRunAlertPanel() while within -[InstallerController openFile:withOptions:]. The guarded NSRunAlertPanel() passes it's msg argument as a vararg argument to the "%@" format string, and the patch is removed when -[InstallerController openFile:withOptions:] exits. A reference count is used to handle recursion.

Soon the month will draw to a close, and with it, The Month of Apple Bugs. Lest you think I've dropped off the map, I've set aside the weekend for addressing the new issues. Apologies for the delay -- it's been a busy work week!

I'm also pleased to announce a number of software updates for outstanding issues. The turn-around time on these updates is admirable.

Panic's Transmit - MoAB #19

The indubitably responsive developers at Panic have released an update to fix the ftp:// and ftps:// buffer overflow announced in the MoAB #19 advisory.

Rumpus FTP Server - MoAB #18

The good-natured fellows of Maxum Development Corporation have released an update to Rumpus, fixing the issues outlined in MoAB #18. Existing Rumpus 5.1 users may use the "Software Updates" feature in Rumpus to automatically download and install the update. All others can download the updated demo application.

0-Day Patches

Where a critical high-risk "0-day" issue exists, and is practical to patch, I plan on continuing to provide patches beyond the Month of Apple Bugs. The Java GIF Patch is the first of these "non-MoAB" fixes.

The MOAB Fixes group remains the primary point of coordination, and I expect to move this effort to a more official home than my own web log.

The latest MoAB issue is a format string vulnability in iChat. The proof of concept uses Safari's auto-handling of aim:// URLs to pass a 'aim:gochat?roomname' request to iChat.

The patch for the issue was adeptly implemented by William Carrel. His patch works by guarding -[ActiveChat showError: (NSString *) error] and escaping any format string specifiers in the error message. As always, patches are coordinated at the MOAB Fixes group if you'd like to participate.

You can download the binary, or the source. As always, you'll need Application Enhancer to use the patches.

Java Patch Update - Firefox

This release also updates the patch for the Java GIF Heap Overflow issue. The update expands the library path regexes to ensure that Firefox-launched JVMs are always patched, in addition to Safari.

Rumpus FTP Server - MoAB #18

The expeditious fellows over at Maxum Development also have a fix in the works for the MoAB #18 issue in Rumpus, and will be releasing an update in short order. Cheers for their fast turn-around on the issue.

Today's post contains a whole slew of goodness:

• A patch for the first non-MoAB bug: ZDI-07-005, Sun Microsystems Java GIF File Parsing Memory Corruption Vulnerability
• A patch for MoAB #19, multiple buffer overflows in Transmit
• Support for user confirmation before mounting DMGs
• A tool for fixing risky disk permissions -- BOM Shelter

More details follow, but for the impatient, you can download the source, or a pre-built binary. You'll also need to install the ever-handy Application Enhancer. See the FAQ for more information. As always, these third party patches are unsupported and without warranty. Please take a look at Anatomy of the Runtime MoAB Patches for more information on the history of 3rd party patches, and their risks and benefits.

MoAB #19: Panic's Transmit

The 19th MoAB advisory documents heap-based buffer overflows in Transmit's handling of ftp(s) URLs. Don't Panic, however -- we have a patch, and Panic is hard at work on an update to fix the issue. The illustrious William Carrel is the author of today's patch.

He tracked the issue down to the -[FTPConnectionWorker _connectTo:port:user:password:initialPath:localPath:redial:listFiles:encoding:] method, where strcpy() is used to copy the user, password, and host to 64 byte arrays. William's patch checks the length of the host, user, and password arguments; if they're too long for the fixed length buffers, the arguments are replaced with dummy, known-safe strings.

Java Extra Credit: Patching the Java GIF Buffer Overflow

This is the first patch for non-MoAB bug, and I'm interested to hear what everyone thinks about providing "0-day" patches to other critical issues.

On January 16th, the Zero Day Initiative released an advisory on a heap overflow in Java's GIF image decoder. The vulnerability allows the execution of arbitrary code within the JVM via any java applet. Sun has released an update, and if you're running a Sun-supported operating system (Linux, Windows, or Solaris), I strongly suggest upgrading.

On the Mac, however, Apple is responsible for porting and maintaining the Java Runtime Environment. It is apparent from my observations of the FreeBSD Java project that preparing and testing a new Java release is an expensive and time consuming process, least of all due to the extensive testing required of Sun licensees. With some prodding from Eric Hall, a friend and security consultant, I decided to release a temporary patch for the issue until Apple can release an official update.

The patch works by guarding the vulnerable Java_sun_awt_image_GifImageDecoder_parseImage(), checking for a image block width of 0, which causes a too-small buffer to be allocated and written to. When a width of 0 is found the patch's guard function throws an ArrayIndexOutOfBoundsException, and the vulnerable code is not run. The MOAB Ape patches both the Java 1.4.2 and Java 1.5.0 runtimes.

Are you sure you want to mount that disk image?

Also included in today's update is an optional feature that requires user confirmation before mounting a disk image, even if "Open 'Safe' Files" is enabled in Safari. The following message is displayed:

A number of disk image-related security issues have
been publicly disclosed, including denial of service
and potential remote code execution vulnerabilities.
Do you still wish to mount this image?

If you'd like to disable the message, simply deselect the "Warn before mounting disk images" in the MOAB Ape preferences (System Preferences -> Application Enhancer -> Moab).

Patching MoAB 5, 8, 15 with Bom-Shelter

If you'd like to fix your local disk permissions to prevent the admin to root issues outlined in MoAB issues 5, 8, and 15, William Carrel has written BOM Shelter -- a tool that corrects the disk permissions hilighted in MoAB issues 5, 8, and 15. You can read all about it on William's Blog.

The Bug

The 17th Month of Apple Bugs issue is a buffer overflow in the legacy SLP daemon. The code in question manages local service registrations on the /var/run/slp_ipc unix domain socket, and the vulnerable appears to be local only.

The buffer overflow is in SLPInternalProcessHandlerThread::HandleCommunication()'s kSLPRegisterURL/kSLPDeregisterURL handler:

char    attributeList[1024] = "";
...
memcpy( &attributeList[strlen(attributeList)], attributeListPtr, attributeListLen );

Work-around

If you're concerned about the issue, disabling SLP is straight-forward, and should not result in a loss of functionality for most users. SLP has not been used for service registration since Mac OS X 10.2, and the slpd daemon is used only to register and announce SLP services to legacy clients. As far as I'm aware, SLP is only started when "Personal File Sharing" is enabled -- however, you won't need to disable File Sharing to disable SLP.

To disable slpd, open /Applications/Utilities/Directory Services.app and deselect "SLP" in the "Services tab. You can re-enable slpd in the same way. To make sure that the SLP daemon is stopped, restart your computer or kill the daemon from Terminal.

Thanks to 'MaxP' of the MoAB Fixes group for determining when, and how, slpd is started.

Today's MoAB issue is another format string vulnerability, this time in Colloquy.

The amazingly prescient Colloquy team has already committed a fix, and a new build is available. The issue was noticed and fixed by the Colloquy team prior to the public release of the advisory due to active use of the vulnerability on the freenode network. The MoAB project disclaims responsibility:

"This is an unfortunate prank, and has no relation with us at all
(except the fact of developing the proof of concept and distributing it
to some people)."

Update

I've been quiet for a few days, (although the group remains active), so I thought I'd write a short status update. For the past few days, the MoAB project has been releasing primarily kernel issues in filesystem (UFS & HFS) and network protocol handling (AppleTalk). While it is technically possible to patch the kernel, I am reminded of a Monty Python quote:

Hello. Now, don't you worry. We'll soon have you cured.
Leave it all to us. You'll never know what hit you.

The stakes are much higher when patching the kernel: a mistake can cause a system crash, complete with the potential for file system corruption and data loss. Count me out -- I don't want to provide a cure that's worse than the disease.

Instead, I recommend some simple steps to help keep yourself safe:

• Disable "Open 'Safe' files after downloading" in the Safari preferences. This will prevent a web page from slipping a disk image in under the radar
• Only open disk images from sources you trust. This is easier than it sounds, as most disk images are used for software distribution -- if you don't trust the source of the software, steer clear of the disk image.
• On untrusted networks, disable AppleTalk (it's off by default) in the Network Preference Pane, and enable your firewall. The latest MoAB issue (14) is a bug in the kernel's AppleTalk implementation. Most (nearly all?) routers are not configured to route AppleTalk traffic.

Other Limitations

Application Enhancer won't (as opposed to can't) patch applications owned by another user for security reasons -- this includes root. I have some code based on mach_star that I have previously used to patch various daemons -- if an issue is announced in a non-user process, I'll look into re-using that code, but it'd be a mighty shame to lose Application Enhancer's functionality.

Alternative Solutions

If you have any thoughts on providing patches for the previous issues, please do drop by the MoAB Fixes group. Matt Beaumont has been working on some ideas regarding pre-mount file system validation (custom tool, or fixed fsck), and could use a hand. I've done some work on deciphering the private DiskImages.framework, and have some code that implements support for pre-attach restrictions on disk images.

Introduction

For the past few days I've been releasing patches for software vulnerabilities in an assortment of Mac OS software. This project was intended to be a technical one, and I've never sat down to explain, in clear terms, how the patches work, what Application Enhancer is, or what the potential risks are in running these patches. I'm also not the first one (not by a long shot) to think of implementing third party patches for unpatched software vulnerabilities, either, and I'd like to discuss those efforts.

Here goes ...

Today's Month of Apple Bugs issue exploits Apple's permissions on /Library/Frameworks to implement local admin to root user privilege escalation. Their proof of concept is Application Enhancer, the tool we've been using to apply run-time patches.

The proof-of-concept local exploit backdoors the aped binary by creating their own replacement framework in /Library/Frameworks. This allows a local administrative user to gain root privileges without supplying authentication credentials. It is the same sort of file system privilege problem that allowed Apple DiskManagement BOM Local Privilege Escalation Vulnerability, and the older /Library/StartupItems admin-writability issue.

Like the previous local exploits, this could be combined with a remote exploit to gain root privileges from an administrator account without user interaction. There are also a number of alternate exploit conditions that could occur due to the admin-writability of other directories in /Library.

One work-around:

sudo chmod g-w /Library/Frameworks

Reversed by running:

sudo chmod g+w /Library/Frameworks

Running "Repair Permissions" will also reverse the changes, but see this, first.

I'd like to state that Unsanity, the authors of Application Enhancer, have provided considerable assistance, and I have respect for their technical abilities.

Today's Patch

After much wrangling, I've cooked up a patch for the denial of service bug in Apple's PDF implementation, detailed in MOAB-06-01-2007.

The patch implements loop detection in CoreGraphic's CGPDFReaderGetPageDictionary(), safely exiting the loop after 500,000 iterations (takes approximately 1 second on my machine). By way of comparison, the full ISO C99 specification PDF hits a maximum recursion count of 24. I'd like to thank the venerable William Carrel for his time reviewing the code, and his ideas on implementing recursive functional call handling. Thanks also to Rosyna of Unsanity for assisting in debugging Core Graphics.

The patch prevents the proof-of-concept PDF from locking up any application that uses the CoreGraphics PDF library, including Safari and Mail.app. Since we're unable to patch mdimporter, it will still temporarily fall into an infinite loop on nefarious PDF files -- fortunately, mdimporter detects the loop and exits.

You can download the source, or the pre-built binary. This release removes the VLC fix, so make sure that your VLC player is up to date.

MoAB Collaboration

I'd like to thank everyone for your input.

As I previously stated, I respectfully disagree with the decision to release exploits with no vendor notification. I must also stress that I am not a security researcher, and as such I strongly prefer to recuse myself from the heated debate and focus on providing fixes.

I think that I will have to respectfully decline LMH's offer of coordination. I genuinely appreciate the gesture of goodwill, but I don't feel that it is the right thing to do. I know some of you will disagree with me (and some will agree) -- but upon further reflection, I can't personally compromise the ethical point, though the offer may make my life easier.

I hope you'll all understand, and we can get back to bug fixes quickly. A sincere "thanks" to LMH and MoAB for the offer.

The ever-talented OmniGroup has already released a fix for this morning's Month of Apple Bugs issue. Talk about snappy! Today's issue is a format string vulnerability in OmniWeb's javascript alert() handler.

Download the latest update to OmniWeb at from OmniGroup

LMH of the MoAB contacted me regarding coordination of fixes. He has posted the conversation.

I should state outright that I respectfully disagree with the decision to release exploits with no vendor notification. I also am not a security researcher, and as such I strongly prefer to recuse myself from the heated debate and focus on providing fixes.

That said, the initial goal of this effort was to have some fun, and to provide a quick fix for some serious issues. I never expected anyone to notice, and was perfectly comfortable labouring away in quiet obscurity. Lots of people noticed, however.

What do you think? Is it worth coordinating? Is it worth continuing providing fixes?

Got a late start today -- I will be releasing the fix for today's MOAB issue tomorrow morning, to give me time for proper testing and a code review. The fix implements stuck loop detection, mitigating the proof-of-concept DoS on CoreGraphic's PDF implementation.

More details tomorrow!

Ever feel like you're watching a game of table tennis? I've never been very good at the game ...

Today's Month of Apple Bugs issue permits a local admin account to gain root access, without any user interaction (ie, an authorization dialog), by exploiting a combination of vulnerable disk permissions and Disk Utility's repair permissions functionality.

When coupled with a remote exploit, such as the Month of Apple Bug's Quicktime RTSP URL Handler vulnerability (patched in the current Moab Ape), today's bug could allow the remote exploit to gain immediate root without any user interaction.

Due to the nature of the bug, a safe runtime patch is not viable without modifying on-disk file permissions.

If you'd still like to protect yourself, the Month of Apple Bugs project provides a temporary work-around in their advisory:

sudo chmod -s /System/Library/PrivateFrameworks/DiskManagement.framework/Resources/DiskManagementTool

This may have an impact on other Disk Utility functions -- you can reverse the work-around as follows:

sudo chmod +s /System/Library/PrivateFrameworks/DiskManagement.framework/Resources/DiskManagementTool

Update on the QuickTime Cross-Zone Issue

I'm pleased as punch to report that the terrific WebKit team is looking into the issue.

Darwin ... Ports! Ports!

A number of publications have done the architects of Darwin a disservice by stating that I'm "one of the principal architects of Apple's BSD-based Darwin operating system core". I just want to set the record straight: I originally wrote DarwinPorts (now MacPorts), with Kevin Van Vechten and Jordan Hubbard. Darwin was architected by minds far brighter than my own.

The magnificent VideoLAN team has released 0.8.6a, which fixes the MOAB-reported vulnerability. You can download it from their web site.

I'll be removing the patch for VLC 0.8.6 in tomorrow's APE release.

Answers to the most frequently asked questions

Q. Are patches cumulative? Do I need to install more than one APE?

A. The patches are cumulative -- each new version contains all the previous fixes (including improvements and bug fixes), plus the new fix. If a vendor releases a fix, we'll remove the patch from the APE bundle.

Q. When does a patch take effect? How do I know if I'm protected?

A. Application Enhancer will apply new fixes when an application is restarted. If you've just installed Application Enhancer for the first time, I suggest logging in/out, or restarting your computer.

Q. How can I determine what patch version I have installed?

A. The current version is displayed under the APE name in the Application Enhancer preference pane. The version corresponds to the day -- today is January 4th, and the APE version is 4.0.

Q. Do you support Panther? What operating systems are supported?

A. Unfortunately, only Tiger. I don't have the resources or time to test anything other than Mac OS X 10.4.8.

Q. What happens when the vendor (eg, Apple, VLC) releases a fix?

A. All of the patches are keyed to specific software versions, and the patches are applied at runtime -- the on-disk files are never modified. When a new release of the vulnerable software is installed, the patch will not apply itself. The APE can also be removed at any time by clicking the "-" button in the Application Enhancer preference pane.

Q. How can I contribute?

A. You are encouraged to join the MOAB Fixes group.

I'd also like to apologize to everyone whose e-mail I haven't yet been able to answer personally. I promise I'll reply, it may just take me a couple days.

Friends to the Rescue

Today's update was masterfully implemented by Finlay Dobbie, William Carrel, and the members of the MOAB Fixes Google Group.

Finlay solved today's Month of Apple Bugs issue -- a format string vulnerability in iPhoto's Photocast support. His patch guards the -[SubscribedAlbum registerPublishError:withTitle:] method, escaping all occurances of '%' in the title argument. -[SubscribedAlbum registerPublishError:withTitle:] passes the title directly to [NSString localizedStringWithFormat:].

William Carrel has skillfully updated the fix for the Apple Quicktime HREFTrack Cross-Zone Scripting vulnerability to use a whitelist exclusion method -- QuickTime movies will only permit http, https, and ftp URLs in the HREFTrack.

You can download the source, or a pre-built binary. As always, you'll need to install Application Enhancer to use this -- once it's installed, simply double-click on the Moab bundle to install the patch.

Today's Fix

The third issue from the Month of Apple Bugs takes advantage of the interaction between web browsers and the QuickTime Plugin, leading to a cross-site scripting vulnerability.

The vulnerability allows an embedded quicktime movie, located on an external site (eg, an ad server), to execute javascript in the context of the enclosing page. Please note that the current 10.4.8 version of Safari does not appear to execute JavaScript from a QuickTime HREFTrack, and thus does not appear to be susceptible to this attack.

Today's fix involves patching the QuickTime Plugin's rNPN_GetURL() function, which is responsible for asking the browser to load a page. The patch replaces any javascript: URL requests with a javascript alert box that reads: "[MOAB] Blocked a QuickTime JavaScript Call. See http://landonf.bikemonkey.org/code/macosx/ for more information."

A huge thanks to both Alexander Strange and Rosyna of Unsanity for doing most of the work to track down the issue. I also owe a big favor to my friend William Carrel, who was kind enough to do a code review of the latest changes, and set up a new MOAB Fixes Google Group.

I'd also like to thank all those who have written kind e-mails, or sent other well wishes. It's very much appreciated.

You can download the source, or a pre-built binary. As always, You'll need to install Application Enhancer to use this -- once it's installed, simply double-click on the Moab bundle to install the patch.

News and Updates

As I mentioned above, William Carrel has set up a MOAB Fixes Google Group, where I will be coordinating bug fixes for the rest of the month.

In addition to the above fix, I've also updated the fix for the QuickTime RTSP issue. I fixed a bug in the argument handling (I miscounted the total number of arguments pushed on the stack), and I've added extra seat belts to ensure that the fix will automatically disable itself upon a new release of the QuickTime Streaming Component.

Cross-zone Vulnerability Update

The astutely observant Aviv Raff noted that the QuickTime vulnerability permits Cross-Zone Scripting, which could allow QuickTime movies to access local reference protocol handlers, and potentially permit remote code execution, in addition to the cross-site scripting aspect we've already patched. Just to be safe, I'll be issuing an update later today with a more restrictive (whitelist-based) protocol filter.

Today's bug from the Month of Apple Bugs is a format string vulnerability in the VLC media player.

The wonderfully responsive and dapper VLC team already has a fix committed, and I'm sure a new release will be forthcoming.

In the meantime, more for completeness than necessity, I've added a fix to the MOAB Ape. The fix is keyed to (and only supports) the latest VLC release (0.8.6) -- the patch will automatically de-activate itself when you update VLC, which you should most certainly eventually do. The patch works by guarding the *_log_handler() callbacks, checking for format string characters before passing the string on.

You can download the source, or a pre-built binary. You'll need to install Application Enhancer to use this -- once it's installed, simply double-click on the Moab bundle to install the patch.

If you'd like to help with tomorrow's MOAB vulnerability please feel free to send me patches or other information. If there's enough interest, I'll fire up a mailing list.

I stumbled across the Month of Apple Bugs today -- a new Mac OS X vulnerability released every day for a month.

Today's vulnerability exploits a stack buffer overflow in the QuickTime Streaming component, and includes a working x86 exploit. More details available here. Short summary: if you visit a malicious web page, a remote attacker can execute code on your machine.

So, part brain exercise, part public service, I've created a runtime fix for the first issue using Application Enhancer. If I have time (or assistance), I'll attempt to patch the other vulnerabilities, one a day, until the month is out.

You can download the source, or download a pre-built binary. You'll also need to install Application Enhancer to use this -- once it's installed, simply double-click on the Moab bundle to install the patch.

I've had one report of the Moab bundle showing up in the Finder as a plain directory. If that occurs, try manually adding the bundle from the Application Enhancer preference pane, in System Preferences (Hit the "+" button).

Technical Details -- How it Works

The overflow is in the QuickTime Streaming component's INet_ParseURLServer() function -- the fix patches that function and pre-validates the URL before passing it off to the real function implementation. If the URL is too long, the patch replaces the Evil URL with a benign, but invalid one, and then calls the original function.

It's worth noting that disabling RTSP, as noted elsewhere, is (unfortunately) not necessarily sufficient -- there are other vulnerable entry-points to INet_ParseURLServer(), as it is used for generic URL parsing.

Update

Please see this page for updates!