12:23 Sat, 26 Jan 2008 PST -0800

Installing the iPhone Toolchain using MacPorts

NOTE: This post has been archived for historical purposes. The toolchain has advanced considerably, and Apple is planning to release their own SDK. I plan to hold out on further development, toolchain or otherwise, until it's released.

Please see iPhone Toolchain Project for up-to-date instructions.

To facilitate my own iPhone development, I've committed three new ports for the iphone-binutils project to MacPorts. Once installed, you're ready to compile Hello, World. Prior to installation, you'll need to acquire a copy of the iPhone root disk image ("Heavenly") and install its contents in /opt/local/arm-apple-darwin/heavenly. The image is required to provide the necessary libraries for linking cross-compiled iPhone binaries, and can't be re-distributed.

Extracting and Installing the iPhone Libraries

To start, download and decompress iPhone1,1_1.0_1A543a_Restore.ipsw:

 user@host> curl -O http://appldnld.apple.com.edgesuite.net/content.info.apple.com/iPhone/061-3538.20070629.B7vXa/iPhone1,1_1.0_1A543a_Restore.ipsw
 user@host> unzip iPhone1,1_1.0_1A543a_Restore.ipsw \*.dmg

This will extract two disk images: "694-5259-38.dmg" and "694-5262-39.dmg". The encrypted "694-5262-39.dmg" disk image contains the iPhone root. The decryption key for this image is stored in plain text within the "asr" binary on the corresponding "694-5259-38.dmg". To retrieve the key, run the following:

 user@host> strings 694-5259-38.dmg| grep "^[0-9a-fA-F]*$" | awk '{ if (length($1) == 72) print; }'

This should output a 72 character hex string, which you'll use as the decryption key.

In order to perform the decryption, you'll need modified 'vfdecrypt' -- a command utility for decrypting Mac OS X disk images. The source is available here. To compile, simply type "make" in the vfdecrypt-iphone directory. The provided version of vfdecrypt was slightly modified to support direct input of the private AES and SHA1 HMAC keys -- these are normally wrapped with a user-supplied passphrase (via 3DES-EDE), which is not available. vfdecrypt was written by Ralf-Philipp Weinmann, Jacob Appelbaum, and Christian Fromme.

Once you've build vfdecrypt, use it to decrypt the disk image:

 user@host> ~/vfdecrypt-iphone/vfdecrypt -i 694-5262-39.dmg -k <hex key> -o heavenly.dmg 

Now, mount the disk image and copy the contents to /opt/local/arm-apple-darwin/heavenly:

 user@host> open heavenly.dmg
 user@host> sudo mkdir -p /opt/local/arm-apple-darwin/heavenly
 user@host> (cd /Volumes/Heavenly1A543a.UserBundle && tar cf - .) | (cd /opt/local/arm-apple-darwin/heavenly && sudo tar xvf -)

Installing the Toolchain

To install the toolchain:

 sudo port install arm-apple-darwin-runtime

You should now be able to compile standard Unix software:

 CC=arm-apple-darwin-cc CPP=llvm-cpp ./configure --host=arm-apple-darwin

For more information on compiling your first GUI application, check out the UIKit Hello World

[/code/iphone] permanent link

FreeBSD's 1.6 JDK on Mac OS X: Status Update

Puzzle Pirates - A More Entertaining Hello World

Status Update

I've spent some more time working on porting JDK 1.6 -- adding support for Tiger (10.4), fixing stack alignment issues for both amd64 and i386, fixing build system issues, and more. On Leopard, the 32-bit JDK now builds to completion. As evidenced by the above "Hello, World", X11 Swing is working on the 32-bit JVM.

Work is still progressing on the 16-byte stack alignment issue. In the meantime, I've enabled the gcc -mstackrealign option to force stack alignment in compiled JVM code. This ensures that calls back into the JVM are properly aligned:

On the Intel x86, the -mstackrealign option will generate an alternate
prologue and epilogue that realigns the runtime stack. This supports mixing
legacy codes that keep a 4-byte aligned stack with modern codes that keep a
16-byte stack for SSE compatibility. The alternate prologue and epilogue are
slower and bigger than the regular ones, and the alternate prologue requires
an extra scratch register

To replace this temporary work-around, I'm currently working on fixing the call stack alignment in the HotSpot Interpreter. Currently most calls are properly aligned, with the JVM running until it hits a JNI call. The JNI code generation assumes that arguments can be pushed directly to the stack prior to calling the stub generator, making alignment more difficult. I have not started work on alignment in the c1 or c2 compilers. Due to -mstackrealign compiler bugs in Tiger, the JVM must be built on Leopard -- however, it should be possible to build a JDK to run on Tiger.

For 64-bit Leopard systems, I've fixed up some latent alignment issues in HotSpot (Leopard is very picky about checking alignment in dyld), and the code appears to work fine with a few caveats. I've only spent a few hours on this today, but 64-bit Java should be the most stable and easiest to port. There is a known issue in the AMD64 JVM that I have not had time to track down, occaisionally triggering the following assert:

 javasrc_1_6_jrl/hotspot/src/cpu/amd64/vm/frame_amd64.cpp:118
  assert((intptr_t) sp() <= (intptr_t) result,
           "result must >= than stack pointer");

As a temporary work-around, the x86_64 JVM appears to run fine (if slowly) in interpreter mode (-Xint).

Future Direction

As a FreeBSD-Java committer, I will be merging low-risk changes upstream to the FreeBSD Java Project (Code). Potentially destabilizing changes to Hotspot interpreter / compiler will not be merged until they've seen adequate review by the FreeBSD and OpenJDK teams.

There are tentative plans to merge the BSD Java changes into the OpenJDK project. This isn't something I'm directly involved with, but I have signed the Sun Contributor Agreement in preparation.

I am also very interested in external contribution -- this is a part time, after-work project for me! See below for code access.

In progress work:

Code Access

I am making the JRL-licensed source code available via a Mercurial repository. Since the OpenJDK project is moving to Mercurial, I thought I'd get on the bandwagon -- it is working out well. The repository is available at:

http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/

To download the source code, you must be a licensee in good standing under the Java Research License. To ensure compliance, the repository requires authentication -- the username is "jrl", and the password is 'I am a Licensee in good standing'. By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Currently, all FreeBSD JDK16 work is under the JRL. In the future, the work may be relicensed, with a copyright grant, for OpenJDK under the GPLv2.

This not something I like to do, but I request that all submissions assign copyright to myself, such that I have the ability to relicense/grant copyright for submission to the upstream projects. If anyone has a better idea of how to handle this, I'm very interested.

To build the x86_32 JDK:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-i586-debug/lib/i386/client \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1 debug_build

To build the x86_64 (amd64) JDK:

cd control/make
env DYLD_LIBRARY_PATH=`pwd`/../build/bsd-amd64-debug/lib/amd64/server \
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=4 PARALLEL_BUILD_JOBS=4 BSD_OVERRIDE_ARCH=amd64 JAVA_JVM_FLAGS=-Xint JAVAC_FLAGS=-Xint debug_build

Remove the "-Xint" flags to disable interpreter mode (and trigger the known issue in frame_amd64.cpp).

Note that the debug build is considerably slower than a production build, and interpreted mode is even slower!

If you're interested in assisting with IA32 stack alignment issues, the current patch is available here: patch-stack-align-darwin-v2.

Looking for a Job?

Lastly, a plug for my group at Three Rings. We're looking to hire Java Engineers (not just billing!) and FreeBSD Developer/Admins. The group rocks (I run it!), so feel free to drop me a line if you're interested.

[/code/macosx] permanent link

12:21 Sat, 26 Jan 2008 PST -0800

Java 6 Port: Developer Preview Release 2 for Leopard and Tiger

Community Process

I'd like to thank everyone who has downloaded, tested, and reported bugs. The result is "Developer Preview Release 2" -- Thanks!

If you haven't already, consider downloading a copy and running your software. Let me know how it works out for you!

Changes since Release 1

Features

Bug fixes

Fetching and Building from Source

Building

The build has been simplified. To build the 32-bit JVM:

cd control/make
make ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Home \
ALT_MOTIF_DIR=/opt/local SYS_CFLAGS="" LANG="C" JAVA_HOME="" CLASSPATH="" \
LD_LIBRARY_PATH="" MAKEFLAGS="" SKIP_COMPARE_IMAGES="YES" \
BUILD_DEPLOY="false" ALT_DEVTOOLS_PATH=/usr ALT_CUPS_HEADERS_PATH=/usr/include \
HOTSPOT_BUILD_JOBS=1 PARALLEL_BUILD_JOBS=1

The following additional make options are available:

To build with extra debugging code and symbols, use the "debug_build" target.

To target Tiger, set the MACOSX_DEPLOYMENT_TARGET environmental variable to "10.4", and pass DARWIN_SDK=/Developer/SDKs/MacOSX10.4u.sdk as a make flag. The Tiger SDK currently must be build on Leopard, due to bugs in the 10.4 compiler's -mstackrealign code generation. This is on the long-term to-be-fixed list.

Fetching the source

Sources are available as a downloadable archive, or from a mercurial repository. To download the source code, you must be a licensee in good standing under the Java Research License. To ensure compliance, downloading the source requires authentication:

Username: 'jrl' and Password: 'I am a Licensee in good standing'

Download: jdk6_devpreview_r2.tar.gz (sig)

The development repository is also available via mercurial: http://hg.bikemonkey.org/javasrc_1_6_jrl_darwin/. The release tag is jdk6_devpreview_r2.

By downloading this source code, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Binary Access

I am making available the SoyLatte binary release. (Red Hat already claimed IcedTea, and I drank a lot of double soy lattes while working on this. Plus, I think it's funny).

SoyLatte is based on the BSD Port of Sun's Java 6 JDK, and is made available under the Java Research License. JDK and Java are trademarks of Sun. Like RedHat, I want to make it exceptionally clear that while SoyLatte is a port of Java, it is not Sun's Java, JDK, or OpenJDK. Unlike IcedTea, SoyLatte is made available under the JRL. Please see below for a licensing discussion.

By downloading these binaries, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

Downloads

Binaries are available for Mac OS X 10.4 and 10.5 (32-bit), and Mac OS X 10.5-only (64-bit). The soylatte directory can be placed anywhere on your system -- I chose /usr/local/soylatte16-amd64. Like other Java platforms, setting the JAVA_HOME and PATH environmental variables to point at these locations will work as expected.

32-bit JDK for Mac OS X 10.4 and 10.5: soylatte16-i386-r2.tar.gz (sig)

64-bit JDK for Mac OS X 10.5: soylatte16-amd64-r2.tar.gz (sig)

To ensure compliance, downloading the source requires authentication:

Username: 'jrl' and Password: 'I am a Licensee in good standing'

Feature TODO List

I'll start looking at all of these eventually, but feel free to jump in:

JSR 223 JavaScript Support

Sun has pulled Rhino from the JRL and OpenJDK (GPLv2) sources, due to concerns regarding licensing. As a work-around, there is a BSD-licensed javascript JSR223 engine implementation, using Rhino, available via https://scripting.dev.java.net/:

We have built and tested with Rhino version 1.6 release 5. There is a Rhino based JavaScript engine bundled in JDK 6 (http://jdk6.dev.java.net). The JDK 6 bundled version is based on Rhino 1.6 release 2. Unlike JDK 6 bundled engine, all Rhino features (optimizer, E4X) are enabled in this version.

This seems to work as a drop-in replacement for Sun's sun.org.mozilla code. Here's JavaScript Hello World using Rhino 1.6r5 and the Scripting package:

java -cp ~/Downloads/jsr223/javascript/build/js-engine.jar:/tmp/rhino1_6R7/js.jar:. EvalScript
Found engine factory: com.sun.phobos.script.javascript.RhinoScriptEngineFactory
Found engine factory: com.sun.phobos.script.javascript.EmbeddedRhinoScriptEngineFactory
Hello, World

I am looking into the feasibility of integrating this solution.

Licensing

The Mac OS X work is based heavily on the BSD Java port, which is licensed under the JRL. The BSDs develop Java under the JRL; FreeBSD has negotiated a license with Sun to distribute FreeBSD Java binaries based on the JRL sources.

As the Mac port stabilizes, I am merging my work upstream into the BSD port, and in turn, it is a goal of the FreeBSD Java project to merge their work into OpenJDK. I've signed a Sun Contributor Agreement in preparation for this, and an OpenJDK Porters group has been proposed: http://thread.gmane.org/gmane.comp.java.openjdk.general/630

While the JRL makes this initial port possible, OpenJDK's GPLv2+CE licensing makes development and distribution far simpler. I hope to contribute this work to OpenJDK as soon as is feasible.

Submitting Bug Reports

There are bugs, and you're likely to find one. The most useful bug report is one that includes a simple, compilable reproduction case -- that gives me what I need to track down the really tricky bugs.

In submitting a bug, please include the following information:

Bug reports may be submitted to landonf (at) macports (dot) org.

[/code/macosx] permanent link

12:18 Sat, 26 Jan 2008 PST -0800

SoyLatte: Release 1.0.1

Minor Bugfix

This release fixes a name resolution bug reported by Leif Nelson of LLNL.

I tracked this down to this copy/paste bug in resolver code:

error = getaddrinfo(hostname, "domain", &hints, &res);

The service argument should have been NULL.

Download

Binaries, source, build, and contribution instructions are all available from SoyLatte Project Page

[/code/macosx] permanent link

12:17 Sat, 26 Jan 2008 PST -0800

Implementing a Better DNS Dead Drop

dead drop (n): A dead drop or dead letter box, is a location used to secretly pass items between two people, without requiring them to meet.

The Original DNS Dead-Drop

Two years ago, I implemented a DNS-based dead-drop, based on an idea presented by Dan Kaminisky in Attacking Distributed Systems: The DNS Case Study.

Using a recursive, caching name server, coupled with a wildcard zone, it's possible to implement double-blind data transfer. In each DNS query, 7 bits are reserved for a number of flags, one of which is the Recursion Desired (RD) flag. If set to 0, the queried DNS server will not attempt to recurse -- it will only provide answers from its cache.

Combine this with a wildcard zone and it's possible to signal bits (RD on), and read them (RD off). To set a bit to 1 the sender issues a query with the RD bit on. The wildcard zone resolves all requests, including this query. The receiver then issues a query for the same hostname, with the RD bit off. If the bit is 1, the query will return a valid record. If the bit is 0, no record will be returned.

So, it's easy to signal a single bit, but what if you want to share more than 1 bit of data? This requires both sides to compute a list of records -- one record for every bit of data we wish to send. In my implementation, I chose to do this with a pre-shared word list and initialization vector (IV). Given the same word list and IV, both sender and receiver can independently compute an identical mapping of words to bit positions. The sender can then signal the '1' bits, and the receiver can query all bits.

Hiding the Trail: Using TTL to Signal Bits

To avoid suspicion, a good dead-drop mechanism should not appear unusual to an outside observer. The RD flag is unusual, and a signature to detect its use can easily be added to intrusion detection systems. It would be considerably more sneaky to use a signaling mechanism that relied on more normal-appearing DNS queries.

This is where the time-to-live (TTL) value can be used. When returning query results, many recursive DNS servers include a TTL -- the number of seconds before the recursive name server will purge the record from its cache. The TTL begins decrementing as soon as a record is cached. Therefor, newer lookups with have a higher TTL than older lookups. Using this property, it is possible to determine if a record was previously cached, and thus signal bits without relying on the RD flag.

To communicate, the sender and receiver need to pre-share a word list, an initialization vector (IV), the IP of a recursive nameserver, a wildcard domain, and a communications window (time of day). Here's how the protocol works:

Sender:

Receiver:

Download

You can download a copy of NSDK here. It's written in Python, and depends on the dnspython library.

The implementation is a proof-of-concept -- the TTL heuristic is very simple, and you'll certainly see bit errors in longer messages. I enjoyed the "Bourne Identity" books way too much, and this is all meant in fun.

Usage

nsdk.py [dns IP] [wildcard domain] [word list] [iv] <message>

Each bit of your message requires at least one DNS query. I strongly suggest testing this implementation against name servers and zones that you control.

To send a message:

./nsdk.py 10.0.0.1 wildcard.example.com /usr/share/dict/words 42 "The crow flies at midnight"
Message sent successfully!

To receive the message:

./nsdk.py 10.0.0.1 wildcard.example.com /usr/share/dict/words 42
Read 208 bits from DNS server
The Secret Message: The crow flies at midnight

[/code/security] permanent link