Landon Fuller

NOTE: For information on Mac OS X Tiger, refer to this article.

In Mac OS X, Apple introduced an additional, non-standard request type to the ptrace() system call - PT_DENY_ATTACH. While an understandable addition, especially in terms of providing plausible defense for their DRM applications, PT_DENY_ATTACH has come to be used by a number of third party developers in an attempt to provide further copy protection.

This is unfortunate for those of us with a genuine need to attach a debugger; There are several circumstances when this ability is necessary, including working with libSystem, writing a runtime patch with APE, writing a kext, writing an input manager, or software auditing.

There are several possible ways to work around this behavior; breaking on ptrace(2) in gdb, recompiling your kernel, or writing a kext. I choose to write a kext that hooks ptrace(2).

ptrace(2) is implemented in xnu's bsd/kern/mach_process.c . When ptrace(PT_DENY_ATTACH, ...) is called, if the process is not already being traced, the P_NOATTACH flag is set for the calling process. If the process is already being traced, then the flag is not set and the kernel will call exit1(p, ...), forcing the proccess to exit.

int
ptrace(p, uap, retval)
	struct proc *p;
	struct ptrace_args *uap;
	register_t *retval;
{
	... snip ...
	if (uap->req == PT_DENY_ATTACH) {
		if (ISSET(p->p_flag, P_TRACED)) {
			exit1(p, W_EXITCODE(ENOTSUP, 0), retval);
			/* drop funnel before we return */
			thread_funnel_set(kernel_flock, FALSE);
			thread_exception_return();
			/* NOTREACHED */
		}
		SET(p->p_flag, P_NOATTACH);

		return(0);
	}
	... snip ...
}

When a process attempts to attach to another, the P_NOATTACH flag is checked. If it is set, the process calling ptrace(2) will be sent a SIGSEGV signal and ptrace(2) will return EBUSY.

	... snip ...
	if (uap->req == PT_ATTACH) {
		... snip ...
		if (ISSET(t_>p_flag P_NOATTACH)) {
			psignal(p, SIGSEGV);
			return (EBUSY);
		}
		... snip ...
	}

The easiest way to work around PT_DENY_ATTACH is to prevent the P_NOATTACH flag from ever being set on the process. I chose the relatively straight-forward method of hooking the ptrace(2) system call with a kext. My code can simply return 0 for all PT_DENY_ATTACH requests, while passing other requests directly to the real ptrace(2) implementation.

In xnu, the sysent array includes function pointers to all system calls. By saving the old function pointer and inserting my own, I can insert my code in the ptrace(2) path. Anyone doing this in production code will be sent home with a note to their parents.

kern_return_t pt_deny_attach_start (kmod_info_t * ki, void * d) {
	real_ptrace = sysent[SYS_ptrace].sy_call;
	sysent[SYS_ptrace].sy_call = our_ptrace;
	printf("Disallowing ptrace(PT_DENY_ATTACH, ...)\n");
	return KERN_SUCCESS;
}

int32_t our_ptrace (struct proc *p, struct ptrace_args *uap, register_t *retval)
{
	if (uap->req == PT_DENY_ATTACH)
		return (0);
	else
		return real_ptrace(p, uap, retval);
}

I've made the source to the kext available here.

Apple's use of PT_DENY_ATTACH makes sense - DRM is a touchy subject, and I imagine there are more legal than technical reasons for the implementation. Further, Apple does not declare PT_DENY_ATTACH in public headers; third party developers had to fish the information out of xnu.

Further discussion on implementing copy protection in your software is available at CocoaDev.