How to Reverse Engineer an Undocumented macOS API to Use It in a Swift Project

Today’s software solutions require continuous improvements. But what used to take weeks or even months can now be done with a couple lines of code, assuming you have the right application programming interface (API) at hand.

By integrating APIs into their software, developers can improve product security and performance and expand existing functionality with new services and features. Unfortunately, some APIs might be poorly documented or have hidden security or performance issues.

In this article, we describe how to reverse engineer a macOS API. We also show how to use a reversed undocumented macOS API in a proof of concept (PoC) Swift application. This article will be useful for macOS development teams who need to work with undocumented or poorly documented APIs.

Reverse engineering undocumented APIs

Not all APIs are properly documented. Some are private with no publicly available information on their internals. Such APIs are primarily intended to be used by in-house developers working on proprietary software. Others might be public but have incomplete documentation due to the continuity of the project, poor development practices, or frequent changes in development teams.

The problem is that when you integrate an undocumented API into your software, there’s a high risk of facing compatibility issues, compromising the solution’s performance, and even introducing new security flaws. That’s why before working with a private or undocumented API, you might need to first reverse engineer and analyze it.

Note: Make sure to check the terms of service before reverse engineering a private API that isn’t yours to avoid any legal issues. Also, remember that unlike open or public APIs, private APIs can be changed without any prior notice, so using another company’s private API in your own project should be a last resort.

There are several scenarios when reversing an undocumented API proves helpful:

1. Research software issues — API reverse engineering can help you determine the actual reason for poor performance or expose security vulnerabilities introduced as a result of integrating a poorly documented API.
2. Maintain legacy code — Using reverse engineering techniques, you can restore lost documentation or improve the quality of existing documentation to keep your API easy to integrate and support.
3. Improve platform interactions — The results of reverse engineering analysis can help you detect hidden bottlenecks and determine the best ways to improve interactions between your software and the platform it works on.
4. Ensure smooth third-party compatibility — Reversing can be instrumental in finding unobvious reasons for your software’s poor compatibility with third-party solutions and services.

When you know exactly how a certain API operates, it’s much easier to identify possible flaws and security issues and figure out the best way to integrate the API into your software. So let’s see how to do reverse engineering of an API for macOS in practice.

Reversing a macOS API

How can you reverse engineer an undocumented macOS API and when do you really need to do it? 

Say you have a project where you need to display the developer mode status. Generally, there are two ways you can do that:

1. Run the responsible utility and parse its output
2. Call the corresponding API

From the technical point of view, getting the needed output directly from a system utility is easy. However, this isn’t always preferred due to possible performance bottlenecks. To enhance your application’s performance, it might be more efficient to identify the corresponding system API and invoke it directly from within the application.

For this article, we’ll be working with an undocumented API used by the systemextensionsctl developer utility to report and modify the status of developer mode on macOS. We’ll need to reverse engineer this macOS API and analyze it before integrating it into our Swift project.

The reverse engineering activities described in this article were performed on macOS Monterey 12.3.1 running on Apple Silicon. As the main tool for analysis, we used Ghidra 10.1.2, a popular suite of tools for software reverse engineering.

Disabling System Integrity Protection on macOS

Before analyzing the systemextensionsctl API, we need to disable System Integrity Protection (SIP). Without doing so, it would be impossible to enable, disable, or display the current developer mode status. Our proof of concept application, however, imposes no such requirement, as it can only display but not modify the developer mode status.

The process of disabling SIP is described in detail in the Disabling and Enabling System Integrity Protection article on the Apple Developer website.

Once SIP is disabled, you can use the following commands to set the desired developer mode:

systemextentionsctl developer on
systemextentionsctl developer off

To query the developer mode status, use the following command:

systemextentionsctl developer

The status is reported as being off or on based on the current setting:

% systemextensionsctl developer
Developer mode is off
 
% systemextensionsctl developer
Developer mode is on

Later, we’ll compare the output generated by our PoC against this output.

Read also

How to Reverse Engineer an iOS App and macOS Software

Ensure compatibility and maintain your macOS and iOS legacy code! Explore the essential techniques for software reverse engineering with our comprehensive guide.

Learn more

Locating the API to be reverse engineered

Next, we locate the API that displays the developer mode status. To do so, we upload the systemextensionsctl binary into Ghirda and get the disassembled binary code. We then perform a textual search using the output of the query command to identify the appropriate function. Ghidra identifies this function as FUN_100003754.

Note: As the aim of this post is to illustrate the overall process rather than provide step-by-step reversing instructions, we won’t cover this step in detail. 

Now we can move on to analyzing the code of the FUN_100003754 function in detail.

Analyzing the reversed code

In Table 1 below, you can see the disassembly of the code block that determines the developer mode status with the accompanying pseudo code, as may be seen in Ghidra’s disassembly and decompiler listings, respectively.

Table 1: Disassembly of the code block that determines the developer mode status

Now, let’s outline the purpose of some of the lines above:

Table 2: Purpose of code lines from the disassembly

The above disassembly would correspond to the following pseudo code generated by Ghidra’s decompiler and shown in the decompiler listing:

OSSystemExtensionClient* client = __auth_stubs::_objc_alloc_init(&_OBJC_CLASS_$_OSSystemExtensionClient); //100003788-100003790
char* local_50 = NULL; //100003794
char* selector = “developerMode:error:"; //10000379c
char* status = (char **)&local_41; //1000037a0
char* error = &local_50; //1000037a4
Bool ret = __auth_stubs::_objc_msgSend(client, selector, status, error); //1000037a8-1000037ac
param_1 = __auth_stubs::_objc_retain(local_50); //1000037b4-1000037b8
if ((ret & 1) != 0) { //1000037bc
    statstr = "Developer mode is off\n”; //1000037cc
    if (*status != '\0') { //1000037c0,1000037d4, 1000037d8
        statstr = "Developer mode is on\n”; //1000037c4
    }
    goto LAB_100003888;
}
…
LAB_100003888:
__auth_stubs::_printf(statstr);

Now that we have successfully reversed the OSSystemExtensionClient API, we want to make sure we can use it in our PoC.

Using the reversed API in a Swift PoC

To use the reversed OSSystemExtensionClient API in a PoC written in Swift, we need to:

1. Obtain the signature for the OSSystemExtensionClient developerMode:error method and put it into the OSSystemExtensionClient.h header file.
2. Import the OSSystemExtensionClient.h header file into the PoCName-Bridging-Header.h header file for external symbols to become available in our Swift project.

Note: The following instructions apply to compiling Swift applications only. They won’t work if applied to applications written in other programming languages.

Let’s begin with the first of these two steps.

Step 1: Obtaining a method signature

The OSSystemExtensionClient API is part of the SystemExtensions framework. The framework is packaged as a dynamically linked shared library, which is part of the dynamically linked shared library cache available at /System/Library/dyld/dyld_shared_cache_arm64e.

There are tools available, such as class-dump and classdump-dyld, that can generate Objective-C header files with method signatures based on their analysis of binaries and libraries. Unfortunately, none of these tools worked for us out of the box in our environment, with each generating multiple compilation errors. The method signature was obtained using Ghidra instead.

To analyze the SystemExtensions framework with Ghidra, we first need to extract the framework library from the cache file. There are various tools you can use for this task, like DYLDExtractor, the dyld_shared_cache_util command-line tool, or the dyld-shared-cache-extractor tool. For the example described in this article, we used dyld-shared-cache-extractor, as it’s readily available as a precompiled binary package through the Homebrew software repository, which makes it immediately usable.

With the help of this tool, we can extract the needed library from the cache file:

% dyld-shared-cache-extractor /System/Library/dyld/dyld_shared_cache_arm64e /Users/user/tmp/dyld_shared_cache

The extracted binary is now available at the following path:

% file
/Users/user/tmp/dyld_shared_cache/System/Library/Frameworks/SystemExtensions.framework/Versions/A/SystemExtensions
/Users/user/tmp/dyld_shared_cache/System/Library/Frameworks/SystemExtensions.framework/Versions/A/SystemExtensions
: Mach-O 64-bit dynamically linked shared library arm64e

After extracting the library and analyzing it with Ghidra, we see the following method signature in the decompiler view:

undefined -[OSSystemExtensionClient_developerMode:error:]
               (undefined8 param_1,undefined8 param_2,undefined *param_3,undefined8 *param_4)

Quick inspection of the pseudo code indicates that the return value represented as a local variable bVar9 is a boolean value, set to zero or one to denote truth or falsehood respectively:

if (iVar2 == 0) {
    bVar9 = 0;
}
…
if (param_3 != (undefined *)0x0) {
    uVar1 = __auth_stubs::_xpc_bool_get_value(lVar6);
    *param_3 = uVar1;
}
bVar9 = 1;
…
return bVar9;

The param1 parameter is a pointer to an instance of the OSSystemExtensionClient class. Whatever is passed as the param2 parameter doesn’t appear to be used within the developerMode method, the value of this parameter being set to zero:

18e5a4fcc 01 00 80 d2     mov        param_2,#0x0

The param3 and param4 parameters are identified as pointers and are the only ones that are passed explicitly.

Note: When reverse engineering system APIs, keep in mind that the same methods might have different signatures depending on the version of macOS you’re working with.

Next, we generate the header files containing the signature of the reversed API and import them into our Swift PoC.

Read also

How to Reverse Engineer Software (Windows) the Right Way

Discover how ethical reverse engineering practices can drive innovation of Windows software, speed up product development, and bring competitive advantages to your business.

Learn more

Step 2: Importing headers

The first thing we need to do is generate Objective-C headers containing the signature of the method we’ve reversed. Based on our method signature analysis through Ghidra, the following signature was found as working:

- (BOOL)developerMode:(char**)arg1 error:(char*)arg2

The method signature can now be put into OSSystemExtensionClient.h and imported into the devmodestat-Bridging-Header.h header file. To do so, we use the following commands:

% mkdir devmodestat
% cd devmodestat/
 
% cat >OSSystemExtensionClient.h<<eof import="" foundation="" h="">
#import <objc/NSObject.h>
 
@interface OSSystemExtensionClient : NSObject
 
- (BOOL)developerMode:(char**)arg1 error:(char*)arg2 API_AVAILABLE(macos(11));
 
@end
EOF
 
% cat >devmodestat-Bridging-Header.h<<eof import="" ossystemextensionclient="" h="" eof="" code="" pre="">
<p>Next, we generate the source file for our PoC with the following command:</p>
<pre>{code}% cat >devmodestat.swift<<eof import="" foundation="" var="" statusstring="Developer mode is " status:="" unsafemutablepointer="" int8="">? = nil
if #available(macOS 11, *) {
    if OSSystemExtensionClient().developerMode(&status, error: nil) == true {
        statusString += status != nil ? "on" : "off"
    } else {
        statusString = "unknown"
    }
}
print(statusString)
EOF
</eof></pre></eof></eof>

Next, we generate the source file for our PoC with the following command:

% cat >devmodestat.swift<<EOF
import Foundation
var statusString = "Developer mode is "
var status: UnsafeMutablePointer<Int8>? = nil
if #available(macOS 11, *) {
  if OSSystemExtensionClient().developerMode(&status, error: nil) == true {
  statusString += status != nil ? "on" : "off"
  } else {
    statusString = "unknown"
  }
}
print(statusString) EOF

After that, we can use the following commands to compile the PoC, link it against SystemExtensions.framework, and verify the resulting executable:

% xcrun swiftc -o devmodestat devmodestat.swift -import-objc-header devmodestat-Bridging-Header.h -framework SystemExtensions
 
% file devmodestat
devmodestat: Mach-O 64-bit executable arm64

Once we compile the PoC, we can compare its output with the output generated by the systemextensionsctl developer utility.

Here’s the output generated with developer mode enabled:

% systemextensionsctl developer
Developer mode is on
 
% ./devmodestat 
Developer mode is on

And the output generated with developer mode disabled:

% systemextensionsctl developer    
Developer mode is off
 
% ./devmodestat 
Developer mode is off

As you can see, the output of the PoC is identical to the output generated by the macOS system API. This demonstrates that we managed to properly reverse an undocumented macOS API and successfully integrate it in the PoC.

Related project

Device Firmware Reversing to Obtain the Screen Mirroring Protocol

Discover how Apriorit reverse engineers analyzed our client’s original firmware to reconstruct the secure connection process and enable cross-platform compatibility for video mirroring. 

Project details

Conclusion

Software developers often need to work with undocumented or poorly documented APIs. Reverse engineering can help them get the information they need to understand the inner workings, recover lost source code, and ensure smooth and secure integration of the analyzed API into the solution they’re working on.

At Apriorit, we have a team of reverse engineering experts who will gladly assist your company with tasks requiring deep expertise in software research and analysis.