How to Control Application Operations: Reverse Engineering an API Call and Creating Custom Hooks on Windows

Improving your software’s cybersecurity posture and preventing data leaks is tricky but essential. One way to safeguard an IT product is by modifying system operations by reverse engineering an API to intercept and modify API calls with custom hooks. This approach helps you augment the solution’s behavior but also requires experienced reverse engineers to execute properly without affecting the solution’s performance.

In this article, we show the main practices for reverse engineering API calls and creating custom hooks to help your team control operations in the application’s processes. We explore how to reverse a target application’s API calls and install hooks via Process Monitor, Api Monitor, and MHook. 

This article will be helpful for project leaders working on cybersecurity improvements and looking for ways to take control over an application’s operations.

Why and how to control operations in applications on Windows

The most common reason for modifying an operation’s behavior is to improve a solution’s cybersecurity and prevent data leaks. By reversing Windows operations, you can improve software security and continue using certain software solutions without needing to rebuild them completely. Also, you can alter application behavior even when you don’t have access to the source code.

In such cases, your developers can influence an app’s behavior with reverse engineering techniques like API call hooking.

To control a program’s work, a developer needs to find a function that’s convenient to place a hook on and find out which operation is running there. By operation, we mean a sequence of program actions that perform a specific function for a user. Examples of such operations are opening a file, editing a file inside a program, copying and pasting, and saving a file.

In this article, we explore examples of monitoring and changing the behavior of two operations for a target program. For each case, we’ll use these two steps:

Before exploring how to reverse engineer the API used by an app using practical examples, let’s briefly discuss each step of controlling operations.

1. Identify the operation in the target program

To block a specific operation in your software, developers need to find specific arguments that could be used inside a hook. And to identify from the inside of the hook which operation is launched, developers need to analyze the current state of the target program while this unknown operation is running.

Your team can use different methods to identify which operation is happening at the moment. The most popular are the following:

1. Check the function’s input parameters. Some functions use specific patterns for input parameters. These could be patterns for file names, flags, and any other parameters specific to a certain function. Analyzing such parameters can help developers understand that a function is used for a certain operation.

Although this method is fast and easy to implement, we recommend combining it with checking the call stack to achieve better accuracy and avoid false positives. The downside of this method is that sometimes parameters can be too unique to identify the operation.

2. Check the call stack. The idea of this method is to receive the call stack of a current operation and compare it against the call stack used for the operation you’re trying to identify. By analyzing similarities and differences between two call stacks, developers can get insights into what the target operation is supposed to do.

In some cases, checking the call stack helps engineers accurately identify an operation. But the accuracy depends on the uniqueness of the call stacks: the more unique they are, the easier it is to distinguish the stack of a required operation from other operations’ stacks. A significant downside of this method is the waiting time for receiving a call stack. Depending on how your engineers do it, getting a call stack can take too much time. Also, the call stack can be less helpful if a program’s code is obfuscated.

3. Use functions responsible for the operation. Another way to identify an operation is to find a function that’s designed to work only with the operation you want to identify and create a hook for it. In this case, developers usually don’t need to perform any checks inside the hook, which saves time. Also, this method ensures fast program response, as a hook will only be called when the required operation is running.

However, this method is risky when working with undocumented functions. Changes in such functions can result in returning an incorrect value or failing to execute the function, which can lead to a program’s malfunction. Also, there might be complications with reversing input parameters, limiting opportunities for using them.

4. Add a hook to another function. Last but not least, your developers can add a hook to another function that will help identify whether the target operation is running at the moment and then save the state of the target operation.

Sometimes, reverse engineers might need to use two different functions: one to help them identify which operation is running and another to change a program’s behavior. Since both functions are used for the same operation, developers can use both of them and install a flag for a required operation. This method is the least optimized among all four, as it requires first using any one of the three methods described above and then using an additional hook.

Once developers identify operations, it’s time to implement hooks that will control them. Let’s briefly discuss what API hooking is and how it works.

2. Create and implement API hooks

The next step you need to take to efficiently reverse engineer APIs is setting hooks.

API hooks allow you to intercept API calls and perform additional actions before and after the actual call. Hooking can provide you with control over software or operating system behavior. Therefore, API hooks are often used in cybersecurity solutions and programming tools.

Using API hooks to modify operations is convenient because you don’t need access to the program’s source code. Also, this approach offers flexibility: even when the target app’s version is changed, API hooks will work as long as the target application uses the same APIs.

On the downside, such hooks introduce additional code, which can cause a decrease in performance and affect the target application in an unexpected way. Not to mention that it’s not always possible to accurately identify which operation is running in the first place. Also, in the case of using API hooks, if the application changes its API calls for the target operation, your team will need to research the operation again and update the hooks.

Below, we explore examples of local hooks that will only influence specific operations of our target program. Since we’ve already described the process of setting hooks in our previous article — Windows API hooking tutorial (example with DLL injection) — in this article, we focus on using previously defined methods for detecting operations inside of hooks.

API reversing in practice: task and examples

To show you how to reverse engineer an API call, we’ve chosen a painting program called Krita that allows users to create raster images. Since it’s an open-source application with the GNU GPL v3 license, we can access its source code to better understand the application architecture.

For the purpose of this article, we’ll assume that we’re working on a prototype for securing files from information leaks. So, we’re going to implement an algorithm that will allow us to safely contain sensitive information inside Krita.

We’ll try to block the following operations:

1. Save as
2. Copying information from a file and pasting it to another program
3. Copying information from a file and pasting it to the same program

As for reverse engineering tools, the market offers lots of useful commercial and open-source tools for different purposes. Check out our comprehensive review of the most popular reverse engineering tools for Windows to learn more about such solutions.

To accomplish our task, we’ll use two popular tools for reverse engineering system operations (Process Monitor and Api Monitor), which we describe below. For hooking, let’s use the Mhook library, as it’s open-source and easy to configure.

Read also

9 Best Reverse Engineering Tools for 2023

Find out the main reverse engineering programs Apriorit engineers rely on and explore practical examples of how to use them.

Learn more

Reverse engineer the Save as operation

Let’s start reversing activities with the Save as operation.

For this task, we’ll use Process Monitor, which is a great tool for monitoring Windows operations. It offers convenient filtering options and a call stack (including kernel and user mode calls), as well as support for message sending. It also responds fast. Process Monitor is easy to use, especially for developers who aren’t proficient in Windows API (WinAPI).

You can use this tool to reverse engineer:

• Register activity
• File system activity
• Network activity
• Process and thread activity
• Profiling events

Note: During reversing activities, we found that it was better to use two of the methods for defining operations:

1. Check functions’ input parameters
2. Check the call stack

How do we identify the save as operation? Since during the save as operation, Krita creates or rewrites a file, we know for sure that a file system is used.

Our idea is to create and save a file named MyPaint.kra so we can detect the call responsible for creating a new file.

First, we create a file named MyPaint.kra. Then, we launch Process Monitor and set the following filters:

• Process Name with the krita.exe value
• Path contains with the MyPaint.kra value

Now, it’s time to launch process monitoring and save a file from Krita to our computer using the Save as operation. To find the call responsible for creating a new file, let’s check the Results column, parameters, and the operation call stack:

In the Details column for one of the calls, we notice the following information:

Result SUCCESS

Desired Access: Generic Write, Read Attributes, Disposition: Create…

Let’s also check the call stack:

In screenshot 3, you can see that the CreateFileW function is being used. We can use this function to check the input parameters and block the creation of a new file.

Let’s also check the call stack to see whether we find Save as operation frames and whether we can add a call stack check to a hook.

Here, we see frames pointing to the Save as operation.

Note: In the call stack, it’s important to pay attention to the offset, since a large offset can be a sign of another function (not a .dll export) being used in the module. This makes the process of checking the operation by signature more sensitive to a change in the binaries of the target program.

Related project

Developing a Custom Secrets Management Desktop Application for Secure Password Sharing and Storage

Explore the real-life case of creating a custom application for secrets management that works on macOS, Windows, and Linux.

Project details

Block the Save as operation

Now, let’s move to blocking the Save as operation.

We know that for this operation, the target application uses the CreateFileW call with the following parameters:

• Desired Access: Generic Write, Read Attributes
• Disposition: Create
• Options: Synchronous IO Non-Alert, Non-Directory File
• Attributes: N
• ShareMode: Read, Write
• AllocationSize: 0
• OpenResult: Created

We also know that the call stack includes the following frames:

• libkritaui.dll _ZN13KisMainWindow12saveDocumentEP11KisDocumentbbb + 0x1689
• libkritaui.dll _ZN13KisMainWindow14slotFileSaveAsEv + 0x32

Let’s create a hook for the CreateFileW call and name it CreateFileWHook:

// Define the function's signature
typedef HANDLE(*CreateFileWFunc)(
    LPCWSTR               lpFileName,
    DWORD                 dwDesiredAccess,
    DWORD                 dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD                 dwCreationDisposition,
    DWORD                 dwFlagsAndAttributes,
    HANDLE                hTemplateFile
    );
 
// Get address of the original CreateFileW function
auto OriginalCreatefileW = CreateFileWFunc(
    GetProcAddress(GetModuleHandleA("kernel32.dll"), "CreateFileW")
);
 
HANDLE HookCreateFile(
    LPCWSTR               lpFileName,
    DWORD                 dwDesiredAccess,
    DWORD                 dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD                 dwCreationDisposition,
    DWORD                 dwFlagsAndAttributes,
    HANDLE                hTemplateFile
)
{
 
    return OriginalCreatefileW(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes, dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);
}

Now, we need to add a check that will compare these input parameters (dwDesiredAccess, dwCreationDisposition, dwShareMode) against those we found when reversing the Save as operation (screenshot 2) :

if (dwDesiredAccess == GENERIC_WRITE &&
        dwCreationDisposition == CREATE_NEW &&
        dwShareMode == (FILE_SHARE_READ | FILE_SHARE_WRITE))
    {
        return INVALID_HANDLE_VALUE;
    }

To minimize the chance of false positive results, let’s also add a check for stack trace:

//  Add GetStackTraceFunctions, a helper function to get a stack trace with function names.
auto st = GetStackTraceFunctions();
 
// Functions in the call stack that we found during reversing activities
std::vector<std::string> funcs;
funcs.push_back("ZN13KisMainWindow12saveDocumentEP11KisDocumentbbb");
funcs.push_back("ZN13KisMainWindow14slotFileSaveAsEv");
 
// If functions are present in the current call stack, we have blocked specifically the Save as operation.
if (FindSubVector(st, funcs))
{
    return INVALID_HANDLE_VALUE;
}

Done! Now data is protected from the Save as operation:

As a result, we’ve introduced security limitations to eliminate the possibility of information leaks when working with files in the Krita application via the Save as operation. Now, let’s explore how to do the same for Copy and Paste operations.

Related project

Developing a Custom Driver Solution for Blocking USB Devices

Find out how to enhance an enterprise product by adding functionality for blocking restricted USB devices.

Project details

Reverse engineer the Copy and Paste operations

Let’s reverse the Copy and Paste operations. Since a file system isn’t being used for these operations, we’ll try to reverse APIs responsible for interacting with the clipboard.

For this task, we’ll use API Monitor. This is a free tool that allows engineers to monitor API calls of a process and its sub-processes, gathering helpful information about a call’s parameters and call stack. API Monitor helps you thoroughly regulate monitoring parameters, offers convenient filtering options, and allows for monitoring of API calls from custom modules. We used API Monitor v2, which was still in Alpha at the time of writing.

Note: During reversing activities, we found out that it was better to apply the third method for determining the target operation: use functions responsible for the operation. Below, we explore this method.

To find hints that point to Copy and Paste operations, we’ll monitor the Windows Application UI Development/Data Exchange/Clipboard category. Let’s start with the Copy operation:

In the screenshot above, we can see that the SetClipboardData API is called from the ole32.dll library and that the format parameter value is 49171.

The SetClipboardData call stack shows us the OleSetClipboard location and the following functions inside Krita that prove that this is a Copy operation:

• libkritaui.dll _ZN14KisNodeManager21copyLayersToClipboardEv + 0x89
• libkritaui.dll _ZN23KisCutCopyActionFactory3runEbbP14KisViewManager + 0x1286
• libkritaui.dll _ZN19KisSelectionManager4copyEv + 0x79

Now, we need to detect the Paste operation. To do that, let’s add the ole32.dll library to the API Monitor filter, since we know that Krita is using ole32.dll for the Copy operation.

Here, we also see the Ole32.dll module and the call to the OleIsCurrent function. The following frames in the call stack additionally prove that this API call is running during the Paste operation:

• libkritaflake.dll _ZN10KoSvgPasteC1Ev + 0x31
• libkritaui.dll _ZN21KisPasteActionFactory3runEbP14KisViewManager + 0x10e
• libkritaui.dll _ZN19KisSelectionManager5pasteEv + 0x76

Read also

Practical Comparison of the Most Popular API Hooking Libraries: Microsoft Detours, EasyHook, Nektra Deviare, and Mhook

Discover what powerful libraries make it easy to implement API call hooking. Find a practical comparison of the top four API hooking libraries as a bonus.

Learn more

Block the Copy and Paste operations

Let’s start blocking Copy and Paste operations to protect data stored in a file from potentially leaking.

We already know that Krita sets the clipboard via the OleSetClipboard API and checks whether the clipboard was changed using the OleIsCurrentClipboard function. Therefore, we can block Copy and Paste operations in the following way:

1. Create a hook for the OleSetClipboard API
2. Change the input parameter to an empty object inside the hook

Here’s the code for OleSetClipboardHook:

// Define the function’s signature
typedef HRESULT(*OleSetClipboardFunc)(
    LPDATAOBJECT pDataObj
    );
 
// Get the address of the original OleSetClipboard function
auto OriginalOleSetClipboard = OleSetClipboardFunc(
    GetProcAddress(GetModuleHandleA("ole32.dll"), "OleSetClipboard")
);
 
HRESULT  HookOleSetClipboard(
    LPDATAOBJECT pDataObj
)
{
    // Create a new clipboard data object
    IDataObject* pNewData = nullptr;
    HRESULT hRes = CreateDataCache(nullptr, CLSID_NULL, __uuidof(IDataObject), reinterpret_cast<void**>(&pNewData));
    if (FAILED(hRes))
    {
        OriginalOleSetClipboard(pDataObj);
    }
 
    return OriginalOleSetClipboard(pNewData);
}

As a result, after the Paste operation, the OleIsCurrentClipboard function will always return False. Thus, there’s no way to copy information from the file and paste it to another Krita file or another application.

Conclusion

Knowing how to reverse engineer an API to efficiently and securely take control of application operations is a proven way to improve your app’s cybersecurity, even without access to source code.

When securing your product from data leaks by intercepting API calls and creating custom hooks, it’s essential to know key nuances of doing so and be aware of pitfalls that might arise along the way. While API hooks help you control operations in the target application, they also might cause unpredictable solution behavior when implemented without thorough research and preparation.

At Apriorit, we have teams of professional reverse engineering and cybersecurity specialists with rich experience analyzing and improving the cybersecurity of software products without affecting their performance.