Windows API Hooking Tutorial (Example with DLL Injection)

The current article is devoted to an easy approach for setting up global API hooks on a system-wide scale. For DLL injection, we will utilize a registry key called AppInit_DLLs, and to perform API hooking in Windows, we will utilize the Mhook library. This article will also provide you a DLL injection example: we will demonstrate how you can easily make the calc.exe process invisible in the running process list.

About API hooking

Windows API hooking is a process allowing to intercept API function calls. This gives you the control over the way operating system or a piece of software behaves. Some of the software solutions that utilize hooks include: antimalware software, application security solutions, security monitoring tools, system utilities, tools for programming, and many others.

API hook types

API hooks can be divided into the following types:

• Local hooks: These influence only specific applications.
• Global hooks: These affect all system processes.

The type of hook technique for Windows that we cover here belongs to the global type. It affects all processes across all sessions (as opposed to the SetWindowsHooks method, which is limited only to a selected desktop).

AppInit_DLLs infrastructure

The AppInit_DLLs infrastructure loads a predefined set of DLLs to all user-mode processes connected with the User32.dll library (in fact, there are almost no executables, which wouldn’t be connected with it). When User32.dll is initialized, it loads the corresponding DLLs, thus performing the DLL injection into processes.

To change the way the AppInit_DLLs infrastructure behaves, you need to configure the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT \CurrentVersion\Windows registry key values. The following values are available:

Mhook library

Several API hooking libraries exist. Typically, they do the following:

• Replace the initial part of a defined function code with our own code (also known as trampoline). Upon execution, the function jumps to a hook handler.
• Store the original version of the replaced code of the defined function. This is required for the defined function to operate properly.
• Restore the replaced part of the defined function.

As I mentioned before, when building our global hooks, we will use Mhook library. It is a free and easy-to-use open-source library for Windows API hooking supporting x32 and x64 system architectures. Its interface isn’t complicated and is self-explanatory:

BOOL  Mhook_SetHook(PVOID *ppSystemFunction, PVOID pHookFunction);
  BOOL  Mhook_Unhook(PVOID *ppHookedFunction);

More details on how to use the library is available further in the article and on the Mhook home page.

Implementation Example

For this example, we will use C++ to write a user-mode DLL to illustrate DLL injection techniques. To do this, the latest version of the Mhook sources is required, which will be added to your project. Please note, any precompiled headers must be disabled for Mhook files.

As we have already said, to provide an API hooking example, we will make the calc.exe process invisible in the list of processes – for any Windows tool representing such list. This example will demonstrate how to create and inject DLL into a process, thus setting up a global API hook – and creating a kind of appinit_dlls rootkit .

Source Function

To list running processes, you need to call the NtQuerySystemInformationNTAPI function. This means that our project requires some NTAPI stuff. As we cannot find the full information in the winternl.h header, the data types must be defined manually:

//
// Defines and typedefs
#define STATUS_SUCCESS  ((NTSTATUS)0x00000000L)
typedef struct _MY_SYSTEM_PROCESS_INFORMATION 
{
    ULONG                   NextEntryOffset;
    ULONG                   NumberOfThreads;
    LARGE_INTEGER           Reserved[3];
    LARGE_INTEGER           CreateTime;
    LARGE_INTEGER           UserTime;
    LARGE_INTEGER           KernelTime;
    UNICODE_STRING          ImageName;
    ULONG                   BasePriority;
    HANDLE                  ProcessId;
    HANDLE                  InheritedFromProcessId;
} MY_SYSTEM_PROCESS_INFORMATION, *PMY_SYSTEM_PROCESS_INFORMATION;
typedef NTSTATUS (WINAPI *PNT_QUERY_SYSTEM_INFORMATION)(
    __in       SYSTEM_INFORMATION_CLASS SystemInformationClass,
    __inout    PVOID SystemInformation,
    __in       ULONG SystemInformationLength,
    __out_opt  PULONG ReturnLength
    );

The creation and initialization of a global variable allows us to store the address of an original function:

//
// Original function
PNT_QUERY_SYSTEM_INFORMATION OriginalNtQuerySystemInformation = 
    (PNT_QUERY_SYSTEM_INFORMATION)::GetProcAddress(::GetModuleHandle(L"ntdll"), 
    "NtQuerySystemInformation");

Read also

Practical Comparison of the Most Popular API Hooking Libraries: Microsoft Detours, EasyHook, Nektra Deviare, and Mhook

Whatever task your team has to tackle ー debug tasks, build sandboxes, or enhance browser security ー make sure to hook APIs efficiently with the most suitable libraries. Read the full text to find helpful advice from Apriorit’s development experts.

Learn more

Function after hooking

After a function has been hooked, first it calls the original function. Then we examine SystemInformationClass. In case it reveals to be SystemProcessInformation, in the list of running processes, we need to find and remove all records related to calc.exe. And that’s it!

Please note that the original and hooked functions must have identical signatures.

//
// Hooked function
NTSTATUS WINAPI HookedNtQuerySystemInformation(
    __in       SYSTEM_INFORMATION_CLASS SystemInformationClass,
    __inout    PVOID                    SystemInformation,
    __in       ULONG                    SystemInformationLength,
    __out_opt  PULONG                   ReturnLength
    )
{
    NTSTATUS status = OriginalNtQuerySystemInformation(SystemInformationClass,
        SystemInformation,
        SystemInformationLength,
        ReturnLength);
    if (SystemProcessInformation == SystemInformationClass && STATUS_SUCCESS == status)
    {
        //
        // Loop through the list of processes
        //
        PMY_SYSTEM_PROCESS_INFORMATION pCurrent = NULL;
        PMY_SYSTEM_PROCESS_INFORMATION pNext    = (PMY_SYSTEM_PROCESS_INFORMATION)
        SystemInformation;
        
        do
        {
            pCurrent = pNext;
            pNext    = (PMY_SYSTEM_PROCESS_INFORMATION)((PUCHAR)pCurrent + pCurrent->
            NextEntryOffset);
            if (!wcsncmp(pNext->ImageName.Buffer, L"calc.exe", pNext->ImageName.Length))
            {
                if (0 == pNext->NextEntryOffset)
                {
                    pCurrent->NextEntryOffset = 0;
                }
                else
                {
                    pCurrent->NextEntryOffset += pNext->NextEntryOffset;
                }
                pNext = pCurrent;
            }            
        } 
        while(pCurrent->NextEntryOffset != 0);
    }
    return status;
}

Windows hook set up

It does not require much effort to set up a hook: you simply need to call Mhook_SetHook from DllMain after loading a DLL to a new process:

//
// Entry point
BOOL WINAPI DllMain(
    __in HINSTANCE  hInstance,
    __in DWORD      Reason,
    __in LPVOID     Reserved
    )
{        
    switch (Reason)
    {
    case DLL_PROCESS_ATTACH:
        Mhook_SetHook((PVOID*)&OriginalNtQuerySystemInformation, 
        HookedNtQuerySystemInformation);
        break;

Unhooking

To reverse hooking, you need to call Mhook_Unhook from DllMain after unloading the DLL from the process:

//
// Entry point
BOOL WINAPI DllMain(
    __in HINSTANCE  hInstance,
    __in DWORD      Reason,
    __in LPVOID     Reserved
    )
{        
    switch (Reason)
    {
    ...
    case DLL_PROCESS_DETACH:
        Mhook_Unhook((PVOID*)&OriginalNtQuerySystemInformation);
        break;
    }

Related project

Improving a Windows Audio Driver to Obtain a WHQL Release Signature

Check out a successful story of improving the audio driver for the Windows version of client’s AI-powered sound filtering application. Discover how Apriorit’s tech professionals helped our client successfully obtain a WHQL certification.

Project details

API hooking sample execution

Now we will demonstrate how our DLL hook works. Follow these steps:

1. Build the project and place the AppInitHook.dll, which you will have in the result, to the disk C root.

   

2. In the Windows Registry Editor, locate the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT \CurrentVersion\Windows key and select the AppInit_DLLs value.

   

3. Edit the value and enter the path to the DLL hook (in our example, it is C:\AppInitHook.dll).
4. After you finish editing the registry, the hooks starts functioning.

Now let’s launch several instances of the process that has been hidden. After that, examine the processes in the Windows Task Manager: the calc.exe is absent from the list.

As the provided API hook is global, we can see that the same result is displayed by other programs with functionality similar to Windows Task Manger. For example, Process Explorer from Mark Russinovich.

The final check: open the command line and run tasklist.exe.

The process of standard Windows calculator and all its instances have been successfully hidden. The API hook works as expected.

Limitations

Now we need to say a few words about the limitations to this method:

• Connection to User32.dll: As we already said in the beginning of the article, only the processes that are connected to User32.dll can be affected.
• Only functions from Ntdll.dll and Kernel32.dll can be called: The reason for this is that DLL hooking takes place in DllMain of User32.dll and no other library is initialized at that moment.
• Windows 7 and Windows 2008 R2 security features: These features require AppInit DLLs with digital signatures. This isn’t really a big issue as the features can be disabled via Windows Registry Editor.
• No spaces in the full path to an AppInit DLL are allowed.

References

1. How to work with the AppInit_DLLs registry value
2. AppInit DLLs in Windows 7 and Windows Server 2008 R2
3. API hooking revealed
4. Mhook, an API hooking library, v2.2
5. Microsoft Research’s Detours
6. DllMain Callback Function
7. Download sourses

I you’re interested in learning more about different API types, check out our article on how to implement the Places API in a web application.

Conclusion

Knowing how to efficiently set API hooks can help your development team enhance your software, especially if you’re working on a cybersecurity solution. To achieve the best results, delegate challenging tasks to engineers with relevant expertise and experience.

At Apriorit, we have dedicated teams with strong skills in cybersecurity development and reverse engineering ready to assist you with any project for Windows or other operating systems.