Detecting Hook and ROP Attacks: Methods with Examples

There are many innovative systems for ensuring software security. But many applications are still vulnerable to hooks and return-oriented programming (ROP) attacks. While it’s impossible to get rid of all vulnerabilities in applications, developers should think about executable-space protection at the programming stage. This article describes some effective methods for the detection of ROP and hooking method attacks that we apply for protecting our applications.

Introduction

Sometimes, our applications suffer from cybercriminals who use hooks or ROP attacks, so we have to find effective methods for protecting them. In this article, I describe a case when I managed to detect that an outsider (a third-party application, malware, or a reverse engineer) was intercepting system calls in our application to change its behavior or monitor its performance.

I also describe two methods of protection against the following types of attacks:

• Hooks that require injecting third-party code into any targeted application to change the rights of memory pages and rewrite source code
• ROP attacks that don’t require any code injection

You can use these methods to protect your own applications or implement them when designing your proactive cyber defense systems for preventing the kinds of attacks just mentioned and even zero-day attacks.

Hooks

Hooks are used for many purposes, but cyber criminals usually use them to change the behavior of applications or operating systems and monitor their performance. There’s a wide range of hooks, but within the scope of this article, I consider only two types:

1. Patching the Import Address Table (IAT)
2. Splicing

In order to hook a function, attackers need to initiate code changes in the memory-loaded application. For patching, they have to rewrite addresses in the IAT. For splicing, they need to change the instructions at the beginning of the function for the JMP instruction. As a result, application code will be changed.

Thus, for detecting a code change, you can substitute all function calls with FunctionForChecking(%necessary API%). Here, you can apply various methods for verifying that the %necessary API% is really the one you need or if it has been replaced by a third party. Here’s how you can do this:

• Check the first bytes of the function for the availability of control transfer instructions that identify abnormal behavior.
• Check all checksums of the function. Changed checksums indicate that instructions have been replaced.
• Ensure that an address for transferring control is within the loaded module where the function should be but not in a third-party-loaded module.

It’s difficult to do this without rewriting the source code. Moreover, now that you know what is hook detection, you can compare the modules loaded in the analyzed process to the original modules. Here’s an example. This kind of hook detection isn’t proactive, however, as it can only detect already installed hooks.

To execute such hooks, third-party code needs to make a write operation to memory. However, to do that, it should obtain rights to write to a memory page. These rights can be received only after the malware code calls the VirtualProtect function. In other words, for intercepting calls to WinAPI in our application, the third-party code needs to use WinAPI itself.

Consequently, we can also intercept VirtualProtect and check it. If VirtualProtect was called by a known module, then we call the original VirtualProtect. We can do this by saving addresses of the beginnings and endings of all modules at the application start and checking whether a calling module is in our list of modules. However, it’s quite difficult to define whether a module has been loaded legally. This method only protects against DLL injection using remote threads. But injection can also be performed using libraries written in the register with the AppInit_DLLs and KnownDlls keys.

For the purposes of attack detection, we can assume that the rights of memory pages should not be changed after modules are loaded into the process’s address space.  

Implementing protection

Code is injected by instancing a DLL, which requires creating a remote thread. During this process, the DLL tries to hook MessageBox. In our case, I used mhook for hook installation. You can learn more about mhook in this article. You can read about the comparison of different hooking libraries here. However, this method also worked when I tried to inject the hook using IAT, as it also requires calling VirtualProtect. After installing the hook, our application displays a message other than the one that was written below for MessageBox.

std::cout << "after pressing ENTER MessageBox will be shown\n";
getchar();
MessageBox(NULL, L"text", L"caption", 0);

After I implemented protection against hooks, the code in my application was the following:

HookDef hookDef;
if (hookDef.Init())ret
{ /* three lines above */ }

Init intercepts VirtualProtect and VirtualProtectEx. When I hooked the mentioned functions using mhook, I got infinite recursion, as mhook itself used VirtualProtectEx for hooking. Thus, I had to add a check, and if the hooking function was VirtualProtectEx, then I used VirtualProtect for applying mhook.

After that, the hook defines a return address by calling CaptureStackBackTrace in order to understand from which module the call was initiated. If it detects that the address transferred to VirtualProtect already belongs to a module loaded to memory, then someone is trying to hook our application.

if (GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS,
    reinterpret_cast<LPCTSTR>(vpAddr),
    &hmodule))
{
    InformAboutHook(callerAddress);
}

Finally, we get the result:

ROP attack

I’ve already explained how ROP attack examples are performed in this article. In this case, I used a ready exploit that lets me call VirtualProtect in the chain of gadgets from msvcp140.dll. You can learn about how this exploit executes in this article. To execute it, I changed gadget addresses, as loading the address of msvcp140.dll is different than it initially was:

The changes in gadget addresses compared to the original ROP exploit are marked with red in the figure above. Addresses are written in little-endian and are 4 bytes each. Thus, you can see that I offset 20 gadgets.

Let’s imagine that in our application a stack is rewritten with the above ROP chain. The exploit execution begins by following the instruction chains, where each ends with a ret instruction for becoming ready to call VirtualProtect. The jump to VirtualProtect code is performed at the last ret instruction.

Read also

Testing Malware Detection Systems: ROP Chain Exploit Example

Explore how you can equip your QA team with knowledge of ensuring software quality when testing malware detection systems.

Learn more

Protection against ROP attacks

How can developers detect ROP attack blocked in applications? It’s obvious that their execution requires manipulation of a stack. Consequently, there’s a range of options for detecting return-oriented programming attacks that are based on the assumption that a stack has been changed. Let’s look at them:

Shadow Stack is a method based on creating a second stack where return addresses are duplicated from the original stack and before returning a function loads the return addresses from both call stacks and compares them: if the records are different, then one of them has been rewritten. What’s interesting is that this method has already been implemented on the processor level.

Change statement call with

push %table_index%
jmp %function%

and ret with

pop ebx
jmp table[%table_index%]

However, this approach doesn’t detect an attack; it prevents an attack and makes it impossible.

The G-Free method adds prologues and epilogues instead of changing instructions. If the results don’t match after encrypting the return address in the function prologue and decrypting it in the epilogue, then the ret instruction won’t execute or will execute incorrectly.

The stack canaries method places known values between a buffer and control data on the stack. These values are checked before returning. If the return address has been changed, then these values have also been changed. Here’s an example of a stack canary implementation in StackGuard.

Anyways, these solutions require binary recompilation. In this article, I want to consider another approach: Last Branch Recording (LBR).

Last Branch Recording

Using gadgets, an attacker can call any system function. A single function call is usually not enough, though; it’s necessary to perform something with more logic. However, creating an ROP chain with more complicated logic requires more time and is also limited by available modules as well as activated address space layout randomization (ASLR). That’s why ROP is often used as a method to bypass some protection and then execute malware code. For instance, you can create an ROP chain for allocating memory for shellcode, calling VirtualProtect, and transferring control for this shell code.

Thus, for detecting ROP attacks we can intercept calls to system functions and try to check how the control flow was transferred to this point: by calling an instruction (normal behavior) or fetching a ret instruction (abnormal behavior).

But how can we get jump addresses? Modern processors are embedded with a mechanism called Last Branch Recording. When activated, the processor records jump addresses into Model Specific Registers (MSRs). The number of recorded branches (which the CPU has jumped from and to) depends on the type of processor. For instance, the Intel Core i5-6200U logs 31 branches. In any case, you still can have an index of the last branch.

You can find MSR addresses for running LBR and reading the index of the last branch and branches themselves here. The hardcoded values mentioned in that article were taken from Intel’s manual, page 1381. Moreover, Intel processors have already applied the Intel Processor Trace feature which allows them to trace more than 32 branches.  

Read also

A Comprehensive Guide to Hooking Windows APIs with Python

Read about the best practices for hooking Windows APIs using Python, as well as the advantages of Python over compiled languages for setting up Deviare and WinAppDbg libraries.

Learn more

LBR implementation

I had to use the kernel module in order to work with MSR. I also tried to activate MSR from user mode by creating a debugged process by installing Dr7 register bytes, but this method is very unstable and limited.

I used the __writemsr and __readmsr functions for MSR input-output. LBR activation was performed by installing the first byte to the address 0x1d9. I used call DeviceIoControl (more about DeviceIoControl) for further communication with the driver in user mode:

IOCTL_ROPPROT_FN fn { (unsigned long long)addr };
IOCTL_ROPPROT_FN fromFnCalled { 0 };
DeviceIoControl(hDriverDevice, IOCTL_ROPPROT_CHECK_LBR, &fn, sizeof(fn), &fromFnCalled, sizeof(fromFnCalled), &dwReturn, NULL);

The address of instruction from which the jump was performed will be recorded after return to fromFnCalled.

A simplified version of the instruction search looks like the following:

do
{
      toBr   = __readmsr(MSR_LASTBRANCH_0_TO_IP + lastLbrIdx);
      lastBr = __readmsr(MSR_LASTBRANCH_0_FROM_IP + lastLbrIdx);
      if (toBr == checkedAddr)
      {           
            *fromAddr = lastBr;
            return ROPPROT_SUCCESS;
      }
} while (lastLbrIdx--);

After that, the caller compares the first byte of the return address with opcodes of the ret instruction (inside the hooked function).

unsigned char byte = *(unsigned char*)fromFnCalled.addr;
if (IsRet(byte))
{
    MessageBox(NULL, L"ROP attack detected!", L"Alert", 0);
    Return false;
}
// call the original function

Here’s how it will look for the user:

After detecting a hooked VirtualProtect, the system appeals to the driver by using call DeviceIoControl with instruction IOCTL_ROPPROT_CHECK_LBR. You can see driver messages in the DebugView window. It goes through the branches beginning from the last recorded one and searches for the address transferred to it by our hooked VirtualProtect (0x402a60). When it finds the address, it defines from where the jump was performed: 0x77696930 (VirtualProtectStub). As it’s just a jmp instruction, it looks at where the jump VirtualProtectStub – 0x6b7a22fc was performed and returns this address. Our application checks the instruction at this address, and if it’s a ret instruction, it sends an alert.

Above, you can see the last gadget that changes the address in esp to the address VirtualProtectStub in eax. After that, the ret instruction redirects to VirtualProtectStub. To the right is a record about this gadget in the ROP chain that gets itself to the stack after the overflow.

If you want to run the driver

If you want to run the driver, you should take into account the following aspects. For driver development, I used the Windows Driver Kit. In addition, I used Visual Studio 2015 for driver compilation.

While the driver most probably will be test-signed, you need to allow the running of such drivers from the command line and then restart your computer:

bcdedit /set testsigning on

In addition, you may need to deactivate secure boot in the BIOS beforehand. Additionally, you may need to add a Debug Print Filter record in the register in order to see the driver output in DebugView.

Read also

Heap Spraying Technique: How to Protect Your Application from the Heap Spray Attacks

Boost your application’s security even further with our experience in detecting heap spraying attacks using hooks and memory protection mechanisms.

Learn more

Conclusion

The described methods are not intended to be the best hooks and ROP detection solutions. They are the only possible approaches. For instance, if you select the last gadget with the jmp instruction instead of ret, then the implemented protection won’t detect an attack. In this case, you need to perform a more detailed analysis of recorded branches or also consider the frequency of ret instructions.

Moreover, I based my defense just on using VirtualProtect, but you also need to protect a range of critical functions like LoadLibrary, which an ROP exploit may use for dynamic loading. The idea of the described method of ROP attack detection was taken from kBouncer, which is included in Microsoft EMET.

As for hooks, you need to take into account that the described protection may interfere with the work of some protectors that use VirtualProtect for their performance. As for Microsoft EMET, it also uses the VirtualProtect hook in its Memory Protection feature.

Here you can download examples of hooking detection and ROP attack detection:

hookdef_src

ropprot_src