Honeypots as a Method of Malware Detection

Have you ever thought about hunting down malware using bait? In cybersecurity, there’s one particular technique that can be used as bait for attackers: honeypots. In this post, we give a detailed explanation of the nature of low-interaction and high-interaction honeypots, describe the most common techniques that malware uses to escape them, determine the value of honeypots, and list the main pros and cons of this approach.

What are honeypots?

First of all, let’s clarify what a honeypot is.

Basically, a honeypot is a security system made specifically for attracting different kinds of attacks. Honeypots work pretty much the same as any bait – they make themselves an easy target so hackers will attack them first. These systems allow you to detect virus attacks and intrusions at an early stage while also preventing these attacks from reaching your real assets, so you can use honeypots as a security measure.

A common honeypot consists of two components: a vulnerability emulation system and a monitoring tool. The first is used to make the attackers think that there’s a weak spot in your system that they can compromise. The monitoring system, in turn, notifies you once attackers take the bait and try to exploit the vulnerability you’ve left exposed.

Developing honeypots

Creating the vulnerability emulation system is one of the biggest challenges in honeypot development, with the level of operating system interactions being the main source of problems. You see, the more malware actions you want to detect, the higher the number of operating system interactions. Plus, you need to know not only a set of possible malicious actions performed by a particular virus or malware but also where exactly you can detect such actions. The complexity of operations required for catching malware with a honeypot may range from writing a file system filter driver and injecting it into the system to measuring system performance and hooking memory allocation functions.

Another great challenge of developing a honeypot for detecting virus attacks and malware hides in the difficulty of emulating the behavior of a real system. Implementing a sandbox paired with a monitoring tool is technically complex.

Since you can’t predict what kinds of actions malware will perform once activated or what kind of data it’ll need, you can’t create a fully secure sandbox. So even when placed in a honeypot, malware should be given some freedom within the real system; otherwise, it may leave the honeypot without initiating any suspicious actions and remain undetected. At the same time, you need to make sure that malware won’t be granted any excessive access to prevent your system from any irreparable damage.

In this regard, honeypots are divided into two categories:

• Low-interaction honeypots
• High-interaction honeypots

Let’s look closely at each of these categories.

Low-interaction honeypots

Low-interaction honeypots emulate only the basic aspects of a particular vulnerability. For example, when you need to detect an attempt to access a computer with a web server on it, there’s no need to install a whole web server. You can implement a small script that will emulate an open port on the system and some basic replies, then use it to deceive the vulnerability scanners used by hackers.

The best thing about low-interaction malware detection systems is that they’re easy to deploy and maintain and work well as a first line of the intrusion detection system. There are low-interaction honeypots that can emulate a sub-network of 65,000 machines with various web servers and services running on them. However, they can capture only a limited amount of information and aren’t suitable for detecting high-level attacks performed through a legitimate connection.

High-interaction honeypots

In contrast to low-interaction honeypots, their high-interaction analogs allow hackers to continue freely with the attack up to a particular point. The best thing is that high-interaction honeypots make it almost impossible for hackers to tell if it’s a real environment or a virtual environment that was specifically crafted to deflect the attack.

Figure 1 shows the basic structure of a high-interaction honeypot.

High-interaction honeypots can redirect the attacker to some virtual machine or a sandbox or use real dedicated machines to make the environment look as realistic as possible. These malware detection systems are much harder to develop due to the difficulty of creating multiple environments simultaneously.

Now, let’s take a closer look at advantages and disadvantages of honeypots.

Pros and cons of using honeypots

Using a bait system to draw the attention of hackers has both advantages and drawbacks. Here are the main benefits of using honeypots for malware detection:

• They help to prevent hackers from attacking your real assets
• They have a lower rate of false positives
• You can use logs collected by honeypots to learn about the capabilities and intentions of hackers

As for the drawbacks of detecting malware with the help of honeypots, there are several challenges depending on the type of malware you need to detect. Below, we list some of the most common types of malware you may try to catch with the help of a honeypot and the challenges that come with them.

Bootkit installation. To detect a bootkit installation, there have to be several environments, one per operating system. This way the bootkit can run successfully and execute its malicious actions so the monitoring system will be able to detect it. When there is only one environment, the bootkit may just check the operating system version and exit the honeypot without manifesting its real intent.

Malware sent via email. To detect malware sent via email, you must have multiple email clients and browsers of different versions used by real users. Plus, your browsers should have some of the commonly exploited vulnerable plugins. The most challenging part of this process is the need to handpick software that’s widely used by both regular users and hackers and to avoid compatibility issues when installing that software on your honeypot. Setting up a high-interaction honeypot for detecting malware sent by email is a non-trivial task.

Fileless malware. To catch fileless malware running as a PowerShell script, you’ll need to create a honeypot environment where that script can still run natively but some events can be monitored during its execution. The monitored events should then be classified to see the difference between the execution of normal and malicious scripts.

Also, high-interaction honeypots must be monitored heavily in order to distinguish malicious behavior from normal behavior and to prevent the malware from escaping the sandbox/VM. It takes time to understand at which points you can make the needed measurements to create a normal behavior profile which is then used to detect anomalies.

Finally, you need to determine the stopping point so that the honeypot can be terminated in time and the malicious code will be prevented from spreading further.

High-interaction honeypots and the catch-22

In some cases, developing a high-interaction honeypot network security system causes a contradictory issue called a catch-22: emulating system components is necessary for detecting malicious behavior, but it also may affect the code flow in the malicious code so that malicious behavior isn’t triggered and, therefore, remains undetected.

For instance, you can virtualize file system access to secure the environment from being broken by a virus. However, since you can’t know each and every aspect of the file system API for each operating system version, there’s a chance of not properly emulating some operating system-specific features used by a particular malware. And without these features being present in your honeypot, the malware will remain inactive and, as a result, go undetected.

Scalability challenges

Despite the development complexity, the main problem of using high-interaction honeypots for detecting virus attacks and malware is scalability. Storing individual environments requires a lot of disk space, and in order to have some available space, you’ll have to spend a lot of time cleaning up these environments. At the same time, it won’t affect the further malware execution since there will still be some artifacts remaining in the system.

Anti-honeypot techniques

Besides mundane attempts to detect a sandbox or a VM, malware may use additional techniques to detect that it’s being executed within a honeypot. Below, we list some of the techniques that malware can use for detecting these systems.

Check the number of connections. Honeypot systems with multiple environments are used to maximize the detection ratio, so it’s common to run the same sample on several environments simultaneously. Malware can connect back to Command & Conquer (C&C) site. Judging by the number of connections from the same network, C&C can determine that something strange is happening and block all further requests. As a result, the malware won’t go through the entire attack chain.

Check the number of infection attempts. Malware may use so-called run once tactics to identify a honeypot and block further execution if more than one attempt is made to infect a particular host. In that case the loader won’t be provided with the payload and no malware behavior will be triggered.

Conclusion

Honeypots present an interesting method of malware detection. You can use honeypots to improve your network security. This approach can help you detect different attacks and understand their nature without putting your real assets at risk. At the same time, building a honeypot-based intrusion detection system is associated with a set of challenges such as development complexity and increased consumption of system resources.

At Apriorit, we feel strongly about ensuring data protection and can offer our assistance with even the most challenging cybersecurity projects.