Mail Server Security: Potential Vulnerabilities and Protection Methods

This article covers the topic of security for email servers. Security measures covered here will allow you to greatly increase the level of protection for your email server and prevent any attacks from succeeding.

1. Introduction

1.1 Challenges of server software security

With the constant development of information technology, the role of cybersecurity is getting ever bigger. While it’s impossible to imagine the modern world without constant communications over networks, almost all valuable data that’s a target for attacks is stored in various forms on servers. Not to mention that the stability of your whole system depends on servers. This is why servers are attractive targets for malicious attacks.

1.2 Email server security

Security for an email server is particularly important because email is one of the most popular means of communication and doing business. And for businesses, in particular, loss of confidential information can result in large financial losses. It’s also important for a server to run stably so that users are able to access it at any time. When a server is unstable, it can lead to the loss of customers.

1.3 Standard methods of software protection

In order to avoid data loss, problems with stability and other troubles, we need to adhere to general recommendations on how to set up an email server and monitor its security in order to detect and immediately fix any vulnerabilities. This is the main topic of this article. We’ll show examples of security measures and various problems that can arise if you don’t adhere to them. We’ll also talk about some features of certain email servers and examples of their vulnerabilities, together with general recommendations for testing.

2. Potential vulnerabilities

Every system has vulnerabilities, so every system is at risk of a cybersecurity breach. Of course, dealing with all of them is impossible, but we can greatly reduce their number. In order to do this, we need to stick to best practices when setting up email server security. These best practices and problems that can arise when you don’t follow them will be the focus of the next part of this article.

2.1 Unauthorized access to data

One widespread type of attack is when a perpetrator tries to bypass the authentication procedure in order to get access to data.

The first thing you need to do to avoid this is to establish strong requirements for the password used to access the server. This prevents the password from being cracked via brute force, which is a universal way to bypass authentication. Everything else depends on the server type. Different types of servers use different operating systems, interfaces, etc.

Another way to protect the server from cracking is SMTP authentication. We’ll look at this in more detail in a section that covers various problems with performance and stability.

2.2 Threat of data leakage

Personal data is one of the key targets for hackers. When an email is sent via the internet, it goes through unprotected communication channels. Passwords, user names, and messages themselves can be intercepted. In order to prevent this, you need to encrypt both incoming and outgoing mail. SMTP, POP3, and IMAP protocols should be encrypted with SSL/TLS.

2.3 Spam

Spam is one of the biggest problems when it comes to email.

From the server security standpoint, we can divide the threat of spam into two categories:

• Sending external spam messages to your own clients
• Sending external spam messages to other clients (In this case, the server acts as an Open Reply.)

To prevent this threat, you need to use content filters. They’re installed either on the mail server or on a proxy application to protect access to the server (such as on a firewall, proxy component of the mail server, etc.). In addition to content filters, you can also use blacklists of known spam servers: for example, DNS-based blacklists (DNSBL), Spam URI RBL SURBL, and local blacklists of IP addresses of spam senders.

To prevent Open Relay, you need to properly configure the Mail Relay parameter of the email server.

When testing anti-spam protection, we analyze the effectiveness of content filters.

2.4 Threat of malware

Both servers and email clients are susceptible to malware. When an email server is infected, the stability of the whole system is compromised. Integrity and privacy of personal data are threatened. Malware spreads among email clients mostly thanks to infected attachments.

Protection from malware involves both built-in tools and third-party antivirus software.

2.5 DoS threat

The damage that a DoS attack can do to a mail server is hard to overstate. It leads to unreceived and unsent emails, not to mention time spent trying to restore the service. Ultimately, the reputation of the whole company suffers.

To prevent such a threat, you need at least to limit the number of possible connections to the SMTP server. In order to cope with SMTP security issues, look into limiting the overall number of connections over time, as well as simultaneous connections.

2.6 Server performance and stability

When we see the words “server” and “performance”, we immediately think about load balancing.

If your server is attacked and stops working, you need to have a plan B. In such cases, we often use a reserve server. For email servers, in particular, this is done by having two MX records for each domain.

Email servers also have an option to use SMTP authentication. If it’s enabled, then in order to send an email to the server you need to provide an additional username and password.

It’s very important to enable SMTP authentication. It allows you to protect the server from an attack consisting of numerous sent requests, ensuring continuous operation.

It’s also very important to configure the mail relay. You can specify from which IP addresses the server can send mail. You can also prevent a large number of messages from being sent to destabilize the server. Another filter for email sent from clients is Reverse DNS. It’s used to compare an IP address with the domain and hostname. It also serves to protect the server from malicious mail.

2.7 Ignoring best practices, and other issues

One of the most relevant best practices is to not store anything unnecessary on the server. You need to carefully consider whether additional software, installed on a server can be used by a perpetrator. Check each of your open network ports to make sure that they’re necessary (and if not, immediately close them) and protected (for example, check whether authorization is required to send data via a port).

You also need to update all the software on the server. Despite the best efforts of developers and testers, no software is 100% error-free. There are whole lists of software vulnerabilities that are easily accessible for malicious hackers. When new vulnerability or exploit is detected, software vendors will usually issue a fix over the course of several days. If the server doesn’t receive this update on time, perpetrators will be able to use this vulnerability to their advantage.

You also shouldn’t dismiss the human factor. Server uptime shouldn’t rely on only one person. Also, you should trust a server only to people who will be responsible for it.

3. Detecting and analyzing vulnerabilities

At the start, it’s necessary to design the approach that will be used to monitor mail server security. In many cases, it’s easier to look at the available solutions. Numerous companies provide cybersecurity audits. But in certain cases, it can be necessary to solve the problem yourself. How tough and formal this process should be depended on the task at hand.

Next, we’ll look at a compromise, where we use a formal, yet flexible approach.

3.1 Preparing the necessary documentation

First, you need to decide what you need to check, why and how. All three questions are important because you need to cover as much ground as you can with your audit without wasting your time on unnecessary details.

• What. Create a list of all data (usernames, contact lists, attachments, etc.) and parameters (performance, uptime, etc.) that you think you need to track and check for vulnerabilities. This list can be divided into several checklists with different areas of responsibility (server, network, operating system). Each entry in the list should be weighted according to the impact that potential problems with it may cause.
• How. Apply two approaches:
  1. Using the previously created list of components, search for tools and utilities that can be used to check whether a particular component is vulnerable. Each component should be assigned a corresponding method of checking. All those methods will be our tests or objects for monitoring.
  2. Using the list of potential vulnerabilities, expand your list of objects to monitor with additional controls, allowing you to check for these vulnerabilities. In case some entries repeat, mark them but don’t remove them.
• Why. Check the target, weight, and how much each individual check covers in order to assign a priority to it. At this stage, you can remove repeat entries from your list of components, but only if the deleted component is fully covered by some other entry or combination of entries. Otherwise, keep the repeated entry but assign it a lower priority.

After that, we need to make a decision on whether each entry is worth the resources necessary to run it. We need to estimate the time and money needed to buy software, train employees and execute checks. Some components with middle and low priority can be excluded from the list if you can’t justify the expense. But they shouldn’t be fully removed from the list; priorities can change any minute.

When creating an application security audit checklist, it’s best to use controls from NIST SP 800-45.

After forming the list, you can establish the scope of work and necessary resources. Depending on the breakdown, you can also create a detailed plan that will fully cover every procedure from start to finish.

3.2 Testing (searching for vulnerabilities)

With the checklist, the process boils down to conducting the checks covered in the Products and Utilities sections of this article. The order of the list will depend on the priority of the task. If there’s limited time then you need to check high-priority components. If there is no time limit than organizing checks based on what’s more convenient can save both time and money.

Some checks can take more time than planned – in this case, you should skip them and move them to a separate pool as well as try to find a way to optimize the process.

All incidents where data and settings from the server were compromised should be logged. Some can be later ignored, but at the current stage, it’s important to not spend time trying to investigate details but rather make sure that the check covers as much as possible within the time available.

3.3 Analysis of detected problems

Analysis boils down to assessing risks based on the formula Impact x Likelihood x Exposure. Each control can be scored from 1 to 5, where 5 is an indication that this problem can’t be ignored and 1 means that solving the problem is most likely more effort than it’s worth.

• The impact of the incident depends on the component compromised. In some cases, it can be extremely small (for example, the server stops working for 100 ms), and sometimes it can be extremely large (database loss). If checklists are correctly constructed, then an impact can be filled in automatically by referencing the checklist.
• Likelihood depends on how repeatable the problem is. It ranges from very rare (running out of memory once a year while constantly sending emails) to stable (always running out of memory when receiving 2²⁵⁶ UDP packets through an open TCP port).
• Exposure refers to whether it’s hard to detect the problem and whether the problem can occur during the course of normal operation of the server. Exposure can vary from impossible (several unlikely problems happening at the same time) to unavoidable (the word “password” used as an admin password, or server down due to receiving 1000 emails).

After an assessment, sort all the problems by risk in descending order (Impact x Likelihood x Exposure). Next, each incident with a risk value greater than 8 needs to be discussed. This discussion should help to categorize incidents into vulnerabilities (for example, loss of data), flaws (a threat of losing loyalty because of possession of excessive data or spam), and problems that can be ignored. Priorities then need to be assigned to each vulnerability and flaw.

3.4 Fixing vulnerabilities

Usually, there are three ways to fix vulnerabilities:

• Switch to a version of other product that doesn’t have the problem
• Install additional software that eliminates the problem
• Disable functionality that exhibits the problem

The main things that need to be considered are the level of associated risks, costs, and the budget for fixes, and necessary resources. All of these things can have a large impact on the creation of a schedule for fixes and the order of tasks.

It’s best to immediately take care of problems that can be solved easily and quickly without putting them off for later. Complex vulnerabilities may require several days to fix, and you’d be better not to leave small, but dangerous problems to simply wait their turn.

In some cases, you can group vulnerabilities that can be solved with a single fix. This can save you money and time, particularly if this approach is considered the most reliable (which means that it wouldn’t require any changes in the future). However, it’s best to not look for silver bullets, as they can prove more complex and expensive than several simple solutions.

4. Example of cybersecurity audit of MS Exchange Server for Windows

4.1 Software and operating system specifics

Microsoft Exchange Server is a popular email server provided by Microsoft. Working only on Windows Server OS, Exchange Server supports both standard (SMTP, POP3, IMAP) and proprietary (MAPI, EAS) protocols.

Let’s look at specifics of Exchange Server with regard to cybersecurity. Here we’ll cover all newer versions starting with Exchange 2010 (official support for Exchange 2007 expired on April 11, 2017).

• Edge Transport Server — This is a server for incoming and outgoing external mail. This server works great for companies with network infrastructure divided into a protected internal network and a protected perimeter, or demilitarized zone (DMZ). An Edge Transport Server is usually located inside the DMZ, while the Mailbox is located inside the private network. An Edge Transport Server provides an additional layer of defense for any messages. This way, the mail server experiences fewer external attacks. Edge Transport is optional during Exchange Server 2010 and 2016 installation but is missing from Exchange 2013.
• Database availability group (DAG) — A component that provides high availability and recovery for data on the server. It was first introduced in Exchange 2010. DAG is a base component of the Mailbox server that ensures availability and data recovery after various incidents.
• Spam protection is achieved via internal antispam agents. They’re available by default on Edge Transport servers starting with Exchange 2010 and can be enabled directly on the Mailbox server (in Exchange 2016)
• Anti-malware protection is achieved with the Malware agent on the Mailbox server. This agent was first introduced in Exchange 2013. In Exchange 2016 it’s on by default.
• Outlook Web Access (OWA) — A mail web client. It doesn’t require the installation of a full-fledged desktop email client.
• Proprietary protocols and services:
  • Exchange ActiveSync — A protocol for email correspondence with mobile devices
  • Exchange Web Services (EWS) — A cross-platform API that provides access to email, contacts, and other data to client applications
  • RPC over HTTP, MAPI over HTTP — Proprietary protocols allowing mail clients to communicate with an Exchange Server

Since a Microsoft Exchange Server supports only Windows Server operating systems, mail server security testing should look into potential malware threats, such as viruses and trojans. In this case, the threat of malware infection exists not only for the client but also for the server.

4.2 Testing scenarios and software to use

Security testing for MS Exchange Server is done with consideration for infrastructure configuration and the environment at hand. To illustrate, let’s look into two examples of infrastructure:

Example 1. The company has an internal network with Internet access and without a network perimeter. An MS Exchange Server is installed with base settings without an Edge Transport Server.

Example 2. The company has an internal corporate network. Internet access is implemented via DMZ. Two Exchange Servers are installed with DAG replication, and there is an Edge Transport Server located inside the DMZ.

The first example is much more vulnerable from the network attack standpoint, but it needs to be considered as there are companies out there that use similar infrastructure. Before we start testing, we configure the infrastructure corresponding to the given use case. We test in ways that correspond to certain types of threats:

Testing for protection from personal data leaks

In this test, we’ll intercept communications between the Exchange Server and email clients.

Testing goes according to the following scenario:

1. The MAPI, SMTP, IMAP, and EAS mail protocols are set up on the mail server.
2. An email client is installed. We’ll test with the desktop (Outlook, Thunderbird), mobile, and OWA web clients.
3. Man-in-the-middle (MITM) hardware is configured, located between the server and the client. This hardware has its own interceptor: Wireshark, tcpdump, or Fiddler.
4. Data is transferred between the client and the server via one of the following protocols: MAPi, SMTP, IMAP, or EAS. Authorizations and further communications are reviewed via the protocol.
5. Network packets are intercepted via MITM and searched for unencrypted data.

Testing spam protection

Example of a scenario for testing Exchange Server spam protection:

1. Configure several local email servers, capable of sending spam to the tested Exchange Server.
2. Send a round of emails to Exchange Server. Use additional scripts to generate.
3. Search target inboxes for spam.

Test with anti-spam filters for Mailbox and Edge Transport Server both enabled and disabled. This will allow you to assess the effectiveness of spam protection.

Testing for protection from emails infected with malware

To test for protection from emails infected with malware, we use a number of test malware attachments. First, we use an EICAR test file called Eicar test virus Standard Anti-Virus Test File. This file actually is not a virus and doesn’t include any parts of a virus, but most antivirus software detects it as malicious.

Also in such a test, we use specially created files: for example, files containing reflective DLL loader code. Essentially, this code isn’t harmful, but most antivirus software will detect it.

An example of a test scenario:

1. Disable the Malware agent for the Exchange Server and Edge Transport Server.
2. Send several emails infected with malware to various clients.
3. Upon receiving this email on a client, search it for malware.
4. Repeat the scenario with the Malware agent for the Mail server and Edge Transport Server enabled.
5. Compare the results.

Testing user passwords

To test the reliability of user passwords, we can use software built into Kali Linux called Hydra. This software identifies weak passwords by tempting to crack them with brute force. Mail server attacks are conducted through SMTP, IMAP, and POP3 protocols.

Testing DoS attack protection

To test DoS attack protection, we need to emulate particular network traffic aimed at destabilizing services of the tested Exchange Server. We also need to emulate various failures in the network. We can use additional software such as WANem to do this.

In this test, we detect how resistant the Exchange Server is to this type of attack and how quickly service can be restored.

While testing high availability, we especially consider whether the database availability group (DAG) component is enabled for an Exchange Server and whether there’s an additional backup server.

4.3 Example of results

Configuration:

Exchange Server 2010 SP3 with Edge Transport Server role enabled.   OS: Windows Server 2008 R2

Infrastructure:

Exchange Server is located on the local network.   The local network is connected to the internet via DMZ.

General Vision:

Exchange Server security was tested for:   Personal data protection Reliability of user passwords Protection from spam While testing, a single user account with a weak password was discovered. In local tests of protection, enabling the anti-spam agent prevented spam emails. Personal data protection testing did not discover any vulnerabilities. Communications via MAPI, SMTP, IMAP, and EAS protocols were tested.

Recommendations:

The security policy with regard to user passwords needs to be strengthened because compromising even a single password can allow perpetrators to compromise private data from other email accounts.   An anti-spam agent needs to always be enabled, and spam server blacklists need to be constantly updated.

5. Example of Zimbra server security monitoring on Linux

5.1 Software and operating system specifics

Zimbra Collaboration Server is a famous product by Sinacor that provides not only enterprise-level email services but also calendar functionality and tools for cooperation used by both large and small companies. The product is free and supports Linux OS. There’s also a paid version with additional features that can be found here:

https://www.zimbra.com/email-server-software/product-edition-comparison

Zimbra can be managed and configured via a web interface.

There are several specifics with regards to the Zimbra mail server and cybersecurity:

1. When updating the operating system, the Zimbra server isn’t updated, but rather removed and installed from scratch. This is why it’s necessary to check all settings after an update. The problem should be solved in version 8.7.
2. Zimbra is open source, allowing anybody to use it and increasing the possibility that vulnerabilities will be discovered.
3. Two-factor authentication: The first part of an authentication procedure involves regular credentials; the second involves some kind of a physical device, such as a USB token or a smartphone.
4. SSL SNI: A server has several certificates for a single IP address and TSP port, which is why it can work with several domains without the need for a unique IP address for each domain.
5. Email security with Postscreen provides an additional security check, protecting from spam bots and limiting the load on the server.
6. S/MIME digital signatures and encryption: You can sign your message and encrypt it before sending.
7. HSM: Zimbra allows for storing an old message in temporary storage, which is cheaper and safer.

5.2 Testing scenarios and software to use

When performing security testing on a Zimbra server, Kali Linux can be used extensively.

• Kali Linux Metasploit even has special modules for Zimbra, such as zimbra_lfi, which can be used to plant your own malicious file on the server.

• Man in the middle. You can also intercept information transferred over the internet between the client and the server. To do this, you can use, for example, arpspoof, which is part of Kali Linux.
• Because the server is controlled via the web interface, you also need to consider the following:

1. First, you can scan for vulnerabilities. Use w3af and sqlmap to do this. With these tools you can, for example, find out credentials for a certain email account.
2. You can also use XSS injection to get credentials of a server administrator
3. It’s also important to check the reliability of the admin password with brute force cracking

• You should definitely check anti-spam and antivirus functionality. To do this, send spam or malicious emails to the server. Zimbra has special settings for these security measures, which is why during testing you can try different combinations.

• Zimbra also allows you to work with an Exchange Server. This feature is available only in paid versions. While testing, you need to both try to intercept information, as well as get Zimbra data stored on an MS Exchange Server.

• And finally, you shouldn’t forget about DoS attacks. Emulate network traffic to try to destabilize the server.

5.3 Example of results

Bug report:	CVE-2013-70-91
Reference:	https://www.cvedetails.com/cve/CVE-2013-7091/
A directory traversal vulnerability in /res/I18nMsg,AjxMsg,ZMsg,ZmMsg,AjxKeys,ZmKeys,ZdMsg,Ajx%20TemplateMsg.js.zgz in Zimbra 7.2.2 and 8.0.2 allows remote attackers to read arbitrary files via a .. (dot dot) in the skin parameter. NOTE: This can be leveraged to execute arbitrary code by obtaining LDAP credentials and accessing the service/admin/soap API.
Global report:
Many file operations are intended to take place within a restricted directory. By using special elements such as “..” and “/” separators, attackers can escape outside of the restricted location to access files and directories that are elsewhere on the system. One of the most common special elements is the “../” sequence, which in most modern operating systems is interpreted as the parent directory of the current location. This is referred to as relative path traversal. Path traversal also covers the use of absolute pathnames such as “/usr/local/bin”, which may also be useful in accessing unexpected files. This is referred to as absolute path traversal.   In many programming languages, the injection of a null byte (the 0 or NUL) may allow an attacker to truncate a generated filename to widen the scope of the attack. For example, the software may add “.txt” to any pathname, thus limiting the attacker to text files, but a null injection may effectively remove this restriction.
Recommendations:
Use an “accept known good” input validation strategy, i.e. use a whitelist of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does.   For any security checks that are performed on the client side, ensure that these checks are duplicated on the server side Use an application firewall that can detect attacks against this weakness.

6. Conclusion

In this article, we covered email server security and various approaches to testing it. We looked into typical threats faced by email servers and also described how vulnerability detection and analysis is conducted.

We looked at Exchange Server and Zimbra, covering popular email servers for both Windows and Linux.

Cybersecurity technologies for email servers are constantly evolving. New versions of popular servers with increased security and fewer errors are constantly released. However, there are certain basic recommendations that you should follow if you want to secure your email server.

6.1 Final recommendations

The question of email server security should be raised at the earliest stages when you plan to install the server because it’s much more cost effective to plan ahead and prevent potential threats.

When you’re planning to install an email server, you need to consider the following:

1. The purpose of the email server. What services will the server support and what kind of data will go through it?
2. Security requirements for the email server
3. Types of users, their privileges, and authentication methods that will be used on the server
4. Where inside the network infrastructure the server will be located
5. What additional software you need to install besides the server itself
6. How the email server will be managed

Consider the expected level of cybersecurity the server will have as well as potential attack vectors. Email server security is also determined by the security of the operating system it’s installed on.

Below you’ll find some basic recommendations on email server security:

• Create network infrastructure in a way that makes the attack surface of your email server as small as possible. For example, create a network perimeter that will shield your private corporate network from the internet. Install proxy applications inside the network perimeter that will serve external emails and will have access to your email server located inside the private network. An example of such an application for Exchange Server is Edge Transport Server.
• Always use an option to encrypt data for every component of your email server, whenever possible. Carefully choose SSL certificates for each component and don’t use self-written certificates.
• Use reliable third-party antivirus software and anti-malware solutions to provide email server security. This will support any built-in anti-malware protection that your server may have.
• Regularly apply updates to your email server. Microsoft releases Security Bulletins with patches for the latest vulnerabilities.
• Create an email backup server and set at least two MX DNS records to ensure stable operation.

Apriorit develops security solutions and conducts security testing, so feel free to contact us.