Software Testing for Antivirus Compatibility

The most of testers at least once performed testing for compatibility of the developed application with the antiviruses. It’s widely known that antivirus software influences significantly the system functioning and also the work of the particular programs, and nowadays, the antivirus is almost must-have part of the computer system. Moreover, there are a lot of diverse malware protection products, and sometimes they can be unpredictable.

So, to adequately check the product for the compatibility with the antivirus software, you should apply some creative approaches. It is natural that you’ll need some knowledge for that – this is what we aim in this article.

In the first part of this article, we’ll consider the main types of antivirus programs, their features and specialties, and also the current statistics of antivirus product popularity. This is the theoretical minimum that a tester should know to start planning and performing antivirus compatibility testing. In the second part, we’ll discuss the role of such testing and its place in the software product life cycle, and also consider the problem situations that can appear during compatibility testing. This information is especially useful when planning and performing tests.

 

General information. Types and specialty of antivirus software

Antivirus Types

There are a number of antivirus applications. And obviously, to perform antivirus compatibility testing, you should know main principles of their work.

By the basic principle of work, we can divide all the antivirus products into 5 types:

• Scanners
• Monitors
• Auditors
• Immunizers
• Analyzers of Behavior

The modern antivirus software product can include a combination of the components of these types and also some additional security elements: firewall, mail protection, etc.

Scanners are present almost in each antivirus. Such antiviruses perform the check of files and sectors in memory or on the hard drive by the user or system request. The check consists in comparing the file or segment contents with the certain masks – specific parts of the malware code, which are stored in the virus signatures. If the virus is still unregistered, or doesn’t have the constant mask (polymorphism), or its mask merges with the common application code, than other threat detection techniques are used. Algorithmization method is applied to track all the variants of code that can generate a virus. Heuristic analysis method includes the analysis of the object code command sequence and statistics gathering; using these data the antivirus decides if an object is infected or no. Using such analyzer, one should remember that it can produce false infection reports.

Resident monitors, work like scanners, but unlike them, permanently reside in the RAM and automatically check all files to be opened or closed – to block the start of the infected file or its modification, if it was infected during its usage. We can mention file, mail and specific monitors for applications. When working, the file monitors is registering as the autorun system service that starts at system start for any user, integrates to the file system driver and check the objects that interact with these files. (As we know, opening or closing of the file requires file system call and thus, file system driver call). Other monitors integrate into the mail program or other applications; check all the objects they use, even if they are just stored in RAM. These monitors reside in the computer memory only when the corresponding application is running.

Auditors of changes proceed not from the image of virus, but from the image of the noninfected file or sector. Auditor gathers and stores the information about all files and sectors in the system. Usually these data include CRC sums, file size, last modification date, etc. Then auditor, by request, compares the actual file state with the one registered before, and thus detects all suspicious changes – like those that malware generates. Modern auditors can call the file via file system driver – thus avoiding the “masking” of the infected file with its clean prototype.

Immunizers are almost out of use nowadays. They add some specific code to the file content to block file infection or at least detect infection if it is already present. To block, the specific signature is added to the file so that a virus thinks that it is already infected and does not touch its “congener”. To detect the infection, antivirus adds a specific code at the end of the file, and this code automatically checks the content for the suspicious changes at file opening, and if something is wrong, notifies user. The viruses, which mask and imitate clear file prototype, meanwhile, infect all the files they can reach (without checking them) and so they can easily overcome this protection. That is why immunizers nowadays are used very rarely and in some special occasions.

Analyzers of behavior intercept and analyze the events, produced by the application, and if an action is considered as a suspicious one, analyzer blocks it or asks user for explicit permission. The suspicious actions can be classified by user, or they can be described inside the AI of the antivirus. In the former case, the administrator must know all the nuances and aspects of the system and internal processes. In the latter, AI makes a decision, basing on its internal logic and knowledge. At the moment, the AI is not so perfect, so such blockers are not used for the whole file system, only for the specific applications with specific file types – in this case, the antivirus has enough information about which actions are normal, and which are not. This is the way how the antiviruses detects marco-viruses in the MS Office, Autocad, mail and some other applications.

Well-known specialties and nuances of the antivirus software

Linux antiviruses. There are versions of antiviruses for Linux for quite a long time already. They are produced by the same companies, which produce Windows-versions – Avira, AVG, Panda, etc. But nevertheless, Linux antiviruses are not so widespread at the moment, so compatibility testing with them is not of high priority. Anyway, tester should take into consideration that the antivirus for any system can provoke bugs.

Virus signature database and update. The most of the antivirus products includes virus signature database, which they use for checking files in the system. To accelerate analysis, this database is entirely loaded to the RAM during scanning, and moreover, it is actively used. If antivirus resides in the system (like monitor), this database constantly resides in RAM (or in the swap file), and can be quickly invoked at each system check request.

To work properly, the program kernel and signature database need permanent update. Usually, it is performed automatically and user is just notified about the results. To perform update, the program establishes Internet-connection, sends requests for the new files and receives necessary data. Frequently, update supposes installation with some services restart and system settings change. That is why the installation of the other applications can cause problems: the system can forbid simultaneous installation of two programs, or the action of two installers can prevent each other.

Monitors of Web and mail traffic. These applications check all the incoming traffic from the Internet in real-time – in RAM, before it is written to the hard drive. Web-monitor also can connect to the server and get information about the potentially dahngerous sites, and then block access to them just after the request is sent from the browser. The common implementation of these mechanisms is services that start in the name of the system and integrate directly to the browser of mail application.

Specific scanning modes. Some versions of the Dr.Web, Eset and probably some other antiviruses include the option of “enhanced scanning” that can use various approaches: it can block user desktop, block any settings and antivirus file changes, or block all network traffic. Surely, the antivirus product will show a warning about what is going to happen, but but it won’t prevent the bad consequences. Users complain frequently about these scanning modes as they limit dramatically the user possibilities or consume too much resources, that is why they are not widespread yet, and anyway these problems are antivirus problems.

Cure and remove While curing, antivirus removes harmful pieces of code or restore the previous clean file version – so we can preserve the file. But the most of the modern malware make irretrievable changes to the files, so it’s impossible to use it then. In this case, this file is blocked or deleted. One should remember that the antivirus doesn’t care – it can block any file, even the one that is essencial for the system or application functioning..

Self-protection is a countermeasure against malware trying to prevent the work of the antivirus program. Almost all interferences to the antivirus configuration or its files are blocked – to perform some changes user must enter the password and captcha for authorization. One should remember that the self-protection mechanism uses its proper driver, which can be really unfriendly to the rest of the applications.

Firewall is usually included to the products of the «security suite» or «internet security» type, which are the universal security products consisting of more than 5 components. Firewall controls the activity at the network ports and the way applications use them. It can block partially or completely user access to the network, it its security policy requires it. Also firewall gathers the network statistics and prevent network attacks.

TOP 10 Antiviruses

As the OPSWAT Inc. Research shows, the top 10 of the most used antiviruses in the world make up the 62,5% of the whole number of antivirus product installations, they are listed in the table below.

Antivirus	Market Share	Price
Microsoft Security Essentials (Antivirus)	10.66%	Free (for licensed Windows)
Avira AntiVir Personal – Free Antivirus	10.18%	Free
avast! Free Antivirus	8.66%	Free
AVG Antivirus Free	7.92%	Free
ESET NOD32 Antivirus	7.15%	Trial license (30 days)
Kaspersky Internet Security	4.31%	Trial license (30 days)
Norton AntiVirus	4.29%	Trial license (30 days)
AVG 10 [Antivirus]	3.68%	Trial license (30 days)
McAfee VirusScan	3.02%	Trial license (30 days)
ESET Smart Security	2.75%	Trial license (30 days)

The first lines are occupied by the free products, and so they are the best candidates for antivirus compatibility testing. But almost each of them has its paid versions with much wider functionality. Here tester can use trial versions – to cut testing cost.

The list below represents the antivirus products I advise to test your application with:

• Microsoft Security Essentials + Windows Firewall + Windows Defender
• Avira Premium Security Suite 10
• Avast! Interner Security 6
• AVG Internet Security 2011
• Eset Smart Security 4
• Kaspersky Internet Security 11

Surely, you should pay attention at other antiviruses depending on the various factors: customer priorities, tested application specifics, market specifics, nuances of the particular antivirus functionality.

Main aspects of the antivirus compatibility testing

Antivirus compatibility testing is important for almost any developed application. 2 main reasons are:

• Antiviruses are extremely widespread;
• Antiviruses can impact to the both system work in whole and work of the individual applications.

Compatibility Testing issues

Just like in the general case, you should think about compatibility testing at the stage of test plan composing. You should define what antiviruses will be used, and at what testing stages compatibility check will be performed. Various factors can be taken into consideration while choosing the antivirus set, but the first one is the customer priorities. The best time for compatibility testing is just after system testing is finished and the product is stabilized. The more you postpone compatibility testing, the closer the release deadline is – and the more expensive the discovered problems can be.

Testing can be performed either on the physical computer or on the virtual machine.

Read more about virtual testing environment.

To perform compatibility testing properly, it is recommended to reconstruct the real life conditions, with the real set of the devices and interfaces. It will help to discover more potential problems and evaluate the application behavior and performance. It’s always important for the antiviruses that use their own driver.

But obviously, sometimes tester doesn’t have enough resources for the testing on the physical machine (hardware, time to reconfigure the system, etc.), or this testing can have irretrievable subsequences, and so virtual environment can be used here. Using virtual machine snapshots tester makes testing process safer and quicker, as he arranges a specific configuration at each snapshot. As virtual platforms like Hyper-V, ESX, Xen Server are rather popular nowadays, such type of testing can actually be the real customer infrastructure testing.

Performing regression or any other kind of testing QA specialist can also use the system configuration with the installed antivirus to cover more potential problems. If there are several machines, e.g. with different OS, different antiviruses can be installed on them, and then, when configuration testing is performed, antivirus compatibility testing will be also partially performed. It would be good to add general for all antiviruses test cases – like “Database Update” or “Scanning Start” – to the test sets, so they will be performed together with all other actions.

If the version passed acceptance testing, and after that the customer has sent a bug report, it’s time to remember about possible antivirus influence. Especially, if compatibility testing was not performed. The first thing is to ask the customer about the antivirus program he uses and how it is configured.

And surely, tester should remember that he checks not only the work of the tested object but also the antivirus behavior. It must not be unnatural, different from the behavior on the “clean” system, i.e. system without tested object installed. If the antivirus works improperly when using tested application – this is the bug of the application, even if this is a problem inside the antivirus. This bug must be fixed or mentioned in the application specification.

Influence on the installation process

The problems with compatibility can appear before the tested product starts its work. Antivirus is interested in the installation of any program, especially the low-level one. If it discovers something suspicious in the installer actions or installed files, it can stop the further product installation by blocking its access to the memory and file system resources. As a result, installer will show an error and report that the process was interrupted, or simply crash. Antivirus, at the other hand, can report that the potentially dangerous application has been blocked or just keep silence and log it somewhere in its database (as users like to turn off notification system that ”distracts” them). And even if the program has been installed successfully, it doesn’t mean that it will start normally, as antivirus will keep monitoring it.

Writing to hard drive

The most obvious danger is that heuristic analyzers will produce false response and accuse some file of the tested product as the potential malware. Installer will be forbidden to copy this file to the hard drive. Even if the installation is finished successfully, this application won’t work properly, so the tester should write the corresponding bug report.

Also, antivirus can produce false reaction to the file copying to the protected system folders. And it will surely suspect something malicious when we try to replace some system component, for example, dynamic library – then we can get the access denial, and the product won’t work properly.

Writing to the registry

As a rule, antiviruses rarely monitor registry changes, leaving this task for the system security products. But if antivirus includes some behavior analyzer, it can qualify as malicious actions attempts to change registry keys, which can be accessed only by system services (as antivirus considers).Also there are some utilities to monitor registry integrity, and if the registry looks like “damaged” at some installation stage, these utilities will try to repair it, in spite of the fact that it will cause incorrect application installation. Such situations are really rare ones, but still it is worth remembering it if the tested product is not installed properly.

Application initialization and start

After the tested product has been installed to the system, it’s time to perform the first start – the moment when all the product component work is initialized and tested. This is the moment when the access and network problems can appear for the first time (details will be provided below), and also some problems concerning the operations that are performed at the first application start. Concerning these operations, there can be such cases:

• Information concerning the system and its components is gathered – if antivirus security policy is set to strict, the tested application can be forbidden the access to such data and even suspected in performing spy actions. So, there can be the antivirus report concerning suspicious object and the application error message;
• Some system device initialization is performed – this process usually does not need special attention, but antivirus can consider suspicious these device start;
• Integration to the system processes is performed – for example, if hookers are injected to all processes (including the antivirus ones), or the tested application subscribes to system events, it can seem suspicious for the antivirus.

Concerning the network, the tested application can use licensing via network or need some additional components, which it will try to download from the Internet, – so, it will have to open the port and establish network connection. For this case, tester should check application reaction if firewall or antivirus traffic monitor has denied Internet access for it. It’s unlikely to provoke unstable application functioning, but still can have unpleasant results: e.g. failed licensing attempt will annoy the user, so he should be at least reported that this situation can be subsequence of the antivirus settings.

Another question is installing antivirus while the tested product is working. In my opinion, this situation doesn’t worth testing, as first of all, without antivirus internals it would be really hard to detect what has caused the installation crash. So this question can be postponed till the beta-testing phase or even support phase.

Influence on the tested application work

Antiviruses are the low-level program systems, so they can influence on the work of the individual components and the system as a whole. Naturally, antivirus vendors try to reduce the probability of the undesirable effects and improve their products so that the functionality of the other applications is not hurt. As a result we get flexible antivirus programs that can be configured for the individual user. But in some situations, undesirable effects can appear – as this software is aimed to protect user’s property, sometimes at any cost.

Writing to and reading from the hard drive

Problems here can appear by several reasons. As file monitors and auditors control almost all hard drive operations, the tested product has to take this into account. The question arises, the question of coordination of file access – during normal work and specific activities (like scanning).

In normal mode, the antivirus intervenes in the tested product work only if qualifies its actions as potentially dangerous. Some antiviruses, e.g. MS Security Essentials, can react when a user or a program has changed something in the disk boot sector, edited hosts file or some other file that is common target of malware attacks. Also antivirus will react if the application actively edits various file content (this is the reason why some immunizers are still considered as malware). The reaction will be denied access to the requested object, and if the antivirus is “cruel”, it can block any further actions of the suspicious process till the moment when user directly allows to continue its functioning. Surely, the process can give up and crash.

Antiviruses rarely crash if they get file access denied while scanning – they write this fact to the log and continue scanning. But the tested application can react improperly if antivirus has blocked some of its files of system files while scanning – so when testing compatibility in this case, it’s good to “set” the antivirus on the system or tested application files when the tested product is performing some work. It helps to make sure that antivirus scanner works properly and doesn’t consider the tested product files as the suspicious ones, and also that the tested application works properly at such file check.

We should mention also the use of file system driver. The major part of the antiviruses work directly with this driver to prevent third-party influence on its file operations. And they do it all continuously by integrating in it with hookers or auxiliary driver. If the tested product does the same, it will surely lead to some bad subsequences: tested application crash, antivirus crash, file system driver crash with BSOD. That is why, if the tested program performs interacts with the file system in some unusual way, than it should be tested against compatibility with the several antiviruses (nowadays, almost all of them include file monitor or auditor).

Registry and system settings changes

Usually, antivirus does not like intervention in its own work, and also changes provided for the system boot, autorun, system security parameters, security policy – in these cases antivirus can ask user for additional approve for such actions, like in UAС. There should not be big problems, but if the tested application actively change the system configuration, then it is recommended to check its work under antivirus control.

Also one should remember that the antivirus itself can be the injured party: if the application provided some changes for the system parameters, which can cause the improper antivirus functioning or block it (e.g. changes in the system firewall can cause the situation when antivirus is unable to update). That is why tester should not only check the correct functioning of the tested application with antivirus, but also make sure that antivirus works as usual.

Though these problems are extremely rare, it is worth checking how antivirus reacts on the operations, performed by tested application on the system. Otherwise, there can be a situation when the tested application appears in the list of the potentially dangerous applications that is clearly not very good for the product reputation.

Network interaction

As we’ve said above, antivirus can include such components as email and web traffic monitors, and also network firewalls. As a rule, they are the services started by Local System name and controlling the work of all system network connections. If the tested product is a network or client-server application, or simply uses network for some tasks, there can be such problems:

• Currently set firewall network policy prevents the tested applications from the proper network use (e.g. firewall blocks opened ports and connection to the server);
• The tested product cannot react properly on the situation when antivirus intercepts its web and email requests and also corresponding responses;
• The tested product makes some changes in network functioning (e.g. hooks network requests, imitates NAT work), and antivirus cannot adapt to it and results working incorrectly.

The unpleasant detail here is that if the tested application and antivirus cannot share the network, this fact can stay undiscovered at first, and when the user finally finds out that something is wrong, it can be really hard to detect the reason.

Influence on the tested drivers

If the tested driver is integrating into the system, then tester should remember that its interaction with antivirus won’t be finished with usual driver file check. Antivirus will monitor it – just like the whole system – it’s very important here that the driver works properly under such conditions, and that it doesn’t prevent antivirus from normal work.

Requests to the system drivers and services

Drivers are usually designed to provide the correct functioning of some system device. Also drivers can change functioning of some already working device – both physical (e.g. keyboard) and program ones (e.g. codec). And the question here is if antivirus allows the tested driver to do everything it wants to do. Thus, keyboard hook or key press monitoring can be considered as the suspicious actions by some antiviruses, and as a result, these actions will be blocked and driver will crash because of this intervention. Even if it does not crash, it won’t be able to work properly, so it is bug.

Another reason of crash can be the situation when antivirus has blocked the access to the necessary resources for the driver: by some security reasons or because it uses these resources itself and cannot unlock them in time. It can be a file, registry key or some device. Driver is not an application so it cannot always wait till the resource is released. It’s critical to find such bugs before the final stages of the release.

Requests to the tested driver

It’s important to check how the tested driver will affect the antivirus work. The most critical situation is when we can be sure that antivirus will use the driver in its work – here tester should check all main antivirus functions, as it can not only crash but also provoke BSOD. There can be many reasons for that: incorrect instructions or input data for the driver, access denial, system conflicts (as it can also use this driver), etc.

The driver can be also “local”, i.e. its work applies only to the specific part of OS and ideally should not affect antivirus work. But still one should remember that this affect can be caused via an intermediary – one of the system components – and it can be a problem. So anyway, the tester should think about what components the driver uses, and if they are the same as the ones that antivirus uses.

It is widely known that drivers frequently have problems with antivirus software, even device drivers of the top vendors. That is why it is insistently recommended: whatever driver you test, check its compatibility with 2-3 popular antiviruses.

Influence on the performance testing

Antivirus software per se can affect the system performance and make testing not very pleasant. But tester still needs to make sure that when the tested product and antivirus work simultaneously, resource consumption of each of them remains normal.

System resource usage (CPU and RAM)

Problems can appear in 2 cases: when antivirus tries to perform some action, but does not manage to do it normally because of the tested product; and when antivirus intervenes in the tested product functioning creating some delays. There can be several situations:

• When antivirus cannot get access to the necessary resource because of the tested application that constantly locks this resource (e.g. network port), or intercepts the requests and responses about this resource usage. As a result, antivirus tries to perform its actions without success using significant resources;
• When the tested product frequently generates files with a rather complicated code – antivirus monitor will always check new files, and if it uses some heuristic analyzer it can hang the system;
• When antivirus uses change auditor or behavior analyzer, and thus checks some actions performed by the tested product. As a result, actions can need more time to be performed, that can also cause imbalance and conflicts among the tested application components.

Work with hard drive

It is not the parameter that causes the most problems, but still some difficulties can appear. There can be such situations:

• Additional logging is applied, when antivirus decides to constantly log all the actions of the tested applications and write the corresponding data to the hard drive;
• When change auditor works and the tested application actively uses database constantly changing and saving it. In this case, antivirus can incessantly read the saved database files from the hard drive, and usually they are not small;
• When the tested application directly influences the work of the file system driver (e.g. hooks it, and antivirus does the same). As a result, tested product and antivirus taken apart can interact with the file system driver correctly, but their simultaneous work can cause significant performance problems.

Network interaction speed

The most attention here should be paid to the antiviruses that include firewall of network traffic monitor. The effects can differ for different traffic types, so it is necessary to check not only the speed of downloading of some file from some ftp source, but also the speed of downloading of Internet-page, media-content, streaming video, p2p interaction, etc. If the tested application actively uses the network, this testing must be performed, as there can be such situations:

• Antivirus checks and analyses all network operations of the tested product (socket opening, incoming/outgoing requests), that extends the time required for these operations, and if the tested application also uses the synchronous data transfer, it can significantly slow down the network functioning;
• Antivirus checks all downloaded content. If the tested application work depends on the downloaded data (like browser or other client-server applications), these checks can significantly delay its work up to fail to satisfy specification requirements.

The work with the modern antiviruses rarely causes some important performance problems in the individual tested applications. So, it is not worth spending a lot of time trying to cover all theoretically possible test cases, but still tester should make sure that the system and individual applications won’t hang because of tested product and antivirus interaction. To check it, it is usually enough to perform simultaneously some typical tested product use case and main operations of the antivirus program: turn on/off, update, scan, “cure” applying different antivirus settings.

Conclusion

Antivirus compatibility testing sometimes doesn’t seem to be worth time resource spending, especially if project is severely limited in timeline and budget. But if the developed software is a low-level specific application, it is recommended to perform this type of testing to avoid significant financial loss at the release stage. Even if all detected bugs are not discovered and fixed, information about them will help to form better product documentation and plan how to avoid these problems in future.

Obviously, it is impossible to check compatibility with all antiviruses and all their versions. It is also impossible to check program functioning with all possible antivirus configurations. The sufficient testing scope depends on the particular case and various factors: client priorities, market characteristics, program behavior, existent experience. In general case, tester should choose 4-5 antivirus systems, set them to the higher security mode (all features are on) and perform compatibility testing – purposefully (if there are sufficient resources) or along with the regression testing.

Moreover, having received customer bug report, tester can compare the symptoms in it with those described in this article and then ask the customer if he has antivirus installed, what it is and how it is configured. It can be the key to the bug detection.