Testing Strategies for the Software Virtualization Systems

In this article, we will examine the specific of testing of the software virtualization systems. We will touch upon the following questions:

• test environment configuration;
• testing types;
• nuances of each of testing types for the virtualization systems.

So, we will examine those important things that should be taken into account when thinking over the testing strategy of the software virtualization system.

This material can be useful for those people who test  systems of described type and also for the managers of these projects.

Software virtualization systems

Virtualization in the broad sense deservedly becomes more and more popular and so do the software virtualization, particularly. Due to virtualization, the end-user can work with habitual things and not think that actually he deals with virtual objects. It lets solve the wide range of problems connected with the usage of real resources.

We will examine the software virtualization systems that deliver virtual applications and desktops to the end-user. It means that we will examine systems that let users use different applications without installing them on the user computer (e.g., working with MS Word without installing the whole Microsoft Office on the computer). Or this can be the case of working with different operating systems without having several real computers with required operating systems installed on them.

In other words, with the help of such systems, the application starts on the client computer and looks as if it was installed locally. The user can freely work with data without making changes on his/her local computer.

The examples of such virtualization systems are Microsoft App-V, VMware ThinApp, Citrix XenApp, and many others.

Test environment

Network

The software virtualization systems are first of all the client-server applications. That is why the interaction with the network is inevitable.

Very often, for a full-fledged testing of the virtualization systems, we need to create a separate test sub-network with a domain controller and several computers added to this domain.

It is enough to have one separate small-capacity computer as a domain controller. We can install several systems on it and in such way test our system for the compatibility with different operating systems on the domain controller (usually, these are Windows 2003/2008/2008R2 server operating systems).

You should install the following roles on the domain controller:

• Active Directory Domain Services;
• DHCP;
• DNS.

You need to configure the gateway to get access to the Internet from the local network. So, you need a switch and several network cables for the network configuration.

Virtual Computers

We need many computers because we need to check the work with all supported clients and servers. Besides, we may need to test the work with different processors, sound cards, and other hardware. Virtual computers are used for these purposes. Utilities that create virtual computers are VMWare Workstation, Microsoft Virtual PC, Parallels Workstation, and Sun VirtualBox.

Though testing on virtual machines is easier, we should not forget about testing on real computers. Results that were obtained in the virtual environment can differ from testing results on real computers.

VDI

If the support of Virtual Desktop Infrastructure is implemented in the tested project (i.e., when the user can get not only the remote desktop, but also the whole virtual computer in his/her disposal), additional test environment for the hypervisor will be required.

It’s better to assign a separate physical computer to install the hypervisor. The most commonly used hypervisors are:

• ESX/ESXi from VMWare;
• Microsoft Windows Server 2008 R2 with installed Hyper-V role;
• Microsoft  Hyper-V Server 2008 R2 (without GUI).

It is recommended to install hypervisors on different hard drives.

ESX/ESXi is very exacting to the hardware (processor, netcard) that is why we should check the compatibility beforehand.

Wide Area Network Emulator

In the process of testing in the local network, we may need the emulation of the global network work and also the emulation of the slow connection (packet delay). We will need the global network emulator for this, e.g., WANem (can be downloaded from http://wanem.sourceforge.net/).

Virtual Printers

If the tested system supports the redirection of printers to the session, we will need not only real printers but also virtual ones to test the system. Virtual printer is a program that emulates the printer work but does not require the real device (as a rule, print action is performed to file).

Examples of the virtual printers are:

• Adobe Acrobat;
• PDFCreator;
• FinePrint,etc.

Additional Devices

Depending on the specific of the tested system and on the redirection of what devices it supports, we may also need the following:

• printer;
• scanner;
• headset with a microphone;
• web-camera;
• usb flash drives;
• card reader;
• usb hub.

Configuration testing

This type of testing is inevitable for the virtualization systems and takes into account a number of parameters. The main parameters of the configuration testing are the operating system and its capacity (32- or 64-bit).

The system should be tested on all supported operating systems, both for the server and the client sides.

The server side can consist of several components, which can or must be installed on different computers. It additionally increases the number of all possible combinations. That is why, the prioritization is required.

It is good if the system is in use, already has its users, and we know what operating systems they use. In this case, we need to test exactly on these operating systems first.

If we do not know anything about end users, we can view the statistics of the operating systems usage in the world or in a definite region (if the system is intended for distribution in some definite region). Such statistics can be easily found in the Internet (e.g., http://www.w3schools.com/browsers/browsers_os.asp).

It is also important to differ and run the testing with different editions of the operating system (Standard, Enterprise, Home, Premium, etc.) and with different service packs, updates, etc.

So, we need to make a table – configuration matrix and place there the configuration parameters in order of the significance. In the simplest case, it looks approximately as follows:

In such table, it is convenient to mark already tested configurations and those in progress with different colors. And also we can mark configurations passed on previous versions.

If the tested system is intended not only for personal computers and notebooks as a client but also for thin clients and PDAs, these also should be included to the configuration table.

If the tested system provides the web-interface to the clients, we should add the all possible browsers to the configuration parameters:

• IE;
• Firefox;
• Chrome;
• Opera;
• Safari, etc.

It is hardly possible to run testing on all configurations. Usually, configurations with the highest priority are covered.

Installation testing

As a rule, the server side of the virtualization systems consists of several components to make it possible to install them on several computers and distribute the load in such way. Or such feature can be used to install or delete some definite functionality.

Hence, there is a set of cases that should be tested:

• Default installation;
• Cancellation of installation;
• Installation of the components separately on different computers;
• Installation of all components on a single computer;
• Additional installation of the components;
• Deletion of one or several components;
• Reinstallation of the application;
• Deletion of all components;
• Installation of the combination of components (to compile the optimal set of cases, we can use the method of all pairs or just select the combinations that are the most suitable for the real situation);
• Installation for all users/current user.

The client side depends on the project. Installation with or without the set of components is also possible. In the general case, the test set will be the same as for the server.

We should not forget to test the installation not only on the virtual computer but also on the real one.

If the system is in use, it is most likely that the new version of the program will be proposed to the users in two ways: as a separate installation and as the service pack. That is why it is necessary to check the update of the previous version to the new one.

Functional testing

It is the main type of testing where the product functionality is checked. Also the testing of interaction with the third-party programs, which the end-user will probably use, is performed.

1. First, it is necessary to check the use cases, i.e. what the user will probably do with the system.

Under the specific character of the program, the last will have at least 2 types of users: administrator, who will deploy and configure the system, and the end-user (the really end one), who will work only with the client side and start from there the required applications and desktops created by the administrator on the server side for him.

Thus, first we need the set of use cases for the system testing.

2. Then we can deepen into the functional testing.

To do this, we need to divide the project into logical parts (features) and write a detailed test plan for each part. We can start testing using the prepared test plan after the smoke testing (see below) was passed successfully and this feature does not have blocking defects.

3. It is also important to check how the tested system works with other applications, which the end-user will use.

For example, if the system works with the redirected audio and/or video only and does not use the standard ones, it will be interesting to see how it will work for programs-communicators, such as Skype, ICQ, and Microsoft Office Communicator.

Performance testing

The performance testing is to define if the developed system is effective and does not overuse resources.

For example, the workload of certain servers is checked when receiving remote applications from these servers by several clients simultaneously.

What to check?

As a rule, we check the memory and CPU usage, and the traffic.

1. To check the CPU and memory usage, we can use standard Windows tools, such as Task Manager and Performance Monitor (run perfmon command from the command line).
2. To monitor traffic, we can use BWMeter, for example (can be downloaded from http://bwmeter.en.softonic.com/).

If the load balancer is implemented in the system (something similar to Load Balancer in Windows Server), it is necessary to load certain servers for the testing purposes artificially (e.g., to increase the memory usage and CPU usage).

To do this, we can use the third-party utilities, such as Rmma for the CPU usage (can be downloaded from http://cpu.rightmark.org/products/rmma.shtml) and AleGr MEMTEST for the increase of the memory usage (can be downloaded from http://www.alegr-memtest.com-about.com/).

High and heightened loads

It is important to check the system performance in conditions that are the most approximate to the reality (at high loads). To check this performance, the load testing is performed. For example, we can check the response time of the server when connecting many clients to it.

If the number of clients, with which we perform testing, exceeds the declared number, this will be testing with heightened loads, i.e., stress testing. In this case, we are interested in how correctly the system reacts to the heightened load.

Regression testing

Testing connected with changes is probably the mostly used type for the virtualization systems. Such systems are not written in a moment, their functionality is gradually extended. That is why, when receiving the version for testing, first it is necessary to check the basic functionality and check if something was broken when making changes in the program.

To check the main functionality quickly, the corresponding test plans for the smoke regression testing are prepared. They are prepared by setting the priorities in the test plan for full testing and by selecting of cases with the highest priority from the prioritized test plan.

It is recommended to run the full testing on 1-2 configurations with the highest priority and the smoke testing on the rest of configurations.

Automated testing

Examining software virtualization systems is a huge set of tasks for cloud automation testing: from the simplest cases that automate the installation to the complicated universal automation systems that start the hypervisor, start the virtual computers that clients need, and provide the user with the session with required parameters.

Development of such “clever” systems is very complicated that is why usually it is enough to have auto-tests for the regression testing. In the simplest case, it is a set of cases for the superficial check of the work of the tested application and the performing of the simplest and the most common actions (acceptance testing). Or, what is better, it can be a little bit extended set of cases for the smoke testing. In this case, when the next version is prepared for testing, first of all, we run automated cases that perform the check of the general operability of the program (acceptance/smoke). And if the auto-testing did not reveal blocking defects, we can then deepen into the testing of one or another feature.

Most auto-tests for the virtualization systems will have the client-server architecture.  First, we need to prepare the client (start it and connect to the server). Then we need to get the session and start the server part of the auto-test in the session. The server part will check what it must check and will send a reply to the client. The client will get the message, write data to report, and close the session.

If we have a number of auto-tests that cover a certain test plan, then we also should provide some solution to run the tests in the required order. This solution will also have the client-server architecture for most tests.

Thus, in the general case, the client and server computers will have parts of the client-server auto-test management system. This system will start the client and server parts of the tests correspondingly and will receive and send messages between client and server.

Conclusion

Among the types of testing, we did not mention the GUI and usability testing. But it does not mean that these types of testing are not performed for the virtualization systems. But usually, the main emphasis is made on the functionality.

As a rule, it is impossible to consider all possible risks while defining the testing strategy of the virtualization systems beforehand. Especially it is because the requirements to the product often change in the process of development, as a rule. And the technology does not stand still.

Strategy is a dynamic structure. It should react to the changes adequately. Only in this case, we can decrease the risks in testing and receive the product of high quality.