Introduction to Mobile Application Testing: Overview of Testing Types, Checklists, and Virtualization Services

Ensuring the smooth and secure work of a mobile application requires quality assurance (QA) engineers to expertly pick which tests to perform for specific types of apps. Only then can they verify an app’s quality and keep up with the fast-paced cycle of mobile app development as well as meet end users’ expectations.

In this article, we discuss how to test a mobile application, which types of testing and checks to apply, and which device virtualization services to rely on. It will be useful for development companies that want to improve their mobile testing and enterprise mobile application management routines.

Why invest time in application testing?

Testing is an essential stage of any software development process that helps to ensure an application’s quality, usability, and security. However, testing sometimes gets overlooked in favor of allocating more time for app development or a faster release. Limiting or skipping testing activities is undeniably a bad development practice, but some organizations use it to save time or because they’re sure that their code doesn’t have major issues.

In fact, paying close attention to testing has many benefits and is definitely worth the time. Here is why mobile testing is important:

• Ensure a flawless user experience. Users expect your application to be easy to use, perform well, and work smoothly on their devices. If it doesn’t meet their expectations, users will use the app as seldom as possible or simply uninstall it.
• Protect user data from leaks and hacks. Some applications need to collect users’ personal, financial, medical, or other types of sensitive data to work properly. The organization that owns an application has to make sure that all sensitive data is well protected. A leak or theft of sensitive information can result in huge reputational and financial losses.
• Pass mobile app store reviews. The Apple App Store, Google Play Store, and other mobile application stores review submitted applications before publishing them. During the review, they check the application’s code, privacy policy, requested permissions, data management processes, and more. Testing your application before it rolls out helps you fix major issues and pass the review faster.

That being said, the testing stage may indeed take a long time, which is not always affordable for mobile applications with a short development cycle. In the next section, we discuss the types of mobile testing that must be performed to ensure your product has the right level of quality, performance, and security.

5 key types of mobile application testing

Mobile application quality assurance is a complex process composed of multiple reviews and checks. You can speed up testing by reviewing the scope of tests and running only those that are necessary. There are five key types of mobile application testing:

Besides these testing activities, each application may require additional checks depending on its goal and characteristics. For example, banking solutions require many more mobile banking app security testing activities than gaming apps, yet the latter require additional performance and usability testing.

In this article, we focus on five main checks that can be applied to a wide range of mobile applications. Let’s start with the definition of functional testing and key checks.

Functional testing

Functional testing of mobile applications is the first and probably the most essential type of testing you should perform. Its goal is to ensure that the software successfully does what it’s supposed to do. Here’s a more detailed definition by the International Software Testing Qualifications Board (ISTQB):

To prepare functional test cases, you first need to gather the requirements that are usually defined in the app’s documentation. If there isn’t enough information, you can also define the needs of your target audience and turn them into requirements. Also, analyze the functions of competitors’ products and applications similar to yours and add relevant functionality to your list of requirements.

The requirements elicitation meaning in software engineering is that it allows you to compile a list of key functionalities your application should have. For example, if you’re testing an app for social media, its key functionalities will probably be user registration, creating and editing user profiles, messaging, and so on. This is the foundation of your app that you should test first.

Here’s a list of core mobile functional testing and QA activities:

Checked functionality	Expected result
Installing, updating, and deleting the application via an app store or using the APK or IPA file	The application launches after installation. During an update, it doesn’t download files that are already downloaded to the user’s device and doesn’t delete user information.
Launching, closing, and minimizing the application	None of these activities cause failures or glitches
Battery consumption	The application doesn’t consume more resources than defined in the requirements.
User interface (UI) elements performance	All application buttons, fields, checkboxes, forms, etc. work as intended.
Switching between pages and UI tabs	The application displays new pages and tabs correctly while keeping current pages in memory.
App notifications	Notifications are sent and displayed according to applied configurations.
Access to native device functionality	The application asks the user’s permission to access the device’s camera, microphone, location, contacts, and other device functionality.
Disk space consumption	The application doesn’t take more space than it needs and stores a reasonable number of required cache files.
Using different types of internet connections	The application can connect to 3G, 4G, 5G, Wi-Fi, a proxy, and a VPN, and it can work with no connection. If the app uses a mobile internet connection, it notifies the user about it. If the app has a poor connection or no internet connection, it warns the user about it.
Receiving a call, text, or notification	When the user receives a call, the application switches to the wait state. When the user receives a text or a notification, the application works as usual.
Support for payment systems	The application can perform needed financial operations via Visa, Mastercard, PayPal, or other payment systems.
Contents of the privacy policy or terms and conditions	If the application handles personal user data, it has a privacy policy or terms and conditions that reflect secure data management procedures. This document describes all required terms and security checks to protect user data.
Application versioning	The app’s version is updated any time there’s a change in the application.

Usability testing

This type of testing helps you ensure that your application is easy to use and understandable for your target audience as well as to find ways to improve the user interface (UI) and user experience (UX).

There are five criteria for assessing an application’s usability that we use in our mobile development projects:

As you can see, these are subjective criteria, which is why it can be tricky for QA specialists to test your application in a completely unbiased manner. Unbiased testing becomes challenging to organize when your application needs several testing iterations. A QA team can get used to unusual behavior and overlook a usability issue.

That’s why using alpha and beta testing with focus groups is particularly useful for checking an app’s usability. Gather a group of people (who can be the end users of your software) and prepare user scenarios that they need to perform. After the testing session, collect focus group feedback on the five usability criteria we discussed. This way, you can get an unbiased and insightful assessment of your application in general and its usability in particular.

Here’s the minimum mobile app usability testing checklist:

Checked functionality	Expected result
Switching between pages and UI tabs	The application has interface elements to conveniently switch between pages and tabs.
Application buttons	The buttons are big enough to be pressed with the thumb. Buttons that perform the actions look identical and are located in the same areas of the screen.
Text display	The text font is comfortable to read. If the application displays a lot of text, the user can adjust the font to a comfortable size. The user can scroll the text if needed.
Support for touchscreen features	The application responds correctly to common gestures like swipes, scrolls, double taps, and long taps.

Performance testing

Users expect your application to work as fast as possible, or at least not to be irritatingly slow. Testing your app’s performance allows you to make sure the app works fine on different devices and under different loads. Performance testing of mobile applications includes checking the productivity of devices, servers, and networks.

Let’s take a look at the key checks you need to conduct.

1. Device performance

To ensure the smooth work of your application, test it using several popular devices with various types of processors, amounts of RAM, battery capaсities, etc. If you don’t have enough real devices to test on, you can use virtualization services that we overview later in this article.

Here are the key tests to run on your devices:

Checked functionality	Expected result
Application launch	The application launches in a couple of seconds. If the application takes more than two seconds to launch, it displays some sort of preview to let the user know that it’s launching.
CPU and RAM load	The application consumes CPU and RAM resources according to the requirements. The load that the app puts on the device doesn’t disrupt the device’s work and doesn’t overheat it.
Battery consumption	The application doesn’t consume too much battery power according to the device’s native tools.
Multi-window mode	When there are several applications running on the device, your app uses the same resources as when working on its own. The app doesn’t cause failures in the device’s work.
Background mode	The application consumes fewer resources when working in the background compared to when working in its opened state.

2. Server performance

Server response times and data exchange can become a performance issue when a lot of users interact with your application. Check the following to ensure that communication with the server doesn’t slow down your application:

Checked functionality	Expected result
Data exchange with the server	The application processes data received from the server quickly and efficiently. Uploading data to the server doesn’t take too much time. If the application needs to transform data from the server before displaying it to the user, the transformation happens fast enough.
Request to server sent via the application’s API	The application sends an optimal number of requests and doesn’t overload the server. There are no duplications in the app’s requests.
Server resiliency	The server doesn’t fail too often or has a reserve server to fall back to. When the server is down, the application doesn’t lose any data.

3. Network productivity

If your application requires an internet connection, you need to test how it works when connected to various types of networks. Here’s how you can do it:

Checked functionality	Expected result
Network speed	The application works fast enough with networks it can connect to (3G, 4G, 5G, Wi-Fi). The application uses all network resources it can access. When the application loses a Wi-Fi connection, it can connect to a mobile network and keep working.
Packet loss	If the application loses a data packet completely, it sends an additional request or informs the user about the loss.

Compatibility testing

Mobile device manufacturers produce many smartphones with unique operating system versions, hardware, screen sizes, resolutions, etc. Testing the compatibility of your application with all devices is impossible. That’s why you need to understand what pool of devices your target audience uses and focus your testing efforts on them.

There are two ways of defining the pool of devices you need to prioritize in compatibility testing:

1. Research the most popular smartphone models worldwide or in your target region.
2. Gather statistics on mobile devices your audience uses with Google Analytics, Mixpanel, Adobe Analytics, FoxMetrics, Fair Analytics, or similar services.

Here are the key compatibility parameters you need to test:

Parameter	Expected result
Hardware compatibility	The application recognizes and supports hardware of popular device models of particular brands.
OS support	The application can work with the latest versions of the target operating systems.
Screen compatibility	The application is displayed normally on device screens of different sizes and resolutions.

Even with a limited number of device models, these three parameters and their combinations take a lot of time to test. To speed up this process, you can use pairwise testing. This test design technique combines tested parameters into pairs and tests all possible combinations for these pairs. With pairwise testing, you’ll be able to quickly detect lots of compatibility issues. But there’s always a risk that minor issues will slip through your tests, so be attentive when applying this technique.

Security testing

An application’s security is a specific area of testing performed by security specialists. It helps ensure that users’ data collected by the application is secure and that malicious actors can’t abuse the app and get a hold of that data. 

It’s hard to define a universal list of security checks since they heavily depend on the nature of your application, its architecture, and the types of data it collects. The ISTQB Syllabus for Mobile Application Testing Foundation Level [PDF] suggests considering at least these tests: 

• Inputs for code injection and overflow
• Encryption of transferred data
• Encryption of locally stored data
• Deletion of temporary data after use or after an abnormal app termination
• Clearing of text in password fields

Of course, this isn’t a definitive list of must-have security tests. When planning your own tests, make sure they cover these widespread mobile security risks:

Now it’s time to take a closer look at ways you can successfully ensure the smooth performance, high efficiency, and reliable security of your mobile application even when you don’t have enough physical devices to test it on. In the next section, let’s take a look at the key mobile virtualization services.

Mobile virtualization services: a brief overview

Using virtual devices brings many testing benefits including speed, scalability, and access to any type of testing environment. We examined the types, benefits, and drawbacks of virtualization in one of our previous articles, where we compared it with cloud computing and discussed key differences between the two technologies. 

Despite offering many advantages, a virtual device can’t fully emulate the behavior of a real smartphone or tablet. That’s why you should run your test cases on at least one real device before switching to a virtual one. Testing on a real device helps you make sure that your app works as intended in real life, and virtual testing allows you to check more mobile hardware and software combinations.

You can add virtual devices to your testing environment in two ways:

1. Create the required virtual machines yourself using official Android, iOS, and other manuals. Go with this option when you don’t need a lot of additional devices.
2. Use a mobile virtualization service.

At Apriorit, we have experience working with different cloud mobile virtualization services in our mobile development projects. Here’s a brief overview of the strengths and weaknesses of the services we use the most:

AWS Device Farm is a paid cloud service that allows you to remotely test your app on physical devices that your users work with. This service helps you see how a user will interact with your app on devices with different processors, locations, firmware, and memory sizes. However, almost all devices provided by AWS Device Farm are Android-based. At the time of writing, this service included only one Apple device model — iPhone 13. 

Corellium is a paid cloud platform that provides access to a large pool of virtual iOS and Android devices, including devices with root privileges. Virtual machines from this service are configured as close as possible to real devices, which allows testing to be as close as possible to the real user environment.

Genymotion is an easy-to-use multifunctional emulator that has both free and paid versions. It works with powerful OpenGL graphics cards, has Android Studio plugins, and is available on several Windows, Linux, and macOS operating systems. Genymotion also allows you to use the webcam for an emulated device as a camera and execute ADB commands using the ADB tunnel. The key drawback of this service is that it only emulates Android devices.

BlueStacks is a popular cloud platform that emulates Android devices. It has free and paid versions, and it works well with both Windows and macOS. Although it can emulate any Android application, BlueStacks is designed specifically to improve the gaming experience of Android video games on Windows.

BrowserStack is a paid cloud-based testing platform that provides access to over 3,000 real on-demand mobile devices and browsers. This service helps to test the compatibility of websites and mobile applications with browsers, operating systems, and real devices. However, BrowserStack doesn’t provide you with full control over device settings.

In case you’re also working on drivers, check out our article on testing a Windows driver.

Conclusion

In a fast-paced mobile app development cycle, testing teams have limited time to ensure an application’s quality and even to define test metrics in software testing. Experienced QA engineers can use this time in the best possible way because they know which tests they should run for different mobile app testing types, which tools and services to apply, and what software performance testing best practices to use.

In this article, we provided you with a brief overview of mobile testing types and must-have checks as well as reliable device virtualization services you can use. Based on this knowledge, you can roughly estimate how much time and effort you’ll need to test your app and find ways for improving the process.

Want to make sure your software is tested quickly and thoroughly? Reach out to our expert QA team to leverage their knowledge!