How to Use Cloud-Based Software Testing Tools: Apriorit Experience with BrowserStack and VirusTotal

Switching from traditional to cloud-based testing can help your QA team significantly speed up the testing process, test on a broader range of devices and platforms, and better scale the testing environment according to a project’s needs. The trick is to choose the tools that best fit your project.

There are too many types of software testing that you can perform in the cloud to cover them all within one article. In this article, we focus on cross-browser and antivirus compatibility testing. We overview the two best automation tools for testing Apriorit QA specialists typically use — BrowserStack and VirusTotal — share our experience working with them, and analyze their benefits and pitfalls.

This article will be helpful for QA professionals who are interested in moving cross-browser and antivirus compatibility testing to the cloud.

Why move testing to the cloud?

As an outsourcing development company that simultaneously works on many projects, we often face a lack of physical test devices. That’s why we began researching cloud-based testing services.

Cloud-based testing helps you assess the performance, security, and reliability of your software using third-party cloud computing resources instead of your own. Compared to traditional testing, a cloud-based approach provides organizations with lots of benefits including cost-efficiency, accelerated testing processes, and the ability to scale the testing environment.

In theory, it’s possible to move any type of testing to the cloud. Since we’re focusing on cross-browser and antivirus software compatibility testing in this article, let’s look closer at these activities first.

Cross-browser testing is a type of non-functional testing that analyzes how your website or web application works in different browsers and on different operating systems (OSs). The key challenge is to obtain the required test devices that run certain versions of browsers and OSs. The bigger the project, the more browser versions and OSs you need to check, and the harder it becomes to get real-life devices for testing your application.

Antivirus compatibility testing is another type of non-functional testing that involves checking your URLs and apps with popular antivirus programs to determine whether those programs:

• Block the application from installing or loading
• Influence the app’s performance
• Flag your URLs and executable files as malicious 

After trying different cross-browser testing tools, we’ve settled on BrowserStack. For checking antivirus software compatibility, our specialists have chosen an easy-to-use online service called VirusTotal.

Cross-browser testing with BrowserStack

BrowserStack is a cross-browser testing tool that provides QA teams with access to more than 2,000 browsers and devices to test their web and mobile applications. The tool offers five services:

Here’s how to use BrowserStack for app testing: First, register on the official BrowserStack website and choose the service you need. After registering, simply go to the Live section of the website, select a browser type and version, wait until it loads, and start testing.

To use BrowserStack for mobile testing, go to the App Live section of the website and choose one of the following options:

• Upload an installation file in APK or IPA format (for Android or iOS, respectively)
• Upload your app from the Google Play Store (for Android apps)
• Upload your app from the App Store or TestFlight (for iOS apps)

You can start with a free trial version to check whether BrowserStack meets your testing needs. It’s best to assess BrowserStack’s services via Google Chrome because this browser has a dedicated BrowserStack extension.

Now, let’s take a look at pros and cons of BrowserStack. After using this tool in our projects, we can outline the following benefits:

• Extensive selection of popular browsers and OSs. You can test your application in different versions of Google Chrome, Microsoft Edge, Opera, Firefox, Internet Explorer, and other browsers running on Android, iOS, Windows, and macOS.
• Large variety of mobile devices. As one of the best tools that support cloud-based mobile testing, BrowserStack provides access to more than 2,000 physical smartphones that use Android versions 4.4 through 11 and iOS versions 8 through 14. This is a significantly wider choice compared to other testing services.
• Built-in automation services. You don’t need to purchase and integrate two tools for manual and automated testing. This way, you can save your time and budget when setting up a testing environment.
• Possibility to change geolocation for mobile applications. Some mobile apps provide different sets of services in different regions. Changing the geolocation on virtual devices helps you test all services from your current physical location.
• Local testing. Transferring data to public resources such as cloud storage can be a security risk for projects that work with sensitive data. If you don’t want to store data in the BrowserStack cloud, the tool enables you to test web and mobile applications hosted on private servers.

However, we also found some disadvantages to watch out for with BrowserStack:

• Slow processing speed. Though BrowserStack allows for decent app testing on real devices, virtual devices process data much slower.
• Limited device settings. If your application requires specific settings like automatic brightness or gyroscope configurations, you probably wouldn’t be able to configure the testing environment in BrowserStack.
• Inconsistent behavior with push notifications. During our testing activities, we noticed that BrowserStack devices don’t always receive push notifications. Getting a notification can be critical for applications that alert users on home security, medical conditions, financial activities, etc. With inconsistent push notifications, it’s challenging to test the alerting features of such apps.
• Inconsistent behavior with geolocation data. Sometimes, devices can’t determine their locations when receiving geolocation data. This can be an issue for location-based applications (taxi, delivery, postal, weather forecast services, etc.), since it’s impossible to properly test location-critical services.

Despite some drawbacks, BrowserStack is an efficient tool that can help you test your applications on a number of browsers and devices.

Antivirus software compatibility testing with VirusTotal

VirusTotal offers a simple way to test the compatibility of websites and Windows and macOS desktop applications (*.exe and *.dmg files) with over 70 antivirus programs.

Here’s how to use VirusTotal: just select a file on your hard drive to analyze it, or input a URL and get results. At this link, you can see the results of scanning a .dmg file. And in Figure 1 below, you can see an example of testing a URL.

We can outline the following advantages of VirusTotal:

• Fast testing. VirusTotal helped us reduce the time for testing compatibility with antivirus software to several minutes. Before using VirusTotal, we needed up to three days to install all the necessary antivirus programs and configure a testing environment.
• License-free testing. There’s no need to purchase antivirus software licenses for real-life testing or to buy several compatibility testing tools. However, VirusTotal has a premium subscription with additional services for malware researchers and cybersecurity specialists.
• Universal testing. VirusTotal covers all major antivirus programs including Avast, DrWeb, ESET, Kaspersky, McAfee, Microsoft Security Essentials, and Tencent. Also, VirusTotal can test both web applications and macOS and Windows installation files.

However, VirusTotal has some limitations you should be aware of:

• False results. With some of our files, the antivirus program reported malware while VirusTotal didn’t. Because of such false results, we still have to manually test some files and URLs when an antivirus program reports them. However, this doesn’t take too much time, since false results are not that frequent.
• Public access to uploaded files. VirusTotal saves all files uploaded by users and makes them available for download to all users with premium subscriptions. That’s why you shouldn’t use VirusTotal to test files with sensitive data.
• Upload limitations. You can only upload files smaller than 32 MB to the VirusTotal cloud. To test files up to 200 MB, you need to request a special upload URL using the /file/scan/upload_url endpoint.

All in all, VirusTotal is a useful tool if you need to quickly test the antivirus software compatibility of websites and small files that can be uploaded to a public cloud.

Conclusion

Cloud-based tools for software testing allow you to overcome the scarcity of real testing environments and broaden your range of tested devices and applications. In this article, we showed you how to use BrowserStack and VirusTotal and discussed their pros and cons. These tools make cross-browser and antivirus compatibility testing faster and more convenient. They have some drawbacks, but you can easily compensate for them with some manual testing.

The Apriorit testing team has extensive experience testing desktop, web, and mobile applications. Contact us if your project requires testing expertise.