Keyword-Driven Testing with Robot Framework

Keyword-driven testing is rather new, yet it’s already one of the most widely used approaches for application testing. While it’s suitable for manual testing, this approach is mostly used for automated testing.

In this article, we describe the basics of keyword-driven testing, highlight its main pros and cons, and explain how to run keyword-driven tests with the help of Robot Framework.

The keyword-driven testing approach

The keyword-driven testing (KDT) approach uses keywords (action words) to determine the functionality under test. Basically, these action words simulate real user actions on the tested application.

KDT separates high-level documentation of test cases from low-level keyword documentation that contains the details of test case execution. This approach is often referred to as table-driven testing or action word-based testing.

This approach consists of two main layers:

• Infrastructure layer – This layer is used as the engine that receives keywords and runs operations on the tested application. The infrastructure layer includes such elements as utility functions, item operations, and user-defined functions.
• Logical layer – This layer is based on test cases and is used for building and executing test scripts with the help of predefined keywords.

Let’s take a look at some basic types of keywords. In the case of KDT, keywords serve as blocks for building test cases. There are three main types of keywords:

• Item Operation (Item) – A keyword that determines an action that performs a specific operation on a given component of the application’s graphical user interface (GUI).
• Sequence – A set of action words determining a functional process, such as Create User. Sentences are widely used for implementing frequently used functional processes.
• Utility Function (Function) – A keyword that symbolizes a script executing a certain functional operation that can’t be implemented as a sequence.

A test case is a table containing keywords and test data, or arguments (see Figure 1).

 

Figure 1. Example of a test case

As you can see from the table above, all keywords are listed sequentially, each starting on a new line. These keywords determine the progress of the test case.

A keyword can be complex, consisting of several action words, or it can be implemented in a specific programming language. For instance, Figure 2 below shows what the keyword Create User might look like:

 

Figure 2. Create User keyword

In this example, the Create User keyword consists of a set of other action words, the implementation of which can be stored in a separate library.

One of the main benefits of using KDT is that it makes it more practical to support tests: high-level test cases can remain constant while low-level details of keyword implementation can be changed if needed. Plus, KDT allows automation engineers to plan test automation at the software development stage, even before the application is ready.

However, this approach has both pros and cons. In the next section, we take a look at the main advantages and disadvantages of KDT for performing a particular kind of tests — automated acceptance tests.

Pros and cons of keyword-driven acceptance testing

Acceptance testing is one of the most important parts of the software testing process. A keyword-driven testing framework can be used to automate this stage and make it more efficient. Robot Framework is commonly used for acceptance testing, and we’ll get back to it shortly. But for now, let’s look closer at the main pros and cons of KDT for automated acceptance testing.

Pros:

• Test cases are easy to read as there are no implementation details.
• Supporting test cases doesn’t require additional expenses.
• Keywords can be reused in different test cases.
• Development of test cases and keywords can be distributed within the team.
• There’s no binding to a particular programming language.

Cons:

• Developing initial keywords can take a long time.
• Keywords need to be modified when changing the environment or implementation.
• Learning a new framework or programming language requires additional time.

Using Robot Framework for automated acceptance testing

The main purpose of Robot Framework is test automation. In particular, this framework is widely used for acceptance testing. It’s an open source project, so you can find all information about the framework and how to use it, including Robot Framework data-driven test examples, on the project’s official website.

With Robot Test Framework, you can develop tests in Python, Java, and .NET. The syntax for describing test cases is based on a KDT approach and a table format.

To install and start working with this framework, you’ll need a Python interpreter and the pip package manager. To install the framework, use the following command:

pip install robotframework

Robot Framework has several key features:

• Simple tabular test format
• Universal keyword libraries
• The ability to create custom libraries in Python, Java, and .NET
• Detailed HTML reports and logs
• Continuous integration plugins

It’s also possible to integrate Robot Framework with an integrated development environment (IDE) such as PyCharm. After installing an appropriate plug-in (for example, robot-plugin or IntelliBot), the syntax of the framework is highlighted and tests can be run directly from the IDE.

In addition to keyword-driven tests, Robot Framework can be used to create data-driven and behavior-driven (Gherkin) tests.

Now let’s see how to compose an automated acceptance test with the help of Robot Framework.

Composing tests

In Robot Framework, tests are sets of tables that contain test data, variables, keywords, and advanced settings (such as library imports and metadata). Each table begins with its name. There are four possible names for Robot Framework tables:

• Settings
• Variables
• Test Cases
• Keywords

Robot Framework supports several file formats for test cases. The preferred formats are HTML, TSV (tab-separated values), space-separated, and reST (reStructuredText).

Let’s take a closer look at the test case syntax in the following example. Suppose the test object is designed to monitor file operations in Windows and consists of a driver and a service. The following test shows how to install the application and verify that the service is logging all file operations.

Driver_test.robot:

*** Settings ***
Resource    Agent_resources.robot
Library     OperatingSystem

*** Variables ***
${msi}         C:InstallationsAgent.msi
${log_file}    C:Program DataAgentService.log
${driver}      fs_driver
${service}     fs_service

*** Test Cases ***
Driver installation
    [Tags]    Installation
    Install The Driver    ${msi}
    Service Should Be Started    ${service}
    Driver Should Be Loaded    ${driver}

File system logging
    [Tag]    Service
    [Setup]    Enable File System Logging
    Generate Read File Operations
    Generate Write File Operations
    Service Log File Should Contain FS Operations
    [Teardown]    Disable File System Logging

*** Keywords ***
Install The Driver
    [Arguments]    ${msi}
    ${install_result}= install_agent    ${msi}
    Should Be Equal As Integers    ${install_result}    0

Service Should Be Started
    [Arguments]    ${service}
    ${result}=  is_service_running    ${service}
    Should Be True    ${result}

This is an example of a space-separated test format where each value (keyword, table, variable) is separated by at least two spaces.

First, we define the settings table. In our case, we use this table to import a standard OperatingSystem keywords library from the framework. In addition to standard libraries, you can import custom libraries that have been specially created for a particular application. In this example, we use the custom Agent_resources.robot library.

Next, we move to defining the variables table, which lists the required variables. In test cases and keywords, variables are used to store temporary information. In our case, we use such variables as the installation path, log file, and names of the driver and the service.

Then we move to the test cases table, which describes high-level test cases. Each test case begins with a title. In our case, we have two test cases: Driver installation and File system logging.

Cases consist of a set of keywords and are described in a user-friendly language, without any low-level implementation details. You can add a specific tag to a test case in order to filter the statistics once the testing is finished. You can also specify the setup/teardown procedures needed for bringing the system to a certain initial state and returning to that state after finishing the test.

Finally, we define the keywords table, which contains details of keyword implementation. Each keyword can consist of other keywords created in the Robot Framework format and can be implemented as a function in a separate Python or Java library.

Running tests: launching and reports

Now, let’s take a look at the process of launching tests and getting their results in Robot Framework. A test is launched from the command line with a simple command:

robot Driver_test.robot

Test results can be displayed on the command line and in an HTML report:

 

Figure 3. Driver test report

This report shows you the statistics for particular tests or a set of tests marked with a common tag.

However, filtering isn’t the only use for tags. You can also use them for including or excluding particular test cases from being executed:

robot --include Installation --exclude Negative Driver_test.robot

Tags can also be used for determining the level of importance of a particular test:

robot --critical Web Driver_test.robot

Since Robot Framework is both system- and platform-independent, you can test your application on Windows, Mac, Linux, or Unix. To increase your chances of finding possible bugs and issues in your application, you can configure the framework to run all tests randomly.

Conclusion

Robot Framework is a universal framework for automated acceptance testing based on a keyword-driven approach. The main benefit of this approach is that it allows you to compile high-level test cases in a user-friendly language.

The framework contains a wide range of keyword libraries. Plus, you can always access detailed test reports and useful tools for working with the framework on the official Robot Framework website.

At Apriorit, we have a team of dedicated quality assurance professionals. Use the form below to contact us. We would be glad to assist you in building efficient test processes to meet any needs, from mobile app testing to driver quality assurance.