Exploratory testing – creation or disorder motion of the ideas?

When you first meet the concept of the exploratory testing you can think that its name fully exposes the sense of this sort of activity. But if you get a little bit deeper – this confidence weakens and disappears. Especially clear it can be seen when balancing between the concepts of the exploratory and wild testing, which in one hand are synonyms in QA industry, and in another (from the good sense point of view) – contrary one to another. It’s because what can have in common the exploration – serious, responsible and well-organized process – and something wild and chaotic, that is wild testing?

 

All testers do this…

The exploratory testing is the only type of testing which does not have a certain scenario. There are no documented Test Plan and test cases. All of scenarios are born in the tester’s mind just in the process of testing and are executed at once. Here is the main difficulty of the process – you should have a considerable base of knowledge and experience.

The main purposes of the exploratory testing are study of the software product (if it’s the first time when tester works with it) and search for the cases untaken into account in the test documentation (if it’s not the first session of work). Watching the program, you hypothesize about its possible behavior, the variants of its responses to your actions and then check up these hypotheses.

And however you can’t do it without a plan. To make the testing more effective it is recommended to create the check-list which will guide you. It’s not obligatory to have it written down but it’s strongly recommended to bear it in mind.

The ideal variant of realization of the exploratory testing is drafting and passing user-cases or tests, which are maximally close to the actions usually performed by user.

If it is the first time when the program is tested, then putting yourself on the user’s place you can find out the most critical bugs for the application functioning. And even if it’s not the first time you work with an application it’s useful to think like a user. The more time tester spends with developers the more he adopts their way of thinking and move away from user. The situation is aggravated by the shortage of time for testing. The more systematic character has the execution of test sets and the less time team has for it the less fantasy is shown by the tester. It’s hard to meet the schedule when you find an «interesting» situation and start to experiment. As the result the vision of the project stiffens, associations become very specific and linked to the concrete steps and terms. It is difficult to find something new or do the same in other way with such adaptation.

So, performing the exploratory testing always try to think in the way a user thinks!

During the testing you should not limit yourself by some rational scopes or paradigms – in fact users very often perform very inefficient actions which can have some unpleasant results.

Try to analyze, what functionality is utilized more frequently and what is touched much rarer. This information will help you to distribute your attention among different features.

The important initial stage of the exploratory testing is formulating the goal. Ask yourself: «Why do I do it? For what purpose?» After completing the testing try to measure your success.

The list of the most widespread goals

• Study of the new product;
• Exposure of omissions in the test-plan. The result is to add non-standard tests to the test coverage;
• Performing the testing of the project with incomplete or inexact specification or when the specification is absent;
• Research of the specific defect;
• Improvement of the product quality after completion of the basic (for example, functional) testing.

Study the new project

Mostly common the exploratory testing is used as the tool for the study of the new project or functionality.

In this situation you should try to learn all accessible information before starting the testing process.

Documentation is on the first place, as usual. If it’s present, of cause… Specifications, requirement specifications will always remain the faithful helpers of tester.

If consider the process of the exploratory testing as the building of a house, then the documentation is its reliable foundation.

The second step is to ask developers and project managers. They always possess necessary knowledge about the project, therefore do not ignore ANY useful information, ask them questions. My experience shows that this meeting can considerably change your priorities and add new test cases.

Choosing approach

The next stage in building of our house is to choose a project. How we will build? What materials will be used? Where will windows and doors be?

Returning to the QA industry terminology we should choose testing approach. There exist a lot of them, we will consider a few basic ones:

• Basic functionality. It is especially important when we have tough time constraints. In this case we shouldn’t get into details and get stuck in some little things. First of all it is necessary to check up main functionality and make sure in its capacity for work.
• Places where bugs appear the most frequently. Such places can be the new functionality or those parts of the program which are the most frequently rebuilt. Certainly, this approach is applicable only in case if it’s not the first time you work with the program, and already know its thin places.
• The simplest use-cases (for the new build validation). If you’ve got the new version, or version after the deep refactoring then don’t be too lazy to check up the most ordinary use-case, and if it fails return this version to the developers with confidence.
• Set of the “favorite” bugs. It will work if the tester knows the programmer well. Each developer has the set of the “favorite” bugs – and each software solution has. When there is much experience of testing even the exploratory testing can be performed by the algorithm.
• Concentration on everything and on nothing concretely. This approach is slightly philosophical. Begin to work with the application as a user and concentrate your attention on everything and on nothing concretely. It’s just like you watch from the outside what is going on. Don’t try to analyze anything while working but be ready to start the analysis at any moment and reproduce all of the steps.
• Vulnerability. Those who know the design and UML will get it easier. The schematic model of the project is created, and the weakest place is determined using different methods and approaches. Analysis can be performed from the position of user, architecture or realization. Unlike the usual test design here we analyze the specific model – mixture of architecture of business processes and human perception.

Choosing the executors

So we chose the project for our house J. Next question is about the workers. Who will be assigned to this responsible task?

The first are QA specialists. When choosing the responsible tester you should estimate his experience. It is not inconceivable that assigning an inexperienced man to perform the exploratory testing of the new product you will spend time for unavailing mouse clicking.

It is good to get not only QA specialists to take part in the exploratory testing but also other team members, for example developers. Frequently a tester thinks deeper than a user usually does and so sometimes he complicates a task while not seeing the most obvious errors. The exploratory testing will be good here with its fresh look to the situation.

After performing the regression testing of the product many companies invite independent experts who perform the exploratory testing, simulating the behavior of end users. Expenses on such testing usually justify themselves, as far as critical bugs will be found and fixed before the release.

3 stages of exploratory testing

Watch Out! Construction Work!

Many sources describe the process of the exploratory testing as the following cycle of actions:

study of the product -> planning of tests -> implementation of tests

And everything this, as already mentioned higher, is not fixed in documents but takes place only in the tester’s mind. It is another argument in behalf on that the exploratory testing should be performed by really experienced specialists: it is the only type of testing, which includes 3 stages of testing of software product simultaneously.

The following questions can help in the process of planning of tests:

• And what will happen if …?
• And how the program will respond on such my actions?
• What data can I get?

Summarizing we can say that the slogan of the exploratory testing is: «Imagine the problem, and then find it».

Summarizing and evaluating: merits and demerits

Walls are ready. Just one small thing is left to do– to construct the roof.

In this section we’ll sum up and, as a result, describe pluses and minuses of the exploratory testing.

Merits of the exploratory testing:

1. Exposes functionality, uncovered by tests;
2. Helps to find the skipped bugs;
3. Allows to look at the product from the end user point of view;
4. Helps to realize the mission of the tested product, improves understanding of it;
5. Requires little preliminary work as far as it’s not too formalized;
6. Helps to find out the majority of critical bugs almost at once;
7. It is acceptable in terms of lack of resources;
8. Discovers usability bugs.

Demerits of the exploratory testing:

1. It is impossible to estimate time for this testing, and, consequently, to set the terms of project completion;
2. It is hard to estimate the project if only exploratory testing is used – it is impossible to make statistics of passed and to failed tests, which is one of basic indexes of product quality.
3. There is no other possibility to check up results but just fully retest the project;
4. It’s impossible to review test cases;
5. The exploratory testing never can be performed identically for several times. Some tests will slip out each time, and so there is a risk to skip some bugs;
6. Results depend much on the experience of tester.

Conclusion

You can read music and you can improvise. Any skills suppose, foremost, that you master some basic techniques and also have certain framework to use as a ground of improvisation.

The same is true for the testing. The exploratory testing is improvisation, the highest degree of tester mastery. It is a complex creative process requiring significant efforts. To perform really qualitative exploratory testing tester should have a lot of experience.

Perform the exploratory testing for the first acquaintance with a product, and it will help to study and understand your application.

Execute the exploratory testing simultaneously with the regressive testing – you will promote the quality of your product, discovering the serious non-standard errors just on the testing stage.

Perform the exploratory testing after basic one and you will be able to discover and remove bugs that slipped out before. It’s also essential to come up with new QA process improvement ideas from time to time.

And let your project be high-quality!