Pairwise Testing or All-Pairs Testing Tutorial with Tools and Examples

The Efficiency of Pairwise Testing as A Technique for Bug Detection:

This piece will delve into a method of test design known as “Pairwise Testing,” which is also commonly referred to as “All-Pairs Testing.”

Effective testing is essential, particularly when system testing teams are frequently working under strict deadlines. Accordingly, test design methods should be highly efficient in terms of achieving maximal test coverage and a high defect yield rate.

Definition: What is Pairwise Testing?

Pairwise Testing is a technique for designing tests that ensures comprehensive test coverage.

If we are to go by ISTQB, All-Pairs Testing (or Pairwise Testing) can be described as a black-box test design method in which test cases are generated to encompass all potential individual combinations of every pair of input parameters.

The output of a software application hinges on various factors like input parameters, state variables, and environment configurations. Approaches like boundary value analysis and equivalence partitioning can assist in determining potential values for individual factors. Testing all potential combinations of values for these factors, however, is not feasible. Therefore, a subset of combinations is generated to cover all factors.

Using the All-Pairs method can prove invaluable when creating tests for applications that include multiple parameters. Each pair of input parameters is tested using all potential individual combinations. The test suite covers all combinations, making it extremely efficient in detecting bugs.

Let’s delve deeper into how to implement All-Pairs Testing with an example.

Example of Pairwise Testing

Example: Application for Ordering Cars:

• The application for ordering cars permits users to purchase and sell cars, offering support for transactions in Delhi and Mumbai.
• The application requires both valid and invalid registration numbers and should facilitate the trade of cars from BMW, Audi, and Mercedes.
• Customers can opt for two kinds of bookings: E-booking and In-store.
• Orders can only be placed during business hours.

Step #1: Determining the involved variables:

1. Order category: Buy or Sell
2. Location: Delhi or Mumbai
3. Car brand: BMW, Audi or Mercedes
4. Registration numbers: Valid (5000) or Invalid
5. Order type: E-Booking or In-store
6. Order time: Working hours or Non-working hours

If we are to test all potential valid combinations:
= 2 X 2 X 3 X 5000 X 2 X 2
= 240,000 valid combinations for test cases.

There are also an infinite number of invalid combinations.

Step #2: Simplifying the combination:
– Use a representative sample.
– Group and establish boundaries, even when the data isn’t discrete.
– Reduce the number of registration numbers to:

1. Valid registration number
2. Invalid registration number

Now, you can calculate the number of potential combinations:
= 2 X 2 X 3 X 2 X 2 X 2
= 96

Step #3: Organizing variables and values:

Arrange the variables along with their corresponding values as demonstrated below.

Place the variables in descending order based on the number of values they contain.

Step #4: Develop a test suite:

Fill out the table column by column. The table should initially resemble this. Repeat the values of the variable with the highest number of values (Product) two times (as the next variable, Order category, has two values).

The Order Category column includes two values. Thus, we must repeat the values of the first column, Product, twice.

For each set of values in column 1, we should include both values of column 2. The same process needs to be repeated for column 3.

We have Buy and Delhi, but no combination for Buy and Mumbai. We have Sell and Mumbai, but lack the combination for Sell and Delhi. Let’s switch the values in the second set of values in the third column.

The table is now much improved!

The same steps should be repeated for columns 3 and 4.

When comparing columns 3 and 4, each value in column 3 should include both values of column 4. However, when comparing the 2nd and 4th columns, we have Buy with Valid and Sell with Invalid. This means Buy lacks “Invalid” and Sell lacks “Valid.” As a result, we must switch the last set of values in the 4th column.

Column 6 (Order time) reveals an issue. We’re missing the combination of Buy/Non-working hours and Sell/Working hours. As all the rows have already been switched, we can’t accommodate the missing pairs by switching. Therefore, we add two more test cases to include these pairs. Hence, the empty rows!

Now, we have the freedom to fill in the vacant cells as we like, because the values of the rest of the variables are discretionary or irrelevant.

Alas! We’ve managed to cover all pairs in just 8 test cases, rather than all combinations in 96!

This, folks, is the power and simplicity of the All-Pairs method in test design. It dramatically enhances the likelihood of spotting bugs, making it both a fun and potent method.

However, the pairwise testing method also comes with its share of limitations:

• It may not function as intended if the values selected for testing are erroneous.
• It may overlook combinations which are most likely.
• It may not work if the interactions between the variables are not properly understood.

Tools for Pairwise Testing:

There exists a plethora of tools that utilize the all-pairs testing method and can simplify the process of designing automated test cases. These tools can generate a compact set of parameters as desired test cases. Here are a few well-recognized tools in the industry:

• PICT – Pairwise Independent Combinatorial Testing offered by Microsoft Corp.
• IBM FoCuS – Functional Coverage Unified Solution provided by IBM.
• Hexawise
• Jenny
• Pairwise by Inductive AS
• VPTag – a cost-free All-Pair Testing Tool

Conclusion:

The pairwise testing method, by decreasing the number of combinations to be covered, remains highly efficient when it comes to uncovering defects. It is a shrewd test design strategy that manages to strike a balance between testing effort and efficacy.

During the test planning phase of software testing, the pairwise testing method should always be taken into account. Regardless of whether it is applied manually or using tools to generate test cases, it forms a crucial part of the testing plan as it impacts test estimation.