Pairwise testing with PICT: Task solution

Software testing is a critical phase in the software development life cycle that ensures the quality and reliability of the final product. However, testing every possible combination of parameters in a software application can be an overwhelming and time-consuming task. Pairwise testing offers a smart and efficient approach to address this challenge.

In this article, we will explore the concept of pairwise testing using the task from the previous week and introduce the PICT (Pairwise Independent Combinatorial Tool) tool—an invaluable resource for automating and simplifying the pairwise testing process.

?? Pairwise testing, also known as all-pairs testing, is a combinatorial testing technique that focuses on testing all possible pairs of input parameters in a system. Instead of testing every conceivable combination, which can quickly become impractical and time-consuming, pairwise testing leverages the fact that most defects are triggered by interactions between just two input parameters. By ensuring that all possible pairs of parameters are tested, pairwise testing can significantly improve test coverage without an exhaustive number of test cases.

? Benefits of Pairwise Testing:

1. Reduced Test Case Count: Pairwise testing dramatically reduces the number of test cases required to cover all combinations of input parameters. The testing effort is optimized, and teams can focus on more critical aspects of testing.
2. Time and Cost Efficiency: With fewer test cases to execute, the testing cycle is accelerated, saving valuable time and reducing overall testing costs. This is especially beneficial for projects with tight deadlines or limited resources.
3. Improved Defect Detection: Pairwise testing targets interactions between parameters, where many defects are often found. By covering all pairs of parameters, it maximizes the likelihood of detecting potential issues.
4. Enhanced Test Coverage: Despite the reduced number of test cases, pairwise testing maintains a high level of test coverage, ensuring that a broad range of scenarios are adequately examined.

?? Introducing PICT.

?? PICT, short for Pairwise Independent Combinatorial Tool, is an open-source software developed by Microsoft Research. It is a command-line tool designed to automate the generation of pairwise test suites. PICT is widely used across industries to streamline the process of pairwise testing and improve testing efficiency.

?? Key Features of PICT:

1. Compact Test Suites: PICT generates minimal test suites, covering all possible pairs of parameters while minimizing the number of test cases needed. This ensures optimal use of testing resources.
2. Input Flexibility: PICT allows you to define input parameters and their respective values through a simple and intuitive plain-text file. This flexibility makes it easy to integrate PICT into existing testing processes.
3. Advanced Constraints: PICT supports defining constraints and dependencies between parameters, allowing for more complex and realistic testing scenarios.
4. Coverage Reporting: PICT provides reports that show the level of coverage achieved in the generated test suites, helping testers and developers assess the adequacy of their testing efforts.

?? Using PICT for Pairwise Testing:

To employ PICT for pairwise testing, follow these general steps:

1. Define Input Parameters: Create a plain-text file (e.g., .txt or .csv) specifying the input parameters and their possible values. Ensure to include all relevant parameters to be tested.
2. Set Constraints (Optional): If certain parameters have dependencies or constraints, you can define them in the input file. This allows you to create more realistic test scenarios.
3. Generate Test Suites: Execute PICT with the input file as the input, and PICT will generate the pairwise test suites in the desired format (e.g., .csv, .txt).
4. Execute Test Cases: Run the test cases outlined in the generated test suites against the application under test.
5. Analyze Results: Analyze the test results to identify defects, potential issues, and any missing test scenarios.

So what was our task? ??

Imagine your company starts working with a new customer, "Patsy's Pizza," a pizzeria that sells pizzas offline. They decided to start online sales as well and want you to create a small app for them which provides their clients with the possibility to order pizza.

?? Here are the requirements: https://drive.google.com/file/d/1K5_r2C3tXcjLMg7Or5h9vAjHUBiUR-B1/view?usp=sharing

?? Task: Please read the requirements attentively, create the mock-up for these requirements (you can use this free online tool: https://app.uizard.io/), and then create a checklist for testing using the PICT tool for pairwise testing.

To visualize the requirements you can create a mock-up:

So, the first part of the task was done!

Now lets move on to the most interesting part connected with the main topic of this article - checklist creation!

As it was mentioned before, to create the checklist that will cover all possible combinations in this web-form for ordering pizza is just impossible! Still there is a pairwise test design technique that arranges the variables (city, pizza type, size, toppings and so on) and select a reasonably sized subset that you can test and still be sure about test coverage and product quality.

There are a lot of online tools which help you to solve this task free and without registration (for example, https://pairwise.yuuniworks.com/ or https://pairwise.teremokgames.com/), but still they will not allow you to set some conditions or restrictions. For example, while generating the combinations for our checklist the tool should take into account that there is only delivery, no carryout in Birmingham or the fact that the clients cannot select the type of crust for pizzas of Small size.

And here is ?? PICT is the best solution!?It provides?scenario generation capabilities. These are described in the manual "PICT 3.3 User's Guide." Feel free to explore them!

So, let's start with PICT installation.

This is is a command-line generation tool, so we'll use CMD to work with it.

But firstly we should create a text file in .txt format where we'll list the parameters and their values that we should should create a checklist on. This file should be added to the installation directory.

Start the command line:

And then using 'cd' command go to the directory where PICT is installed, and your .txt file is saved:

You can use the following command to generate the file with test combinations:

And the resulting file will appear in the directory where PICT is installed, and your .txt file is saved, along with its parameters:

Open the file and you'll see the set of randomized combinations (78 combinations) that are ready for testing and can be used as a checklist:

Still, if you analyze the provided checklist, you'll see a lot of issues:

? Negative checks for text fields are combined

? The limitations identified in the requirements were not taken into account. For example, you can see the carryout for Birmingham or crust selection for pizzas of small sizes that should be impossible.

Yes, sure, you can do these checks, but they are extended. And you'll have to look through all of these combinations to verify that smoke and critical path tests are created. If no - you are going to have a lot of manual work when adding them!

But that's a cool piece of news about PICT: all these limitation can be set in the initial .txt file! See the full user guide: https://www.amibugshare.com/pict/help.html

What should we do? According to PICT Guide the constrains can be set this way:

Let's edit our file:

?? Note: this is not the most rational way to add constrains here, we'll use it that way to show you basic PICT possibilities. Of course, feel free to explore PICT Guide and use more convenient advanced options to set the limitation!

If you use CMD and execute the same command one more time, we'll have:

Let's verify that the constrains we set in PICT, we considered.

For example, what about Small pizzas? Do they have only 'Original' crust?

What about Birmingham and Leeds-Bradford cities? Do they have only delivery option as it was described in the requirements?

What about our text fields? We wanted they had no combinations of invalid input. Let's check it:

?? Note: you can use equivalence partitioning technique here and add more options for negative testing for text fields, not just valid/invalid!

Feel free to explore the created file (checklist here):

https://docs.google.com/spreadsheets/d/1HnhW_SjQYO9DXWOmBSyYoxXklpQFAlcV/edit?usp=sharing&ouid=101103919551467550725&rtpof=true&sd=true

In the ever-evolving landscape of software development, efficient testing techniques are invaluable to deliver high-quality products on time and within budget. Pairwise testing, driven by the powerful PICT tool, offers a compelling solution to streamline testing efforts while maintaining a comprehensive coverage of test scenarios. With PICT, software testers and developers can unlock the potential for faster testing cycles, reduced costs, and improved defect detection. Embrace the prowess of PICT, and take your software testing process to new heights of efficiency and effectiveness.