Importance of Reporting in a Test Automation Framework

I thought of re-visiting the reporting element of a test automation framework as it plays a crucial component in a well-designed test automation tool. The success of the designed test automation framework and its survival also depends on how effectively reporting mechanism will be implemented. Test automation report is the blue print that shows how the script was executed. It shows the scripts that were executed, steps in the test script which were executed, execution time, check points that were passed or failed, if failed what was the actual results and failure scree shot. Test execution report should be detailed enough when there is a test execution pass as well as a failure.

When there is a test execution failure the report should be detailed enough to identify where the test failure has occurred and the reason behind the failure. The report should be detailed enough to identify whether it’s a script failure or an application failure.

The test execution report also should be detailed enough even for a manual tester to open a defect utilizing the information provided in the report.

There are various reporting libraries that an automation framework designer can use for the reporting component rather than creating it from scratch.

They are;

1. TestNG reporting –TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use. In simple words TestNG is a tool that help us to organize the tests and help us to produce the test reports. This reporting mechanism can be used simply by adding the TestNG library to your automation framework. After adding the TestNG library we can easily add the necessary TestNG annotations and execute the test automation script. We can add BeforeTest, Test and AfterTest annotations to organized pre-conditions, test and post-conditions after executing the test. TestNG has an in-built HTML reporting mechanism, in test-output folder.

The default HTML report does not provide much information. The test engineer should add greater level of detail by adding assert statement covering all the verification checkpoints and add images when there are failures.

If we want to know the result of each run along with some custom data, we can implement our own logs and reports. TestNG provides a way of implementing custom reports and logging.

Simply put, we can either implement the org.testng.ITestListener interface for logging or the org.testng. .IReporter interface for reporting.

Following will implement simple custom login.

public class CustomisedListener implements ITestListener {
     
    // ...
    @Override
    public void onFinish(ITestContext testContext) {
        LOGGER.info("PASSED TEST CASES");
        testContext.getPassedTests().getAllResults()
          .forEach(result -> {LOGGER.info(result.getName());});
         
        LOGGER.info("FAILED TEST CASES");
        testContext.getFailedTests().getAllResults()
          .forEach(result -> {LOGGER.info(result.getName());});
         
        LOGGER.info(
          "Test completed on: " + testContext.getEndDate().toString());
    }  
  
    //...
}

TestNG also produce ‘index.html‘ report and it resides in the same test-output folder. This reports gives the link to all the different component of the TestNG reports like Groups & Reporter Output. On clicking these will display detailed descriptions of execution.

2. Extent Reporting – This is much more interactive reporting mechanism that can be integrated with a Selenium test automation framework. This facility can be availed by simply downloading the extent report libraries and adding them to your Selenium project.

There are many features provided by extent report.

• Interactive – Extent report is provided in HTML format but its more interactive in nature with lot of UI widgets.
• Multiple reports - ExtentAPI can produce an interactive report, real-time reports with slf4j and also email reports which can be mailed right after your unit test session.
• Supports many test automation frameworks - Can be configured to support almost all Java and .NET test frameworks such as TestNG, JUnit, NUnit etc.
• Historical Test Analysis - ExtentX allows you to track test history from your builds via the Test view to see how your application functionality is behaving overtime.
• Dashboard support - ExtentX provides detailed and graphical analysis for your projects, reports, categories, authors and tests to enable you to find the strong and weak areas in your application.

3. Allure reporting – This is a test reporting framework developed by Yandex QA Team – Allure. The Allure libraries can added by adding the Maven dependencies to your Mavanized test automation framework. This reporting framework also combines with TestNG.

Example 01

To show the Display name in the report we use description attribute in @Test annotation

@Test (priority = 0, description=”Invalid Login Scenario with wrong username and password.”)

Example 02

To display the test steps in the Allure report we use the @Step annotation.

Allure reporting framework is also shipped with an inbuilt web server which provides a nice dashboard with a very detailed, interactive and attractive report. For this open a command prompt screen, go to the project directory, and write below command.

Compared to extent report I see that new Allure reporting is much easier to work with and has more features in built, specially the nice dashboard it provides.

4. Integrating JIRA reporting with your Java Selenium test automation framework – If your project or your organization is using JIRA as the project management and as the bug tracking tool, JIRA can be utilized as the test execution reporting tool. This can be done by integrating your test automation framework with the JIRA Rest API provided. With such API we can do wonders like remotely login into JIRA and uploading already generated HTML report or maintaining a test execution dashboard within JIRA itself.

So there are so many alternative options that we can use when it comes to the implementation of test execution reporting in your perfectly designed Java Selenium test automation framework. This article is published to help everyone to know the available options and how we can make reporting better.