Part 4 - Hands-On Test Automation Project with Java + Cucumber + Selenium + Spring + Docker

This article is a continuation of Part 3 - Configurations. The implementation of Part 3 can be found in my demo-spring-selenium project tagged as v3.0. You can download this release to use it as a starting point for this part of the article.

Part 4 - Reporting, Screenshots and Logging

This part will go through more on how to add custom reporting, embed screenshots to the report, and add logs with customized outputs.

Reporting

Reports are very important to have in any test project. A good report allows us to evaluate the current status of the project and the quality of the product. It makes it easier to take corrective actions if it is necessary. The test report can also determine if the product is ready for release or not. The key thing on setting your report is to make sure that the report you generate will give visibility to every stakeholder (team, product owners, managers, etc.) of the tests that passed and/or failed, and if it failed for what reasons. 

Cucumber have different formatting options for the output reports: 

• pretty: prints the gherkin source with additional colors and stack traces for errors. 
• html: generate a HTML report.
• json: generate a JSON report with all the information from the gherkin source.

For example, to generate all these reports you need to add them as a plugin in CucumberOptions of the CucumberRunner.java class. It would look like as follows:

plugin = {
    "pretty",
    "html:build/test-results/html-report.html",
    "json:build/test-results/json-report.json"
}

Execute the test again and you will see that the "pretty" option will provide a more verbose output, see the image below.

You will also notice that on the folder build/test-results the HTML and JSON reports were also generated. The HTML report will look like this:

The Cucumber HTML report is very poor in usability. There is a report plugin option that is pretty cool and I like more, which is Masterthought Cucumber. Since Cucumber generates a report in JSON format as well, the JSON file can be used as an input for this plugin to generate a nicer HTML report version. 

We need to create a task inside build.gradle that will generate the report right after the tests are done. First, add the following at the top of your build.gradle.

import net.masterthought.cucumber.ReportBuilder
import net.masterthought.cucumber.Configuration

buildscript {
    repositories {
        jcenter()
    }

    dependencies {
        classpath "net.masterthought:cucumber-reporting:5.3.0"
    }
}

And then add the following at the end of your build.gradle file. As you can see we created a method that generates the report. We are calling this method inside the task generateReport. The following line test.finalizedBy generateReport is saying to Gradle to execute the report task after the test execution.

def generateReport() {
    File reportOutputDirectory = new File("build/test-report")
    reportOutputDirectory.deleteDir()

    def jsonReports = fileTree(dir: "build/test-results").include '**/*.json'.toString()
    List<String> jsonReportFiles = new ArrayList<String>()
    jsonReports.each { File file ->
        jsonReportFiles.add("build/test-results/${file.name}".toString())
    }

    String projectName = project.name
    Configuration configuration = new Configuration(reportOutputDirectory, projectName)
    configuration.addClassifications("Version", project.version);

    ReportBuilder reportBuilder = new ReportBuilder(jsonReportFiles, configuration)
    reportBuilder.generateReports()

    println("\nReport: file://" + reportOutputDirectory.getCanonicalPath() + "/cucumber-html-reports/overview-features.html")
}

task generateReport() {
    doLast {
        generateReport()
    }
}


test.finalizedBy generateReport

Execute the following command to run the test:

./gradlew test

You should see at the end of the execution a link to the custom report and then open it.

Screenshots

Capturing a screenshot is one of the frequently desired actions when we automate a test with Selenium. The screenshot can be useful in different ways, such as providing evidence of the test execution, or evidence of a problem encountered, or serving to analyze and compare the state of the interface, etc.

We need to use TakeScreenshot class from Selenium to take screenshots of the browser and then use the Cucumber class called Scenario in the @After method to embed the screenshot to the report. Let's add the following lines to our @After method inside Hooks.java class.

@After()
public void closeBrowser(Scenario scenario) {
	byte[] screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
	scenario.attach(screenshot, "image/png", scenario.getName());
	driver.quit();
	log.info("[ENDED] Scenario: " + scenario.getName());
}

Execute the test again with the following command:

./gradlew test

Open the report and navigate to the scenario details. Expand until you see the After step. You will be able to see a link with the scenario name that once clicked will expand and show the screenshot. This is evidence that the scenario was executed successfully.

The way we set up the project all scenarios will have a screenshot. In case you want a screenshot just for the failed scenarios you just need to add a conditional statement as shown below.

if (scenario.isFailed()) {
  byte[] screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
  scenario.attach(screenshot, "image/png", scenario.getName());
}

Logging

Sometimes reports do not give all the information necessary to any investigation. Tests may fail due to different reasons: application issue, execution timeout, or even test code issue. You should be able to log things properly so you can find out the failure reason quickly. 

Some projects (like the one we are building) are very simple, and sometimes additional logging is not needed, but I want to mention how I add logging to my projects when I need it. Sometimes using System.out.println() seems to be a simple and easy option, however it is not a good approach. Loggers are libraries we can add to our project that gives us a more structured way of logging information. The main advantages I see on using appropriate libraries for logging are:

1. Logging Levels: It allows to log information with log level (i.e., ERROR, INFO, DEBUG) which later can be used as a filtering criteria (e.g., you might get interested in DEBUG messages only when creating/changing a test scenario, and be interested in suppressing DEBUG messages when executing scenarios on a server).
2. Customized Output: It allows you to print formatted output by specifying a formatting pattern, which can include a timestamp, class name, thread, etc. A formatted output helps troubleshooting and debugging.

In Java, we have several ways of doing it. Apache Log4j is one popular sample of logging utility. Logback is another option, and it is intended to be a successor of the Log4j project, picking up where log4j leaves off. You can also use the Simple Logging Facade for Java (SLF4J) that serves as a simple facade or abstraction for various logging frameworks (e.g., java.util.logging, logback, log4j). You will need to choose one for your project. But using Lombok makes all this simple to add to your test project.

In our project lets make the @Before and @After methods to log when a scenario starts and ends. Inside the Hooks.java class add the annotation @Slf4j on the top of the class as shown below.

@Slf4j
public class Hooks {
  // class content
}

This annotation causes Lombok to generate a logger field, so you can replace the @Before and @After methods with the following code:

@Before
public void openBrowser(Scenario scenario) {
  driver.get(properties.getHost());
  log.info("[STARTED] Scenario: " + scenario.getName());
}

@After
public void closeBrowser(Scenario scenario) {
  driver.quit();
  log.info("[ENDED] Scenario: " + scenario.getName());
}

Execute the test again and you should see the following line in the console.

2020-08-01 22:26:44.074 INFO 82698 --- [  Test worker] d.spring.selenium.stepdefinitions.Hooks : [STARTED] Scenario: Login successfully

The six logging levels used by SLF4J are (in order):

1. Trace (the least serious)
2. Debug
3. Info
4. Warn
5. Error
6. Fatal (the most serious)

For example, FATAL is the highest logging level. If you set the logging level to FATAL, you will only see the logging messages with FATAL type. If you set the logging level as TRACE, you will see the logging messages belonging to all 6 logging level types. Here is a table explaining these options.

Let’s say we want to change the logging level of our project to ERROR. This means only ERROR and FATAL logging messages will be shown in the console output. We should add the following to our application.yml file. Execute the test again and you will observe that the log.info() we added to the @Before and @After methods will not be shown anymore.

logging:
  level:
    ROOT: ERROR

I just wanted to introduce the Logger concept and I will not expand the explanation on other functionalities, but you can check more about it on:

• SLF4J Documentation: https://www.slf4j.org/
• Eugen Paraschiv's page: https://www.baeldung.com/java-logging-intro.

We are done with Part 4 of this article. :)

Remember to commit your changes to Git:

git add .
git commit -m "Adding Reporting, Screenshots and Logging"
git push

The implementation of this part can be found in my demo-spring-selenium project tagged as v4.0. You can download this release to check how it looks and/or use it as a starting point for Part 5 of this article.

Thanks for reading! I hope you found it helpful.

If you enjoyed it, I appreciate your help in spreading it by sharing it with a friend.