Automating screenshot and log file capture for Allure reports by Nick Karamaniolas

"Strategic tooling can lead to substantial improvements in software development workflows. It guarantees that comprehensive logging and diagnostic information are consistently captured and reported, supporting robust testing practices and fostering a culture of continuous improvement."?

Solving problems and automating work across projects: our shared QA library??

If your team has to navigate the complexities of multiple software projects, you know that it demands more than just expertise. It calls for innovative solutions.?

My team builds several different products. To ensure that we can keep delivering consistent, high-quality results across all of those projects, we developed a shared library that unifies test-oriented functionality across our projects and promotes reusability. The components of our shared library do many different things for us, and today I'll share with you how one of those components helps us report test outcomes and understand why a test failed.?

A star of our QA library: our very own Allure Attachments Handler

At the heart of our reporting process is Allure, a lightweight test report generation tool that produces detailed and interactive HTML reports for test execution. It supports multiple languages and testing frameworks, making it versatile for different development environments. Allure helps teams visualize test results by providing a clear and comprehensive overview of the test execution process, allowing teams to better understand test outcomes and pinpoint issues quickly. (You can find out more about Allure?here .)

Allure out of the box is a great solution, but without some custom work, you can't automatically attach artifacts like screenshots and log files to test runs. And so we developed the Allure Attachments Handler. Designed to support both Selenium and Playwright testing frameworks, it ensures comprehensive documentation and reporting of test artifacts, which streamlines the debugging process and enhances the clarity of test outcomes. This powerful tool assists debugging and reporting efforts by seamlessly integrating screenshots, log files, and Playwright trace files into Allure reports.?

A standard Allure report with no attachments.

Which problems were we trying to solve with respect to Allure??

Before we developed the Allure Attachments Handler, our teams lacked clear, useful and concise reporting from Selenium and Playwright tests. Each project operated in isolation, resulting in duplicated efforts and inconsistent approaches to capturing and reporting test artifacts.?

The resulting increase in development time and cost made it difficult to standardize testing practices across the organization. And the lack of a unified method for capturing and reporting test artifacts often led to insufficient debugging information, hindering our ability to identify and resolve issues quickly.

Build or buy? Exploring the testing toolbox?

Before we wrote a single line of code for a solution or spent any money on a third-party solution, we explored some alternatives and approaches that could potentially address our challenges. Here are some of the notable alternatives:

1. pytest-html:?This?pytest?plugin generates attractive HTML reports. While useful, it lacked the comprehensive artifact attachment capabilities we needed, especially for capturing visual elements and detailed log files. The plugin's simplicity and ease of use were appealing, but it did not provide the depth of reporting necessary for our multi-faceted test environments.
2. Custom-built solutions:?We considered developing a custom reporting solution tailored to our specific needs. This approach would give us full control over features and integrations. However, it also meant significant initial development and ongoing maintenance efforts, diverting resources from core development tasks. Creating a solution from scratch would have required a considerable investment in time and expertise to achieve the level of functionality and reliability we required.

Ultimately, we chose to develop the Allure Attachments Handler. This decision was driven by our need for a unified, flexible solution that could integrate seamlessly with our existing testing frameworks and provide comprehensive reporting capabilities. Our solution incorporated several existing libraries to streamline the development process and ensure robust functionality:

• Selenium and Playwright:?We use these well-established testing frameworks to support both browser automation and comprehensive artifact collection.
• Allure reports:?By building on the powerful Allure reporting framework, we ensured that our reports would be visually appealing and rich in detail, meeting the needs of various stakeholders.
• Existing?pytest plugins:?We used?pytest plugins to integrate with our test suites, allowing us to build on familiar tools and minimize the learning curve for our team.

By combining these established libraries with our custom enhancements, we created a solution that addressed our specific needs while benefiting from the reliability and community support of widely used tools!

Our solution at a glance?

Our primary focus for this tool was to capture and attach detailed testing artifacts to Allure reports, enhancing our test reports with crucial data we would need in case of a failed test.

The key features are:

• Screenshots and log files for Selenium:?The Handler automates the attachment of browser state screenshots, DOM sources, cookies, and console log files, providing a comprehensive view of the test environment at any time.
• Trace files for Playwright:?For projects using Playwright, the Handler includes trace files in the reports, which are crucial for a detailed analysis post-test execution. These trace files encompass screenshots, network log files, and more, offering a granular view of the test conditions and outcomes.
• Cross-platform compatibility:?Designed with versatility in mind, the Handler supports both Selenium and Playwright frameworks, ensuring that teams can use its functionality regardless of the underlying technology stack.

This integration not only simplifies the debugging process but also significantly boosts the efficiency of our reporting mechanisms. This allows stakeholders like engineers, managers, product owners (and hey, even us testers!) to gain deeper insights into the outcomes of automated tests.?

Inside the Allure Attachments Handler: a technical deep dive

Here, we delve into how the Handler operates with both Selenium and Playwright, providing insights under the hood.

Integration with Selenium

For Selenium-based tests, the Handler employs a multi-faceted approach to capturing test artifacts:

• Conservative use of WebDriver initialization:?The?webdriver?property initializes the Selenium WebDriver instance only when first accessed, reducing unnecessary overhead for tests that don't need it.
• Attaching log files and screenshots:?Methods like?attach_selenium_screenshot?and?attach_selenium_logs?capture and attach various Selenium artifacts (such as screenshots, page source, cookies, console log files, and network log files) to the Allure report for detailed test analysis.
• Network log file processing:?The?_process_selenium_network_logs?method filters and formats raw performance log files from Selenium to include specific?XMLHttpRequest?(XHR) events, which are then attached to the Allure report.

Using Selenium Webdriver and attaching useful info to Allure reports

?@property

??? def?webdriver(self):

??????? """

????????Lazily loads and returns the Selenium WebDriver instance when needed.

????????This property ensures that the WebDriver is only initialized when it's first accessed.

????????This approach is beneficial for scenarios where the WebDriver might not be needed for all tests,

????????reducing unnecessary initialization overhead. The WebDriver instance is obtained from the

????????Pytest fixture named "driver", which should be defined in the test setup.

????????Returns:

????????????WebDriver: The Selenium WebDriver instance obtained from the fixture.

????????"""

??????? if?not?hasattr(self, "_webdriver"):

??????????? self._webdriver?=?self.request.getfixturevalue("driver")

??????? return?self._webdriver

??? @staticmethod

??? def?_process_selenium_network_logs(selenium_network_logs:?list)?->?str:

????????"""

??????? Processes?Selenium?network?log?files?to?extract?and?format?network?request?and?response?events.

??????? This?method?filters?the?raw?performance?log?files?obtained?from?the?Selenium?WebDriver?to?include?only

??????? XMLHttpRequest?(XHR)?events,?specifically?"Network.requestWillBeSent"?and?"Network.responseReceived"?events.

??????? Args:

??????????? selenium_network_logs?(list):?A?list?of?raw?log?files?obtained?from?the?Selenium?WebDriver's?performance?log.

??????? Returns:

??????????? str:?A?JSON-formatted?string?of?the?filtered?network?events.

??????? """

??????? logs_list?=?[

??????????? json.loads(log["message"])["message"]

??????????? for?log?in?selenium_network_logs

??????????? if?json.loads(log["message"])["message"]["method"]

??????????? in?["Network.requestWillBeSent", "Network.responseReceived"]

??????????? and?json.loads(log["message"])["message"]["params"]["type"]

??????????? == "XHR"

??????? ]

??????? return?json.dumps(logs_list,?indent=2,?sort_keys=True)

??? def?attach_selenium_screenshot(self):

??????? """

????????Attaches a screenshot of the current state of the Selenium WebDriver to the Allure report.

????????The screenshot is attached as a PNG image with a predefined name.

????????"""

??????? allure.attach(

??????????? self.webdriver.get_screenshot_as_png(),

??????????? name="Screenshot ???",

??????????? attachment_type=allure.attachment_type.PNG,

??????? )

??? def?attach_selenium_logs(self):

??????? """

????????Attaches various log files from the Selenium WebDriver to the Allure report.

????????This includes the page source (DOM), cookies, browser console log files, and processed network log files.

????????Each log is attached with a specific name and format to the report.

????????"""

??????? allure.attach(

??????????? self.webdriver.page_source,

??????????? name="DOM ???",

??????????? attachment_type=allure.attachment_type.JSON)

??????? allure.attach(

??????????? str(self.webdriver.get_cookies()),

??????????? name="Cookies ??",

??????????? attachment_type=allure.attachment_type.JSON)

??????? allure.attach(

??????????? str(self.webdriver.get_log("browser")),

??????????? name="Browser console ??",

??????????? attachment_type=allure.attachment_type.JSON)

??????? allure.attach(

??????????? self._process_selenium_network_logs(

??????????????? self.webdriver.get_log("performance")),

??????????? name="Network logs ??",

??????????? attachment_type=allure.attachment_type.JSON)

Integration with Playwright

With Playwright, the Allure Attachments Handler extends its capabilities to include comprehensive trace files, which are pivotal for post-execution analysis:

Conditional trace file attachment:?The?attach_playwright_tracefile?method conditionally attaches Playwright trace files to the Allure report, based on whether the test failed and the?trace_on_failure?flag.

Trace file generation and attachment:?The?_generate_and_attach_tracefile?method generates a Playwright trace file, attaches it to the Allure report, and deletes the file after ensuring it's created within a specified wait time.

Arguments and operations:?The methods use the Playwright?BrowserContext?to manage tracing, with options for conditional attachment based on test outcomes. And they include specific messages and filenames for the Allure report.

Using Playwright’s?BrowserContext to generate an information-rich trace file:

?def?attach_playwright_tracefile(self,?context,?trace_on_failure=True):

??????? """

????????Conditionally attaches Playwright trace files to the Allure report, based on test failure status.

????????If trace_on_failure is True and the current test is marked as failed, a trace file is generated and attached.

????????Otherwise, if trace_on_failure is False, a trace file is always generated and attached.

????????Args:

????????????context (BrowserContext): The Playwright BrowserContext instance.

????????????trace_on_failure (bool, optional): Determines if trace files should only be attached on test failures. Defaults to True.

????????"""

??????? if?trace_on_failure:

??????????? # Check if the current test is marked as failed

??????????? if?hasattr(self.request.node, "pytestmark")?and?any(

??????????????? mark.name?== "failed_test"

??????????????? for?mark?in?self.request.node.pytestmark

??????????? ):

??????????????? message?= "Step failed: A trace zip file was produced. Head over to https://trace.playwright.dev/ and simply drag & drop the file there"

??????????????? self._generate_and_attach_tracefile(context,?message)

??????? else:

??????????? message?= "A trace zip file was produced. Head over to https://trace.playwright.dev/ and simply drag & drop the file there"

??????????? self._generate_and_attach_tracefile(context,?message)

??? def?_generate_and_attach_tracefile(

??????? self,?context,?message="traces generated."

??? ):

??????? """

????????Generates a Playwright trace file, attaches it to the Allure report, and then deletes the file.

????????Before calling this method, tracing must be started using context.tracing.start(...)

????????at the beginning of the test or during the setup phase. This ensures that the entire test execution

????????is captured. After the test actions are completed, this method can be called to stop tracing,

????????save the trace file, attach it to the Allure report, and clean up by deleting the file.

????????Args:

????????????context (BrowserContext): The Playwright BrowserContext instance used to stop tracing and generate the file.

????????????message (str, optional): A message to be attached to the Allure report along with the trace file. Defaults to "traces generated.".

????????"""

??????? allure.attach(

??????????? message,

??????????? name="Test failure!",

??????????? attachment_type=allure.attachment_type.TEXT,

??????? )

??????? with?allure.step(message):

??????????? timestamp_now?=?time.strftime("%Y%m%d-%H%M%S")

??????????? trace_filename?= f"{current_dir_path}/trace_{timestamp_now}.zip"

??????????? context.tracing.stop(path=trace_filename)

??????????? failed_scenario_name?=?self.request.node.originalname

??????????? max_wait_time?=?10? # maximum time to wait for the file to be generated after tracing.stop (in seconds)

??????????? start_time?=?time.time()

??????????? while?(

??????????????? not?os.path.isfile(trace_filename)

??????????????? and?(time.time()?-?start_time)?<?max_wait_time

??????????? ):

??????????? if?not?os.path.isfile(trace_filename):

??????????????? raise?FileNotFoundError(

????????????????????f"The trace file was not created within the maximum wait time of {max_wait_time} seconds."

??????????????? )

??????????? allure.attach.file(

??????????????? trace_filename,

??????????????? name=f"trace_{failed_scenario_name}.zip",

??????????????? extension="zip",

??????????? )

??????????? os.remove(trace_filename)

A step-by-step guide to running the Allure Attachments Handler

Using the Allure Attachments Handler within our testing framework and projects is straightforward, thanks to its seamless integration with Selenium and Playwright environments.?

Below are step-by-step guides on how we use the Handler in our projects.

Running the Handler with Selenium

Setup:

• Ensure that?commons_lib?(the name of our shared library) is included in your project dependencies.
• Configure Selenium WebDriver to suit your testing environment.

Implementation:

• Instantiate the?AllureAttachmentsHandler?at the beginning of your test or in a setup method.
• Use the handler methods to attach desired artifacts at different stages of your test.

Example usage

from?selenium import?webdriver

from?commons_lib.allure_attachments_handler import?AllureAttachmentsHandler

driver?=?webdriver.Chrome()

allure_handler?=?AllureAttachmentsHandler(driver)

def?test_login_functionality():

????try:

??????? driver.get("https://example.com/login")

??????? driver.find_element(By.ID,?"username").send_keys("myusername")

??????? driver.find_element(By.ID,?"password").send_keys("mypassword")

??????? driver.find_element(By.ID,?"loginButton").click()

????????assert?"Dashboard" in?driver.title,?"Login failed or did not redirect to the Dashboard."

????

??? except?Exception as?e:

??????? allure_handler.attach_selenium_screenshot()

??????? allure_handler.attach_selenium_logs()

????????raise?e

????

????finally:

??????? driver.quit(

For Selenium-based projects, browser console log files, a screenshot of the app, cookies, and network data are all captured and attached to the reports!

Running the Handler with Playwright

Setup:

• Include?commons_lib?(the name of our shared library) in your project.
• Set up the Playwright environment and configure the browser context as needed.

Implementation:

• Start tracing before beginning any actions within your test so that all interactions are recorded.
• Conditionally attach the trace file at the end of the test based on the outcome.

Example usage

import?pytest

from?playwright.sync_api?import?sync_playwright

from?commons_lib.allure_attachments_handler?import?AllureAttachmentsHandler

@pytest.fixture

def?browser():

??? with?sync_playwright()?as?p:

??????? browser?=?p.chromium.launch()

??????? yield?browser

??????? browser.close()

def?test_page_interactions(browser):

??? page?=?browser.new_page()

??? page.goto("https://example.com")

??? page.click("#some-button")

??? # Check some conditions

??? if?not?page.query_selector("#expected-element"):

??????? allure_handler?=?AllureAttachmentsHandler(page.context)

??????? allure_handler.attach_playwright_tracefile()

??? page.close()

For Playwright projects, a trace file is attached when a test fails. This file helps us immensely in debugging and troubleshooting test failures.

Benefits of using the Allure Attachments Handler

The introduction of the Allure Attachments Handler has greatly improved both the efficiency and clarity of our test reporting.

• Enhanced debugging:?By systematically capturing detailed artifacts such as screenshots, log files, and trace files, developers can quickly pinpoint the root causes of failures without the need to replicate test scenarios manually.
• Improved test transparency:?Allure reports enriched with comprehensive artifacts provide a clearer picture of test scenarios and outcomes, making it easier for team members and stakeholders to understand what went wrong and why.
• Resource efficiency:?The Handler's conditional attachment feature ensures that resources are used efficiently, attaching detailed artifacts only when tests fail, which helps in managing storage and processing power more effectively.
• Standardized reporting:?With a standardized approach to attaching test artifacts, the consistency of reports across different projects and teams is greatly improved, facilitating better cross-team collaboration and knowledge sharing.

Looking back and ahead

A single solution for test artifacts

The Allure Attachments Handler has revolutionized our test automation processes by providing a unified, automated solution for capturing and reporting test artifacts across both Selenium and Playwright projects. This tool has effectively tackled the issues of inconsistent artifact capture, labor-intensive manual efforts, and incomplete debugging information. What we have now: enhanced debugging capabilities, improved test transparency, and standardized reporting practices throughout our organization.

Greater transparency and informed decision-making?

Effective reporting and logging are fundamental to the success of test automation. Clear and detailed reporting gives stakeholders the power and all the essential information they need, to understand test outcomes. This facilitates informed decision-making and timely intervention when issues arise. Comprehensive logging offers the detailed context necessary to diagnose and troubleshoot problems efficiently, reducing the time spent on debugging and ultimately enhancing software reliability. These practices ensure that everyone involved has a clear view of the testing process and can address any potential issues promptly and effectively.

A moderate, thoughtful investment yields huge reward

By efficiently serving both Selenium and Playwright frameworks, the Allure Attachments Handler exemplifies how strategic tooling can lead to substantial improvements in software development workflows. It guarantees that comprehensive logging and diagnostic information are consistently captured and reported, supporting robust testing practices and fostering a culture of continuous improvement. This tool has not only streamlined our debugging process but also elevated the overall quality assurance practices within our projects, underscoring the critical role that effective reporting and logging play in achieving high-quality software development.

For more information

• Smart Test Cases with Allure TestOps , George I
• Testing Ask Me Anything - Test Reporting , Elizabeth Zagroba
• Snapshots Through Time - How We Evolved Screenshot Testing at Our Company , Alyssa Burlton

"As Head of Testing, I am responsible for the professional development of my people and have MoT Team Membership to support this. I want to have a good atmosphere and up-to-date information for the exchange. The TestBash conferences and the software testing content are excellent sources for this. Great speakers and the latest content and news from the testing community!" - Sven Schirmer

Support your testing and quality team with Ministry of Testing Professional Membership

"Ministry of Testing is the one-stop shop for everything testing." - Ashutosh Mishra

Discover what's happening in the testing community every week by subscribing to the Ministry of Testing newsletter