The Pros and Cons of Different Testing Tools - XRay

We started the year looking at JIRA, but before I look at other similar tools, I wanted to spend some time looking at some added extensions to JIRA that can add a lot of value to JIRA and help make it a more expansive set of tools. And the first extension I want to look at is XRay – a test management extension for JIRA.

XRay was initially developed by the software consulting company XpandIT as a way of adding value to their testing – built around the Atlassian JIRA eco-system and has since become a company and product all on its own, under the Atlassian umbrella.

Whereas JIRA is fantastic at tracking work tasks and issues of different types – it doesn’t do a good job by default in tracking testing activities. So, even if you customize it to issue types that can work for testing, you won’t get ideal usage out of it as it tries to fit testing into the idea of agile workflow items when in reality, tests are repeatable processes that are executed across sprints and projects.

Reporting is another specific area where a lot of the default JIRA dashboards just don’t provide you with the right amount of detail and would often leave teams needing to push data into another visualizing or reporting tool (like PowerBI, Grafana or even Excel) to try and provide accurate reporting on the testing effort. Something that takes a considerable amount of unnecessary time.

And lastly is the idea of traceability and linking a requirement (which is not the same as a user story) to a user story, development task, testing item, and possible issues. I’ve written about traceability before and its usefulness, but essentially it is the ability to link work items together and allow teams to make data-driven decisions by knowing the importance of certain development or testing tasks when it comes to prioritizing issues, requirements, or other work. A vital component of an effective team.

So, even though JIRA is highly customizable and can be configured for a large number of different uses, it doesn’t quite work as a test management tool even with all of its extensibility and so if you want to make full use of it as a tool and have it drive your end-to-end workflow – having a test management plug-in like Xray is essential.

Below is a list of features that XRay brings to JIRA:

Test Case Management: XRay allows users to create and manage test cases directly within JIRA. It supports the definition of test steps, expected results, and other relevant information.

BDD (Behavior-Driven Development) Support: XRay supports BDD by allowing the creation of tests using Gherkin syntax. This enables teams to write tests in a human-readable format, promoting collaboration between developers, testers, and other stakeholders.

Test Execution: Testers can execute manual and automated tests and log results within JIRA. XRay provides integration with various test automation frameworks, allowing automated test results to be linked to corresponding JIRA issues.

Traceability: The tool offers traceability features, linking test cases to requirements, user stories, and other JIRA issues. This ensures comprehensive coverage of requirements and facilitates impact analysis.

Reporting: XRay provides reporting capabilities to track test execution progress, generate test coverage reports, and analyze testing results. Customizable dashboards and gadgets are available to monitor key testing metrics.

Integration with CI/CD Tools: XRay integrates with popular CI/CD tools, enabling seamless collaboration between testing and development teams in continuous integration and delivery processes.

Test Sets and Test Plans: XRay supports the organization of test cases into test sets and test plans, providing a structured approach to test management.

Requirements Coverage: XRay helps ensure that all requirements are covered by corresponding test cases, promoting a comprehensive and structured testing approach.

These added features don’t come without some constraints – particularly on the performance side. Though, over the past few years with heavy investment from Atlassian on the cloud front – a lot of the performance issues have been addressed – though if you are running a local instance of JIRA with Xray, you will likely find a few performance issues – especially when trying to work hundreds of different tests across multiple different teams and projects.

Below is a full breakdown of the pros and cons of Xray:

Pros:

• Integration with JIRA: Seamless integration with JIRA makes implementing it into teams already using JIRA a breeze and significantly quicker than other testing tools will do.
• Comprehensive Test Case Management: With a focus on different testing types – manual, automated, and even non-functional, it adds a lot of usefulness to JIRA. Tracking these across different projects and sprints, as it manages these separately to the work delivery – unlike a few other JIRA extensions that try and do the same thing. ??
• Traceability: Offers traceability features to link test cases to requirements, user stories, and other JIRA issues, ensuring comprehensive coverage and visibility into testing efforts.
• BDD Support: XRay’s support of BDD allows the creation of tests using Gherkin syntax and can save considerable time for teams using a BDD approach to their test automation design. ?
• Test Execution and Logging: XRay offers extensive tracking of all testing activity providing information needed to help with the test reproduction or debugging effort -? or even trying to identify the effectiveness of different tests.
• Integration with CI/CD Tools: Integration with popular Continuous Integration/Continuous Delivery (CI/CD) tools allows is quite seamless for automated test execution and reporting in CI/CD pipelines. Essentially anything that works for JIRA works for XRay and JIRA has one of the widest sets of integrations of software development tools in the industry
• Customization: Unlike many other similar testing tools that can operate rigidly, XRay offers flexibility in customizing test case fields, workflows, and other aspects to adapt to the specific needs of different teams and projects.
• Reporting and Metrics: On top of JIRA’s default dashboards, XRay provides reporting capabilities and customizable dashboards for tracking test execution progress, analyzing results, and monitoring key testing metrics.
• Cost: I mention this in the cons section as well, because it is an added cost if you just use basic JIRA. However, while that may deter some companies from using it, it is still significantly cheaper than most other test management tools and so still offers a lot of value for money.??

Cons:

• Learning Curve: Users unfamiliar with XRay and JIRA may experience a learning curve, especially if they are new to test management tools or agile methodologies.
• Cost: Depending on the licensing model and the specific needs of your team, the cost of using XRay may be a consideration, especially for smaller teams or organizations with budget constraints. It is an added expense and its usage needs to be factored into your JIRA costing.
• Dependency on JIRA: While integration with JIRA is a strength, teams heavily reliant on JIRA may find themselves dependent on its features and limitations. If you plan to move away from JIRA at any point in time, you may need to reconsider XRay.
• Continuous Improvement: While XRay is regularly updated with new features and improvements, the pace of updates and the response to user feedback may vary from typical JIRA updates.
• Performance: Depending on the size and complexity of your JIRA instance and the volume of test data, there could be performance considerations.

When to use XRay:

• If you are already using JIRA and need a comprehensive test management tool
• You need added traceability across your testing effort beyond your usual user stories
• You use BDD extensively and need a test management tool that easily works with this testing approach.

Despite its excellent customisability, JIRA does have its limitations when trying to modify it for other purposes and testing is one of those where an additional extension like XRay can add a lot of value. If teams are already embedded into JIRA to track their workflows, then this is certainly an extension worth considering if you can afford it.