How to integrate your automated tests pipeline with Jira and Xray?

A blueprint for complete visibility of your tests results, right from Jira.

15 min. read

Let’s say that you are committed to automated tests and have also built a fluent CI (Continuous Integration) pipeline. Each commit triggers the pipeline and runs a set of automated tests. However, your development cycle starts in Jira, and you would like to close the loop in Jira too. Hence, you want that the automated test cycles will be reported into Jira, providing greater visibility into the status of development. Moreover, in situations where evidence of successful testing is required; i.e. from compliance reasons, it would be easier to tie all the traceability together if test results will be logged in Jira.

I will describe here how to design this last step of the integration pipeline, ensuring the development and testing loop is fully documented in Jira.

Our automated tests’ tools and key assumptions

The technical details here assume we are running on Jira Cloud. Same output can be achieved for Jira Server or Data Center but the technical implementation will be different. We are also assuming that Xray for Jira is the test management App in use. Xray is one of the most popular test management tools on Jira and over the years has been my personal favorite. Beyond these two key assumptions, our technical description does not restrict or assume a specific CI tool.

All the modern CI tools can support programmed/scripted pipeline steps which will allow you to achieve what is described here.

Let’s call the last leg of the pipeline, our test result importer, or importer as shorthand.

Setup and terminology

1. Jira Cloud instance installed with XRay, and a project is configured to include the XRAY issue types: Tests, Test Executions, etc.

• Note: it is possible to have two separate projects: one for Test Cases, and second one for Test Executions. This can be useful if you are importing automated tests results with high frequency – which generates many test execution issues and hence there might be a desire to separate them to a different project (to keep this clutter separate from the main project). We will call these: TESTPROJ , and TEXPROJ – although in reality they can be exactly the same Jira project.

1. Test (sometimes called “test cases”) can be executed many times. Each execution is captured in an issue of type: Test Execution. For example:

• TESTPROJ-1 test has been executed three times:

1. As part of TEXPROJ-200 (25-12-2020, PASS)
2. As part of TEXPROJ-201 (26-12-2020, FAIL)
3. As part of TEXTPROJ-202 (27-12-2020, PASS)

• If you are practicing BDD (Behaviour Driven Development) perspective, a Test would likely be mapped to a Scenario. Xray has native support for Gherkin tests and these may also be imported as part of the pipeline.

1. A single Test Execution is one series of running one or many automated tests. A single Test Execution issue will be linked to all the automated tests that it executed, like this:

• TEXPROJ-200 included two test cases:

1. TESTPROJ-1 (PASS)
2. TESTPROJ-2 (PASS)
3. A Test Run is the event (and result) of running a single test as part of a specific execution. So- TESTPROJ-1 has three related Test Runs, and TEXTPROJ-200 includes two Test Runs.
4. From Jira structure point of view: Xray Tests and Xray Test Executions are regular Jira issues (with some extra fields and capabilities). A “Test Run” is a “virtual issue” which is not recognised or know to Jira. To access Test Runs you have to use the Xray API.
5. There exists a unique “Cross Platform Test ID” that identifies each test and which can be used to link between test report information to Jira issues. We call it “Cross Platform Test ID” because it is known to both Jira and the automated tests suite, and exists in the report which is imported to Jira. See in following section more details.

One nice feature of Xray, when it deals with Gherkin tests, is that the scenario description is copied from the Test Issue to the Test Run when the Test Run is created. This means that even when the scenario changes in the future, the Test Run always contains the applicable automated tests scenario version.

The “Cross Platform Test ID”

As mentioned above this is the ID which is used to identify a test (a test case) between it’s Jira issue and the code representation of the test (ie- as a scenario). We propose to implement this in the following way:

1. Each scenario or test method will be labeled in the code with a unique ID, which we will call here “Cross Platform Test ID”. Each test case ort scenario needs to be annotated, in the source code, with this unique ID. As part of the design of the CI pipeline, you need to decide how to generate these IDs.
2. It is important that this “Cross Platform Test ID” will be existing in the Test Report file (which is then imported to Jira) or in other way available to the importer.
3. A Jira field is configured to contain this “Cross Platform Test ID” information (simple short text field is probably OK)
4. When a Test issue is created in Jira (whether manually by a user, or more typically- automatically by the importer) the “Cross Platform Test ID” is set to link it to the related test or scenario in the code.

Other implementations could be problematic:

• Use of the scenario name – this is problematic because the name might change and then will require manual adjustment
• Use of Jira issue key – is problematic because cannot be put in GIT before we actually create the Jira issue, and also- Jira key can change (ie: if we move the issue to another project)

The import flow

The following steps describe in high level the sequence of steps taken by a single execution of the importer. We assume that the importer has a single test report to import, and that all the information it needs is in this report. The sequence might be tweaked if you have other CI sequences (like if you use multiple reports as inputs or if you want to add many reports top a single execution). We want to achieve the following:

1. The report will be imported into a new single Test Execution issue. All the results in this report will be included in this single Test Execution. No other results (ie from previous executions) will be included there.
2. Each of the executed scenarios within this report will be recorded as a single Test Run within that Execution.
3. Each of the Test Runs will be related to the correct Test issue. If a Test issue does not exist then a New Test issue will be created “on the fly” and will be related to this Test run.

Note: What specific data bits are imported to which data fields is not described here- refer to other section for overview of data fields mappings options.

Process steps:

1. Parse the test report file and load it into an internal data model (in the importer).
2. Loop over all tests:

• Note: we use issueid and not issuekey for referencing Jira issues, because the Xray API works with issueids. An issueid is an internal ID which is not seen from the Jira UI.

1. Check if the test already exists in Jira, using the “Cross Platform Test ID” to determine this (see the API Examples section):

• If the test exists, keep the issueid of the test (you will need this for later).
• If the tests does not exist, create it and retrieve the issueid from the result (you will need this for later).
• By the end of this step all your “Cross Platform Test ID”s are mapped into a Jira issueid of the corresponding test.
• Note: this step can potentially be optimised by using one search to find all existing tests (instead of querying one by one), but this might run into jql string limitations and other problems, so suggest to avoid for now. Can be considered later if communication time becomes an issue. Alternatively you could also query for all tests in a project- but this may run into other limitations.

1. Update the test fields according to the data in the input file. Tests may be updated from time to time and the data in Jira needs to reflect the current data that arrives from the pipeline (see the API Examples section).

• It is preferred that fields are only updated when they have actually been changed (do not update for each import because it will create noise in your history in Jira).

1. Create the Test Executions, including already all required test cases (see the API Examples section)
2. Update the Testrun results, comments→

• Get the Testruns ids from the Test Execution you just created (see the API Examples section)
• For each Testrun: 

1. Update status (see the API Examples section)
2. Update other fields as needed.

API Examples

Authentication

Both Jira and Xray apis require an api key to authenticate.

Jira rest api: obtain the api key from your Atlassian user profile (under Security). Reference: https://confluence.atlassian.com/cloud/api-tokens-938839638.html

Xray rest api: obtain the api key from within Jira → Apps → Xray → API Keys. Then use this to create a Bearer token which you use in the api calls. Reference: https://docs.getxray.app/display/XRAYCLOUD/Authentication+-+REST+v2

Jira Rest Api Examples

Search issue with specific “Cross Platform Test ID”

This search is achieved using Jira’s rest API.

The JQL search it uses is this:  project = XT AND “Cross Platform Test ID” ~ “123555aaabb”

curl --request GET \
  --url 'https://YOUR-SITE.atlassian.net/rest/api/3/search?jql=JQL-QUERY' \
  --header 'Authorization: Basic XXXXXXXXXX=' \
  --header 'Content-Type: application/json'

For the rest of the code snippets go to: https://www.radbee.com/how-to-integrate-your-automated-tests-pipeline-with-jira-and-xray/