X-ray test management for jira

Xray is built by Xpand IT, a leader in IT services and digital transformation. Over 4,000 customers trust Xray to test their mission-critical code.

Xray for jira is a complete, full-featured Atlassian verified test management add-on for Jira. Xray supports both manual and automated tests and a complete testing life cycle: test planning, test designing, and test execution to help developers improve the quality of their systems.

Xray for Jira is an add-on developed by Xpand as I mentioned before, an Atlassian Solution Partner.

On this article:

• Xray Data Import
• Xray Dimensions
• Custom Dimensions
• Xray Measures
• Xray Calculated Fields
• Properties
• Private eazyBI
• Xray Report Examples
• Overall Xray Test Run Results
• Created and Executed by Test and Test Set
• Test Status History Report
• Top Defects Report
• Xray Data Import
• Xray Dimensions
• Custom Dimensions
• Xray Measures
• Xray Calculated Fields
• Properties
• Xray Report Examples
• Overall Xray Test Run Results
• Created and Executed by Test and Test Set
• Test Status History Report
• Top Defects Report

Xray Data Import

Xray import option is available for Jira server version and currently is not supported for Jira Cloud version.

If you have installed Xray add-on for Jira Server then in Jira import options you will see Import Xray option.

Xray Dimensions

After the import the following additional dimension will be added:

Xray Requirement -- >Requirement issues which are linked to test cases. Will be imported all issues with types selected in "Requirement Issue Types" and with link type "Tests". The dimension organized in Project and Requirement levels.

Xray Test Plan --> Test plan issues which have assigned at least one test or executions with tests. The dimension is organized by default hierarchy (in Project and Plan levels) or by Test Plan boar hierarchy (in Project, Plan, Folder and Subfolder).

Xray Test --> Tests issues; all created issues with Xray specific issue type "Test". The dimension is organized by default hierarchy (in Project and Test levels) or by Test repository hierarchy (in Project, Folder, Subfolder and Test).

Xray Test Set -->Test set issues which have at least one test. This dimension is organized in Project and Set levels.

Xray Test Pre-condition --> Test pre-condition issues. This dimension is organized in Project and Pre-condition levels.

Xray Test Execution --> Test execution issues which have assigned at least on test. This dimension is organized in Project and Execution levels.

Xray Defect --> Defect issues added during testing. This dimension is organized in Project and Defect levels.

Xray Test Run Assignee -->Users who are assigned to a test runs.

Xray Test Executed by --> Users who executed tests.

Xray Test Execution Status ---> Test execution statuses – by default there are TODO, PASS, EXECUTING, FAIL and ABORTED statuses, but also there can be other user-defined statuses.

Xray Pre-condition Type --> Types of pre-conditions - Manual, Cucumber.

Xray Test Type---> Types of tests (Manual, Cucumber, Generic, etc). The dimension is organized in Type and Scenario Type levels.

Test Environments -->Test environment where tests are executed or scheduled for execution.

Revision --> System revision for test execution.

Custom Dimensions

If you do not find Xray custom field Test Environments in import options, you may import Xray custom field Test Environments using Advanced settings for custom fields. Use following custom field definition example (replacing NNNNN with a corresponding field ID in Jira):

[jira.customfield_NNNNN]
data_type = "string"
dimension = true
multiple_values = true
name = "Xray Test Environments"
dimension_annotations = {group = "Xray"}

Xray Measures

The following measures will be imported. These measures can be analyzed by the mentioned Xray specific dimensions as well as by Issue dimension (where individual tests executions are registered as issues with "Test Execution" type and tests as issues with "Test" type) and by other standard dimension members which correspond to the issue. Note that, measures used in the report defines the context of standard dimensions. For example:

• When using execution related measures, like, Xray Tests executed, Xray Test Runs, Xray Tests defect count, then standard dimension Status is applied to Xray Test Execution as execution is the core to gather information about executed tests, execution outcomes, fix versions, found defects, executors, etc.
• When using test related measures, like, Xray Tests created, then standard dimension Status is related to Xray Test.

Xray Tests created ---> How many tests are created. Count of issues with type "Test".On the Time dimension data are grouped by the issue creation date of test.

Xray Test Executions created --> How many test executions are created. Count of issues with type "Test Execution". On the Time dimension data are grouped by the issue creation date of test execution.

Xray Tests scheduled --> How many tests are scheduled within selected dimensions but not yet executed (test has no dates for execution "Started On" and "Finished On"). If a test is scheduled for two separate test executions, then eazyBI count it as one test. On the Time dimension data are grouped by the date when test is added to the execution.

Xray Tests executing --> How many tests are executing within selected dimensions (test has execution "Started On" date and no "Finished On" date). If a test is executing in two separate test executions, then eazyBI count it as one test. On the Time dimension data are grouped by the date when test execution (run) is started.

Xray Tests executed -> How many tests are executed within selected dimensions (test has both execution dates "Started On" and "Finished On"). If a tests is executed in several test executions, then eazyBI count it as one test. On the Time dimension data are grouped by the date when test execution (run) is finished.

Xray Tests defect count --> How many defects are registered for a test execution (including individual test steps). Unique count of different defects is returned.

Xray Tests with executions --> How many tests have at least one test execution (scheduled, in progress or completed) within selected dimensions. If a tests has several test executions, then eazyBI count it as one test.

Xray Test Runs --> How many times tests have been executed, are executing or scheduled for execution.

Xray Calculated Fields

Also following calculated members will be created.

Xray Test last executed date -->The last execution date for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension.

Xray Test last executed by --> The last executor for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension.

Xray Test last execution status --> The last execution status for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension.

Xray Tests executed first date --> Date and time for first executed test within selected dimensions. 

• If a test run has Finished date, then return the timestamp when it is finished.
• If a test run does not have Finished date but has Started date, then return timestamp when it is started.

Xray Tests executed last date -->Date and time for last executed test within selected dimensions. 

• If a test run has Finished date, then return the timestamp when it is finished.
• If a test run does not have Finished date but has Started date, then return timestamp when it is started.

Xray average Test Run duration --> How much time (hours and minutes) it took on average to execute a test.

Xray Tests execution summary --> Test execution statuses within selected dimensions.Xray Test Execution Begin DateBegin date of a test execution issue.This is an execution property and should be used together with Xray Test Execution dimension.

Xray Test Execution End Date -->End date of a test execution issue.This is an execution property and should be used together with Xray Test Execution dimension.

Xray Test Execution Revision -->Current revision assigned to a tests execution issue. This is an execution property and should be used together with Xray Test Execution dimension.

Xray Test Execution Test Environments -->Current test environment assigned to a tests execution issue.This is an execution property and should be used together with Xray Test Execution dimension.

Xray Execution Test Plans -->Key of test plan to which a test execution issue is assigned. This is an execution property and should be used together with Xray Test Execution dimension.

Xray Test Test Plans --> List of test plans in which test is executed or scheduled for execution.This is a test property and should be used together with Xray Test dimension.

Xray Test Test Type --->Current test type of a test issue.This is a test property and should be used together with Xray Test dimension.

Xray Test Cucumber Test Type --> Current Cucumber tests type of a test issue if it has a test type "Cucumber".This is a test property and should be used together with Xray Test dimension.

Xray Test Plan Begin Date -->Begin date of a test plan issue .This is a test plan property and should be used together with Xray Test Plan dimension.

Xray Test Plan End Date -->End date of a tests plan issue.This is a test plan property and should be used together with Xray Test Plan dimension.

Xray Defect status --> Current status of a defect issue.This is a defect property and should be used together with Xray Defect dimension.

Xray Defect created --> Creation date and time of a defect issue.This is a defect property and should be used together with Xray Defect dimension.

Xray Defect assignee -->Current status of a defect issue.This is a defect property and should be used together with Xray Defect dimension.

Properties

Xray tests, test executions as well as defects, requirements and test plans are Jira issues and also represented in Issue dimension. You can define calculated measure to get any issue property which is imported in eazyBI also for Xray specific dimensions. To access issue property use function get().

For example, to get priority for Xray Defect, copy a formula of issue property Issue Priority and replace dimension name [Issue] with [Xray Defect] like this:

[Priority].[Priority].getMemberNameByKey(
  [Xray Defect].CurrentHierarchyMember.get('Priority ID')
)

Private eazyBI

You can import Xray custom fields also when using Private eazyBI. Add the following settings in the configuration file config/eazybi.toml configuration file to enable Xray data import. There are more details on the needed configuration for data from Jira apps.

Xray Report Examples

Overall Xray Test Run Results

You can create a simple test execution report to quickly see the overall snapshot of your test execution results. You can specify additional filters, for example Xray Test Plan, Test Environment, Test Set, Time period, or other Xray dimensions.

Report definition

{
  "cube_name": "Issues",
  "cube_reports": [ {
     "name": "Sample Xray Overall Test Run results",
     "folder_name": "Samples Xray",
     "result_view": "pie_chart",
     "definition": {"columns":{"dimensions":[{"name":"Measures","selected_set":["[Measures].[Xray Test Runs]"]}]},"rows":{"dimensions":[{"name":"Xray Test Execution Status","selected_set":["[Xray Test Execution Status].[Status].Members"]}]},"pages":{"dimensions":[{"name":"Xray Test Plan","selected_set":["[Xray Test Plan].DefaultMember"]}]},"options":{"nonempty":true},"view":{"current":"pie_chart","maximized":false,"pie_chart":{"data_labels":"values_percentage","donut":false,"relative_size":false,"show_labels":true,"show_legend":true,"swap_axes":false,"series_options":{"PASS":{"color":"#95C160"},"TODO":{"color":"#A2A6AE"},"EXECUTING":{"color":"#F1E069"},"FAIL":{"color":"#D45D52"},"ABORTED":{"color":"#111111"}}},"table":null}}
  } ],
  "calculated_members": []
}

Created and Executed by Test and Test Set

Analyze created and executed Xray tests by project and test set. This will give you a full overview of test execution status across all your project portfolio.

Report definition

{
  "cube_name": "Issues",
  "cube_reports": [ {
     "name": "Sample Xray Created and Executed by Test and Test Set",
     "folder_name": "Samples Xray",
     "result_view": "bar_chart",
     "definition": {"columns":{"dimensions":[{"name":"Measures","selected_set":["[Measures].[Xray Tests created]","[Measures].[Xray Tests executed]"]}]},"rows":{"dimensions":[{"name":"Xray Test","selected_set":["[Xray Test].[Project].Members"]},{"name":"Xray Test Set","selected_set":["[Xray Test Set].[Set].Members"]}],"nonempty_crossjoin":true},"pages":{},"options":{},"view":{"current":"bar_chart","maximized":false,"bar_chart":{"stacked":false,"vertical":false,"swap_axes":false,"data_labels":false,"series_options":{"Xray Tests created":{"color":"#4572A7"},"Xray Tests executed":{"color":"#89A54E"}},"y_axis":{"0":{"step":"1"}}},"table":null}}
  } ],
  "calculated_members": []
}

Test Status History Report

Detailed information about each test: which tests have been scheduled, executed, or completed most often; how many times and how long each test has been run; what statuses have been returned; which tests have been failing most often.

Report definition

{
  "cube_name": "Issues",
  "cube_reports": [ {
     "name": "Sample Xray Test Status History Report",
     "folder_name": "Samples Xray",
     "result_view": "table",
     "definition": {"columns":{"dimensions":[{"name":"Measures","selected_set":["[Measures].[Xray Tests scheduled]","[Measures].[Xray Tests executing]","[Measures].[Xray Tests executed]","[Measures].[Xray Test Runs]","[Measures].[Xray average Test Run duration]","[Measures].[Xray Tests execution summary]"],"members":[]}]},"rows":{"dimensions":[{"name":"Xray Test","selected_set":["[Xray Test].[Test].Members"],"members":[],"bookmarked_members":[]}],"nonempty_crossjoin":true},"pages":{"dimensions":[{"name":"Project","selected_set":["[Project].DefaultMember"],"members":[{"depth":0,"name":"All Projects","full_name":"[Project].[All Projects]","drillable":true,"type":"all"}],"bookmarked_members":[],"current_page_members":["[Project].[All Projects]"]},{"name":"Fix Version","selected_set":["[Fix Version].DefaultMember"],"members":[{"depth":0,"name":"All Fix Versions","full_name":"[Fix Version].[All Fix Versions]","drillable":true,"type":"all"}],"bookmarked_members":[],"current_page_members":["[Fix Version].[All Fix Versions]"]},{"name":"Xray Test Execution Status","selected_set":["[Xray Test Execution Status].DefaultMember"],"members":[{"depth":0,"name":"All Xray Test Execution Statuses","full_name":"[Xray Test Execution Status].[All Xray Test Execution Statuses]","drillable":true,"type":"all"}],"bookmarked_members":[],"current_page_members":["[Xray Test Execution Status].[All Xray Test Execution Statuses]"]}]},"options":{},"view":{"current":"table","maximized":false,"table":{"row_dimension_headers":{"Xray Test":true}}}}
  } ],
  "calculated_members": [{"dimension":"Measures","name":"Xray average Test Run duration","format_string":"MinutesFormatter","formula":"[Measures].[Xray Test Run duration] / 60","annotations":{"group":"Xray","drill_through_dimension_levels":"[Xray Test].[Test], [Xray Test Execution].[Execution]","predefined":true}},{"dimension":"Measures","name":"Xray Tests execution summary","format_string":"","formula":"Generate(\n  Filter(\n    Descendants([Xray Test Execution Status].CurrentMember,\n      [Xray Test Execution Status].[Status]\n    ),\n    [Measures].[Xray Tests executed]\n    + [Measures].[Xray Tests scheduled]\n    + [Measures].[Xray Tests executing] \u003e 0\n  ),\n  [Xray Test Execution Status].CurrentMember.Name || ' ' ||\n  Format([Measures].[Xray Tests executed]\n         + [Measures].[Xray Tests scheduled]\n         + [Measures].[Xray Tests executing], '#'),\n  ', '\n)","annotations":{"group":"Xray","drill_through_dimension_levels":"[Xray Test].[Test], [Xray Test Execution].[Execution]","predefined":true}}]
}

Top Defects Report

List all distinct tests who executed with defects. Find out what defects are captured in each version and how many tests does that affect?

Report definition

{
  "cube_name": "Issues",
  "cube_reports": [ {
     "name": "Sample Xray Top Defects",
     "folder_name": "Samples Xray",
     "result_view": "table",
     "definition": {"columns":{"dimensions":[{"name":"Measures","selected_set":["[Measures].[Xray Defect status]","[Measures].[Xray Defect created]","[Measures].[Xray Defect assignee]","[Measures].[Xray Tests with executions]"],"members":[]}]},"rows":{"dimensions":[{"name":"Xray Defect","selected_set":["[Xray Defect].[Defect].Members"],"members":[],"bookmarked_members":[]}],"order_by":{"expression":["[Measures].[Xray Tests with executions]"],"order":"DESC"},"filter_by":{"conditions":[{"expression":["[Measures].[Xray Tests with executions]"],"operator":"\u003e","value":"0"}]},"nonempty_crossjoin":true},"pages":{"dimensions":[{"name":"Fix Version","selected_set":["[Fix Version].DefaultMember"],"members":[{"depth":0,"name":"All Fix Versions","full_name":"[Fix Version].[All Fix Versions]","drillable":true,"type":"all"}],"bookmarked_members":[],"current_page_members":["[Fix Version].[All Fix Versions]"]},{"name":"Test Environments","selected_set":["[Test Environments].DefaultMember"],"members":[{"depth":0,"name":"All Test Environments","full_name":"[Test Environments].[All Test Environments]","drillable":true,"type":"all"}],"bookmarked_members":[],"current_page_members":["[Test Environments].[All Test Environments]"]}]},"options":{},"view":{"current":"table","maximized":false,"table":{"row_dimension_headers":{"Xray Defect":true}}}}
  } ],
  "calculated_members": [{"dimension":"Measures","name":"Xray Defect status","format_string":"","formula":"[Status].[Status].getMemberNameByKey(\n  [Xray Defect].CurrentHierarchyMember.get('Status ID')\n)","annotations":{"group":"Xray","drill_through_dimension_levels":"[Xray Test].[Test], [Xray Test Execution].[Execution]","predefined":true}},{"dimension":"Measures","name":"Xray Defect created","format_string":"yyyy-mm-dd hh:nn:ss","formula":"[Xray Defect].CurrentHierarchyMember.get('Created at')","annotations":{"group":"Xray","drill_through_dimension_levels":"[Xray Test].[Test], [Xray Test Execution].[Execution]","predefined":true}},{"dimension":"Measures","name":"Xray Defect assignee","format_string":"","formula":"[Assignee].[User].getMemberNameByKey(\n  [Xray Defect].CurrentHierarchyMember.get('Assignee name')\n)","annotations":{"group":"Xray","drill_through_dimension_levels":"[Xray Test].[Test], [Xray Test Execution].[Execution]","predefined":true}}]
}