Basics of TestRail for QA | Study Notes #2

As I was looking at job openings for QA positions, I’ve come across the name “Testrail” multiple times as a common bug tracking tool used by companies, so I thought it would be useful to get myself familiar with it, so I looked on their website and found a course which works as a perfect follow-up to the QA lessons I studied before, I decided to to take note mostly of QA notions that aren't super specific to this tool and could be relevant wherever you apply them.

So let’s start!

Key Terms

Test Case

In TestRail, a test case consists of:

• A summary of what you need to test
• A description of how to run your test, including prerequisites, steps, and expected results
• Additional information to help you prioritize, group, and track your test case like priority, type, and linked requirements

Test cases are the basic building blocks in TestRail and are contained inside projects. They can be reused in multiple test runs over time.

Write test cases so that each one can be verified by a single tester in a short period or corresponds to a single test in your automation suite. Each test case should verify a requirement, document a task, or confirm some functionality is working as intended

Sections and Subsections

In TestRail, you can organize your test case repository into sections and subsections. Sections and subsections are like folders for your test cases, allowing you to organize your

test cases in a hierarchical tree structure.

How you organize your sections depends on the size of your project. If you have a lot of test cases for a project module, it is recommended to break down your sections further and create subsections for specific project modules, areas of functionality, or processes within your application.

Test Suite

In TestRail, you will only find test suites when opting for the multiple test suites project type to manage your test cases.

Test suites give you an additional layer of separation between groups of test cases. Each test suite is independent and not related to other test suites within your project.

You can use sections and subsections to further organize test cases within test suites, like in a single repository-type project.

Tests

When a test case is selected to be part of a test run, TestRail creates a test with a unique ID for that specific test run.

That test can then be assigned to a team member, who will add the test results and

set a status after all the testing steps described in the test case are completed.

Test Run

You will most likely start a new test run every time you test a new version or update of your application. You will also test many of your test cases across multiple test runs over time.

A test run consists of individual tests for each test case that you add, and it represents what you actually plan to test as part of a specific test cycle, sprint, or milestone.

When you create a test run, you can choose to include all cases of a test suite, section, or subsection, select a specific set of test cases, or set a dynamic filter.

Whereas test cases, sections, subsections, and suites document how to test your application, a test run is an actual set of tests you are conducting.

Results

After a test is executed, the tester will compare the actual outcome to the expected results and record their findings as a test result.?

Test results can include:

● Status (i.e. Passed, Failed, Blocked)

● Comments

● Images or files like screenshots or logs

● Elapsed time

● Linked defects

● Other custom fields

Embedding images to the test results can be very useful in displaying additional details as part of a test step or expected result.

Test Plan

When you need to manage multiple test runs for a single release or milestone, a test plan comes in handy.

Test plans allow you to create multiple test runs at once and group them for easier tracking. Test plans also allow you to generate identical test runs for various configurations automatically.

A configuration can be anything you need to test your projects against, such as different operating systems or web browsers.

Milestone

A milestone can represent any project milestone you deem to be important, such as a planned software release, phase of a project, development sprint.

You can define a start and end date, add a description and attachments, and mark once a milestone has been completed.

Once you have added your milestones to TestRail, you can link test plans or test runs to specific milestones so that you can easily track all of their test activities and results in a single dashboard.

Milestones help you align testing cycles with external development goals and timelines.

They can also help if you have many active tests running at the same time.

Reference

After integration, you can copy and paste a URL or ID into the References field to quickly access the resource as a reference during testing and for coverage-tracking purposes.

Reference is the generic terminology TestRail uses to link to an item in an external system that reflects what the test case is designed to validate, like a requirement or user story.

You can use the References field to document a specific requirement number, a string of text or URL, or the unique ID for an entity stored in an external tool like an issue tracker, product planning tool, or other agile project management tool.

TestRail integrates with many common requirements managers or issue trackers like Jira, Azure Boards, Asana, and others to make it easy to reference user stories, product requirements, and tasks in your test cases, runs, plans, and milestones

Defect

A defect is when a tester runs into a condition which does not allow the test to be completed satisfactorily.

Then, the tester can use the Defects field to link a defect or bug report to an external bug, defect, or issue tracker.

Or, if you are using one of TestRail’s defect integrations with tools like Jira, GitHub Issues, GitLab Issues, Azure Boards, or others, you can automatically create or push new issues, bugs, or defect reports to your issue tracker from your test results in TestRail.

You can also copy and paste unique IDs for defects or issues that you’ve already created in your external tool if you prefer.

Testing Process

In general, you can break the testing process down into four steps:

1. Design - Analyze requirements and write test cases (for manual and automated tests)

2. Planning - Create Milestones, Test Runs, and Plans with tests that you will run during this test cycle

3. Execution - Begin testing, add test results via the TestRail UI or send test automation results via the TestRail API, and link or generate defect reports

4. Reporting & analysis - Track test progress in real time and generate test coverage and traceability reports to identify risks or gaps in testing

In agile testing, you may iterate through this process with each new sprint or multiple times during a larger release cycle.

If you use a waterfall testing approach, you may do this entire process once per release.

How to design a Test Case

When developing a product, whether it is a piece of software or any other type of product, tests need to be conducted in order to assure its quality and fit for purpose. Test case design is the process of planning and documenting these tests.

During test case design, you will typically think of:

????What you want to test

• ????How will you cover your requirements or user stories?
• ????Will you conduct functional tests only, or are you testing other aspects?

????How you will test

• ????What are the steps to test a functionality??
• ????What test data will you use?

????What is needed to test

• ????What are the required preconditions to execute a test?

????Where you want to test

• ????What environments are you going to run the test on?
• ????Do you need any particular hardware to run the tests on?

????When you want to test

• ????What is the importance or priority of each test?
• ????In which stages of development will you run each test?

So imagine you have a simple login functionality that you are in charge of testing. The first step is to know what you want to test. When designing test cases, you will usually want to add tests up to a certain level of detail so as not to unnecessarily bloat your test suite unnecessarily. In this case, you may want to test:

• Successful login?
• Failed login with an invalid email
• Failed login with an invalid password
• Multiple failed logins trigger a captcha

After you have your list of test cases, you proceed to how you will test by writing the test steps that will accomplish the verification you want to make for each.?

You must also check what is needed to test, like completing a previous step, specific test data, or other preconditions. Keeping with the login functionality example, if you’re running an end-to-end test, your precondition is that you have a registered user.?

You also have to assess where you want to test, because certain environments may not allow testing this feature due to, i.e., not being fully integrated with other essential systems.?

As for when you want to test, you can use the test case properties to decide what the priority of the test case is. And if this is not enough, TestRail allows configuring custom fields so you can fully customize how you catalog your test case.

What do you do after you have designed test cases?

After you have designed your test cases, you will plan your test cycles using test runs, plans, and milestones, and begin testing!

Important: Test cases and tests are different artifacts in TestRail.?

Test cases describe what, where, and how you will test your application.?

Tests represent a case ready for execution as part of a particular test cycle, and they exist in the context of test runs.

As you start testing, you will upload test results to your tests and update the test status to something like “Passed” or “Failed”.?

Take, for example, a test run where you are testing the test case for your app’s login functionality against two different devices

The first group of data contains results for one test (T7579), executed using a Galaxy S22 device. The second group contains results for a second test (T7833), using an iPhone 14. These are two tests, included in two separate test runs, but originating from the same test case.

Basics of Test Planning

Why Test Planning?

Planning is (or should be) the starting point of any new project or activity, including software testing.?

The type of test planning you do will depend on your team’s development style, the size of your project, and other factors. Teams that work in heavily-regulated industries or have more waterfall-style development approaches may need to develop robust, detailed test plans, while fast-paced agile teams tend to cut down formal planning and documentation.

No matter how you test, having a test plan can help prioritize the right tests, coordinate efficiently, and maximize the value of the effort put into testing.

A test plan should map the risks and priorities. It should also give your team an opportunity to think through key questions about the test cycle, like:

1. What needs to be tested?
2. Who is performing the tests?
3. What is the time frame?
4. What kinds of testing will you do?

With that map on the table, testers can find defects as early as possible, providing information to the development team and speeding up delivery.

What is the time frame?

Teams are always under pressure, working under tight timelines and with no time to waste. That is why the test plan should consider the time frame.?

You may be deciding on your test plan for a specific sprint. Or maybe you need to develop the test plan for a new feature or release. Your test strategy will be a lot different for a release that happens in two weeks than one in two months.

What kind of testing?

Automated or manual? Structured or Unstructured? Smoke, regression, functional, acceptance… Each type of testing differs slightly from the other and has its own advantages (and disadvantages).

Automation can allow you to speed up repetitive product validation or increase coverage for your regression tests, but developing and maintaining automated tests takes a significant investment of time. Finding the balance between automation and manual testing is the key to get more value from the time and the money spent on testing.?

Analyze your risks and priorities, the intent behind each testing type, and its feasibility, as well as the value that you might get out of it.?

The size of your testing team also affects the testing plan in relation to the kind of testing you choose to use. With more work power, there are more opportunities to diversify your testing.

What are test results?

Test results are the outcome of your testing cycle. In TestRail, each test result can include updates to test statuses and other test fields, comments, and attachments.

After test planning, test cases design, and execution, the result is a collection of data and documents that verifies that the software works as intended and highlights any areas of risk.?

Test results allow the QA team to document and communicate the findings from their testing, and allow project stakeholders to make decisions regarding the functionalities of the software project and release timelines.

Test statuses

Comparing the test results with the expected results allows the tester to assign statuses to the tests.

By default, each test has one of five different statuses that need to be signaled.

• Untested: By default, new tests have the status Untested. Once a test result has been added to a test, it can never receive the Untested status again.
• Passed: A test is marked as Passed when a tester verified the test steps and their testing demonstrates that the functionality of the application matches the expected results.
• Failed: A tester marks a test as Failed if one of the specified test steps resulted in an error or if the expected result differs from the actual test result.
• Retest: If a developer or tester determines that a test should be tested again, they can mark the test as Retest (e.g., if a test originally failed and the developer fixed the problem, the developer can mark it to be retested).
• Blocked: The Blocked status is used to signal that a test cannot be executed currently because of some external dependency (such as a bug that needs to be fixed before being able to complete the test). It is often used in combination with the Retest status.