TMI: Can Your Test Cases and Bug Reports Be Too Detailed?

Everyone Wants Detailed Test Cases and Bug Reports

Ideally, or at least what I glean from job postings recently, companies want testers who can create very detailed test cases and bug reports. As someone who comes from a technical writing background, I can write step-by-step procedures in excruciating detail. Sometimes, your test cases might need that level of detail, but you don’t want to write a Faulkner novel when a Hemingway short story would do.

Let me illustrate what I mean by a detailed test case or a detailed bug report. Basically, both share a similar structure:

• An introduction. In a test case, this includes preconditions, user type/access rights, and perhaps a goal; in a bug, the introduction includes a summary of what is happening.
• Steps to perform the test or steps to reproduce the issue. Each step in the procedure should include what displays next whether to validate it or to simply confirm that the tester is where he or she should be.
• A conclusion that includes the final result of the test or the expected behavior that needs to be corrected.

So a simple test case might be:

• Administrative user has access to the search screen.
• System has records in all three states (incomplete, complete but not submitted, complete and submitted).

1. Go to the test site. The Home page displays.
2. Click the Search link. Search page displays with the search form which includes the following edit boxes: ID, Title, Customer, and Opened Date/Closed Date along with a Search edit box.
3. In the Customer edit box, type International Exports, Ltd. The value displays properly.
4. Click Search. A table of search results displays, including the following columns: ID, Title, Customer, Opened Date, Closed Date, and a column of action buttons (View, Edit, Submit).

The search results match expectations.

A defect has a similar structure; in both, the procedure represents most of the word count. 

Procedures in test cases and bugs can get quite lengthy. I once wrote a test case with 157 steps describing how to change a setting in a green screen application, where a mainframe presents individual prompts one at a time, and how to test that change. The prompt where the tester needed to enter test data appeared several screens down in both the application and the test case management software. I am pretty sure you could win at Zork with fewer steps.

Detailed test cases and defects can have their benefits. You can hand them to an intern, a developer, or off-shore resource who can then walk through the steps without experience in the application under test. However, great detail comes with its own set of risks, and by understanding those risks, you can mitigate them.

The Risks of Too Much Detail

Cut and Paste Errors

If your test team needs to create a large set of test cases, such as individual test cases for individual acceptance criteria in a rigid process with required traceability, the individual test cases will be very similar to one another.

For example, think of a single user story whose acceptance criteria include:

• The page reloads with a table of results from the database.
• The ID number and title display in individual table rows.
• The title displays in red if the record is incomplete.
• The title displays in yellow if the record is complete but not submitted.
• The title displays in green if the record is complete and submitted.

At minimum, a Requirements Traceablility Matrix (RTM) would need at least five test cases for the user story or maybe ten if you need a negative test for each one—or more. With a larger set of acceptance criteria grows, your number of test cases can grow rapidly to check all the boxes you need.

Tester might write a single test case that goes through the steps of logging in, navigating to the search page, and then testing one of the acceptance criteria. Then, the tester might clone or copy the first test case and only edit the steps as needed to cover the next acceptance criteria. Or, in the case of defects, the tester might copy and paste a set of steps from one bug to another and add new steps where the procedures diverge.

If a tester makes an error in the ‘template,’ the original test case that he or she later clones, other copied test cases will include that error. I have seen this happen, and I have done it myself (unfortunately). If the tester uses a cloned copy containing an error as a ‘template’ for other test cases or defects, you can introduce any number of errata in downstream step-by-steps. While errata make valuable collectibles—just check eBay—they make for numerous headaches in trying to interpret problematic test cases or trying to reproduce tricky defects—not to mention the chance of rejection in some external quality audits. Instead of communicating clearly, you’re likely to confuse the next person who needs to use that test case or defect.

Maintaining Accuracy

Even if you or your compatriots do not make an error in the unofficial test case ‘template’ and make bad photocopies of the original photocopy, you might find it difficult to keep a large number of test cases up to date. If the interface changes somewhere that impacts the early steps of a large number of your test cases, you’ll have to go back and update those test cases individually to include the new steps or to reflect changes to prompts/controls/edit box names. In some test case management systems, you can easily insert new steps, but others require almost retyping the whole test case to update early steps.

If you’ve done any work in test automation, you already know the problem of maintaining your automated tests when the GUI or API changes. Detailed manual test cases mirror record-and-playback scripts in this regard—but rest assured, you can use similar strategies to mitigate this risk as I will explain later.

Missing Subtle Differences

When someone else needs to run the test cases or to reproduce defects, he or she might miss the important distinction if the only change is on step 14 of 26 especially when step 14 is the same in most of the test cases or defects.

My own eyes glaze over a bit when dealing with the same material, the same steps over and over again for a couple of hours. I might miss that I should have typed P at the V/P/F (View/Print/File) prompt—especially since P looks a lot like an F in a small font or amid a large screen full of text.

Important information gets lost in the repetition of the basic steps. So how can you mitigate these risks?

Communicating Effectively and Efficiently

You hear a lot about Minimum Viable Product—building just enough software for users to try it out, to see where to improve and add flourish. I’d like to add another concept to your pantheon of laziness, I mean efficiency: Minimum Viable Communication. Your communication should be just enough to convey the important information. Save the flourishes and the repetition for novels and poetry.

The following techniques can help minimize the verbiage needed to convey information in your test cases and your test plans.

Shared Vocabulary

Way back in my technical writing days, we had the Microsoft Manual of Style to guide us when we were writing manuals. Applications took place within the context of a window (well, for a windowed application anyway—work with me here). The user might have several documents open in windows within an application like a word processor; a dialog box appeared over the main window of the application. Every manual used the same terms for list boxes, edit boxes, and buttons.

Web software and design changes faster than the style guides can keep up, but your team has to agree on what to call that spot in the browser window where you make the shapes to call out attention to something someone has created—a canvas? An editing panel? What about the things that slide out from the side with properties of the selected item—a drawer? The slide-out? The panel? To quote what the wise label of a set of radio buttons once said to me, “Pick one.”

Your whole development team needs to define single words or phrases to describe the same thing the same way. You can do this de facto with everyone just getting into the habit or de jure by creating a style guide of your own to standardize terminology for your interface and your documentation.

Shared Shorthand

With a smaller or more closely knit team, such as an early-stage startup or with a product or project with a fairly limited feature set, everyone will probably know what it means to log into the system (you click the Login link to expose the Login panel; you type a username or email address; you type a password; you click Login).

You can capture this in a single step in a test case or bug report as:

1. Log into the system as an administrator.

As I mentioned, this shorthand can work well in certain situations:

• If your test team only includes two or three testers—or maybe just you. 
• If only a couple of developers who wrote the software from the ground up will see your bug report. 
• If the product is new and only has a limited feature set.

However, as your team, product, or organization grows, you might need to better specify what this shorthand means without including the steps in each test case. You can do this by using pointers to shared procedures in each test case.

Pointers to Shared Procedures

In software documentation, you don’t have to spell everything out every time. You don’t tell the user how to log in on every procedure or page. However, you do have to provide that information once and then hyperlink it or include a “For more information on logging into the application, see ‘Logging In.’” That way, users who already know how to log in don’t have to skip those steps and get lost in the ordered list of actions to take. 

Likewise, you can capture basic steps in one test case or one defect report and then include pointers and hyperlinks in subsequent places where the user would have to follow those steps, either in the prerequisites or the introductory text above the steps you’re focusing on now. 

For example, a test case for adding a new user might look like this:

• Administrative user has access to the Users section.

1. Log into the system as an administrator (as described in Test Case 1.3).
2. Go to the Users List page (as described in Test Case 2.2).
3. Click the Add New User link.
4. Add New User form displays. In the Username edit box, type the new username.
5. In the Email Address edit box, type a valid email address.
6. Click Add User.
7. Log out as the administrator (as described in Test Case 1.8).
8. Wait for the email sent to the new user’s email address.
9. Click the link in the email.
10. The Welcome, New User page displays.

Verify that the new user name matches the email address and that the Please set your password and Please type your password again edit boxes display followed by the Set password button.

Using this simple technical writing technique can help you abbreviate your test cases and defect reports so that they focus on the most important information that you need to convey.

This approach works for larger, distributed teams where some of the team members might need the pointers for procedures they’ve not done in applications or parts of the application where they’ve not worked. 

Consistent Communication

If you have a team of testers writing test cases and defect reports, you need to encourage consistency in how the team writes their test cases. Of course, as a sometime technical writer, I like style guides for everything from marketing to graphical user interfaces. When your team composes test cases using the same style, anyone can pick up a test case or defect report without needing to translate from Robert’s quirky shorthand or from Michele’s way of writing.

For example, Michele might write:

1. Go to the test site. The Home page displays.
2. Click the Search link. Search page displays with the search form which includes the following edit boxes: ID, Title, Customer, and Opened Date/Closed Date along with a Search edit box.
3. In the Customer edit box, type International Exports, Ltd. The value displays properly.
4. Click Search. A table of search results displays, including the following columns: ID, Title, Customer, Opened Date, Closed Date, and a column of action buttons (View, Edit, Submit). Verify that International Exports, Ltd., displays as expected.
5. Click the International Exports, Ltd., row in the table. The Customer Details page displays with the International Exports, Ltd., information populated and the New Order and Edit Customer buttons at the bottom.

Where Robert might put it:

1. go home
2. hit search.
3. type customer name
4. hit search
5. click customer

Certainly these represent extremes in how different people write test cases or bug reports, but even slight variations can slow people down when it comes time to repeat the steps, and different interpretations of different styles can introduce problems.

Minimum Viable Communication

Of course, you want the ability to expound in great detail about your software, its technology, its workflows, and its interfaces—the more you understand and can explain it, the better. However, you don’t want to be that guy at the coffeepot, in the break room, or at the party that launches into long, boring expositions at every opportunity.

You don’t want to do that in Jira, either. You want to keep your procedures only as detailed as they have to be to convey the important parts: the places where they differ from the other similar procedures or in the places where the problems occur. Because if you insist upon complexity for completeness sake, your devil will be in the details.