Reporting a Finding

WHO

Anyone who needs to describe an issue can file a report

The majority of the reports are written by Quality Assurance team but this should not prevent any other role such as Product Manager, Developer, Project Manager, Designer, Sales, Support or anyone else from filing a report.

WHEN

Have you spent one minute or less to search in Jira if a similar report is already filed ? It might save from creating a duplicate report.

If you truly believe that an issue is that critical that others should abandon whatever they are doing to deal with this issue then do NOT write a report but rather communicate with the people who can help. But this normally should be very rare.

WHY

Why should you even write a report of an issue ?

You write a report in order to help the team. You write a report to describe an issue in a way that will not interrupt other people’s work and it will be picked up by a dev lead or project manager to be prioritized accordingly.

Why write the report now ?

Indeed before writing a report you should consider everyone’s time consumed by the report:

• Yourself: Your own self is going to spend time writing a useful report. Is the issue worth the time in comparison to all of the rest of your tasks ?
• Dev Lead, Product Manager, Project Manager: Is the report clearly separating the actual pain or concern that the customer is facing from the rest of the technicalities in order for someone to prioritize it?
• When one is going to be looking at multiple tickets in a list, will this person be able to deduct from the title alone the true concern of this issue ?
• As a reporter can you have a gut-feeling of the priority of this defect to help with the initial estimation ?
• Developer: Is the report containing all of the information in order for an engineer to dive in debugging mode without any unknowns ?
• Given that you have a feeling of the pace of bug-fixing, is the report going to be picked up or are you spending time writing a report that will be picked up much later due to other priorities ?

If the report is not helping anyone to be written now, do NOT lose it, rather leave aside the defect in your notes to refer to it later to describe it fully rather than alerting now with a half-baked report.

Why write a long report ?

The report needs to be as useful as possible, answer as many questions as possible without requiring the presence of the report, and include as much of metadata as possible, in order to make it reproducible, if possible, and for one to pick it up without any unknowns.

Why write a short report ?

Because you want your report to be picked up and it is most likely to do so if it is concise, understandable and directly to the point.

As anyone in the company can file a report, there can be cases where it is justifiable that you file a report which is shorter than required, because you do not have the luxury of time to write a full fledged report. Note that this is NOT recommended, but in case this happens you should be prepared to make yourself available to whoever picks up the report and answer all of their questions as the report does not automatically answer them as it should.

WHAT

What happened in a nutshell ?

Summarize in the title what the issue is about, by avoiding any technicalities, focus on the impact that the issue has on the customers.

Summarizing the title should be done diligently as this will be the only content one can see when navigating through a long list of tickets.

What kind of report ?

At the majority of time you will be filing reports that impact the Functional part of the system, this is the default, no need to mention anything.

If other than functional, then specify what is the pain/concern/issue that the report is related to. The list can be very long but here are some kind of common kinds of reports:

• Usability: The system works but the users suffer or cannot even use it because it is unintuitive
• Performance: The report is related to an issue that has occurred due to a large load of traffic
• Compatibility: The system does not work only in some specific cases of software installations and/or devices
• Translation: The system is not translated properly to a non-default language
• Security: The report is referring of how the system could be vulnerable when attempted to be used by a malicious user
• Concurrency: The system is behaving normally when used by a single device or single browser tab but it is misbehaving when attempted to be used by multiple devices or having multiple browser tabs open
• Findability: The system is behaving normally but nobody can find what was built because search is not working properly or the product cannot be found by google search because of lack of SEO, etc.
• etc.

What priority ?

Use your domain knowledge, gut-feeling, experience and everything you can to give an estimation that you believe is appropriate for the defect

• Critical: Blocker and the release should be blocked for any related functionality, where related functionality could be a specific feature or an entire app depending on the context
• Major: At some extent the app can survive temporarily with the current defect, but it is either hurting the acceptance criteria or it is a great pain for the customer and in general it is considered of high importance
• Minor: The app can survive for quite some time having this defect exposed to the customer but for reasons of identity, to provide an elegant app, also for building trust with the users and in general offering a more than positive user experience this needs to be resolved at some point in the near future
• Trivial: These defects can be neglected and the user can easily find a workaround to accomplish his/her goal. These defects should be resolved if we want a pixel perfect app and we want to offer a premium user experience to the end customer.

If you can separate easily you may even provide two priorities. Namely how severe the report is, on both the specific Feature and on the Product overall, even if the latter is usually at the ownership of Product Managers or Product Owners.

Under which context ?

This should be done as automatically as possibly to prevent human input error and reduce time of creating a useful report but should include inside a video and/or in text:

• User: Either write or show in the video the login credentials of the user or users utilized during the scenario
• Link: If not shown in the video specify a web link if relevant
• Version: Grab or ask the version of the API, the website, the app etc.
• Product: Which product this defect is referring to
• Device/Platform: For mobile mention if it is Android or iOS and specific device if necessary
• Browser: If the scenario was executed on Safari/Chrome/Firefox/Opera etc.
• Operating System: If you believe that the scenario can be prone to MacOS versus Windows vs Linux please specify as well
• Logs: As they are generated from the execution itself
• Other metrics that might be available/relevant: CPU usage, Memory Usage, Battery usage etc.

What was expected ?

Describe what was the expected outcome from the scenario that you followed.

In case you are unaware or uncertain of what the expected outcome should have been, you need to chase for a concrete answer or call for a meeting if necessary as this will be what the fix of the engineering team will be focussed on.

What happened ? [The full story]

Use an appropriate recorder to record a Video, speaking out-loud, describing with your voice the steps that you are doing either in the mobile app or in the desktop, to highlight the experience of the customer.

Rule of thumb: Minimum recording time: 30 seconds, maximum recording time: 4 minutes

For more simple to understand issues you may include a couple of screenshots instead of a video.

Complement the video/screenshot with text in order to:

• Give a summary of steps of what happened but do NOT fully repeat what one can see from the video already.
• Include steps that happened before the recording of the video in case this is necessary
• Include steps and perhaps extra findings that followed the recording of the video if this is applicable

What is the optimal length of the report ?

This can be optimized through practice and while being adaptive take into account your audience:

• Are they experienced with the system ? You may be more short and concise
• Are they newcomers ? You may need to be more thorough explaining also the obvious parts

What to do if an issue is non-reproducible ?

It is happening quite often that you may encounter an issue because you do not start from a clean state and that the issue is not easily reproducible.

You need to mention the fact that your report is non-reproducible and that the steps to reproduce may not lead to the defect one can observe in the video or screenshot.

HOW

How to deal with non-reproducible defects ?

An engineer needs to carefully add the necessary logs in order to have the full-story the next time this defect is encountered.

Until then an engineer needs to consider the trade-off of how long will it take to discover the root cause versus how long will it take to build an extra layer that handles the defect at hand.

Therefore the initial strategy should be to have a deeper understanding of the system and discover the root cause BUT if this is consuming large time-resources then one may need to create an extra layer of code that is not causing other side-effects to the business logic, and manages the defect.

How to assert that you have written a useful report ?

This is NOT by blindly following what is written above but:

• by observing if you receive many questions after filing a report. The fewer questions you receive benefits both yourself as you are not interrupted by questions and the one who is picking up the report as he/she has all the relevant pieces of the puzzle at hand
• by having team members expressing their gratitude when reading they read your reports, finding them useful and helpful
• by observing less incidents to the end-customers as your reports are actually being picked up and resolved
• by observing/feeling gradually an overall increase at the quality of the product which could/should impact the customer reviews, the revenue etc.

EXTRA TIPS

• Reports are here to help the team and enable asynchronous communication but cannot always replace team member communication
• Reports are written from people who may have different styles of expressing themselves
• Reports are written for people, therefore should not be as rigid as an api
• Did you find yourself enjoying writing the report ? If not, then you are surely doing it wrong