When Testing is Enough?

When Testing is Enough?

A familiar question every IT person struggles with is, when testing is enough for releasing a software. To be very precise and clear, Testing can never be enough, as there is no software for which you can take ownership that this is a bug free

software. Because 100% bug free software is a myth. But yes, you can stop when all relevant risks have been covered to the point you can afford and accept.

I) Guidelines: The following points can provide some guideline to help you stop the testing tasks:

• Testing Deadlines.
• Completion of test case execution.
• Completion of Functional and code coverage to a certain point.
• Bug rate falls below a certain level and no high priority bugs are identified.
• Testing budget
• Management decision.

II) How much testing is really enough?

Step 1: Determine what the team means by enough

Asking a question about enough testing might be confusing to some organizations.

They might understand testing that consists of positive confirmation of what is expected and not looking beyond the happy path.

The idea of one requirement: one test is normal.

The challenge is, while this may be acceptable for some organizations, it is something less than that for many others.

Step 2: Avoid the not as much testing as we think trap

Many organizations are closer to the minimal one requirement: one test than they realize. A simple script is created and intended to be an open-ended question.

The steps are expected to be run several times with different values which can be correct or incorrect in various ways.

Step 3: Find the right balance

What does the idea of enough mean? There is a balance between the two extremes.

What and where that balance is depends on the situation.

I find a level of testing which allows for in-depth exploration of key features, along with reasonable coverage of secondary features to work much of the time.

What will be tested more and less than other features should be discussed with the stakeholders and project team so everyone is in agreement.

Then, regression and integration tests need to be updated accordingly to handle the changes.

III) Seven ways you can decide you are done testing:

1. Walked through all the test ideas which were given

2. The time for testing ran out

3. Experiencing diminishing returns

4. Testers are exhausted

5. Test ideas are out of scope

6. Remaining test ideas are below the cut line

7. Tests are below an agreed-on priority

IV) Checklist

You can use this checklist to determine whether or not you have enough confidence for completing the testing:

• Stop the testing when deadlines like release deadlines or testing deadlines have reached.
• Stop the testing when the test cases have been completed with some prescribed pass percentage.
• Stop the testing when the testing budget comes to its end.
• Stop the testing when the code coverage and functionality requirements come to the desired level.
• Stop the testing when the bug rate drops below a prescribed level.
• Stop the testing when the number of high severity Open Bugs is very low.
• Stop the testing when the period of beta testing/alpha testing is over.

V) Software Metrics: Some useful metrics you could use :

• Calculate the percentage of test cases that have passed.
• Calculate the completion percentage based on the number of test cases executed.
• Calculate the percentage of failed test cases.
• Calculate the percentage of code coverage.

VI) Summary

Creating a comprehensive qualification process and testing strategy to answer the question ‘How much testing is enough?’ can be a complex task. Hopefully the tips given here can help you with this. In summary:

• Document your process or strategy.
• Have a solid base of unit tests.
• Don't skimp on integration testing.
• Perform end-to-end testing for Critical User Journeys.
• Understand and implement the other tiers of testing.
• Understand your coverage of code and functionality.
• Use feedback from the field to improve your process.

VII) Conclusion

So, how do you know when is the best time to stop testing?

The correct answer is to combine several of the mentioned above practices/metrics and determine what is the definition of DONE in your test plan/strategy documentation. If you can select most of the items from the checklist and tick them off with a yes, that's when you know you can potentially stop testing. On the other hand, if you see more ‘No’ than ‘Yes’, you know that something might be missing and you can work on it and avoid any bug heading into production!

HAPPY TESTING!