Defects Don't Matter
How are you measuring quality? Is it the number of defects in your backlog? Code coverage? I've been pondering how to measure quality recently and the best I can tell, none of that matters.
First, let's define some terms because any domain needs specific language and for the domain of this discussion, we'll use these.
- Deploying: applying an application change to Production. Test environments don't count.
- Defect: Anything that doesn't meet expected outcomes.
- Issue: A defect that causes a variance outside of the defined bounds required for stable operation.
Given those:
Quality = Deploy frequency / Issue frequency
Defects that are not Issues don’t measure quality. The reason is that Defect is anything from "I don't understand how this works, make the documentation better" to "OMG!! The server is melting down!!" It's completely subjective. Using it as a quality metric for a team or application encourages meetings, arguments and prioritizing "the font isn't the shade of green I expected" over higher value features. None of those increase our ability to deliver value more rapidly. Also, using defect count as a measure of quality will generally cause the number to increase.
Deploy frequency is the numerator of quality:
To accelerate your deploy frequency, you need to focus on standardizing and optimizing your process flow. This is true in any manufacturing process but for software development, that means focusing on your development processes deployment process, and methods testing. Deploy frequency will plateau early if you are depending on manual “testing”, manual approval, complicated branching structures, and manual or multiple installation methods. As you automate and optimize these things, you remove issue generating variance injected by human fallibility. In addition, the humans who were doing those things now have time to do smart and creative things and can continue to find more ways to optimize. Humans cannot beat machines at repetitive work. Stop trying.
Another reason that frequency matters is that the higher your frequency of deploy, the higher your learning bandwidth is. Just as high frequency X-rays can resolve much finer detail than lower frequency radar, high frequency deploys require smaller changes that are much easier to identify when an issue happens. In addition, it’s rare that a tiny change will cause major issues. However, when they do (and they will) you have a much better chance of finding them quickly and reducing impact. Let me be clear on this, MTTR is the reason. Rapid feature delivery is awesome, but being on call for production support will make you optimize for MTTR. That, in turn, allows for rapid feature delivery.
So, stop tracking defect counts. Just prioritize them in the flow. There’s no difference between features and defects anyway because both mean the system isn’t doing what I need it to do. Create a test for the desired behavior and make sure it does what you want tomorrow.