Check capabilities, not ingredients.

If someone told you that their house is squeaky clean because they bought a large bottle of Mr. Clean, you'd likely be a bit skeptical as it doesn't imply they actually cleaned. However, in corporate IT statements like "we have great code quality because we use SonarQube" are all too common. Now, SonarQube is a fine tool, just like Mr. Clean is a great cleaning product, but having either in the shelf won't do much good. It's about how you use these tools, not how many of them you have,

What Unit Tests Coverage Should You Have?

When looking to assess the quality of automated testing, we commonly look at test coverage, i.e. what percentage of the lines of code are exercised as part of automated unit / integration tests. Is 80% coverage sufficient? Is 85% better? Can you get to 90%? Aside from the usual caveat that you can get "coverage" with tests that contain no asserts and thus never actually test anything, the real test (pun intended) for whether your coverage is sufficient requires you to answer why you have tests in the first place.

Most organizations will state that they have unit tests so they can find defects early before the code reaches production. If that is so, a simple experiment can assess whether your coverage is sufficient: allow me to delete a dozen or so lines from your code base (that's surely a defect!) and run your tests. If the automated build remains green, we'll immediately push the updated code into live production. And go for a drink.

Testing is about confidence

If the developers start panicking or open Monster.com to look for a new job after hearing this proposal, you know that your test coverage isn't sufficient. It's not about having a tool and also not about a percentage of code covered, but about the certainty of knowing when code is broken.

Hardly any team would take up this challenge, which admittedly is a bit exaggerated. However, the premise is valid: the purpose of automated testing is to give teams confidence that their code is of good quality and that nothing broke due to a recent code change.

Without confidence, teams will hesitate and thus be slow. Or they won't fix things out of fear of breaking more things. The digital world demands fast teams, so testing tools are needed to give confidence. If you want to test whether the tools are sufficient, test the team's confidence, not the tools.

Anything that's not tested is broken

I once observed a major hardware outage due to a failed power supply. Of course, the system featured redundant power supplies, so the list of ingredients would make us believe we were fine. However, the second power supply did not kick in when the first one failed. Apparently no one ever tested the failure of a power supply. The existence of the "redundant power supply" ingredient was deemed sufficient to assume the capability of operating with a single power supply.

While architects are employed to make design decisions that yield desired behavior, the ultimate proof is in the pudding. Trust yourself to make the right architecture decision but don't trust your system. Check every ability you rely on. Assume that any ability you didn't test doesn't exist.

Another case of the boxes

Focusing on tools or attributes can be another case of putting boxes ahead of lines. Having a box in place is easy to check: we use Nagios, so we have good monitoring; we have SonarQube, so our code quality is good; we use Jira, so we are agile. Reasoning on the desired ability or behavior is much more meaningful, but admittedly also harder. No one said IT is easy...

Read more about what one chief architect knows about IT transformation at https://leanpub.com/37things

If you appreciate my articles, please share them with your network.