Unit Tests: Good or Bad? (part 2)

Part two of three: The Dark Side.

In the first part, The Mind Set, I have provided some contexts to demonstrate that unit tests can be either good or bad. In this second part, I will attempt to isolate some bad practices that lead to poor unit testing. So let’s explore the dark side together...

“You have to go through the darkness before you get to the light.” -- Haruki Murakami

Indeed, it is quite difficult to isolate the issues at stake because they are all intertwined. So let's try to split apart some key factors that lead to poor testing. I have decided to break down this section according to the main tools that are commonly used when building unit tests.

Things we should beware of:

• XUnit and Coupling
• Mocks and Fragility
• Coverage vs Testing

XUnit

Ok, my point here is not to say that you should avoid using XUnit which are excellent tools. But using these XUnit tools may give a false sense of what good testing should look like.

One-to-One Correspondence

When I started being asked to write unit tests, my first reflex was to use the Eclipse built in JUnit plugin. Within it, it is very easy to create a new Test Class with all its public methods prefilled. So this must be the best way to do things, right? Wrong!

Indeed, JUnit gives us the impression that a “Unit” is associated with a “Class”. It also gives the impression that each “Test” should be associated with one “public method” of that same class, and this is... bad.

“The problem is – and I want you to think carefully about this next statement – a one-to-one correspondence implies extremely tight coupling.” -- Uncle Bob

Mocks

I’m not saying you should not use the Mocks but indeed, yes, you shouldn’t use the Mocks…

“Mocking is a Code Smell.” -- Eric Elliott

Testing the mocks

Looking at many of the Mock tutorials [7] is a very good way to learn how you should not use them. Indeed, when reconstructing the implementation of the tested method in a mirror image way, you are in fact testing the mocking framework itself. As you can see, it is fairly easy to write useless [8] unit tests and it can also be fun!

“The team was new to TDD, and they got hold of a tool, and perhaps read a book or article, and decided that TDD was done by using a mocking tool. This team is not the first team I’ve seen who have fallen into this trap. In fact, I think that the TDD industry as a whole has fallen into this trap to one degree or another.” -- Uncle Bob

The Fragile Test Problem

This “mirror image” aka test doubles [5], leads to what is called Fragile tests. Fragile tests are easily broken because strongly coupled to the implementation [4]. This is bad since it prevents the code from being easily refactored without side effects. The main reason being the obligation and the time required to re-write all the unit tests. At the end, Fragile tests will prevent the refactoring they are supposed to promote.

“I don’t see any compelling benefits for mockist TDD, and am concerned about the consequences of coupling tests to implementation.” -- Martin Fowler

Code coverage

Code coverage deserves a section of its own simply because it is too often mixed up with unit testing. For some people, it’s almost like the two expressions are interchangeable. Code coverage is to pinpoint the code areas that are definitely not tested and has nothing to do with good testing. 

“I’m not saying code coverage is a bad metric, it’s just dumb” -- Chris Cooney

As a matter of fact, the actual amount of useful information one can extract from the code coverage metric is somehow quite limited...

But the most important thing to remember is that code coverage is a measure of quantity not quality [6,2]. Good tests can do good coverage, but so do bad tests. On the other hand, a good testing design does not imply full coverage. Be warned!

“They (The teams who try) also tend to miss the whole point of TDD. It's not about test coverage, it's about design.” -- Ben Scott 

Coverage will not tell you which pieces of code and what code related tests are missing. It will only run through whatever code it is given. Coverage should not be mixed with testing especially at the design phase [3] since doing so will effectively prevent the finding of those previously mentioned missing parts. Also, not allowing enough time for an effective analysis of coverage issues will induce a patching the holes in testing instead of a reflection about why a certain section is not covered accordingly...

“If a part of your test suite is weak in a way that coverage can detect, it's likely also weak in a way coverage can't detect.” -- Brian Marick

Conclusion

In this second part, I have shown that the most popular tools that are supposed to help you in building your unit tests can actually induce some wrong doings. Indeed, knowing how a tool works does not mean you know how to use it. The variety of websites that can be found about all those tools and their various approaches using these tools is the perfect example of this principle. So now that we are aware of this confusion, how can we tell the good from the bad?

Please look at the conclusion in the last part: Use the Force!

References

[1] Yoda image by Mario Eppinger, https://pixabay.com/en/yoda-star-wars-jedi-figure-toys-3888783, downloaded January 8th, 2019.

[2] Efficient Code Coverage with Eclipse by Frank Appel, https://www.codeaffine.com/2014/04/07/efficient-code-coverage-with-eclipse/, consulted December 3rd, 2019.

[3] How to Misuse Code Coverage by Brian Marick, https://www.exampler.com/testing-com/writings/coverage.pdf, consulted December 3rd, 2019.

[4] TDD Harms Architecture by Robert C. Martin, https://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Architecture.html, consulted December 20th, 2018.

[5] When Writing Unit Tests, Don’t Use Mocks by Seth Ammons, https://sendgrid.com/blog/when-writing-unit-tests-dont-use-mocks/, consulted December 20th, 2018.

[6] Is Test Coverage a Good Metric for Test or Code Quality? by Chris Cooney, https://hackernoon.com/is-test-coverage-a-good-metric-for-test-or-code-quality-92fef332c871, consulted on February 18th, 2019.

[7] Mockito Mock Database Connection Example by Mohammad Meraj Zia, https://examples.javacodegeeks.com/core-java/mockito/mockito-mock-database-connection-example/, consulted January 8th, 2019

[8] Don’t write useless unit tests by Brandon Savage, https://www.brandonsavage.net/dont-write-useless-unit-tests/, consulted December 10th, 2018.