Unit Tests: Good or Bad?
Part one of three: The Mind Set.
Even though everybody seems to agree that unit tests are good, not everybody seems to agree what good unit tests are. Indeed unit testing is an intricate matter which may be one reason for its still much debated status [2,9,11,14].
This article can be seen as a follow-up to one written by a good friend of mine. Upon reading his paper [3], two questions immediately popped up in my mind:
- Why aren't we doing it, if it is so great?
- Is there a right way of doing it and why is it right?
Why aren’t we writing more unit tests?
Indeed, it is not really possible to have a definitive answer to the first question without answering the second one. This series of three articles is my attempt to provide a plausible explanation in this matter. So let’s start by giving some contexts in which we are writing the unit tests.
At this point I must confess: I rarely write unit tests and when I do so it is because I do feel it is the right thing to do. That said, I would agree to write more of them if I could relate them to some tangible benefits. Maybe this is the actual reason I'm writing this paper after all. Let me explain what I mean.
Why should we?
A quick tour on the web [4,5] gave me the following reasons why one should write unit tests:
- Make the process Agile - By making the code easier to change.
- Quality of code - At least on the testability side of things.
- Find software bugs early - Reduce the number of bugs when updating and refactoring the code.
- Facilitates changes & simplifies integration - When refactoring.
- Provides documentation - But who will ever use it for that matter…
- Support for debugging process - This is my favorite anyway.
- Improved design - Or at least testable.
- Reduce the costs - This is mainly true in production [13].
- Automated tests - The obvious use case.
- Inspires confidence - Safety net for refactoring. Increase testing coverage.
- It is fun - Not everybody seems to agree with this last one though.
All of which could be grouped in three main categories:
- To verify the code - Is it doing the thing right?
- To increase quality - Better design, less errors.
- To increase agility - Ease of debugging & refactoring.
Why are we not?
That sounds great! But between you and me, did you ever experiment that feeling of making worthless efforts when writing and rewriting those unit tests? You are not alone!
Here are some documented reasons that I found in literature and over the web [2]:
- No exposure to real testing culture - Which often leads to bad practice or no practice.
- Unclear educational resources - The testing boundaries are not explained, good examples are hard to find.
- No focus in universities - No support or motivation provided when learning to code.
- High test passion required - Learning to love better coding instead of quick coding practices.
- Hinders development and refactoring - because they need to be designed in parallel of the actual functionality according to the Three laws [6,7], so that resulting design, although it is being improved, is not trivial and so requires more time.
- Hard to retrofit existing code - Unit testing is a Design Consideration [8]. Good tests should not need to be rewrite when retrofitting the code.
- Learning curve - Which is steep and long. Uncle Bob himself admits it took him many years before he got it right in his “TDD Harms Architecture” paper [8].
- No obvious short term benefits - Apart from very nice looking code coverage reports.
So why are we?
The actual reasons why we are (mostly) doing unit testing if you ask me:
- Please our customers - Because it looks good on paper.
- Increase code coverage - An immediate and objective metric that can be interpreted in a beneficial way.
- Learn a new mock library - The only cool, but not so useful reason.
- Dogmatism - It must be done because it is... good.
Each of these reasons, but the learning one, is reflecting the fact that you have been asked to write them. This is the exact opposite of “feeling” it is the right thing to do, which should be the ultimate reason why Unit Tests should be written. Which leads me to this unfortunate conclusion:
We are not writing the unit test in the right mindset. And this, in turn, has a significant impact on their actual usefulness.
What science has to say?
Many studies point to the fact that TDD does not actually provide much positive effect on productivity, quality or even any reduction over other conventional testing methods [9,10,11].
Some studies even point to the fact that TDD can actually be harmful:
- Making refactoring more difficult and costly.
- Can reduce productivity, creativity and motivation.
- Create a false sense of correctness (covered code does not mean better code).
This is probably the main reason for the relatively low adoption rate: more than 10 years after their inception: slightly above 50% (was reportedly 53% in 2010 [12]).
Conclusion
After reading this first part, it is easy to conclude that unit tests can be either good or bad. It can also be argued that the context in which they are written will have a substantial incidence on their quality. But yet, it hasn’t been told what is good or bad?
Please check the following parts of this serie: The Dark Side... and Use the Force!
References
[1] Yoda image by Mario Eppinger, https://pixabay.com/en/yoda-star-wars-jedi-figure-toys-3888783, downloaded January 8th, 2019.
[2] Observations on the testing culture of Test Driven Development by Doug Arcuri, https://medium.freecodecamp.org/8-observations-on-test-driven-development-a9b5144f868, consulted December 20th, 2018.
[3] A Case for Unit Tests by Frederic Simard, https://www.dhirubhai.net/pulse/case-unit-tests-frederic-simard, consulted December 17th, 2018.
[4] Top 8 benefits of Unit Testing by Ekaterina Novoseltseva, https://apiumhub.com/tech-blog-barcelona/top-benefits-of-unit-testing/, consulted December 20th, 2018.
[5] 7 Benefits of Unit Testing by Leah Grantz, https://www.typemock.com/7-benefits-of-unit-testing/, consulted December 20th, 2018.
[6] The Three Laws Of TDD (video) by Robert C. Martin, https://youtu.be/qkblc5WRn-U, consulted November 20th, 2018.
[7] The Three Rules Of Tdd by Robert C. Martin, https://www.butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd, consulted January 9th, 2019.
[8] TDD Harms Architecture by Robert C. Martin, https://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Architecture.html, consulted December 20th, 2018.
[9] Effectiveness of Test Driven Development and Continuous Integration – A Case Study by Chintan Amrit and Yoni Meijberg, https://ris.utwente.nl/ws/portalfiles/portal/6943854/TCM_IEEEProf_final.pdf, downloaded January 22nd, 2019.
[10] An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach by Davide Fucci et al, https://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf, downloaded December 20th, 2018.
[11] Why Most Unit Testing is Waste by James O Coplien, https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf, downloaded December 20th, 2018.
[12] Introduction to Test Driven Development (TDD) by Scott Ambler, https://agiledata.org/essays/tdd.html, consulted January 8th, 2019.
[13] The Outrageous Cost of Skipping TDD & Code Reviews by Eric Elliott, , https://medium.com/javascript-scene/the-outrageous-cost-of-skipping-tdd-code-reviews-57887064c412, consulted february 4th, 2019
[14] Don’t write useless unit tests by Brandon Savage, https://www.brandonsavage.net/dont-write-useless-unit-tests/, consulted December 10th, 2018.
[15] Light saber image, https://pixabay.com/illustrations/star-wars-lightsaber-laser-sword-2369317/, downloaded January 8th, 2019.
[16] Dueling lightsabers image, https://en.wikipedia.org/wiki/File:Dueling_lightsabers.svg, downloaded January 8th, 2019.