Getting the Most Out of Test Coverage

Having a solid test suite is instrumental to having confidence in your system, but how do you know if your test suite is good?

All metrics for evaluating the quality of a test suite have to be applied carefully; if taken to an extreme blindly, they all can cause harm. Common metrics include:

• count of lines of test
• maintainability of the test suite
• speed of the test suite
• how much of the code is run by the test suite

This last metric, called coverage, is what we will look at today.

What is Test Coverage

Test coverage counts how much of the application code is used by the test suite. Consider the following example code:

def application_code(animal)
	if animal == "cat"
		puts "felines rock!"
	elsif animal == "dog"
		puts "canines are better"
	end
end

def test
	raise "Test failed" unless application_code("cat").include?("feline")
end        

In this example, the test only tests the cat side of things, the dog is left out. In terms of test coverage, this means that of the 7 lines of application code, 6 are being run by the test, giving 86% test coverage.

Why is Test Coverage Good

Test coverage gives you an easy, understandable metric for your test suite. I can easily say that a test suite with 1% coverage is almost certainly not well tested, whereas at 99% coverage, it almost certainly is. In addition to how much coverage you have, where you have coverage makes a big difference. If you are a bank, having 100% coverage on the finances section is much more valuable than 100% coverage on a “where’s the closes branch” feature.

When you have high coverage and a quality test suite, you can be more confident that the system works as specified. This increases the speed at which new features can developed and deployed.

Why is Test Coverage Bad

Test coverage can easily be misleading:

def divide(a, b)
	a / b
end

def test
	raise "Test failure" unless divide(10, 5) == 2
end        

In this test, we are getting 100% code coverage. Congratulations!

Just don’t pass 0 in for b. Or do 11 / 5 cause that is 2 instead of 2.2 And good luck if you pass in something that isn’t a number. Currently, the code will happily try to compute 11 / ??.

Poorly written tests can increase the coverage metric without testing the code well. This leads to a false sense of confidence.

How to Use Test Coverage Well

I typically recommend testing the critical sections of your code first. This is the checkout path for an e-commerce site, posting on social media, or searching on a knowledge-sharing service.

Having high coverage of this critical code means you can deploy with confidence, and make updates without worrying about spooky actions infecting your code.