Let’s Start Writing Tests (Correctly)

Are you grappling with the process of writing tests? Pondering over questions like, how many tests should you write? For what purpose should they be written, and what purpose should they not serve? Should you always adopt TDD (Test-Driven Development)? If these questions resonate with you, then you're in the right place. I have personally written over a thousand tests across diverse platforms and have embraced TDD in a multitude of scenarios. Throughout this journey, I have installed a culture of testing within few teams and projects, and will now summarize and share my insights.?

Fundamentally, test writing in programming has prone to becoming a cargo cult activity. Developers often chase trending methodologies, blindly trust documentation, or aimlessly pursue a 100% code coverage with tests. I have even observed instances where an entire test folder, comprising a staggering 40,000 lines of code, was discarded as it was deemed unwieldy. This conventional approach to testing often leads to counter-productivity, as it slows the development process, and, while it may have some positive effects, they are far too pricey.

In this article, I aim to present a comprehensive understanding of the core purpose of test writing. With a clear concept of the essence of testing, you can think critically and establish the right direction forward along with valuable practical tips.

To kick off, let's address the all-important question: Why do we need tests?

The answer is simple yet critical: We strive for confidence in our product's performance. Note that I didn't mention "function," "module," "code," or "project." At the end of the day, what truly matters is that the end product functions impeccably. While this may seem apparent now, maintaining a high degree of focus on this objective will enable us to make informed decisions.

Types of Tasks

Moreover, it's important to recognize that tasks can generally be categorized into two types: binary (either completed or not) and continuous, where completion lies on a scale from 0% (nothing accomplished) to 100%. Pursuing a solution on the continuous scale often incurs exponentially soaring overhead costs as we approach perfection.

Take the instance of service availability, or SLA, essential to many services. Hosting sites may pledge a server availability of 99.9%. Let's do the math here: 0.001 365 24 = approximately 8.7 hours of possible unavailability per year. That seems reasonable.

However, if it costs the company $1,000 to provide this level of availability, how much would it cost to add another nine to that percentage (achieve 99.99% or 99.999% availability)? This situation sounds like an exponential cost increase, pulling 100% availability into an unrealistic territory.

That example illustrates that, in continuous tasks, the overarching principle is to "maximize results with minimal resources." We should, therefore, strive for a balance where the desired results are delivered to stakeholders within an acceptable budget and timeframe.

Applying This to Tests

Applying this framework to test writing, we understand that testing is also a continuous task. Writing the first few tests in a new project can create a tremendous impact. A code coverage of 50% can be achieved swiftly, which is a massive step up compared to no tests at all. Post this level, the coverage growth rate begins to decline drastically, with tests becoming more specific and expensive. The complexity of test maintenance and refactoring also begins to soar. Achieving 100% test coverage is an expensive and likely an unnecessary pursuit, given the fact that even full-coverage tests cannot provide a complete guarantee of functionality.

In addition to the number and quality of tests, the cost factor significantly depends on the test type deployed. When it comes to classifications, there are various schools of thought based on system knowledge, automation level, testing time, etc. In this context, let’s concern ourselves specifically with "the degree of isolation of components". There are three primary categories:

1. Unit Testing
2. Integration Testing
3. System Testing (Acceptance)

These segments are immersed with endless debates on what qualifies as a unit test and what does not. Often, these tests have strict requirements to fit into the categories, and developers find themselves focusing more on conforming to the rigid canons than on the more important deliverable, the end result.

In reality, no clear distinction exists among these three levels. When you're testing pure functions or unit testing, you are in fact executing it on specific hardware where there could be a deviation in results. This brings an element of integration testing to the table, thus proving there aren't rigid boundaries between all these categories.?

Fixating on test type conformity can lead a tester to lose sight of their prime objective: to develop tests that ideally capture broad scenarios at a low cost. Some may resist this notion, believing that developers should be writing isolated unit tests, whereas testers should focus on acceptance tests. However, the reality varies across projects. There are projects where unit tests form a fraction of a percentage of all other tests, and others where only acceptance tests are written by specific individuals.

Example: Command Line Utility

Let's take an example scenario – you're writing a code for a program (a command-line utility) that searches for a specific word within files. This utility already exists, and it's called the "grep" service that most developers are familiar with.

Apart from the initial inputs, additional details like output formats, supported input formats, directory traversal (recursive), fuzzy search, and more, make the eventual program architecture unclear. Tests for small internal components are typical, but counterproductive in such scenarios. These 'unit tests' might appear cheap and high quality, but they carry concealed costs. With the uncertain project architecture and rapid changes in modules, classes, and functions, maintaining these tests becomes tricky which may frustrate the developer, and could lead to halting the test writing entirely.

The key here is to identify the highest entry point into the program that remains independent of the internal implementation while satisfying the task's primary objective. While commencing testing at the highest level (like directly running the program in the console) might seem a good idea, it usually entails issues like launching a separate process or reading standard streams, and so on. In our sample program case, such level of testing can be classified as "systems testing," given that it examines the function at the highest possible level, without grappling with the internal implantation. Though such a test would not challenge an experienced developer, the cost of such a test for this library would be at the maximum.?

A more efficient approach would be testing at a slightly lower level. For instance, consider a function that takes a file path and a substring to be searched as input, and gives a finished result that only needs to be printed. This kind of test strikes a perfect balance between ensuring that everything works and reducing the cost. These tests indirectly influence all internal components used, stand independent of the implementation, are extremely easy to write, and are cheap to maintain.

Here’s how you might write such a test in Java

Test-Driven Development (TDD)

The described technique works well with TDD, where tests are written before or along with the code. TDD is often misunderstood as only useful for regression testing. However, writing tests before the code can speed up development, especially when dealing with complex data transformations.

Benefits of TDD

• Faster Development: Writing tests before the code helps clarify requirements and design early. It forces you to think about the API and expected behavior, reducing the need for rework later.
• Better Design: TDD encourages writing smaller, more focused functions that are easier to test and maintain. It also helps avoid over-engineering by focusing on the minimal code needed to pass the tests.
• Regression Safety: Although not the primary benefit, TDD ensures that you have a suite of tests that can catch regressions when you make changes to the codebase.

Designing Tests with TDD

When designing tests with TDD, start by writing a test for the simplest functionality. For example, if you’re building a calculator, you might start with a test for adding two numbers:

Testing is about finding a balance. Aim for maximum results with minimal resources. Understand the purpose of your tests and choose the right entry points for writing them. Use TDD to design better code and speed up development. Remember, the goal is to ensure the product works well, not just to follow methodologies blindly.

By adopting a thoughtful approach to testing, you can achieve better outcomes without unnecessary overhead. Happy testing!