Should You avoid End-to-End Tests?

In his book Succeeding with Agile Mike Cohn outlines a model called test pyramid to help explain what types of automated tests are needed. The pyramid helps us think about not only the scope of tests, but also the proportions of the different types of tests we should aim for. Cohn’s original model split automated tests into unit tests, service tests, and UI tests.

?

The key thing to take away when reading the pyramid is that as we go up the pyramid the test scope increases, as does our confidence that the functionality being tested works. On the other hand, the feedback cycle time increases as the tests take longer to run, and when a test fails, it can be harder to determine which functionality has broken.

As we go down the pyramid, in general the tests become much faster, our continuous integration builds are faster, and we are less likely to move on to a new task before finding out we have broken something. When those smaller scoped tests fail, we also tend to know what broke, often down to the exact line of code – each test is better isolated, making us easier to understand and fix breakages. On the flip side, we don’t get a lot of confidence that our system as a whole works if we’ve tests only one line of code!

?

Unit Tests – Typically tests a single function or method call.

Service Tests – These are designed to bypass the user interface and test our microservices directly. In a monolithic application, we might just be testing a collection of classes that provide a service to the UI. For a system comprising several microservices, a service test would test an individual microservice’s capabilities. The scope of the test is somewhat isolated to the service.

End-to-End (UI) Tests – These are run against entire system. Often driving GUI through a browser. End to end tests gives a lot of confidence that the entire system is working fine but they can be very tricky to do well in a microservice context.

?

Should You avoid End-to-End Tests?

End-to-End tests can be still manageable with a small number of microservices, and in these situations they still make lot of sense but with 3,4, 10 or 20 services they very quickly become hugely bloated, and in the worst case they can result in a Cartesian-like explosion in the scenarios under test.

What is one of the key problems we are trying to address with end-to-end tests is to ensure that when we deploy a new service to production, our changes won’t break the consumers of that service. Although end-to-end tests catches such breakages, but they do so at a great cost. Ideally, we would like to have some type of test that can catch semantic breaking changes and run in a reduced scope, improving test isolation and therefore speed of feedback. This is where contract tests and consumer driven contracts come in.

?

Contract Tests and Consumer-Driven Contracts (CDCs)

With contract tests, a team whose microservice consumes an external service writes tests that describes how it expects an external service to behave. This is less about testing your own microservice and more about specifying how you expect an external service to behave. One of the main reasons these contact tests can be useful is that they can be run against any stubs or mocks you are using that represent external services – your contract tests should pass when you run your own stubs, just as they should the real external service.

?

Contract tests become very useful when used as part of consumer-driven contracts (CDCs). The contract tests represent how the consumer microservice expects the producer microservice to behave. This is done by having the producer team run the consumer contracts for each consuming microservice as part of its test suite that would be run on every build.

?

A consumer could be a web app, mobile app or another service. Anything that consumes an event or API service, can be a consumer. Within consumer driven contract testing they are the creators of the contracts. A provider is a service providing a message or API. Providers verify the contracts are compatible with how the service has been implemented. The contracts are validated against the provider’s real instance partitioning a complex system into manageable, testable units. This leads to quicker, simpler and more stable tests, instilling confidence in the release process.

?

Because these CDCs are expectations how the provider microservice should behave, we need only the provider microservice to run, meaning we have the same effective test scope as our service tests. CDCs require us to only run provider microservice with any external dependencies stubbed out.

?

CDCs sits at the same level in the test pyramid as the service tests, albeit with a very different focus. If one of the CDCs breaks during the build of the provider service, it becomes obvious which consumer would be impacted.

?

The Final Word

End-to-End tests have lot of disadvantages that grow significantly as you add more moving parts to the system under test. As we start with creating CDCs the end-to-end tests may just for a useful safety net, where you are trading off cycle time for decreased risk. But as you improve more on CDCs and as the relative cost of creating the end-to-end tests increases, you can start to reduce your reliance on end-to-end tests to the point they are no longer needed. But yes, ditching end-to-end tests without fully understanding what you’ve lost is probably a bad idea. Challenge yourself to think long and hard about how much end-to-end testing you really need to do.

Credit - https://samnewman.io/books/building_microservices_2nd_edition/ ?

?

?