Better Testing Pyramids

The Automation Triangle (or pyramid) has continually caused a bit of a stir in the testing world. It has been mislabelled many times to be called a “testing” triangle. It has been abused in many companies by being followed as a “test strategy”. It has been butchered in ways that cause further misunderstandings and has caused many a headache to many different people involved in software for years.

Recently, I decided to try and change my stance on the model which was previously to cringe and moan any time I saw the triangle being used in a way that only tells half a story about testing. I decided to intentionally look at the triangle from a perspective of awesomeness.

That didn’t quite work out for me – BUT, I did have an idea on some different ways to draw different testing pyramid models!

Here’s one

From this perspective, the triangle could be useful to see that testing (both investigative testing and assertive testing approaches) are done at all the different layers from the user interface right down to code level.

Lets explore this idea for a second. If you are investigating risks to discover unknown information, you can test at all layers. If you are writing automation scripts to assert any expectations that you have, then you can write the automation at any layer too. The pyramid is still based on value, like Mike Cohn’s original automation pyramid. It’s cheaper and takes less time to test the things at the bottom, then the middle, then the top. So that means we should test for more risks at the bottom if we can, then test at the middle layers if we can, then test through the UI where necessary.

Here’s another one:

This triangle represents the amount of tests and test ideas that we can generate across the abstract layers of ignorance. There is information that we know. This is usually very limited compared to the information that we are aware that we don’t know. And beyond that there is even more information that we are unaware of…

Although the triangle isn’t based on value, it can help us to focus on trying to uncover more information from our testing, after all the purpose of testing is to provide information, right?

Here’s another one:

With this one, we can think about the different environments and where we should conduct our testing. The majority of the risks that we are investigating should be done on a Dev environment. Ideally while pairing with a developer. This offers the fastest feedback loop and any problems can be resolved there and then when discovered.

A test environment is still useful for testing risks such as integration risks, where you are merging your developed code in with other developed code that might not have been available on your device environment. This is a slower feedback loop, but offers you a safe space to test and build confidence in the quality of the software before you release to prod.

Testing in prod is also an option. Although this offers the slowest possible feedback, and has a higher risk of users becoming pissed off with any bugs you might have released (this risk is especially high if you *only* do testing in production), there are some benefits to testing in the production environment. One common one is if you are releasing to a set of users for them to trial in a beta test/acceptance test sense. Additionally, it’s very hard to mimic the production system with a “like-live” system, so if you need the exact hardware and exact amount of data, then you should investigate whether it’s feasible enough to just use the production environment for testing those risks… But I strongly advise to weigh up the business risks in leaving that testing till the end with the risks of users seeing problems first before you make that decision.

Here’s another one:

This triangle has a focus on the different approaches to testing.

The idea is that a traditional scripted testing approach (using test cases that list steps and expectations for a person to check) should be reducing with the rise of using an automation approach to be able to automatically assert those expectations through code.

But with us having very limited information regarding expectations compared to the unknowns around all the different product risks, especially in the complex software that we see today, then the majority of our testing will use an exploratory testing approach to investigate these risks. Especially when we think about testing the ideas of new features, and testing the artefacts that we create to describe our expectations, and testing the UX/UI designs, and testing the code design, and testing the code through code reviews, and testing our processes, etc. It becomes clear how much exploratory testing we do (or should do, if you aren’t doing this) all the way through the SDLC.

ALL MODELS ARE FALLIBLE!!!!

So, there are many different ways that we can draw a triangle, or pyramid. Each model has a specific purpose, based on the creator of the model and what they needed the model for at that point in time.

Lumping a different meaning onto a model to try and portray something different to what the model was created for can be dangerous and ignorant. I’d strongly advise just to create a new model. It can be in the same format (as I have demonstrated in my above “triangles”), or it can be in a completely different, unique format – whatever helps you regarding your need for the model.

For example, if you are looking to create a model for a “test strategy”, then check out an example of a model for test strategy that I created previously. Or if you want to create a model for showing where testing fits within Agile, you can look at the Agile Testing Quadrants model for inspiration. (Coincidentally, you will also see on this blog post from Lisa Crispin how easy it is for someone to misunderstand and misuse a model in a wildly dramatically misunderstood way than the original model was intended to be used).

Share your models!

I’m keen to see some of your own models. Rather than try and fit other models into your needs (like some people have tried to do by using the Automation Pyramid as a test strategy), have a go at creating your own! Share it with me on twitter or on here in the comments section. Good luck!

(This post was originally posted on danashby.co.uk - https://danashby.co.uk/2018/05/03/a-better-testing-pyramid/ )