Continuous Testing in DevOps...

I’ve recently attended a number of conferences, some testing ones, some agile ones and some dev ones too. Although some of the talks were painful due to misunderstandings of one thing or another (be it testing, automation, agile, BDD, TDD, etc…), overall, I thought the conferences were pretty good and I met some new people too and got to share some stories with them.

One thing I heard a fair amount was about DevOps. Many people were talking about it. It’s a big topic! But too many people seemed hugely confused about where testing fits in this wonderful world of DevOps. Some suggested that you only need automation in DevOps, but when asked to explain, their arguments fell floppily by the waist-side. Some people blatently refused to guess at how they’d try and implement testing in DevOps.

And one person even said that no testing was required at all, as he pointed to a DevOps model on a slide, which had no mention of testing on it whatsoever.

Here’s my hand-drawn attempt at replicating that model:

It transpired that this model was presented quite a lot throughout the conferences (or extremely similar ones were presented) and you can see why people struggle to understand where testing fits in a model that doesn’t mention it at all.

For me, testing fits at each and every single point in this model. Let me break this down for you in an updated version of that model to show you what I mean:

We can test the plan. If, in this DevOps world, plan comes to mean design, then we can test the design. Testing the design of the software is exploratory in nature. It can still be risk based, as we use the risks as heuristics to focus our investigation. Our ideas that we explore through questions will help us uncover more information. Information that will help us refactor the design (or plan) to make it better!

We can test a branch (not a physical branch on a tree, although I’m sure that would make an interesting testing challenge for someone…). We can test individual development branches, or we could test our branching strategies. See all the conversations that you have about your branching strategy, and you ask questions and investigate ideas and refactor your strategies… That’s testing.

And of course we can investigate our code! We can also check our code too against any expectations that were discussed as part of our planning! Code reviews – exploratory testing to investigate to find any mistakes and to see if the code is written in a really great way or to investigate if there is a better way to write it… Unit checks – awesome automated checks at code level to ensure that an expectation is met by the code… And if you are a developer writing some code, then you can even pair up with the tester who will share some really cool test ideas about different risks, different interactions, different perspectives, different variables and different combinations of all these things to help you right there while you code the software! (Yey!! How cool!!)

Merges need to be tested and checked too! Watch out for the hidden risks of different integrations (and different levels of integrations), and also watch out for merge clashes too!

We can certainly test the build too! Why not test your build process and enhance it?

On top of that, Why not spin up an env and do some exploratory testing of the product too? ET is the most effective approach to testing and helps us uncover a perceived level of quality.

When it comes to the release, we can certainly test that process too. And we can test our deployment (on a test env & on production too). Some people like to do “sanity” or “smoke” testing and checking. Explore your heart out! Just make sure that your testing (or your test data) isn’t disruptive to the users.

And once its live, no doubt that your users will see the shiny fields and buttons and will be clicking away to appease their curiosity of solving the “what if I do this…” mysteries that live inside us all.

And finally, here comes the monitoring… But in no way is it least by being last. Monitoring is powerful in supplying us information on what we tell it to look for. And that’s where we can test it! Are we telling it to look at the right things? What should it be looking for and what should it be expecting so that it can automatically transmit to us those informative little bundles of messages about those expectations? Lets explore these ideas! And lets keep an eye out for that information and investigate what its telling us too.

But what is missing in the diagram…? I can’t help but feel there is something missing…..

Oh yeah! It’s so obvious now! We can’t start planning unless we have the idea, right??

The model doesn’t mention all that work that happens before that planning bit. Here’s an explosive fluffy-clouded gif to help explain my thinking on this:

Testing the idea is something that some people completely miss the boat with… Just like the model did. Testing the idea is vital; investigating to uncover information, refactoring the idea to enhance and solidify it. It allows us to stem lots of different artifacts and we can test those artifacts them to refactor them too. And we can then use those artifacts to stem all of the activities that we conduct, from; designing and planning, development, testing and checking… And those activities produce lots of useful outputs too – Quality software for one!

So hopefully this blog post has been entertaining and has enabled us to take one step further forward in helping to solve that puzzle of where testing fits within the world of DevOps.