Smoke and a Pancake?

Ok. Blog post number two. Let's start this one by addressing the title. The Austin Powers fans among you will be all too familiar with this line from Goldmember. Admittedly, this tickles me and my mind wanted to come up with something which would grab your attention but also include the word 'Smoke'.

How many times has somebody asked you to perform a quick smoke test? I'm guessing lot's. This has a tendency to be a get out of jail for many companies. The deadline is looming. The word on the street is it's going to be released no matter what - wait a minute - no matter what!? When I hear these sorts of things, my heart sinks. Is there any point in testing something when it's practically a foregone conclusion this is going to be shipped regardless as to what you uncover? To put this in perspective, then you could argue a massive 7-mile wide comet from space colliding with the Earth could put the mockers on the release. A remote possibility perhaps, but still possible nonetheless. I could still find a Showstopper defect though couldn't I eh? Well this depends on how much test effort has happened beforehand to some extent, though it's still entirely possible for a variety of reasons. For example, the test coverage may have been as uneven as the middle lane on the M62 motorway - things could have been missed and there's a real risk of something being identified at the eleventh hour. Or maybe, there's been a shed load of testing done previously but the code is in a constant state of flux - so invalidating some of your previous test effort. Shifting sands so to speak. Maybe a late bug fix has introduced another issue, etc.

Anyway, back to that 'quick smoke test'. I've been in situations where it's left to the individual tester to determine what is tested. In others words, there are no actual smoke tests to execute but rather you are entrusted to ensure what needs to be tested, is then tested. No pressure then. What if my idea of a quick smoke test differs from yours? What if we both performed a quick smoke test against the same build (how quick is quick?) and ended up verifying differents aspects of behaviour and identified different issues. Some far more severe than others. The danger here is that something could still be seen to have passed a 'quick smoke test' but still contain defects you care about.

I'm a big fan of even test coverage. I love it. However, it's something which can easily be forgotten about in the rush to ship as fast as humanly possible. I take great satisfaction in the knowledge that I've tested a build in such a way that I've covered what needs to be covered (in the time that's been made available to me) and most importantly, verified the aspects of behavior the whole team cares about. The primary focus over and above assessing risk is 'value'. If you have a finite amount of time to perform your testing then ensure what you do test is of value. Ensure those common user journeys are behaving as expected. There's a real temptation for testers to become side tracked with all sorts of weird and wonderful permutations and you end up disappearing down a proverbial rabbit hole trying to come up with steps to reproduce, etc. Yet, your Product Owner may not even care about something which a very small percentage (if any) of the audience will discover. What they will care about are those common (I view as 'core') user journeys. I call this the 'Ronseal' approach - does it do what it says on the tin?

To instil confidence I'd always recommend your high value 'smoke tests' are written down somewhere. Store them in such a way, anyone within the team can access them. Anyone should be able to run them. Make them visible and love them. By love I mean 'maintain'. Ensure they are actively updated. That they are still meaningful, valid, and which absolutely must pass. One could argue if any one single smoke test case fails, then it warrants a fix (since something you consider high value has a problem). Whether this precludes the current release remains to be seen but at least you have uncovered that defect. It is now a 'known issue' and has now appeared on your defect management radar (insert a beep, beep sound effect here).

I guess these are a prime candidate for automation, assuming you have an automated solution in place and perhaps even more importantly, assuming that they can be automated at all. Since we all know you can't automate everything. That's another blog post idea right there.

Your high value smoke tests are very important. So important in fact that they are worth their weight in gold. With that in mind I'll close with another Goldmember quote "You see Austin Powers, I love goooold...".