Test Automation- Silver Bullet? Or Can Of Spinach?

A Silver Bullet, by definition, is "a simple and seemingly magical solution to a complicated problem". This is what Test Automation has become to most people; the silver bullet for all of their testing problems. I think Test Automation is more like a can of spinach. In the right hands, it's magic just ask Popeye: it strengthens test solutions, it resolves problems, it helps you get the girl. In the wrong hands its mush, unpleasant, unappreciated mush: it's ill-placed (like between your teeth), sloppy, and for goodness sake, canned. Who cans spinach?

So why does everyone regard Test Automation as the silver bullet? As more and more companies move towards iterative development models, the increased amount of testing required to deliver high-quality products puts more strain on already strained budgets. And this is where things start to go spinach-shaped. Before the great Agile/DevOps movement, Software Testing in its entirety was a grudge purchase. Enter Agile, cape fluttering in the wind, the methodology designed to solve all of our problems. Except for cost.

Agile is notably meant to reduce the cost of failure. As far as testing is concerned, it does not address this particular cost factor. In fact, if you are doing it right, you are going to pay far more for quality in an Agile environment. Simply put: more build releases=more regression=more testing. It's now that we realise that the two testers we have allocated to our development team of ten might not be enough. Especially when there are 13 856 regression test cases (and growing) that we need to execute every time we release a build. Test Automation becomes a non-negotiable – if we want to do this right, we need to start automating. We need this silver bullet!

One of the most significant issues when we first start with agile is not addressing the size of the test team, but we require more testing more often. Automation makes sense, but does it solve the fundamental issue of under-resourcing we undoubtedly have when we first migrate to Agile? Not to mention the tremendous technical debt we generally start off with, like having a regression pack in place but no automation.

The first solution everyone has to all of their problems is to start automating upfront. We need to adjust the types of test resources we have, including our methodology. Test Driven Development? Behaviour Driven Development, maybe? But then does everyone else prescribe to this model, or is the test team alone in this? This is sometimes the case; the testing team follows formal Test/Behaviour Driven Development practices, but the developer isn't involved... how does that even make sense? As soon as we change to a technical testing model, we immediately require resources with programming knowledge, so we hire traditional Test Automators.

The problem with this is that the majority of Test Automators don't actually have a testing background. They are not formally trained in Test Analysis and Test Design techniques. As a result, we have automation up front, but it is more of a "user acceptance" test than a proper functional test, so we have far more defects making it into production.

Our other flavour of spinach is not quite as complicated but is definitely far more messy and rife with dodgy preservatives. We often get requests from prospective clients to quote on an automation tool. So we do our due diligence and ask if we can assess their test landscape for the best fit... only to find out that there is no test landscape. There isn't even a test team. There is no regression pack, there is no testing process, there is no test tool, BUT there IS some user testing. Go forth and conquer!

This is the most traditional of the silver bullet requests. But, it is difficult to get people to understand that automation will not add quality if you don't have a quality process in place. Automation will not speed up testing if the automators have to do all of the analysis work and test design first because it doesn't exist.

Forget crawling before you walk... these people want to run like Usain Bolt, straight off the delivery table! Unfortunately, this is the approach that is destined for failure. It's like being on that first date and smiling at your date for hours with creamed spinach between your teeth.

We regularly get approached by potential clients about our test automation solutions. We are very proud of our Automators and the wizardry they produce. We are proud of them because they aren't just Test Automators... they are proper Quality Engineers (QEs). They understand the fundamentals of quality and have all, at one stage or another, been manual testers (trained in the fundamentals of testing). In fact, they still are because QEs do EVERYTHING and inherently understand the value of testing and how it fits into the bigger picture.

Our team are first and foremost analysts, they don't just automate solutions. They analyse the big picture and custom-make a solution – fit for purpose – that gives our customers the most bang for their buck. They also don't work in isolation. They work very closely with the entire team, ensuring seamlessness between where manual testing ends and automation begins. If you were to ask our QEs questions about the percentage of automation coverage a particular regression pack has, they can answer you. If you were to ask them how much execution time has been reduced, they can answer you. If you were to ask them to fly around the room backwards, they'd tell you not to be ridiculous because they wouldn't be able to see where they were going. Our QEs are the IT world's Popeyes, and their "spinach" is fuelling our testing processes and procedures.

Be that as it may, our QEs can only do so much. Testing is, regardless of how you look at it, context-specific. If you don't have your quality policies in place and they're not adhered to from start to finish by the entire team, you are destined to be the guy on the date with spinach teeth. And no silver bullet is going to fix that.

If, however, you understand the value of quality, are prepared to make the initial investment to get it correct from the beginning and not take shortcuts, your automation frameworks are going to be everything they were prophesized to be by the Agile guru's out there. Then you get to walk away with Olive on your giant arm while whistling around a pipe.