Clarity and the Agile Testing Pyramid

I have been in the software testing space for the past 20+ years and never experienced greater clarity regarding the testing challenge than after a session with Wolfgang Platz, the founder of Tricentis. 

Recently, Wolfgang hosted a Masterclass on testing where he discussed his vision of the Agile Test Pyramid.

Part of my confusion over the years has been the great variety in testing approaches adopted by my clients. Much of this I attributed to the pendulum swinging between centralized and distributed testing and the move from waterfall to agile software development. But there was also the broad adoption of Selenium and other open source tools versus embracing commercially available tools like Tricentis Tosca and my old favorites, WinRunner and QuickTest Professional (QTP).

As illustrated in the original testing pyramid, in days of waterfall development, testing was a daunting task relegated to the final phases, far removed from the actual development activities. Testers were required to infer extensive "end-to-end" use cases from documentation that was likely outdated and was not well suited to automation. None the less, in an attempt to improve efficiency, "record and playback" technology was embraced in an attempt to reduce the efforts.

As a result, commercial testing tools seemed to have their heyday when centralized testing teams were attempting to tackle this enormous workload. As the pendulum swung over the years to a decentralized QA model, development managers would tell me "I want my developer to do the testing" and "I want to hold my developers responsible for delivering quality code." Looking back, it only makes sense that these "testers" would embrace a coding-centric approach to test automation.

The testing envisioned by these managers is really limited to "unit testing", answering the question "Does my individual piece of code work?" Combined with a move to agile development, this momentum has actually flipped the testing triangle. However, as Wolfgang illustrates in his updated pyramid, unit testing remains only a part of the problem.

Above Unit Testing are Integration Testing, System Integration Testing (SIT) and end-to-end User Acceptance Testing (UAT). These are typically not the realm of developers. As you move up the pyramid, the participants change to SDET (Software Development Engineers in Test), architects, business testers and, ultimately, actual business users.

While developers may be able to approach 75% test coverage with Selenium at the unit test level, you will almost certainly never be able to accomplish this at the higher levels. There simply aren't enough people skilled in Selenium coding to initially develop the tests, let alone maintain and run them. This explains why most organizations only report overall test automation coverage rates closer to 15% (per the 2020-21 World Quality Report).

Opening up test automation to non-developers is exactly what the first generation commercial tools like WinRunner, Segue, Compuware and later QTP/UFT tried to enable. Their "record and playback" capabilities were meant to let non-programmers quickly and easily create scripts. And they worked well at doing this…but quickly exposed another challenge. Dealing with change.

Change comes in a number of forms. As applications evolve, the scripts need to be maintained. Keeping up with this is a huge effort. But even developing re-usable scripts the first time with a stable application means dealing with changing fields, potentially hidden session ID's and a myriad of other items that needed to be addressed. Ultimately, these required the tester to tweak the script (ie code) hidden behind the tool's "business friendly" user Interface. It once again required a programmer.

This is where Wolfgang pops up again. The fragility of these "record and playback" scripts was ineffective. Most organizations struggled to achieve 20% test coverage in the Integration, SIT and UAT layers of the pyramid. There had to be a better way…and he figured it out. (Read the full story here.) Wolfgang's "model" based approach abstracted many of the problem with scripts but retained all of the benefits. Users were now able to regularly achieve 90% test coverage…and test maintenance issues were greatly reduced. 

But even Wolfgang acknowledges that you will never be able to achieve 100% test coverage. Some things change too fast and there will always be some manual testing. While I have always known this to be true, I think his annotated pyramid illustrates this quite well.

 The reason I find clarity in Wolfgang's pyramid is that is illustrates the various types of testing while simultaneously explaining where API, UI and manual testing fit into the picture. It illustrates why solutions like Tricentis Tosca will not likely replace Selenium and other scripting tools in an organization, but explains why Selenium will never be the solution to the entire problem.