The Reality of Testing

When discussing the process of software testing, we typically start with the idea of a "requirement". This is then sent to the Developers to create the code and the Testing team to develop meaningful test cases and tests. Once the code is ready, the tests are executed and the results are compared to the original requirements.

In my test automation world, many customers rely upon an integrated suite of tools to manage requirements, tests and results. A typical set up would have User Stories and Requirements in Jira, test cases and historical results in qTest and automated functional/regression tests in Tosca.

Better ideas

This process outlined above is an ideal case. In reality, it seems that this process may only be a rough approximation. At this process occurs over time, there is the danger of "drift" being introduced into the system. As the code is developed, things change. New ideas and brilliant inspiration enter the equation along with the inevitable bugs. The problem is that these alterations are often not reflected back in the requirements...and rarely communicated to the Testing team.

What then happens is that there needs to be some heroic efforts to rework the the tests, extending testing phase and increasing costs considerably. I have heard of companies needing to bring on 100's of additional staff to keep up with the resultant backlog of testing effort.

An additional problem is that if the Tests are modified to allow the actual code to pass, the reports going up to management could be misleading. For example, consider that case where the Requirement says a button should say "OK" but the developer has a better idea and creates a "Submit" button. The tester could fail the test...or modify the test to accommodate the idea (and avoid a dispute with the developer). Unfortunately, if this test result is linked back to the original requirement as a "pass", the system of record thinks the button says "OK". This would not only be misleading to management, it would likely result in poor documentation and perhaps even compliance violations and fines.

Upstream mismatches

Another situation that might arise during testing is that the systems surrounding the Code (Application Under Test - AUT) are in flux. The code associated with a prior step might not be ready...or have it's own "undocumented features". There could be external systems and/or APIs that are not ready or unavailable for a variety of reasons. These challenges can often be lumped together as "environmental issues".

I have again heard of situations where the API challenges go back to the requirements stage. API definitions are often documented in something known as a "Swagger" file (or specification). Developers use the swagger file to govern communications with external systems and form a type of "contract" between systems. Test automation tools like Tosca can import these swagger files and test the APIs.

In this story, the Testers were not able to get the swagger files from the Developer team. (There could be many reasons for this but it is clearly not an ideal situation.) Given that the Testing team still needed to do their testing, they developed tools that would scrape the application's log files for API artifacts and use that to drive the testing. As this process became institutionalized over time, the team lost sight of the fact that this was not testing to the Requirement, but is in fact simply testing that the application does what the application does.

To help address some of these environmental issues, the industry has come up with the concept of endpoint simulation. Testers and Developers first came up with the idea of "stubs". These are bits of informal code that can emulate these external interfaces during testing. Products like ITKO's Lisa and Tricentis' Orchestrated Service Virtualization (OSV) provide a collaborative enterprise ready solution that can be used at scale to ensure these environmental issues are minimized.

Your experience?

What are your real world experiences that deviate from the "ideal"? How do you address these in your world? I'd love to hear your stories in the comments below.