The Politics of Load and Stress Testing

I spent several years at major financial institution as the Performance Test Lead for the Risk Technology division, and we had 50 applications that we were responsible for testing.?Not all of them required testing at the same time, thank God, but we would often have as many as 10 projects going on simultaneously.

Every project followed the age-old format – gather requirements, build a test plan, write the scripts, build and run the scenarios, and analyze the results. ?Since each project usually included a baseline test, load test, peak test, and endurance test, you can imagine how many times tests were run and their results were analyzed. If a test didn’t pass, the issue had to be fixed and the test repeated, ad infinitum.

An analysis consisted of a report generated by the performance engineer with graphs, tables, and a narrative description, sent out in an email to everyone on the team.?The test never simply passed or failed, that was up to the team to decide.?So, an email dialog usually ensued, with people basically voting about whether it was a go or no go but the project manager would make the final decision.?

Some of the discussions went like this:

Martha from QA – “It’s written in the requirements that the test must run without errors, and I am seeing plenty.”

Drake from Dev – “Those 500 errors are outliers; they won’t show up in production.”

Martha – “Outliers, inliers, I don’t care what you call them they’re still errors.”

Bob from Business – “Hey folks we have a tight schedule, and I don’t want to hold it up because of some error our users won’t see.?Can’t we just ignore those?”

Martha – “Ok but what about those response times on the Create New Case page??They’re 20% higher than the SLA so we have to find out why and fix it.?Them’s the rules and we have to follow them.”

Drake – “I’ve always said the SLA for that page was not attainable with the current database configuration.?I vote we increase the SLA by 25%.”

Bob – “Our users will go crazy if they have to twiddle their thumbs for 60 seconds just to create a new case. How much would it cost to upgrade the database??Users have been complaining about slowness for some time”.

And so on and so forth.

What were small decisions often ballooned into big ones, and the nitpickers picked nits while the broad strokes folks tried to overlook them and push forward.??The nitpickers often won because that’s what you expect to happen in a bank. But these discussions could go on for days, I'm not kidding.

You may not have 50 applications to test, maybe only one.?But agreeing up front as a team about what constraints to apply to your load and stress tests, inside or outside the CI pipeline, will save everyone a lot?of time and headaches in the long run. If a test passes, it passes, and nobody has to discuss it or even examine the results if they don't want to.