The Politics of Load and Stress Testing

The Politics of Load and Stress Testing


I spent several years at major financial institution as the Performance Test Lead for the Risk Technology division, and we had 50 applications that we were responsible for testing.?Not all of them required testing at the same time, thank God, but we would often have as many as 10 projects going on simultaneously.

Every project followed the age-old format – gather requirements, build a test plan, write the scripts, build and run the scenarios, and analyze the results. ?Since each project usually included a baseline test, load test, peak test, and endurance test, you can imagine how many times tests were run and their results were analyzed. If a test didn’t pass, the issue had to be fixed and the test repeated, ad infinitum.

An analysis consisted of a report generated by the performance engineer with graphs, tables, and a narrative description, sent out in an email to everyone on the team.?The test never simply passed or failed, that was up to the team to decide.?So, an email dialog usually ensued, with people basically voting about whether it was a go or no go but the project manager would make the final decision.?

Some of the discussions went like this:

Martha from QA – “It’s written in the requirements that the test must run without errors, and I am seeing plenty.”

Drake from Dev – “Those 500 errors are outliers; they won’t show up in production.”

Martha – “Outliers, inliers, I don’t care what you call them they’re still errors.”

Bob from Business – “Hey folks we have a tight schedule, and I don’t want to hold it up because of some error our users won’t see.?Can’t we just ignore those?”

Martha – “Ok but what about those response times on the Create New Case page??They’re 20% higher than the SLA so we have to find out why and fix it.?Them’s the rules and we have to follow them.”

Drake – “I’ve always said the SLA for that page was not attainable with the current database configuration.?I vote we increase the SLA by 25%.”

Bob – “Our users will go crazy if they have to twiddle their thumbs for 60 seconds just to create a new case. How much would it cost to upgrade the database??Users have been complaining about slowness for some time”.

And so on and so forth.

What were small decisions often ballooned into big ones, and the nitpickers picked nits while the broad strokes folks tried to overlook them and push forward.??The nitpickers often won because that’s what you expect to happen in a bank. But these discussions could go on for days, I'm not kidding.

You may not have 50 applications to test, maybe only one.?But agreeing up front as a team about what constraints to apply to your load and stress tests, inside or outside the CI pipeline, will save everyone a lot?of time and headaches in the long run. If a test passes, it passes, and nobody has to discuss it or even examine the results if they don't want to.

Mauricio Medina

CEO at Consequor Tecnologia

1 年

Será que isto n?o acontece por aqui?

回复

要查看或添加评论,请登录

Randy H.的更多文章

  • Online Tsunami

    Online Tsunami

    In 2009 a company called Lockerz was launched by Kathy Savitt, the sister of the man who was CEO of Amazon Web Services…

  • To Stress Test or Not to Stress Test, that is Congestion

    To Stress Test or Not to Stress Test, that is Congestion

    In the early days of web load testing, it was commonly assumed that you only had to do it if your site was expecting…

  • JMeter Results Analysis with Robust Error Handling

    JMeter Results Analysis with Robust Error Handling

    In this example I’ll be using the campsite booking app I’ve referred to earlier. First, we define the usual performance…

  • Separating the Wheat From the Chaff in JMeter Stress Test Results.

    Separating the Wheat From the Chaff in JMeter Stress Test Results.

    As I laid out in the previous article “Why Error Rates Are Overrated”, having a load or stress test pass or fail in the…

  • Why Error Rates are Overrated

    Why Error Rates are Overrated

    An automated test that requires someone to examine the results to see if it failed is not an automated test, it’s a…

社区洞察

其他会员也浏览了