The necessity of continuous system and performance testing.

The importance of requirements

Requirements drive everything. Being agile, lean or whatever you wish to call it does not dispense you from that quintessential fact. 

You test against requirements. If your requirement is inexistent, vague or god forbid, wrong, the best vision, developers, testers, automation, methodology, tools, technology can do nothing for you.

Requirements need to be specified and understood at every level. A good requirement can be explained in such fashion than a smart 5 year old can understand.

Requirements are the driving purpose of testing, do not forget that. It's a key factor for a successful lean development philosophy implementation. 

The backlog trap

Continuous delivery makes lean development possible, however, it does not mean delivery over quality.

Often times (that's an understatement) as teams go for the "delivery first" they pile up defects and end up doing damage mitigation by managing defect backlog and risks.

By increasing the backlog of defect you increase not only the time between when a defect is introduced, detected and/or fixed but more importantly, you also increase the complexity of understanding the issues exponentially.

As the contingency of undetected/unfixed defects rises , it prevents you from detecting/fixing secondary and tertiary defects.

Testers end up with the task of understanding increasingly complex issues and developer face the even more daunting task of addressing these new issues nobody understands. These new issues can't be troubleshot because of their complexity as they factor multiple distinct defects which are clogged together.

In software development and testing we must build up to complexity rather than build from it.

Do the team a favor, break free from the defect management trap.

Here’s a simple mathematical fact, if you introduce more defect than you fix, your backlog will grow.

If you continue delivering while your backlog keeps growing it will reach the point of no return. A defect threshold where the defect backlog management and risk assessment becomes your predominant issue, that's the backlog trap. You'll lose the ability to detect and assess defects efficiently due to their increasing complexity.

From that point on, shall you keep continuous delivery your quality will be lower and lower and you won't have any efficient path to address your quality issue.

You will have to stop new development and solely focus on your forest of defect. By then the complexity of your most critical issues will be so high and the time which they have been introduced will be so far that fixing them will cost you much more.

Continuous testing and not only unit and functional tests, but continuous system test is a vital part in avoiding the backlog management trap.

Testing is not a process that comes after development, testing is a core component of the development itself. 

The cost of quality is at its cheapest at the beginning of the development cycle. 

Continuous system test

Continuous system tests which are hands free and effortless build a feedback loop into the development process.

They provides accountability. A level accountability that is a notch higher than the functional "you better not break anything" modus operandi.

In addition to knowing if something is broken (functional aspect) you want to know if things have gotten slower, or use more resources (nonfunctional aspect).

Software performance does not improve easily, software defects multiply on their own, and they will bring performance down even if they don't breakdown the software functionality .... or at least not immediately.

The more commits are made to the code, the more defects are introduced.

The earlier you are able to detect these issues the more efficient you will become at fixing them.

While it is true that you can't test everything all the time, you can test a lot more than you realize you can.

And here's where you need to start to achieve that.

Start by having automated tests that focus on the primary question you want to answer.

Start with one or multiple tests which are focused performance tests.

Focused performance tests:

The focused mode of performance analysis is concerned with how a product performs with a workload made up of uniform requests that are static in their count, complexity, and concurrency.

Tests employing this aspect are designed to measure how a product performs with regard to a specific type of work that it’s intended to handle.

A focused performance test usually involves:

• A single transaction type
• A single transaction/file size
•  A specific complexity of transaction
• A single, non-varying, workload level

Once you have accomplished a satisfactory level of focused performance tests. That means you have a full suite of focused tests that runs without any human intervention several times a week against the latest development build.

You can graduate in complexity and build Comprehensive performance tests. 

Comprehensive performance tests:

The comprehensive mode of analysis involves measuring behavior of the system under test while simulating the conditions that a product might experience during the course of real-world operation.

A comprehensive performance test usually involves:

• Multiple transaction types
•  Multiple transaction/file sizes
• Multiple transactions of different complexity levels
• Multiple varying workloads

Just as with focused performance tests, each comprehensive performance test seeks to answer a single question.

However they are slightly more relevant and interesting questions such as:

How quickly does my product complete a backlog of work which simulates a system that has been down for two hours?

What’s my average throughput with a workload that simulates 24 hours of expected real-world use, including a varying workload from nominal through peak and back?

With comprehensive performance test we step into the world of interesting questions that we want to answer when we want to plan our capacity.

Remember you can’t start with comprehensive tests as we graduate to complexity and can’t build from it.

The importance of good automation.

Interesting how subjective "good" can be, here's what I mean by good automation that is hands free and effortless as I explained in a previous article.

Why is automation key to software development?

Additional problematics of scalability, resilience, availability or capacity planning are not covered in this article.

I'll cover them in future articles.