Why Your Project Doesn’t Perform – Or the Subtle Art of Non-Functional Requirements and How They Should be Used in Testing

For some reason, many people don’t seem to “get” Non-Functional Requirements (henceforth referred to as NFRs) and don’t treat them with the respect they deserve. The only time people seem to really care about them is when the system clearly doesn't perform, people are getting very noisy and things are metaphorically or actually on fire. I'd rather we dealt with such things far sooner into the process, ideally from the inception of the project.

As usual, this is my opinion and not that of my employer.

The Problem Statement

I hear three common scenarios, which encapsulate much of what's wrong with NFRs and NFT today:

• The system must perform adequately – Tell us more… No, really do. I’ll discuss why this isn’t sufficient below
• But X is a functional requirement, not a non-functional one – I don’t care, as long as it gets tested. Whining that something isn’t an NFR is no good if somebody then forgets to test it or both sides of the coin insist it’s “not their job”
• The test run was inconsistent, but I think it performed well enough – This is like waving a lucky rabbit's foot at the server and hoping you haven't angered the database gods

All of these are the common smells of bad/lack of NFRs leading to a broken Non-Functional Testing (NFT, which once upon a time was referred to as technical testing) process.

Let me be very clear about this, if you don't agree your NFRs up front, you could end up with an unsuitable architecture. Then you have to rearchitect, overspend on your infrastructure or accept your project will not meet its goals... None of which are pretty and could lead to loss of business, reputation or legal action.

I’ve had NFR meetings in the past where the question of it being met or not has been very subjective, which in my mind is a bad sign. This is a lack of linking NFRs to test artefacts. A tool like JIRA is excellent for this purpose. Lack of an objective pass or fail criteria or other subjectivity in the process not only wastes time, but means that unexposed issues are probably present in your system.

What is an NFR?

Before we delve too deeply into this article, it'll be helpful if we're all on the same page with regards to what an NFR is.

A non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. They are contrasted with functional requirements that define specific behavior or functions. The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture, because they are usually architecturally significant requirements. - https://en.wikipedia.org/wiki/Non-functional_requirement

These NFRs come in many categories, which Ian Sommerville succinctly diagrammed in his timeless book "Software Engineering".

Although this diagram is at least 18 years old, it mostly stands as-is, which is astounding in itself.

NFRs, NFT and Automation

The main difference between types of NFRs really matters when it comes to how you test them. This may occur as part of your manual or automated functional testing or it may be as part of your NFT process. Before we get to that, let’s make sure we don’t confuse “automated” and “non-functional” bleed over into each other, so I prefer to use the following:

• Automated functional testing – Computer driven scenarios that test the system, usually in a single thread with a single execution per test scenario
• Non-Functional testing – Computer driven scenarios that test the system, usually in multiple threads or run over multiple iterations per test scenario

The traditional view of NFT was one mainly around perfomance testing, but that's far too simplistic a view of the world and excludes most categories of NFR.

In order to usefully run any of your NFT plan, you have to establish a baseline, which must be based on the Scientific Method, which goes something like:

• Make an observation or observations
• Define the Problem
• Research - Ask questions about the observations and gather information
• Form a hypothesis - a tentative description of what's been observed, and make predictions based on that hypothesis
• Test the hypothesis and predictions in an experiment that can be reproduced
• Analyze the data and draw conclusions; accept or reject the hypothesis or modify the hypothesis if necessary
• Reproduce the experiment until there are no discrepancies between observations and theory

This predicates two key things: the first being a concrete set of testable parameters and the second being repeatability with consistency. But why are these important?

• Without concrete parameters and constraints, you cannot fail or pass the test, it’s not even a test really until the hypothesis can be disproved!
• You introduce opinion into the process e.g. “the system performs adequately”, adequate is a subjective term
• If you cannot repeat the test with consistent results, you are missing some of the parameters and thus you cannot interpret the results as they have no meaning
• If you cannot repeat the test, you also cannot make singular adjustments and observe the results

If you do not address the above, YOU WILL FAIL; trust me on this one, it’s 100% guaranteed. Also, notice that I mentioned “singular adjustments” above. If you have a failing test, you MUST make singular changes and repeat the test, otherwise, if you’ve changed 100 variables, which one of them fixed the issue?

You must also observe the pass criteria. If you need to reach 1,000 logins per second and you hit it, don’t be tempted to see if you can get 1,500. You meet the pass criteria and you move on, otherwise you could run out of time to address other important tests.

Writing your NFRs

In order to succeed in using your NFRs as part of a successful test scenario pack, they require the following qualities:

• Clear, singular and unique test purpose – Do not try to test multiple concepts at once
• Clear story/requirements – As a Test Case will be written off the back of this, you must be clear
• Clear pass criteria – This must be measurable and 100% objective
• Clear fail criteria (if not part of the pass criteria) – Including any kind of expected defensive outcomes

To expand upon something I mentioned earlier, NFRs must be then backed by linked Test Scenarios. The NFR cannot be marked as complete until the test cases have been executed and passed. Also, your performance and volume based NFRs are best served if you have created a volumetric model first, including growth predictions and peak loads, but that’s a topic for another day.

The exceptions to the above are NFRs like “The entire programme must be ISO27001 compliant” these should be labelled and squirrelled off away from the main effort. Don’t over-think it, just divide up the NFRs into ones that are policy/untestable, ones that are what people may consider to be a functional test (such as usability or session timeouts) and ones that are a traditional technical test (such as the system must allow for 50 concurrent registrations per second).

Who Does What?

I hear a lot of "not my problem" when it comes to NFRs and NFT, which is due to a lack of understanding of their part in a project. I'd expect every member of the team, to be familiar with the NFRs and their status. Have they been agreed and signed off? Are they linked to concrete test scenarios? If they're not, go and speak to your friendly Business Analyst, NFT Lead or Technical Architect. If you are those people, it is your collective job to get this defined and in place as soon as possible (preferably from the bid stage).

My belief is that the Business Analyst, Technical Architect and Test/NFT Lead must work with the client SMEs and Architects as early on in the project as possible. This is a collaboration, as all good Agile should be!

End Transmission

I hope you've enjoyed this little dive into NFRs. Does the above ring true? Don't just take my word for it, have a think about your own experience and maybe comment as I always welcome your feedback!