Performance Stats - Why Average is Wrong

This came up recently  in a discussion. The idea was, is anything average can be used as a metric of something as serious as a performance benchmark? We have all seen stuffs like this :

Do they actually mean something? Do they mean anything at all? Unfortunately, for all the common people reading this graph, they simply lack the foundations which are needed to interpret these sort of data. That is partly also because these data are not what is termed as meaningful by any professional.

Observe that, when some system runs over an input, the time taken to produce outcome varies not only from input to input, but also from time to time. Formally, then, given the same input, the time taken can be any value, even an infinite value. That qualifies the time taken to respond as a Random Variable. ( These random variables are not variables, per say, they are a very special type of functions called : Measurable Functions, if you are mathematically inclined ).

What are the examples of a random variable? Well, the result of a die throwing works, so is the result of a coin toss. Suppose there is this guy standing at some point. A coin is tossed, and if head comes, he moves left one step. If tails appear, he moves right one step. The position of the guy from the current base point is another random variable, and this specific phenomenon is called Random Walk.

There is a hypothesis that the stock market price follows random walk. These things are important to understand performance of a system, why? Because the current queue size on a system under test is nothing but a random walk. It is easy to think why it is, and thus I am omitting it. Just think that there is a chance that a new request will come per time slice, and that there is another independent chance that a request would be fulfilled and dispatched to client. The balance is stochastic in nature. 

So, how to characterise the time taken to respond back to the client? As it is obvious, the only characteristic of a random variable is it's Density Function. The density function can be experimentally obtained by the methods of the histograms - observe the below image : ( which shows why average is bad)

But the image is wrong. To actually generate the density function, one needs to normalise the data, that is, needs to plot the densities over time slices. The time slices read - between 0 to 1 sec, x_1 % of request, between 1-2 sec , x_2 % of requests.... thus, between n to (n+1) sec, x_n % of request. Thus, a curve can be established over histogram, which is known as the pdf. Interestingly, a PDF, f(x) has a very nice property because probability is a measure normalised to 1 :

Also,  more interestingly, it tells you, between time T1 to T2, how many requests ( in percentage ) are done, rather what is the probability that a response time would be between T1 and T2  :

That, is precisely what a pdf is known for. This also gives a glimpse on what we should actually measure, given there is a pdf. Suppose we are having a real time system -  that would mean, I would have a deadline Td, below which we we should have responded back. What we would really want, is to take almost all of this pdf to the left side of the deadline Td, that is :

Note that, this holds because no system should be able to respond back in less than 0 time. Now, what do we mean when we say approximating 1? We probably mean ( pun intended ) that the probability that the system responds before Td, is almost 1, which in most of the practical scenarios are not to be. So, we do an engineering trade off, we choose a critical probability, instead of 1 :

 Most of the cases *people* have chosen this Pc as 0.9, or in English, it is called 90 percentile mark. Where they would be useful? Everywhere. The formula is self sustaining, it basically says, when you are giving 90% data : 

"90% of the time, the response time taken for an user would be less than or equal to this time : T90"

Hence, it is the only viable idea to reduce performance metrics to a single scalar. There are fancier stuffs, to find the spread of the distribution, to check if it is not really a long tailed one ( see below image) but they are too fancy stuff. 90% works, and works well for 80% of the applications. For more fancier applications, we can choose to use 99%, or rather if  you are into the whole idea of  six sigma, then 99.97 %. 

Choosing these parameters are not of much importance here, because they are a matter of choosing what suits your need. But one thing for certain, averages do not work for you, ever. A much better alternative would be Median value, which is nothing but the 50%, thus giving you a 50/50 chance. Choosing median makes your Pc as 0.5, while choosing 90% makes your Pc as 0.9. Now, take a look around the poster image of this post. Given the curve is that way, average would most certainly fail to showcase any meaning. Averages also fails in the cases of asymmetric distributions, including long tailed distributions.  In case of Poisson distribution, there is only a single parameter, which is the average, and which defines the distribution, hence, the 90% is a function of the same parameter.

Addendum:

Normally the craze of a post wanes away over time. But looks like this particular post is still alive and proverbially kicking some very old long held spurious belief systems. Thus, I would stoop to down from the theorists to an performance minions level - and showcase real practical example of what is really happening. Here is my favourite ( and created ) nJexl code to perf-test Bing.com :

all it does : calls 30 times bing server with the query "average", to showcase the following result on response times (this is do it at home yourself stuff): 

thus, in short, the claim that a stable system generates responses which are not much different from the average is a very wrong one. The notion comes when one does not comprehend the Dirac Delta function, which  is a limit to 0 centred Normal distribution. Averages are almost surely a bad statistic of a population. And yes, I am telling this fully knowing the statements of law of large numbers and Central Limit Theorem. Observe that in reality, 90% would significantly differ from Max, as well as average.

Thus, the summary:  Do not use average, anywhere, ever in performance testing. That is the worst cardinal sin one performance tester can commit. Something is easy to calculate, does not mean it would be meaningful, and in this case it is not. But averages are important too, when we have to go with the Martingale Hypothesis, generally it does not appear in performance testing (you can not have an unbiased random walk as the model of the queue size build up, that is not engineering).