Performance - Being Lazy or Greedy - Response Time, Resource Utilisation and Locality
What do we mean when we say a given object performs well?
This is an important question for testers, because unless this is answered for a context, one can not design and interpret a test primarily focused on performance, or derive meaning of performance from tests that are not directed primarily at evaluation of performance.
Response Time / Latency / Delay
These terms represent the same notion of time although with a little different way to think about it:
How long does a test object take to perform an action?
The difference is subtle as asking "What time is it?" vs "How late is it?".
It falls into the category: The lesser the better.
Resource Utilisation
To performance an action, there is a cost in terms of resource utilisation.
An action could be processor-intensive and/or memory-intensive and/or network-intensive and/or disk-intensive and so on.
It falls into the category: Optimum is better.
The Reality of Performance
There is a problem with nice-sounding words :-) and the obvious. Let's have a look:
Do you want an object to perform an action in the least possible time?
Do you want the object to perform an action with the least resource consumption?
You will end up answering "Yes" for both these questions.
That is without realising that these goals are mutually conflicting goals.
If you want less response time, you might end up increasing resource utilisation or spending on a costly resource. For example, the nature of a disk will determine the read time. Whether something is stored in RAM vs disk will have an impact on access time. If you want to tune the object for better response time, you end up spending more on the resource.
If there is a limit on the resource budget or constraint because of the nature of the device or because the resources are outside your control (e.g. a local software deployed on a user's system, a mobile application etc.), then response time is a goal governed by these constraints. The response time goal here may not be the best you would want, but the best one you can manage to deliver.
领英推荐
Locality
Somewhere in the equation of response time and resources is another factor - how "local" are the ingredients which a test object needs to perform an action.
For example, if the object wants to do a + b:
The more local (close to each other in time and space) you make the above two, the better (lesser) is the response time.
This is documented as The Locality Principle in Connie and Lloyd's body of work.
This is one of the most important generic principles I ever learnt in performance engineering.
Think how any web application by its very nature breaks the Locality Principle by bringing in the latency introduced by the network between the browser and the server. Now, think how client-side Javascript is trying to cater to Locality principle. How the larger apps via local data centers (google.com -> google.co.in) try to reduce its impact. Think about caching.
Lazy vs Greedy
A related concept is whether a logic is lazy or greedy in terms of whether it reads & processes all the data at once and loads it in a resource with faster access or does it do the other way round - only process a chunk of information at a time.
Evaluate SAX vs DOM for XML parsing in this light. Which one is lazy/greedy?
Look at generative logic in Python. Is it lazy or greedy?
Food for thought: You want to read a 10 GB text file on a machine with 4 GB RAM. Using programming, how will you write a program to solve this? What is the nature of this logic? Lazy or Greedy?
Thinking Lazy or Greedy is basis of design of a lot of stuff. When one designs test automation frameworks as well, this thinking is critical, but I don't see it being discussed in testing community.
That's all for now.
Some related articles: