The GOLF Heuristic - Go One Layer Further

I had a long, interesting conversation with Wayne Roseberry on various topics. One of the pieces of conversation was on The GOLD Heuristic . He asked shouldn't Go One Layer Up be also important. I agreed. I had a separate model for that. However, after this conversation I thought to revise the GOLD heuristic to accommodate this duality. That's why I like conversations with good testers. They help you see your blind spots. Thanks Wayne!

So, here I introduce the GOLF Heuristic for the first time.

Go One Layer Further

The Direction of "Further" is either downwards/upwards, outwards or inwards from your current position. Both directions can co-exist from your current position. The idea is to think about what you are testing in terms of Test Action Layers .

When I say position, I mean the current Point of Primary Control . Or you could think a bit more open-ended like me, you can imagine yourself flying to a box in an architectural diagram (actual or notional). Just be there and look around while keeping the notion of direction w.r.t. the overall architecture in mind. I use this trick all the time - boxes and directions, that's all there is. That doesn't work? Become the test. Look around yourself in the context of the position in test object where you sit. That doesn't help? Then find a way and share with me :-).

As a tester, this is one of the most useful ways to get more ideas for testing as well as breaking free of constraints that prevent you from exercising some test ideas.

For example, rather than going for blind boundary value arithmetic, you will start thinking in terms of the boundary itself. You will explore undocumented, notional boundaries and so on.

Going One Layer Down/Inwards

I have found this very useful when I need to break the constraints which a Test layer puts on exercising test ideas. The web interface, for example, puts such restrictions via the client side scripting that prevent experimenting with various types of inputs; the browser puts a restriction on the URL size; the flow of an application via the browser puts a restriction on playing with the sequence of operations; a local GUI/CLI interface puts a restriction on the sequence of API calls and so on.

The concept is simple, although from one implementation to another the ground work required might vary.

There are two variants of this:

Variant 1 - Direct Access to One Layer Down

Here, you have access to the underlying layer to see the exact implementation.

While testing a web application, applying the GOLD heuristic could mean understanding the underlying protocol (mostly HTTP) and how the functionality of the application could be making use of the same.

As an example, the moment you know that HTTP is a stateless protocol and the application under test requires a unique business flow be established for an end user, you know that there would be some sort of state management in play and one or more of the headers/cookies/hidden variables could be used to depict the same.

In such a case you can do a good amount of testing by skipping the web interface altogether and talking at the HTTP layer to the server. This can be done as tool assisted exploratory testing using web proxies like Fiddler/WebScarab/BurpSuite.

This style of testing is an example of shifting the Point of Primary Control one point further in the Action Layers.

It can also be automated using tools or custom-built modules that talk at HTTP level. This is a great supplement to the GUI-click/mouse emulation tools commonly used in test automation.

Variant 2 - No Access with Actual/Notional Knowledge of One Layer Down

In contrast to the example in Variant 1, sometimes you may not have the option of directly talking to the application logic at the underlying layer. The GOLD heuristic stills comes in handy.

In such a situation, you gather information beyond the current interaction layer by asking questions about the same from the rest of the team, studying similar applications, looking through the various possibilities in which the actual implementation could exist and ruling out the same with careful observations.

Let’s take the example of a command line interface. Some common tests which a tester conducts are duplicate switch testing, non-existent switch testing etc. If a tester is able to think how internally the different command lines switches are going to be represented, s/he can come up with a precise list of tests. E.g. if the internal data structure that is used to represent the switches is a dictionary, the test for duplicate switches is not as important ( as the dictionary would disallow duplicate keys). Duplicate boolean switch testing would also be purposeless. The duplicate parameter switch testing becomes important as one would want to know whether the second entry is discarded or is used to update the parameter switch option value, in which case one should see the second value overriding the first one.

Another example of the GOLD heuristic in such a case is thinking about queues and servers in the underlying logic. One need not know the exact implementation. Just acknowledging that there are various points in the execution where queuing takes place, a tester can come up with questions like – how can the queue get filled up and what happens then? Is the queue size enough for anticipated usage scenarios? etc.

Thinking in this manner helps testers in coming up with questions that lead to powerful test ideas.

Going One Layer Up/Outwards

When you are testing at the lower most layers e.g. a component or its integration with another one, or a class or a function etc, as a tester you suffer from the same blindness/bias that a developer does. This layer is far far away from the point where the user needs were translated into some documents, diagrams and models. These DDMs were further translated into technical specs or still further into some mid-low level design details.

A lot is lost in this translation. Some have tried to solve this problem as a process + technology solution with concepts like continuous integration, customer demos etc to build the bride between business and technology continuously. However, as a tester I can tell that the subtle bugs, assumptions beyond published interfaces still remain until they cause a mess.

Just like the downwards direction it has 2 variants:

Variant 1 - Actual Knowledge about One Layer Up

For example, at a component layer, thinking in terms of public interface of the component vs the integration needs of the other components will give you various insights into the usability, performance, security etc.

Some examples of missing this approach is:

• Focusing too much on just the technical details without paying much attention on how a component, class is invoked by the actions of the end user.
• Assumed data like current timestamp treated as a locally calculated value rather than provided value.
• Random boundaries assigned to internal state variables e.g. queue lengths without considering all the environments or usage scenarios.
• Input interfaces ignored. This especially becomes critical for identifying the complete attack surface.

Variant 2 - Notional Knowledge of One Layer Up

At especially early stages of development, when the top layers are not available or only partially available, you will need to rely on available information, conversations and exploration.

You will need to rely on notional top layers. This style of thinking will anyways help you testing those layers when they become available.

Some Further Thoughts

Beyond the generation of test ideas, going one layer down helps a tester in having a high level logical system view of the application under test. When a tester can see a system as a combination of various smaller components, it gives clarity of thought and while communicating with rest of the team, s/he can communicate with precise information or atleast educated guesses about which part of the system might be causing the problem.

Thinking one layer up helps questioning the way input is assumed. The role of tester is not to just act on inputs as seen in these components and protocols. The role also asks us to question what is not being considered as input. Techniques for test design come later.

You can as well use the GOLF heuristic recursively – apply it at one layer further (top/down), then further and so on. You can also think in both directions.

Try it out and you might find it useful. Your experience could be different, it’s a heuristic after all.

That's all for now.

Some related articles:

Performance - Being Lazy or Greedy - Response Time, Resource Utilisation and Locality

AEIOU Model Explained 1 - Test Actors, Test Object, Participants, Layers and Points of Control