On the experiment to COLLAPSE COGNITIVE BIAS (in support of decision making in the oil & gas sector)

Guy Loftus (K2V Ltd) and Marc Bond (Rose & Associates), March 2019

For the last 2 years, K2V Ltd has been experimenting with crowd-sourced opinion to establish if the “wisdom of the crowd” can help to support investment decisions for the oil & gas sector of the 21st century.  The experiment employs a technique called “Knowledge Stacking” which was launched in 2017 [1] using a simulated data room of a real prospect made available online and presented in plenary sessions around UK universities. The premise was simple: does crowd-sourced opinion of an opportunity converge on the actual value of that opportunity?  What has emerged is a gap between participants’ automatic responses and their evidence-based opinions. This article lists some preliminary results without giving away too much to avoid distorting ongoing participation.  A more complete article can be accessed by clicking HERE[2], which incorporates method and a discussion.

BACKGROUND

Estimating the number of almonds in a bottle (a singularity) is a task that is hard for individuals to get right [3] but is very easy and normally correct if you average crowd-sourced opinion using some basic ground rules [4].  The reason it is hard for individuals is because a single deterministic estimate is only part of a much broader forecasted reality which the crowd is adept at scoping.  The singularity lies within an uncertainty range that can only be properly scoped when a large enough sample of independent guesses have been made that are equally informed (or ill-informed).  In the absence of any hard evidence, we are free to make a quick guess using our gut-feeling unencumbered by the need to over-engineer the answer; with nothing to lose, we do it for fun in the hope that we have defied the odds by being correct.  Reflexive responses using gut-feeling make quick work of single estimates but switch to slower, more demanding (reflective) ways of thinking when presented with complex uncertainty [3]. 

There are just three attributes (simplified here) that contribute to uncertainty when counting almonds in a bottle: the internal dimensions of the bottle, the average almond size and the pore space.  This creates a multi-variate distribution of uncertainty, which, even in this simplified form, is difficult to estimate without hard data:

Number of Almonds = total bottle volume / the volume occupied by an average sized almond + pore volume

When tasked with guessing the number of almonds in the bottle, our reflexive thinking instantly collapses all the attributes in to a single question to deliver an estimate, which is inherently inaccurate [3].  But what if we were to de-construct the estimate based on its component attributes, ask separate questions for each attribute (1- what are the internal dimension of the bottle? 2- how big is the average almond? 3- what porosity can we expect?), “stack” the deconstructed attributes using the formula above to calculate the number of almonds and then compare that with the original guess?  It is tempting to suggest that you might be comparing a reflexive response to a complex problem (a quick, possibly unconscious response rooted in instinct, experience, mood and intuition) with a reflective response (a slower, more controlled response rooted in rational, logical, critical and deductive thinking). But that may not be what is happening here.  

INITIAL RESULTS

The standard (to some of you) chart above shows the total number of participants to date plotted on a geo-commercial versus geo-technical scale, each bubble representing stacked individual scores based on presented evidence.  The bubble colours denote current roles, with the colour of the “halos” describing the gut-feeling for the prospect.  The total population is distributed about a mean (white bubble) in what is likely to be a non-uniform way with no obvious clustering.  Note that the choices of reflexive gut-feeling are the same for the evidential class limits of optimism.  Despite that, the definition of the “halos” frequently do not match the position of the bubble on the same scale of optimism e.g. some claimed that they felt “bullish” when doing the survey but their evidential scores demonstrated that their analysis was actually “neutral”.   That disparity represents a gap between deconstructed thinking (stacked from 15 evidential metrics) and reflexive thinking (gut-feeling). Those whose reflexive estimates fall in to the same class as their stacked estimates are called here realists.  Those whose stacked estimates are more optimistic or less optimistic than their reflexive estimates are referred to here as optimists and pessimists respectively*.

As results are still coming in, we are unable to reveal too much without distorting the outcome but some preliminary observations can be shared:

• The gut-feeling of two thirds of respondents disagrees with their stacked evidence-based opinions

------  The percentage of realists, optimists and pessimists are roughly evenly split at around 33% each. The percentage split changes with the level of experience of respondents

• Approximately half of the participants plot in the neutral "safe" zone

------  The other half are more or less evenly split between "opportunistic" and "cautious", with a slight emphasis on cautious

------  The results probably conform to non-uniform distributions

• There is no discernible difference between early responders in plenary sessions (who took less than 5 minutes to complete their scores) to those who took 15 minutes or more
• Different disciplines and different demographics exhibit different biases but on the whole, the range of evidential outcomes are strikingly consistent

------  It is estimated that a minimum of fifty responses for each peer group is required to converge on a stable aggregate outcome 

• Despite having 7,353 opinions offered by 387 geoscientists from around the world, which is sufficient in number to test whether crowd-sourced opinion converges on the truth, the experiment continues to expand the diversity of opinion to examine the role of demographics on those outcomes

CONCLUSIONS

The oil & gas sector relies heavily on expert knowledge for new business development.  But as geoscientists, we are only too aware that the right answer to complex problems is not simply a case of "ask the experts", because that assumes that the experts agree on the answers.  To consult the “wisdom of the crowd” effectively, we need to preserve all views (however extreme), ensure that they are independent (no one influences another’s opinion), that they are decentralized (we don’t all have the same experience) and that we have the means to aggregate.  If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out [4].  The method of aggregation previewed here demonstrates that crowd-sourced opinion delivers results that are both rational and measurable.  The question under-pinning this investigation is what role does expertise play (if any) in making the right decision for any enterprise.

This abbreviated article is a call for more participants in the experiment– we are short of opinion from people in their early and mid careers as well as geochemists, geophysicists, petrophysicists, economists, production geologists, asset managers and many others.  If you would like to contribute your opinion to the experiment anonymously or to know how you score, click HERE.  If you would like to arrange a group plenary session amongst your peers, please email us on [email protected].

REFERENCES:

[1] Knowledge Stacking [www.k2vltd.com/article-6]

[2] Collapsing Cognitive Bias [www.k2vltd.com/article-21]

[3] Kahneman, D, 2011 – Thinking Fast and Slow. [Farra, Strauss and Giroux, ISBN 978-0374275631]

[4] Surowiecki, J, 2005, The Wisdom of the Crowds [Anchor Books ISBN 978-0-385-72170-7]

**The terms realist, optimists and pessimists have implications for attitudes that are potentially misleading. These will be redefined in the future when we understand what they really mean.