Reference-class forecasting for oil and gas exploration

Probabilistic prediction of exploration volumes suffers from four fundamental systematic biases. These are

• Optimism (P50s predicted too high - this is common)
• Pessimism (P50s predicted too low - this is less common, but not unheard of. It sometimes comes about after compensating for having been too high)
• Over-confidence (P10/P90 ratios too small - this is very common)
• Vagueness (P10/P90 ratios too large - this happens; usually an artefact of mathematical malpractice, but occasionally motivated by KPIs based on delivering volumes in range)

Over-confidence is often mistaken for optimism because when our ranges are too small, we tend to fall outside of them and because of the skew in volume distributions, we are more likely to fall on the low side. Conventional lookback methodologies often miss these distinctions, especially if they don't include uncertainty ranges.

The theory behind this taxonomy of bias is explained in detail in my article Precisely Vague, as well as my presentation What it costs to be rubbish at risking. This article is about how to compensate for them.

Cassandra's quantification and correction for bias

Cassandra has been tasked with putting together a probabilistic prediction for the year's drilling programme. Unfortunately the results of the last lookback revealed modest, but unequivocal biases in the teams' predictions. How does Cassandra correct these biases?

For speakers of Math, the mathematics are set out at the bottom of the article (after the footnotes), but the essential idea is to build models for how optimism / pessimism and confidence / vagueness distort the probability parameters we use to describe uncertainty. Both bias models are described with a single number, respectively the factor by which we over- or underestimate P50s and the factor by which we under- or overestimate P10/P90 ratios.*

Starting with optimism, by adjusting the sequence of delivered volumes using the (biased) predictions, we can generate a sequence of numbers whose mean is exactly the bias parameter we need for our model. A standard Bayesian analysis of this sequence gives us the uncertainty distribution on this parameters, such as the one shown here from Cassandra's lookback. The optimism bias parameter is the logarithm of the factor by which P50s are over-estimated, so Cassandra's teams typically over-estimate P50s by a factor of around 1.5. This is not unusally high. Once the bias parameter has been extracted in this way, it is straightforward to incorporate it into future predictions (see below)

There are two ways to correct the variance. The first pessimistically assumes that teams are so dismal at predicting these ratios that it is better to derive variances directly from the historical data. With a judicious selection of reference class (see below), this is often the best recourse. The second method corrects the teams' P10/P90 ratios using a correction factor similar to that used for the P50, inferred in a similar way. This parameter is the factor by which the P10/P90 ratio is underestimated, so Cassandra's teams' P10/P90 ratios are about 1/1.4=70% of what they ought to be; again, pretty modest against the range of industry practice.

Reference classes

Cassandra can choose to adjust all the prospects in the drilling programme with correction factors derived from all the historical data, or she can divide both historical and future prospects into reference classes.

Reference classes are simply categories of prospect within which we might reasonably expect similarities in the characteristics of assessment, specifically similar levels of uncertainty as captured in the P10/P90 ratio. Trap type is an obvious candidate (four-way, three-way against salt, three-way against fault, stratigraphic, etc.) Geological era is another, but region, data quality and provenance, on- versus offshore / DW / UDW and play maturity (frontier, emerging, mature, EoL) are others.

The idea with a reference class is that both the uncertainty and the bias parameters are expected to be more consistent between prospects within a single reference class. Assessment methodologies respond differently to different categories of prospect, but consistently within those categories.

There is a trade-off. The fewer classes applied the greater the number of prospects within each class and the smaller the statistical uncertainty around the bias parameters (the width of the distributions shown above are mainly dependent on the number of prospects in the historical reference class). On the other hand, the more classes applied, the greater the discrimination between classes and the greater the consistency within class.

Ultimately, Cassandra needs to experiment to discover how quickly the bias parameters converge for different choices of reference class and whether there are enough data points in the classes she chooses to give that convergence. Luckily Cassandra knows a consultant who can automate this process, so this is a morning's work.

The proof of the probabilistic pudding

This is a so-called percentile plot from Cassandra’s lookback, which contains 100 discoveries. The discoveries are sorted into bins by the percentile on the predicted distribution on which they came in (p).

If the predicted distributions were faithful then the distribution of the percentiles on which they land should be uniform and we should expect to see roughly 20% in each bin. Of course, even with 100 prospects, we don't expect to see exactly 20% and the uncertainty range (P90-P10 for the number falling in each bin) is also shown in the figure.**

Clearly, there is a bias here as far too many are falling in the low P100-P80 range (and too few are falling over P60). This looks like pure optimism, but as we can see from the bias parameter distributions above, there is actually a fair amount of over-confidence here too. This is one of the advantages of the methods outlined below: a substantially more insightful lookback.

Cassandra's not convinced, so she takes the first 50 of the prospects in her historical sequence and derives the bias parameters. She will then use these parameters to correct the predictions for the next 50.

Here's how those latter 50 manage without correction.*** The over-confidence is a little more apparent here, holding up the high end. Although four out of five of the bins are in range, the consistent tendency to the low side is indicative of systematic biases.

Now after extracting bias parameters from the first 50 prospects in the sequence and correcting the subsequent prediction, the percentile plot looks like this - entirely consistent with statistical variation.

----

Footnotes

*We assume the resource distributions are log normally distributed, which - as I explain in my article The perils of extreme percentiles - is fine as long as we don't venture too far from the centre of the distribution.

**Note this is a 80% confidence interval. The probability that all five bins fall in this interval is (0.8)^5, which is slightly less than a third. So we shouldn't expect more than to see a more or less flat distribution, with most, but not all falling in range.

***Note the larger uncertainty range (+/- 40% cf +/-25%) coming from the fewer prospects in the historical sequence.

---

The math