The Parker Challenge (ii). The interaction of uncertainty and classification

The headline images in this post were taken from my presentation of the Parker Challenge results. It’s an attempt to use an analogy to show why we have so much more to understand if we want to produce more fit-for-purpose classification of mineral resource estimates.

One of the more unique aspects of resource estimation is that there is no objective truth (the target) so knowing if your model is reasonable is an always a subjective judgement. This is something the judges of the challenge came up against when determining the winner. And, being technical people the idea of subjective judgement can be uneasy to accept!?

I kept getting strange looks when I likened the judging process to a beauty contest or diving competition.

Why do I say there is no objective truth, no ‘real’ resource, no target? Because of uncertainty. Because a ‘resource’ is a concept, a mental construct we use on the way to estimating an ore reserve. You can never directly measure a mineral resource - at least with our current technology. You may think that’s an extreme position - hey we can mine the resource and that will tell us if you were right or wrong and by how much!

Not so. The act of mining (or the application of modifying factors) affects the outcome. You are not measuring the resource. You are measuring the resource PLUS all the machine, human and other factors involved during mining. Each of those have their own errors and uncertainty, their own noise. For instance ore/waste misallocation, dilution, imperfect selection to name a few. And need I mention the capability of different operators, different ore blocking, different organisations?

A ‘resource’ is part of an ensemble system. A system that only makes sense if all the pieces are assembled and considered together. You cannot understand the system by studying its individual components. That would be like trying to understand an ant colony by looking at the individual ants. A futile exercise and one likely to leave you with the wrong impression.?

Any ‘target’ or ‘reality’ we may have to assess the accuracy of a resource estimate is at best blurry and imprecise. And the further away you are from production the blurrier that vision of ‘reality’ becomes. Think about that for a moment. Then think about how your mind works. What happens when you have no target, no goal? You tend to invent one. And that leads to cognitive biases - anchoring, recallability and the like.

That’s a problem. That blurry target plays games with our thought processes.?

Look again at part of the headline image.

Be honest. If I asked you to say which of the cluster of crosses on the left were the best you would make a judgement. You cannot not make an instantaneous assessment. What’s more I reckon you would be more likely to select option A as ‘the best’. Why? It’s subtle but the crosses in option A are more centred under the letter and they are closer to the letter. This arrangement connects with our ideas of accuracy and we conflate accuracy to precision. We perceive the closely spaced points as more like our expectations of a ‘good’ outcome and forget about the potential for an underlying bias. We unknowingly substitute precision for bias in our judgement.

Phew… that’s hard. It’s so much easier if you have a target! Until you understand the target is probably invalid.?

It gets a bit worse.?

Let’s say we are sophisticated resource specialists and we understand the need to assess the precision of our models. We may even use conditional simulation to determine probability distributions in preparation for resource classification. You know, simulate and then check the variance of the simulations. Does that help us? Can we assess accuracy using probabilities in the absence of a target??

Look at the following image. The orange cross is a new estimate you are assessing against multiple previous estimates (the blue crosses). Is the orange cross more accurate or less accurate than the previous models??

Look again. We have no target. In the absence of a target you cannot fully assess accuracy. All you can do is assess the relative position of the new estimate compared to the earlier estimates. I’ll play the same trick again and superimpose a target.

Now the orange cross in A is clearly more accurate. And I bet that is different to your original perception when the target was unknown.

The problem is knowing if we have a central estimate or not. Option B looks great because we are so familiar with thinking we know the target. Estimation under uncertainty is not so simple.?

Let me explain a bit further. Imagine you are estimating a mineral resource. You diligently check the data, its quality and distribution. You investigate the geology and consider the genetic model. Using your expertise you devise geological interpretation and estimation domains. You estimate the grade within those domains using state-of-the-art (ark?) geostatistics.?

Can you honestly say that your estimate is the central case of every plausible alternative? Are your volumes, your tonnes, your grades the centre of the distribution? That is unknowable. We use ‘good practice’ and ‘industry standards’ in a vain attempt to ensure centrality but where there is uncertainty there is risk. And I haven't even discussed the geometric and location aspects of estimation that come into play when we think about local instead of global...

So we have an estimate that may out may not be the central case. We then assess the risk-state of that estimate using simulation or some similar tool. I think you can see that the risk distribution we have assessed is not the risk distribution of the model vs our unknown reality. No. It is the risk distribution of our model which itself may be biased. We have found Option B - the spread of possibilities around our estimate without first understanding that Option B may not be the central case.

Oops..

If you really want to make things difficult to understand, try adding non-linear effects and skewed distributions into the mix. But that might have to wait for a later post. Those two factors mean that not only do we not have a target but that the unknown ‘real’ bullseye is moving in discontinuous jumps…