That's appalling!

Sometimes I find my work equally humorous? and frustrating. Sometime it’s just downright tragic. When you review as many resource estimates as I do you see the full gamut of performance. From the most sophisticated approaches that go beyond understanding to the epitome of keep-it-simple models. Then there are the models that, you know… sound great but have the substance of a piece of fluff. One strong breeze and they disappear never to be seen again.

Possibly the most frustrating and tragic cases are those estimates where it is clear the practitioner is in way over their head. Often without realising their ineptitude. Sometimes I see the classic ‘recipe book’ approach - someone who has followed a predefined approach that is not appropriate. Other times it’s a case of a little knowledge being stretched beyond breaking point. And all too often it’s the Dunning-Kruger effect in full swing.

In my opinion this is a complex problem and the solution is unclear. How do we, as an industry, improve the quality of our mineral resource (and ore reserve) estimates?

You may think it’s simple - just ensure the person doing the work is a Competent Person or Qualified Person. Sure, it sounds great but our existing frameworks around competency are loose. Furthermore tightening the framework is unlikely to materially improve the underlying problem unless there are some material changes elsewhere. Remember the JORC Code and its ilk are about reporting not about doing. The Codes do not mandate estimation approach (nor should they) and they do not provide practitioners with any guidance on how estimation works. No, they simply mandate you what you must reveal to third parties.

OK, if improvement does not sit within our reporting codes, maybe it can be found in industry practice? Again, it sounds great and is a wonderful ideal but, in my experience, such lofty goals fail when exposed to the reality of costs. Our current systems and culture favour lowest-cost solutions. It’s a brave person who accepts the highest bid as opposed to the lowest… not to mention that sometimes price is not a great indicator of quality.

By now you might be wondering - what has raised Scot’s ire this time? Why is he off on another rant? You guessed it. I’ve been coming across an increasing number of estimates that are just bad. Poor quality work with obvious mistakes, omissions and, in some cases totally erroneous thinking. In one recent example I’d say the resource estimate was a large contributor to operational failure. No doubt there were other factors in play but that estimate underpinned everything from mining method selection, resource recovery thinking, considerations of dilution through to the final production schedule. And it-was-fundamentally-flawed!

The most worrying aspect of this case is that the estimate was from a reputable international consulting group. The sort of people you’d go to when you need help and expertise. Reading the resource documentation I’d say no one above about 5-10 years of industry experience reviewed the document much less the estimate itself.

Look, I’ve been there. Running a large consulting group is stressful and challenging. With growth you have more people, more salaries to pay. That means you need to win more work. Your focus is dragged from quality to quantity - that old cost-of-doing-business is as hungry as any ore treatment plant chewing away on your diminishing bottom line. It’s easy to slip into the mode of winning more work by being to lowest cost solution. Unfortunately, that road leads to decreased standards, poor product and, if you are unlucky, ruination.

This problem of estimation quality and the suitability of any estimate for planning and forecasting is widespread. It’s also lumpy. Some organisations take things seriously (to the extreme). Others are led by people who just assume their models and estimates are ‘the truth’ without question. This lumpy, mess is the playing field and we expect our investors to navigate it sure-footedly. Are we naive? Are our investors equally naive?

Some recent examples for you to ponder. Let’s start with a quote out of a resource report…

“…proper selection of the kriging parameters will apply the proper smoothing of the exploration data such that it accurately predicts the tons and grade on both a local and global basis.”

Sounds plausible doesn’t it? To me, it’s an immediate red flag. It’s not possible for an estimate to be both locally accurate and globally unbiased. Smoothing and conditional bias raise their heads one way or another and no amount of optimisation of the search neighbourhood can completely ameliorate the problem. When I read that sentence I immediately knew the practitioner did not understand what he was doing nor what the implications of his decisions could be.

A second example - possibly more confronting. What is one thing we all depend on when we are making estimates and constructing resource models? One, thing that is so ubiquitous that it’s a bit like breathing air? If you said our estimation software you’d be right. No one, and I mean no one, completes any type of real world resource estimate by hand calculations these days. It’s not only impractical but many of the skills required for the types of estimates that abounded in the 70’s, 80’s and early 90’s have long faded. Today, in our brave new world, we are completely at the mercy of our estimation software - a relatively few dominant players.

We depend on that software to work the way we expect. Sadly, that’s not always the case. Sometimes our expectations are wrong and sometimes the software engineers don’t understand our expectations. There’s a wide chasm between the resource estimator and the software engineer. One knows what they want or need. The other knows how to write code. When there’s an incomplete specification or misinterpretation of requirements you get… well let’s be kind and say things get a bit odd.

My most recent example of software not being exactly what we expect relates to sub-blocks, parent block estimation and locally varying search orientations. For the uninitiated, that last point is where the search ellipse is defined on a block-by-block basis allowing it to change orientation to better align to the interpreted geology - a great idea (sometimes).

In a recent review my client documented their estimate as being a ‘parent block estimate’. By definition, the grade was estimated at the volume of the parent block even if that volume was comprised of sub-blocks. A technical point but sometime very important - estimation error is inversely correlated to the estimation volume (i.e., error is higher for small blocks than for large blocks if all else is equal). Thus, my client’s decision to estimate parent blocks was a sound choice. There was only one problem. When I checked the model, the sub-blocks all had different grade estimates. How could that be?

After a fair degree of investigation the answer was ‘it’s the software’. Here’s what happened.

In this particular product, the search rotations used for the local orientation of the search ellipse are typically derived from the wireframe. If the model has sub-blocks that means the rotations for each sub-block can be (are!) different. If, these sub-blocks with different rotations are then used for grade estimation the estimate is at the sub-block scale. Worse, it’s at the sub-block scale even if you specify that you want a parent block estimate!

Ouch! As I sit here writing this I wonder how many people do not know this is an issue. How many people are blithely ticking the ‘parent block estimation’ box and yet producing a sub-block estimate?

No, software doesn’t always behave the way we expect. Do you think our lowest-cost practitioner understands this? I don’t.

A third and final example. I’ll call this one “unintentional misdirection”. Many of the resource reports I review are full of this type of nonsense. The author focuses on details that have little to no impact on the outcome and omit critical information you need to be able to understand their work. This can be subtle. For example, a discussion of some statistical property observed in the data, something that will impact on the quality of the estimate, with no detail of how, or why it was managed. It’s like saying the light was red without letting people know you must stop at a red light. These unintentional misdirections are everywhere. A problem is identified but no solution proffered. Great detail is provided about some esoteric data wrangling practice whereas the search neighbourhood is omitted - the list goes on. Even when there is a mandated table of contents, people gloss over things - the optimist in me sees no nefarious motive, more it’s a case of “I don’t really understand this so I’ll just focus on what I do understand and that will do!”.

Where does all this leave us? In a quandary I think. We need more discussion, more debate, more innovation and more informed decision making. We need to improve cross-disciplinary understanding, not only amongst mining professionals but to financiers and policy makers. Fundamentally we need to get to grips with uncertainty. We need to stop believing that are models are all equally predictive regardless of the practitioner. We need to stop believing that one person’s Measured Resource is the same as another.

We need something different. Without meaningful and well directed change we will continue to build our house of cards on an unstable foundation.

Sometimes you need to start from scratch… What is that estimate for again?