Improving evaluation criteria in public procurement

How can we improve evaluation criteria in public procurements??Robert McNamara summed it up when he said "Measure what is important, don’t make important what you can measure.”

Robert McNamara was the US Secretary of Defence during the Vietnam war; some would call him the architect of that affair.?He also worked as president of the Ford Motor Company and the World Bank where quantification of objectives featured highly to be certain.?What strikes me most about this quotation is how clearly it comes from lessons hard learned.? McNamara has been on the record as quite self-critical of his role in the Vietnam war, in particular the contribution that hyper focusing on quantifiable metrics (body count ratio) had in detracting from a wider geo-political assessment of success (stalling/stopping the spread of Communism in South East Asia).

[Side note: I feel like there is a paper from Australian Command and Staff College that I would like to go back and rewrite - Daniel Marston drilled into us that studying history has immeasurable value in the present; you were so very right sir.? Hit me up Australian Defence College if you want a guest lecture in what history can teach future military leaders about procurement.]

But here’s the crux of it - many procurements fail to achieve their best possible outcome because of a frenzied desire to choose quantifiable, perfectly objective, unchallengeable evaluation criteria.?The problem is: they measure the wrong things.?We use all kinds of words to describe this problem: McNamara’s Fallacy, surrogation, association fallacy, hasty generalization, secundum quid, etc. but the principle is the same.

Let me give an example:?

Say you are procuring a new Ground Based Air Defence system (seems apropos in today’s geo-political climate) and you have identified a strategic risk that the supplier will not be able to manage their delivery on time/on budget.?In short, you are concerned about their project management.

Traditional approach - Tell me how many years of experience your Project Manager has on Major Defence Projects (>$100m involving ammunition and/or complex command and control systems).?5 years to be compliant.?10 point per year up to a maximum of 100 points (15 years).?Provide a detailed CV as evidence.

I see this criteria or a version of it all the time.?There are a number of problems with it:

1. While there may be correlation between years of experience and performance, it’s not a universal truth.?You can be amazing with few years of experience (Sydney Crosby, circa 2008) or you can be terrible with plenty (how I play hockey).
2. The experience of the project manager (even if we accept it as a proxy for competence, which I do not) is one of dozens of elements that lead to the successful management of a project.?Many teams have succeed in spite of their boss.?But you need systems, tools, collective experience, education, process, and training.
3. An individual Project Manager is unlikely to be enduring for the life of a project.?Years of experience is actually a very good proxy for likelihood of retirement.?Yes, you can build in contractual terms and conditions to ensure like-for-like replacements; however, we are all aware of the bait-and-switch.
4. While we may think it’s a perfectly measurable criteria, the reality is that we’re not able to assess the relevance of experience really.?I suppose that a project that delivered 9mm bullets from a factory in Albania would qualify (it involves ammunition) as would a project that constructed a command centre (exclusive of the contents) could be written in a way as to comply (it “involved” complex command and control…)?We can address this by tightening up the definition in the criteria, but we’re going to really start to narrow the field and we open ourselves to the challenge of steering this to a particular bidder.

The Traditional approach leads to a terrible criteria.?It makes important what we can measure rather than measuring what is important.?Why do we see it used all the time??Well, it’s easy - just copy/paste what was done last time.?It seems highly objective - not much room for subjectivity in counting years of experience (except that the acceptability of relevance may be open to interpretation).?And in doing so, it mitigates the risk of challenge. ?

So how can we do better:

1. Determine your overall objective
2. Measure that as directly as possible.?Avoid proxies.
3. Identify the characteristics of a winning bid that align to that objective and include those in your criteria.?This will improve the objectivity you need to mitigate the risk of challenge.
4. Accept that measuring what is really important is rarely a perfectly countable attribute. Embrace qualitative measures. ?
5. Employ effective evaluation techniques, such as consensus, to drive out residual bias and evaluative noise, further mitigating challenge risk.

If you are interested in a better version of the Project Management criteria, contact me directly. I would love to help.