The Inconsequential Repercussions of Poor Estimation in Project-Oriented IT

Estimating – the art of practicing educated guesses on how much time and money are required to perform something – is a difficult task, particularly in corporate IT. I have provided them, collected them, validated them, compiled them, suffered from them and abided by them, and let me assure you that this whole estimation business is far from trivial.  Being a difficult task is one thing, but it should not be a reason to push the subject aside.

So let’s look at a classic scenario that I have seen in all corporate IT projects that I’ve been involved with:

• The first estimations are made with very little knowledge about the requirements during the IT investment budgeting cycle, starting six months to more than a year before the project is effectively launched.
• The budgeting cycle directly involves the IT managers who will be responsible for building the solution. It is their opinion that carries the most weight in the balance.
• In the best-case scenario, technical experts, designers and architects will be involved in a quick tour of the requirements and a high-level design of the solution. In other, less ideal cases, the managers will make the estimates.
• Estimates are made with very little time allotted for the exercise, with managers and experts busy delivering current-year projects and dozens of other projects to evaluate within just a few weeks.
• No quantitative method is used because the IT team has never developed such methods. There is little usable historical data, apart from the actuals of past projects. The identification of analogous projects is left to the memory of people rather than a rigorous classification of past expenses.
• After several rounds of investment prioritization, the remaining investment projects will be challenged on estimates.
• Based on the same limited knowledge of the requirements and with still very little quantitative data to back-up their argument, IT managers, sometimes with the help of their experts, will come up with more stringent assumptions in order to reduce the estimates and fit the expected budget.
• At this point, the fear of having a given project cut from the investment list will have a definite effect on the level of optimism of the involved parties, both on the business sponsoring side and the IT team.
• If the project makes it through the cuts, then in the next fiscal year a project team will be assembled. Only then will the true requirements be fleshed-out with the help of business experts, leading to more complete business and IT designs.
• This detailed knowledge will lead to re-estimation of the cost and schedule. Most of the time, the new estimates will be higher than the ones from the budgeting cycle estimates. If the budget cannot be trimmed, then features will be cut.
• In some organizations, a gating process may be put in place to reassess the net business value of the IT investment in view of the more accurate costs and schedule. The project may not pass the gate, at which point it is cancelled.
• However, in many organizations, IT investment gating is avoided – or is nonexistent – and the business sponsor, project manager and IT managers will work on the expected scope and schedule in order to deliver something of value within the current year.
• If the business value cannot be achieved within the available budget/schedule, a change request may be issued, frequently justified by the falsehood of one or more of the original estimation assumptions.
• Since there is no formal quantitative estimation model in place, there is no process to assess if the change requests are caused by flaws in the estimation practice, nor is there a way to address how it could be improved for future projects.
• Upon completion, the project may deliver fewer functions or less business value than expected, but since the original requirements were pretty vague, it is difficult to assess the delta.

This typical and classical sequence of events is one of the many variations that occur in IT organizations. Estimation-wise, the most important characteristic of this scenario is that the estimation duty suffer from little rigor, no repeatability, absence of relevant data collection, and archaic tools.

In short, the corporate IT estimation discipline is so immature that it can’t be called a practice. Things are mostly left to good intentions and experience.  

Even the Agile? tidal wave isn’t bringing much improvement in that area. An iterative development method is a blessing for avoiding large projects to become white elephants. It is also a benediction for eliciting requirements when complexity, unknowns, or ignorance significantly raise the risk levels. But the Agile deployments I have seen are misleading many actors into thinking that the need for knowing in advance how much something is going to cost has suddenly become obsolete.  Because plans -and strict adherence to them- has caused failure in the past within completely different processes, the is a vogue of rejecting anything that looks or smell like a plan, including estimating. But there will always be someone that is investing some amount of money to get some result.  I have yet to see, read or hear about any improvement in the rigor and effectiveness of the estimation provided by any development method, Agile or other. The Agile way of tackling IT-related change has taken the ignominious waterfall method and sliced it to shorten delivery times, and allows to reorient work.  But still, work has to be estimated before action and calling it Poker Planning or T-shirt Sizing doesn’t make it more rigorous than any other technique I’ve witnessed in the past 30 years. Rigor comes with systematization: using it across teams and time, as well as collecting data with the intent to improve the estimation process.

Agile? methods have brought tangible improvements in corporate IT’s delivery effectiveness. But from an estimation point of view, apart from cool names, the techniques are still based on good intentions and experience.

Corporate IT is nowhere close to being mature in the estimation practice. If someone in your IT function ever tries to talk you into the difficulties of building a reliable estimation process due to the newness of IT, spare your tears and start with this interesting quote:

“False scheduling to match the patron’s desired date is much more common in our discipline than elsewhere in engineering. It is difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by hunches of the managers. […] Until estimating is on a sounder basis, individual managers will need to stiffen their backbones, and defend their estimates with the assurance that their poor hunches are better than wish-derived estimates.”

This may look like an excerpt from a blog or a recent report from one of the IT observatories, and may appear quite apropos and contemporary. But here’s the embarrassment: this quote is from a landmark book, The Mythical Man-Montn[1], published in 1975!

Does this mean that the estimation practice in corporate IT has been at a standstill for 40 years? I’m afraid so. 

This standstill has occurred despite research on the subject, text books and the development of estimation software, and active organizations dedicated to that subject. It’s happened in the face of the pitiful track record of corporate IT for being on-time and on-budget. All of this while some organizations spend hundreds of millions of dollars on IT projects over multiple investment cycles. To make it short: accuracy of estimates is secondary, and it explains the generalized laxity on this topic across organizations and over decades.

How can such a serious weakness with such considerable monetary consequences not be the driver of a relentless quest for improvement? The answer is simple: there are no incentives to get any better.

There are very little consequences in corporate IT for bad estimates. Worse, there are tangible benefits not to improve.  As I explain in my book there is no such thing as an IT Machiavellian plan to entrench in your organization a system to milk your hard earned funds. There is simply an engagement model that doesn’t foster improvement in several key areas, estimation being one of them. By changing the game, IT will need to improve, will adapt and develop what it needs to get much better at estimating. Yes they can.

[1] F.P. Brooks Jr., The Mythical Man-month: Essays on Software Engineering, Addison-Wesley, 1975.