Keys to Successful Software Estimation

This is the 2nd in a two-part series on software estimation. The 1st (“The Trouble With Software Estimation”) dealt with estimation bias relative to technology choices, domain knowledge, and the perceived recipient of the estimate, and is targeted more at technical estimators. This article discusses estimate types, the stages at which these estimates are created, and the keys to successful software estimation at various stages of the process. We also discuss important parameters for the estimation model, which embody the values and constraints discussed here.

1. Summary

Software project challenges stretch budgets to 189% of original estimates, on average. The interplay between software estimates and other success factors that drive software project performance are difficult to disentangle. Estimate precision at any given time is a reflection of project readiness.

Good software estimates contribute to the success of a software project by:

• setting expectations for time, budget, and contingency
• promoting delivery team engagement and performance
• giving visibility into progress relative to plan
• promoting trust between client and vendor
• indicating when further discovery is necessary

Furthermore, if project estimators understand how their estimates will be used at any given time, they are much more likely to deliver accurate estimates that are appropriate for the context in which they are required.

Broadly speaking, different types of software project estimates are produced at different project stages (pre-sale, post-sale, delivery phase), each having a different utility and precision. The estimation context must be understood by all stakeholders in order to properly set expectations.

Estimate precision is a proxy for the completeness of the requirements, client readiness, vendor readiness, and technology risk. Expectation management, sales and delivery team alignment, and a healthy estimation culture also are pillars of good estimates. Estimators need estimation models that can account for these variables, to ensure consistency of estimates across the organization.

2. Industry Performance

The Standish Group produces the CHAOS report, a summary of software industry project success rates, derived from anonymized surveys of enterprises that produce software. For their latest attempt in 2015, they surveyed 365 respondents and 8,380 software projects, gathering company-private metrics and aggregating them to establish their reported industry-wide metrics.

The numbers indicate that software projects end up spending about 189% of the original estimate on average across all companies, with schedule overruns averaging 222% of the originally estimated timeline. These numbers speak to the fact that the software industry has a ways to go before it reaches maturity with respect to estimation, or fulfilling expectations relative to the estimate, or both.

According to Standish, the top 10 reasons for project challenges (time, cost, features) are as follows:

Standish Group draws no direct causal relationship between poor estimates and project challenges, but there is an obvious interplay between many of the project challenge dimensions and the software estimates that preceded them. Consider the following table, in which I imagine how estimates might affect these dimensions, (“Affected By Estimate” column), and the extent to which estimates themselves may be affected by them (“Affects Estimate” column). Where I see no significant effect, I’ve left the square blank.

For example, estimates have an obviously high effect on time and cost expectations, but high expectations can also depress estimates in an estimator’s attempt to satisfy his or her customers. Changing requirements may completely invalidate estimates, but high estimates may also cause the requirements for the system to be reduced, to match budget constraints.

Note that all of the major project challenge factors affect software estimates. This suggests that it is very important to understand this landscape before the estimate is produced, and further that these factors could introduce much more uncertainty into the estimate than is immediately apparent.

Loosely speaking, the precision of an estimate can be seen as a proxy for the level of challenges a project is likely to experience. For example, having a precise estimate likely implies that you have complete specifications, realistic expectations, and clear objectives, etc.

It’s not possible to address all challenge dimensions at all times during a project’s life-cycle. The key is to have a good idea of how uncertainty or readiness affects estimate precision. Thus, we must change the estimate and its associated precision whenever the context changes. A failure to do this means that the estimate will be no better than a shot in the dark, and the project will be on shaky ground from the start.

3. What’s the Context?

We consider here some of the different estimates one might see during the course of a software project, and describe the context for each. We aim to be as accurate as possible given what is known, and to be diligent and open when describing the precision of the estimate as the context changes over time, as well as the reasons for the level of precision. Together, these constitute the estimation context:

• estimate of cost to build a set of features (e.g. $100K)
• precision of that estimate (e.g. a $20K contingency budget might represent a 20% risk premium)
• text describing the current current set of issues introducing risk and uncertainty given current project conditions

Fixed-price quotes would naturally include some kind of contingency fee such that the customer and the vendor share the risk of overages. If the precision produces too wide a margin (and thus higher-than-acceptable contingency fees) for the customer’s liking, further discovery is required before entering into the contractual phase.

4. Types of Estimates

Different types of estimates are appropriate for different contexts.

4.1 - Back-of-the-Napkin Estimate

The back-of-the-napkin estimate depends wholly on an estimator’s experience with software projects, a very high-level understanding of the problem at hand, and their ability to extrapolate based on similar projects they’ve seen. It is usually delivered to set broad, high-level expectations, and to determine whether or not the conversation between the client and the vendor should even continue. It is not used to commit either party to a project.

This conversation is further complicated by the fact that pre-sale, the vendor’s salesperson is attempting to figure out how to price the project, whereas the technical sales engineer is attempting to figure out how to cost the project. Note that this article is about costing (i.e. estimation), not pricing.[1] A level of trust is required between the vendor and the customer to make sure that the delta between cost and price (i.e. the profit) is reasonable.

4.2 - Contractual Estimate / Quote

Your vendor may deliver a simple fixed-fee quote for a software product to be developed, or an hourly rate for each resource on the project, or both. Irrespective of the contract style, a cost estimate should always be delivered, unless there is simply no way the project scope can be reasonably defined at the outset.

The estimate provided by a vendor at the time of sale is used to establish the proposal and contract, a legal document attesting to what the client and vendor have agreed to do, and what the client will pay for it. The ability to produce such an estimate usually requires the execution of some sort of Discovery phase during which all features to be developed are broadly explored, and risky items are explored in detail, to increase estimate precision. See Keys to Successful Estimation: Discovery, below.

Note that in general, the team responsible for fulfilling the terms of the contract may not yet have had a chance to vet the estimates! Without processes to counterbalance the absence of the delivery team at the point of sale, a misalignment of expectations will almost always ensue.[2]

4.3 - Delivery Estimates

Agile software development practices employ frequent sprints in order to establish the velocity at which the delivery team can produce the system. They measure velocity for past features already implemented, and extrapolate this velocity to the remaining features to come up with the projected total estimate and cost.

This technique addresses both estimate accuracy and estimate precision. The estimate precision increases over time because over time the joint vendor / client teams learn what it takes to deliver features on their project. This knowledge is then built into the forward-looking plan during planning poker, which occurs once per sprint. This process builds a loose consensus among the delivery team about how long a particular set of features should take to build, and how many such features the team can fit into the next sprint. It also occurs at a time when the feature is considered “ready for development” – i.e. most of the uncertainty has been removed by detailed specifications and acceptance criteria.

This is the most accurate of the three main estimate types. And accuracy of these estimates may also improve over time, since their own team’s performance relative to typical project tasks is calibrated, and estimators become more familiar with the business and technology domains, as well as what it’s like to work with this client.

Of note is the fact delivery estimates generally occur post-sale. As such they cannot be used to establish the price of a project already sold.

This does not decrease their utility. The planning poker session is usually the first time the developer gets a chance to “have their say” in terms of how long a given feature will take to develop. It is important to generate developer buy-in through this process, without which the developer is at liberty to disconnect from the sales estimate completely, and claim that upstream estimators (who generally know less about the vagaries of software development) shouldn’t have sold the feature at that cost without consulting them first.

To a developer, an estimate is a kind of promise. Involvement and public commitment to the estimate during planning poker is incredibly important for two reasons – agreement by the team is generated, and personal commitment is achieved. And since the fulfillment of that promise will also be measured (at the end of the sprint), the likelihood that they will work hard to fulfill it also increases. Things that are measured (and committed to) are much more likely to happen.

But what happens when developer estimates conflict with contractual estimates? At the best of times, the following things happen:

• Some contractual estimates end up being lower than some developer estimates, and some end up being higher. In aggregate, they cancel each other out.
• Contingency costs (built into the contractual estimate) are used to cover a reasonable amount of overages
• Project overheads (meetings, communication, and management, analysis, design, testing, deployment) temper the build estimate variances. Overhead costs are typically higher than developer build costs on software projects, so a variance in estimated overhead is generally more impactful to the project than a variance in build time.
• The project manager uses tools to help the team notice how far ahead / behind they are relative to plan, and communicates this to the customer so they are not in the dark
• The team uses one of the following techniques to get “back on track” if they are behind:

- Increase productivity (usually a by-product of being measured relative to promises, as opposed to a conscious decision)

- Reduce scope or deliverables

- Work some overtime to get back on track relative to plan

- Re-baseline the project with new expectations

Note that for the last two techniques, costs still increase for vendor and/or client, depending on the agreement. But the customer experience is generally still good if the communication about reasons for variances is also good. This of course assumes that the overages are not due to vendor negligence.

Agile estimates are best used as an expectation-setting measure during the project, as opposed to a pricing strategy: they generate project progress visibility for the team and customer. They are also useful in committing the team to their promises on a regular basis (every 2-3 weeks).

4.4 - Earned Value / Projected Cost

The earned value isn’t really an estimate – it’s a measure of how much of the contractual value associated with a set of features has been earned. Loosely, it is the percentage of features that have been completed multiplied by the agreed-upon contractual value of the project. It is independent from the amount that has been spent to earn those features, which is equal to the number of hours spent multiplied by the blended hourly rate for the team. Typically, the earned value of a project that is behind is less than the amount spent. Comparing the earned value to the spent value thus gives you an indication of where you will end up by the end of the project if current conditions persist, which is a useful estimate in and of itself.

In Agile terms, an earned value analysis is similar to a “velocity” analysis: X features remain, implemented at velocity Y, delivers a time-remaining estimate of Z.

If a project is cancelled, the earned value usually represents the amount that the customer owes the vendor under a fixed-bid arrangement.

4.5 - Summary

The table below summarizes the kinds of estimates we have on software projects, their approximate accuracy and precision, and the utility of each type.

5. Keys to Successful Estimation

5.1 - Discovery

Discovery is both a requirements gathering exercise, and a risk-mitigation exercise. It takes time and effort, but aiming to complete a full analysis of the requirements for the system is basically the beginning of a waterfall process, which can be risky (See Agile Development and the Mona Lisa for a discussion of why).

Hinchey, in his seminar “Evolving Critical Systems” draws the following curve to help explain the utility of Discovery. It indicates that when very little is invested in up-front requirements gathering (Discovery), eventual overruns are very high.

This curve plots eventual NASA space mission cost overruns (as a percentage of the original estimate) for different levels of up-front requirements gathering cost for those missions. It shows that cost overruns decrease and level out to an acceptable eventual overage when the team spends about 15% or more of the budget on up-front requirements gathering. Indeed, comparing the shape of this curve to the typical overages seen on software projects [3] (according to Standish Group, 189% of estimate on average) provokes the question “is lack of up-front Discovery the main predictor for the significant project budget overruns?”

5.2 - Client Readiness and Maturity

7 (!) of the 10 challenge dimensions seen in section 2 above relate to project / client readiness or maturity:

• Lack of User Input
• Incomplete Requirements and Specifications
• Changing Requirements and Specifications
• Lack of Executive Support
• Unrealistic Expectations
• Unclear Objectives
• Unrealistic Time Frames

It is thus a fool's errand for a vendor to imagine that estimates will be accurate before taking an account of these factors. Vendors should build in contingencies based on the client readiness context, and indicate to customers how increasing their readiness can increase estimate accuracy and precision, and thus decrease contingency fees.

Mature vendors will actively evaluate client values, maturity, style, and transparency – the same things that the client is evaluating in the vendor, and account for these in their estimate models. Admittedly, this is not a very precise exercise, but attempts should be made to quantify these in estimation models.

One technique to assess customer readiness for the project is to deliver a short survey meant to help the customer understand where they are, and whether more preparation or discovery is necessary, given the estimate accuracy and precision they’d like to achieve before starting the project. Scoring the survey gives you a "readiness" input parameter for the estimation model.

5.3 - Vendor Readiness and Maturity

2 of the 10 challenge dimensions seen in section 2 above relate to vendor readiness or maturity:

• Lack of Resources
• Technology Incompetence

When the vendor isn’t scoring well on these factors, estimates are not likely to be accurate. In the prior article, we showed that technology factors usually cause estimators to inflate estimates by about 8% as a contingency, and that domain ignorance causes them to inflate it by 27%.

Early on in their history, vendors don’t have many reference-able projects, customer logos, testimonials, experience, or money in the bank. These companies are hungry and will eagerly clamor for any project they can get! This can have positive and negative implications for the customer vis à vis the estimate. On the positive side, young vendors:

• will more likely over-deliver relative to their own costs, because the value of each new project sold is disproportionately high for their company
• are more likely to make concessions when subjected to price and schedule pressure from the customer, which could mean a lower price
• are more likely to be catastrophically affected by a failed project, so are very motivated to ensure that this doesn’t happen; every customer is a big fish

On the negative side, young vendors:

• have less experience with estimation and delivery, and thus more likely to be “off”
• are more eager to throw caution to the wind in order to sign new deals with new accounts, which has implications for both their estimates and their ability to fulfill their promises
• are more likely to disappoint the customer when they are unable to meet the high expectations they’ve set for themselves
• may understand the cost of what they do, but not how customer involvement impacts their estimates

Mature vendors, on the other hand, have experience with both software development and customer experience management. With their customer logos, case studies, and testimonials, they have proven that they can be successful if the right conditions are met. More importantly, they know what those conditions are, and how they impact their estimates.

Note that “internal delivery team” maturity may be substituted for “vendor” maturity above, if the project is being in-sourced.

Since vendors are generally the ones responsible for producing estimates, it is incumbent upon them to prepare themselves and the customer as fully as possible, and to indicate how the current level of preparedness affects estimate precision and contingency fees, as a matter of managing expectations.

5.4 - Technology Readiness

Finally, 1 of the 10 main challenge dimensions has to do with technology readiness:

• New Technology

All feature estimates contain a certain amount of risk. The most complex features to be developed generally carry the most risk, which have real potential to blow up an estimate. At Jonah Group, we call these Factor “X”, representing unknown variables. It is prudent to place these at the highest priority in the product backlog (the list of yet-to-be-built features) and begin to tackle them immediately with research and proofs of concept during the Discovery phase. Risk and associated course corrections can thus be made early on, and big surprises don’t arrive late in the game.

Software project estimates have to account for the level of technology uncertainty inherent in the project at any given time. For the most problematic of these, risk and complexity factors should be incorporated into estimates for features slated to use those technologies.

5.4 - Expectation Management

Customer expectation management is one of the most important and subtle aspects of project success. It plays a role in both the sales and delivery experience for the customer, and depends largely on trust that is developed at the point of sale, and throughout the project.

Everyone knows that customers want low prices and a timeline that is as short as possible. But good customers also want to ensure that the vendor is successful, because their fate is inextricably entwined with that of the project. The role of the good customer at the sales stage is to push the vendor hard enough to get a reasonable price, but not so hard that expectations are unreasonably heightened.

On the flip side, we all know that vendors want to make a good profit on their projects. But good vendors push themselves to commit to costs and timelines that are both difficult and make-able, and “know when to say when” in the face of cost and budget pressure.

The dynamics for arriving at a good estimate for a software project are thus very delicate. Good estimates depend on both parties’ ability to apply an appropriate amount of tension in opposite directions, but to also be reasonable, open, and willing to embark on a shared journey with just the right amount of risk to make it both worthwhile and exciting.

5.5 - Sales / Delivery Team Alignment

A canonical problem on projects at any company is that the sales and delivery team incentives are misaligned: sales people want to do everything possible to get customers to buy, in order to fulfill quotas and increase their sales commissions. Delivery teams want to make sure there is enough contingency built in to ensure that they are successful in meeting the budget.

It’s thus best to have representatives from the delivery team cost the estimate (i.e. produce the estimate), even if the sales team ends up pricing it for the customer.

The sales team may decide to deliver the project at a loss, but pick up the value elsewhere. For example, the following may be good reasons to do that:

• having a new marquee client
• using the first engagement to impress the customer
• generating trust for larger projects to follow
• making quarterly quotas

The sales team can even be explicit to the delivery team that while prices may not reflect their estimates, they are still only responsible for delivering on the promises embodied by their own estimates – not those of the sales team.

5.6 - Estimation Culture

The prior article also spoke briefly to estimation culture, and the need for teams to understand how their estimates will be used, and the nature of the promises they are making.

Within the culture of the organization, it is thus important that delivery teams understand that the sales and deliver teams are interested in the so-called baseline build estimate, before overheads and contingencies for project, client, and technology readiness have been applied.

From the point of view of achieving baseline estimates, it is also important that the estimation team understands that they will not be “punished” for inaccurate build estimates. With a little help from good tools, estimate accuracy can be tracked for each team member to help them understand how they’re doing relative to actuals, which helps delivery team members get better over time.

Finally, it’s important to allow delivery team members to change their estimates during the build phase, and to accept this as a normal practice. Tool support for allowing estimates for features in-flight to be refreshed over time helps delivery team PMs communicate better with their customers, as well as lowering the temperature for the estimation team, and connecting them more fully to a culture of improving estimates.

6. Wrap-up

This article argues that software estimators need estimation models that can be parameterized not just by feature lists and requirements, but also by:

• Client and vendor readiness
• Engagement style
• Technology and Business domains
• Feature Complexity and Risk
• Project Overheads

Delivery teams also need processes and tools that promote better estimates throughout the delivery phase:

• Estimation model for producing contractual estimates, including the parameters above
• Planning poker
• Automated tracking of actuals relative to estimates (time tracking and reporting)
• Ongoing re-estimation by the delivery team
• Healthy estimation culture

7. Addendum: The Jonah Group Estimation Model

Over the years, Jonah Group’s proprietary estimation model has been tuned to serve the way we work with our clients. It collects many years of institutional knowledge around software estimation, costing, and pricing, and is the primary tool we use to establish the estimation context at various stages of the engagement process.

It is essential in both setting expectations and allowing easy removal or addition of features and functions.  Discovery and client interviews and conversations establish requirements, client readiness, and complexity / risk parameters, all of which feed into the model. Project overhead effort levels are established based on this readiness, and the nature of the deliverables clients expect on any given project. These are calculated as a percentage of feature build estimates from the sales engineer, who normally becomes a member of the eventual delivery team.

Our project management and tracking tool (Lychee) allows project managers to measure earned value, velocity, efficiency, and predicted eventual project costs and end dates at any stage of the project, and at any task node in the project hierarchy. The reports it shows are based on the collection of time-sheet data and "time remaining on this task" estimates.

The task breakdown and performance widgets, shown below, track important data for any task in the project hierarchy.

We’ve found that a bottom-up approach to the collection of ongoing estimates enables real-time progress visibility, and ends up being much more accurate than traditional top-down status gathering by a PM in a weekly status meeting. Any any point during the project, developers can update time-remaining estimates, as well as request them of others, or override them if they know better. Estimates roll up to any level in the project hierarchy the PM needs to report on. The dialog below shows an example of what that looks like.

[1] Pricing obviously depends on a knowledge of estimates and costs, but prices can be manipulated up or down based on numerous factors. For example, vendor prices can be increased if there is a shortage of delivery talent, or they may be decreased if the value of the client partnership is of significant value on its own.

[2] Clients should be clear with the vendor on whether the number supplied is an “estimate,” in which case they are responsible for any overages that may occur, or a “quote,” in which case the vendor will be responsible for overages. You must both clearly understand the answer to the question “if you get to the end of the estimated budget, and the system still isn’t complete, who is responsible for the additional costs to finish the project?”

[3] Comparing NASA project overruns to software project overruns may seem like a stretch, but both kinds of projects involve the coordination of teams of people, any single one of whom cannot know everything about the system, and all of whom are engaged in abstract thought-work using models for communicating unseen structure. We also imagine that the estimation process for any large project breaks down requirements for the system into components, until they are estimable by technical experts, which is precisely what we're doing here.

At Jonah Group, we’ve chosen “15% of the likely eventual budget” as the right amount to spend on the Discovery process, before feature development begins in earnest.