Death by Documentation: How excessive planning can sink software

By Firefield CEO, Adam McGowan

To most executives, the idea of embarking on a project as complex as a software application without a plan is insane. I couldn’t agree more. However, it’s often equally problematic to go too far in the opposite direction. I frequently meet innovators who assume that if no plan equals failure then it’s sensible that extensive planning should protect them from the miscommunications, budget overruns, or lack of cohesion and accountability that often sink software projects. That also sounds really good when explaining their vision to their board.

For early planning, this logic holds up well. There is, however, a point where additional preparation generates diminishing returns, and eventually things actually invert: adding additional documentation not only wastes time, but it actually begins to REDUCE the likelihood that a software project is going to be a success. Let me explain…

Every portion of an application’s description of features and functions introduces what I like to call “documentation debt” to a project. I describe it like a loan because in one way or another, it needs to get repaid. Feature by feature, that has to happen in one of three ways:

“Do it”: Develop and deploy a feature exactly as the documentation and plan described.

“Change it”: Agree to modify a feature (likely after more documentation) and then develop (or rebuild) and deploy it as adjusted.

“Dump it”: Conclude that the feature should not be developed and officially agree to remove it from the project plan.

Obviously “Doing it” is the ideal result. Heavy-documenters bank on it because otherwise a huge volume of planning would have been in vain. Unfortunately, for many projects this outcome is the exception rather than the rule.

You might be asking yourself, “How can that be possible? What’s so hard about building something according to the specifications?”. The issue isn’t that it’s hard to build to spec. Rather, in the real world, building software is fluid (maybe less than some agile development evangelists would have you believe, but fluid nonetheless). Requirements and resources can change quickly and without warning. That’s why for many projects, “changing it” is the norm.

And unfortunately “Dumping it” generates a lot of waste. As the saying goes, “you’ve got a break a few eggs to make an omelette”, but not everyone prepares for this. Your goal should be to limit these outcomes as much as possible, but regardless of the project, they are inevitable.

If we assume that a software project’s execution results in too few “Doing it’s” and too many “Dumping it’s”, then the process certainly fell short on productivity. But just how does that support my point that too much documentation actually INCREASES software’s likelihood of failure?

Well, for starters, every project requires the efficient use of scarce resources: time, money, people, ideas, etc. Unless the well is endless, these assets eventually dry up. When too many of these resources evaporate as the result of changed or unused documentation, then there are fewer available for product development and execution.

But in my view, a bigger issue is that thorough (and often high cost) documentation is really hard to let die. If the “right” answer regarding a proposed feature is that is should be dumped, stakeholders frequently don’t appreciate that its planning was a sunk cost and should be forgotten. Rather they rely on justifications like “Our team did three weeks of great work to spec this feature. It would be a waste if it wasn’t included in the product.”.

In the end, these features often remain in the project plan. They consume resources to build and maintain. They can result in a cluttered user experience that diminishes the effectiveness of key features, or worse, they result in an application that is so unintuitive that it actually turns customers or users away.

When even the best laid plans start to feel like their crumbling, it becomes harder and harder to maintain perspective and objectivity. Imagine you’ve just received user feedback suggesting less adoption than expected for a key feature. It’s time to decide whether to divert the pre-planned product roadmap and investigate, or stay the course and assume “those users just need more time to figure the app out”. When fatigue and pressure (key symptoms of excessive “documentation debt”) start piling up, it’s easy to make the wrong choice.

So what’s the solution? Well, given the uniqueness of every project, there is no “one-size-fits-all” answer. But here are a few takeaways that should help you find the right balance:

Avoid extremes. Too little planning leads to chaos and waste. Too much and you’re battling frustration, cost overruns, and possibly failure. Finding the right balance means planning enough to engage your stakeholders and team, yet letting up before the details blur the big picture.

Make ongoing planning part of the process. Too often a team will design, strategize, and document up front – leaving it behind once development starts. Don’t. This creative and iterative process should persist from start to finish.

Keep it flexible. Show me a development team that recalls a project running exactly to plan, and I’ll show you a pack of liars. The landscape is going to change repeatedly on you. If your approach to planning doesn’t account for that, you will set yourself up to fail.

Don’t double down on sunk costs. No one likes waste, but the notion that “if we just spend a bit more time and money, we can make that unused feature great” is dangerous. It’s like your favorite jeans from 2004. Just let them go. Seriously…it’s time.

Until you validate your assumptions, you’re just making things up. In the earliest stages of planning before you release something into the wild, no amount of powerpoint presentations can predict how users are going to respond to your product. The reality of those variables only come with real usage and testing. If you forget that, it’s going to be a very long and expensive road ahead.