Why Projects Get Delayed and How to Solve It in 4 Steps

Are projects in your organization taking longer and longer to complete while you need to deliver results? Do people say they're not estimating correctly? You're not alone. Learn about the potential causes and how to solve this problem systematically.

Jan'25 Update: This article was originally published on May 26th 2021 in Spanish. Due to significant interest, I revised it and translated it.

Several years ago, I worked for a major bank in Buenos Aires. The team was developing a new ATM network system to replace the current one -still operational but obsolete and without vendor support.

When I joined, the project was already two years behind schedule with several deadline renegotiations. As the new system wasn't ready, what was initially a technological migration evolved into new projects: they couldn't wait any longer and needed to implement new features in the current system, plus the rework of replicating these features in the new one.

During those years, they had tried everything possible: paying overtime, replacing most team members with more senior staff, expanding capacity with more teams, training, and even changing the project manager, but nothing worked. Distrust and damaged relationships generated more status and control meetings that took away development time. Teams felt highly pressured and stressed. However, the project didn't seem to progress.

When the root cause isn't understood, only symptoms are addressed. They might say estimates are wrong and teams need to improve their estimation skills. Consequently, more time is invested in increasingly detailed requirements analysis instead of solving the underlying problem.

What Was Failing?

I knew this was just the tip of the iceberg. To understand the causes, we need a deep understanding of the organizational system. That's when I decided to create a systemic model using causal loop diagrams (CLD) to see the complete system.

"When we don't see Systems, we are at their mercy" - Barry Oshry, Seeing Systems

Cause #1: Work in Progress Volume

The well-known "Little's Law" explains the systemic relationship between work in progress (WIP) and delivery times: for a given process, generally, the more items you work on simultaneously (on average), the longer each item will take to complete (on average).

The more projects in progress, the longer the average delivery time for each project. Moreover, the longer projects take, the less patience stakeholders have to wait for completion before starting new projects. In other words, there's increased pressure to begin working on new projects representing new business opportunities.

Additionally, the "sunk cost fallacy" explains why many lagging projects aren't cancelled. It's a bias that makes us think that since we've already invested significant time and money in a project, we must continue until completion. Cancelling would mean acknowledging that everything invested was a complete waste, which often carries a high political cost.

Note: I adopt the graphical notation used in LeSS (Large Scale Scrum) courses

How does this relate to organizational agility?

The popular phrase "fail fast, learn fast" refers to conducting short experiments to learn from them. Each executed project is an opportunity to learn what worked and decide what the next most important project should be. Organizations need to adapt quickly to take advantage of new opportunities with increasingly shorter time windows. This isn't possible when projects take too long.

When projects are delayed, the organization loses its ability to deliver value and adapt.

While each delay cause individually contributes to the problem, what's truly alarming are the systemic effects of reinforcing cycles.

Cause #2: Overly Optimistic Estimates Treated as Commitments

The Cone of Uncertainty is a model describing that at the start of a software project, estimation error oscillates by a factor between 0.25x and 4x. The gap reduces as the project progresses and only closes when the project is completed. In other words, for projects with uncertainty and variability, trying to predict total scope and delivery date from the beginning is an estimate with too much error margin to be useful or to coordinate commitments with other teams.

Sometimes projects start off on the wrong foot when teams provide overly optimistic estimates. This happens either due to external pressure or the phenomenon known as "wishful thinking" - a cognitive bias of optimistic thinking that makes us assume there won't be unforeseen issues.

I've seen countless times how estimates are treated as certain dates and commitments signed in blood, and based on this, managers coordinate other activities and make new commitments with clients and stakeholders. Thus, deadlines and commitments become tied to teams' original estimates.

Often these managers are the visible face to the rest of the organization and clients, and when plans aren't met, they hold teams responsible. Traditional managers manage people seeking to increase delivery pressure, assuming the root problem is in the people's attitude (lack of commitment), and fail to "see the complete system." That is, how the organization is designed to favor the results obtained.

“A bad system will beat a good person every time." - Edwards Deming

The tense conversations resulting from these situations reduce trust in both directions of the relationship, which in turn negatively impacts motivation and emotional commitment that teams are willing to invest in the company, worsening the situation once again with another reinforcing loop.

In other companies, I found a variant of this factor: a third party unilaterally defines delivery dates. For example, a salesperson closes a sale and signs a contract with a client promising a date that wasn't validated with the teams that will develop the product. While the situation is different, the impact in this analysis is similar.

Cause #3: Unattended Technical Debt

The term "technical debt" was coined in 1992 by Ward Cunningham - one of the co-authors of the Agile Manifesto. It's a metaphor describing the decision to postpone certain activities, generally related to product quality, to accelerate software delivery in the short term. But like any debt, it must be paid later with interest. In software, this implies future rework to address product quality and make it sustainable.

The bigger problem occurs when paying technical debt is indefinitely postponed. This is how delivery pressure negatively influences product quality, which in turn generates consequences impacting delivery times. For example, tests aren't automated, code isn't reviewed or improved, quality controls are rushed with superficial testing. Eventually, defects occur and time is used managing them, managing complaints, rework to solve problems, maintenance, and greater effort to make changes to a poorly constructed product. "Technical debt" is a major cause of delays. As can be seen in the following graph, here too another vicious cycle is generated, both from the rework of paying technical debt and the rework from defects.

Cause #4: Time Fragmentation

This factor describes the situation where available time for product development is divided into small time portions during the workday. When teams estimate required time, they often assume they'll have the day free to work in a single block. However, this rarely happens.

In the analyzed case, after each missed deadline, a new timeline agreement had to be made. A re-planning and re-estimation of pending work was carried out, this time with a greater level of detail. Due to reduced trust, more control mechanisms were introduced: status meetings, progress report updates, documenting commitments in emails, hour tracking, and continuously interrupting the team for follow-up questions.

There are two negative impacts produced by these activities. The most evident is that valuable time is consumed that could be dedicated to project development. The project is paused every time the team or key people are in a follow-up meeting or completing forms and reports. Paradoxically, people with greater expertise who could contribute most to project progress are those who spend their day going from meeting to meeting because they're also the ones who have the answers managers need.

Moreover, interruptions during a creative thinking process have a great impact on productivity, since after an interruption - no matter how short - time and energy are required to resume the train of thought. In an analysis conducted by the Georgia Institute of Technology in 2010 of 10,000 programmers, it was found that on average it takes between 10 and 15 minutes to return to writing code after an interruption.

Also as a result of cost optimization and maximizing people utilization, and the amount of work in progress, team members had to divide their time between different projects. This generated a great waste of time in context switching and dependencies (when one person needs another, they can't attend because they're in a meeting for another project).

When people are assigned to multiple teams and projects, the sense of belonging and team cohesion is lost, which also reduces motivation and commitment that people are willing to invest in the team.

Cause #5: Adding More People

Another factor that generates interruptions and distractions is incorporating more people into the project. Brooks' Law (1975) indicates that when a software project is delayed, assigning more people delays it further. This is because new incorporations must learn about the project context, requirements, tools used, and adapt to team dynamics. This situation consumes time from people who already had their hours assigned to development.

In the analyzed case, to optimize costs, students were hired through an internship agreement with two universities. The lack of experience consumed more training time. After not having favorable results, a complaint was made to one of the universities to replace them with more experienced people; training had to start again.

Another additional impact is that a larger team entails more complexity. Communication lines between people increase exponentially following the formula N(N-1)/2, where N represents the number of people. For example, in a team of 10 people there are 10(10-1)/2 = 45 communication channels, while with 14 people there are 91 communication channels between team members. Team size notably influences meeting duration and team cohesion.

Cause #6: Dependencies between Teams

There's a reason why agile organizations promote autonomous teams: so they can advance projects without depending on others.

When a project in charge of one team requires collaboration from another, there may be a conflict of priorities. As a result, the project again falls into dead times. Usually, faced with the impossibility of advancing with prioritized work, the team decides to take a new task, further increasing the amount of work in progress.

For example, in a company where I worked, team "A" had to use a service developed and maintained by team "B". However, the system had a quality defect, which for team "B" wasn't relevant at all and therefore they weren't going to fix it because they had other priorities to attend to. However, for team "A" it was blocking for a priority project.

The existence of dependencies between teams often responds to an organizational design decision that seeks to optimize costs and maximize occupation at the expense of workflow and delivery speed.

The Complete Model

This is where we see what's really interesting: when we combine these isolated causes and partial analyses and diagram the complete systemic model, we find a very complex situation, due to the fact that each of these causes reinforces the others and multiple reinforcing loops are in play.

Thus, new projects will become blocked increasingly quickly. Projects spend more time "waiting" than in progress, despite all people working at their maximum capacity or even stressed with overtime. Dead times and bottlenecks make the organization stagnate completely.

Systemic Delays: Delays are the factor that most hinders seeing systemic correlations. Often in systems, an indeterminate time passes between a cause and its systemic effects, making it very difficult to relate both factors. Systemic analysis allows us to see the complete system to identify these correlations beyond the time variable. In the diagram, it's represented with an interrupted causal relationship line.

The Solution in Four Steps: Systemic Levers

Resolving this situation requires an organizational leadership strategy with a systemic view. Teams alone cannot solve the problem; they need leadership support. The following four steps can lead the organization to a better context.

Step 1) Identify the situation to make it visible. Systemic analysis serves us for this. If we don't see the problem, we won't be able to act on it.

Step 2) Leadership must make the decision to optimize organizational results. That is, global optimization over local optimization. Optimize flow over people occupation.

• Local optimization refers to resource efficiency and productivity of what each team (or worse, each individual) produces. Related metrics are generally "output": hours worked, story points, amount of software produced, occupation level.
• Global optimization refers to what the organization manages to deliver to the client. Related metrics are generally "outcomes": number of completed projects, total delivery time to client, and metrics that reflect user or client satisfaction and achieved business results.

Step 3) Identify Systemic Levers and act on them.

Levers are points where we can intervene in the system to achieve results. For this particular case, you can find blue variables in the diagram; these are the key leverage points I identified.

All of them are there by someone's decision, therefore they are within leadership's control scope. Like any lever, with a little effort, great results can be achieved.

With this information, a global improvement strategic plan can be created that permanently reverses the problematic situation.

Step 4) Continuous improvement to optimize flow in the organization.

Leadership strategy must prioritize projects flowing, minimizing wait times, dependencies, bottlenecks, and other organizational impediments. This being more important than maximizing people occupation.

Henrik Kniberg calls it "escaping the resource utilization trap" and demonstrates it with a simple metaphor in this video. Just like Niklas Modig in his book This is Lean.

Conclusions

One of the famous phrases in the Kanban method is "stop starting, start finishing". For this, it proposes as a central practice limiting work in progress to manage limited capacity and preserve flow.

A popular saying goes: "if you're in a hole, stop digging". In an organization experiencing project delays, the best thing is to stop taking on new commitments to allow ongoing projects to complete.

By limiting work in progress, impediments become visible and that's the first step to resolving them. Without this limit, it's easier to work on "what can be done" instead of "what really should be done" and problems hide within high occupation.

In Lean, this concept is illustrated with a metaphor, where water (amount of work) covers problems.

This rule leads us to consciously prioritize projects that have the most value for the business and helps teams maintain focus on completing them.

Once workflow is achieved, we'll be in a position to adjust the amount of work in progress. The objective will be to find how many projects the organization can manage without harming delivery times.

Without wait times, projects complete so much faster that it will be easier to prevent projects from accumulating.

These four steps aren't simple to implement, but doing so will generate an important transformation in the organization, leading it to a leaner and more agile way of managing work.

Perhaps a smaller step to begin with is to use these 10 strategies to speed up any project.

This is a very illustrative example of how you can apply systems thinking to an organizational problem to make impediments visible to those who lead the organization.

Agile leadership must work on the system, that is, see all factors and causal relationships to make systemic interventions. If we fail to see the system, we only intervene in the parts we see and as we saw in the example, this can worsen the situation. The solution must be systemic. All mentioned factors must be addressed simultaneously to achieve a more effective organizational design.

While we see how multiple factors interrelate, we can also see that the six factors we described in this analysis are within the control of leadership and teams if they work together.

The problem is resolved if it's decided to have dedicated teams, attention to quality, limiting work in progress to maximize flow, and changing management style regarding deadlines and estimates. A systemic solution addresses multiple factors simultaneously.

I invite you to share this article with the next manager who says teams need to learn to estimate better when you actually know there's a bigger underlying problem.

What did you think? Let's continue the conversation on the comments below and feel free to share it.

Thank you very much for reading this far and much success on your agile journey!