The Real Reason IT Projects Fail and the Key to Delivery Assurance

Did you know that 70% of IT projects fail? Whether it’s missed deadlines, blown budgets, or abandoned systems, the financial losses and reputational damage can be catastrophic.

Introduction

(This article was originally published on Medium. You can find the original version here.)

IT projects fail at an alarming rate. Despite decades of research and countless frameworks, organisations continue to struggle with missed deadlines, budget overruns, and outright project cancellations. The collective cost of failure is enormous, running to hundreds of billions for the private sector alone.

If you work in IT or change management, you’ve probably encountered this problem firsthand. You will have seen projects start with great optimism, only to be derailed by delays, re-plans, and firefighting. This is often true of well-managed projects led by experienced professionals who follow best practices. This makes the problem all the more perplexing.

In this article, we’ll explore why this is. For well-run projects, the answer isn’t a lack of methodologies, leadership techniques, or stakeholder engagement. The real issue is something more fundamental: non-delivery. In the modern matrix organisation, project managers usually lack the leverage to enforce commitments. When critical tasks are not completed within agreed timescales, projects unravel.

Most delivery assurance strategies miss the mark because they fail to create strong enough incentives to drive prioritisation. But after 25 years of delivering IT change, I’ve found one approach that works. This article will explain why the most cited strategies are doomed to fail and the one effective way to ensure project deliverables are prioritised and completed.

The Problem of IT Project Failure

According to any number of credible studies, IT project failure rates across the corporate sector are alarmingly high. Here’s a sample of the headline findings of some of these, which makes for sobering reading:

• The Standish Group’s CHAOS 2020 report found that 66% of technology projects end in partial or total failure, based on an analysis of 50,000 projects globally
• Research by McKinsey and the University of Oxford revealed that large IT projects with budgets over $15 million run 45% over budget and 7% over time, delivering 56% less value than predicted.
• A study by McKinsey in 2020 discovered that 17% of large IT projects perform so poorly that they threaten the company’s existence.
• According to BCG (2020), 70% of digital transformation efforts fall short of meeting targets.
• The Standish Group also reported that 31% of US IT projects were cancelled outright, while 53% faced significant performance issues.
• The failure rate is even higher for public-sector IT projects, with 81% overrunning their schedules compared to 52% in the private sector.
• Impact Engineering’s study found that 65% of software projects adopting Agile requirements engineering practices fail to be delivered on time, within budget, and to a high-quality standard.

It’s no secret that IT projects are routinely delivered late, and this is about the best that can be hoped for. At worst, whole programmes can be scrapped mid-flight and, in extreme cases, can pose an existential risk to the corporation (Kmart in the US and Auto Windscreens in the UK are notable examples). Whatever the degree of failure, these situations can be immensely stressful for the people involved and often entail recrimination and damaged working relationships or even careers. Furthermore, the financial and reputational costs are staggering. CISQ estimates this cost for US firms alone at $260 billion annually.

This topic has, of course, been written about extensively. After all, it’s a longstanding and well-known problem. However, there is an issue with the existing body of project management theory: even well-run projects have a high risk of failure. Many of you will know this from hard experience. You can hire the most competent project managers, and they can follow all the best practices and implement all the success strategies routinely trotted out by contemporary literature. Still, these projects can and do fail. This highlights the persistent challenges in IT project management and the critical need for improved practices and strategies to increase success rates.

The Problem of Non-Delivery

Where well-run projects fail, by definition, failure will not be due to the absence of generally accepted success factors. So, how do well-run projects fail? In my experience, they fail due to a problem we can call non-delivery. I define non-delivery narrowly here as the failure to complete critical project deliverables in agreed timescales.

In a typical scenario, the project is well planned, and the planning has been done collaboratively, bottom-up and subject to the explicit agreement of those responsible for doing the work. However, critical tasks are not completed on time, resulting in project delays and re-plans. It’s been my experience that in high-functioning organisations (where projects tend to be well-run), this is almost always the cause of failure.

But how is it that plans that have been agreed to by all concerned can so regularly go awry? To answer this, we must first understand the context of the problem. That context is the matrix organisation.

The Matrix Organisation

A crucial dynamic in modern project management is the "matrix" organisational structure, where project managers have to achieve results through influence rather than direct authority. This is the norm in large corporations implementing business systems. Let me explain why the matrix organisation is necessary.

The root of this challenge lies in organisational design. Companies typically organise their permanent structure around functional departments (IT, Finance, Operations, etc.) with traditional line managers. When a cross-functional project like a system implementation comes along, a temporary project structure is overlaid on this permanent organisation. This creates a matrix where non-ring fenced resources (i.e. not dedicated to the project) have dual reporting lines - to their line manager for their career development and permanent role and to the project manager for project-specific work.

In theory, the matrix organisation can come in three forms:

1. The Weak Matrix. Here, line managers retain most authority, and project managers act more as coordinators or facilitators.
2. The Balanced Matrix. Here, authority is shared equally between functional and project managers.
3. The Strong Matrix. Here, project managers have more authority than line managers.

However, in practice, I’ve only ever encountered the weak matrix. This stands to reason as authority derives from the power to control a person's career, and this inevitably resides with the line manager.

In this context, it’s not hard to see how project deliverables ‘slip’. Non-ring fenced resources (invariably a significant proportion) must balance the responsibilities of their permanent role and project work, and they often work on multiple projects simultaneously. They provide commitments based on their anticipated workload; when unanticipated work inevitably arises, their capacity becomes constrained, and something must slip.

What is delivered and what slips becomes a matter of priorities. Furthermore, those priorities will be set by the line manager in whom formal authority resides. It falls to the unfortunate project manager to seek to indirectly influence these priorities in their favour. Their strategies for doing this can be called ‘delivery assurance’. This is the key challenge that we must address. Let’s start by looking at what doesn’t work.

Ineffective Delivery Assurance Strategies

The received wisdom, by which I mean generally accepted best practices, suggests there are several effective strategies for ensuring project work is prioritised and delivered on time. In my experience, there is only one. The rest are all worthwhile or even essential practices for other purposes. But they have no material impact on delivery per se.

Before we look at why that is, here is some context. One of the mechanisms for driving accountability is the RACI matrix, which defines, among other things, who is responsible (i.e. for actually doing the work) and accountable (i.e. the manager who is answerable) for each task. Strategies designed to influence responsible individuals will be ineffectual because these individuals don’t have a high degree of discretion over their priorities. Their line manager, who holds their career in his hands, will set their priorities. Therefore, delivery assurance strategies should target more senior, accountable individuals.

With that in mind, let’s look at the most commonly cited strategies. In doing so, I hope we can dispel some of the notions that have clouded the issue of non-delivery and distracted from effective strategy for too long.

1. Build Strong Relationships: The idea here is to invest time in building relationships and understanding team members' motivations and constraints to create a ‘foundation of trust and cooperation’. Of course, it’s essential to foster good relationships with your colleagues. The problem with this strategy is that it simply does not create the strong incentives required to influence priorities. Your colleagues have careers to manage. They have families that rely on those careers. These considerations drive their priorities: not grinning project managers making small talk at the beginning of every meeting in a desperate attempt to ‘build rapport’. By all means, seek to get on well with your colleagues. But don’t expect them to place your priorities above their own as a result.
2. Leveraging Executive Sponsorship: This entails securing visible support from senior leadership to create organisational alignment and prioritise project work. The problem is that projects/programmes are typically cross-functional (at minimum, IT and business functions). Sponsors can’t effectively motivate Individuals in adjacent functions. Hopefully, this stands to reason. Furthermore, within the function, the executive is likely to sponsor multiple initiatives as well as emphasise the importance of BAU activities. This will inevitably dilute the motivational effect of any one of these.
3. Managing Through Influence: This strategy involves articulating project benefits to individual contributors and using ‘negotiation’ to create ‘win-win solutions’. Hmmm. I do agree that benefits must be clearly, widely and repeatedly communicated from the off. However, this will not significantly impact how work is prioritised for the same reason expressed in 1. above - it does not create strong incentives. As for negotiation, the project manager won’t ordinarily have anything to offer over and above the project benefits, i.e. they usually have little or no negotiating leverage.
4. Creating Visibility and Recognition: This strategy emphasises the importance of highlighting team members' contributions to make project work attractive and beneficial to their career development. Again, I agree with the importance of this. Leaders should always publicly recognise individual contributions. I’d go further and say that this is one of the most important aspects of leadership. But it’s not a delivery assurance strategy for the same reason expressed in 1. and 3. Above.

In summary, in the modern matrix organisation, the key people you will need to rely on to complete project deliverables will often be working under pressure and juggling the competing priorities of different projects and BAU work (which ultimately must take precedence). Influencing how they prioritise work will require you to create strong incentives that work in your favour. The strategies above don’t achieve that and can’t be relied on as delivery assurance strategies.

The Key to Delivery Assurance

In my career, I’ve encountered only one strategy that effectively motivates accountable individuals to prioritise project work. Like all good strategies, it’s a single idea, which gives it a distinct advantage over the all-too-common ‘laundry list’ of tactics masquerading as a strategy. I’ve come to view it as the most critical concept in project management. The idea is simply this: align status reporting to accountability.

Let me explain why this is effective. As a project manager within a ‘weak’ matrix organisation, you have little leverage, but the one thing you do preside over is status reporting. This can be a powerful tool for incentivisation if wielded correctly. By aligning status reporting to accountability, you showcase individual performance. The status reporting becomes a de facto report card. You won’t call it that. You’ll object to any such suggestion and point out that the status report is entirely objective (which you must ensure it is). But that’s how it will be perceived - as an assessment of individual performance - and that’s all that matters.

This strategy taps into the powerful incentives created by the fact that no one wants to be seen to be doing a bad job (much less failing). Many people are happy to do a bad job, but few are happy to be seen to. Several studies have examined the impact of transparent individual performance metrics in the workplace and found this to positively impact employee motivation. Most of us would have guessed as much - it’s common sense.

I received a lesson in this dynamic early in my career as part of a finance transformation project. The new system allowed us to see how many invoices each Accounts Payable team member entered, which was impossible before. Some were managing thirty a day, some five, and some none! As soon as the manager revealed that these metrics were visible to her, performance across the team rose to around thirty invoices a day. Of course, this outcome was predictable but nevertheless interesting to witness.

Some may argue that making individual accountability so transparent could create a culture of blame and discourage collaboration. I would argue that the opposite is true. Bringing objective clarity to the ownership and status of deliverables obviates the need for finger-pointing and blame. It’s been my experience that these behaviours arise out of frustrations caused by a lack of clarity over accountability. In these situations, colleagues fear being unfairly blamed or scapegoated and try to preempt this by pointing out where they believe the failings lie. If the status reporting is accurate and transparent, this is unnecessary.

Transparency is also as much a tool for proactive problem-solving as it is for motivation. If an accountable individual struggles to meet commitments, early visibility allows leadership more time to provide support, remove blockers, or adjust priorities.

Of course, defining accountability is already a cornerstone of project management; there is nothing new about that. However, what typically happens is that accountability is defined in a RACI matrix buried in a programme directory and is not readily apparent from the project plan or status reporting. Unless accountability is linked to performance and made transparent, it will have no effect. Furthermore, the execution of this strategy must be precise, as there are many ways to undermine its efficacy. I’ve provided a step-by-step guide to execution below.

Execution

1. Align the plan to accountabilities: For status reporting to be correctly aligned to accountabilities, you must first ensure that the project plan is correctly aligned. Start by ensuring that every line in the plan has both a responsible and an accountable individual assigned to it.
2. Ensure there are no hidden accountabilities: ?Review each line item to ensure there are no instances where the task will require significant input from more than one team. Accountability will be hidden in these cases unless the task is split out to reflect all responsibilities. I’ll give you a typical example. Requirements are often shown as being the responsibility of business analysts alone. However, requirements must be elicited from users who must make time for this. Frequently, BAU priorities mean that user availability is more constrained than anticipated, resulting in delays to requirements. BAs are often seen as accountable for these delays because user responsibilities are not correctly reflected in the plan. This obscuring of accountability leads to dysfunction. Instead, the requirements section of the plan should include a line item to reflect that users have a responsibility to provide requirements in specific timescales.
3. Get responsibilities and accountabilities explicitly agreed. Hopefully, this one goes without saying. For the avoidance of doubt, there should be a record of this agreement (email is fine).
4. Group by accountability for status reporting. How you organise your project plan is entirely up to you. However, for status reporting purposes, group the plan line items by accountability. This is, of course, the crux of our strategy. (If you have many accountable individuals, you’re probably defining accountability at too low a level.) Once you’ve done this, you can, for status reporting, bring the plan up one level or more (i.e. zoom out), safe in the knowledge that this will not obscure accountability.
5. Provide a RAG status for each task in the status report. Try to make the RAG status rule-based. This will help avoid endless disagreements.
6. Provide an overall RAG status for each accountable individual. This is a key metric. A red status may well be contentious and ruffle feathers. However, it’s essential that project managers have the courage of their convictions. You will come under pressure to delay reporting a red status to give an accountable individual time to remediate the situation. You must resist that pressure. Senior stakeholders rely on this so that potential problems are not concealed and can be addressed as soon as possible. It’s hard to overstate the importance of this. If you’re a project manager, as you value your job, avoid surprises.
7. Distribute appropriately. This requires understanding whose perceptions accountable individuals actually care about and visibly including those individuals in the distribution list. If the status reporting is not both visible and seen to be visible to the right people, its effect will be diluted.

Most projects don’t implement status reporting in this way. In my experience, doing so, as I have described above, will be the single most impactful thing you can do.

Conclusion

IT project failure is a persistent and costly problem, but you can significantly improve your chances of success. To do so, you must recognise that many of the commonly recommended strategies—relationship-building, executive sponsorship, and influencing through persuasion—are ineffectual for delivery assurance purposes.

The key is to create real incentives for accountable individuals to prioritise project work. Aligning status reporting to accountability and making performance transparent is a powerful way to do this. It’s anathema to ambitious professionals to be seen to be failing, and that visibility alone is often enough to ensure commitments are met.

This strategy won’t magically eliminate competing priorities - sometimes, you simply come up against corporate priorities that trump yours. Nor will it guarantee success in every case. But it will give your project the best possible chance of success in a world where competing demands constantly threaten delivery. If you’ve struggled with non-delivery, try this approach - it might prove to be the most critical tool in your project management toolbox.

— Stuart Bowes

Have you faced similar challenges in IT project delivery? I’d love to hear your experiences—what’s worked for you, what hasn’t, and how you’ve tackled non-delivery. Let’s discuss in the comments!

Or contact me directly to work with me or discuss your project and programme management challenges.

?? Get in Touch

?? Email: [email protected]

?? LinkedIn: Stuart Bowes

About Me

A seasoned Project/Programme Manager with 25 years of experience delivering complex business transformation initiatives in financial services, commodities trading and blue-chip organisations. Part-time founder of Simpbills and Sea Change AI.

#ProjectManagement #ProgrammeManagement #BusinessTransformation #DigitalTransformation #FinanceTransformation