Agile is dead, long live the Waterfall

It's fascinating how quickly businesses blame a methodology for project failure instead of reflecting on their own operations. A recent study found 268% higher failure rates for Agile software projects compared to waterfall projects. Interestingly, projects with clear requirements documented before development starts are 97% more likely to succeed.

Many might think this means waterfall is better and Agile is too fast and loose. However, good Agile programs often have better structure and success rates than many waterfall programs. Complaints about Agile being more expensive or causing more issues due to changing requirements are common, but these issues often stem from poor implementation and lack of accountability.

The Agile vs. waterfall debate continues, highlighting that we haven't learned the critical aspects of Agile: Go slow to go fast, Collaboration, Evolution, and Metrics that matter. Perfect Agile programs are rare, and each organization's implementation looks different due to their unique challenges.

Rather than debating which methodology is superior, the focus should be on understanding the specific needs of the project and the organization's readiness to embrace change. Successful implementation of any methodology hinges on strong leadership that not only understands the chosen approach but also actively champions the cultural shift required for its success.

The iterative, cross-functional approach of Agile can unify fragmented organizations and help develop new capabilities like Generative AI progressively such that it is embedded into the organization to realize persistent value. In the fast-paced digital era, Agile allows businesses to pivot and adapt quickly based on internal and external insights.

Key Success Criteria for Agile:

Go Slow to Go Fast: When leaders get approval for a new project, there is a desire to ‘get on with it,’ which means teams are asked what they will build first and how quickly we will see value. This is usually linked to an SI being onboarded and pushing to execute so that paid resources can be put to work. However, in a world where quick wins are often prioritized, 'Go Slow to Go Fast' challenges the notion that speed is synonymous with progress. Slowing down at the beginning to spend more time on defining the journey, the MVP (as in what will do as a first release – i.e., not minimum marketable), the flow between systems, the new processes, quality of data, etc., can help provide a runway for teams. In any program, I would encourage a runway of +3 months of dev backlog for teams to work on so that the engine can run, learnings can be incorporated to adjust direction, and resourcing can be aligned to be as effective as possible. By investing time upfront in thorough planning and backlog development, organizations can avoid costly rework and setbacks later.

Collaboration: The traditional boundaries between tech and business need to be addressed at the start of any program, as running Agile means a degree of collaboration that has likely not been seen previously within the business. There are many examples where I have seen the role of ‘Product Owners’ being picked up by IT or Digital teams in the form of ‘Shadow product owners,’ which in the end becomes a band-aid for a bigger issue. Alternatively, assigning junior representatives who are not empowered to work in a squad means teams can’t operate as a single unit and end up feeling a stop-start to all work being done. Ultimately, functioning as a cohesive unit with shared KPIs to evaluate quarterly performance is essential. This approach also ensures that each team member—from development, product ownership, and design to data and security—has a clear understanding of their specific role. Successful Agile programs are recognizable where squads work with minimal guidance, and in many cases, the distinction between roles is difficult to spot as team members support each other toward a common goal.

Evolution: There is this amazing expectation that programs will deliver immediate value or that all squads are just as successful as each other from the start. We often talk about a Human transformation; however, while we can transform systems or processes, it is very difficult to transform people in the way they work or how they deliver outcomes. Think about what it’s like to immerse yourself in a completely different culture—the customs, how things are done, the nuances of communication—all of this takes time. Similarly, in setting up squads, it takes time for the harmony within a team to settle, and creating an environment for each squad to learn and evolve is important. We also need to ensure we focus on how each team is maturing on their own path versus comparing teams, as this can create a culture of success vs. failure and teams not wanting to be on a perceived failing team. This doesn’t mean you shouldn’t look to manage team performance, either through coaching or trying to change individuals in the squad to create the right dynamic within each team. However, ultimately knowing where a team is starting from and how they progress their maturity in terms of increasingly independent, supporting each other, and recognizing how to partner with other squads are all things that should be looked at progressively.

Metrics that Matter: In the excitement of getting on with execution, it can seem obvious that the metrics will simply be the outcomes we are looking to realize from the Agile program. However, the metrics that you start with shouldn’t actually be the final metrics you use to assess the success of the program. Early metrics for a customer service transformation could be the transactions that are possible via self-serve vs. assisted; this could evolve to the number of transactions now being realized in self-serve, to finally the reduction in the number of required frontline customer service team members. This is important to understand as jumping to the final outcome metric can leave both leaders and squads frustrated in the lack of progress, unnecessarily adding pressure and increasing pivots in deliverables with outcomes being missed. Whether using Agile, Waterfall, or a hybrid approach, the key to success lies in an organization's ability to remain flexible and adapt its strategies as the project evolves. It can be difficult for leaders to understand how metrics can evolve; although, this approach can provide validation of the progress being made and the link to trusting teams to be successful with metrics that matter at each point of the journey.

Of course, there are other aspects that are important here, including creating performance criteria for vendors and SIs you are working with, ensuring a greater focus on ‘Out of the box’ capabilities vs. customization, as well as defining the funding framework. Regardless, with a goal of understanding what can make Agile programs more successful, the aspects I have described above can provide the right guardrails to set squads, a transformation program, and the organization as a whole up for success.

As we navigate the complexities of modern project management, I would encourage leaders to critically evaluate their current methodologies and consider how these best practices can be integrated to drive meaningful change and sustainable success. Despite the study results, in my view, Agile is not dead. The risks highlighted emphasize the need for proper preparation and guardrails to ensure successful adoption and execution.

Would love to hear other views on what determines a successful Agile program or if Agile is viewed as no longer being fit for purpose.