The Cascade – An Agile Waterfall - Part 1

During my career, I’ve seen different variations of the how projects are executed and I am sure so have you.

The most common turf war I encounter is the whole Agile vs Waterfall debate - and I wanted to weigh in not so much because I have a preference for one over the other - it's because over the years I've come to realize the debate itself is futile.

Let me explain.

There are various flavours of Agile & Waterfall. I’ve also been in the middle of many a an argument on the benefits and applicability of each and I have to admit I’ve found myself in different camps at different times.

Over time, I’ve realized that we rely on these methodologies to attempt to guarantee a delivery outcome within time & within budget, however in reality their execution is never an exact science and depends so much on the team involved. It wasn’t till I went through the ‘crucible’ of several really tight projects, did the full magnitude of it hit me.

My current and evolving position is that neither method is in orders of magnitudes superior to another and when one groups talks about the superiority of one method over another, they are viewing both methods through a restrictive lens, biased in favor of their organization culture, personal preference or perceived benefits and pitfalls.

Criticisms towards the rejected methodology tend to be in the extreme.

Recently a very astute project manager mentioned to me that these methodologies are for our benefit, they are here to help ease our burden and they are here to work for us. We are not here to work for them.

With that simple observation, I thought I’d share some of the common areas where different elements fall through.

This blog post will be broken up into multiple parts to facilitate ease of consumption . This is part 1 .

Note: I am not a certified Agile Coach or Scrum Master so these experiences are purely based on my first hand experience on the ground across several projects

A special thanks to Eugene Colborne for his incisive insights for this blog as well as helping me appreciate various nuances in this area over the last few years

Agile Project Methodology

Basic principle: At its heart, ‘Agile’ simply refers to an iterative method of developing, managing and delivering an outcome or software which is of value to the customer. One of the most widely adopted flavors of Agile is Scrum wherein the team typically works in short iterative sprints, using a Kanban board to organize their work, and daily standups to communicate progress or blockers.  Theoretically this allows them to achieve the most valuable requirements until the “ budget runs out” or “a deadline is reached”. In this way Scrum , again theoretically “ avoids budget overruns and delays”.

If you don’t completely agree with the bit about budget or delays, congratulations! You are also in the same camp as I am. More on that in later sections on how Agile can get out of hand

What Agile Has to Offer :

Agile methodology allows you to focus on quality of the solution up front . If there are different teams involved within the same project, it allows you to manage their different paces together by allowing for greater flexibility of what work can be picked up in specific sprints as long as all teams attend scrums or scrum of scrums.

Egos have no room in an agile process and over time, the’ True’ solution can be discovered as customers’ requirements get validated and refined based on the how sprints progress, as long as performance radiators (Burn Down charts or Scrum Task boards ) are diligently maintained and sprint reviews and retrospectives conducted.

What you need for Agile Scrum to Work for you :

Every project has a limited time and budget availability and to believe  that the Agile process will simply allow the development of the true solutions within cost and deadlines on the fly is not advisable

What you need at a bare minimum are:  

1. A detailed enough plan or vision for the overall project of what needs to be done what would the end result enable your customers to do (Project Goal, Backlog and High-Level Design)
2. A backlog that is at least 75% to 85% complete high level requirements that qualify for a minimum viable product or MVP. This backlog must clearly call out requirements that make up the foundation layer vs add on features
3. An established and accepted  process for getting detailed acceptance criteria ( definition of ‘done’ ), sprint show-and-tell, and dealing with changes, and managing enhancements. Velocity charts and sprint reviews are essential to demonstrate impact of any changes on sprint and project goals
4. An open culture of communication, freely, and without prejudice. This I would say is the single most important requirement whether you are working onshore or offshore. A willingness to speak up and to be concise during scrums is also imperative as well as knowing what to communicate. A good Scrum Master is the answer. An engaged and knowledgeable Product Owner is imperative.

How it can get out of hand

1. While it allows you to discover the true solution over time, the Agile process can seduce you in believing that you can forever tweak a solution. There will ALWAYS be improvements that can be made and it is imperative that both the dev team and client team agree, on a “good-enough baseline” set of requirements that suit their overall goals and is on the roadmap for future enhancements. This baseline could come within a single sprint or several iterations.
2. Meetings and more meetings - believe it or not there can be something as too much communication where your technical team is spending the majority of time in assessing complex requirements,  incoming enhancement and changes for technical feasibility This is best avoided by having an excellent technical BA and some up front thinking from the business product owner on what they want out of the solution
3. Build Now – Adjust later – Following this to the letter of the law is very dangerous. Just because it’s agile, doesn’t mean you forgo design patterns and future proofing your design. Quite often a development feature can be prematurely included in a sprint just to mark it done It then has to be uprooted or re-written to accommodate incoming specs from other teams, or evolving requirements. While this is not bad per se , it will have an impact on your budget spend and timelines. It is very important to balance out prioritization of feature development with progress demos to sponsers  
4. Tasks that are important but do not account for overall sprint velocity – These are tech or business sinks that can be essential to get the project moving along ( e.g. setting up a dev environment as a simple environment) . This can lead to a unfair portrayal of the team’s productivity as these activities do not add directly to deliverables . This leads to undue pressure on the team and can be skipped quite often to the detriment of the project. In essence , the Scrum Master needs to showcase any and all work done by the team in sprints
5. If you do not execute sprint reviews or retrospectives, ( and quite often they are relegated in favour of ‘getting more work done’ ) , then you will never be able to truly understand any challenges in the project. This combined with Agile’s openness to change without documented change control ( in terms of why and who authorizes the change ) , will easily cause you project to overspend

Most misunderstood aspects of Agile

Requirements cannot be fully known in advance : Yes and No. At the very least there has to be an appreciation of underlying information architectures and how different foundational elements interact with each other before the build has begun. Over time I’ve seen projects spend at least an extra 30 to 40% more in operational and dev costs alone when a little proactive forethought would have helped layout foundation to build on

This is especially exacerbated when you have a solution that works across different scrum teams and projects ( that may not be running on agile ) working along separate timelines

Agile gives better estimations:  Yes but only over time with a consistent team. As the dev and client team work together more and more with a high degree of communication, they will get better at understanding requirements and thus estimating the user stories.

Agile means you can change your requirements whenever you want: Sure as long as you have the time and money to support the change :) . You also need to ensure that resources are available and have not been pre-assigned to other engagements.

Agile allows iterative software releases over short durations: This can require organizational process change to allow iterative releases. This can also come with the addition of specific software to manage and synchronise change/release management.

So Agile has can be really powerful provided you pay attention to how the methodlogy can work for you . What about Waterfall ? Surely that is definitely an inferior methodology and can't compare ? - Well, not quite - it's not that simple. You can read all about it in an upcoming Part 2 in a few days

Till then , tell me what you think. What have your experiences been with Agile and Waterfall processes ? Where do you see yourself ?

P.S. The title picture is Wentforth Falls in the Blue Mountains just outside of Sydney