Don’t always believe what the experts say – 2

Welcome back and thanks for hanging in with me as we continue to explore a set of “rules” that have been part of project managers’ life for years. These rules are typically treated as sacrosanct by many experts when, in fact, any reasonable research shows that they are, at best mis-leading and, at worst false. As I covered in Part 1, I have been calling these rules – Poonsbain Rules – as they seem to have appeared like aliens from the planet of Poonsbain (Confused? Then please read Part 1 of this series).

Let’s get to a relatively new and prevalent Poonsbain Rule.

Poonsbain Rule 3: Agile is better than Waterfall

To many organisations and experts, this rule is almost gospel today. Agile has almost become a form of religion. True believers, Agilists versus Waterfall-ists and so on.

A very quick history lesson is required here. In 1956, yes 1956, Herbert Benington at the Symposium on Advanced Programming Methods for Digital computers proposed the Waterfall System life cycle – Requirements, Design, Coding, Test, Implement with each phase discrete and the next phase could not commence until the previous one was complete. In 1970, Winston Royce, who worked for Lockheed, presented a paper on the Waterfall which many see as when the Waterfall SDLC became of age. As a baby project manager in the 70’s I could access formal methodologies such as Arthur Anderson’s Method 1 and SDM/70 which were pure Waterfall.

Interestingly, the Royce paper also covered iterative and prototyping concepts predating Agile by decades, but Waterfall ruled.

So, let’s unpick what we mean by Waterfall. The fundamental concept of Feasibility, Requirements, Design and so on are embedded in a standard problem-solving process (I have a great book called The Universal Traveler by Koberg and Bagnall, written for architects in 1972, which has the Waterfall problem-solving model as its core tool for architects). In fact, if you take a moment to consider, the same problem-solving approach is alive in Agile. How could you begin coding without understanding requirements and some design? Well you could but ….?

Simply, the underlying concept of Waterfall is not the problem .. it is how people used it to manage and govern projects.

The biggest failure in organisations implementing Waterfall was not in the nature of waterfall problem-solving process but in all the bureaucracy, paper-work and heavy governance that they added to it.  You know, mandated stage gates and so on. Waterfall became a standardised, rigorous and a failing attempt to force what is a creative process (developing new systems and products) into a repeatable, enforced process.

The second big problem is that standardized Waterfall methodologies were based on the assumption of perfect quality assurance. They assumed no defects in Requirements flow on to Design and so on. Iterative looping back to Analysis or Design to correct errors found during later stages was always required but caused havoc with governance, tracking and reporting mechanisms that assumed all progress was forward. All of you will be familiar with teams pushing forward into later stages of the Waterfall straight-jacket to keep the project on schedule until they reach the final phase – Test and Ship. There, defects that have “flowed down” from earlier phases are finally confronted, resolved or worse, shipped. Think of a balloon being squeezed at one end.

As early as the 1980’s, modified approaches to Waterfall designed to minimize bureaucracy and the impact of defect rippling such as Rapid Prototyping, Iterative Development, Time-boxing (my favourite) and Rapid Application Development (RAD) emerged to address the quality challenge as well responding to increasing pressure to compress time-frames and costs. In construction and engineering areas, radical approaches such as Fast-tracking emerged (Fast-tracking was widely used in the construction of Australia’s new Parliament House). Basically, construction starts before design is complete and increased risk and defects are accepted in a trade-off against time.

These “radical” Waterfall models also included more focus on delivering value to business folk faster and included processes such as JAD which greatly increased business IT partnerships. My collaborative approach to project planning (RApid Planning) which was collaborative and inclusive of all experts (business and IT) was pioneered in the late 1980’s.

As a side note, I consulted on the first $1 billion, 10-year Tax Modernisation program in the late 1980’s and early 1990’s, where by using 6 – 9 month time-boxes or increments, we delivered major components using Waterfall with minimum bureaucracy and maximum business engagement (all Program Directors were from ATO business units – see my next article). 

Concurrently, technology and programming language evolution together with innovations such as Kent Beck’s eXtreme programming (XP) approach, lead to the seeds of Agile being planted by the 1990’s.

At its basic level, the Agile Manifesto and Agile development was, in reality, a back-lash to the persistence and further growth of bureaucratic governance processes associated with  “Waterfall control-freaks”. 

Another argument put forward by hard-nosed Agilists is that Agile is based around long-lived teams. Again, quick history lesson, I was part of the same team for over 7 years in the 1970’s.  We moved from project to project using Waterfall. I worked with long-lived teams throughout the 1980’s including the ATO Modernisation program. Eventually, outsourcing and off-shoring, driven mainly by false cost considerations, led to the loss of long-lived teams which, to give Agile its dues, has now become more acceptable again to smart organisations. What’s more, variations of Daily Stand-ups were common in some of the large “waterfall” projects I consulted in during the 1980’s and 1990’s.

To conclude, if you strip out all the bureaucracy, handoffs, governance and associated baggage that became associated with Waterfall and, provided you have great QA processes coupled with some form of time-boxing, Waterfall is still a highly relevant development approach. Simply, Agile is great for some projects and real Waterfall is great for others. A Hybrid approach, where parts of the product or system are built using Agile and some parts using Waterfall will probably be the most common approach going forward.

So, I rest my case. Agile is not better than Waterfall, it’s just different. Another Poonsbain Rule is de-bunked.

In writing this article, you might have noticed, I avoided the questions of large-scaling Agile and Agile as an organization-wide concept. I’ll get to those later this year …

In my last article in this series, I will explore another Poonsbain Rule that even I had become a follower of.

Poonsbain Rule 4: You need to manage stakeholders

Keep tuning in ...