Why we can’t get rid of the waterfall

Even though iterative software development is the dominant methodology for more than 20 years now, the single-step waterfall approach keeps making appearances, it is not “dead”. In this short essay I will explore how, while it is on a purely technological basis highly flawed, the waterfall “V”- model could have other reasons for its survival.?

The 5 steps of the iterative technological process

When I was 12 or so, we learned of the 5 steps of the technological process in school. It made such a profound impression on me that I still remember them... or I actually don’t and it’s just that these steps have reappeared in related forms (Deming Cycle, Agile, DevOps) over and over through my career that it just seems I remember them verbatim…

Here they are:

1. You have a certain pain and an intuition that a technological means, a tool, can improve your condition
2. You design the tool, first in your head, then on paper (or CAD :-)
3. You implement, build the tool
4. You start to use it (!!!!)
5. You evaluate your experience and consider improvements, which are the start of a new cycle, and you’re back to square 1!

The idea to see step 4, use, as an integral part of what is essentially technical design is crucial. Waterfall (“V”-model) processes, which we will come back to, run the capital risk of over-disentangling all the humans into separate designers, coders, testers, sponsors users, etc. where magically the user gets a product that is supposed to work, because it has been “tested”, “verified” and “validated”... but forgotten to be used as in for real!?

To overstate the case: at Apple they design products that they themselves like to use, not to test.

Look at it this way: If you zoom out for 20, 50, or a hundred years and you look at say the automobile: the process of stepwise refinement of that tool is obvious. So obvious that no one in his right mind would conceive that a 1960s car could emerge in the 1950s as the result of some definitive process consisting of single steps of Conception → Architecture → Design → Integration → Commissioning/Acceptance. Could you build even one??

Particularly in software development the parallels with life’s evolution itself are striking. There’s evolving, i.e. improving, in a non-linear, quasi-random way and not by grand design (by committee). Worse designs and techniques are slowly abandoned, they fade out, better ones are slowly adopted, and they fade in (into the mainstream).?

And yet, the waterfall, the single V-shape, just keeps coming back in project plans. Why? Because of the limited investment horizon of the sponsor?

Projects are by definition finite

When is an iterative process finished? Never. Will we need one, two, three, or ten cycles? Who knows? When we are honest, we don’t know when we’ll be happily satisfied in the use of our tool. And that does not change by now calling it now a product, because we’ll monetize by replicating said usage/usefulness.

Indefinite simply won’t do for approving investments. By definition, the investor does not tie up indefinite amounts of money indefinitely. How to calculate a return on investment on that?

So sponsor Philips, in 1995 would say: look people, you get two cycles, “call them what you will”, but after those, we must go to market. (They were called basic functionality and full functionality). Oh, and here’s a fixed amount of money and time. Sure, in the case of a strategic product, like our first DVD player, we’ll be more sensitive to the time than the budget.

Projects have risks

The primary fear of a (software) project sponsor is that the project will go on forever, given the notorious overruns of the past. Hence a contract is made between sponsor and team that sets both time constraints and formal acceptance criteria. By this time, such and such is to be delivered. That defines a discontinuity in the eternal improvement cycles and rules out pure, unbounded iterating, doesn't it?

But did this contract effectively eliminate the risk that the unknowns to overcome are properly understood? Experienced engineers know that we tend to find out when it is too late.?

And so how do they mitigate the risks concerning delivery on time, on budget and with satisfactory quality, i.e. satisfactory usefulness, given their grave doubts about the effectiveness of the single-step waterfall process to this end? Well, by employing hidden cyclical processes anyway. Not an outward extra iteration of this instance of a software product, but iterating by re-use from within:

• The experience of the engineers and project managers coming from sufficiently similar projects in the estimation phase
• Reuse of working parts, and components (stand-by-themselves results of previous “V”s)
• By using that one large “V” as a scaffold, get going… and in the heads of the people do immediately something else: Pull all the activities way forward, push the buffers to the end (as per Goldratt) and start iterating as hell.?

Some do this only in their head, others are in a dialectical process at the coffee machine, and still others (the most experienced ones?) as much as they can in the real world, looking at similar use (cases)

Is Waterfall a mere illusion?

Is all this to say that Waterfall is utter madness? The countless successes of cloud-based software-as-a-service offerings that were done in pure agile and went out as Minimal Viable Product, or an Alpha version for the older among us, seem to suggest it?

But the answer is more nuanced. Such pure applications sit on top of an enormous iceberg of proven “stuff” underneath, that is not “built” (by the last party in the value chain, in the same “V”) but… well let’s say it’s “grabbed from a shelf”. A brick and mortar house is not grabbed from a shelf, it is clear that there is an order, sequencing going on between pouring concrete foundations, up to applying the wallpaper… paper that you can change in a quick last sprint… unlike those foundations, right? So “systems” are not “apps” and visa-versa. Please keep that in mind.

The other aspect I’d baptize product, or service, or organization, or even continent Grand Strategy.

A grand strategy “simply” matches ends to available means and it “simply” holds up a single and consistent north star as the far aiming point. The grand strategies’ end goal can be stated in one sentence (as opposed to 500 pages of requirements specification)

• We will win this war
• The U2 will have 2 big wings, a hull, and engines like any airplane before it, but it will fly way higher than any other (the example of amazing fast project management in Goldratt’s book the Critical Chain)

Waterfall as the rallying “North Star”?

In large software projects, the goal of the “V” is not pure technical excellence, i.e. (Software) Project Victory are other things, over and on top of just happy users.

The strategic goal of a time-limited project can be, maybe should be, to make a major push in one key dimension, keeping novelty in the other dimensions reasonably low (by reuse).

Examples of such dimensions: build a new partner eco-system, gain a critical market share, change the car engine, …??

When executed well, the outcome is first and foremost a new organization that is ready for the next "V", while it kept you in whatever game it is that you are in, or better, got you a step ahead.?

To put it philosophically and/or psychologically: the physical stuff that technologies are made of does evolve/improve incrementally and in the world, but groups of humans, each of them with a mind and spirit of their own need that big single, lofty and final light in the distance to effectively rally to. You can’t rally to a process, can you?

That, more than technologies, might explain the resurgence of the Big “V”, time and time again. Indeed, while we are sliding down that project waterfall adventure slide, all the time in the distance we keep the other leg of the V, pointing the way out, steeply upwards!