Why Agile Should Be More Predictable Than Waterfall

This is a chapter from the upcoming book: Achieving Business Agility at All Scales: Transforming Your Organization with Lean Thinking and Lean-Agile Patterns

Many executives have been led to believe that Agile is inherently less predictable than a waterfall approach. However, Agile, when wrapped in Lean-Thinking, can be more predictable because it enables working directly on the true causes of unpredictability in software development. Waterfall’s large projects and stage gate approach cause delays in feedback, workflow, testing and integration. These delays inherently create a significant amount of rework (redoing requirements, reworking code that missed requirements, finding bugs, thrashing during integration). This work, of course, is never planned for, and therefore result in inherently bad estimation, not to mention that eliminating this extra work improves productivity.

In addition, Lean-Agile methods have us focus on smaller, higher density pieces of work. This alone helps bring our systems under control, as observed by Eli Goldratt – “Often reducing batch size is all it takes to bring a system back into control.”

The key to having predictable results is to eliminate delay, which creates unplanned for work. This requires smaller batches of work that are properly sequenced in importance, properly formed teams (or groups of teams) and solid technical practices. Agile suggests additional methods such as Acceptance Test-Driven Development, automated testing and continuous integration. These directly improve understanding of requirements (through quick feedback), code quality and risk reduction.

Although waterfall and Agile approaches have different mindsets, some of Agile’s methods can be incorporated into a waterfall. For example, making both projects and planning cycles shorter. Improving methods will not make an organization Agile, but it may help set the stage for a true transformation.

I’ve heard it said that “we can’t go Agile because we need predictability.” This is an understandable sentiment because many Agilists have dismissed predictability as something impossible to achieve. However, there are two camps in the Agile community – one that is team centric and another that wraps Agile with Lean-Thinking (we call this Lean-Agile). A team centric approach, when applied at scale, will often not achieve predictability because the team is only one part of the value add cycle. But a Lean approach is holistic and, although it may not be possible to predict exactly what must be built, it is possible to predict how much can be built. With proper selection of the work to be done,

However, more than simple iterations are required to achieve this. Lean-Agile should not be just a mindset shift, but must incorporate solid product management and technical practices. Let’s consider why projects (regardless of approach) often do not achieve the predictions they make. The most discussed are:

• what the customer wants is not really understood (note, throughout this blog I will use the word ‘customer’ to refer to anyone who is driving requirements on behalf of the real customer(s))
• the time to build something is unclear
• the code is fragile and hard to change

There’s an even bigger one lurking here (which I’ll discuss a little later) but for now let’s look at these.

What the customer wants is not really understood. Customers often don’t know what they want until they see what they don’t want. This implies they should be shown “what they don’t want” as soon as possible. This is why it’s important to build small segments of work quickly. This also shortens the time between all stages of the work (requirements, design, code, test, integration). This combination of quick feedback with coding and testing being done together enables quick course corrections as both the customer and the development group learns what is truly of value.

Acceptance Test Driven Development (having customers, developers and testers, specify requirements in the form of test-specifications) achieves better clarity on what is needed and avoids many misunderstandings altogether. Getting the customer to provide requirements in this precise manner changes the nature of the requirements. Vagueness is no longer allowed and customers’ understanding of they are looking for improves as well.

The time to build something is unclear.  The answer to this is again to go small. Estimating the time it will take to complete a large chunk of work is typically less accurate than estimating the time for smaller things. Some people have ridiculed estimation because they are often very inaccurate and managers sometimes abuse the teams for that. But bad estimates are more of a symptom of other issues – unclear requirements, poor code quality or even fear of management. But avoiding estimation abandons looking at the true underlying reasons.

The code is fragile and hard to change.   Poor code quality will slow things down, but this does not have to be the path taken. When code is fragile and/or is difficult to change it will not be possible to accurately predict how long things will take, even for small changes. But that’s because of the poorly written code which is both difficult and risky to change.

Waterfall’s approach addresses this issue by doing all coding first and then testing afterwards. Their is often a perceived need to do this because of the need to batch test. But this doesn’t get at the true issue (poor code quality). Accommodating poor code quality will not improve it, but tends to get the code in a slow death spiral that will eventually need to be re-written.

Poor code can be improved with automated acceptance tests. The time test automation saves is only part of its value. It also enables re-factoring of the code in safety, because defects introduced will be detected quickly. Developers fix these bugs much faster as well because they are detected while they are still fresh in their mind. This is a huge time savings. Most developers think they spend a lot of time fixing bugs. But on reflection they see that most of the time they spend is in finding the bug. This is not mere semantics. Bugs detected immediately are fixed much faster than bugs detected weeks later (even when the code hasn’t changed).

The Common Themes

The first common theme here is that smaller batches of work allow for greater efficiency by providing quicker feedback, better estimation, and shortening the delays between requirements, code and test. This efficiency is due to the fourth issue I earlier mentioned I was going to discuss. This is that delays in feedback, workflow (e.g., waiting for people) and detecting errors will cause additional work to be done.

It is common to focus on how to work faster or better. But it is more effective to discover how to avoid creating the need for rework. Although much rework is due to misunderstanding, having quick feedback significantly lowers the amount of rework required. Consider how much time is spent on redoing requirements, building features of less value than originally thought, fixing bugs, manual testing, thrashing during integration, …. It should be apparent a significant amount of work done in projects was created by the afore-mentioned delays. The focus must shift from doing things faster to avoiding delays between the work steps – achieving fast feedback along the way.

Why Waterfall Will Inherently Provide Bad Estimates

Consider the waterfall planning process.   Larger chunks of work are planned to be done in a sequence of separate steps – requirements, design, code, functional test, integration, and system test. This exacerbates the afore-mentioned issues by increasing delays in feedback between each step and overall feedback achievable at completion. This creates a significant of work due to these delays. However, how much of this time is planned for at the start of the project?

None. Zero. Nada. 

Estimates in waterfall don’t consider the work that derives from delays – work that often takes as much or more as the planned work. Planning session typically don’t include the question “how much time should we set aside for missed or wrong requirements, pitfalls in our design or all of the extra time it’ll take to find the bugs because testing was delayed?” In other words, one reason estimates are bad is that they do not include a significant amount of the work that needs to be done.

So Why Should Lean-Agile Be More Predictable?

Lean-Agile takes a different approach with requirements, design and coding and testing. By working in small batches and continuously course correcting, the amount of time spent correcting misunderstood requirements is lessened. However, technical methods are still needed. Test-first and automated testing also results in better code. Considering writing tests before writing code is a major lesson from design patterns (although not stated explicitly in the iconic Gang of Four Patterns book, Both Design Patterns Explained and Essential Skills for the Agile developer describe this in depth). And finally, by using smaller batches of work the time between code and test decreases, thereby lowering the finding time even more.

Another difference exists in the approach to architecture. In waterfall, developers are supposed to design for the stated requirements. They have learned, however, that requirements will change on them and tend to over-design to accommodate this. Lean-Agile, on the other hand, suggests to design in a way that accommodates change. This is known as emergent design.

An Extra Benefit of Lean-Agile

Building things in small batches results in learning faster about what is needed and enables pivoting as required. Feedback from the customer uncovers the true value of each feature or story. Furthermore, customers learn that they don’t need to ask for everything up front. Instead, they are allowed to adjust their requirements as they see what’s being built and better understand their needs.

Waterfall, oddly enough, encourages customers to ask for everything up front because little justification is required other than “I need it” when initial requirements are stated. Anything specified later, however, must go through a change review board. Customers quickly learn to ask for whatever they might need up-front. Lean-Agile teaches customers they only need to specify what they need soonest, and that they can adjust as they learn more. While a general understanding of the entire system is required to enable an estimate for building, only the general size of the work is needed. The details will come later as more is learned. Accurate estimates on the rate of value delivery enables this

Management’s Role in Lean-Agile

Management’s role is to create a great work environment for teams to autonomously work in a manner that achieves the strategic vision of the organization. They should leave teams to self-organize, but need to help create the context within which the teams can work.  This, of course, is done by getting input from the teams as well as other parts of the organization (e.g., product management, business intelligence, ops).  Managers are their not to manage people, but rather to improve the eco-system within which people work.

Lessons from Lean-Agile Can Improve Waterfall

Although Lean-Agile is as much a mindset shift as it is a change in how work is being done, it is possible to take some of its lessons and apply them to a waterfall approach and get immediate benefit. The key is to better work on smaller chunks of value. In other words, two 3-month waterfall projects will almost certainly deliver more value than one 6-month project. The reasons are delays are removed by having a smaller development cycle and that pivoting on requirements is possible after the delivery of the first chunk of value. A further benefit is that half the value of the entire project is delivered in half the time. To learn more about this see The Business Case for Agility.

Summation

It is true that many companies that are not hitting their targets using Waterfall methods may not hit them initially when they switch to Lean-Agile methods. But Lean-Agile provides a vehicle for improving the underlying causes of lack of predictability. Repeated failings in waterfall persist because management often thinks it would work if “people would just follow the plan.” But the plan doesn’t include things that are certain to happen (extra work due to delays). This makes “following the plan” inherently impossible. Lean-Agile provides the opportunity to get to the root cause of delays that are creating additional work that isn’t being planned for. By working on the root causes of the problem instead of the symptoms, Lean-Agile promises a path for true improvement.