Fixing Agile won't be easy, but it is possible
In 'Beyond Agile' we examined why Agile software development failed to meet the expectations of its founders, the catastrophic impact this had on project outcomes, and ways to fix it.
I recently stumbled across a talk given by one of the creators of the Manifesto for Agile Software Development, Dave Thomas, which underlines the dysfunction caused when the principles or values of the manifesto are misinterpreted or distorted to fit a corporate IT agenda. He half-jokingly attributes the downfall being our tendency to turn adjectives into nouns, so instead of the title Manifesto for Agile Software Development, people turn it around a truncate it to — Agile Manifesto, thus losing its meaning and purpose.
Beyond Agile explores many of the noteworthy 'lightweight' methodologies, some of which pre-date Agile. We did this to provide context for Agile and explain why it often fails. As one of the founders of the Manifesto for Agile Software Development, Alistair Cockburn, puts it:
"The Agile Manifesto was the product of seventeen people from different schools and backgrounds. No one person is responsible for the words we came up with — it is clear that it was the product of all seventeen people. The addition or removal of any one person would have changed the outcome, something we recognised and discussed at the end of that meeting.
Whether you think 'Agile' saved the world or poisoned it, be sure always to recognise that it grew from a rich compost (joke intentional) of backgrounds. The next time you read a would-be history of the Agile movement, look for all those names. If you don't see them, it is not a history. It is one person's recounting of their journey, years after the event (as indeed, this one is)."
Cockburn's statement gets to the heart of why Agile has died: few people understand or respect it for what it is, choosing instead to follow second-hand interpretations of the manifesto or, worse still, some management consultant's cheap imitation of it.
Agile failed not because it was inherently wrong or misguided. We said at the beginning of this chapter that the Agile Manifesto was an obvious thing of beauty, and its core values are spot-on. The problem with Agile is that it was released into the wild and then co-opted by people unwilling and unable to think about it or understand what it meant. Kent Beck, the creator of extreme programming and one of the original signatories to the Agile Manifesto, calls this the 'staring dog problem.'
"If you try to point something out to a dog, it will look at your finger," he explains. "If you explain an idea in terms of concrete practices — like test-driven development, pair programming, continuous integration — people will fixate on the practices and stop thinking."
Scrum and other spawns of Agile fail precisely for this reason. They shroud important ideas like user-centricity and fast iteration in practices, rituals, silly names, and guiding principles. The Agile manifesto itself, despite its brevity, left too much room for interpretation. And now it feels like we're back where we started in the late twentieth century with powerful technology and no idea what to do with it.
In the second chapter of Beyond Agile, we describe the three main reasons why the method was created :
1. We wanted to eliminate (or at least reduce) waste in software development;
2. We wanted to create software that people love to use; and
3. We wanted to deliver projects on time and budget.
An analysis of fifty thousand software projects conducted by The Standish Group found that in 2015, claimed an average of 29% of projects were successful (meaning they were delivered on time, on budget, and to a satisfactory standard). The remainder, 71%, were either failures or 'challenged' to meet expectations. Just think about that for a moment: starting a large scale technology project, you only have a 1 in 3 chance of being able to celebrate success — not very good odds. The same report in 2018 showed the situation had got worse, with only 23% of projects successful, and 77% wither challenged or failed.
Other sources corroborate these worrisome facts. In a paper presented to the 7th International Conference on Knowledge Management in Organisations, Stanley and Uden proved that software projects typically overrun their budgets by two hundred percent, and exceed their schedules by fifty percent.
But it's the cost of the failures that's the real issue here. The financial impact of project failure is no trivial thing. The data speaks for itself:
- In 2004, the total cost of project failure across the European Union was estimated at €142 billion
- The cost of re-worked and abandoned systems costs the US economy an estimated $75 billion per year
- In Australia, $5.4 billion is wasted each year on IT projects that don't deliver value or are abandoned completely
- One project alone — the infamously abandoned National Health Service patient record system in the United Kingdom — cost taxpayers ￡10 billion.
To put those numbers into perspective, the United Nations estimates that a yearly investment of $267 billion would end world hunger by 2030. A meager $175 billion per year would eliminate extreme poverty globally.
It's been a long time since the Agile Manifesto was written, and even longer since Winston Royce published Managing the Development of Large Software Systems. But nothing has changed, and no single methodology has improved software development. Software development is still a hugely expensive, wasteful endeavour that rarely meets expectations. And that's unacceptable.
As we approach the third decade of the 21st century, it shouldn't be beyond humans to come up with a better, more reliable way of developing software solutions. Just as garment makers and car manufacturers in past centuries industrialised their domains, so too software coders need to do the same. Digitisation begets automation.
Beyond Agile addresses some of the shortfalls of other methods, but it's by no means a silver bullet. It requires skilled people, disciplined practices, and full engagement with users and problem owners. Indeed it's quite likely that many of the principles distilled in Beyond Agile can and should apply to other development methods; this would go some way to fixing Agile.
In summary: to make Agile methods work, there are five principles to consider which align closely to the original manifesto:
- Start by defining the challenge or problem to be fixed
- Define the outcome in terms of a quantifiable measure
- Describe the 6–12 features or functions the solution must have
- Identify the first half a dozen users you'll engage at the outset and keep them engaged throughout
- Find 'The One' business problem owner who can represent and arbitrate all stakeholder requirements
Rinse and repeat.