Hubris at the Heart of Agile

[originally published on medium.com as The Hubris at the Heart of Agile]

A large company I was working with (and shall remain nameless) had commissioned a vendor to build and supply a software product. At the first major review it was discovered that the software was largely incomplete, full of bugs and the vendor wanted to issue a change request for several million dollars.

While there was much noise around the circumstances under which this happened, no one seemed to know what the real root cause was.

I did a little digging.

The locus of the problem seemed to be the old chestnut that the vendor didn’t understand the requirements… or that the customer didn’t specify them properly, depending on which view point you took.

But looking at it, it seemed there was a deeper problem?—?a more fundamental misunderstanding.

In the words of the PM brought into troubleshoot the debacle, “the Vendor doesn’t appear to know how to manage and execute a fixed price contract (maturity).”

Hmmm….. why would you enter a fixed price contract if you thought you couldn’t deliver?

A little digging unearthed some interesting artefacts.

The vendor used user stories to document requirements. Their schedule was defined in terms of sprints. They had weekly project meetings and status reports and show cases.

In short, they were an agile shop.

Now the customer and the industry they’re in understand fixed price contracts?—?they use them all the time. In fact the level of sophistication I’ve encountered in their contractual framework makes most supply contracts I’ve seen look quite mickey-mouse by comparison.

The position the two parties had come to was something just short of armed hostilities. There were lawyers in every meeting, lots of exchange of documents and claims and a very, very uneasy negotiated settlement.

So what ultimately caused this schism?

I think it was a simple misunderstanding.

You see, in the customer’s mind, a fixed price contract implied fixed cost AND fixed scope.

But (I think) to the agile vendor, the contract implied fixed cost and VARIABLE scope.

In agile, the emphasis is on close conversation with the customer to discover what benefit can be achieved with the money available. At each iteration functionality is delivered and the remaining scope is reassessed to see what can and should be delivered.

The onus is on the customer to select what they want delivered from a prioritised menu of options on offer.

Now this led me to an interesting question.

Why, as a customer, would you ever take fixed cost/variable scope, when you could have a fixed price contract (scope+cost)?

But, why as an agile vendor, would you not offer to fix price and scope?

The great claim of agile is that it offers more certainty and less risk than methods like waterfall, which are built on the false premise of being able to specify upfront what should be delivered. Waterfall it’s argued hides an ugly reality behind promises of fixed milestones and deliverables (and I’ve certainly found this to be true).

If agile is more reliable, more certain and delivers more value why wouldn’t an agile team offer to fix price and scope? Why wouldn’t you offer to fix the delivery date? The commercial risk would be lower, so managing a fixed price contract would be safer?

I’ve certainly never heard of it happening.

As a customer I would prefer the certainty, but why wouldn’t an agile vendor or team offer it to me?

Because they don’t honestly believe they can deliver.

They certainly believe the process is better for them, it feels better. They (should) get treated more humanely than in a waterfall project managed with s-curves and risk logs. But they feel that the ability to deliver value is actually out of their control, it’s dependent on outside factors?—?like the client, or the technology or management. Otherwise they should be happy to take the risk of a fixed delivery… and own it.

Certainly every agile team I’ve ever reviewed is no less evasive about their role in the process than any waterfall team. If a project goes wrong it was always ‘the business’ or ‘the code’?—?never the process.

Maybe agile or waterfall aren’t the problem?—?maybe discipline is the problem?

Maybe you should pick a method and actually stick to it and try and improve it as you go.

But process is an anathema to many people and we’re taught at an early age that success comes from personal effort so people don’t like the ‘constraints’ of process. Except in a few lucky professions, we equate process with rigidity, loss of control and boring repetition.

Agile was born as a reaction to overly formal models of software development.

And it wasn’t wrong. The models didn’t work, the analogies were wrong.

But there’s an underlying and unspoken assumption in agile that it delivers better outcomes than previous methods. Now, I certainly agree that agile, done well, can deliver better outcomes than waterfall… but agile done badly, delivers just as badly (if not worse) than other methods?—?as the example above shows.

Waterfall has some discipline built in, process discipline. But agile relies on personal discipline.

To assert that agile by-and-of-itself delivers better results is hubris beyond belief.

Mind you it’s the same hubris that’s been dogging software development for the past 20 years.

“We’re not like engineers/architects/anyone else! We’re special! We’re creative! Every piece of code is a unique and beautiful snow flower! ”

Oh dear.

In an agile gathering someone once asked about the apparent assumption in agile that you must build an extraordinary team to deliver extraordinary results (people over process). What do you do if you only have ordinary people? In reality, how many teams will have extraordinary developers?

There was some confusion and then one of the louder mouths in the crowd said, “But agile *naturally* attracts smarter people, so that’s not a problem.”

He was serious.

I couldn’t let this one lie and turned to the audience, “Put your hand up if you’re an above average programmer?” I asked.

Guess how many did?

Half of them were wrong.

Maybe we need to stop focussing on the people so much and focus on the process. Maybe if we focus on the process then ordinary people will produce extra ordinary results.