Q: Is our need for certainty an anti pattern for Agility?

I've spent most of my work life creating plans only to watch my beautifully crafted visions of a utopian future be cruelly disfigured by life until they are no longer recognisable.

I imagine that, like me, many of you who are responsible for planning and delivering things have sat in front of your line managers, colluding in the illusion that because the team have spent a little time up front imagining how the project will play out, that you can predict with a relatively high degree of accuracy how long a project will take and how much money it will require.

Sure, deep down, you knew that the plan was nothing more than a fantasy, a fictional story with a fairy tale ending, yet we all played the game, including the person you were selling it to! In truth, even though we knew it was all a lie, the plan still made us all feel warm and fuzzy inside.

At this point I'd expect every Agile evangelist to explode into action, chanting chapter 1 bullet 4 of the Agile Manifesto "responding to change over following plans" at their screens. However, I've been "doing Agile" exclusively since 2008 and my experiences do not go far towards solving the question posed by this article.

I remember being so excited when I first read about Agile, as a person who already had several battle scars, inflicted by my faulty crystal ball which had been used as a weapon against me during the Waterfall years, I felt sure that Agile was going to be my "knight in shining armour" here to save me from the scourge of the evil Prince2. But on reflection, some of the most dysfunctional teams, projects and management regimes I've experienced, have been under the banner of the new Agile world order!

So why does Agile still often fail? After all, it was specifically created by some the finest thought leaders in the world to address (amongst other things) the very notion that we can't possibly plan everything up front, yet I've spent a large chunk of my career going into businesses to assess and revive failing Agile practices.

My services are typically procured at the behest of an exasperated exec or senior leader who is not seeing the ROI they were promised, but always, it's the people in the trenches who are most confused and lost as to why this perfectly logical, common sense approach is making them fall out of love with their jobs.

Now, obviously, there are many reasons why Agile adoption can turn toxic, however, I believe one of the core reasons Agile fails is how people are hard wired to crave certainty.

I've sat in front of quite a few executive boards and explained the common sense logic behind Agile, watched the pennies drop, only for them to revert back to long term plans within weeks. I've seen delivery managers focus on output instead of outcomes, often turning story points into hours so they can track progress in a way that senior management can easily see in a Gannt chart. I've seen Burn Down charts being used to single out developers who appear to be under-performing and at the heart of it all is the human need for some degree of certainty.

It's not the processes, it's the way people interpret and abuse them that is the consistent ingredient in why Agile fails.

So what can we do about it?

Whenever it's a people, rather than a process issue, we first need to understand how we are programmed, what makes us tick. I'm no Socio-biologist but I may be a bit of a "Hippy" at heart so I believe the future of Agile needs to incorporate "mindfulness".

But what do I mean by that? A definition I quickly found on Google search describes mindfulness as: "a mental state achieved by focusing one's awareness on the present moment, while calmly acknowledging and accepting one's feelings, thoughts, and bodily sensations, used as a therapeutic technique".

C Level leaders at forward thinking organisations are already starting to incorporate mindfulness training into their skill set as a way to help them make the right calls under intense pressure but I believe it's a critical skill that all people should have.

Thus, in the example I've used, whereby business leaders clearly understand the common sense logic behind Agile principles but still struggle to disregard their need for certainty in the form of a plan, even though deep down, they know it's only giving them a false sense of security. A moment of mindfulness could help to rationalise those concerns and give them the clarity and courage to trust in the Agile process and accept the uncertainty that is innate in life.