Agile Is Not Enough

Alan Kay once compared the “Web” to a pop culture:

“Pop culture lives in the present…People who live in the present often wind up exploiting the present to an extent that it starts removing the possibility of having a future.” His point was that programmers jump to the popular tool and do whatever everyone else is doing, instead of making reasoned choices. [ref: Beyond The Box, February 1, 2007, By Allan E. Alter]

Agile rapidly became a pop culture, and for this reason, Agile software development — as practiced today — is horribly broken.

Agile software development — as practiced today — is horribly broken.

Let me be clear: I am a great fan of Agile. My own company — the one that I co-founded in 1995 — adopted eXtreme Programming (XP) back in 2000, and was one of the very few Agile oriented consulting firms in the Washington DC region. I have completely embraced Agile since then; but I also have seen that the Agile community — and particularly the Scrum community — so focused on certifications and promulgating their Scrum Guide — have dug themselves into a set of anachronistic practices, many of which have been embraced by the community as “the way to do Agile”. The community became inflexible on the so-called “Agile practices”, and as a result (1) DevOps was eventually born to escape from that stranglehold, and (2) the Agile community is far less effective than it could be. For example, the Scrum community—overseen by the Scrum Alliance—is notoriously unwilling to consider the ideas promoted by the Scaled Agile Framework (SAFe). Since Scrum is the most widely used Agile methodology, this has a stifling impact on Agile as a whole.

Because of the community’s dogmatic inflexibility, Agile is in danger of becoming bypassed entirely by a better model, like DevOps, or the next thing. Indeed, Andrew Ng, a well known pioneer of deep learning, asserts that, “adding Deep Learning to an internet company does not make an AI company”, and that “AI Era” companies will focus on building enterprise-wide platforms and unified data solutions, and will organize data and intelligence functions horizontally. If true, this means that today’s Agile practices — which emphasize a vertical-first focus — might not be well suited to AI, and that a new model is needed — not the old pre-Agile waterfall model, but something different altogether. The Agile community — locked into the Scrum model — might be too rigid to adapt sufficiently and so it seems like its replacement is inevitable. Ironically, the Agile community is not Agile.

Ironically, the Agile community is not Agile.

I am not alone in thinking that the Agile community — really mostly the Scrum community — has been too rigid and dogmatic. Dave Thomas, one of the authors of the Agile Manifesto, has given many talks about this rigidity. Here is one: https://www.youtube.com/watch?v=a-BOSpxYJ9M. Ron Jeffries, another of the Agile Manifesto’s authors and one of the inventors of eXtreme Programming, recently wrote that “developers should abandon Agile”.

That Agile is broken flies in the face of the hype, that Agile is the answer for the abysmal history of software projects that end up way beyond schedule and hopelessly over budget. Yet, Agile does not always help: the procurement for the healthcare.gov website specified that Agile should be used, and the contractors claimed — and thought — that they were complying, yet the launch was a cataclysmic failure.

Agilists— including me — will be quick to point out that in cases like healthcare.gov, they were probably “doing it wrong”. (I know for a fact that they were doing it wrong.) But it turns out that if you merely inject Scrum into a large organization, it does not work well, because in a large organization, teams are not self-contained— they can’t be. They depend on the rest of the organization, and Scrum does not account for that. But when pressed on how it should work, the Scrum community’s response has mostly been, “The organization should be Agile, not do Agile”— a singularly unactionable and unhelpful response. It turns out, they don’t really know the answer.

And that is why DevOps arose. DevOps is not a methodology: it is a collection of approaches that evolved at companies like Google and Netflix, that enable those companies to deploy new features frequently, rapidly, and at massive scale, while simultaneously managing risk. The practices came first; the term “DevOps” and the philosophy behind it came later, as a kind of “refactoring” of the approaches.

So it is not that Agile does not work: it is that Agile does not work by itself — it is not enough. DevOps provides critical elements that help Agile to work in a large organization. Given what DevOps provides, there are still big missing pieces, for example the kinds of leadership that are needed: Scrum’s “Scrum Master” role is ill-defined and insufficient for a team in a large organization.

In a later post I will examine what Agile really is, and how so many popular Agile practices do not work well in large organizations and why, and what to do instead.

(Second post in this series is here.)