Is Agile always the right answer?

TLDR - Agile is great, and not just for software development. Have a nice day!

Whilst enjoying my post-run coffee and browsing LinkedIn this morning, I came across an older article which proposed that Agile isn't always the right answer and that, particularly for large projects, Waterfall might be the right approach (I contacted the author to see if they'd be happy for me to namecheck them - it really was a very good article - but they'd rather I didn't). It was an interesting article and it struck a chord with me because I used to be of the same opinion.

Ten years ago, when Agile was first put to me as the method by which all software development should be done, I scoffed at it. Agile was Fragile (chortle chortle), it was uncontrolled and light on process. Sure, it was fine for a spot of airy-fairy web development - but for real software developers, those big-beasts who laugh in the face of anything higher-level than C, it was a poor tool and not up to the job. Yes, I'm afraid to say that I really did think like that, I really was that tribal.

Part of the problem, of course, was that I hadn't been formally taught Agile - it was nothing more than a buzzword. My early attempts at 'doing Agile' were little more than shoehorning Waterfall methodologies into Agile terminologies. Not a happy marriage, and unsurprisingly doomed to failure.

A lot can happen in ten years. For one, I've come to respect what can be done with web-development (it's all about choosing the right tool for the job). I'm older now, and a little wiser. So what do I think about Agile? (It goes without saying that this is all my opinion - yours may differ and I'm not saying that you're wrong, I'd really like to hear your thoughts).

Let's look at the complaints levelled against Agile (with thanks to the original author)

You have a deadline to hit, but that's at odds with the Agile ethos that software is delivered when it's ready.

I'd respond that, with the traditional 'Waterfall' approach to project management, the time quoted for the delivery of a project is always a finger-in-the-air best guess (1) as to the time required for the project, plus extra for contingency. Provided that no disasters occur, the project will run according to the schedule - and, if you're lucky, you might even get it a little early. If a disaster occurs, and bear in mind that the disaster might be building slowly rather than a sudden catastrophe, then the project will be late - and may not even be delivered at all. History is littered with such examples, one recent one being the NHS Plan for IT, and poor project management is inevitably blamed. I'd say that the overarching problem is one of poor methodology - Waterfall.

With Agile, the backlog dictates the length of time for the project, and the duration of each story (item to be delivered) is arrived at in consultation with the developer (2). This is as it should be - after all, who knows better how long something will take to be delivered than the person who'll be doing the work? If a problem occurs in the development of a story, a delay, then it will be re-scoped (perhaps even split into multiple stories if it was incorrectly scoped to start with) and returned to the backlog (3). Oversight of this will likely be daily, in a short daily meeting (4), so that problems don't have the opportunity to turn into disasters. The overall delivery time for an Agile project will always be greater than the imaginary most optimistic estimate of Waterfall, but I've never known it to be greater than Waterfall's realistic estimate plus contingency - and, because it's been managed properly, it's less likely to fail altogether too.

As for delivering when it's ready, I don't want software (or hardware) that's insufficiently tested. It could be argued that had Comet been designed according to best Agile principles many lives could have been spared, and had Chernobyl been designed according to Agile principles maybe large parts of Ukraine would still be habitable. Perhaps I'm old fashioned - but I like to think that the products I use are correctly designed, and correctly tested - rather than just pushed out according to an arbitrary timeline to suit a project managers spreadsheet.

Agile is not well suited to large development projects.

An Agile sprint does not have to deliver full and finished products - what it has to deliver is full and finished discrete units (stories). Imagine building a bridge. You can break that down into foundation, piers, abutments, girders and deck. Can each be delivered in a sprint? It's unlikely - but each can be further broken down; Foundation might be broken into excavation, reinforcement, pouring concrete and so forth. Keep going, keep collaboratively breaking the work up, and eventually you'll have all the stories you need to complete the project in the backlog - and a better understanding of the time and cost required to deliver than you might have done if you'd stuck with Waterfall. From websites to operating systems, tricycles to supertankers, parish council projects to government (5), if the project can be managed then it can be managed with Agile.

But wait a moment - surely Agile doesn't work when there's a fixed timeline, like a budget year end?

By now it's not too much of a surprise for you to hear that I dispute this too. Just because an item is in the backlog doesn't necessarily mean that it has to be delivered. Every project I've ever seen has items in it which are desirable - perhaps even very desirable - but not absolutely essential. Prioritisation is a key part of Agile sprint planning, and the lowest priority items in the backlog may forever remain there. By understanding completely the work which needs to be delivered (collaboratively, remember), the project manager will have a clear understanding of the work that needs to be done, the time required to do it, and whether or not to request additional resource in order to ensure that the essential work gets completed in the time available. Waterfall might deliver on time - but what it delivers may very well be a fudge.

Agile is undocumented.

Some Agile projects might be undocumented, but I'd argue that that might be bad Agile. There's nothing in Agile that says you shouldn't have UML requirements documents, for example, or that stories have to be understandable by the layman. If you need a document to describe the requirement though, the writing of that document may itself be a sprint or sprints. What there shouldn't be is unnecessary documentation. If the Requirements Spec and the Functional Spec, for example, can be combined then they should be (this isn't mandated in Agile either - this is my own opinion on good practice).

Government Projects, and particularly projects like the Affordable Care Act, tend to be deeply partisan and therefore they make bad examples in a discussion like this. If you don't like the Affordable Care Act (6) then you're unlikely to be in favour of any part of it, including its management. If you do like it then you'll probably dismiss the problems that it experiences as foibles. For what it's worth (very little), from what I've read about the Affordable Care Act, the management of it was hamstrung by a hard deadline and insufficient contingency (which would have stymied Waterfall too), insufficient resourcing, and a lack of rigour in deciding which items from the backlog were inessential and could be dropped from the day 1 release.

It's worth remembering also that Agile is a flexible foundation - but it is a foundation. Without the foundation you can't build all kinds of useful extra management tools on top - tools like SAFe or DevOps. They aren't rocket science - but they are enablers of rocket science, and much else besides.

I'll leave you with this final thought - any project can be broken down into discrete units, that's the job of a good project manager. If it can't be broken down into units that fit into a sprint then that's not a problem with Agile, it may be a problem with the PM.

(1) Maybe an educated guess, but a guess nonetheless - you can't know how long it'll take until the work is complete.

(2) Collaboration is key. The project manager cannot say how long each task will take to complete, and neither can senior management. The only people who have the domain expertise to say how long a story will require to develop is the people who'll actually be implementing it. The developers. Similarly, it's unlikely that the developers have the domain expertise to prioritise each story; they may do - but it's more likely to be a call that the stakeholders need to thrash out. Nor can the developers make decisions on funding - you get the gist. In good Agile planning, everyone has a role to play.

(3) I've tried not to use too many Agile terms but a Story is an item to be delivered, a Sprint is a short period of time in which the story or stories will be delivered (typically a week or two), and the Backlog is the list of outstanding work. It is unlikely that an entire project will be delivered in a single sprint - it is likely to be made up of many stories and many sprints. This isn't supposed to be a discussion on how to do Agile well however - if that interests you then I'll leave that as the homework!

(4) Which might be a Scrum meeting, if that's the way you do things. Agile is nothing if not flexible (there are other Agile methodologies that you might use other than Scrum).

(5) Collaborative government? Now there's an idea! Who knows what they might achieve?

(6) Yes, yes, I know I'm a hypocrite because I brought up the problems with the NHS Plan for IT only a few short paragraphs ago (I'll merely observe that the vast majority of people in the UK are in favour of the NHS - it's a less divisive topic).