Why is Agile STILL not working for your project!

In a previous post over 3 years ago https://www.dhirubhai.net/pulse/why-agile-working-your-project-harmesh-mehrotra/, I had argued that Agile wasn't delivering on it's promises of faster ROI, improved visibility, lower costs, adaptability & decreased risk. The reasons for those were that it promised too much too soon, MVP (Minimum Viable Product) was misunderstood, people didn't learn from past failures and that jargon became more important than delivery. During these 3 years, I've had a chance to lead a few more data-centric engagements and thus revisit how Agile was contributing (or not) to their successful outcome. As you may have guessed from the title of this post, the answer is it didn't. This post is an attempt to reason why Agile isn't the ideal methodology for enterprise scale data-centric projects.

The purpose of data-centric projects (for e.g. setting up a data lake, delivering analytics use cases, brilliant visualization etc.) is to effectively provide actionable insights to business users, period. To deliver such insights, there's a whole data organization engine working in the background that's constantly performing a lot of activities - automated data ingestion, stream processing, edge analytics, active data curation, modeling, ad-hoc querying, automated refreshes and publishing etc. etc.

Firstly, several of these activities are not client facing but need completeness with accuracy (hint: ETL/ELT), hence the concept of MVP doesn't quite apply; thereby hampering Agile.

Secondly, modeling, be it persistent or evolving is of paramount importance, and in an enterprise context is generally complex. To time-box such an activity with any reasonable accuracy would be a difficult exercise, thus likely to be eschewed by architects and developers alike. Thus estimation of story points goes for a toss; thereby hampering Agile.

Thirdly, self-service is increasingly becoming the name of the game. Business users and citizen data scientists are more equipped than ever to conduct their own free flowing data analysis. They want to discover the expected and uncover the unexpected. Such a boundary-less approach, doesn't lend itself well to Agile.

To emphasize, data modeling and curation are key facets of a data-centric project, and they are fundamentally different to areas where Agile has established itself quite strongly: application development, particularly those with a heavy UI/UX aspect. I'd stick my neck out to say that even those areas are being transformed through use of thin shells (HTML+CSS) with limited need for customization, thus reducing the relevance of Agile.

To summarize, Agile has had it's moments in the Sun, and is here to stay, but there's a good chance that it's still not working for your project.