The End of Agile? No, the End of Undisciplined Agile

Kurt Cagle recently wrote an article for Forbes, entitled The End of Agile. Although Forbes’ regular Steve Denning responded effectively a few days later with Why Agile’s Future is Bright, I’d like to chime in with several thoughts beyond what Steve has offered. Here they are:

1. Look beyond Scrum. Kurt started out with a tongue-in-cheek description of an agile team, but it was really a description of a small team at the very beginnings of learning Scrum. This is the classic “Scrum = Agile” mistake that many people make, and it plays well to people who only have a passing understanding of how agile works in practice. We recommend that you look at teams that are succeeding with agile, rather than teams that are clearly struggling. Better yet, let’s help the teams who are struggling — rather than making fun of them.
2. Beware of agile purists. Kurt’s observation that agile has become a religion wasn’t far off. It might be more accurate to say that there are Agile purists out there, people who often prove to only have experience with applying agile in straightforward situations — rather than in the enterprise situations that Kurt describes. But there are also pragmatists out there in the Agile space (pragmatism is one of the seven Disciplined Agile principles) who are actively addressing how to apply agile and lean strategies in less-than-ideal situations. We have always recommended that you be open-minded and pragmatic, to do the best you can in the situation that you face and always strive to learn and improve.
3. Agile works in a very wide range of situations.There is ample research data and case studies showing that agile works in a wide range of situations. For example, in our 2016 Agility at Scale survey we found that organizations were in fact applying agile on large teams, in complex domains, using legacy technologies (including legacy data sources), in regulatory environments, and multi-organization situations including outsourcing. And we have data from other studies going back over a decade that show that Agile strategies have been applied beyond the “small teams with straightforward problems” scenario for quite a while. We recommend that you look around and see how agile is being applied in practice.
4. Scaling agile requires discipline. The article also states that agile doesn’t scale well to address big problems, yet many organizations are doing just that. To be fair to Kurt, this is a common misunderstanding that is a result of the Scrum = Agile mindset because Scrum purposefully doesn’t address architecture (amongst many topics). But other agile methods do — in particular Agile Modeling and Jim Coplien’s excellent work around Lean Architecture — strategies which are explicitly part of Disciplined Agile. In Disciplined Agile Delivery (DAD), we explicitly include initial architecture modeling early in a project via the Identify Architecture Strategy process goal, we show how to reduce architectural risk via the Prove Architecture Early process goal, and how to evolve architecture throughout Construction via the Produce a Potentially Consumable Solution process goal. DAD teams also have someone in an explicit Architecture Owner role who is responsible for guiding the team through architecture decisions. Furthermore, there is an explicit Enterprise Architecture process blade to support organization-level architecture issues. Finally, DA the idea that Architecture Owners and Enterprise Architects should work closely together. Our recommendation is to explicitly address architecture in your way of working, and this is something that Disciplined Agile provides clear, proven options for.
5. Data projects can be difficult. The same can be said of non-data projects too. When either domain complexity or technical complexity rise, or both, you need to become more disciplined in your approach to requirements and architecture modeling. As Cagle implies, data projects often suffer from domain complexity, this is particularly true for the enterprise data systems he mentions. Similarly data projects tend to suffer from technical complexity in that they need to integrate data from many disparate sources that often use different technologies, apply different semantics, and are accessed via different strategies. Once again, the context-sensitive choice-driven strategy promoted by Disciplined Agile wins the day over the prescriptive methods and frameworks Cagle appears to be familiar with. To confuse matters further, prescriptive frameworks such as Scrum and SAFe have virtually nothing to say about addressing data issues. Disciplined Agile, on the other hand, encompasses proven strategies from the Agile Data method which are critical to success on data projects. Our recommendation is to embrace the complexity that you face and to recognize that you will need to explicitly tailor your way of working (WoW) to address that complexity.
6. Agile is applicable to data projects…and may be the superior approach. I’m going to be blunt here – Kurt Cagle was completely and utterly wrong concerning the application of agile strategies to data projects. While traditional data professionals often prefer to do extensive data modeling up front, this strategy has been shown to be ineffective in practice. What does work well is an agile data modeling strategy where high-level modeling is performed early in a project and detailed data modeling performed as needed during construction. You can be very agile when supported by agile database practices such as database refactoring, automated database regression testing, and continuous database integration. And his claim that the focus on enterprise data is relatively new clearly doesn’t reflect the wealth of techniques around enterprise data modeling, master data management, and meta data management that the data community has been following since at least the mid-1990s. And yes, there are agile approaches to all of those practices. As an aside, I started writing this blog in the evenings while attending the World Wide Data Vault Conference in Hannover. Most of the presenters have discussed how they’re taking, or at least supporting agile — and often a Disciplined Agile — approaches on their data projects.
7. Many other claims the article makes are observably false. Contrary to what Kurt claims, there are many large and complex open source projects, and they clearly benefit from agile and lean strategies. Examples of such projects include Linux (operating system), Hadoop (big data processing), MongoDB (database), Numenta (machine learning), Angular (web application framework), and Docker (software infrastructure) to name but a few. But hey, it’s not as if any of those technologies have become mission critical for your organization. And his discussion of games development? Having actually consulted to several commercial game companies, I can safely say that they were all working in a very agile manner.
8. You must adapt your approach to address the context that you face. One process size does not fit all. Two of DA’s principles are Context Counts and Choice is Good – different teams in different situations will choose to work in different manners. If you want your teams to be effective then you have to allow them — and better yet, enable them — to choose their own way of working (WoW).
9. Adapting your approach requires skill and experience. In some ways, Kurt is right – enterprise-class problems require an enterprise-class agile approach. But he was wrong in assuming that because people were struggling to apply undisciplined agile strategies to these situations that agile didn’t work; the real problem was that they didn’t know how to make it work. The fact is that others have figured these things out, and we’ve captured a lot of these strategies in PMI’s Disciplined Agile (DA) toolkit.

Let’s hope this is the end of undisciplined agile. But as Steve Denning points out, it certainly isn’t the end of agile. I suggest this could be an important beginning for Disciplined Agile approaches.