What's Wrong With "Agile"

If you know me, you probably know I'm a huge advocate of the now decade-and-a-half old collection of IT software development trends that collectively get labeled as "Agile." I read Eli Goldratt's "The Goal" back in the 1990s, and dreamed of what it would be like to work in a technology role where the ideas presented in "The Goal" were taken seriously. When I first heard about the Agile Manifesto in the early 2000s, I immediately recognized it was something like Goldratt's ideas applied to a software development context. Some time around 2005 I was asked to help frame up one of the first agile practice teams at my employer, and I spent the next decade in various capacities (team member, Scrum practice coach, engineering practice coach, and so on). I love what we've collectively done to our profession under the agility banner.

As many have posted elsewhere, "Agile" is sort of a screwed up name since "Agile" isn't a thing, but "agility" is. For a few years now some of the luminaries of the movement have been saying as much, and I agree with them. When our industry began to productize agility practices as service offerings that people made money from, we slowly began walking away from the intent of the movement.

However, I really think there's something bigger that's wrong with these practices (or rather something is missing), and if you synthesize the work of some thought leaders, you can begin to get your hands around what's wrong. What's wrong is actually, in my opinion, bigger than the fact that some "hacks" have started claiming "agile" as their property and started selling coaching services or branding their own methodologies as "agile-derived." No, the problem is more foundational than that. Before we get into that though, let me introduce you to Dave Snowden and his work. If you're already familiar, you can skip the next paragraph.

Dave Snowden is the founder of a company called Cognitive Edge. Before founding Cognitive Edge, he worked for a company called Data Sciences Limited. Data Sciences was acquired by IBM in the early 90s, and for about a decade Dave worked for IBM Global Services. While at IBM he was Director of "The Cynefin Center for Organizational Complexity" where he and his team used complexity theory to devise some very useful tools and practices. One area of focus was the use of narratives ("stories") as an alternative way to gather data from people in order to determine sentiment for decision making, and for other purposes. Modern AI "sentiment analysis" tools like IBM's "Watson Tone Analyzer" are almost certainly inheritors of some of Dave's team's work. Another area of focus for Dave can be characterized as "understanding what approach to apply to solving a given problem," in other words strategic problem solving. It was in that pursuit that Dave and his team devised "Cynefin" which is a remarkably versatile approach to establishing strategy (and problem solving in general.) Dave calls it a "sense-making model," which is very apropos, because that's what it helps you do -- it helps you make sense of what kind of situation you are in (in real time), and gives you clues about how you should approach addressing the problem/situation. The way you use Cynefin in business contexts was discussed in a Harvard Business Review article a few years ago.

It's worth watching Dave's introduction video to Cynefin above, but briefly speaking, you can identify the situation you're in by determining whether you're in a Simple, Complicated, Complex, or Chaotic situation. Simple is the domain of "best practices." Complicated is the domain of "good practices." Complex is the domain of "emerging practices, hypothesis, and experimentation." Chaotic is the domain of "unknown practices" where action is more productive than any of the other approaches mentioned before.

How many times have you heard someone say the phrase "best practices?" I've heard it many many times in my career. However, how often has the speaker asserted why he or she believed that the application of best practices makes sense in the context of your discussion? If you're like me, it's almost never. To a limited extent, when the agility movement emerged, we sort of "cast off" things like "best practices." We started talking about engineering culture, empowering engineers, and honing our craft. In other words, we started talking about "good practices." (It's no coincidence that we had this collective confusion about "architects vs. engineers" when we did this. We were essentially confused by the fact that we were trading two different kinds of "experts," which is an attribute of the Complicated domain -- the use of experts for problem solving.) Here's the $50,000 question though. Are agility practices (engineering or otherwise) the right approach for all software endeavors? Should agility concepts be applied to our cash cows? If so, should they be applied in the same way as those things we're doing a lot of feature delivery on? Well, let's keep working our way around the Cynefin framework and ask ourselves some more questions.

The next domain after "Complicated," (which is where agility practices seem to have taken root as a replacement for waterfall) is "Complex." (As an aside, if you think about it waterfall was an attempt to cast software engineering into the "Simple" domain -- best practices like the highly ceremonial project management discipline were considered "mature"). In the Complex problem domain, the behaviors of the players determine what the best choice is, but it's not really possible to determine ahead of time which choice was even good. Where does that domain apply in business? If you said "maybe when you're establishing new markets, or trying to stay current on emergent trends" you'd be correct. So then that begs a question about applying agility practices (especially engineering practices, but also things like Scrum) to that domain. If you're working on a project where the primary organizing principle isn't feature development (at least not yet), but rather it's to explore whether a market exists for an idea, then many of the things we take for granted as "part of being a good software craftsman" suddenly become questionable. Does it make sense to build a CI/CD pipeline for an idea that you're not sure even has a market yet? This, and many questions like it, are pretty important, and we don't even ask ourselves about it -- we do some hand waving about "scaling agility as necessary" but then we reach for our metaphorical shotguns when someone suggests we might not want to do TDD in every situation.

The last domain is the "Chaotic" domain, and this is generally an area where you don't want your company spending a great deal of time. This is the domain where there are no defined emergent practices. This is the domain of disruption. In this situation, action is king. You need to be releasing your team to explore lots of options, because until you can get a foothold in something that seems to be working (the "Complex" domain), you're burning through money and you don't have a solution. This sounds dire, and often it is, but another thought leader, Simon Wardley, offers a glimpse of how you might actually take advantage of this domain to purposefully disrupt your competitors (or even downstream suppliers who are exerting more leverage on us than we want). People like Jeff Bezos and the late Steve Jobs have almost proven themselves to be masters of this. We'll have to save Wardley's material for a later discussion, but it's symbiotically related to Snowden's.

So, getting back to the point of this article, what's wrong with Agile? It's not so much that it's been "corrupted by infidels" which seems to have been the movement's diagnosis thus far. It's actually a lack of understanding of context that bedevils our beloved movement. I strongly suspect that all of our clutching for "scaling of agile" is actually about this lack of understanding. In other words, large enterprises more often have to work in all four of these domains (or at least in all three of them), and leaders in those enterprises have been guilty of two things. First, they haven't been able to articulate that need very well (which has enabled hand-waving and blind faith by agility advocates), and second, they have clung to the notion that everything can probably be forced into either the Simple and Complicated domains, and their biases tell them that Simple is better, so they keep asking for artifacts from that perspective (which causes agility advocates to bemoan their leaders' "lack of support from the top.") So, when our friends at ThoughtWorks for example assert things like "Why all this talk about scaling agile when teams aren't even doing basic stuff like source code management well?" they are potentially leading us astray by giving us a half-answer to a legitimate question. It might not be possible to take a "bottom up" approach as is often advocated, because agility practices might be the wrong solution for the problem domain a team is in. What's wrong with "Agile?" Well, nothing dire, other than we're possibly misapplying it, and to the extent that we're doing that, we might be sabotaging it.

In my next article I'll examine Simon Wardley's complimentary ideas. Simon addresses an important notion of "culture" (in addition to his ideas about utilizing disruption with intent) that is certainly related to Snowden's Cynefin concept. It's an important follow-on to the ideas expressed in this article.