What are the Root Causes of Agile Disappointments?

There are 6.1 million companies with employees in the US. That implies 6.1 million agile journeys–from not yet started, to fully mature agile companies (whatever that means). Some have begun the journey and abandoned it. Some have been journeying for years. Some have been wildly successful, others mildly successful, and still others have little to show for their investment.? (See Steve Denning reference)

Anyone, pro or con in their view of agile can make a case from 6.1 million (or many more worldwide) data points. In this post, I’d like to point out two things: a preferable alternative to success or failure, and exploring root causes rather than relying on divisive rhetoric. Rather than success or failure, I prefer accomplishments and disappointments. To me, the only failure is a failure to learn from a disappointment.??

For this article, I’ll focus on agile disappointments, but skip over the rhetoric such as: agile is dead, scrum poisoned agile, scaled agile isn’t agile, frameworks don’t work, the Agile Industrial Complex strangles innovation, etc., etc. While some of these statements have a kernel of truth, they are by and large, unhelpful. While I value constructive debates, I particularly dislike discourse that attacks another’s integrity, intent, or competence.? So, I offer six candidates for the root cause of disappointments with agile that hopefully embody this constructive debate sentiment:

• Capabilities are spread too thin?
• Performance-People imbalance
• Doer-Enabler imbalance
• Hierarchical Organization Design
• Confusion about Methods, Methodology, and Mindset
• Fixed Mindsets?

Capabilities are spread too thin

I like to invoke Jerry Weinberg’s Law Of Raspberry Jam, "The wider you spread it, the thinner it gets.” Think of the degree of capability as a huge pancake with hordes of elves with jars of jam racing around spreading jam. Towards the edges, it thins out. Consider your agile jam – with people as the spreaders. The further out on the edge you get, the greater the incidence of beginners teaching beginners.?

This thin jam problem runs headlong into what constitutes capability. Capability is equal to knowledge, plus experience, plus decision-making. Thin jam training courses teach knowledge, sending people back to offices where it is difficult to get the right experience to further one’s capability improvement.?

Let’s worry more about helping people and organizations where they are on their journey rather than demeaning them because they might not be as far along the way as we think we are.

Performance-People Imbalance

Organizations that create the optimum balance between Performance (Customer Value, Financial) and People (healthy work environment) have the best results according to a February 2023 McKinsey article: “P+P Winners were roughly 1.5 times more likely to remain in the top tier year after year, and they had about half the earnings volatility.” The core of the Agile Manifesto and the thinking of its authors reflect a similar idea – Deliver valuable software and Create healthy work environments. Have we gone far enough towards this goal? Absolutely not. Are work environments generally healthier than they were 23 years ago? Absolutely yes.?

The work environment prior to 2000 was heavily steeped in traditional command-control practices that led to unhealthy work environments. There was tremendous pressure to build new internet apps, but Y2K work had strained resources just when organizations needed them – Ed Yourdon’s “Death March” book detailed the harrowing times on many projects. The Agile Manifesto provided a beacon of light towards a healthier path. But we have overshot the balance point. Senior executives I’ve spoken with are concerned about this imbalance. When the agile literature is full of management words like empowerment, self-organizing, empathy, servant leadership, and more, they are concerned that there isn’t an equal commitment to accountability and financial success.??

Doer-Enabler Imbalance

Colleague Phil Abernathy told me the story about a consulting gig where “They had about 100 developers.” He asked me to guess how many project managers and program managers there were. Phil responded, “Ninety-eight, right, 98! And I was like, You have got to be kidding me. They have to coordinate everything. They had functional silos of business analysts on this side, devs on that side, and testers on this side. And people were working on seven projects at the same time. And of course, they needed 98 project managers to coordinate everything. We got rid of that whole group. We reorganized the teams. They reduced 30% of their staff without firing a single person with contract reductions and retirements.”?

This company had a ratio of about 50% doers and 50% enablers whereas an 85% to 15% was considered an appropriate balance. Phil calls this ratio a “Bureaucracy Mass Index (BMI).” (See Phil’s video reference)

Other companies who have been dragged into the press recently as “getting rid of agile,” have actually been doing no such thing – they have been rebalancing. That is, reducing the number of Enablers--project managers, agile coaches, and scrum masters--while increasing the number of Doers--technical talent.

In today’s high-tech, AI+ environment “doers” are technologists – software developers, data scientists, etc. But it’s not enough for the front-line doers to have technical talent, the enablers – from scrum masters to CIOs must be tech-savvy. Look into the engineering talent in high-tech companies like Google and Microsoft – the front-line, mid-level, and executives including their CEOs are tech savvy.

Hierarchical Organization Design

While overall organizational design is beyond the scope of this article, progressive organizations are trying to reduce the “friction” of multiple organizational levels that hinder change. Communication between levels includes Alignment, Accountability, and Autonomy. For example, if every manager or executive wants some level of autonomy and there are 12 layers of management, how do you slice it? This is how you get first-line supervisors approving $5 expenditures, the next level can approve up to a whopping $15, and on up the chain.?

Bayer Corporation executives, managers, and staff are implementing what they call Dynamic Shared Ownership (DSO). One of the changes they made was eliminating management approval of 140,000 annual expense reports. Another was reducing management layers and implementing cross-functional customer and product teams. In an industry where the manager-to-subordinate span of control hovers around 3-5, Bayer is achieving a span of 10-15 and relabeling it “span of consulting.” Bayer reduced management by 40% as many of the individuals moved to their cross-functional teams. Bayer’s results to date have been spectacular. (See references include a thought-provoking video of DSO.)

This is where agile gets hard. Making these kinds of organizational changes, whether in IT or more widely in your organization takes courage and persistence. However, to reach higher levels of agility, organizational changes will be required, including mindset, management structures and operating models.

Confusion about Methods, Methodology, and Mindset

Have you encountered fuzziness around the words “method” and “methodology?” To be candid, I have probably contributed to the fuzziness a time or three. In scientific research, one’s methodology—the strategy, and the steps to collect data—the methods, are often defined early in the process in research work.?The definitions of these terms are edited versions from my “Wild West to Agile” book (see reference).

?A software method defines the detailed steps to deliver artifacts identified by your chosen methodology. Refactoring is a technical method for improving the quality of code. Daily stand-ups are a collaborative method to keep teams in sync. Product planning sessions are management methods to plan how to move forward – for a day, a week, or a month.

A software methodology defines your strategy. It divides software work into activities or steps, each of which may include the use of specific methods, the definition of specific deliverables (a requirements document), and more. Software development life cycles (SDLC) define the highest-level sequence of processes, with names like waterfall, spiral, or iterative.??

A mindset is an attitude, a set of beliefs we use in making sense of the world. Our mindsets cause us to think about issues from a certain perspective, to have feelings about events, and to act accordingly. For example, In software development, a person with a cautious mindset will interpret an event differently from a person with an adventurous mindset.

A methodology defines your strategy while a mindset identifies your attitude. Your attitude influences how you implement a methodology or method. Long ago, at a conference, the organizers handed out a feedback form to attendees. Getting feedback on your product or service is a good thing, isn’t it? In this case, the feedback form was 20 pages!! The sponsoring organization was noted for its monumental approach to software development, so the lengthy form was in keeping with their mindset.?

Some might say, “This methodology stinks!” Another way of thinking about it is, “Their methodology conforms to their mindset, but it does not conform to my mindset.” Methodologies or Frameworks are not inherently good or bad, their implementation is a function of mindset and appropriateness for the job at hand within a particular organization.

Fixed Mindsets

According to author Carol S. Dweck, ”Mindsets frame the running account that’s taking place in people’s heads. They guide the whole interpretation process.” Dweck identifies two mindset categories: Fixed and Growth. As the names suggest, fixed-mindset individuals are slow to change, mostly satisfied with the status quo, while growth-mindset individuals are open to change. Obviously, there are degrees of each mindset. Dweck indicates that the population as a whole is about 60% fixed and 40% growth.

The issue with agile journeys arises because “being agile” not only prospers in a growth mindset environment, it introduces enough organizational change to require a large number of growth mindset people. That’s why when agile is introduced to high-tech Silicon Valley individuals, the response is often, “Okay, what’s new? We’ve been doing it that way for some time, you have just put a name to it.” Introduce the same ideas to a traditional conservative bank and the response is often, “That’s too radical, it will never work here.” And, guess what – it doesn’t.?

As companies grow over time the tendency is to become and to attract more fixed-mindset employees. Maybe the 60-40 mix becomes 80-20. How difficult will it be to introduce agile to an organizational culture with an 80-20 fixed-to-growth mindset??

Dweck says, and I agree, that there is nothing inherently good or bad about a growth or fixed mindset. The rub comes when a job or an organization is tackling a fast-moving market and the need for a growth culture exerts itself.

Mindset operates at the method and methodology levels. For example, those with a fixed mindset tend to implement what I’ll call “prescriptive agility.” I know that is an oxymoron, but it happens. Companies implement a methodology for example, but it becomes prescriptive – ”Follow these 8 steps and don’t deviate.” Therein, they eliminate the most critical aspect of agility – the ability to adapt over time. Going back to the issue of capability improvement, they short-circuit adapting and thereby relegate themselves to a permanent beginner state, they never adopt adaptive agility.

If you don’t develop an agile mindset for continuous capability improvement, you will never rise above a beginner level and will never fulfill the outcomes envisioned for your agile journey.?

Conclusions?

In wrapping up this nuanced landscape of agile disappointments, let's remember that the journey toward agility is as diverse as the individuals and organizations embarking on it. The challenges range from spreading capabilities too thin to navigating the tricky balance between performance and people. Yet, amidst these trials, the underlying message is clear, learning from our setbacks, rather than labeling them as failures. Agile is not a one-size-fits-all solution but a mindset that encourages adapting and evolving. It invites us to embrace change, question our assumptions, and continuously seek improvement. So, as we reflect on our agile experiences, let's hold onto the notion that every disappointment carries the seeds of knowledge and growth. After all, in the grand tapestry of our agile journeys, each thread of challenge is interwoven with opportunities for learning, innovation, and, ultimately, transformation.

References:

Discussion of the BMI see Phil Abernathy’s video.

Steve Denning:? https://www.dhirubhai.net/feed/update/urn:li:activity:7169138342700249088/?

Bayer’s DSO:? https://www.youtube.com/watch?app=desktop&v=_mubMfgrdIQ?

Reimagining Agile: Back to Basics, Forward to the Future: https://www.dhirubhai.net/groups/14366064/??

Wild West to Agile book: https://www.amazon.com/Wild-West-Agile-Adventures-Development/dp/0137961006?ref_=ast_author_dp?