Agile and Scrum Transformations Will Always Fail - Organizational Transformations May Succeed.

I wrote my article about the Product Owner challenge sometime back and its had me thinking continually about how to solve this problem. It has also had me thinking about the challenge of being agile at scale and I’m starting to see a pattern. Failing Scrum is caused by ineffective Product Ownership. Ineffective Product Ownership is caused by lack of organizational change. Yes, there are other points of cause, but I warrant that the root causes will all lead to the same cause. Lack of organizational change. Further analysis will probably lead to leadership as the root cause. I might have a real basis for saying this.

The key to solving failing transformations is to address effective leadership, understand complexity, and learn the science of teams. As an organization scales it needs to transform as a whole if it is to succeed.

By solving the role of the Product Owner at scale, you will solve the key reason transformations fail.

The Product Owner

The role of the Product Owner is predominately to prioritize, which in my own experience is the biggest challenge. 

The Scrum Guide states:

• The Product Owner is one person, not a committee. The Product Owner may represent the desires of a committee in the Product Backlog, but those wanting to change a Product Backlog item’s priority must address the Product Owner.
• For the Product Owner to succeed, the entire organization must respect his or her decisions. The Product Owner’s decisions are visible in the content and ordering of the Product Backlog. No one can force the Development Team to work from a different set of requirements.

Deciding on what’s important is rarely a single person role, and the more complex the organization the harder that is to do. The ‘real’ people making those decisions are more often than not very much detached from the actual work that is being done. Once you have a team working on many different products, as well as supporting day to day operations, this becomes almost cripplingly difficult for the PO.

I’ve spoken at a number of events and also written some articles where I have used the phrase ‘there is no such thing as an Agile or Scrum transformation, only an organizational transformation’. I’ve also said ‘agility is an outcome and not a goal’. If you focus on 'doing' agile you'll miss the whole point of delivering value to your customer more effectively, and if you fail to become agile as an organization you'll fail to address the reasons you are not agile to begin with.

Toyota is a Customer 1st focussed company. Since 1946 this has been our focus. Agility is about ensuring you are delivering the right product at the right time for the right customer. By focussing on the customer first you will prioritize the right work your teams should be doing.

I have been using this slide for a while to to describe my thoughts. In essence, Systems Thinking or Lean (Toyota Production System) is focussed on removing constraints, building in quality, and reducing all non value added activities. Agile is focussed on maximizing the amount of work 'NOT' done to achieve the best product or service desired by a customer. Scrum is an empirical planning process that instills a team behavior in the people doing the work, and ensures we make evidence based decisions.

Agility at Scale

My boss remarked to me a few months ago that Scrum always seems to work at small scale but struggles at large scale. That led me to talk to people I respect as knowing more than me, including one of the creators, and some of the influencers. Here’s what I learned.

Scrum as designed only works for a single product and a single team. That is if you follow the Scrum Guide and 101 teaching it is designed to work this way. Sure, many folks are using it beyond this pattern, but it is inherently designed to work at this simple level. That’s why many startups and small companies succeed with it.

Scaling approaches such as Nexus (SPS) are designed to work for a single product with up to 9 teams. LeSS also uses a similar approach for 8 teams, and both are extremely good in this configuration. One product with one to many teams. It’s obvious really when you focus on just one thing at a time it’s much easier and faster to deliver great results. Our own experience at Toyota Connected supports this. If you notice in the diagram above it does not say 'Products <--- Backlog', it says singular 'Product' Backlog.

A Product Owner is meant to own a single product, and a team is meant to work on a single product. In Scaled Professional Scrum (SPS) the addition of roles and events is meant to aid the additional coordination needed when more than one team works on a single product. It will not solve multiple teams working on multiple products in some matrix managed nightmare.

As the number of teams grow keeping them small and enabling cross team communication helps you maintain that agility and speed. However, once you throw into that mix multiple products and multiple interdependencies between teams the complexity increases and as a consequence the speed and agility decreases. Trying to force fit Scrum into this world will only result in failure, and the inevitable 'Scrum Doesn't Work' rhetoric. A better phrase might be 'Scrum only works if you use it as designed.'

Now map that to an organization’s growth and as that growth increases the communication becomes more complex as the speed and agility decreases. We start adding roles and hierarchies and management, and our ability to move faster and make decisions more rapidly declines. Toyota recently recognized this and as a result Akio Toyoda (President of Toyota) announced the collapsing of several layers of leadership to enable more rapid decision making.

Organizational Scale Reduces Efficiency

As reported on Toyota's Global Newsroom Akio Toyoda is quoted as saying: "The revisions to our organizational structure are designed, by reducing the number of structural layers, to allow rebirth into a Toyota that is able to reach conclusions more swiftly, make prompt decisions, and take immediate action faster than ever."

Conway's Law explained this in 1967.

Some so named scaling frameworks attempt to address this by adding more roles and more coordination, but this only leads to reinforcing the level of abstraction between those who want something, and those who build something, and increases complexity and inefficiencies. Another impact of scaling in this way is ineffective communication. As more people are added into the decision making mix the ability to make a decision decreases.

The Scrum Guide suggests a range of 3 to 9 members of the development team (the team actually executing work) would be about right.

A framework should by definition work in any and all contexts. Alas there are no scaling frameworks, only tools and methods to address certain needs. A better term might be a scaling approach. Where one approach, tool, or method may work in one context it might fail in another.

According to Merriam-Webster the word 'framework' was first defined in 1578 as 'a basic conceptional structure'. Therefore to create a methodology, which is a series of processes or methods, and call it a framework is incorrect.

The Need to Descale

The issue of scale is one that constantly besets organizations seeking to become agile. We must start by descaling the organization.

Professor Robin Dunbar, a fellow Brit studied primates and noted as they expanded in number they naturally divided into new troops with new leadership. In human studies he determined that humans are only capable of sustaining 150 stable relationships, after which communication and other social behaviors become difficult or impossible.

Bill Gore of Gore-Tex fame knew this so structured his company of around 9000 title free associates into self contained operational units of 150 or less people. One of the reasons the approach works so well is that everyone in a given plant knows everyone else, knows what everyone's role is, and perhaps most importantly, knows what they can expect of each other and where they can find expertise. In other words, by limiting its facilities to 150 employees, Gore has solved the issues around collaboration and communication that have plagued so many other large companies. 

This is the concept of Value Streams.

The idea is that you have a self contained group of people and resources that can deliver value from concept to cash without outside support or dependance, and yes that includes legal and compliance. To ensure we maintain the rapid decision making and engaged leadership we limit this to the Law of 150. This is how most small startups function, and how Toyota Connected was originally designed. Other examples are Toyota Research Institute who are working on our autonomous future.

So the first thing we need to do is reduce the size of organizational units. Re-focus people on a smaller group or products and services, and prioritize on Customer 1st value. This will lead to an increase in FLOW, and a reduction in waste.

This requires Organizational Transformation, and there are no 'scaling frameworks' that address organizational change. For this to happen you need engaged and participatory leadership who are working in FLOW daily. Yes you need a hierarchy to manage the structural parts of the company so separate that from the product delivery organization. If the product delivery organization has to serve the needs of the hierarchy then those needs have to be prioritized with those of the customer focussed product and service delivery.

Scaling the Product Owner

If you have one team and one product then use Scrum 101 and a single Product Owner. It works.

If you have one product but multiple teams use one PO until it becomes apparent that the capacity of that single PO is exceeded. Now we need to divide the teams into two groups and have one PO per group, and a Scrum of Scrums between the groups to coordinate delivery and integration etc. Approaches such as Nexus or LeSS are applicable patterns here. Both Nexus (SPS) and LeSS offer ideas to go beyond 8 or 9 teams, but remember this is one product only.

If you have one team servicing multiple products and/or services then your Product Backlog becomes more of a Team backlog prioritized by one product owner. The product owner now is really a team backlog owner. 

You could argue that each product should have its own product owner and the team resolves prioritization issues, but good luck with a non empowered team trying to solve that with a group of senior stakeholders all demanding their slice of the pie. You need someone to focus on the value of the work the teams does, and working with other people in the organization all vying for the same resources. That someone is the Product Owner.

If like many organizations your picture looks like this one, you are in need of organizational transformational design. You just entered dependency and prioritization hell. Agility is NOT possible in this scenario and Scrum could make it even worse as it forces you to face the realities of what's not possible without organizational change.

Attempting to implement Scrum or any Agile approach into an organization without addressing the causes of what's preventing you from being agile will only result in more frustration, and ultimately a list of reasons why Scrum doesn't work, or excuses why your organization is different in some way.

Ken Schwaber is quoted as saying something like "Scrum is like your mother-in-law, it's constantly pointing out your shortcomings." The reality is Scrum does not fix anything! What it does do is change behaviors and through doing so your shortcomings and organizational challenges are exposed. This is painful and requires embracing transparency, one of the pillars of empiricism Scrum and PDCA is based on.

This takes engaged and participatory leadership. Leadership who themselves are willing to change their roles and their actions in solving structural and systemic problems. If you continue to do what you've always done, you can expect to get what you've always gotten. If you truly want to be an agile organization then you have to change everything, and not just the technical organization, while the others continue without change.

Summary

If you want to transform into an agile delivery organization, you are going to have to transform your entire organization. Remember, the focus is on delivering better and faster value for your customers. Lose sight of that and fail faster with agility.

Leadership must address the organizational structure and reshape and restructure that into small, adaptable, and flexible workforces that can deliver while being unimpeded by the corporate colossus. Move the noise to the hierarchy and away from the value delivery units.

Create true value streams that can deliver capabilities without dependance on external groups by building in independent shared services and of course leveraging capabilities such as DEVSECOPS, and compliance as code.

Build an effective Product Ownership team that meets continuously to review progress, value, and priorities. Be fully engaged in that team or truly empower the team members to operate with full autonomy.

All Agile and Scrum first focussed transformations will fail if they do not address the organizational and structural issues associated with legacy corporations. Strong leadership with a will to change and restructure is necessary.

My team is focussed on training organizations in these and other effective techniques to design effective value delivery organizations by understanding complexity, effective leadership, the science of teams, and of course The Toyota Production System.

Footnote

I started by saying that by solving the role of the Product Owner at scale, you will solve the key reason transformations fail. Product Ownership starts with the CEO focussing on the customer priorities for the enterprise. The organization is then able to rapidly turn that into customer value. If you designed an effective Product Ownership organization you will have designed an agile organization and effected a true transformation.