Why do Agile Transformations fail?

Agile is the ability to give the customer what they want quickly. People who create products or services orient themselves towards the customer to deliver value under a guiding vision from management who facilitate and empower teams rather than direct predefined work. Simple, eh? It is a simple idea but stubbornly difficult to do because a current way of working is so entrenched and reinforced by compensation, cultural expectations and mastery of current processes developed by people over many years.

The big picture is workers now need to listen to customers, rather than to functional managers on what to produce and when. The smaller picture is workers, working in teams, have to decide among themselves how to do things rather than having tasks laid out for them by managers. This is a difficult shift to make, both by team members who do the work and by managers who must shift from directing work to facilitating teams to create value.

Most organizations I speak with or hear about want to become Agile, says they are transitioning to Agile, or that they already are Agile.

With such interest, and an endless parade of consultants set to transform your business, one would think an Agile organization would be common. The actual landscape does not match the public face. Most organizations are not Agile and the majority of those that are have been in business for fewer than twenty years. Transitioning to Agile is hard.

Why is this. In my observations there is a common lack of understanding of what Agile is and, closely tied to it, an insufficient effort, backed up by a poorly aligned commitment, that is needed to successfully transition an organization from a command and control mindset to a focus on the customer one.

From my consulting work and speaking with others I have identified the following failure modes to a successful Agile transformation. Below and in further updates I will address each mode in turn.

1. Lack of commitment by management

2. Management does not understand the scope of what has to be done and the effort that will be necessary to make a change to Agile.

3. Misconception that this is simply another management reorganization

4. Misconception that Agile is something that takes place in the IT department for everyone else in the company to ultimately enjoy.

5. Misconception that and Agile transformation will be disruptive to productivity so it needs to be implemented slowly.

6. Misconception that Agile will not work with our products, or with our career staff or wit our internal or external stakeholders. It may work for others, but won’t work here, mentality.

The section below is a work in progress. These are my thoughts but I have not yet been edited and integrated into the above presentation. Deliver value quickly as in the first page above, develop more based upon reader feedback, Agile!

=======================================

To support an Agile transformation an iterative development team needs to be fully cross functional and self organizing. Cross functional refers to the idea that all of the talent necessary to bring user stories from the backlog to done lies within the team. There are no hand offs to external groups or individuals to perform work and no occasional members who join the team temporarily or to do a task. This, by the way, includes code reviews. If a code review or test plan approval is done by an external product owner or QA lead then the team must rely on that external resource to complete it before the team can move on. Reviewers of work done during a sprint should lie within the team.

The team is self contained and can perform the work, as they see fit, without coordinating or collaborating with other individuals or teams. Team members focus their efforts on completing user stories. Working together, they feed off each other’s expertise. At the end of each sprint team identify ways to improve. Each sprint becomes more productive and with fewer defects as the team moves toward High Performing.

Teams should only accept user stories that meet their definition of ready, have clear acceptance criteria and can reasonably be delivered to the agreed upon definition of done by the end of the sprint. Only when they have a clear and well defined set of tasks can they plan out their work and collaborate to get it done by the end of the sprint.

The second part, self organizing, is more important, and more difficult to achieve. In the case of being cross functional, an organization can adjust its definition of done to avoid the need of employing others to help out. Self organizing, on the other hand, requires empowerment by the organization.

The team must be given freedom and be insulated from outside influences to allow them to perform the work as they see fit. Teams can then focus on user stories, developing a sense of ownership for the user stories and their work.

An agile transformation can only take hold and stick when there is a consistent message from management. Once product owners, project managers and business leads come to accept that they have the freedom to be agile without management constantly demanding waterfall reports, they can stop issuing performance orders and leave the team alone to do their work.

It will take some time for product owners to change their focus from giving work to teams to the creation of a book of work for teams to draw from. Product owners will need help to develop INVEST user stories, create a backlog and to keep it prioritized.

Product Owners, Project Managers and Leads often all bear requirements. In these cases, it is important that they speak with one voice. I have heard of several teams where it seems there are multiple Product Owners making it difficult for a team to know who represents the voice of the customer.

To set the stage for a self organizing team to choose their work for a sprint, a good backlog which has been vetted by the team should be created that is at least two sprints long. Teams that can choose their work are in a position to take down the right amount of work so that they finish a bit early.

Work is not decomposed into actionable user stories or even well understood requirements with clearly identified dependencies before a sprint begins. Refinement of the work begins too late to permit the team to make informed choices on the work they accept into their sprint. Refinement of requirements and dependencies continues throughout the sprint and is often the focus of the “second half” of many daily stand up meetings. Without knowing what the work is, it is impossible for a team to lay out their work, decompose it, and work out for themselves how to complete it. This “figure it out as we go” makes it very difficult for the team to intelligently accept the right amount of work to take into a sprint in the first place. This lack of understanding of the work to be done, coupled with a strong willingness to please, leads teams to accept far more work than they can properly complete in a sprint.

Agile team members will often say work is agreed to at the sprint planning meeting. However, I have attended a number of sprint planning meetings and found it is more of a waterfall assignment than an agile selection by the team. Product Owners very often assign work, rather than offering the team prioritized list of work for them to select from. Typically, more work is assigned than can be completed during the sprint. The thinking it is better to give more work then have the team sitting around doing nothing. Under these circumstances, the team does not have the opportunity to fully test in extensive and creative ways or refactor their code as much as is possible. A team that buttons up everything has the luxury of being able to reflect upon their work and how they did it. The following sprint retrospective can then be a real opportunity to identify an area of improvement to take into the next spring.

I am in no way faulting product owners, project managers and development leads. They are operating in an efficient, waterfall manner. I will say there is a lack of understanding that can be addressed with formal training, coaching and mentoring. It takes time to see things in a different way than the one you have practiced for so many years. Developing agile requirements is easy, but only when you have learned how to do it and have practiced it for a period of time. This is why agile support is needed for the better part of a year.

Currently, incomplete stories are simply tacked onto the next sprint along with more, newly assigned work. Older user stories are completed, newer ones are deferred, and the cycle continues. I have not heard any pushback by the team to rein in this excessive work in process for a spring. In fact, in wanting to please This is really a waterfall process with check points every three weeks. 

This again is not the deliberate fault of those who assign the work. Team members too, in their exuberance to deliver more accept far more work then they can complete. I have seen teams complete 50% of their user stories in a sprint. And the work that has been assigned often not been decomposed into small work packages, established clear acceptance criteria and sized before the sprint planning meeting.

Collaboration versus Coordination

All of the daily stand up meetings I have attended have primarily been dedicated to coordination. Team members report on the progress of work that had been assigned to them and raise any issues of dependencies with other components. Sometimes these dependencies are with other members of the same team, sometimes with other teams or business units outside of the team. Except in limited cases, developers (any team member in SCRUM, XP or Kanban is called a developer) work in their domain and does not cross over from QA to Dev, BA or DevOps. Even within domains, incomplete user stories are rarely passed from one developer to another. A culture of collective ownership toward user stories has not been developed on any of the teams I have observed.

Unfortunately, I have not heard a single instance of collaboration discussed at a daily stand up meeting. In other words, nobody says, “I’ll help you with that” or “Let’s all work on this user story today”. Some of the meetings are brief and efficient, others are as long as an hour or more. While longer meetings are not optimal, they do serve a very valuable purpose where team members are not co located. They are used to resolve dependency, user story clarification and even renegotiation of user stories during a sprint.

I strongly recommend you do not unilaterally require stand ups to be 15 minutes long. They are a solution to other issues that you have and are a valuable bridge. Fix the underlying issues and those meetings will become shorter on their own.

It is perfectly to modify meeting lengths to suit a team’s circumstances. On a related topic, I would like to mention team size. It is perfectly ok to stray away from the ideal team size of 6. While communication channels increase quickly with more members, it is often difficult to be fully cross functional with 6. I suggest teams be formed around total capability, cross functional, rather than trying to limit them to an artificial limit such as 8.

On one particular team, I have listened in on the demonstration, sprint planning and grooming meetings for more than a month. I have also attended most of their daily standup meetings. There is an assigned Product Owner. That Product Owner decides what work will be done in each sprint. The PO assigned work at the Sprint Planning meeting. The work had not yet been sized by the team. The team was not asked how much work they can accept. The technical specifications had not yet been finalized so there was no clear acceptance criteria. This person said they had no problem with the team being agile. Yet, it is this stream of waterfall requirements that have not met the requirements of the definition of ready, with clear acceptance criteria that will make it impossible for the team to become a well functioning iterative development team. There is nothing the team can do to become agile without this.

An agile team themselves prioritizes and divides up the work to be done during a sprint. A team that has worked through the phases of forming, storming and norming enters the performing phase. Achieving this state enables teams to continuously improve and accelerate their output. High performing team members have a good feel for the capabilities, interests foibles and personalities of each other. This is how they optimize their process for getting the work done within a sprint.

Communications

The ability of team members to collaborate is affected by both physical separation and time zone differences. For US based team members the primary difficulty is team members are typically not in the same room. There is no ability to share a Kanban board or the non verbal cues we all exhibit in our face to face conversations. Under an ideal scenario, all members of a team will be co-located. In this case team members typically come into the office at least a couple of times a week so are able to work together in the same room to pair program or swarm on user stories.

But, ideal is often difficult to achieve. With technology it is possible to pair program with another team member and hold high fidelity team meetings where everyone can be seen on webcam. 

To collaborate, teams need to be in close contact with each other. While I have said not all teams need to be 100% colocated, the caveat is that team members must be able to collaborate effectively with technology.

Transitioning a well functioning, complex group of development teams from a waterfall approach to an iterative agile one takes time and patience. This is not a reorganization but a change in the way we work together, driving decisioning down to the person who is doing the work and knows the most about it. Managers will have to learn to let go and trust their teams to figure out the work and do it. Teams will have to learn to focus on getting user stories done and not always thinking about a legacy job function. This will take time and trust.