The Deficits of Agile Coaching - Challenges and Solutions

The other day (July 20th) I had the honor to participate in an Event at the Agile Munich Meetup. The format of the event was an "Oxford debate", a highly structured debate where proponents and opponents are supposed to do their best to convince the audience. The facilitator takes polls both before and after the debate.

The leading question of the debate was: Are Agile Coaches Catalysts for Teams and Organizational Progress? I was part of the opposing team and had to convince the audience the coaches are not catalysts – an interesting feat with my background as a long-term agile trainer and coach.

The very inspiring and interesting debate leads to this article: I try to rephrase my arguments – slightly more balanced – and to add some consequences. I want not just to point out deficiencies, but also contribute to a longer-term debate: if agile goes into the direction of a commodity, what ist the next big thing? Should we really think beyond agile and what are the candidate theories / methods / frameworks / none-of-the-above?

Introduction

Agile coaching has gained significant popularity in the software development and project management realms due to its ability to foster adaptability, collaboration, and continuous improvement.

Agile coaches have been helping teams and organizations implement agile methodologies. However, despite its numerous advantages, agile coaching is not without its deficits. In this article, we will explore some of the common challenges faced by agile coaches and discuss potential solutions to address these deficits effectively.

The Challenges

• Lack of Clarity in Roles and Responsibilities.?One of the primary deficits in agile coaching is the lack of clear definition and understanding of the role itself. Agile coaching can be ambiguous, and organizations may have varying expectations of what a coach should do. This often leads to confusion within the team (and the organization as a whole), with some team members, unsure about the coach’s authority or how they fit into the existing hierarchy.
• Inadequate Experience and Expertise.?Agile coaching requires a deep understanding of agile principles, methodologies, and best practices. But knowledge about agility is not enough: the coaches also need the expertise to deal with the existing structures, reward system, and all the other realities in an existing organization. Unfortunately, there is a shortage of experienced and qualified agile coaches in the industry. This is made worse by the fact that the expectations of the coaching role are so unclear. As a result, organizations may look for a pink unicorn and end up with coaches who lack the necessary expertise to address the specific set of complex challenges effectively.
• Difficulty in Measuring the Impact.?Measuring the success of agile coaching efforts is challenging. Unlike other roles in a project or the roles of managers, the impact of coaching is often intangible and difficult to quantify. Coaches in a company-wide role also run the risk of taking on a proxy role to a manager. This makes the roles and their accountabilities even more unclear. Managers may feel challenged in their role and Organizations may therefore struggle to see the return on investment (ROI) for hiring agile coaches, leading to skepticism about their effectiveness.
• Resistance to Change. Agile transformations (like other change initiatives) often face resistance from various levels of an organization. Team members may be hesitant to embrace new methodologies, and management might be reluctant to relinquish control. Agile coaches are then faced with the challenge of navigating these resistances and at the same time fostering a culture of openness and collaboration. If the coaches are hired to overcome resistance, without changing the system as a whole, they are doomed to fail – and they leave often a lasting negative memory of the initiative or agile as a whole.
• Building a shadow structure in the organization. Agile Coaches need to justify their role and secure their position as everybody else. Especially for external coaches, this means a preference for long-term contracts and getting embedded in the organizational structures. This mirrors the development of the classical consulting industry when huge consulting firms deepen dependencies and amass more and more competencies away from their clients. It makes accountabilities less clear, it can become toxic when coaches are used by managers to get "off the hook" for driving change and it is counterproductive: organizations tend to react like a metaphorical rubber band that returns to its initial position when the pressure is released.

Do things better and do better things

It’s more than a platitude that it’s necessary to both do it right and do the right thing. I’ll try to give a few nudges in both directions.

A first set of ideas to improve things – in coaching and beyond

None of these ideas are really new, but many people still engage in stubborn "try harder" rather than change the way they are doing things. I’ll try to collect a few of these ideas here:

• Make the responsibility transparent. Edgar Schein [1] distinguishes "expert consulting" and "process consulting". While an expert consultant has expertise in, let’s say Diesel engines, a process consult helps the organization to perform. In doing this, the process consultant develops a "helping relationship" with the client. Sounds familiar? Edgar Schein wrote about this in the 1980s. When we add Kühl′s "person-oriented consulting" [2] to the mix, which includes mentoring, coaching, and similar activities, you have a much clearer spectrum of roles. Whenever you can not get hold of Chuck Norris or a pink unicorn, you might be better off when you have a clear term for the specific job to be done and can hire on this basis.
• Educate and enable. Do not just change role names. Scrum Masters get a certificate in a two-day course. This is a license to learn how to be useful for a team if they have some expertise in teamwork and some talent. They need a learning environment and can profit from a mentor. If a company provides opportunities to learn, this adds up to a significant investment and a shift towards a learning organization. This will pay off in many ways, more motivation, a more flexible workforce, more effectiveness, and more adaptivity and viability. On the other hand, if a Scrum Master becomes a coach by attending another 5-day class and manages not to die during that time, this becomes ridiculous. We need a more serious approach to consulting/coaching.
• Do not try to change a system against its will. For lasting change and improvement, it must be embedded in the structures and the culture of the organization. If it is easier to fall back to the old ways, people will fall back[4]. No amount of mindset work or admonition will change this. This means an intervention should be an experiment, which may improve the conditions for subsequent initiatives, like micro nudges[3] (found thanks to Dave Snowden) slowly turning the tide.
• Change the system, do not rely on crutches. Consultants (again, this includes coaches) can be very helpful as temporary scaffolding but may become harmful if you let them morph into permanent crutches. Do not let the managers from the hook. Make them do their job and do not allow them to rely on crutches. Their job is to provide an environment in which everybody can be effective, creative, and thrive. This can not be delegated to a coach or a consultant, it must be an integral function of the system. Managers must be enabled, empowered, motivated, and rewarded to fill these responsibilities.
• Look beyond Agile. Well, this belongs already to the category of "do the right thing", so let’s switch to the next section.

A few directions to go beyond Agile

There is a big shift underway in Agile. The shift towards commoditization. This means standardization, but it means also that there is a relatively stable scope and state-of-the-art. Some try to extend the meaning of Agile into other areas – I am not a friend of this. It might be better to leave Agile where it makes an impact and look for greener pastures.

This is the result of my research in the last months (some goes back years). It is a personal selection of tools I found useful, plausible, or enlightening. It is by no means complete, it is – at least for me – a starter kit for the next big thing.

• System theory – more than system thinking. System thinking, i.e. thinking and working with feedback loops and complex interactions, is a strong tool in itself. It can help to question and break the control illusion of die-hard machine logicians. But there is more. First, there is a whole family of system theories, each with a rich practice and many with a solid theoretical foundation. The arc ranges from cybernetics as the science of control, the source of all of them, to human system dynamics, soft system theory, and many others. For me, one of the most important is the?Viable System Model, which applies to decision-making and communication processes in organizations. You will find a myriad of practices that are also used in agile contexts. The great strength of these books and papers is that these authors put a lot of emphasis on empirical verification of their work and cite their sources extensively – which is not the case for all authors in the agile field.
• Complexity tools – the champion skeptic among theories. While Cybernetics – and the theories derived from it – claims to be the science and theory of control, complexity theory says: in a complex adaptive system, there is no such thing as control. This is an interesting challenge. It prevents systems theorists from selling their thing as a silver bullet and brings them down to the level of tools. Tools can be combined; their effectiveness depends on the people who use them. In addition, in the area of complexity theory, there are some unconventional ideas and approaches that can be used to break down entrenched structures and old thinking and gain new insights.
• Go back to the roots of Agile: look into Lean. Lean is one of the main sources of Agile. If you look at the early Scrum papers of Jeff Sutherland, you could say that if you port Lean to the software industry, you end up with Agile – well, at least Scrum. One example of going back would be: Teach structured problem-solving. Structured problem solving is tailored for complicated, non-complex environments, but first, in every complex situation there are also complicated sub-areas, and second, and more importantly, it is an excellent example of how to plan your work thoughtfully and execute it well. and that, dear fellow monastics, is not a given everywhere.
• Organization Design and Organization Development. If you play with the acronym OD, you could add: Organization Debugging. While many of the people in the field are firmly rooted in traditional change processes, it is not all of them. OD has one huge advantage over Agile: OD tries to get a complete view of the different dimensions of an organization, while Agile and Lean focus heavily on processes. Again, while Agile has advantages, it is time for a more general approach. And, by the way, one reason I fell in love with the Viable System Model, is because it provides very precise tools for Organization Debugging, i.e. identifying deficiencies and pathologies.
• Work on the base of how people behave, not how I want them to. Working on the mindset of people is intrusive and, which is my argument here, it does not work (pun intended). If and when we want to change the way people behave in an organization, we need to change the conditions. This includes the opportunities and resources they have, the degree of freedom, making a real impact with their actions, and the way relationships work at the workplace. This includes also trust and personal safety: they are part of the real environment and not a part of mindsets.

If you also would like to discuss new directions to work and act, and my ideas resonate with you, give me a hint what are the most promising candidates. I would absolutely love to initiate a broader debate to avoid the commoditization trap.

---

[1] Schein, Edgar H. 1999.?Process Consultation Revisited: Building the Helping Relationship. Reading, Mass: Addison-Wesley.

[2] Kühl, Stefan. 2008.?Coaching und Supervision. Wiesbaden: VS Verlag für Sozialwissenschaften.?https://doi.org/10.1007/978-3-531-91136-6 .

[3] ?Nudge theory – Wikipedia“. o.?J. acccess: 28.07.2023.?https://en.wikipedia.org/wiki/Nudge_theory .

[4] ?Assemblages, Agency and Affordances- a discussion with Dave Snowden and Ellie Snowden“. o.?J. access: 28.07.2023.?https://cognitive-edge.wistia.com/medias/otsev5jq9e .