Agile Transformation or Agile Disformation?

(Lutz Malburg, Novatec Consulting GmbH, 12.12.2023)

In this article I want to share my experience why many Agile Transformations get stuck or fail and end up in a state I become to call “Agile Disformation”. I will not talk about change management issues like building a guiding coalition or achieving quick wins. You need great change management to do a great Agile Transformation. But if your basic setup is wrong, change management won’t come to the rescue.

Please let me know in the comment section what are your experiences and thoughts.

Introduction

To get started I define what is my understanding of an Agile Transformation and what I mean by Agile Disformation.

While there is not one defined target state for all companies and individual results differ widely, there are certain aspects an Agile Transformation needs to achieve to be successful, e.g.

Product orientation

• Development is done based upon products and not in projects.
• Every product has one or more teams assigned, that develop, maintain, and operate the product throughout its entire lifecycle.
• Products are released early and often and are evolved based on customer and market feedback.

Focus

• Focus on the right products that are in line with your business strategy and that your customers really need.
• Make sure that the people doing agile product development can focus on developing one product instead of hopping around constantly, trying to balance the parallel development of many products.

Empowerment of agile roles

• The agile roles involved in the chosen agile approach are empowered to take decisions and to lead the development of their product.
• The empowerment of agile roles requires trust and the right mindset to enable the required self-management for agile teams.
• As stated in the principles of the Agile Manifesto: “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

Learning organization

A learning organization understands that

• despite their best effort, people will make mistakes. I am not talking about negligence.
• learning from mistakes is often faster and more targeted than trying to work out everything up front.
• instead of looking for the guilty person it makes much more sense to analyze mistakes and derive measures to make sure the same mistake will not happen again.

Agile budgeting

• When products are evolving based upon customer and market feedback it is impossible to fix a budget upfront.
• Instead, agile budgeting needs to allow for ongoing budget decisions based upon the success of a product and the additional business value that can be generated with further investments.

Top-management is involved

• The top-management is involved and fully backs the agile product development.
• It is understood and accepted that agile product development affects the whole organization, not only parts of it.

Ability to scale agile development

• When switching to agile product development you will inevitably reach a point where one team is not enough to develop a product.
• You will need to be able to orchestrate three, for, ten, twenty or even more teams that work on a product.
• You will also need to manage the interaction between teams working on related products.
• This means you must have processes and structures to manage dependencies and to make sure you integrate all relevant parts frequently in accordance with your development cadence.

If only parts of the required changes for a successful Agile Transformation are implemented, e.g., if “Agile” should be done by “the IT” only, I call this an “Agile Disformation”. This means an intermediate state where your former way of working has been changed, but not far enough to enable agile product development. As a result, former ways of working are disturbed but a clear new way of working is not yet established. This leads to a loss of performance and frustration for the people caught in this in-between state. This frustration is even bigger if you brought in experienced people who already worked in a real agile environment. For them an Agile Disformation is a disappointment and a regression in their way of working. This holds true for internal as well as for external people.

Main Reasons for Agile Disformations

Based on my experience the following are some of the main reasons why companies end up in an Agile Disformation.

No clear vision and goal

An Agile Transformation is started without a clear vision and goal. There is no common understanding what should be achieved with the Agile Transformation. The same transformation means different things and is supposed to solve different problems for different people. Consequently, there are no clear criteria do validate if the goals of the transformation have been achieved.

Top-management is not involved

Top, management is only informed but not involved. There is an “I don’t care what you do, make it work” attitude. But there is no understanding for the necessity and no will for a real change. Therefore, the basic way of working within the company remains unchanged. Essential decisions like adopting organizational structures, changing the way how a product development can be started, or how budgeting is done are not taken. Setting up cross functional agile teams clashes with department borders and unaligned goals.

Agile is not understood

An agile way of working and frameworks like Scrum are misunderstood as another and maybe better way of project management. It is not understood that working in an agile way means a fundamentally different approach to how products are developed, tested, released and how stakeholders and their feedback are an integral part of the development process of a product.

Organizations fall short to understand that that agile product development affects the whole company and almost all processes within the company. Very often discussions are centered around the notion that a product may only be released when it is completely done. This misses the whole point of doing agile development where the goal is to develop a product that delivers the best value possible based upon stakeholder feedback for product increments that are released early and often.

The way agile frameworks are set up and the empowerment needed by the included agile roles to be able to do their job is neither understood nor supported. Therefore, people in the new roles get stuck in existing and unchanged processes and are not able to do agile product development.

An agile way of working should be a reaction to a situation where you are dealing with a complex environment. In a complex environment you can’t predict what will happen. You can make assumptions and try to verify those assumptions as quickly as possible based on constant feedback. In this sense when you are working in an agile way you are basically exploring the future, not predicting it.

You need to understand the essence of an agile way of working. Alistair Cockburn describes this essence as “The heart of agile”. If you take away all frameworks and methods what remains is: Collaborate, Deliver, Reflect, Improve, and then do it all over again.

Doing it for the wrong reason

Many companies are struggling with their classical project management approach and fail to deliver projects or programs successfully. This leads to an increasing pressure to become faster, cheaper, better, and more reliable. Unfortunately, many companies turn to agile in this situation without rethinking their overall approach. The fixed scope for a new product or project remains, but the expectation is that “agile” will make things somehow faster and cheaper, but the scope is fixed and the work that needs to be done remains the same. Therefore, you don’t get faster or cheaper. While you might theoretically be able to save some time by setting up some real cross functional teams, most companies fail to do so.

Expecting people to know how agile works

Many companies take the decision to turn to an agile way of working. While this is a big transition, companies expect their people “to know” how agile works. Teams are not trained together and are not supported by an agile coach in their early startup phase. People in the company are expected to work out themselves how agile is done and how to get the ball rolling. Consequently, people within the teams and in the management have a different understanding about agility and lose a lot of time by misunderstanding and misinterpreting agile frameworks like Scrum and by doing things wrong.

Copy, paste, Spotify

As human beings we are often trying to avoid risks. If confronted with challenges, we try to find proven solutions that seem to make sure we will be successful. Unfortunately, this often leads to copy, paste approaches for an Agile Transformation.

The most prominent example for this is companies saying, “We are using the Spotify model”. There are at least two problems with this approach. First, there is no “Spotify model”. What people refer to is a flashlight on the way Spotify was working many years ago, that Hendrik Kniberg gave in some short videos. Second, you are not Spotify.

The presented approach worked for Spotify at that point in time, as the result of a journey the company had gone through to reach this point. If you take this approach, and simply paste it to your organization, you will very likely fail.

The people in your organization are in a different situation, have a different mindset and typically a lot less experience than Spotify back then. That is why frameworks like Scrum or Kanban define a certain structure and a set of rules but require you to fill in a lot of additional parts that make sense in your situation.

Expecting the change to happen instantly

Working in an agile way is foremost a mindset change in the way things are done. This means rethinking and changing values, principles and practices based upon positive experiences while doing things in a new way. This takes some time and does not happen overnight.

An example for this is that people coming from classical project management often struggle with the idea to start the development of a product before the business requirements document is finished. They need to experience that starting the first development iteration with a small set of requirements deemed the most important ones really works. They need to experience that releasing early versions of a product and guiding the future development of a product using the feedback provided by stakeholders really works. Only then will they be able and willing to change their mindset and start to embrace an agile way of working. This holds true for the people in the agile development teams as well as for the upper management within companies.

Planning an agile transformation with fixed milestones

Sometimes companies try to plan an agile transformation upfront with predefined milestones and a fixed end date. With an organization being a complex adaptive system, you cannot predict what is going to happen when you start an agile transformation. Some things will be faster than expected, others will be slower or will not work and new opportunities will show up.

There will be unexpected developments that will require a flexible response according to the situation, adopting the plan. There is nothing wrong with having a plan and breaking down an agile transformation into milestones. But you must expect the unexpected and need to be willing to react to the things you learn along the way.

An agile transformation is a complex undertaking. The further you plan into the future, the more likely your plan will need to change to keep up with reality. If you neglect that you cannot predict the unpredictable and are not willing to change your plan along the way but try to bend reality instead, you will get stuck in an Agile Disformation.

Permanent overwhelm

Many companies fail to do a stringent portfolio management. Yes, it is sometimes hard to take the decision not to start a certain product development. But as a company you only have a certain capability to develop products for your internal and external customers. If you are working in a regulated environment, you must also take care of the respective regulatory requirements. You may enhance your capacity by hiring external support, but you also need the internal capacity to steer and control external partners.

If you try to push to many product developments in parallel through your organization, you will permanently overwhelm your people and you will pay for it immediately and again later.

The short-term price is that you delay the development of important products because people are jumping between parallel developments. With deadlines approaching and increasing pressure to deliver all products in parallel, people start to take shortcuts, create quick and dirty solutions creating technical dept, postpone necessary updates of e.g. basic software and libraries, reduce tests , and sacrifice the product quality.

The long-term price is that you end up with products that are hard or impossible to maintain or evolve and product environments that are outdated and no longer possible to support. This typically leads to unplanned migration projects that require a lot of capacity and jeopardize planned product development.

What can we learn from all that?

Be very clear about what problem you want to solve

First, it is necessary to be very clear about what your real problem is. If for example project management is done badly, jumping to agile without understanding what agile really means, will lead to a bad adoption of agile that will not solve your initial problem. You need to be very clear about what is the problem you are trying to solve and what is your vision of the aspired future state.

If the problem is that market and customer requirements are changing faster than your company can deliver suitable products, agility might come to the rescue. But you need to invest enough time, especially on top management level, to understand that switching to an agile way of working is a disruptive change. You will not be able to achieve substantial benefits, without fundamentally changing the way products are developed, reported, maintained and how budgeting for agile product development is done. These kinds of changes require decisions on top management level which is why top management needs to be involved.

Find the right solution for your situation

Your situation within your organization is unique. This does not mean agile does not work for you. It means you should not try to find a ready blueprint that solves all your problems and eliminates the necessity to think.

You should look at the problem you are trying to solve an than look for a suitable solution. Depending on your requirements, the size of your undertaking and the number of people involved, different approaches will make sense. If you do not have experienced people within your company, bring in experienced agile coaches to find and evolve the right setup for your situation.

Be ready to change your mindset

You need to be willing to let go the idea that you already know what your customers want and that a product can only be released after it is completely done. You need to change your mind set in a way that allows you to put your customer needs at the center of your development efforts and to constantly evolve your product based on the feedback you receive for incremental product versions that are released early and often.

The key is to ask customers for their needs, not for solutions. A good approach for that is the “Jobs to be done” framework which is aimed to the question which jobs or tasks customers need to get done and then designing solutions that help customers to achieve that easier, better, cheaper and or faster.

Take care about your ability to scale

Scaling agile development is a demanding task. Instead of developing your own model from scratch, use the experience people gathered in frameworks like Nexus, LeSS, Scrum@Scale or SAFe and decide what is the best fit in your situation. Don’t expect to get everything right the first time. Try out approaches, do retrospectives constantly to understand what is working for you and what isn’t, and adjust accordingly.

When you start to scale, make sure the people involved are properly trained and experienced to master the basic framework (e.g. Scrum) that you are trying to scale.

Understand the importance of quality

The classical iron triangle of project management scope, time and budget often neglects a very important 4th dimension: Quality. If the scope increases, time stays the same or is shortened and the budget remains unchanged or is cut, typically the quality of the results suffers. This means hidden costs that will hit you later in your product lifecycle in many ways: Low acceptance of your product, unstable operations, ongoing bug fixes, technical dept to name just a few. In an agile approach we try to provide the product with the highest possible value for our customers. Therefore, quality is not discussed but is considered a non-negotiable product property.

Create focus

There are always more ideas for things to be built than you have time and money. This means you must prioritize on company level and decide which product development is in line with your business strategy and which is not. To do that you need a first-class portfolio management.

If you fail to prioritize accordingly, you will end up with people hopping between two, three, for, or even more projects. While it might seem that you are doing more, you are actually slowing down and are compromising your quality. There is a reason why “focus” is one of the values in Scrum. This is based upon scientific research and the insight that jumping between tasks and constantly de-focusing people causes a lot of lost time and energy.

The better you manage for people to focus on developing one product, the faster the teams will be able to go and the better the quality you achieve will be. The faster your teams can go, the faster you can release product increments and the faster you get feedback for the further development of your product.

Actively manage your change

An Agile Transformation is a big change that requires appropriate change management, e.g. You need to understand what you need to change and why, tell the people in you company about the urgency of the change – why do you need it now? Since change management is a big topic on its own, I will not elaborate deeper on it in this article.

Be aware that the change will take time

An Agile Transformation takes time and needs the space to develop. There will be a lot of changes you need to think through, and the people will need to get accustomed to. For example, there will be new roles within your company, exiting roles will change or disappear.

Many processes like starting a product development, budgeting, reporting, or operating a product will change. Often the existing mindset and rule-set are too strict to allow this to happen. Therefore, it often makes sense to create new organizational structures to provide the necessary freedom for the required change. This might be a new division or a subsidiary company that is explicitly created to allow for an environment with different rules to allow for an agile way of working to develop and blossom.

Invest in your people

You are trying to introduce a fundamentally new way of product development into your company. Do not expect your people to just know how to do it or to figure it out along the way. Get help from experienced people to figure out what makes sense in your situation. Then invest in appropriate training to bring everyone involved to a common understanding of your situation and your approach. This explicitly involves top management to make sure that agility is really understood on all levels. Bring in experienced agile coaches to guide you through the process to avoid pitfalls and plain bad decisions.

Do your Agile Transformation in an agile way

An Agile Transformation is a complex undertaking. You can’t predict what will happen along the way and how people will react. Consequently, you can’t set up a milestone plan and a fixed budget upfront and expect your transformation to follow your milestones and budget plans. You need to do your Agile Transformation in an agile way. You need short feedback cycles in your process to learn what is working and what isn’t and adopt accordingly.

There is nothing wrong with milestones and setting up budgets if you are willing to change those according to what you learn during your journey. If you start to try to bend reality to your plans, your transformation will suffer, and you are likely to end in an Agile Disformation.

An Agile Disformation is not necessarily the end of your attempt to do an Agile Transformation. If you are willing to be honest with yourself and your situation, you can initiate steps to start moving forward and fix your situation. But you will have lost time and money. Therefore consider the success factors above and bring in experienced people to avoid getting caught in an Agile Disformation.

Final thoughts

When you start an Agile Transformation, it is typically something you have never done before as a company. Even if some of your colleagues have been part of agile transformations in other companies, keep in mind that what worked in their situation is not sure to work for your company in your situation.

Management and Leadership

To do a successful and sustainable Agile Transformation you also need to evolve your management style end develop agile leadership capabilities. On the Management level you need to understand that your company is a so called Complex Adaptive System (CAS). This means you can’t predict how your organization will react to changes.

One way to help you deal with complexity is to have self-managed teams that are empowered to take fast decisions within their realm of responsibility. To make self-managed teams possible, you need to develop agile leadership skills to create an environment in which self-managed agile teams can arise and prosper.

A great tool to support you in this is Management 3.0.

The whole Management and Leadership issue is worth an article on its own.

Conway’s law, organizational structures and unFix

Conway’s Law tells us that “Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.” Since the communication structures of an organization are closely linked to its organizational structures, you need to develop the capability to change your organizational structure to be able to develop the products you want.

Since the world around your company is ever changing, you need to be able to easily reflect those changes in your products and therefore in your organizational structure.

Different organizational structures like hierarchical and networked have different strength and drawbacks. So, the question is not hierarchical or networked but how to manage the right balance between the two. In his book “Accellerate” John Kotter calls this “The Dual Operating System”. Jurgen Appelo calls it “Ambidexterity”.

Management 3.0 helps again to understand the complexity behind organizational structures and how to approach it. Another great tool to understand an evolve organizational structures is the unFix model developed by Jurgen Appelo.

The organizational aspects of an Agile Transformation are also worth a separate article.

No static state

A well-done Agile Transformation will not lead you from one static state to another. It will transform your organization into a fast learning, constantly adopting and changing company. In the end it comes back to the essence of agile on all levels: Collaborate, Deliver, Reflect, Improve, and then do it all over again.

?

I hope that the experiences and thoughts I shared in this article will help you to reflect what problem you want to solve, provide some guidance, and help you to do a successful Agile Transformation and avoid ending up in an Agile Disformation.