Agile for Today

During the last year or so I became discontent with Agile. Right of the bat, I should qualify this by saying it is probably more about processes such as Scrum and not so much about Agile. But regardless, it is something that should be investigated.

This article documents my search to discover what exactly is bothering me, what is wrong with Agile, what are the alternatives and how to fix it. It is also an invitation for discussion on this. So here goes…

The problem

The Agile Manifesto was formulated in 2001, and it took about a decade for Agile to become the de facto standard of software development and to replace the waterfall method. And it was awesome. It improved the way we worked; it made us more efficient, more responsive to (paying) customers’ needs, and invigorated development teams and environments.

But during the last few years some things happened. Or didn't happen:

• Technology changed. A globally distributed workforce emerged on the backbone of technology that enables it (read Slack, Skype, or whatever communication/chat app you use). Individuals and interactions seem to be less important than the ability to interact across continents and time zones. Working together and face-to-face communications are just not that important anymore. On top of this, it seems like developers favour using Slack over talking to each other when discussing work, even if they sit right next to each other. Right or wrong, it is a fact that we need to deal with. Furthermore, DevOps and Continuous Delivery teach us that we shouldn’t wait until the end of a sprint to release stuff: just release it when it is ready to go.
• Lean principles are catching up. Specifically the one about eliminating waste. Companies (and paying customers) are no longer happy to see an entire development team spending 10-15% of their time in meetings. Not to mention the fact that developers are getting bored with daily stand-ups and fortnightly planning, review and retrospective meetings (as illustrated by scrum masters’ continuous quest to find more engaging ways to keep these meetings interesting).
• Agile became institutionalised. These days any man and his dog can become a certified scrum master or product owner. That in itself is not a problem, but the fact that some organisations require these certifications certainly is. “Agile at scale” is becoming a thing, even though it is nothing more than a backlash from project-minded dinosaur organisations that feels a need for relevance. (see Marty Cagan’s brilliant article on this here)
• Agile never got the support it needed. This is a hugely loaded statement, and forgive me; I am generalising. But it is true. For Agile to really work a complete organisational change in mindset is needed - making the development team Agile is just not good enough. I’m talking about things like bad/non-existing product strategy, the continued existence of middle management (i.e. the lack of self-organising teams and the lack of trust required to have self-organising teams), the non-participation of senior management, and most importantly, the continued mindset of project over product.
• Agile became stale. All the previous points could be argued away or worked around, but this one is the essence of the problem. But I should probably state it like this: “some Agile implementations became anti-patterns”. The Agile Manifesto says we value “individuals and interactions over processes and tools”, yet the dogmatic approach practiced by “evangelists” concerning their specific way of doing things is driving a wedge into the development community. While all the above problems/challenges still exist, we seem to be stuck in an existential crisis as we are more concerned with promoting the “right way” of doing Agile as opposed to being Agile and solving the problems.

OK, enough about the problems. But before looking at solutions, I need to be clear on two things:

• I still believe in Agile. But I need to be specific: I believe in the Agile Manifesto and the Principles behind it. I believe in being Agile, and practicing Agile as a philosophy, as opposed to doing Agile as a methodology. In other words: I’ve lost my faith in the multitude of processes and methodologies (i.e. Scrum and everything that derived from it or are trying to compete with it) that profess to be Agile.
• Following from this, I believe the solution is in evolution. When Agile became the new thing, it was a revolutionary challenge and replacement of the waterfall process. Since I still believe in Agile, I’m not suggesting a similar revolution to something new, but instead an evolution of an almost 20 year old concept into a fresh new way of doing it, as influenced by both technology and other Agile-associated ideas/methods/tools/philosophies such as Lean, Kanban, System Thinking, etc.

Let’s start by looking at a few alternatives that are attempting to address a solution to these problems.

Evolution of Agile processes

There seems to be a plethora of alternatives, extensions or redefinitions to Agile. A few are listed below. Follow the links for more detail.

Beyond Agile

Kent Beck, one of the original signatories of the Agile Manifesto, proposed a next step in the Agile evolution (https://beyondagilemanifesto.org/):

• Team vision and discipline over individuals and interactions (over processes and tools)
• Validated learning over working software (over comprehensive documentation)
• Customer discovery over customer collaboration (over contract negotiation)
• Initiating change over responding to change (over following a plan)

(Note: I could not verify that this actually came from Kent Beck, but several websites credit him.)

The Heart of Agile

Another signatory of the Agile Manifesto, Dr Alistair Cockburn, says: “Agile has become overly decorated. Let’s scrape away those decorations for a minute, and get back to the center of Agile”. Following from this, he emphasizes four words:

• Collaborate
• Deliver
• Reflect
• Improve

(https://heartofagile.com/)

Async software development

The “Manifesto for Async Software Development” is here: https://asyncmanifesto.org/. In short, it says:

• Modern tools and flexible work environments over meetings and office hours
• Flexibility in prioritization over detailed planning
• Comprehensive documentation over tribal knowledge

Modern Agile

From https://modernagile.org/:

• Make People Awesome
• Make Safety a Prerequisite
• Experiment & Learn Rapidly
• Deliver Value Continuously

Scaled Agile/SAFe

SAFe’s Core Values:

• Alignment
• Built-in quality
• Transparency
• Program execution

(https://www.scaledagileframework.com/safe-core-values/)

Learning from terrorists

These things scare me. I remember growing up in the 70s, when a new terrorist group declared itself every so often in Europe, Africa or the Middle East, and they did so with an attack on the establishment and the release of a manifesto. This reminds me of those times. But I get it. The people doing this are attempting to distinguish themselves from those doing it in the same old way. And I think therein lies the rub: all the above are still Agile, just different. And doing it in the same old way is also still Agile, just same old.

So what can and should we take from these new manifestos?

The Ugly

Modern Agile

Maybe it’s just me, but looking at their website makes me think this is just a gimmick designed to sell stuff.

Scaled Agile/SAFe

This one just gets to me; it embodies everything that is wrong with Agile. Let me explain:

• It is an Agile anti-pattern. It’s like “individuals and interactions over processes and tools” got extended with “and we have mandatory processes and tools to control how those individuals (we prefer the term ‘resources’) interact”. (credit to Manifesto for Half-Arsed Agile Software Development)
• It is a command and control system that is about output instead of outcome, about project focus over product focus.
• And finally, the website tells me that it is mostly about training and certification, i.e. selling stuff.

The Bad

Here I’m going to focus on the Async Software Development manifesto. Don’t get me wrong: I think they have a few good ideas, but there are two things that I strongly disagree with:

• When reading their website (not just the three bullet points quoted above), it feels like an outright attack on Scrum. And although I agree in my heart, it feels like a dogmatic attack. It reminds me of the atheist attacking the Christian with an almost religious fervour.
• Following this process will make planning much harder, but more importantly, it will almost certainly skip proper design: because of the lack of planning meetings, it will expect the developer to do the architect and designer’s jobs.

The Good

It feels like there is a common theme to the above five “manifestos”:

• Individuals, teams and teamwork
• Experimentation, learning, reflection, improvement
• Deliver value and quality, customer discovery

There is also the unwritten point:

• Don’t be dogmatic

I’m going to summarise the above four points as follows:

• Focus on people and how they collaborate
• Learn through experimentation
• Continuously improve how you work
• Deliver value to satisfy the customer’s needs
• Don’t be dogmatic

Yeah, I know: it sounds like the beginnings of another manifesto, but the goal is to learn and to discover how to improve.

So, what should we do?

Let’s investigate these points further:

Focus on people and how they collaborate

Sir Richard Branson said: “Clients do not come first. Employees come first. If you take care of your employees, they will take care of the clients.”

I’m not sure if I agree with this; I think clients do come first, after all, they pay the bills, but employees should be a close second, or shared first place. Why? Because that is how you attract and retain the best talent. And the best talent delivers the best value.

I’m a fan of Jurgen Appelo’s Management 3.0 and Managing for Happiness. According to him everyone is responsible for contributing to the success of the organization. These books are dedicated to cultivating a workplace that enables, encourages and rewards innovation, productivity, self-managing teams and employee empowerment. To me one of the more important aspects here is the cultivation of a shared team (and organisational?) vision. And it is quite obvious that high levels of teamwork are also at play here.

It is all well and good to talk about valuing teamwork and collaboration, but there is an important problem here: the globally distributed workforce. Face-to-face communications is just not that viable anymore. We have teams distributed over continents and time zones, and “working from home” is becoming something that cannot be ignored or rejected. How do we maintain a healthy team in this scenario? There is an abundance of views, ideas and research on this, but I think Adel Taweel got it right at the International Workshop on Distributed Software Development conference. See here for the full paper. His list of things to do for successful distributed software development:

• Use suitable collaboration tools. (Including wiki pages, discussion forums, video and teleconferences tools, email and shared repositories.)
• Establish working relationships through face-to-face meetings at the beginning of the project.
• Precise and complete documentation, including requirement specifications, design method and symbols and coding conventions. (Suitable collaboration tools come in handy here.)
• Flexible planning, clear objectives and work schedules. With a special mention of self-managing teams.
• Defined sets of interfaces and development methods and tools.

In summary:

• Enable and encourage people to be innovative.
• Enable and encourage self-management and self-managing teams.
• Share the vision (product vision, team vision, organisational vision).
• If applicable, enable effective remote collaboration.
• Focus on people’s outputs (results) instead of inputs (time spent in the office).

Learn through experimentation

"I never lose. I either win or I learn." - Nelson Mandela

In the world of software development, Rapid Prototyping is a relatively old concept; it predates Agile by more than a decade. Lately, the concept was rebranded to Experimental Learning and it is becoming hugely popular in several aspects of software development:

Solution exploration

Extreme Programming (XP) introduced the “spike”; it is used to experimentally determine how much effort is required to address a specific problem and involves evaluating several alternatives, gathering additional information and/or testing edge cases. Through experimentation we learn which solution works best, and we fail fast with unsuccessful solutions (we win or learn).

Product Development

Product development and management is covered in more detail later, but experimentation is an important aspect thereof. By involving customers and users in product development and presenting them with several possible solutions (A/B Testing) and/or beta releases, we learn what works; which product, which concept, which user interface, or just about anything. And again, we learn what doesn’t work.

Process and best practice

This concept also applies to how we work: what process do we follow, what tools do we use, and what best practices do we implement? We should experiment with this as well: we should test different ideas, processes, tools and practices. Try something new for a predetermined time, see how it works, and then make it permanent, abandon it, or modify it and try again. This could apply to what repository we use, how we ensure quality, or even how “we do agile”.

What is needed to achieve this?

Simple: An environment where it is safe to fail. But only while we learn from it.

Continuously improve how you work

This follows from the previous point: experimenting with process and best practice. But now the actual focus is not to learn, but to improve how we work. The Scrum practice of retrospectives sets the tone here: we should continually evaluate how we work by asking ourselves what went well and what can be improved. And then we need to take action to implement improvements.

But we need to do more than what Scrum tells us to do: we need to experiment with everything, we even need to experiment if something other than Scrum works better, and take action if it does.

We need to investigate and experiment to implement different approaches and combinations of approaches. In the end we need to find what works best for the combination of organisation, team, project and customer. But even if we found what we think works best, we should continue exploring, experimenting, evolving and improving.

I can easily imagine following a process that combines Agile, Lean, Systems Thinking, Product Management and Lean Startup principles into a single systematic approach tailored for a specific team.

What is needed to achieve this?

• Regular retrospectives. Not at the end of each Sprint (because we may not have Sprints), but at some regular interval, say monthly.
• During these retrospectives we need to critically evaluate ourselves, innovatively imagine new ways of doing things, plan the experiment (who, what, how and for how long), assign tasks and work the plan.
• Rinse and repeat.

Deliver value to satisfy the customer’s needs

This requires a radical change from a “project mindset” to a “product mindset”.

In a Project it is all about delivering on time, on budget and on specification. This works well if projects earn money, like defence and government contracts or enterprise projects. But in most other cases a project earns money as a consequence of something else; when end users are spending money. Doesn’t it make more sense than to focus instead on satisfying the customer? And we satisfy the customer by providing a product that satisfies their needs. Enter Product Management. This doesn’t mean we should not be concerned with delivering on time, on budget and on specification; it means we should be more concerned with satisfying the customer.

In Scrum there is a Product Owner, and in XP the customer (or a representative) must always be available to the development team. In both cases these roles devolved into something that was not the original intent; they simply became the guardians of the backlog by being responsible for generating tasks or user stories and prioritising them. This is not a bad thing, but another level of product management and customer interaction is required.

Here are a few things that are essential to the product mindset. Note that each bullet point leads to the next one.

• Teams should not serve the needs of the business, but rather serve the needs of the customer within the context of the business.
• Instead of “gathering requirements from stakeholders”, we should work with customers and users, through discovery and experimentation, to determine their needs: what do they need, how should it work, where should the focus be? It is essential to understand that our initial ideas and plans may be wrong, and we need to be open to allow customer and market research to dictate the direction to take.
• We should have product managers and UX designers instead of project managers, architects and business analysts.
• We need multi-skilled teams. A development team should not be limited to developers and testers, but the product managers, UX designers, UI designers and DevOps engineers should also be part of the team. (As opposed to being part of a PMO, an analysis and design team, and an IT support team).
• Organisational change is essential:

1. Product, marketing and development should be on the same branch of the organisational chart.
2. The entire business should be product-focussed and agile, not just the development team.
3. Both accountability and funding should be based on outcome instead of project.

Much of the content in this paragraph was inspired by Marty Cagan’s article. It is worth reading.

Don’t be dogmatic

The first point in the Agile Manifesto is “Individuals and interactions over processes and tools”.

This point was probably the single most important factor that led to Scrum and other agile-based methods supplanting the Waterfall process. Ironically it is also the single most damning piece of evidence against Scrum, SAFe and other dogmatic agile processes.

I think the multitude of post-agile manifestos and alternatives are, at their hearts, not truly meant to be alternatives or even improvements to Agile, but rather lashing out against stale and dogmatic approaches. This is part of the bigger problem: the old versus the new, this approach versus that approach, the dogmatist versus the maverick.

Instead, we should realise what every tradesman knows: don’t use a hammer to drive screws in (unless it is a hammer screw). We should use the right tools for the job; there is no playbook. Instead of focussing on process, we should focus on success, and select a process that will help us to achieve success. And, as discussed before, through experimentation we should continuously improve the process that works for us.

Here are a few pointers:

• Your process should work for your team, your organisation, your project, and your customer. If needed, it should be tailored to work.
• Don’t be locked in. Change the process when needed, and always improve it.
• And stop criticising those following another process or variation of your process.

Conclusion

Here is a summary of what I said, in easy to read list.

Focus on people and how they collaborate

• Enable and encourage people to be innovative.
• Enable and encourage self-management and self-managing teams.
• Share the vision (product vision, team vision, organisational vision).
• If applicable, enable effective remote collaboration.
• Focus on people’s outputs (results) instead of inputs (time spent in the office).

Learn through experimentation

• In solutions, products and process and best practice.
• Implement a safe to fail environment (but only while we learn from it).

Continuously improve how you work

• Regular retrospectives.
• Evaluate what we do, imagine and experiment with new ways of doing.

Deliver value to satisfy the customer’s needs

• Teams should serve the needs of the customer within the context of the business.
• We should work with customers and users, through discovery and experimentation, to determine their needs, and allow research to dictate the direction.
• We should have product managers and UX designers instead of project managers, architects and business analysts.
• We need multi-skilled teams.
• Organisational change to support product-focussed development.

Don’t be dogmatic

• Process should support what we do.
• Always improve the process.
• Stop criticising other processes.

This is not my manifesto; this is the key elements of how I see Agile should be practiced today. But more importantly: we need to realise that Agile is a philosophy: we need to be Agile, rather than trying to do Agile.

And the next time an old school project manager tells me his team is agile because they have stand-ups and a Kanban board, I’ll show him this.