A perspective on scaling Agile

These days everyone seems to be doing Agile, from small start-ups to large blue chip companies. But what does it actually mean when they say they've moved to Agile, particularly larger organisations? Has it turned them into the next Spotify? Has it transformed their oil tankers into speed boats? Are they delivering products magnitudes faster than before? In most cases the answer is simply, no.

Scrum, one of the more well-known Agile methods, is relatively simple and quick to learn – at least the theory – and trials often yield good results so why aren’t organisations seeing the expected benefits when ‘rolling it out’ to hundreds of teams and thousands of employees across the organisation? That’s the billion dollar question, isn’t it?

There is unfortunately not a simple answer to this question, but by reflecting back on my 20 year software engineering career I hope that this post will give you some ideas and make you think different about Agile. And if I were to explain in one sentence why so many organisations struggle with Agile, particularly when trying to do it at scale, I'd say it’s because they fail to recognise that it isn’t about picking the right method, it isn’t about the latest tools and technologies, it’s all about people, culture and successfully embedding the belief system that underpins Agile into the entire organisation.

First a few words on the Agile Manifesto, given it started the movement

It was in 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, that seventeen thought leaders met to talk, ski, relax, and try to find common ground on a problem they were all facing.

They were the people behind Scrum, Extreme Programming, DSDM, Adaptive Software Development, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes that dominated the industry in the ‘90s. The outcome from that weekend was the Agile Software Development Manifesto, which consists of four core values and twelve principles that embodies the ethos of most Agile methods in use today.

Take principle #5 as an example, "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done".

This is easier said than done, particularly in large and highly regulated organisations. I don’t know how many times I’ve sat down with a new team and asked them what their biggest challenges are and “we don’t feel trusted” is right up there at the top. While trust sometimes is a real problem, it is more often down to people looking at situations from very different perspectives. Having people from Risk sitting with engineers and explain their concerns, and having the engineers find ways to address these concerns rather than simply being told what they can and cannot do, will go a long way to help build that trust.

It is well recognised that methods like Scrum, XP and Kanban require additions and adjustments for portfolio management, governance, and risk to scale beyond a single team of about 8 – 12 people. Scaling Agile isn’t a new question, in fact Martin Fowler, one of the authors behind the manifesto, wrote a post about it in 2003. In it, he makes an interesting statement:

“Scaling Agile methods is the last thing you should do”

This might raise some eye brows, but what he’s actually saying is that you shouldn’t try to scale Agile techniques up to big projects and a better approach is to try to “scale down” a project, i.e. use smaller and more focused teams. He goes on to say that that “smaller teams staffed with more able people is usually faster and cheaper, even if everyone is more individually expensive”.

Based on my own experiences I couldn't agree more with this last statement, and when it comes to solving complex problems, or turn difficult engagements around, I tend to rely on a handful of trusted and talented engineers. One such scenario was for a major government client many years ago when I assembled a small "SWAT" team to rescue a €100m delivery. With only four engineers we stabilised and optimised the application in less than three weeks resulting in operational cost savings of €1m per year - something the incumbent team of 20+ people had spent months trying to achieve.

Over the years there have been several studies where researchers have found a 10-fold difference in productivity and quality between different programmers with the same levels of experience. While you find a good summary of some of these studies in The Origin of 10x I'd say that right-sizing the team is equally important to talent when it comes to creating highly effective teams.

So does it mean that organisations used to deliver large and long-running projects and programmes can’t adopt Agile methodologies? And do they need to rely on external talent to make it work?

No, it just means that they need to think about Agile in different ways than what most mainstream Agile methods promote. And while external talent can help, it is important to build up the understanding from within if long lasting change is to be achieved. I will not try to cover everything here, but I’ll touch on a few things that I believe are fundamental in order to scale Agile techniques across a large organisation.

Method

Traditional portfolio management is focused on top-down planning with long delivery cycles, but Agile portfolio management takes the concept of build-measure-learn cycles used by individual Agile teams and applies it on a larger scale, which is where so called “scaled Agile frameworks” comes into play.

This is where you have frameworks like SAFe, or the approach used by the likes of Spotify, Google and Netflix with a mantra of “highly aligned, loosely coupled” teams.

However, as many organisations have learnt the hard way, success can’t be achieved by simply picking up and rolling out one of these frameworks, there is a lot more to it than the method itself.

A method can act as a catalyst for change, but use it as a guiderail rather than following it religiously. Use it to “test and learn” what works in your organisation, tweak it when it doesn’t and expect it to evolve over time - make it your own.

What’s common across these frameworks though is that they all centre on small, talented and autonomous teams. They focus on value delivered, not cost per head, which is exactly the point Martin was making back in 2003.

People & Culture

People build solutions - not processes, not tools, not practices - people. While adopting an Agile method can be a catalyst, it’s really the DNA and mindset within the organisation that need to change. With the right people and mindset you can achieve wonders with almost any delivery method, including Waterfall, but the method won’t get you anywhere if culture and behaviours are wrong.

Spotify, with their well-published engineering culture, alongside Facebook and Google are envied for their engineering cultures and there are many organisations that are trying to emulate their success by adopting similar approaches. The challenge is that most organisations aren’t young, homogeneous and equally fast-paced, which means embedding a similar approach is all but trivial.

Many also see Scrum as the de-facto Agile method, but it is simply one out of many, and most scaled Agile frameworks aren’t prescriptive about which method to use within a team. Scrum can be good for establishing fast feedback loops when working closely with customers, but adopting two weekly sprints for more operational teams that get called off to deal with production issues will simply not work.

So it is important that teams have some freedom around the method they use, but it is equally important not to get too focused on the rituals that the chosen method prescribes. This may sound contradictory, but in order to make a method like Scrum work effectively within a large organisation a dose of pragmatism will be required as teams very rarely operate within the autonomous environment Scum was designed for.

I’ve seen more than one Scrum Master follow the method to the letter and say all the right things, but when you probe deeper you find out that it isn’t working very well. A typical challenge is when delivery pressure mounts and the Scrum Master, without properly engaging the team, let delivery pressures trump any sound engineering principles. “We refactor later”, or “we automate our test scripts another time” are excuses used to get things out the door quicker, unfortunately this rarely delivers even short-term benefits as quality is compromised, not to mention the long term cost of building up technical debt.

Sometimes it is down to inexperience, or people simply thinking it's OK to revert back to a command and control behaviour when delivery pressure mounts. Nevertheless, it is important that everyone stays true to the expected values and behaviours, and when this doesn’t happen, acknowledge it and then work as a team to get things back on track. 

Leadership

We often hear that in order to make Agile work across an organisation we need buy-in from the top. While support from the top is an absolute necessity the bigger challenge is actually winning around “middle management”, which often consist of a hierarchy of more traditional project and programme managers that separate the executive team from the people on the ground.

The challenge is amplified by the fact that Agile methods focus on self-organisation and empowerment of teams, leaving managers sometimes wondering what their role becomes in the new world. The mindset needs to shift from a traditional command and control attitude towards strategy and guidance, becoming servant-leaders who help their teams succeed. It can be very unsettling to ‘let go’, but success will only come through empowered teams trusted to do the right thing. This also include how funding is spent, if teams are forced to create a ‘business case’ for every small tech debt reduction they quickly run out of steam as approvals get in the way of doing actual work.

It isn’t just middle management that can struggle with the idea of servant-leaders and empowered teams. I remember when I was asked by an executive, who had just found out that we’d removed hard reporting lines from the teams into myself, how I could control what they were doing. He looked very sceptical when I said I didn’t need hard lines for this, all I needed was to paint a vision that we all stood behind – he eventually got to see that a common vision that inspires is far stronger than reporting lines when it comes to building talented and high performing teams.

Reality over Rhetoric

Considering there is a fair bit of rhetoric surrounding Agile methods I thought I wrap up by recapping on a few things that I hope you will take away from this post.

Agile is not a method, it's a mindset. People often ask when I started with Agile and I get that puzzled look when I answer “I’ve always been using agile”. The term Agile may have been coined in 2001 at The Lodge, but methods like XP, Scrum and ADM, and the principles that underpins the manifesto, were things that I took influences from during the ‘90s when I started my professional career and it is something I’ve championed ever since.

I also want you to remember that while an Agile method can provide you with a good framework, it won’t transform the organisation on its own. So if you pick up one of the mainstream methods, take a pragmatic approach and don’t follow it to the letter, be ready to tweak and make it your own. 

And to change the mindset and culture within an organisation its employees need to start thinking differently. The best results often come from mixing new talent with old, and by coaching and encouraging existing colleagues to challenge the status quo.