Agile Is Not Enough: Part 2

In a previous post, I chronicled the dysfunction that exists in the Agile and Scrum communities, why DevOps came to be, and how real Agile — not what is proselytized by the Scrum community— is powerful but is incomplete, and needs other ideas like DevOps to work well in large organizations.

It is almost a joke in the Agile community that if Agile is not working for you, then you must “not be doing it right” — that is, Agile can never be to blame. But what is Agile? It depends who you ask. Most proponents of Agile will point to agilemanifesto.org — a website created by a group of experienced IT folks who were fed up with how software was being approached, and who wrote down a set of ideas to set things on a better path. It was — is — a brilliant document. But as it turns out, very few Agile projects actually follow the Agile Manifesto. Instead, they adhere to specific methodologies such as “Scrum” — assuming that such approaches must be Agile since they are sold as such — yes, “sold”, because Scrum is a commercially marketed process.

Very few Agile projects actually follow the Agile Manifesto. Instead, they adhere to specific methodologies such as “Scrum”.

As a result, not too many practitioners in the Agile community actually question what they are doing — but instead blindly follow what Scrum says, as well as following the advice of a handful of other thought leaders in the Agile community — few of whom will challenge Scrum since it is so entrenched. In fact, there are a whole suite of entrenched ideas that cannot be challenged without peril — yet, most of these ideas are very destructive if implemented in their extreme form, which is often the case, because as Alan Kay has said, IT is a pop culture, and pop cultures often subscribe to the idea that more is better — whatever the latest fad is. Judgment goes out the window. Thinking is not a factor in pop culture — following the crowd is.

Let’s look at some of the sacred ideas in the Agile community and see how they hold up. Mind you — none of these are stated by the Agile Manifesto, yet these ideas are as much part of today’s Agile as the Manifesto is, because they are so embedded in the culture.

Everyone should be together in the same room

This originates from the Manifesto’s statement that,

“The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”

Yet, the Agile community has taken this to an extreme, and today an “Agile team room” is considered to be inextricably linked with Agile, so that if you are Agile then you must have an Agile team room — with everyone all sitting together and no walls.

How well does this work? There is research that shows that putting people in an open space makes them substantially less collaborative, not more. And try this: search the Web for “open plan office” and count the number of articles that push back on the team room idea. The book Deep Work by Cal Newport has been on the New York Times bestseller list. In that book Newport explains why sitting together — compulsory collaboration — keeps people from being able to think deeply. Thinking deeply is necessary for breakthrough insights, and so while sitting together might be enjoyable — it is social — it comes at the potential cost of losing our best ideas. In short, it threatens breakthrough innovation.

There is research that shows that putting people in an open space makes them substantially less collaborative, not more.

Self organization, and “trust the team”

This originates from the Manifesto’s statement,

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

Somehow, the Agile community — especially the Scrum community — has translated this into an extreme, whereby team leadership is seen as hands-off, where the team should always make every decision and originate every idea. The team lead is reduced to a mere facilitator. Is this best?

In Walter Isaacson’s recent book The Innovators, he paints a very different picture of what kinds of teams tend to work well and produce innovation. The teams that he describes, which include diverse domains from the development of the hydrogen bomb to the leadership of Intel, are teams that are able to complement each other and that are able to engage in dialectic discussion. The right team mix seems to depend itself on the mix. Very often there is a combination of leaders: someone who can relate to those beyond the team, someone who relates to people within the team, and someone who keeps things running well. But in essentially all cases described in the book, leadership entailed deep knowledge of the subject matter. I did not see a single example of a team lead who was a mere facilitator.

Indeed, a recent Harvard Business Review article claimed that harmony on a team is an indicator of low creativity; which implies that disharmony is necessary to a degree when creative exchanges are occurring. But disharmony is dangerous, in that it can destroy the team. Thus, the oft-cited Tuckman model of “storming, norming, performing” is an ideal and does not actually reflect what goes on in a creative team. Unfortunately, actual authority is sometimes needed to be able to intervene and address inter-personal problems on a team, as well as personnel issues.

Absolute trust in the team is also problematic for security considerations. Recently Tesla discovered that an employee had been sabotaging code. Security is really important — a major security incident can destroy a company. Also, people don’t know what they don’t know, and most programmers don’t know much about security, which can lead to terrible mistakes, and a security expert is needed when developing code that protects critical resources. There are ways to do this in an Agile manner: my book High-Assurance Design is about that and rejects the traditional security review in favor of a design pattern approach, which is much more compatible with Agile. This means that collaboration with and learning from experts is the right model — not complete autonomy by team members. It might even mean, in some cases, that only certain people can modify certain code modules. Scrum’s “generalizing specialist” is an ideal, but not practical for complex systems, and that fact needs to be incorporated into the team models that are used for creating security sensitive applications — which are most.

Games as a way to teach

Everyone likes games, right? So games must be a great way to engage a room full of people.

Actually, my wife and I once had some friends over and my wife — who likes games — tried to get everyone to play a game called Crimes Against Humanity. I groaned, and so did my friend Kevin. Kevin does not like games, and I don’t either. We were outvoted, but it made me realize that I am OK for not liking games — there are others like me.

Games are a great learning tool for some people, but not for everyone! We are all different. And even for those who enjoy games, it is not always the best tool.

The purpose of a game-based instruction is that it enables the participants to experience a situation in a tangible way — they live it. However, my wife is a behavioral therapist and she explained to me that for experiential learning to be effective, it is crucial that the context be the same as the context in which the new skill will actually be applied. Thus, for example, a game in which someone cooperates with other players to build a tower of cards does not effectively teach someone to cooperate with others in a very different task such as writing software. The reason is that the contexts are so different that people do not automatically translate the experience from one domain to the other. Thus, games can be entirely ineffective if they do not simulate the specific context in which people are expected to later apply the new knowledge.

What works much better is simulation — going through the very motions that they will use, in a simulation of the setting in which they will later be applying the knowledge — the more realistic the simulation, the better.

Yet games have taken hold as a popular tool for teaching people about Agile, and Agile team coaches all nod their heads when hearing about the use of games for this purpose, when instead they should be shaking their heads.

Dot voting

A colleague of mine recently participated in a management planning session with the CEO of a several thousand person health care organization. The session was facilitated. Afterwards, the colleague told me that it was very ineffective — that she would have greatly preferred to have deep discussion about the issues, either one-on-one or in a very small group with the CEO. Instead, the participants voted on issues and discussed them around a table, and then dot voted on strategies — ending up with a list of strategies that, my colleague claims — is disconnected from the reasoning that went into reaching each strategy.

I have been in many sessions like that, and have always felt the same way afterwards; yet this form of “collaboration” is accepted as a norm in the Agile community. What seems to occur much less frequently is the old form of collaboration, where people discuss things in a logical manner around a table, perhaps with a knowledgeable team lead provoking questions, offering ideas of his or her own, and helping the team to talk things through and reach consensus. That was the approach that Socrates used, and I have always found it to work really well — much better than dot voting, which seems to replace logic with popularity.

An alternative idea

Agile is indeed broken because people are “doing it wrong”, but things have gone so far — there are so many entrenched practices that are simply wrong — that I find that Agile is no longer a useful paradigm for success — we have strayed too far from the original ideas in the Agile Manifesto, by having embraced so many extreme practices that defy common sense. The Agile Manifesto was about balance — not extremes.

Instead, I offer an alternative idea: that we treat Agile as just a set of ideas — ideas that are not necessarily complete or right for our situation — and that we go back to using our brains — which was the intent of Agile in the beginning — and strive to think holistically — consider the whole picture before proposing any strategy or practice; that we never pull a practice out of our pocket and apply it automatically or mindlessly — that every practice be the result of consideration of the whole — the organization, the individuals, the challenges that are faced, the short term and long term goals, and the resources at our disposal. At the same time, don’t try to “boil the ocean”. One of the messages of both original Agile and Lean Startup is that one should try small experiments — not try to create a perfect product or solve every problem in the first attempt. Accept that you don’t know what you don’t know.

This might sound like a cop-out — no better than the Agile community’s “don’t do Agile, be Agile” — so in another post I will offer some actual alternative practices to consider — but only to consider, because there can be no escape from choosing your own way. There is no approach that works for all cases.

(Third and final post in this series is here.)