Agile Is Not Enough, Part 3

Following Part 2, this is my final post in the three part series, Agile Is Not Enough. After criticizing the Agile community and the dysfunction that exists in it (sorry, but change is hard!), proposing instead that we stop viewing Scrum and Agile as a set of rigid practices, but instead free ourselves from those constraints and think “out of the box”, I had promised to provide some actual ideas beyond platitudes. Here are my ideas. Remember, these are not proposed as replacements for current practices, which—as I have explained at length—have become a kind of dogma. I don’t want to create a new dogma to replace the old. Instead, these are approaches that have worked for me, and I merely offer them as approaches to consider. My best advice remains, think for yourself.

Instead of the open team room:

Instead of an Agile team room layout, consider a mixed coffeehouse layout. That is, don’t assign desks at all, and don’t force everyone into the same room. Instead, provide lots of places for people to work—that’s what I mean by “mixed”: there are a variety of spaces, some that are quiet, some that are social. There should be enough space so that it is not crowded: crowed equals noise and distraction. Have carpets and surfaces that absorb sound, and have big windows: windows are important because people don’t want to feel like they are locked away from the world.

People don’t want to feel like they are locked away from the world while they work.

In a mixed coffeehouse layout, people will gravitate to where they work best on a given day. The coffeehouse layout, with an actual coffee bar, is reminiscent of a social scene and is more fun, and people will be more enthusiastic about their work at a result—they will actually enjoy being there! And don’t crunch people in: space is really important for focus.

If your CFO complains that the density is not high enough, ask her if she remembers how much she is paying these folks, and if she wants them to be productive.

Scrum enthusiasts will complain that in a coffeehouse layout, people won’t overhear conversations. Well yes, that’s the point. When I have been in team rooms, I recall so often stopping what I was doing, with my hands on the keyboard, pretending to work, but really I was waiting for a nearby conversation to stop.

People will still hear things in a coffeehouse setup: because they move around, they mingle. There are conversations. But there is enough space to feel somewhat private but still not isolated, and there should be enough soft surfaces that the ambient noise is low.

Instead of the Scrum Master role:

Choose team leads who have a natural servant leadership style. Give them authority, but also give them training in servant leadership. Teach them about conflict resolution, and about the Socratic style of discussion. Teach them about forms of leadership, and leadership models. (An overview of some applicable models of leadership is here.) Emphasize that as leaders they can have ideas too—that offering an idea does not silence the team if you present it as just another idea candidate and demand that it be challenged. Contrary to what the Scrum community claims, a team lead should not be just a facilitator: the lead should also be a mentor and a source of valuable experience. What the lead should not be is dominant, or the person who fixes everything—those traits are toxic. Teach them also about externally facing leadership, and how to advocate for a team in an organization. Learn what servant leadership really is—not what the Scrum community says that it is.

Learn what servant leadership really is—not what the Scrum community says that it is.

Instead of the team-centric view:

Think holistically, end-to-end. This is a DevOps principle. In a large organization, a team is not self-contained. It relies on others outside the team. If that were not the case, then an organization would merely be a collection of teams. Instead, adopt the DevOps approach of viewing the organization as a system, consisting of teams that work together. In the DevOps view, there is a “pipeline” that begins with an idea, progresses through implementation and deployment of the idea, followed by validation of the idea and feedback on whether the idea was a good one. Thinking of it that way, it is possible to shorten the time—the “lead time” (a Lean idea)—from idea to feedback on the idea.

Instead of imposing test-driven development:

In the Agile community, the phrase “test-driven development” (TDD) is very specific: it is not a general term. It is not related to Acceptance Test-Driven Development (ATDD) or Behavior-Driven Development (BDD). TDD refers to a software development practice whereby one creates very low level tests, internal to the code, to validate each low level piece of code. There is not a consensus on whether this practice is effective—some highly respected people actually claim that it is destructive and results in “test-induced damage” to the design. Most likely TDD is effective for some people, but not others: those who think inductively versus those who think reductively. The value of the practice might also hinge on the software language being used, for example, a type-unsafe language such as Ruby versus a type-safe language such as Java. In any case, don’t force people to use TDD: let them decide; but the rub is that a whole team must decide or it won’t work, and that is a strong argument for only using TDD if everyone on the team likes that process. In any case, allowing them to decide for themselves is consistent with the Agile principle of letting team members decide how to do their work.

Test-Driven Development (TDD) is not related to Acceptance Test-Driven Development (ATDD) or Behavior-Driven Development (BDD).

TDD is not an “advanced” Agile practice in the sense that those who do it are “more mature” Agilists or better programmers: TDD is merely a chosen approach and is not better or worse than other approaches.

Instead of the generalizing specialist:

Scrum teams work best when most people on a team are able to perform most of the tasks that the team does. That enables anyone to “pick the next story”, and prevents individuals, such as a DBA, from becoming a bottleneck.

The problem is, in a large and complex organization, it is not always feasible. There are often tasks pertaining to security or infrastructure that require deep expertise that took years of experience and special training to acquire. Those cannot be learned rapidly through pairing, which is the Agile model for cross-training. That means that experts are sometimes needed.

You might have heard that in some organizations, like Netflix, development teams do everything themselves—they don’t need outside experts. However, not ever organization is like Netflix: Netflix was able to achieve this because they have an immense tools team that does nothing but create self-service tools to enable dev teams to do things themselves. Most organizations cannot afford to have a large tools team pumping out and maintaining sophisticated custom tools.

A very effective approach for accommodating experts is to cultivate centers of expertise, such as for security, test automation, release management, enterprise architecture, and then assign members of those specialization to development teams to help those teams. That way, each development team has a single individual to contact for a given special expertise, and that individual comes to learn the team’s work. Further, the specialist interacts with the team as a coach would: teaching them what they know when possible, attending the standup when their expertise is needed based on the stories that are being worked that week, and checking in with the team lead and team’s technical lead.

The people in these centers of expertise do not work from the team’s backlog: they have their own backlog. Their work for the team can be represented in the team backlog, as a dependency, but should not be defined there. This is necessary because to maintain a center of expertise, the experts need to be able to define their own work, which includes things needed to maintain the expertise, such as learning new tools, prototyping, and supporting enterprise level discussions about technology strategy. They will need to coordinate their priorities with the priorities of the teams that they support: there is no avoiding competition between various short term and long term interests in a large multi-product organization, and so there needs to be a forum for that coordination. Such a forum is not defined by any of the Agile scaling frameworks that I am aware of, so you will have to invent your own. I talk more about this below, under “Scrum of Scrums is not enough”.

Teams need technical leads:

The Scrum community strongly promotes the idea that anyone should be able to work on anything. This is a rejection of the old practices of technical silos: that certain people “own” certain modules. However, as with so many good ideas that the Agile community has had, it has been taken too far. Not everyone is qualified to work on any part of the code. There needs to be a middle ground. The horrible unreliability and insecurity of today’s applications is a partly result of not paying attention to skills sets.

Instead, code modules need to be considered with regard to their sensitivity in the overall design, and someone needs to be on point to watch for which code modules are becoming sensitive. By “sensitive”, I mean that it changes to the module might have atypically large unintended consequences, with regard to security, privacy, reliability, or maintainability. If someone needs to work on sensitive code, they should be paired with someone who is experienced with that part of the code.

For complex applications, there needs to be someone whose job it is to think end to end.

There also needs to be someone whose main job is to think end-to-end. During a sprint, people are way too focused on getting their story done: they don’t have time or mental space to be worrying about the larger end-to-end issues. Someone needs to be thinking about the overall system every day, and asking hard questions: “Will it be able to handle this case?” “—Oh, we didn’t think of that, no.” They should also be looking through people’s code to see how it really works, and—importantly—learning what the team knows (because as a whole, the team knows more than the tech lead) and how things work throughout the entire system—and also talking to architects outside the team, because in a large organization, there are ever-evolving architecture questions that affect the various teams, and teams don’t have time to be in all those discussions.

The person who has this end-to-end focus is the team’s technical lead. It does not mean that they are the only ones who think end-to-end—a good tech lead will constantly be sharing and discussing issues with the team, and encouraging and empowering the team to start their own discussions—but someone needs to be given the time and capacity to think end to end most of the time, or it will not get done—not when building complex systems.

Instead of just the standup:

Standups are important because they keep everyone focused on getting the stories done. I personally don’t like the “stand up” part—to me it feels like an interrogation. I like keeping it time-boxed, but I have been in standups where people sat, and those worked better that way, in my opinion.

The standup—whether standing or sitting—is important, but it is often not enough. This is important to realize, because most Scrum teams that I have seen assume that the standup is enough, if augmented by ad-hoc conversations that the team or Scrum Master initiate. However, it is often not enough. The ad-hoc conversations often don’t occur with sufficient frequency or depth, and a non-technical Scrum Master will not know when technical discussions need to happen but are not happening.

This is a place where the team’s tech lead comes in: the tech lead should be watching to see if more technical discussions are needed. People will often do that on their own, but often they won’t, or don’t know that they need to, because they are so focused on their story and don’t know or simply forget how some of the things others are working on function.

In one project that I was on, the tech lead decided that the work was getting very complex and people were getting bogged down in the complexity, so he scheduled a one-hour discussion every afternoon at 3pm about technical challenges, which he would facilitate. Attendance was optional. It was amazingly effective: people who were having problems would attend and talk through their technical approach. After a few weeks the sessions were not needed anymore, and the application started to work better. That particular tech lead also made it his business to talk one-on-one to each team member every day, to ask them how their stuff works and have a dialectic discussion about it. That’s not something that happens in a standup. The standup is not enough.

Scrum of Scrums is not enough:

Having a Scrum of Scrums is important: it helps to coordinate work across teams. The discussion is mainly about impediments. It is not a backlog refinement. This is very useful, but is not enough.

There should also be a regular, e.g., weekly, scheduled discussion between the tech leads of each of the teams that are working on things that affect each other. This is important for sharing and working out technical issues that span teams.

There also needs to be a regular planning adjustment, with regard to the work backlog. In the SAFe model, this regular adjustment is called Product Increment (PI) planning. That’s a great process, but there also needs to be a more frequent adjustment. This might take the form of all of the team leads and Product Owners discussing the progress of the work, and making adjustments of the many team backlogs.

Sometimes an adjustment is needed right away. Things happen: there might have been an incident that requires that some teams take on some product support work as soon as possible. If teams support their product—the so-called “No Ops” approach—they will always be receiving new tasks from the Tier 1 incident response. Such teams might have a set capacity allocated for handling break fixes, but sometimes that capacity will be exceeded, or it might be too much if things are running well.

If fixing a defect is the priority—and it can happen in the middle of a sprint—the Product Owners need to understand the issue so that they can re-prioritize, and they need the guidance of the team tech leads on the impact. It is okay to stop work on a story that has been started, if something urgent comes up, as long as the Product Owners knows the impact and agrees to it. Thus, there sometimes need to be urgently scheduled ad-hoc meetings to adjust backlogs. That’s the reality of an enterprise. Some organizations create a separate break-fix team and rotate dev team members through it, and that greatly alleviates the need for real time dev team sprint priority adjustments, but even so, it is important the the planning increments such as SAFe PI planning or whatever you use not be seen as a mini waterfall that fixes a set of features that can't change until the next planning increment. You need a lightweight process for doing multi-team backlog fine tuning along the way.

Stories don't have to be “end-to-end”:

So often I hear, “There is no end user, so how can we write a user story?” I have heard Agile coaches give a variety of very different responses to this—and that should tell you that there is no consensus about it. So let me tell you what I think.

If one is building a Web service or a microservice, there is no end user: the service is accessed by other software applications. The response of the Agile community is often that one has defined the story at the wrong place: there must be an end user somewhere, and the story should start there. However, while that might work for a simple website, it is not always practical for the complex applications that enterprises have.

For example, one system I worked on consisted of events that originated from embedded C code running on electronic control modules in machines: the events were sent over satellite links to a messaging gateway, which translated the messages into a stream that was managed by Azure Event Hubs and then processed by Hadoop, Spark, and Storm, which generated event entries in various event sourced databases, which were consumed by Web services, which in turn were called by various applications there were aggregated into a number of customer portals—one for each line of business. I would be really impressed if someone wrote an end-to-end user story that a single programming team would be able to implement.

One system used embedded C code, Azure Event Hubs, Hadoop, Spark, Storm, microservices, and Web apps using Tomcat, and Angular—creating an end-to-end story for that system was not practical—no one team would be able to implement it.

The fact is, the old paradigm, component oriented development, sometimes makes sense. It is a judgment call. Component boundaries need to be defined by contracts, the contracts maintained in source control, and regression tests maintained rigorously against those contracts. That is a very effective practice. Today there is an approach called “API-driven development”, which is a slight twist on Behavior-Driven Development (BDD), which is an approach whereby one writes acceptance level tests prior to or in parallel with the application code. This aligns very well with Agile: the tests match the story acceptance criteria. However, sometimes there is no user. That’s just a reality.

If there is no user, there is the danger that stories will not actually have business value to someone. The Agilists are right on that one. But it is a situation that must be managed. Forcing all stories to be end-to-end or have a user is not workable.

Summary

The practices that I have explained above are only options. There is no “best way” to be Agile, or to “do DevOps”. Agile correctly threw out the notion that you can create a standard repeatable process for building software—you can’t. It is too varied each time, and the choices that need to be made throughout are too contextual. Instead, Agile is about making judgment calls continually. There is no escaping that, and that means that there is no standard set of Agile practices that everyone should use.

Never assume that things should work a certain way because that is how everyone else is doing it. Think for yourself. Shun the comfort of the norm, because, as Michael Tiemann (former CTO of Red Hat) recently pointed out, there is no “normal” and those who do what everyone is doing are bound to be doing the wrong thing.