Misconceptions about Agile

On a recent application development project, I worked with a product manager who, whenever I attempted to think ahead to the pitfalls I foresaw or delve into details about requirements, accused me of “being waterfall.” This product manager stubbornly clung to the notion that “being agile” means do it fast and keep it surface-level.

Writing code requires details: you can’t really write vague code. Either the product team defines the details or the developer makes assumptions about them — escaping the need for details is not an option.

Likewise, whether a project is agile or not, planning is necessary. Anticipating risks is necessary. And getting into details is necessary for both of these things.

Oversimplifying the agile manifesto

Where do these misconceptions about agile come from? I believe the culprit is the agile manifesto. Not the manifesto itself, per se, but the natural inclination to latch on to the simplest explanation of a complex topic and lose sight of crucial context and nuance.

The agile manifesto basically divides concepts into two columns, with this statement: “while there is value in the items on the right, we value the items on the left more.” Unfortunately, the typical fast take is to focus exclusively on the left-hand items and claim that the right-hand items are irrelevant in an agile world.

These three elements of the agile manifesto are at the heart of misinterpreting agile: planning, processes, and documentation.

Planning

“Responding to change [left-hand column]?over following a plan [right-hand column]”

In the project I referenced at the top of this article, one huge risk I wanted to address upfront was where the application would be getting its data.

Building a data-centric application requires detailed analysis of the data needs, especially in a large organization with hundreds of interrelated applications and databases (both legacy and modernized versions) sprung up over the course of many decades. Given the time-consuming effort of identifying potential source systems, analyzing and modeling the data, learning the data’s limitations and dependencies, and setting up integration methods (including possibly going through lengthy business processes for getting the data into a data lake or data warehouse), you want to do this analysis as early as possible.

But this product manager insisted on relying upon one source system based strictly on that system’s team assuring us it had all the data elements we needed, after a discussion that never went beyond high-level. Unfortunately, the project team discovered too late that this system did not have certain critical data elements, which ended up rendering the MVP release useless for the targeted users.

This product manager is not alone in believing that agile means not having anything more than a very high-level plan. But think about the wording that the agile manifesto uses: “responding to change.” Implied in this statement is the existence of a baseline — what is “change” if not a difference from a plan? And how can you define that “change” if your plan is defined at a very high level?

Planning means thinking through potential implications and deciding which risks merit mitigation plans. It means having a good enough understanding of the overall solution, requirements, and key deadlines to know the project’s dependencies and priorities, which gives you the judgment needed to decide what to focus on with your first sprints.

In agile, the point is to not allow a plan to remain static, but rather be flexible knowing that change is inevitable. As I’ll reiterate with processes and documentation in the next sections, the tyranny of planning is what must be avoided, as well as the tendency to make detailed plans too far ahead of time.

Good planning itself is not optional for any type of project.

Processes

“Individuals and interactions [left-hand column] over processes and tools [right-hand column]”

Though the waterfall model is still predominant in some industries, agile has become so ubiquitous that many people don’t know (or don’t remember) what the pre-agile world was like. The pre-agile context is crucial to understanding the agile manifesto because that’s the construct the agile model was designed to challenge.

The antipathy to process derives from the tyranny of process in the waterfall model.

Process is what keeps the software development lifecycle and related activities running smoothly. Process is how you keep obstacles from making the entire project grind to a halt. Process is how you avoid time-killing uncertainty and promote consistent results.

But not when following a process becomes a goal unto itself.

A good process is outcome-driven; a well-functioning team is dedicated to regularly reevaluating whether a process is facilitating or impeding the desired outcome and will improve the process whenever needed. When the organizational culture makes continuous improvement to processes difficult or impossible, the results are predictable and frustrating.

If the desired outcome is quick turnaround for functionality to get stakeholders’ input early (a key reason for going agile) and the process is getting in the way of that outcome, then the process must change. Or if the process is blocking the team from achieving quality metrics, gaining necessary access or knowledge, or whatever other outcome is desired: the process must change.

It’s not that we should ignore or eliminate processes, but that we must make process subordinate to and supportive of the desired outcome.

Documentation

“Working software [left-hand column] over comprehensive documentation [right-hand column]”

As someone whose first role in tech was as a documentation specialist, I am aware that not everyone approaches documentation with the same relish I do. Most people find it tedious and will find any excuse to avoid doing it.

Yes, of course, the point of a development project is to produce working software, not documentation, as the agile manifesto states. However, we must not discount the importance of good documentation.

Looking back again to the pre-agile world, we see another tyranny closely linked to that of process: the requirement that exhaustive documentation be produced before a single line of code is written. Anyone who has ever worked on a complex software development project knows how nonsensical it is to think that a system can be developed in exact accordance with a 3” stack of predefined specs with no changes needed. Software development is not nearly as predictable as those far removed from the code believe. The problem is that once the 3” stack of specs has been produced, making any changes to the documentation is quite burdensome, forcing the development team to forego innovations or, when the specs are simply undoable, wait out a lengthy change request process. Hence the need for an agile approach.

Documentation is critical both in the short term and the long:

• Now: Written artifacts are infinitely more reliable than conversations to confirm requirements with stakeholders, ensure understanding of the requirements among the team, and assist with scope and prioritization discussions.
• Future: Years from now, when everyone’s memory is a little fuzzy, you’ll want to have good documentation for the purposes of troubleshooting, upgrading, and training.

However, an intelligent approach to documentation is necessary to avoid that burdensome process of making changes as the team’s knowledge grows and desired approach changes (not to mention the typical evolution of requirements from stakeholders).

In an agile world, you want to take a just-in-time approach: get the right level of detail for what you’re working on now and immediately after. The level of detail will depend on the complexity of the functionality, the development team’s familiarity with the product and functionality, and other factors.

But you do not want to take the approach of making documentation a secondary priority that might or might not ever get done.

Your turn

Do you agree or disagree with my take on processes, documentation, and planning? What misconceptions about agile do you see repeatedly? Please share in the comments!