Handling the Crossover Point between Requirements and Design Work

Summary

Backlog grooming may involve crossing over from the “what I want” mindset into the “how to do it” mindset. This is often necessary to boil out the Backlog contents, discovering what needs to be done. But it is risky, and can lock a system into a design too early, without including all the right people. The secret is to be aware when we are moving into “design think”, understand why we are doing it, and stopping ourselves before we encroach too far across the line.

Requirements, Features, and Stories

Understanding how to navigate the dividing line between requirements (what is needed) and design (how to do it) is not an agile-specific issue. It was addressed in ISPI’s “Requirements Engineering” courses in the 1990’s, and in industry literature long before then. Now, we tend to refer to requirements as “Features” and “Stories,” but the question of how deeply we should define a story is the same one as when we call them “requirements.”

In conventional systems development, we realized that having one-directional progression between requirement and design activities is not realistic. Agile methods are more explicit about this being a bad idea. While requirements/stories guide design, it’s not a one-way flow of information. Design work also results in discovery that helps us refine our Features and Stories.  

Defining Features and Stories means identifying a combination of things we want the system to do, and things that constrain how we can do it. Sometimes these constraints are iron-clad, defined by law, policy, or financial constraints that can’t be budged. Sometimes they evolve from knowledge of the technical environment and limits on how solutions can be implemented. 

Systems thinking often involves stepping across the “what versus how” line to discover and evaluate perceived constraints. This risks leaping to design, so be aware when the conceptual divide has been crossed, learn what needs to be learned (via a prototype, proof-of-concept, or other activity) and then step back to the “What” mindset to define the learning within your User Stories. In a strong, dynamic team this line will be crossed and re-crossed on a frequent cadence.  

The “three amigos[1]” Backlog Grooming concept means that those with extensive “what” knowledge (Product Owners) frequently meet with those having extensive “how to do it” knowledge (developers and SMEs) and “how to prove it” knowledge (test designers.) This collection of skills allows User Story decomposition to navigate the what/how dividing line without straying across too far and for too long.

The second type of constraints mentioned above are those based on existing system knowledge. They can be dangerous. Often such constraints come from a precursory exploration of the “problem space,” followed by leaping so solutions. Such leaps give a comforting feeling that the way forward is known, but can limit our willingness to consider other options when designing and developing a solution. A clear danger sign is when a Product Backlog is filled with stories that read like design constraints. It could mean one of several situations have happened:

1.     Sponsors and Product Owners have been doing “armchair designer” work, possibly without involvement of the development team. The fine line between requirements and design has been crossed without looking back. (We’ll discuss this a bit more, farther on.)

2.     Big Design Up Front (BDUF) has taken over, which risks emotional and financial attachment to decisions that probably are subject to change.  This can happen when Product Owners abdicate their role of defining business need and value, leaving it to the developers to "wing it."

3.     As more is learned, the make-versus-buy analysis (which shouldn't be a one-time thing) is shifting strongly to using existing components. This may not be entirely bad -- sometimes "buy" or assembling existing components is the right way to go.

4.     Preconceived notions are being pushed for political or other reasons. Addressing that topic is figuratively opening a whole 'nuther can of snails; we won't "go there" in this article.

Acknowledge Constraints, Refine Through Recursion

BDUF risks high investment in work that is subject to change. We don’t want the backlog full of design constraints. But opposing too much design up front often causes a fanatical opposition to any design up front (NDUF); such agile purism stops any design discussion in its tracks during backlog grooming. As we’ve said before, that approach is restrictive. Do avoid having a backlog filled with design constraints, but don’t stamp out all design discussions during story elaboration. Build the recursive relationship between “what” thinking and “how” thinking.

In Agile Software development, the recursive aspects of needs definition (which was used in the Modified Waterfall, spiral, and other pre-manifesto approaches) is made a constant behavior. Evaluation activities happen frequently, with participation required by key players. This shortens timelines by lessening the “information float” between figuring out what I want, and whether I am limited in how I can get it. Done well, it limits time lost on defining Features and Stories, and may limit the number of times they have to be revisited before and during their development.

A Contract Scenario

Software work sometimes takes place in contract situations, wherein the sponsoring organization doesn’t have software engineers as direct employees. This sort of situation was the cradle in which the Waterfall software life cycle was born. (By the way, the first descriptions of Waterfall by Winston Royce were instructions on how to make the best of it in DoD contract situations; they were not promoting it.) 

In contract situations, the project sponsors often will attempt to make good use of time by specifying the system as much as possible. This can be false economy. Significant backlog grooming before the systems development team is present poses some risks:

·       “Specification fatigue”

·       Design lock-in

·       Impatience with developers (“Get out of your analysis paralysis and just do it the way we specified!”)

When the sponsorship/product ownership team is highly technical, that can mitigate some risks but create additional ones.

·       There is a tendency to specify things based on deep knowledge of existing systems and constraints. Some system aspects may appear to be “immovable objects” and lead to story descriptions and prioritization that appears obvious. This is the root of “throwing requirements over the wall” behavior (described by Rob England as the Dead Cat Syndrome.[2]) The Product Owner team and Subject Matter Experts should be flexible in their thinking, and cautious about creating design constraints.

·       The PO team brings experience with the operational environment in which the new system will run. They may be familiar with its volatility, constraints, and peculiarities. This provides understanding of how to ask for data or external processing that is needed. The risk is overly specifying where data will be and how to access it; those may be subject to change. 

Mitigating Activities

·       If the Development team is not yet onboard, slow the pace of story definition. This will address the temptation to justify backlog grooming time by overly specifying stories.

·       Discuss existing systems and how they function in order to inform the process and stories. View all resulting stories with caution; note the possibility that systems specifics are by way of example and are subject to change.

·       Stories/requirements that deal with information handoffs and interfaces represent higher risk. Touch on these early, revisit them often, and tag them to be revisited throughout the iterations.

·       Don’t lock in interface and information handoff-related stories early, however. Commit to them at the latest responsible moment to mitigate the risk of changes to these critical system aspects.

[1] https://www.agileconnection.com/article/three-amigos-strategy-developing-user-stories

[2] https://www.itskeptic.org/dead-cat-syndrome