Engineering-Driven Stories

This article describes the traditional agile approach to defining engineering backlog to deliver upon software features incrementally. It highlights some of the common problems that Agile teams encounter, and suggests a solution that still delivers on the incremental delivery promise of Agile.

The Traditional Agile Approach

The Product Management team or Product Owner defines one or more User Stories in an Epic. The definition of a User Story is defined as "The purpose of a user story is to articulate how a piece of work will deliver a particular value back to the customer."

The Epic typically lives on an Engineering Team’s backlog. It’s a good idea and a premise of agile to break a large feature down into smaller pieces so they can be delivered incrementally, or even in parallel by multiple engineers. So, the Product Owner will define a series of Stories to represent pieces of the functionality. The Stories are then implemented by engineers until eventually all Stories defined for the Epic are complete. Theoretically, this results in the delivery of the new feature. Voila!

The Problem

I don’t mean to be patronizing but I want to use a basic well understood analogy instead of a software system so that we don't get bogged down with technical details. Let’s imagine the Epic is to build a house. That perhaps seems like a large Epic but it’s all relative so please bear with me on this as an example.

If we break down the components of a house functionally to be delivered incrementally, we might end up with stories like:

• As a homeowner, I need a kitchen so I can store and prepare food.
• As a homeowner, I need a dining room so I can consume food with my family.
• As a homeowner, I need a bedroom so I can sleep.
• As a homeowner, I need a bathroom so I can…

You get the idea. Now, if these stories are assigned to different engineers to build the house, or even implemented one at a time by a single engineer, it is pretty obvious that things will quickly break down into chaos. Where will these rooms be built? How will you get from one to another? Where does the plumbing or electric wiring go? What about a foundation or a roof?

When this approach is applied to define the backlog for a software project, it results in similar chaos in engineering teams. Significant work is overlooked. Bad designs are implemented. Work is duplicated. Quality issues are found late in the development cycle. Etc.

The Solution

Of course, mankind has been building buildings for a long time and a house hasn't been built like this since shortly after we stopped living in caves. It seems the software industry has yet to reach this level of maturity.

The instructions given to construction workers are not functional but are instead derived from an engineering plan with blueprints that are designed by an architect. Only then can the actual engineering tasks be defined to deliver upon the design. Of course, the design should be reviewed by the stakeholders that it meets their requirements before being translated into work instructions that describe what needs to be built.

This design work and definition of building specifications is performed by engineering specialists, not by the property developer or business owner. Once the technical specifications are understood, the work to deliver upon the design should be defined in terms of what needs to be built, and not the end-goal functionality. Of course, these work-instructions will ultimately deliver upon the functional goals because the design will have been explained, reviewed, and approved by the Product Owner before implementation begins.

The design phase is a great opportunity to consider other non-functional requirements that can be derived from a list of emergent properties such as Security, Scalability, Availability, Reliability, Maintainability, Compliance, Usability, etc. Once these potential 'risks' have been identified, further designs can be put in place to mitigate or solve for them and appropriate Component Specifications (or requirements) can be defined.

It should be noted that this 'design phase' is itself actual work and should be a recognized as part of a software development process and plan. You should acknowledge that this work takes time and resources and identify it on your backlog as such. I call these Analysis Stories and they have story points like any other story. You don't need a dedicated architecture team, but you do need your best engineers to wear an architect-hat occasionally.

You can still build most features incrementally with an Analysis Story, followed by one or more Development Stories that are derived from the engineering work that needs to be implemented to deliver upon the design. For new projects or larger features, your analysis may identify that large pieces of supporting technology need to be built to deliver upon the feature. A talented senior engineer will find a way to design a system that can be built incrementally but that can be extended in the future for anticipated or known features.

Be careful of the slogan YAGNI (You Ain't Going To Need It). You can't apply that to everything so tread carefully. Build what your team knows and agrees you will need.

Estimation, Planning Poker, and MVP to Schedule

Another benefit of breaking a set of Product Requirements down technically is that the engineering team can provide a much more accurate estimation of the scope of each piece of work once it has been defined. My teams have even had great success SWAGing the scope of a product by estimating the Analysis and Dev Stories for each feature prior to investing the time to perform the actual analysis and define the design. It turns out that if you SWAG enough features at once, and have a good team conversation around a planning-poker game, that you'll be surprised how accurate the total estimate will be.

Once you have a pre-filled backlog with estimates, you can work with the stakeholders to prioritize or delay scope to deliver to a desired schedule and deliver incremental releases.

It's useful to understand the dependencies between Functional Epics and Development Stories so you know the true engineering work required to deliver a feature. This isn't usually supported well by Agile tools but is typically easy enough to manage. E.g. that features A,B,C all have stories X,Y, Z as a dependency.

Conclusion

I used to think this problem was caused by Product Managers not being technical enough to define Development Stories effectively. I had the epiphany recently that the root of the problem is actually caused by the mindset used to break the problem down.

An Epic should not be broken down functionally but instead should be broken down from the perspective of what engineering work need to be performed.

I.e

Stories are not subsets of Product Requirements.

Instead,

Stories describe the engineering work needed to deliver upon a Design.

It also turns out to be very useful to have a cohesive set of Product Requirements defined outside of your task management system, as opposed to scattered across a thousand ephemeral stories.