Managing the Product and Sprint Backlogs

One of the most frequent challenges that I come across with Agile / Scrum teams, may they be established or going through an Agile transition, is with the proactive management of their Product and Sprint Backlogs.

Despite the proliferation of many great Agile / Scrum application lifecycle management (ALM) tools, such as Jira, VersionOne, Microsoft Team Foundation Server, or others, teams struggle with the process of creating good backlogs, maintaining relevant information in backlogs, and regularly grooming them.

Based on my observations, the problem arises from a combination of issues such as:

• The Product Owner being overwhelmed or spread too thin
• The lack of a clear product vision that translates into a product roadmap
• The lack of a product roadmap that translates into clearly expressed release goals
• Insufficient tools expertise or lack of understanding Agile / Scrum best practices
• Problems estimating the backlogs in time

Prerequisites

To reiterate some of the basic preconditions that have to be met in order to effectively manage the Product and Sprint Backlogs, they are:

1.      Good user stories

2.      Good estimation techniques

3.      1. and 2. result in the ability to perform effective release planning

Without good user stories and applying good estimation techniques, the product and sprint backlogs become less useful and will ultimately prevent the Agile / Scrum process from becoming predictable.

The Product Owner provides strategy and direction for the project, which means he is responsible for providing the vision, product roadmap, release goals, sprint goal. The Product Owner is expected to insert, re-prioritize, refine, or delete items from the product backlog; this can happen any time until the sprint scope is defined and committed to by the development team.

Sizing and estimation of the product backlog usually occurs in sprint planning meetings or at regularly intervals during ongoing sprints. Depending on how fast the Product Owner adds user stories, more frequent, maybe even daily estimation sessions might be needed. The Development Team and Product Owner must work together to estimate the backlog items, may that be in sprint planning meetings or during ongoing regular sessions.

The important point here is that product backlog items need to get estimated continuously in order to avoid the big bang estimation effort at one critical juncture of the effort. The frequency of the estimation effort is depending on your product / project development effort. This has the added benefit of exposing the entire team to the Product Owners’ thinking, keeping everybody in sync and with an improved understanding of requested product features and functionality.

Sprint Planning meetings or more frequent estimation session need to follow established Agile Estimation best practices, meaning Product Backlog Items are estimated using either Estimation Poker or T-Shirt sizing.

The Product Owner is responsible for initiating requirements, creating user stories, and exerting scope control.

The Development Team is responsible for estimating the backlog in close collaboration with the Product Owner.

Product – Release – Sprint Backlogs

Many efforts require you to manage three backlogs:

1. The Product Backlog, which essentially is the requirements database for the product, expressed in user story format – exists for the lifetime of the product
2. The Release Backlog, which is the subset of functionality that will have to be delivered in a specific release, according to the product roadmap and release plan – active for a specific release
3. The Sprint Backlog, which represents the work to be done for the upcoming sprint – active for the sprint duration

All tools in the market allow you go create tags or categories that allow for quick and easy views of what user story falls into what bucket.

Using such tagging lets you easily view relevant information about the specific user story and what state it is in.

Usually the next upcoming sprint is sized and estimated in detail using Estimation Poker, with the following two to three sprints and their contents being sized and estimated using either Estimation Poker or T-Shirt Sizing. Remember that once a sprint is committed to by the team, the scope cannot change any longer as you are “in-flight”. Future sprints can still be adjusted and items can be reshuffled as the Product Owner and Development Team agree on.

Product Backlog Items and Sprint Backlog Items may contain more than just user stories, but regardless what exactly is in the specific backlog, it is supposed to represents all work items that consume team capacity. This includes the actual user story describing the product feature to be implemented, defects, research, and other technical tasks.

Remember that the Sprint Backlog is supposed to be the most precise and fleshed out backlog due to its immediacy to actual delivery of a working product increment:

• Consists of committed PBIs negotiated between the team and the Product Owner during the Sprint Planning Meeting
• Scope commitment is fixed during Sprint Execution
• Initial tasks are identified by the team during Sprint Planning Meeting
• Team will discover additional tasks needed to meet the fixed scope commitment during Sprint execution
• Visible to the team
• Referenced during the Daily Scrum Meeting
• Used to measure Sprint Acceptance by the Product Owner based on the Definition of Done

Backlogs are the Fulcrum of Agile / Scrum

I hate the old saying “Garbage in, garbage out”, but in the case of Agile / Scrum backlogs, it is true.

The single most important artifact impacting the delivery success of the current sprint is the Sprint Backlog.

The single most important artifact impacting the success or failure of your product / project is your Product Backlog.

Agile / Scrum is a simple process, with few defined roles, few artifacts, and few events.

Backlogs hold the essence of what needs to get implemented, and as such need to be created, maintained and groomed with great care.

Unless the Product Owner and the Development Team focus on keeping backlogs up to date and well groomed, the Agile / Scrum process will not be able to deliver what is most important to organizations: Predictability.

Common Issues and Challenges

The section addresses some common issues and challenges that make managing product and sprint backlogs more difficult than it has to be.

Poorly Written User Stories

User stories should follow the INVEST principle. They should be:

• Independent (to the degree possible, user stories should not rely on other user stories to be implemented)
• Negotiable
• Valuable (meaning they are feature focused, not task oriented, and are written in the user’s language to the extent possible; where domain specific language is required, such as in genomics or medicine, a subject matter expert will have to be available)
• Estimable
• Small (half a Sprint or less for one Dev Team member)
• Testable (needs to have acceptance criteria and not be subjective)

The testability of a user story is usually documented in the form of Acceptance Criteria on the reverse side of the index card (or the appropriate place in the ALM tool of your choice) and lists pass/fail testable conditions that help us verify if the user story is implemented as intended.

All sounds simple enough, but frequently teams and Product Owners really struggle with writing good user stories, and 90% of all problems encountered are based on user stories being too big.

When a user story is too big, it is hard understand, estimate, implement, and prone to wildly different opinions. So what is a good size for a user story? Basic guidance is that the user stories at the top of the product backlog (the ones that should have sufficient specificity in order to be worked on by a developer) should be sized such that a team member could easily complete two user stories during a sprint.

Good user stories are the basis for good backlogs.

Rapid Requirements Inflow

Sometimes the Product Owner might start adding user stories to the backlog far faster than the Development Team can estimate the effort.

Or, the Product Owner, out of desperation because the Development Team is busy with implementation tasks, starts estimating the effort himself, without the Development Team being involved. This is a bad idea!

One of the core Agile principles is that “business people and developers must work together daily”, as such the Product Owner cannot simply hand over user stories to the Development Team as it defeats the purpose of jointly understanding and estimating user stories. One of the key characteristics of a good Product Owner is the ability to explain, in detail, what is meant by a user story.

If the inflow of requirements outpaces the teams’ ability to understand the requirements in detail and estimate them accordingly, the best course of action is to set up additional backlog grooming sessions to get things back on course.

Fantasy Wish List

The backlogs need to contain user stories that are relevant to the product, the release, and the sprints. Cluttering the backlogs with fantasy requirements that do not directly contribute to the product, release, or sprint goals is counterproductive.

Such wish list items need to be kept separate and managed outside of the backlogs, until such requirements become tangible goals at the product, release or sprint level.

Backlogs = Requirements Specifications

Especially during Agile Transformation / Adoption efforts, teams often times deal with a long tail of old artifacts available from previous software development methodologies.

For example, you might have many UML diagrams, functional or technical requirements specifications, etc. Why not put them to good use and just make them part of the user stories?

The reason not to do this is because specification have the side effect of shutting down discussions, as in “this is the specification we need to implement, nothing more, nothing less” – essentially specification hinder the discover of newly emerging requirements.

However, this is not to say that once a user story is well understood by the team, you will never, ever, write a technical specification document in order to document the implementation.

Unavailable Product Owner

One of the most common challenges with Product Owners is their lack of availability to work with the Development Team. Product Owners are enablers, making the Development Team more productive – or less!

Without an available Product Owner, the Development Team will get stuck – this might manifest itself either in sprint goals not being met, in poor quality, or in low velocity.

Lack of Backlog Grooming

There really is not excuse not to regularly groom and estimate your backlog. As mentioned before, if for whatever reason your backlog is not being groomed, the best course of action is to set up additional backlog grooming sessions to get things back on course.