The Never-Ending User Story

Are you tired of user stories that seem to go on forever? Are your user stories poorly defined? Are you stuck in a quagmire of invisible work in progress, struggling to understand the status of your tasks and make progress on your projects?

You're not alone!

Before starting, I would highly recommend "Learning Agile" for anyone interested in learning more about agile software development or some of more overarching topics. This book is a comprehensive guide to the most popular agile methodologies, and it does an excellent job of explaining the underlying principles of agile and how to apply them to your team. The book covers four specific agile methods - Scrum, extreme programming (XP), Lean, and Kanban - and provides a clear and engaging description of each one. Whether you're new to agile or looking to try it again, this book will help you understand how to choose the right method for your team and company. Overall, "Learning Agile" is an essential resource for adopting agile practices and principles.

Specifically, the book explores the concept that, while most people may still need to fully realize all the potential benefits of agile methodologies, they are still generally experiencing some improvement by using these approaches. This idea is captured elegantly in a quote from the book that often hights early experience of the agile process:

"Better than not doing it"

This expression suggests that even if teams are not fully maximizing the potential of agile, they are still seeing some positive impact throughout their software development process. This can be attributed to the fact that agile methods prioritize frequent check-ins, regular communication, and adaptability, leading to more efficient and effective project management. Even if teams are not fully following the agile process, improved communication across the team can be a major benefit; hence it is better than not doing agile at all.

The book explores common problems that teams experience when implementing agile practices and approaches to addressing these problems. Today want to write about a common problem I have experienced and witnessed: the phenomenon of "the never-ending user story". This problem is often a side effect of teams being new to the agile process and its elements or when organisations cannot fully support the agile process. When teams are unfamiliar with agile principles, user stories can become overly large and complex, making them difficult to understand and complete. When user stories are not clearly defined or sprawling, it can be difficult for team members to understand what needs to be done and when a story is considered "done." This can lead to delays, misunderstandings, and a lack of transparency, all of which can have a negative impact on the success of the project and the motivation of the team.

A simple solution, contributed by a fellow LinkedIn user, recently discussed the idea that we shouldn't estimate the length of user stories but rather limit them down to their smallest elements, such as jobs that can be completed in a day. While this is an insightful perspective, it is not always practical. User stories often involve various stakeholders who contribute to their development and execution. These stakeholders may include the end users who will be interacting with the product or service, the product owner who represents the business and prioritizes the work, and any other individuals or groups who have a stake in the project's success. Each of these stakeholders may have different perspectives and requirements.

While there may be an argument for reducing the field of stakeholders to streamline the process, in some cases, it is necessary to include certain stakeholders due to regulatory requirements or the need for input from key business owners. This is a good example where the organisation constrains the agile process. In this environment, we often give birth to the neverending user story due to the business not signing off an item being "Done" and ready for business as usual.

The Iceberg

With that, I want to explore a strategy that I find effective with my stakeholders: to treat a deliverable as an iceberg. Visualising a delivery as an iceberg can be a useful tool for understanding the division of responsibilities between different stakeholders in a project. The ice above the water represents the aspects of the project that are visible to the wider business and are of concern to stakeholders such as product owners and executives. These may include high-level business goals, customer needs, and strategic priorities. On the other hand, the ice below the water represents the elements that the development team is concerned with to achieve delivery. These may include technical details, implementation approaches, and work breakdown into smaller tasks. By understanding the division between the ice above and below the water, teams can better align their work with the needs and priorities of the wider business and ensure that they can deliver high-quality results in a timely and efficient manner.

This lack of conceptual separation can lead to executives being pulled into the tall grass, and becoming over-involved in the process to a detrimental effect. The long-term goal in this situation is to build confidence in a higher-level objectives plan driven from a roadmap or milestone list, however, to build this confidence we have to take small steps on this journey.

Taking this analogy further, we can seek to build confidence by applying the idea to user stories. We can look at a user story as "tip of the iceberg" - the high-level description of the functionality that the user will see or experience. Or a body of work that thought of as the "ice above the water," and it helps us communicate the concept to wider stakeholders there is often a much larger body of work that takes place "below the ice," representing all the smaller tasks and details that go into building the deliverable as a whole.

This may include investigating different paths that ultimately prove to be not fruitful, as well as the volume of smaller elements that are required to complete the user story. By understanding the full scope of the work involved, teams can better plan and execute user stories and deliver high-quality results. Defing as business what is above the ice and below the ice can help as it allows us to better track progress against some of the slower moving elements in the project.

Azure DevOps, specifically, the boards, is a powerful tool for managing agile projects and supports this distinction for teams to build a clear distinction between work items which are "above the water" and "below the ice". Item "above the water" are those that are visible to the stakeholders such as product owner, users and executives. These include user stories, features, and epics.

The work item types that are "below the ice" are those that are primarily of concern to the development team and are focused on the implementation details of the work. These may include tasks, bugs, and issues, as well as other derived work item types.

DevOps reinforces this distinction by allowing only "below the water" items to be assignable to sprints or specific iteration, while "above the water" items are visible on custom boards. Where as the above the water items, define the swimlane for work that is in progress, i.e the themes for the sprint.

This helps teams focus on the implementation details of the larger body of work, maintaining a distinct hiracharal relationship of the driving force of the smaller tasks. Product Owners can drive the generation of story items and the development team then can focus on smallest element that contribute to the overall delivery. Chunking the work down like this, also supports the sprint planning process, allowing developers to brainstorm the elements that go into building out a specific task.

Below The Water: Tasks

The rules for these work items, such as tasks in Azure DevOps should be:

• Clearly defined peace of work that must be completed which support goals of the user story or feature. I.e what can I get done today.
• These tasks should have a clear purpose and objective, in practice should aim to no more than days work.
• An ethos it is important to note that a task cannot "succeed" or "fail" on its own - the work can only be done or not done. If a task misses an element of additional work or requires rework, this should be identified as a new task that must be added to the backlog.

There may be times when there is a task of broader scope, which is expected to run over several days, such as "Update the database schema to support roles for contacts". While it is important to have a clear understanding of the overall goal of the task, it is also important to be willing to break it down into smaller tasks during sprint into work can realistically be completed in a given day. We should embrace creating task that represent granular peaces of the work throughout the day to help teams better understand the work that is being undertaken.

It is also important to remember to estimate the broader tasks, this allows us to manage sprint capacity and provides immediate visabilty when chunking the work down of any differences between the assumptions made and the reality of the work. This process of recording smaller, manageable tasks daily keeps total visabilty of work in progress and we can clearly see what taken place and identify any constraints that maybe present. The process of chunking down should be considered a continual.

An Example:

To elaborate further, taking our previous example of a four day task, "Update the database schema to support roles for contacts"; it could be broken down over course of the week.

Day 1: Add the new tables and foreign key relationships into database update scripts.

Day 2: Ensure default data is created and Populate the roles table with required roles.

Day 3:Identify any stored procedures and views that need updating, the process of this task identities two further tasks:

1. Review and update the complex stored procedure that needs more consideration
2. Update remaining views to use new tables.

Day 4: Review and update the complex stored procedure that needs more consideration.

Day 5: Update Remaining views to use new tables.

These are obviously fictious tasks, however we immediately see that are identifying further work which will push out the scope of this item by one day. Rather than allowing larger item to roll over the time without any visibility to why. It is fairly normal for work to deviate from our estimates, however this process gives us visabilty imidately to how much longer something has taken.

This continual process of breaking things down to elements that can be completed in day or less allows teams to clearly report progress and velocity. Allowing a team to clearly demonstrate the work that gone into a deliverable, the areas that it has touched and related implications and risks.

Managing Above the Water

Above the water, we have user stories, features and epics or simular derived work items. Sometimes when teams are new to agile, or from different development backgrounds, it can be challenging to determine whether an item should be a user story or something larger, like a feature or epic. In these cases, I follow a simple rule of thumb: everything should be considered a user story, i.e a user requirement, whether it is a large-scale feature or a smaller requirement like adding delete permissions. This helps keep things simple and allows for the possibility of promoting items up the hierarchy.

When building out the "above the water items", in environments where requirements are loosely defined or evolving it can be difficult to manage expectations or define when something is "done". One way to solve the problem is to introduce the concept of promoting user stories larger scale element like a feature. Providing this, allows product owners to reevaluate "done" when the scope is extended.

Imagine context of story item which is a report. It might be during the sign off process the business had made an assumption that report would be printable and distrubtable as PDF document and realise current implementation as Excel document is insuficant. In these situations, the business can be reluctant to sign off on work done on current story as is not currently meeting there requirements, irrespective of whether these where clear at the onset of the task.

In this situation, the report can elevated to Epic, the work in current story item can be redefined in new story to meet the work that is done and signed off in isolation, and additional user stories can be generated to reflect the additional requirements.

This process of promoting and redefing the scope of tasks helps twofold. Firstly we demonstrate and report meaningful "above the water" progress to our stakeholders while building confidence that further requirements are being taking into account. Secondly, we can ensure that existing progress is thoroughly tested under the scope of the new completed story item, rather than having a single item that has sprawed over a significant number of weeks.

Conceptually, we have to consider that above the water, we are working in the frame of refreance of business. In that sphear something is delivered when it is 'adding value'. In some envirmeonts, especially those new to agile, it is not uncommon for large volumes of work to remain idle, not adding value because it has not delivered 'all' the requirements.

The process continually evaluating done, is process which allows the business to continually assess whether how completed work will effect there day to day operation and ultimately support the iterative journey to reach the end goal.