Experiments – How to Not Blow Everything Up

One of the most interesting bits of feedback I ever received in a retrospective came from a team that believed Scrum was holding them back.?Those team members enjoyed our culture of collaboration, inspection and adaption and continuous learning. From an Agile standpoint, the team was working great.

But good Agile teams have a continuous improvement culture. My team members felt we could track and manage their work better. They would be more efficient and deliver more value quicker if they performed their work in a Kanban structure instead of a Sprint structure.

I was stuck for a minute about what to do when my ace team members called me with this idea. Should I immediately say no, because it would affect our team velocity? Should I immediately say yes, because I trusted that group to think through their request? Or is there a middle ground? For a moment, I felt caught in a real dilemma.

Thankfully, there is a smart and well-defined approach to take to this request: an Agile Experiment. With the encouragement of ace Agile Coach Chris Moody, I worked with my team to create an experiment that would allow them to try Kanban for a fixed timeframe and see how it works. You can think of an Agile Experiment as a kind of “try before you buy” idea, a chance to, well, experiment with a concept before you put it fully into force.

Here are a few guidelines to structuring and managing an experiment: First, target the experiment to the team’s maturity. Once you have that in hand, consider the problem the team is looking to solve. Create a hypothesis, a tangible, measurable goal and a time-box for the experiment. Monitor the experiment as it runs, and don’t be afraid to cancel or revise the experiment as it’s running. Finally, when the experiment is complete, assess the results based on your criteria and then determine next steps.

I go more into detail on each step below.

Consider the team’s maturity level before taking on experiments

Experiments can be great ways to enhance the team’s maturity levels. In fact, one of the key indicators of a team’s progress along the ShuHaRi?scale – a way of assessing a team’s mastery of Agile skills - ??is in its interest in taking on experiments. Teams in the “Ha” phase may have experiments suggested by the Scrum Master or Product Owner, while teams in the “Ri” phase might consider experiments as part of their adaptation of Agile to their specific circumstances.

In fact, this would be a good place for an experienced team leader to suggest changes to help make the team more efficient and, more importantly, help kickstart the team to a “Ri” mentality.

Clearly my team was already on the “Ri” phase of maturity. Team members felt safe suggesting their idea because we had experimented before. For instance, we experimented with providing Kudos in all Retrospectives and in canceling Daily Standup in the first working day of each Sprint.

Identify the problem – not the solution

Before committing to an experiment, try to define it into clear terms. Make sure all stakeholders understand the challenge and opportunity, and think through whether the experiment will actually solve the problem at hand. Treat the conversation around experiments as similar to a backlog refinement session – team members should understand the problem and solution in simple terms.

So my team’s problem statement was:

Our work is managed on a continuous flow, so the start-and-stop boundaries of Sprint cycles cause our team to deliver fewer stories than we could.

Here are a few more examples of problem statements:

Define an actionable hypothesis

Write the story in clear terms. User story format (“As a… I want to… So I can…”) is a good way to start. Anybody should be able to understand the problem and potential solution.

Make the experiment tangible. In Agile, we believe in using tangible metrics to track success.

Size the experiment to a reasonable size. Again, common sense applies: Don’t boil the ocean.

So my team created this hypothesis:

As the configuration team I want to track my work on a Kanban board so I can have a continuous flow of work.

Here are some hypotheses based on the problem statements above:

Define a tangible goal

Set a goal that can be tracked using metrics that can be easily defined.

My team set this goal:

10% increase in story completion in a rolling two-week period versus the average of the last 5 Sprints.

Here are some goals based on the problem statements above:

Timebox the experiment

Set a time limit for an experiment to run. The ideal timebox should be long enough to see the results of the experiment over a reasonable amount of time, but not too long that an unchecked experiment will cause problems.

Make sure the team gives itself sufficient time to learn what they are experimenting on. For instance, if the team is trying mobbing, the team may see decreased productivity while first starting out. Therefore they may want a longer timeframe for that experiment.

My team set this timebox:

We will track this work over 2 Sprints (4 weeks)

Here are some timeboxes based on the problem statements above:

Don’t be afraid to adjust or to complete?early

My team quickly discovered they had too many columns in their Kanban board because they found some statuses were uncommon. No problem, we did a huddle, made an adjustment, and continued the experiment.

If the problems with columns had been a major problem, we could have completed the experiment early. If it’s obvious before the completion date that an experiment has failed, don’t hesitate to call that experiment done early. Just document your learnings and move on. Remember, it’s better to learn fast than to continue to be frustrated.

Encourage the team to embrace a no-failures mindset. The experiment simply didn’t work out as expected. That’s not a failure.?That’s an important learning that leaders should praise.

Inspect the results

At the end of the timebox, the team should review the results of the experiment. The Scrum Master or other team leader should lead a discussion of the metrics and about team members’ feelings about the experiment. My guru Chris also recommends the team asks themselves the following questions to help assess their success.

• Did everyone do what they needed to do to make the measurements valid? Be honest with yourself.
• What did you learn throughout the course of the experiment?
• Did the hypothesis support the outcomes you were expecting?

Decide on next steps

Once the team has inspected the results, they should decide how to adapt. Some options:

• Pivot – Some aspects of the experiment were proven true, or the experiment showed that something else might be a root cause. Reframe the experiment and try again.
• Adopt?– the hypothesis was proven true, and the team is ready to adopt this as part of its standard work. Make sure to add this change to your team’s Working Agreement
• Complete – The hypothesis was wrong or the experiment didn’t work as expected. Move on and treat this as a lesson learned.

A culture of experimentation

Team leaders can help the team create a culture of experimentation in several ways.

One way is to discuss new experiment ideas during Retrospectives or Inspect and Adapt events.

Another is to create a doc in a shared place – Teams, SharePoint or a wall in your team’s work area – to list all experiments and help make it a part of team culture that you perform experiments and try new and different ideas.

Share results with your Community of Practice or Scrum of Scrums?

Finally, and most importantly, share your results with your peers. ?They’ll be interested in hearing what your team tried and how your experiment might influence their team’s experiments.

The Benefits of a Hypothesis

Have you tried experimentation as part of your work? What has succeeded or failed for you? Please share your results in the comments.