How to embed experimentation into Agile product development

Introduction

The way Agile was supposed to work and the way it actually works in many real companies are often incredibly different.

The idea was that software teams can be directly responsible for the fluid development of products according to customers needs. The reality is often that Agile is used by senior management to get what they want done, but faster.

The issue in many cases is who/what feeds the road map? It ought to be the customer, but more often it is ego and opinion.

This post aims to show how Agile can reclaim its original aim by embedding research, analytics and, most importantly, experimentation practices into its core ways of working

My take on the original Agile vision

Agile means many things to many people, but for me the concept of agility is ultimately about iterative customer collaboration — developing products or software gradually based on what customers want and need, rather than using internal opinion about what that software should be.

Now, if you had 3 customers you could simply put them in a room every week and ask them how they are using it and what else they need, but when you have thousands or millions you cannot do this. This means that Agile relies on 2 things:

1. Understanding of what customers want and need; how they use your product, the struggles they are having and the opportunities to make it more useful. This is data, research and analytics.
2. Experimentation which allows you to test whether your interpretation of the findings of #1 are correct when delivered back to the customer, because some of those interpretations will definitely be wrong.

Therefore, these skill-sets must be central to scrum ways of working and mindsets, otherwise agile becomes simply a different way of responding to internal business demand, even if that demand is masquerading as customer insight.

Why embed experimentation into Agile ways of working?

In many companies there is some form of insight and analysis feeding the road map, but I don't believe that is enough. Transforming your Agile operations will maximise the ROI of investment in these resources because:

1. It is the foundation of iterative, continuous improvement

Iterative improvement is always improvement towards something and that goal can only really be customer experience, the secondary benefit of which is business value. What customers want cannot be intuitively known; only through actually listening to real customers and understanding their needs will any real progress and innovation happen.

2. It puts the scrum in control

Requirements can either come from the customer or they can come from the whim or desire of someone internal to the business. When the latter is the primary driver of the work flowing into the squad this generally results in a demotivated team who feel disenfranchised from their work. If, conversely, the requirements are both data-driven and originate from within the squad, the squad gets the sense of control, autonomy and mastery of destiny which is the intention of Agile.

3. It delivers value

When executed effectively, a data-driven approach to product development will always deliver the most value, both to the customer and to the business, because it is focused on directing resources at where the greatest impact can be achieved. It doesn’t mean that intuition and creativity have no place, it simply points those efforts at the right part of the experience.

Step 1 - define your continuous improvement strategy

The most important foundation stone of the experimental approach is a strategic framework - essentially a description of what a successful customer experience looks like. The following example describes digital online self-serve help and support operations.

The purpose of this framework is, very simply, to guide what it is you are seeking to improve. This lays out the steps of a successful self-help engagement and defines both the metrics needed to measure the effectiveness of that step in the journey and the components which can be improved through experimentation.

A very different example for a more sales focused website:

Step 2 - create an experimentation process which sits before your road map

The ultimate goal is that you are not investing any significant resources (and therefore money) into developing anything which hasn't been proven through experimentation. This might have involved MVP development or similar, but at each step the amount of development is proportionate to the risk involved due to the validation it has received through live experimentation.

Experimentation does not mean testing features as they are released. This is the cart before the horse.

I wrote a specific article on this recently. In summary though, you are creating a funnel process before your main road map which translates insight and data into ideas into product developments of gradual size and risk as the feedback through experimentation validates these developments. If you are testing major features when they are released, you have already invested the time to develop them, which defeats the point of experimentation.

Step 3 - embed digital experimentation resources into your resource structure

In an ideal scenario, a product owner would be aligned to and supported by an Experimentation Analyst who is integral to and dedicated to the scrum team, and who is responsible for project managing the experimentation processes, including co-ordination of the many different specialist resources required to run such a function

The experimentation analyst is ideally:

• Embedded — physically located with the squad or tribe if resources don’t allow for 1 per squad. They should be involved in all aspects of the daily life of the squad; stand-ups etc. They will particularly partner with the product owner/manager and also interface with any wider business stakeholders.
• Lean — they are uncovering customer pain points and suggesting fixes in a fluid, always-on manner, not (or maybe only very occasionally) going away into a black box to produce large presentations. They are entirely pro-active in their pursuit of results.
• Product-specialists — the experimentation analyst is multi-skilled and covers all aspects of insight, analytics and experimentation, but is dedicated to a particular product team or area, and knows that product inside out in the same way the product owner does. It is virtually impossible to hire someone with all these skills out of the box, but if they are pro-active and analytical the rest of it generally comes fairly easy to them.

A culture of analytics and experimentation should be championed and driven by the analysts within their respective areas, and relies on:

• Clear KPIs and reporting of the benefits of features
• Gamification of the experimentation process — healthy competition about whose ideas deliver the greatest impact. Getting teams to vote on which variants of tests they believe will win is also a time-worn but incredibly effective way of creating this culture.
• Democratisation of tools/data and a focus on simplicity in those tools, for example ensuring that analytics implementation is clean and intuitive. The analyst should also regularly train the other team members on using these tools.

They will be required to co-ordinate other specialist resources in order to achieve the desired outcomes.

Conclusion

The whole point of Agile is to listen to customers and develop based on their needs, but for that to be successful the scrum team must be in control of this listening. What I have tried to show is that ‘listening’, really means ‘understanding customers frustrations and needs via data’ and ‘development’ really means ‘development driven by controlled experimentation’ which allows the team to truly validate what that listening is telling them.

But, above all else, the approach is about embedding the skills and resource within scrum, not taking a ‘requirements’ approach.