Taking Bets Further With Experimentation

Every day of your life you're making bets.

Perhaps you're not sitting down at a roulette table for a spin, though the is that we gamble with many decisions we make.

Back in April 2020 (the depths of the COVID panic) my wife and I were looking for our first house while 6 months pregnant. At that point, Sydney had been on a 10+ year property bull run and we were looking at house prices rapidly out-scaling our ability to afford.

We found a house that we loved, and were lucky enough to move in on 27th July, 2020. Our daughter was born 3 days later.

On reflection, and even though it didn't always feel like it at the time - we were taking an absolutely massive (perhaps reckless) gamble.

• Betting on our continued income to pay off a mortgage for most of the rest of our lives
• Betting (in 2020 remember) that our world wasn't irreparably changed by COVID
• Betting that our daughter was going to be born AFTER we moved in!

?

It probably sounds obvious that taking on a massive amount of debt is a gamble, though in the context of buying a home it's one we see as a normal part of life?

Stepping up to the table

In the exact same way, making decisions every day in our professional lives is betting, even if we may not realise it.?

• A measured cost-reduction approach that cuts mandatory risk and compliance training by 10 hours per year is betting that this reduction won't result in an increase in safety incidents or violations.
• Hiring a new creative agency for a series of large advertising campaigns is not only betting on increased sales, but on the campaigns not triggering backlash, PR challenges or negative customer sentiment.
• Developing and launching a new feature for a SaaS product is betting that it will make a meaningful, positive difference to end users and increase product usage and retention.

Decisions like these are made every day. They may be the best decision, the only decision available to us, or an informed, calculated decision - and they are all bets.

As decision makers in businesses, your role is to mitigate the risks from bets made, and to prioritise what bets have the biggest potential pay-off. It might sound like I'm about to start extolling the virtues of a proper risk assessment framework. Instead, I'd like to raise a different toolkit that doesn't just help decision making, and allows taking big risks while protecting you from the potential downsides: Experimentation.

Big 'E' Experimentation

As we've covered, Experimentation is a big tent that encompasses decisions ranging from strategic through to the day-to-day.

While a scientific purist approach to running experiments may lead you to think of a randomised, controlled trial (RCT) like you'd use for cancer treatment testing for example.

Our messy world, filled with imperfect information, forced choices and a constantly-shifting competitive environment means that we rarely get the chance to apply such a rigorous method. Instead, Experimentation with an initiative requires a few simple ingredients:

1. A goal
2. An idea of how to achieve that goal
3. Expectations of the outcomes from your idea
4. What to do if you were right
5. What to do if you were wrong

?

A simple example: Imagine you're the Head of Product for a Buy-Now-Pay-Later (BNPL) product. You need to hire a Product Manager to lead a new offering for the business.

• Your goal is to see the new offering land in market and hit (or exceed) the revenue and customer expectations
• How to achieve that is through hiring a new Product Manager who has experience in BNPL products and expertise in aligning products to customer needs
• Expectation is 2-4 months hiring the role, 3-6 months of ramping up and hiring a team with a product launch in 18 months. If you're right, then your timeline makes sense, you'll hit your market entrance period and can hope to hit revenue targets If you're wrong, you may need to hire someone with less specific qualifications to hit the timeline, take longer to hire for the right person, delay launch of the product or revise (down) your revenue projections for the new offering.

?

Where is the experiment? Testing on this bet could happen at multiple stages, and allow you check your assumptions and your method, for example:

• Reach out to a dozen recruiters to ask about the feasibility of your asks re: hiring (and fend off the follow ups!)
• Discuss with other non-competitive finance product companies what they saw in their product development and roll out that might impact your timelines or expectations
• Pay for market research looking at historical growth of new BNPL products into market to refine revenue expectations

(Astute readers might notice that in this example these tests are a combination of user and stakeholder interviews taken from a UX playbook!)

You know what's missing from this example?

• A control group
• Statistics of any kind
• A lift %?

Experimentation is a structured way of making confident decisions (bets) by mitigating risk and looking for value. That doesn't mean just an AB test!

Experimentation as a mindset

Why think in Experimentation?

It might seem like an extra step, after you've already done all the hard work of figuring out what you should be doing, to also consider how you're going to be testing your approach.

The reason an Experimentation approach is so valuable is to provide trust and confidence in a choice, to take risks without over-committing and to be able to track and share how you're finding success.

The ingredients that power experimentation can be applied by practitioners at every level across digital channels and decision making. The ingredients don't change, however the method you use to test your bets shift and change.

Remember Experimentation sits after a you're clear on the bet to be made, and is a part of confidently bringing to life a more significant change. In the classic 7 step decision making process, steps of experimentation are incorporated in formalised research, and testing alternatives:

1. Identify the decision
2. Gather information (Research)
3. Identify alternatives
4. Consider the evidence
5. Choose among alternatives
6. Take action: Experiment!
7. Review Experiment
8. Take action: Double Down, or
9. Take action: Choose a different alternative

?

It's natural to be unsure about a course of action, or to be challenged to demonstrate the research and data behind? a decision. Experimentation provides both confidence and validation in spades that allows consistently better decision making.