Experimentation: An Australian Story

In a previous article, I explored a paper from Bojinov and Gupta that defines 3 key pillars of digital experimentation:

• The Data Platform
• The Experimentation Platform
• People and Process

This conceptual idea is helpful to break down what digital experimentation is, though if you are running experiments (or want to) you're probably more interested in how you can deliver better experiments, and more of them.?

There’s a challenge for us in Australia, as most of the leading thought and practice examples of digital experimentation are from the USA or Europe - both regions that benefit from a lot of people and have a strong history of digital maturity. Common case studies of companies showing massive growth through experimentation - Booking.com, Netflix, Facebook - are also tech-focused with huge audiences.

So where does that leave the rest of us?

Succeeding in experimentation in ANZ requires just a few keys, which when used together can overcome the natural barriers that many businesses see when trying to do more, better experiments.

The Australian Landscape

Down here in the lucky country, we need to think a little creatively about experimentation. Australia and New Zealand are in the enviable positions of being some of the best places to live in the world, though with some complexities relating to digital testing.

Less People

Australia has less than 1/10th the population of the USA, and this is critical because without people visiting websites, gathering data about experiments can take some time! If you’re experienced in running tests, you probably know the pain of thinking of a great experience, then calculating that it would take 6 months to reach the sample size for a rigorous test.

Smaller Teams

We have smaller teams (and smaller businesses more broadly) than the USA and Europe, meaning if you're sitting in a digital team, you might have a few other team members, but you all likely wear multiple hats! The huge 50+ teams of digital experts is much rarer here, and is usually relegated to the larger enterprise brands.

When it comes to running tests, it's often a small part of one or two-people's job, rather than a practice with dedicated headcount.

Happier to Buy than Build

Our market has historically been a boon for software vendors, with brands marketing, digital and IT teams happy to splash cash out for software compared to build things in-house. Broadly, this strategy makes sense, however it also leaves a legacy of expecting a technology solution when there might be a better human one available.

We've seen this come in cycles: DMPs came (and went), CDPs are still happening (but less often), and now AI orchestration vendors are in the limelight. Remember that when it comes to making better decisions that are backed by data, technology is rarely going to solve the problem - it's one of working together between people.

Experimentation (and AB testing) is often categorised into a software vendor RFP that is 'ticked' for procurement, when it should be thought of more like JIRA, Monday.com or Asana: a tool that needs adoption to see the value that's promised.

Keys to Successful Experimentation in Australia

Only Customer Needs

It's only too easy when you're thinking about your website every day to consider problems for the business. There's a reason it takes 3 months to be 'onboarded' in most companies - it takes time to learn the language and modes of thinking internally - and it's the exact same for your customers!

Experiments that you design should be based on the problems a customer has - not the problems the business has. To improve purchases on an ecommerce site, consider: what is stopping the customer from buying? Are they being rushed to purchase before product decision? Are they having trouble finding products? Do they have the right social and emotional proof for the products?

Running tests that are only focused on customer needs removes a lot of bad test ideas, and forces you to solve real customer problems. Along the way, you’ll show a commercial impact and be moving your engagement metrics in the right direction.

Top tip: Creativity is needed in test design, through using more UX testing, feature testing and well-defined measures of success.

Considered tests only

You're reading this on LinkedIn, so you might have seen a post like "147 AMAZING Landing Page Optimisation Tactics that WORK - comment 'Optimize me!' for the full list".

That belongs in the bin for us down here in Aus - we don't have the traffic or the time to benefit from trying dozens of (probably terrible) ideas across our limited real estate on site.

Each experiment run should be considered: have a known customer problem that's being solved for, be designed to capture learning no matter what the outcome is, and be scoped for how quickly and easily the test is expected to run.

Full-person Ownership

Lots of Aussie businesses survive with lean digital teams, and that means wearing multiple hats in roles. Unfortunately, too many businesses are trying to get value from experimentation where the 'owner' is only dedicating <25% of their time to it.

Experimentation needs full-person ownership - in larger teams that means a dedicated strategist who drives the program and advocates for adoption. For smaller teams (of which there are many), it means relying on partners who are dedicated and focused on the job of experimentation.?

Top tip: Partners are often needed to create a systematic process for teams to follow, as with 30% of one person's role, it's unlikely to develop the consistency or expertise required on their own.

With these three keys, Australian businesses can still reap the same rewards seen overseas in Europe and USA, despite the unique challenges we face in our market.

What other keys to experimentation success have you seen for ANZ businesses?