Why do we test? Narratives for our experimentation and CRO industry

Understanding WHY we test can help those who do test do it so much more effectively.

And make a TON more money and effective change along the way.

And the why is found in stories. Stories point us in the same direction. They make us look the same way. We can’t always do this, nor should we. But if we feel like it’s a worthy cause then that’s the goal.

I want the experimentation industry to look in the same direction. Or at least be aware of the options of where to look.?

My motivation here is awareness and growth of the industry. If it grows we all grow with it.?

I’ve worked with our marketing manager Morteza Maleki on helping me refine these sets of narratives for our industry. This post is an aggregation of a lot of thoughts and discussions, therefore the post itself is a bad story…we’ll work on that. But for now, let's use it as an anchor.?

The key exploration here is where and how "story telling" applies to our Experimentation/CRO industry right now? Where are the stories in our field right now? What are we telling ourselves are the core justifications and obstacles?

E.g.,

• Data are not being understood
• Insufficient leadership and decision models
• A focus on results without understanding process
• We should 'nudge' our customers.....but not too much?

As we seek to elevate our industry, to elevate experimentation, these could help you differentiate and advance the conversations and applications of testing and science applied to business generally.?

To this goal, below I provide

1. some ad hoc narrative headlines within three categories (think of these as genres of what's going on in our world right now, and
2. two fleshed out strategic narratives using a cool framework by Andy Raskin.

First, three categories of narratives for the experimentation industry:

Category 1: 'decision support' (the point of it all right?)

Linkedin post with discussion

There are a three good ones here:

1. The cost of decision-making inefficiencies

>Businesses are a series of people making decisions, resulting in processes and outputs. How is the efficiency of decision-making directly linked to the success of the business?

2. Decision Intelligence Protocols and developing Business DiQ

>How to use experimentation programs to advance your businesses Decision making efficiencies [DiQ]; the ability to make better decisions, faster?

3. Holistic Experimentation; company wide Exp Proficiency, Agility, Community for Decision Intelligence.

>How do orgs create DiQ via operational process standards?

Category 2: What stories are we telling ourself (experimentation/cro industry) about DATA right now?

Linkedin post with discussion

Data is a huge narrative for the CRO industry, and probably the most important current narrative IMO. Data is the core ingredient for a decision. The better the data, the better the decision.

Here are two core 'narratives' related to data that I see:

1. What is data efficacy

> As we shift from the race to collect data, to the race to best understand and action that data, our data processing infrastructure becomes crucial to success. So where are the gaps and how can we develop robust ‘rituals’ to better leverage the data for business decisions.???

2. Changing perspective on data for decision support (with EXP)

> What if EXP data was a product? What are the expectations of this product? How do you maintain this product? How do you continue to improve this product. (I see 'data-warehouse as the new backend' being a part of this narrative)

I am purposely skipping over 'statistics' as a core narrative as I think it's implied, always in the background at this point. It's the furniture in the scene that we're creating with the other stories.

Also see Chad Sanderson's recent post on this topic which is great.

Category 3: How experimentation helps orgs with innovation and change:

Linkedin post and discussion

1. Experimentation and the Speed Of Innovation**

> Creating confidence in product dev and innovation, with better faster decision making

2. Digital Transformation Investment without the research, data and experimentation culture to support it

> The illusion of being ready for digital transformation

3. Role of EXP in experience vS. role of EXP in business and product strategy

> The value of Exp beyond customer experience and conversion

This set of narratives speaks to a group of folks that are pushing for experimentation to be more a core part of a business operating systems. Manuel Da Costa is pushing a lot on this, and we see top tech firms like Netflix, Microsoft, Booking, Uber have testing in their DNA...sometimes to the degree that EXP 'centers of excellence' aren't even needed.

Search 'CRO' vs 'Experimentation' in Linkedin, under the 'people' category. In the first you'll see people with roles in marketing, in the second they will be on data and product teams. This narrative set above speaks to the need to bridge these worlds.

Second, two strategic narratives for why we experiment

Linkedin post and discussion

Why do we experiment? What would the company do otherwise? What are the stakes? How do companies that experiment win?

I've created two 'strategic narratives' using a rough version of Andy Raskin's framework for strategic narratives:

1. there's a new world
2. there's stakes/costs in this new world
3. therefore there's a new game to play, with winners and losers
4. but there's a cheat code, a magic capability we should know about
5. and the proof?

My two narratives have two audiences:

1. CMO/CPO - this is the PRACTITIONER narrative.

I'm sure most people reading this will be familiar with this one. It preaches to the choir. It speaks to the flood of (shitty) data at hand and the need to make decisions quickly. It speaks to the democratization of the randomized controlled trial and science principles generally. The narrative here isn't new, but I do like to see it broken out like this, it makes it graspable and wieldable in convos.

2. CEO - this is the RESOURCER narrative.

This is new, I don't see people talking about this but it's a HUGE hidden narrative. In small orgs, and even 'small' 100mil ecom orgs, the CEO is also the practitioner...so this narrative doesn't apply there.

This narrative is for larger, legacy companies (think any company that existed before the internet and still might have a fax machine). Large and mid-sized enterprises. They need innovation and digital transformation to stay relevant, they need to attract modern talent that wants autonomy in work and needs an operational system (experimentation) that pushes decision making down corporate ladders.

This strategic narrative speaks to the need for testing and decision system to promote the workplace to attract better talent. Do you want to work for Walmart or Walmart Labs??

Narrative #1 above has driven CRO growth and industry $ for the past 10 years. This $ will be eclipsed by the potential in narrative #2.

Think about what the consultative and change management agencies/firms like Accenture/McKenzie/Bain have been doing and charging. But now get tactical with proper data management and decision systems based on science. This new narrative will dominate a small group of agencies working on the bridge between digital transformation and practical data and CRO work.

To summarize, stories are powerful, and can be used to elevate a cause and an industry or group of practitioners. In the 1980s Sony didn't have a mission to elevate Sony, they wanted to elevate Japan. And it worked. For Japan and Sony. Lets point ourselves in the same direction, identify the problems and opportunities to change how business work. To democratize decision making and data into and down organization ladders.