The Power of Business Experimentation: 5 Questions with Alex Atzberger

Alexander Atzberger is the CEO of Optimizely , a digital experimentation platform that helps businesses conduct website testing at scale, enabling developers to optimize the customer experience.?

Alex, business experimentation is obviously a major part of what you offer as a company. What is your overall perspective on the subject?

For me it’s both a philosophy and a practice. It’s a whole mindset of being a learning culture, of not being afraid to fail, being able to understand how to build a good hypothesis and test. And giving your team a way to do that with the right tools. As a personal practice, oftentimes as a CEO I say, “Look, we have a new business idea. I’m not sure if it’s the right direction. How fast can we find out if this is a good or bad idea, and how can we get some data?” A lot of it is about velocity. It’s probably our most important operating goal, and the same is true for our customers. I talk to their leaders all the time, and it’s “How quickly can I get the information I need to this or that?”

When you look at what we can do for our customers, it’s clear that they have been able to capture a great deal of value in terms of growth by actualizing this mindset, of just quickly trying things out and learning by testing everything. The abundance of data now available makes that all feasible, where it wasn’t a few years ago. Being digital is so much of the customer experience today, a primary touchpoint. It becomes super easy to quickly understand what may or may not work, which speeds up innovation and decision-making and ultimately growth.

When it comes to creating that learning culture, what are some of the things you do?

Culturally, we talk a lot about values, especially the behaviors that?embody those. We have these five values, and we designate leadership time to discuss those. What behaviors do we as an organization need to embrace around those? One is to keep asking ourselves, Are we actually doing what we tell our customers they should be doing: experimenting? And are we applying it ourselves? We have made some acquisitions where the company culture is not experimental, and none of the people have ever really done this sort of thing before. They would just build and launch and then fix it when things went wrong. Which they did a lot. So we have to insist on it. I talk about it constantly in all-hands meetings. It takes time to permeate the organization. But it is making a difference. Where people used to come to me and ask, for example, if they can have this or that in the top navigation of our site, now they find out for themselves through experiments and data and don’t even come to me anymore. That’s what I like to see. That’s how I know we are making progress.

Is there a common experimentation methodology used across Optimizely?

We have a five-step process that we have actually published a short playbook on. Of course, it begins with a research step where you collect relevant customer data, both qualitative and quantitative. Next we look at that data and identify the problem we want to solve for customers, then settle on a goal and metric for an experiment. Then we create a hypothesis that states our belief about what needs to change, what the outcome will be, and why. Finally, we roll out the experiment to a small subset of users, then review results to see if the outcome has proven or disproven our hypothesis. We want to share the results internally and define next steps with data-driven decisions.

What do you find is your biggest cultural challenge around experimentation??

There is a tension around not wanting to do little tests all the time versus being a part of a big project. There can be this bipolar feeling that yes, I want to experiment, but let’s do it with a big idea. I’ve pulled some big projects because I just didn’t see an angle of how to use experimentation on something. I think the first thing that you can do as a leader is just ask the team, “Have we looked at it differently? Have we actually tried to define a much smaller project that would give us some data?” There are always times you may have to make decisions in the absence of data, and that’s judgment, but for business ideas, looking for evidence needs to come first. Then we can make a better decision. And then what would the data look like that would inform us if it’s really a better decision or not? The only way I know how to get to that is through running an experiment, or experiments.

As CEO, are there any specific focal points that are your top priorities?

Yes. One of the things I continually test with the team is how we pitch our solutions. Because that’s a very simple thing to test. And you don’t say, “This is the pitch, it’s not going to change.” You say, “Look, this is going to be changing as quickly as we get feedback.” I’m forever driving to make it better and better. I think too many times people decide to be perfect on the first go. It doesn’t need to be perfect; it needs to be better through trying new approaches, soliciting feedback, listening, and learning what we can do to improve it for the next time. I think this whole notion about incrementalism as a means of transformation is a powerful concept. I get exposed to big-budget decisions that are black and white, yes or no. A lot of people want a big, giant, one-shot transformation. The math just doesn’t pencil as well as taking one thousand smaller, faster, cheaper steps that let you get better each time. Oh, and with one thousand times less risk.

[Adapted from What a Unicorn Knows: How Leading Entrepreneurs Use Lean Principles to Drive Sustainable Growth (Matt Holt Books/BenBella Books, 2024)]