3 non-obvious experimental design principles, that will dramatically improve how you run experiments -in software and in science

I often talk to either product people or founders, and everyone universally agrees that using the scientific method is important. Even more importantly, everyone would tell you that they are are building an experimental organization. However, once you look at the way they are running experiments, unfortunately in 90%+ of cases, the system to experiment doesn’t actually yield good experiments.

One of the ‘easiest’ ways to see how experiments aren’t run very well, is A/B testing. This twitter thread highlights how ineffective A/B testing can be in ‘average’ circumstances:

Fundamentally, the scientific method is actually largely about asking useful questions the right way. The mistake most people make is they assume they should test everything. This is simply false. In fact, when I was a scientist (I was a lab scientist for 9 years, having gotten my PhD), one of the most important things I did was figure out what not to test - either because it was unimportant, or important to hold constant.

So here are 3 non-obvious experimental design principles, that will dramatically improve how you run experiments:

1. Learn the prior art in the field, and then talk to the relevant people about their learnings
2. Know what assumptions were being made for others’ experiments, and what’s changed
3. Know whether you are running a ‘discovery’ based process or a ‘hypothesis’ based process

Prior art and talking to the right people

It may seem obvious that the first thing to do is to see what has already been done. In science, this is by reading papers, and in software, it’s by looking at other products, and their evolution over time. However, most people stop there. This is the big gap. What gets published is the ‘last 5%.’ 95% of the work is hidden when all you read is the final output. So a lot of the learning is in the author’s experience and unpublished work. The only way to get it, is to reach out to them and talk to them.

The most common reasons people don’t reach out to the author:

• They don’t think about doing so - somewhat simple
• They think reaching out is competitive - this is unlikely, the bigger problem is likely whether enough people care about what either of you is doing
• They don’t think the other person will reply - as Wayne Gretzky said: You miss 100% of the shots you don’t take

However, talking to the author is generally very useful. I have done this every time, as a scientist, a product leader, and now as a founder. People will happily share the most useful information so you don’t have to relearn the 95% that was unpublished. As long as you are genuine in your curiosity and generous with your own learnings.

With this knowledge, you are now much better equipped to decide what experiments to run, and what aspects you can assume will pan out a certain way. So lean on this approach as much as possible.

Knowing what assumptions were made, and what’s changed

In conjunction with learning from others’ work and experiences, an important factor is understanding what assumptions were previously being made, and what assumptions are changing. I experienced this first as a scientist, where I asked a professor whether a certain reaction would occur, to which he said no. It turned out his assumption based for why the answer was no wasn’t valid in my case. So when I ran the experiment, it worked. The important part of this anecdote was in discovering what assumptions were being made.

The same thing is true in software. A subset of this is the ‘why now’ question people often ask. Which is really a question of what assumptions are changing that enable a new opportunity. But sometimes, nothing obvious has to change. There could just be a latent opportunity that exists as a consequence of an incorrect assumption that was held for a long time. Probably the best examples of this are places where founders built companies in what appeared to be a ‘solved’ market - like zoom in video conferencing, or google search. Nothing actually changed, but prioritizing the right hidden variables enabled very large, successful companies to be built. Google’s decluttering of the home page is perhaps the best known specific example, where everyone assumed you needed an information dense landing page - turns out, maybe you don’t.

Discovery vs Hypothesis based experiments

This is probably the dimension I hear people talk about the least often. There are two approaches to running experiments.

Discovery based experimentation assumes that if you take enough approaches, you’ll discover something interesting. Fundamentally, this is the most empirical of approaches one can take, where you don’t rely on any previously stated theory per se. A lot of great science and businesses are built this way. In fact this is how most pharma companies operate to develop a new drug - they try a very large number of compounds and see which one has an effect on the pathway of interest. Or for that matter 95%+ of A/B testing is discovery based - no prior theory, just testing till there is an observable result.

Hypothesis based experimentation on the other hand requires that there is some theory that you are trying to prove is true empirically. This is what most people think they are doing, even when they are often actually doing the above. The most important thing you need in this approach is a sound theory that help you understand how to design the rest of the experiment. In science, the best example is almost all of quantum mechanics. Most of quantum mechanics is easier to derive in theory, with experiments to prove the theory coming after. In business, this is also highly effective, it just doesn’t happen that often. For example, Tesla is run largely hypothesis based - where things are first derived in theory, and then the operations push the limits of approaching the theoretical designs.

I don’t necessarily have a strong point of view on which is better. I think the approach that is best depends on the field, the people, and the resources available. However, the most important thing is knowing which approach you are taking so you can design the experiments correctly, and with a reasonable perspective on how you are going to find good solutions.

Put these principles to the test, and let me know how they work for you!