Startups and the Art of Asking Questions

One of the primary purposes of a startup is to effectively explore a vast search space. Startups are inherently case studies in optimization?—?trying to determine the “best fit” solution to a problem (and in some cases, the best problem to fit an existing technology or solution). The space of possible solutions is typically vast, poorly understood, and poorly explored. At any given time, there are many directions in which the company can go, and many different outcomes that result by following each path.

“Two roads diverged in a wood…”, except that the there are dozens or hundreds of roads, and they are not so much roads as faint markings on the ground (if that).

The primary purpose of an early startup team is to construct the experimental apparatus necessary to effectively traverse the solution space. This apparatus consists of people, processes and tools at a minimum.

This is not to say that startups proceed by simply randomly trying out product experiments?—?there are often strong intuitions about the right path to take, based on the prior experiences of the early team or other factors. However, the experimental apparatus helps the company ask and answer questions of the market at a high velocity.

The questions can be very broad or very narrow, very exploratory or very directed, very qualitative or very quantitative. But once posed, the job of the company’s experimental apparatus is to get to a reasonably good answer?—?typically by building product features?—?as quickly as possible. Startups cannot wait for unambiguous answers to emerge, so “good enough” is often good enough. The team needs to be able to:

1. Come up with a metric to optimize for
2. Create a hypothesis about what product strategy will move the needle on that metric
3. Devise a feature that implements that product strategy
4. Ship that feature quickly?—?and with good instrumentation?—?to test whether that feature actually does move that metric.

In this model, the metric is usually beyond reproach, or obviously worth striving for?—?it is something fundamental, like “get more users” or “get more visits per user per month”. The rest of the steps are usually much more controversial, involving a mix of intuition, qualitative customer discovery, and data from past experiments.

A sample set of steps might be:

1. We want to optimize the metric “users who log in to our product”
2. Our hypothesis is that making login easier will make significantly more users log in?—?move the needle on this metric
3. The product strategy is to implement login by email, so that users don’t have to remember a password
4. Our engineers will work closely with product managers to define what metrics we need to collect to track the success of this feature?—?perhaps the ratio of homepage visits to successful sign-ins, or something else. In addition, we will release this feature as an A/B test for 20% of users.

The experimental apparatus necessary to be able to ask such questions of the market at a high velocity typically consists of the following:

1. People: Engineers, product managers, designers, analysts
2. Process: The techniques the team uses to ship high quality software quickly, including techniques for greenlighting ideas, defining an MVP, doing technical due diligence, writing bug-free code, testing, and deployment
3. Tools: Continuous integration systems, QA tools, A/B testing tools, clickstream generation libraries, analytics visualization tools

I’ll write more about building out these areas in future blog posts.

It’s useful to think about this experimental apparatus as the core piece of an early stage startup. Being able to ask questions of the market quickly and efficiently is one of the fundamental building blocks that early startup teams must strive to put in place as quickly as possible.