How I Break Down Hypotheses to Make Them Easier to Test

For years I’ve been advocating for teams to reframe their product and feature ideas as hypotheses. There are four reasons:

1. Hypotheses reflect the real world doubt inherent in digital product development
2. Hypotheses change the definition of done from “works as designed” to achieving an outcome
3. Hypotheses tie every feature idea to a specific target user
4. Writing a believable hypothesis for your feature idea is the first test of that idea

When teams start writing hypotheses for the first time, though, they end up writing huge hypotheses. The scope of the hypothesis makes it difficult if not impossible to test. Here’s how I work through a big hypothesis to break it down into testable components.?

First, I break the feature down into individual steps

Here’s the hypothesis template that I use:

We believe that [this outcome]

Will be achieved if [these people]

Attain this [goal/benefit]

With [this feature].

A consumer lending team I was working with recently wrote this hypothesis for a new mortgage application process they were working on:

We believe we will increase mortgage application completion rates by 55% if first time homebuyers don’t have to prepare any documents upfront with a broadly integrated, end-to-end mortgage application process.?

I said to them, “Great! Go test that.” They looked at me and smiled. They knew there was no way to test this whole thing in one attempt. And it quickly dawned on them that they didn’t know where to start.?

The first thing I did was ask them to break down the components that made up the “broadly integrated, end-to-end mortgage application process.” This they could do!

The hypothetical system will:

• Not ask for any client documentation up front
• Integrate with a broad set of databases including:National credit reporting agenciesPayroll systemsReal estate sales systemsOther lending systemsSocial networksMLS (US real estate database)Geographic and address databases
• Include AI integration that will parse all of the relevant user data and provide practical next steps in the mortgage application process

Next, I asked them to think through the various risks involved in each of these components. What’s the most important thing to learn about each of them? The team chimed in with answers that varied from the valuable (Will anyone want this?) to the technical (Can we build this?) to the practical (How long might this take to build?).?

Anytime you’re building a new system, regardless of how technically complex it might be, testing the value of the idea is the best place to start. If no one wants it, it doesn’t matter how hard or easy it is to build it.?

Next, I identify the learning goal

The team settled on a value question – will first time home buyers feel comfortable starting and completing a mortgage application process without having to do any preparation in advance? Most people assume this is an onerous process and one that doesn’t make that heavy of a request may seem sketchy, untrustworthy or worse, fraudulent.?

Notice how far we’ve already come. We started with “end to end integrated system” and are now already focusing on a very specific step in the customer journey. Will people even give this thing a shot?

Now that we had a question, I asked the team to tell what they wanted to learn about the question. They made a list:

• What makes a mortgage application system trustworthy?
• What do you expect the process to be? How long should it take?
• What would make a mortgage application process seem untrustworthy?
• What would make you abandon a mortgage application?
• Does AI enhance or erode the trustworthiness of a mortgage application process?

There were more questions but this list gave us a clear direction. We needed to get a better sense of what first time homebuyers trust a mortgage lending system.?

Finally, I can design a simple experiment

If, “What’s the most important thing we need to learn next?” is the opening question in our test and learn loop, “What’s the least amount of work we need to do to learn that?” is the closing question. When we started out, the hypothesis was so huge the only realistic answer to the second question was, “build the whole thing” – the riskiest and most expensive way to find out if something works as we hoped.?

Now, however, we have a clear question around the trustworthiness of mortgage application systems. The smallest thing we can do to learn that seems much more feasible. We can easily:

• Interview first time home buyers before they apply for mortgages
• Interview them after they’ve applied and/or bought a home
• Observe users as they apply for mortgages

The data the team collects with these experiments starts to inform other risks they’ve identified and they have a better direction of where to head next. Each time a cycle is complete, the questions come up again, “What’s the next most important thing we need to learn?” We take another part of the hypothesis and work through it. In a perfect world we derisk the whole thing and build it over time. In the real world, we’ll adjust our plans, pivot and in some cases decide not to build the thing at all.?

Practice Makes Perfect

The more I work with teams, the more hypotheses they write, the better they get at them. They become more specific, smaller and, most importantly, testable. However, most teams will start with huge hypotheses. The process I’ve described here has worked for me week over week for nearly 15 years in breaking these down with teams to get them unstuck and moving forward in their discovery and delivery process. I hope it works for you too.?