80% of Agile companies are building the wrong things faster

80% of Agile Companies are building the wrong things faster, according to Jeff Patton

Let's dig into the science

There are two types of evidence:

1. Experimental

2. Observational

Let's take a look at how the medical field does science. This is called a hierarchy of evidence, and they use it to rank knowledge in what they refer to as "evidence-based medicine". The most abundant but weaker form of evidence is at the bottom of the pyramid.

Users Lie

At the bottom are observations/correlations. This is analyzing some data and finding trends. We have all heard that correlation is not causation. User Interviews, Surveys, and looking at Analytics are some evidence that can lead you down the wrong path.

Actions Don't Lie

Experimental evidence is the strongest form. Nothing is stronger than working software in the hands of users. We are getting real interaction with the product. But we don't need to build the product to get this evidence. Not even the MVP.

A good Continuous Product Discovery process will consist of experimental and observational research methods. Product experts such as Teresa Torres and Marty Cagan will advocate for a good mix, but they heavily focus on experiments. There is a reason for this.

If your discovery process consists of only observational research, then the evidence will be weak. You have an 80% chance of acting on information that will fail and your CEO will be mad at you. Don't just talk to users, experiment on them.

So how do you experiment?

The experiment must be conducted in a real environment with real people for hard unbiased evidence. This means your product and tech stack must be treated as a research tool.

How should your toolset work?

1. Select a sample population or cohort.

2. Show a mock of the feature to those users.

3. Collect user behavior and feedback.

4. Analyze the result.

If you're not building this into your tech stack, then some tools can help you here:

Feature Toggle - LaunchDarkly , Optimizely , and Google Optimize

Analytics - Amplitude , Heap | by Contentsquare , and Segment

Feedback - Sprig , @chameleonio Chameleon , and Intercom

You will need to write many scripts to connect all these tools just to show something to a sample population and collect some feedback when the user interacts with the mock. Many teams can't even afford these tools or build their own, especially if they have to be lean.

We have to coordinate with UX and Engineering just to place a simple mock or fake door in front of users. On some teams, there is no capacity and the setup cost is significant. Look at the number of tools you need.

Why can't Product Managers just drag a component from their design systems like Figma , Sketch , or Storybook right into their website, turn it on to a segment of users, and get results in a single dashboard?

The field is so fragmented that it costs so much to experiment.

We hypothesize that we can make this process seamless and affordable for everyone involved. Get up and running with a fake door test in minutes with our integrations to CDPs like Amplitude here: