Outsmarting Bias with Experiments

Ever stuck to your guns even when the evidence said otherwise? Or maybe you ignored some facts because they didn't fit your worldview? These are examples of how biases can creep into our lives, both personal and professional.

In the world of product development, biases can be a real roadblock. There are different types of bias; Cognitive Biases, Social Biases, Research Biases, and Unconscious Biases.

In this article let's explore how things like confirmation bias (cherry-picking data that supports our ideas) and optimism bias (overestimating our chances of success) which are sub topics of cognitive bias can lead to product flops.

Confirmation Bias

Confirmation bias is the tendency to seek out and interpret information that confirms our existing beliefs while ignoring or discounting information that contradicts them. For example, if you're a die-hard fan of a particular sports team, you might constantly predict they'll win, even when the odds are stacked against them.

In a product scope this can be impacted in the following ways;

• Ignoring the haters: nobody enjoys negative feedback. But ignoring it can be a major mistake. Imagine your team launches a new app feature, only to find users are frustrated and confused by it. Instead of listening to their concerns, you stubbornly push ahead, forcing users to adapt.
• Jumping the gun: Sometimes, we need to test different approaches to see what works best. For example, you might be trying to figure out which of two promotional links gets more clicks. But once you see early results favouring one link, you might prematurely end the test, missing out on valuable data.
• "My way or the highway": We can get so attached to our own ideas that we fail to consider alternatives. Let's say you're developing a new product feature that costs heavily. To cover the costs, you have to increase the product price. But maybe there were simpler, more cost-effective ways to achieve the same goal.
• Assumptions-based limited user base: Testing your product with a select group of users and assuming those results apply to everyone is a recipe for disaster. For example, you might test a new feature with a group of existing teen users and find it's a hit. But when you release it to the wider public, you discover most users don't find it useful at all.

Optimism Bias

The tendency to overestimate the likelihood of positive events and underestimate the likelihood of negative events.

• Unrealistic timelines and budgets:??underestimate the time and resources required to develop and launch a product, leading to delays and budget overruns.
• Inadequate risk management: overlook potential risks and challenges, leading to unpreparedness when unexpected issues arise.
• Poor market research: conducting the research to support the optimistic view and ignoring the appropriate samples.
• Ignoring warning Signs : ignore the warning signs due to high optimism. The warnings could be indicated from similar products in the market, regulatory concerns etc.

You might be thinking, "Isn't it a good thing to be optimistic? Shouldn't we try to confirm our beliefs?" And you'd be right! A healthy dose of optimism is important. However, it's crucial to maintain a balanced perspective and not let it blind us to potential risks, issues, and potential failures.

So how can we make sure we're not letting our biases get the best of us? The answer lies in conducting product experiments.

Product Experiments

Product experimentation is the process of systematically testing hypotheses about how to improve a product. It involves conducting controlled experiments, such as A/B testing, to gather data and make informed decisions about product features, design, and user experience.

What are the types of experiments?

• A/B Testing: known as split testing or bucket testing. Provides the ability to compare two versions of a single element.

Example: Testing two different email subject lines to see which one has a higher open rate.

• Multivariate Testing: tests multiple variations of multiple elements simultaneously.

Example: Testing different headline styles and button colors and image placements on a landing page to find the optimal combination.

• Usability Testing: observe users interacting with your product to identify usability issues and gather qualitative feedback.

Example: gather a group of users and observe users navigate a website to identify areas of confusion or frustration.

• User Interviews: conduct one-on-one interviews with users to gather in-depth insights into their needs, motivations, and experiences with your product.

Example: releasing a beta version and interviewing early adopters of a new feature to understand their satisfaction and identify areas for improvement.

• Focus Groups: facilitate discussions with small groups of users to gather feedback and insights on specific product features or concepts.

Example: conducting a focus group to discuss user reactions to a proposed redesign of the product's homepage.

• Beta Testing: release a limited version of your product to a select group of users for early feedback and testing.

Example: releasing a beta version of a new mobile app to a group of early adopters to identify bugs and gather feedback on the user experience.

• Conjoint Analysis: determine how users value different product attributes by asking them to choose between hypothetical product options.

Example: asking users to choose between different pricing tiers with varying features to understand their willingness to pay.

• Fake Door Testing: gauge user interest in a potential new feature by making it "clickable" even though it's not yet implemented.

Example: Adding a "Request Early Access" button for a new feature to see how many users click it.

• Event Tracking: track user clicks on different elements within your product to understand user behaviour and identify areas of interest.

Example: Tracking which links users click most frequently on a product page.

• Session Recording: record user sessions to observe their interactions with your product and identify areas of confusion or frustration.

Example: recording user sessions on a website to see how they navigate through different pages.

How can product experiments support overcoming biases?

Confirmation Bias

• Experiments provide measurable data, which leads you to view the reality of the product performance rather than relying on interpretations.
• Feedback from users from different backgrounds helps to identify the blind spots.
• Continuous testing and refining will support to capture the evidence that proves or disproves the initial hypothesis.

Optimism Bias:

• Provides insights into the actual performance of the product which helps to adjust the expectations.
• Capture the risks and support to defined mitigation steps.
• Experiments provide data to base the decisions to remove the influence of overly optimistic assumptions.

Let's be honest, no one's perfect. We all have our biases. But by embracing experimentation, we can create a feedback loop that helps us learn, grow, and build products that truly delight our users. Isn't that what we're all striving for?

#productdevelopment #experimentation #coginitivebias #productmanagement #businessanalysis