Determining the Right Sample Size for Reliable Validation of Your SaaS Business Model

Intoduction

In this article, I describe the process of determining the sample size for interviews and experiments used in validating SaaS business models, and the implications of sample size on associated risks.

In today’s increasingly data-driven business landscape, the validation of SaaS B2B business models remains reliant on traditional tools such as customer interviews and experiments (Talking to Humans , Testing with Humans ) especially at early stages of validation of initial idea / business model. While low- and high-fidelity prototyping and data insights support this process in later stages, interviews and deep experiments with customers remain indispensable (Hypothesis-Driven Entrepreneurship: The Lean Startup).

The rationale behind the predominance of interviews and experiments in B2B SaaS validation is clear: the quest for repeatable and scalable patterns amidst the complexity of business problems and processes, compounded by the unique characteristics of each enterprise. Only through direct engagement with process owners, users, and decision-makers can the necessary insights be collected.

Despite their importance, determining the requisite number of experiments with customers (or interviews) can be a challenge even for seasoned Product Managers. Let’s take an example, we are considering a market of, say, 100 production plants. The question arises: How many confirmations are needed to validate the business hypothesis? Surprisingly, the answers vary widely among Product Managers.

In response to this ambiguity, I propose a short guide that draws upon my expertise in statistics, experience in conducting A/B tests for ML systems, and track record in validating and investing in B2B SaaS products. This guide will outline a methodology for determining the sample size necessary for Product Managers to validate their business hypotheses.

• Part #1 of the guide will delve into the fundamentals of statistics for sample size selection, laying the groundwork for subsequent discussions
• Part #2 will explore the application of these principles to the unique context of B2B SaaS validation, providing actionable insights tailored to this domain
• Part #3 offers simplified practical guidance on implementing the proposed methodology in real-world scenarios, equipping Product Managers with easy to use tools they need to navigate the validation process with confidence

By bridging the gap between theoretical statistical concepts and practical application in the realm of B2B SaaS validation, this guide aims to empower Product Managers to make informed decisions and drive successful outcomes for their businesses.

Part #1: fundamentals of statistics for sample size selection

Let’s take a simple example. Imagine, that you are a doctor residing in a small village of 500 inhabitants, you’ve discovered the spread of a disease impacting all demographics. Suppose you estimate that around 30% of the population is affected. To confirm this with 80% confidence and a margin of error of ±5%, you’ll need to calculate the necessary sample size.

Thanks to the statistics, this problem was solved ages ago. You can find more details here on selection of sample size and justification quality metrics for experiments (sample size justification ), and here with a strong focus on A/B-test , even the sample size calculator

To estimate the disease’s prevalence accurately, you utilize a proportion sample size formula that considers the total population size (N), the estimated proportion affected (p), the desired confidence level (Z-score), and the margin of error (E). Specifically:

where:

• n is the sample size needed
• Z is the z-score corresponding to the desired confidence level (for 80% confidence, Z approx 1.28)
• p is the estimated proportion of the population with the disease (30% or 0.3 in this case)
• E is the margin of error (5% or 0.05 in this case)

This formula assumes an infinite population. However, because the village population is finite (500 people), we need to use the finite population correction formula:

where

• n_adj is the adjusted sample size accounting for the finite population,
• N is the total population size.

Let’s calculate the required sample size.

To have an 80% confidence that the prevalence of the disease in your village is 30% ± 5%, you should test approximately 108 people from the population of 500. This sample size will give you a statistically reliable estimate within the specified margin of error.

At the end to determine sample size n we need to know the following parameters

Now let’s transform this simple case to the case of business model validation for the business model we are considering for investing.

Part #2 will explore the application of these principles to the unique context of B2B SaaS validation

Assume we are evaluating a business model for a SaaS product tailored for Automotive OEMs that manufacture passenger vehicles with an annual output exceeding 20,000 units. Our initial business operations will be launched in Germany.

The primary objective of this validation is to ascertain whether the customer needs, which our SaaS product intends to meet, actually exist within this target market. This scenario mirrors the earlier example involving disease prevalence; here, however, the ‘condition’ we’re assessing is the presence of a specific business need among potential customers.

To determine the appropriate sample size for this validation, and to apply the statistical formula we’ve discussed, it is essential to accurately define the following three parameters:

• Population Size: The total number of Automotive OEM plants in Germany that meet the criteria of producing more than 20,000 vehicles annually. This defines the scope of our target market
• Population Proportion (Where Hypothesis is Positive): The estimated percentage of these plants that are likely to have the business need our product addresses. This is akin to estimating the prevalence of a condition in a population
• Margin of Error: The acceptable range within which we are willing to tolerate error in our estimation. This determines how precise our conclusions need to be to proceed confidently with business decisions

By meticulously specifying these parameters, we can employ the adjusted sample size formula to ensure our business model validation is both statistically reliable and relevant to our market entry strategy in Germany.

Determining the Appropriate Market Estimation for Sample Size

When preparing to validate a business model, especially in specialized sectors such as SaaS for Automotive OEMs, one must first decide which market estimation to consider for the basis of population size. The terms TAM, SAM, and SOM are pivotal in this decision-making process:

• TAM (Total Addressable Market): Represents the entire global demand for a product or service, illustrating the ultimate revenue potential if one were to achieve 100% market share. It is the broadest metric, providing an overview of the total market opportunity
• SAM (Serviceable Available Market): Refers to the segment of TAM that is within your product’s or service’s geographical reach and is accessible under current business capabilities. This market estimation is more focused and realistic than TAM, as it only includes those potential customers that can realistically be served
• SOM (Serviceable Obtainable Market): The fraction of SAM that is realistically achievable within the near future, taking into account existing competitors and other market constraints. It represents a more conservative and immediately targetable segment

For the purpose of validating a SaaS business model for Automotive OEMs, the most relevant metric is the SAM. This measure accurately reflects the subset of the market that not only has the need for the product but is also within the operational reach of the business, thus providing a practical target for initial market entry and hypothesis testing. Unlike SOM, SAM is not limited by current market share or competitive dynamics, which might unduly narrow the scope of potential customers without reflecting the full extent of market needs.

Applying SAM to Our Scenario: In our specific case, the SAM consists of passenger vehicle production plants operated by Automotive OEMs within the DACH region (Germany, Austria, and Switzerland), supplemented by an additional 10% from other parts of Europe to account for broader market engagement. This effectively places our target population size at around? 50 plants. This figure may vary slightly based on brand representation and specific geographical focus but serves as a solid basis for conducting a statistically valid market validation.

Determining Population Proportion and Margin of Error for Customer Need Validation

Estimating Customer Need: The crucial question in validating our SaaS business model is to estimate the proportion of the target population that exhibits the specific business need our product addresses. The most robust method to ascertain this is through a detailed analytical estimation, which involves understanding the intricacies of the customer’s problems and how they align with our solution. An alternative, more straightforward approach is to utilize established B2B sales benchmarks to gauge customer interest during the sales process, specifically looking at the conversion rates at critical stages.

Sales Process and Lead Qualification: In our sales funnel, a pivotal point of interest is the transition of leads to the Sales Qualified Lead (SQL) stage. At this stage, a lead is considered qualified based on certain criteria, primarily the presence of a need and the authority to make purchasing decisions, although not necessarily having confirmed budget and timing (commonly referred to by the acronym BANT: Budget, Authority, Need, Time). The subsequent opportunity stage focuses more on our ability to close the sale rather than on the customer’s initial need, which is the primary focus of our hypothesis validation.

Using Sales Benchmarks to Estimate Need: According to SaaS sales benchmarks, the progression from SQL to an actual sales opportunity (Opp) typically occurs with a probability of 30–40%. For the purpose of our validation, we can reasonably set the estimated population proportion (where the hypothesis of customer need is positive) at an average of 35%. This figure strikes a balance between the lower and upper bounds of our observed conversion rates, providing a realistic estimate of the market segment that could potentially benefit from our product.

The sales process stages and the relevant one (taken from here: https://firstpagesage.com/seo-blog/b2b-saas-funnel-conversion-benchmarks-fc/ )

Setting the Margin of Error: To accommodate potential variances in our estimation and ensure robustness in our business model validation, we set a margin of error at 5%. This allows us to account for uncertainties and fluctuations in customer behavior and market conditions without significantly compromising the accuracy of our validation process.

Estimating sample size

To confirm the customer need hypothesis with different levels of statistical confidence and precision, we apply the previously discussed formula for sample size in proportion studies. Below are the results for varying confidence levels and margins of error, based on the assumptions specified for our customer validation scenario:

1. High precision (low error margin = 5%):

1a) 80% Confidence:

• Z-Score: 1.28
• Estimated Need (p): 0.35
• Margin of Error (E): 5%
• Required Sample Size: 38 customers

1b) 70% Confidence:

• Z-Score: 1.04
• Estimated Need (p): 0.35
• Margin of Error (E): 5%
• Required Sample Size: 34 customers

2. Low precision (high error margin = 10%):

2a) 80% Confidence:

• Z-Score: 1.28
• Estimated Need (p): 0.35
• Margin of Error (E): 10%
• Required Sample Size: 22 customers

2b) 70% Confidence:

• Z-Score: 1.04
• Estimated Need (p): 0.35
• Margin of Error (E): 10%
• Required Sample Size: 17 customers

These calculations provide a robust framework for determining how many customers you should validate with to have a statistically reliable estimate, guiding investment decisions and product development strategies with empirical evidence.

Let’s summarize in the table

In statistical sample size calculations, the margin of error (E) and the confidence level (reflected in the Z-score) are both crucial factors that influence the required number of observations. However, it is observed that changes in the margin of error tend to have a more pronounced impact on the sample size compared to changes in the confidence level. This effect is due to the mathematical formulation used in determining sample sizes for population estimates:

• Square of the Margin of Error: The margin of error appears in the denominator of the sample size formula as E2. A smaller margin of error (indicating a need for higher precision) substantially increases the required sample size because it reduces the denominator significantly.
• Square of the Z-Score: While the Z-score (which adjusts for the desired confidence level) also appears in the formula, it is not squared directly; rather, it multiplies the squared term of the estimated proportion and its complement. The influence of the Z-score on sample size is less dramatic because changes in confidence levels typically result in smaller proportional changes in the Z-score values.

For example, decreasing the margin of error from 10% to 5% more than quadruples the required sample size, assuming all other factors remain constant. In contrast, increasing the confidence level from 70% to 80%, which involves a smaller relative change in the Z-score, results in a less substantial increase in the required sample size.

This analysis demonstrates why precision (as dictated by the margin of error) is a more sensitive parameter in sample size calculations than confidence (as adjusted by the Z-score). Understanding this dynamic is crucial for researchers and analysts when designing studies or validation tests to achieve both practical and statistically robust outcomes.

Part #3: Practical Usage of Sample Size Calculations

Understanding Risk and Decision-Making Implications

When we apply the 70% confidence level with a 10% margin of error, practical usage suggests that interviewing 17 customers might be sufficient to gauge if the need is relevant for between 25% and 45% of the potential customer base. This broader range implies a higher risk compared to a more precise approach:

• Risk of Higher Customer Acquisition Costs: If the actual need is at the lower end of the estimated range (25% instead of the average 35%), the cost per acquisition could increase significantly. For instance, costs could rise from €100k to €140k, increasing by a factor of 1.4 due to fewer customers than anticipated requiring the service.
• Potential for Lower Revenue: Similarly, if the need prevalence is lower, revenue projections based on the average need might also be overstated by the same factor.

Strategic Implications for Business Decisions

The choice between these two sampling strategies?—?low precision versus high precision?—?depends on the specific goals of the validation effort:

• Low Precision Validation (Fast Check): Appropriate for initial explorations of customer interest or market need, where speed is prioritized over precision. This approach uses a 70% confidence level and a 10% margin of error. It’s typically used when the goal is to quickly assess the viability of a concept before committing more resources.
• High Precision Validation: Necessary for detailed assessments related to pricing strategies, customer acquisition costs, and lifetime value calculations. Here, a higher confidence level of 80% and a tighter margin of error of 5% are used to ensure that the data collected is robust enough to inform significant financial decisions and long-term strategies.

Empirical guidance for Product Managers

Below, I propose an empirical framework that suggests the number of interviews needed for both low and high precision validations, tailored to the customer segment size:

• Empirical Table: This table would list different segment sizes?—?10, 20, 50, 100, 500, 1000 customers?—?and provide corresponding sample sizes for both validation approaches. For example, a segment of 100 might require 20 interviews for low precision and 61 for high precision, offering product managers a practical tool for planning their validation studies based on specific market segments

What comes?next?

Next articles I will write about building up SaaS business cases, Unit Economics and their metrics and how to validate them.

PS

Thanks to Simon Weiss and Felix Hajdinjak for reviewing the article!

References

1. Talking to Humans: https://www.talkingtohumans.com/
2. Testing with Humans: https://testingwithhumans.com/
3. Hypothesis-Driven Entrepreneurship: The Lean Startup: https://www.hbs.edu/faculty/Pages/item.aspx?num=41302
4. Sample Size Justification: https://www.researchgate.net/publication/359409294_Sample_Size_Justification
5. Sample size calculator: https://www.calculator.net/sample-size-calculator.html
6. A/B Testing: A Complete Guide to Statistical Testing: https://towardsdatascience.com/a-b-testing-a-complete-guide-to-statistical-testing-e3f1db140499
7. B2B sales pipeline benchmarks: https://firstpagesage.com/seo-blog/b2b-saas-funnel-conversion-benchmarks-fc/