How precise does measurement need to be?

We have all heard we need precise metrics and analyses to make decisions. On the surface, it makes sense.

However.

When we look closer, we find several factors contribute to the precision of the analysis and can conflict the decisions that need to be made. One factor to consider in measurement planning is how long will it take to get the precision necessary for a decision. Most analysts will want a longer time for the analysis to ensure a higher level of precision and confidence interval.

That makes sense, but the longer the time period of the analysis the more factors can influence the results. For instance, an A/B test is in market and the analytics team is saying that six months are necessary to get the statistically valid analysis. The problem is that in those six months changes were made to the sales force, the competitor reduced their prices, and a new competitor entered the market. All of these can affect the test and may not be evenly distributed across test and control groups.

Determining what decision will be made from the analysis is critical in advance. Deciding before the campaign launches of the analysis timeline ensures that the marketing decisions are made in the best possible circumstances.

How long will it take to get precise measures?

The longer it takes, the more likely other factors can influence the results.

How precise does the analysis need to be?

What decisions will be made from the analysis?

Next up tomorrow: Is personalization worth the effort?