A Problem With Aggregated Metrics

As a marketer, you likely spend a lot of time looking at data.

Here’s how to avoid a common analytics mistake: Simpson’s Paradox.

Simpson’s Paradox occurs when the trending direction of an aggregated metric is reversed if the supporting data is separated out into segments.

Here’s a simple example.

You check an average session duration on your website in a month-over-month view. You see a +33% increase. Looking good!

However, you double-click to break down desktop vs. mobile as audience segments. When you compare the average session duration for February against January, both desktop and mobile engagement decreased. It’s the opposite of what the previous table showed. What happened?

Hidden variables, such as the sample size or quality of grouped data sets, can be distorting.

Pulling in session count as an additional dimension reveals that sessions reapportioned in February. This created a weighted-average distortion that’s easy to miss, especially since metrics like avg. session duration are not often viewed with volume metrics like session count.

This phenomenon highlights the importance of good data intuition; data is often simplified into dimensional representations of a much more complex situation.

Common reporting tools like the Google Analytics reports contain entirely aggregated data, and are susceptible to fallacies like this one.

The solution:

• Use your intuition, and double-click into your data sets if something feels off.
• When running multi-variate tests, don’t stop a test until you have a high degree of statistical confidence and a low margin of error.
• Don’t generalize a test result without first exploring the dynamics and audience segments affecting behavior.
• Build your own data systems that allow you to use event-level data and reduce situations like these.

Continue to get to know your data better, because mining it might hold the keys to your most pressing growth challenges.