# The concept

When comparing trends in large chunks of data, it’s easy to arrive at the wrong conclusions when you don’t understand the underlying data in full context. When a segment makes up a bigger piece of the pie in one group vs. another, that segment can over-influence the aggregate results. If you are unaware of this over-influence, you could make the wrong decision for your business.

# The example

Let’s say you are deciding on two types of advertising for a nationwide promotion coming up. To aid in your decision, you run each ad in test city and monitor performance for 5 days.

After the 5 days are over, you put together a basic data table by day and by city, to compare conversion rates:

Day City A visits City A conversion City B visits City B conversion City A vs City B
1 50 10.00% 60 9.50% +5.3%
2 75 10.00% 90 9.50% +5.3%
3 (holiday) 200 5.00% 150 4.75% +5.3%
4 75 10.00% 90 9.50% +5.3%
5 75 10.00% 90 9.50% +5.3%
————– —————— ——————– —————— ——————– ———————
Totals 475 7.89 % 90 8.02% -1.5%

You notice that when you compare the totals for each city, that city A has a 1.5% lower conversion rate than city B, but when you review the performance by day, city A wins by 5.3% every single day. So which city had the better results and which ad should you choose?

# The math

If you look at the visits by day, you can see that on day 3, city A had 200 visits, which represents 42% of the total visits for city A in the 5 days. City B had 150 visits on that day, which represents 31% of the total visits. So even though both cities had much lower conversion than normal on day 3, and city A had a 5.3% higher conversion that city B, the impact of that bad day on city A’s totals was much higher than on city B’s totals because the bad day was over-represented in city A relative to city B.

If you had chosen the ad that ran in city B because of the overall better performance, you potentially would have given up the 5.3% additional orders you would have gained had you chosen city A.

# Conclusion

Make sure to dig a bit deeper and review your data across a few different (and relevant) segments, to ensure there aren’t inherent biases in the data driven by differences in mix. It can make a substantial difference in your outcomes and prevent big surprises down the line.

Contents