The concept
When a key metric changes, there’s often a clear driver that impacted all users or customers. Your checkout process is broken, you launched a special sale, you kicked off a loyalty program, and so on. But sometimes there isn’t an obvious reason, and and a deeper understanding of underlying marketing channels (or other user segments) is required. This allows you to determine whether the cause is missed performance in a given segment, or a change in size of a segment relative to other segments. The latter example is mix shift, and it can meaningfully impact if it persists.
The example
Let’s say your conversion rate in the latest week dropped from 5.05% the week before to 4.90%, and you want to know what caused the drop. You decide to start looking at the behavior of traffic coming through each of your 3 marketing channels.
So you put together a basic data table with traffic and conversion by each channel, and the totals:
| Channel | Last week visits | Last week’s conversion | This week visits | This week’s conversion |
|---|---|---|---|---|
| Search | 30 | 2.00% | 50 | 1.90% |
| 100 | 6.00% | 110 | 6.20% | |
| Direct | 70 | 5.00% | 75 | 5.00% |
| All channels | 200 | 5.05% | 235 | 4.90% |
The math
First things first, let’s break do some quick math to make the data more usable:
| Channel | Last week share | This week share | Change in share | Change in conversion | Last week conversion diff from avg. |
|---|---|---|---|---|---|
| (channel / all channels) | (channel / all channels) | (This week share - Last week share) | (This week conversion - last week conversion) | (channel - all channels) | |
| Search | 15% | 21% | 6.3% | -0.10% | -3.05% |
| 50% | 47% | -3.2% | 0.20% | 0.95% | |
| Direct | 35% | 32% | -3.1% | 0.00% | -0.05% |
| All channels | 100% | 100% | 0.0% | -0.15% |
The formulas used to calculate each field are right below each column header in the table above. Here are the steps we went through:
- Translate ‘visits’ to ‘share’, in order to make traffic apples-to-apples from one week to the next
- Calculate ‘change in share’ to understand how each channel changed in size relative to other channels
- Calculate the ‘Last week conversion diff from avg.’ to understand whether a channel improves or hurts conversion when its share increases
Now, to understand the different effects:
| Channel | Intra-channel | Mix shift | Interaction | All effects |
|---|---|---|---|---|
| (Last week share) X (Change in conversion) | (Last week conversion diff vs avg) X (Change in share) | (Change in share) X (Change in conversion) | ||
| Search | -0.02% | -0.19% | -0.01% | -0.21% |
| 0.10% | -0.03% | -0.01% | 0.06% | |
| Direct | 0.00% | 0.00% | 0.00% | 0.00% |
| All channels | 0.09% | -0.22% | -0.01% | -0.15% |
- Intra-channel represents the portion of the change in conversion due to changes within a channel. In this case, Email generated a positive effect of 0.1% because its conversion increased from 6% to 6.2%
- Mix shift represents the portion of the change due to changes in the relative size of a channel. In this case, Search generated a negative effect of -0.19% because its share increased from 15% to 21% of the total traffic.
- Interaction represents a second order effect caused by a combination of mix and intra-channel. It’s the least actionable effect, but it’s also usually much smaller than the other effects
As you can see, there are many effects taking place that impacted the drop in conversion 0.15 percentage points down to 4.90%, but the Search channel Mix effect stands out as the largest. Search, which has a much lower conversion than other channels, increased meaningfully from 15% to 21%. If this channel remains elevated, you’ll need to ensure that the underlying assumptions and economics account for this change lest it hurts overall profitability.
Conclusion
Depending on how you look at it, mix shift can explain the large majority of changes in key metrics. Even special sales and loyalty programs represents segments that change the overall numbers when increasing or decreasing in size. Locating the mix shift and understanding why it happened is key to ensuring that your numbers don’t run askew and that your business stays healthy in the long run.