Marketing mix shift quick analysis

The concept

When a key metric changes, there’s often a clear driver that impacted all users or customers. Your checkout process is broken, you launched a special sale, you kicked off a loyalty program, and so on. But sometimes there isn’t an obvious reason, and a deeper understanding of underlying marketing channels (or other user segments) is required. This allows you to determine whether the cause is missed performance in a given segment, or a change in size of a segment relative to other segments. The latter example is mix shift, and it can meaningfully impact if it persists.

The example

Let’s say your conversion rate in the latest week dropped from 5.05% the week before to 4.90%, and you want to know what caused the drop. You decide to start looking at the behavior of traffic coming through each of your 3 marketing channels.

So you put together a basic data table with traffic and conversion by each channel, and the totals:

Channel	Last week visits	Last week’s conversion (CVR)	This week visits	This week’s conversion (CVR)
Search	30	2.00%	50	1.90%
Email	100	6.00%	110	6.20%
Direct	70	5.00%	75	5.00%
All channels	200	5.05%	235	4.90%

The math

First things first, let’s do some quick math to make the data more usable:

Channel	Last week share (channel / all channels)	This week share (channel / all channels)	Change in share (This week - Last week)	Change in CVR (This week - Last week)	Last week CVR diff from avg. (channel - all channels)
Search	15%	21%	6.3%	-0.10%	-3.05%
Email	50%	47%	-3.2%	0.20%	0.95%
Direct	35%	32%	-3.1%	0.00%	-0.05%
All channels	100%	100%	0.0%	-0.15%

The formulas used to calculate each field are right below each column header in the table above. Here are the steps we went through:

Translate ‘visits’ to ‘share’, in order to make traffic apples-to-apples from one week to the next.
Calculate ‘change in share’ to understand how each channel changed in size relative to other channels.
Calculate the ‘Last week conversion diff from avg.’ to understand whether a channel improves or hurts conversion when its share increases.

Now, to understand the different effects:

Channel	Intra-channel (Last week share) X (Change in CVR)	Mix shift (Last week CVR - avg) X (Change in share)	Interaction (Change in share) X (Change in CVR)	All effects
Search	-0.02%	-0.19%	-0.01%	-0.21%
Email	0.10%	-0.03%	-0.01%	0.06%
Direct	0.00%	0.00%	0.00%	0.00%
All channels	0.09%	-0.22%	-0.01%	-0.15%

Intra-channel represents the portion of the change in conversion due to changes within a channel. In this case, Email generated a positive effect of 0.1% because its conversion increased from 6% to 6.2%.
Mix shift represents the portion of the change due to changes in the relative size of a channel. In this case, Search generated a negative effect of -0.19% because its share increased from 15% to 21% of the total traffic.
Interaction represents a second-order effect caused by a combination of mix and intra-channel. It’s the least actionable effect, but it’s also usually much smaller than the other effects.

As you can see, there are many effects taking place that impacted the drop in conversion by 0.15 percentage points down to 4.90%, but the Search channel mix effect stands out as the largest. Search, which has a much lower conversion than other channels, increased meaningfully from 15% to 21%. If this channel remains elevated, you’ll need to ensure that the underlying assumptions and economics account for this change lest it hurts overall profitability.

Conclusion

Depending on how you look at it, mix shift can explain the large majority of changes in key metrics. Even special sales and loyalty programs represent segments that change the overall numbers when increasing or decreasing in size. Locating the mix shift and understanding why it happened is key to ensuring that your numbers don’t run askew and that your business stays healthy in the long run.

Note - article was ported from a deprecated version of this blog