The Pitfall of Averages in Campaign ROI Success for e-commerce marketing businesses is often represented in terms of Return on Investment (ROI), and while it is convenient to think of ROI as low value, looking specifically at average ROI can be very misleading. Classic campaign measurement tends to compute a single point estimate (it's usually the mean ROI) implicitly believing everything behaves like a pleasant symmetrical bell curve around that point value. Marketing outcomes can have "fat tails": rare but extreme events on the downside which conventional models’ underestimate. By focusing on average ROI alone, we miss out on such negative tail events that might underperform the most successful campaigns at a market stage (or worse yet, a company's reputation). As one marketing postmortem put it, many “marketing fails” come about when one of your smartest campaigns "sparks negative viral attention that outweighs results". A colossal backlash or PR disaster in the end, in other words, can obliterate the positive side of dozens of ordinary campaigns. For instance, let’s suppose a fictional online retailer ShopCo carries out a cheeky piece of social media marketing. It typically anticipates a healthy 20% ROI. But there’s a very small likelihood that the campaign goes viral for the wrong reasons, might possibly an insensitive slogan or violating the policy inspires public outrage. The campaign will probably face big losses (lost sales, refunds, PR damage) in that rare example. Even traditional analysis might overlook this possibility because it is “only 1 percent likely.” But if it occurs, the downside effect is so considerable that it could nullify many successful campaigns. This is the essence of tail risk: low probability, high consequence adverse events that the conventional ROI calculations ignore. Just as in finance, where missing fat-tail market crashes led to nasty surprises, marketing analysts also need to defer to the „fat tails“ of consumer behavior and public opinion. When Marketing Campaigns Go Wrong: The Downside Tail We’ve all seen campaigns that blow up in a bad way. A classic example was American Eagle’s “Great Jeans” ad starring Sydney Sweeney, which “went viral for the wrong reasons” which is a clever tagline was considered tone-deaf, prompting online push back. The backlash drove a slew of hits but little sales lift, highlighting that extreme negative reaction can swamp the average benefits. The marketing teams have begun to realise that planning campaigns should be viewed as risk management, not just creative production. That is to predict not just the desired ROI but also the worst-case scenario if it goes south. Negative tail events in marketing come in all different guises: A social ad gets banned for policy violations (frequently crashing traffic, thus wasting spend), an influencer program fucks up, or a tag-along campaign is used to steal a hash tagged campaign. Such events are rare, so they hardly register in average ROI metrics, but they have outsized downsides. A common reporting method might say “Campaign ROI = +15%” and say no again, losing sight of the 1 in 100 probabilities that ROI is –100% (that is, total loss) as a result of a disaster. To bypass these tails is to invite disaster. Just as prudent asset managers are concerned about “black swan” losses exceeding three standard deviations, prudent marketers should be concerned about just that one-in-a-thousand campaign fiasco event that might destroy a brand’s quarter. Embracing Uncertainty with Bayesian Modeling How can we explain away these unlikely-but-impactful events? And this is where Bayesian probabilistic programming comes in. As opposed to classical analytics, which are a typical point estimate of the point, Bayesian modeling treats ROI as a complete probability distribution – encompassing in your input all probabilities and all the way through and the possible consequences of every individual uncertainty. When we fit a Bayesian model (such as we use with PyMC and NumPyro libraries), our model outputs not only, “ROI = 0.2” but it is approximately ~0.2 on average for the average but with long left tail (left tail). We obtain a posterior predictive distribution: basically, a set of plausible distributions of possible ROI values to the right (or posterior predictive distribution), weighted with respect to their probabilities given our data and prior knowledge. This posterior distribution is a treasure trove for risk analysis. Bayesian methods are crucial, however; it is easy to transition away from naive normal assumptions. For example, if we think that our campaign outcomes are skewed or have heavy tails (which often will happen in real data), we can then select a better likelihood. In fact, a good example would be to model campaign ROI with either Student’s t-distribution or a mixture model that captures outliers on the rare occasions. With a tool like PyMC, often it's as simple as swapping pm.Normal for pm.StudentT in the model specification. The implications of such a decision are substantial: the model will intuitively give greater predictability to extreme negative ROI events, the odds as in real life. For instance, in a finance context, if a heavy-tailed Student’s T model was used the Bayesian forecast would show “higher likelihood of extreme negative returns than on the normal assumption”. The analogy holds true for marketing ROI: a Bayesian scheme acknowledging fat tails will expect a higher probability of a campaign blow-up than a naive model accounting for mild, Gaussian variation. Also, we already knew that the parameter uncertainty is intrinsic to our Bayesian inference process. If you have limited campaign data, a typical Bayesian model reflects this uncertainty in larger predictive distributions. Rather than being too confident or underestimate the risks in your tail paths, you'll see a spread out of some possible ROI values, some of which may be quite negative. This makes for a more genuine risk assessment. Treating model parameters as if they were fixed can “lead to severe underestimation of tail risks”, whereas, the PyMC Labs team says, a Bayesian approach can “help practitioners avoid overconfidence and better understand the range of possible outcomes”. In a nutshell, Bayesian programming makes us accept uncertainty instead of skating over it. Modeling ROI Distribution (With a Little PyMC) How would we implement this for our fictional ShopCo campaign? We start by specifying a probabilistic model. For example, suppose past marketing campaigns suggest an average ROI of around 20%, but with occasional disastrous outcomes. We might encode this as: Baseline ROI ~ Normal(μ, σ) for typical variability (most campaigns hover near μ).
Rare disaster event ~ with small probability p, ROI draws from a separate “disaster” distribution (e.g. very negative). Baseline ROI ~ Normal(μ, σ) for typical variability (most campaigns hover near μ). Rare disaster event ~ with small probability p, ROI draws from a separate “disaster” distribution (e.g. very negative). In practice, we can build a mixture model or use a heavy-tailed distribution to capture this. Using PyMC pseudo-code, it could look like: with pm.Model():
    p = pm.Beta("p_bad", 1, 19)
    good_roi = pm.Normal("good_roi", mu=0.2, sigma=0.1)
    bad_roi = pm.Normal("bad_roi", mu=-0.6, sigma=0.3)
    # Mixture likelihood: ROI comes from good_roi with probability (1-p), or bad_roi with probability p
    roi_obs = pm.Mixture("roi_obs", w=[1-p, p], comp_dists=[good_roi, bad_roi], observed=campaign_data) with pm.Model():
    p = pm.Beta("p_bad", 1, 19)
    good_roi = pm.Normal("good_roi", mu=0.2, sigma=0.1)
    bad_roi = pm.Normal("bad_roi", mu=-0.6, sigma=0.3)
    # Mixture likelihood: ROI comes from good_roi with probability (1-p), or bad_roi with probability p
    roi_obs = pm.Mixture("roi_obs", w=[1-p, p], comp_dists=[good_roi, bad_roi], observed=campaign_data) Once this model is fitted to past campaign data (or with informed priors if disaster data is sparse), we can simulate the posterior predictive ROI for a new campaign. If PyMC’s sample_posterior_predictive is used, this resulting distribution will produce thousands of ROI outputs from the learned distribution. The net result is not a single ROI forecast but a range of return potentials – some with high or moderate returns, some with low or negative returns. Why is this valuable? Because we can now quantify stuff like: “There’s a ~5% possibility ROI is worse than –50%.” We’re able to calculate Value at Risk (VaR) and Expected Shortfall (CVaR) for the campaign ROI just as finance people do for portfolio return. Instead “Bayesian inference” creates a strong method for estimating the VaR/CVaR, as it “reflects not only the risk in future returns but also uncertainty in model parameters”. And, unlike a fixed VaR formula, Bayesian posterior provides a conditional distribution for extreme cases which should be used for scenario analysis, and stress tests, by sampling from those tails. Predictive Distributions vs. Point Estimates I find it fascinating to see the difference between an estimate of ROI and a distributional prediction. Think again about ShopCo’s campaign. A traditional analyst might say “we see 20% ROI, give or take 10%” and act under a bell curve. Using tail-risk modeling a Bayesian might say “we are still expecting ~20% average but are a tiny bit closer to a much worse outcome.” The above Chart shows that both predict a 20% average ROI (their peaks at ~0.2 x-axis). The naive normal model (blue) has a thin left tail though – no chance of a worst ROI (< -20%). However, still in this simulation, its 5% worst-case ROI remained positive. By contrast, the Bayesian heavy-tailed model that (orange) represents is a much fatter left tail: a small but very powerful bloc of probability at –60% ROI (or worse, if we permitted it). The 5th percentile (VaR₅₀) for the orange distribution is approximately –8%, which means that at least 8% ROI can be lost 1-in-20 times. And the expected shortfall (CVaR) which is the average risk with that worst 5% which is even worse (roughly –30% in this case). The naive model grossly underestimates risk; it would force you to think extreme losses are almost impossible, while the tail-aware model recognizes that they’re plausible. A tail-aware predictive distribution, used to solve questions like: “What is the probability that our ROI is negative, or our losses outstrip our campaign budget?” and “If it does go wrong — how bad can it get on average?”. Measuring Downside: VaR and Expected Shortfall Now let us unpack those risk metrics. Value at Risk (VaR) at 95% confidence (the standard choice) basically asks: “What threshold of ROI will we cross 95% of the time?” Or for that matter “There is only 5% chance ROI falls below X.” If, then 1-in-20 campaigns went to perform worse than –20% ROI. VaR, on the other hand, has a weak area: It shows the cutoff, but “is indifferent to anything beyond its threshold”. If the 5% worst cases are –21% or –80%, that’s not differentiated by standard VaR – it only knows they are beyond –20%. This is where Conditional Value at Risk (CVaR), or Expected Shortfall (ES), comes in. CVaR is the average outcome in that worst 5% tail. It answers, "If the worst-case zone happens, how much do we lose on average?" Example: VaR would be -20%, CVaR could be -60% (meaning if things go wrong, usually you lose 60%). CVaR offers a more detailed understanding of tail risk overall and wins out for its conservative approach - even in volatile scenarios. As an important point made by Investopedia, "while safer investments often have small CVaRs, those with significant upside potential usually exhibit larger CVaRs, highlighting their risk–reward trade-off" and it is the same for marketing campaigns: the ones promising great ROI (e.g. viral hits) might also carry tremendous downside risk (e.g. viral flops). Mathematically, if  is ROI, the CVaR at 5% (for losses) can be written as: X → campaign ROI (random variable) VaR₅%(X) → 5th percentile of ROI X ≤ VaR₅%(X) → the worst 5% of outcomes E[ · ] → average (expected value) the expected ROI considering that we are in the worst 5% tail. (For most cases, if ROI is positive, it will be a negative number representing net loss.) Bayesian simulation enables the estimation of such a metric – simply look at the posterior predictive samples of ROI, identify the bottom 5% and average them. No need to use closed-form equations or use hard distribution rules. Tail-Risk–Adjusted Decision Making Half the battle is knowing the distribution of outcomes. The other half of it is doing something about it. How do we mitigate downside ROI risk when selecting campaigns? We need a strategy that doesn’t get blinded by high average ROI, but punishes campaigns that incur nasty tail risk. Optimization of a risk-adjusted objective function is one proposed way. For instance, one decision could be to optimise a “risk-adjusted ROI” described by: Here is a risk aversion factor (of the deal we’re making between the mean return and lower tail risk). If for simplicity, this essentially means we choose the campaign with the highest average ROI less its expected loss in the worst 5% cases. A campaign with some really big upside but an extreme downside can still hit low CVaR (risk-adjusted) or have quite a negative CVaR (adjusted risk). This is but one example – in reality one could restrict your efforts to “only select a campaign whose 5% worst-case is not < –X%” or create a portfolio of campaigns with high CVaR for optimal return. The key concept is to make sure that you spell out a tail-risk in your decision criteria. Let’s bring back our fictional example to see this in action. Suppose ShopCo has five potential campaigns (A, B, C, D, E) for the next quarter. Their estimated average ROIs and downside risks (CVaR at 5%) are as follows: Campaign

Average ROI

5% worst-case ROI (CVaR)

Selected by naive?

Selected by tail-aware?



A

20%

–60%

Yes

No



B

18%

–20%

No

Yes



C

15%

+3%

No

Yes



D

10%

+5%

No

No



E

25%

–100%

Yes

No Campaign

Average ROI

5% worst-case ROI (CVaR)

Selected by naive?

Selected by tail-aware?



A

20%

–60%

Yes

No



B

18%

–20%

No

Yes



C

15%

+3%

No

Yes



D

10%

+5%

No

No



E

25%

–100%

Yes

No Campaign

Average ROI

5% worst-case ROI (CVaR)

Selected by naive?

Selected by tail-aware? Campaign Campaign Average ROI Average ROI 5% worst-case ROI (CVaR) 5% worst-case ROI (CVaR) Selected by naive? Selected by naive? Selected by tail-aware? Selected by tail-aware? A

20%

–60%

Yes

No A A 20% 20% –60% –60% Yes Yes No No B

18%

–20%

No

Yes B B 18% 18% –20% –20% No No Yes Yes C

15%

+3%

No

Yes C C 15% 15% +3% +3% No No Yes Yes D

10%

+5%

No

No D D 10% 10% +5% +5% No No No No E

25%

–100%

Yes

No E E 25% 25% –100% –100% Yes Yes No No Here, a naive selection strategy (predicting campaigns by highest average ROI) would select Campaign E and Campaign A – Campaign E and Campaign A have the tantalizing top-line numbers of 25% and 20% ROI. Campaign E, though, is a gamble: it has a small chance of a disastrous –100% outcome (maybe it’s an edgy viral stunt that can either spike or blow up into hell), which is the point. Campaign A also conceals a prospective –60% tail event. The tail-risk–aware approach prefers Campaign B and Campaign C with slightly lower expected returns (18% and 15%), respectively, but much milder worst-case scenarios (at worst circa –20% for B) and Campaign C’s worst 5% outcomes delivering only a slight beneficial return on investment! Campaign D is very safe (almost no downside), yet the return is lowest, so neither strategy appears to favor it as much as the alternatives. The contrast is striking. Naive selection selects an average, chasing a high one, and therefore can result in two very risky bets, whereas tail-aware selection seeks safety in the tails. If the average performance is similar for both options, naive strategy wins (average ~22.5% vs ~16.5% ROI). But the naive picks could implode if a tail event happens – e.g. if Campaign E goes south, that –100% loss would obliterate all of the gains, giving the naive selection a terrible overall result. Those tail-aware picks, with limited downside, would be able to weather the storm and yet still produce modest positive returns. This is similar to an investment portfolio itself: High flying stocks can deliver greater average returns, but a couple of nasty crashes can wipe out the portfolio as a whole. Over few weeks, we can understand the performance of model as this below. Towards a Safer (and Smarter) Marketing Strategy Now, the takeaway should be obvious: Distributional and tail-risk learning makes smarter marketing decisions. Rather than the myopic view of campaigns (average ROI or determinism), we measure them as an intelligent investor would measure assets — by the return distribution across the period, but also their loss tail left. And Bayesian probabilistic programming is applicable even when things get complicated. With PyMC or NumPyro we can create custom models that learn to cope with our marketing-specific idiosyncrasies (seasonality, virality etc.) and take uncertainty and rare events naturally into account. We get posterior predictive distribution of ROI and can compute metrics such as VaR or CVaR easily. With those then we can create a tail-aware campaign or risk-adjusted optimization which will not funnel our entire budget into a one-potential “Pepsi ad fiasco.” In practice, taking this approach doesn’t mean becoming overly conservative and not engaging any campaign with risk. It means we assess the risk-reward trade-off more globally. We can take on a risky campaign but do so mindfully, perhaps we spend a smaller amount going into the high-tail-risk experiments, then we have a stable campaign that is all the better for it. Or we purchase “insurance” in the shape of PR crisis management plans (akin to hedging a tail risk). The thing is that we’re not being blindsided by the downside. Just as Bayesian finance models provide “stronger estimates of possible losses” and demonstrate extreme scenarios more effectively, Bayesian marketing models can flag the kind of scenarios in which a campaign might tank. From our conclusion, our work, distributional & tail risk learning for marketing ROI – whether it is distributional and tail risk learning for marketing ROI is simply about an expectation of unexpected events. We can use Bayesian programming to model the risks at the margins – bad viral hits, policy bans, and bad PR nightmares – then the bad viral hits, and the pandemic-buzzy PR nightmares at the perimeter using Bayesian programming models, to model how things that could happen – we can also keep our e-commerce projects protected from the downside surprises, in order to prevent such negative surprises. The next time you plot a daring marketing stunt, don’t just calculate “expected ROI.” Inquire about the worst-case ROI and probability of it. If those answers are not going to make you sweat any more, a Bayesian tail-risk model will save you a long, hard time (and a good budget!) And in the uncertain world of digital marketing, this model can transform “unknown unknowns” into known risks with measurable outcomes empowering data scientists and marketers to develop plans that can extract the highest upside and plan for the possible tail points with confidence.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

How Bayesian Tail-Risk Modeling can save your Retail Business Marketing Budget

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

From Time Series to Causal Scenarios: A Statistical Guide to Counterfactual Forecasting

What Are Convolution Neural Networks? [ELI5]

The Noonification: Have U Been Pwned? (1/12/2023)

Goldman Sachs, Data Lineage, and Harry Potter Spells

People are still crazy about Python after twenty-five years

10 Questions to Consider when Setting up a Corporate A.I project

From Time Series to Causal Scenarios: A Statistical Guide to Counterfactual Forecasting

What Are Convolution Neural Networks? [ELI5]

The Noonification: Have U Been Pwned? (1/12/2023)

Goldman Sachs, Data Lineage, and Harry Potter Spells

People are still crazy about Python after twenty-five years

10 Questions to Consider when Setting up a Corporate A.I project

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps