When A/B Tests Aren’t Possible, Causal Inference Can Still Measure Marketing Impact

Ever tried to prove a marketing change was effective, but you’re stuck without a clean A/B test? You’re not alone. In many real‑world settings—legal constraints, limited traffic, or budget—running a randomized experiment is simply impossible. In this post you’ll learn how to turn those constraints into opportunities by applying causal‑inference methods. We’ll walk through Diff‑in‑Diff, Synthetic Control, and Meta’s GeoLift, show how to prep your data, and provide ready‑to‑run code using libraries like causalpy and GeoLift.

1. When A/B Testing is Off‑limits

When you can’t do a classic experiment, you can still estimate causal impact by leveraging existing variation in the data—if you handle the assumptions carefully.

2. A Very Quick Primer on Causal Inference

All of these methods rely on observational data, so the key challenge is to approximate the conditions of a randomized trial through clever modeling and careful data prep.

3. Data Preparation – The Common Thread

1. Panel structure – you need observations over time for each unit (user, region, product, etc.).
2. Pre‑ and post‑treatment periods – at least 5–10 periods before and after the intervention.
3. Covariates – variables that predict the outcome and are not affected by the treatment (e.g., seasonality, marketing spend).
4. Treatment flag – a binary column that is 1 during the treatment period for treated units and 0 otherwise.
5. Balance diagnostics – check that treated and control units are similar on covariates before the event. Once you have a tidy dataframe, you can plug it into any of the libraries below.

4. Difference‑in‑Differences (Diff‑in‑Diff)

Diff‑in‑Diff is the workhorse of quasi‑experimental design. It’s simple, fast, and works well when you have a clear before/after signal and a suitable control group.

Assumptions

• Parallel trends: the counterfactual trend for the treated group would have matched the control group.
• No spill‑over: treatment on one unit does not affect the outcome of other units.

Implementation in Python (statsmodels)

import pandas as pd
import statsmodels.formula.api as smf

# df: long format with columns `y`, `treat`, `time`, `unit`
model = smf.ols(
    formula='y ~ treat * time',
    data=df
).fit(cov_type='cluster', cov_kwds={'groups': df['unit']})

print(model.summary())

The coefficient on treat:time is the Diff‑in‑Diff estimate.

Quick check: Parallel Trends

import matplotlib.pyplot as plt
pre = df[df['time'] < 0]
plt.figure(figsize=(8,4))
plt.plot(pre.groupby('time')['y'].mean(), label='Control')
plt.plot(pre.groupby('time')['y'].mean() + pre.groupby('time')['treat'].mean()*pre['treat'].mean(), label='Treated')
plt.axvline(0, color='k', ls='--') plt.legend(); plt.show()

If the lines are roughly parallel, your assumption is plausible.

5. Synthetic Control

When you have a single treated unit (e.g., a city that receives an ad campaign) and many potential controls, synthetic control offers a principled way to construct a counterfactual.

Concept

• Compute weights for each control unit so that the weighted average of their pre‑treatment outcomes matches the treated unit’s trajectory.
• After the treatment starts, the weighted control series becomes your counterfactual.

Library: causalpy

The research output shows how to run a synthetic‑control analysis with causalpy.

from causalpy import SyntheticControl
import matplotlib.pyplot as plt

# Suppose df_sc is a dataframe with columns:
# 'unit', 'time', 'Y', and a column 'treated' that is 1 for the treated unit
df_sc = causalpy.load_data('synthetic_control')
result_sc = SyntheticControl(
    df_sc,
    treatment_id=1,
    treatment_start=51,
    outcome_var='Y',
    control_ids=[2,3,4,5]
)
print(result_sc.summary())
fig, ax = result_sc.plot()
plt.show()

Interpreting the Output

Synthetic control is powerful but requires many pre‑treatment observations and a decent pool of control units.

6. GeoLift – Geo‑Targeted Incrementality

Meta’s GeoLift is an open‑source implementation of an augmented synthetic control tailored for advertising campaigns that target whole regions (cities, states, etc.). It adds a ridge‑regularized forecasting layer to improve out‑of‑sample performance.

Key Features

• Geographical focus – works at city or state level.
• Power analysis – helps you pick the right number of test markets.
• Augmented Synthetic Control – adds a ridge‑regularized regression on pre‑treatment data.

Quickstart in R

#  Install GeoLift
install.packages('remotes')
remotes::install_github('facebookincubator/GeoLift')
library(GeoLift)

#  Load example data
data(GeoLift_PreTest)   # 40 cities, 90 days pre‑campaign
data(GeoLift_Test)      # same cities, 15 days post‑campaign

#  Convert to GeoLift format
pre  <- GeoDataRead(
  data = GeoLift_PreTest,
  date_id = 'date',
  location_id = 'location',
  Y_id = 'Y',
  format = 'yyyy-mm-dd',
  summary = TRUE
)

post <- GeoDataRead(
  data = GeoLift_Test,
  date_id = 'date',
  location_id = 'location',
  Y_id = 'Y',
  format = 'yyyy-mm-dd',
  summary = TRUE
)

#  Market‑selection & power analysis
sel <- GeoLiftMarketSelection(
  data = pre,
  treatment_periods = c(10,15),
  N = 2:4,
  effect_size = seq(0,0.25,0.05),
  cpic = 7.5,
  budget = 100000,
  alpha = 0.1
)

#  Run GeoLift inference
gl_test <- GeoLift(
  Y_id = 'Y',
  data = post,
  locations = c('chicago','portland'),
  treatment_start_time = 91,
  treatment_end_time = 105
)
print(gl_test)
plot(gl_test, type = 'Lift')
plot(gl_test, type = 'ATT')

What the output tells you

• Lift – percent increase over the synthetic control.
• p‑value – statistical significance of the lift.
• Weights – which control cities contributed most to the counterfactual. GeoLift also offers a best = TRUE flag that automatically augments the synthetic control with ridge regression, giving tighter confidence bounds.

7. Choosing the Right Tool

Libraries you’ll need

8. Pitfalls and Best Practices

1. Check assumptions – Always plot pre‑treatment trends and balance tables.
2. Avoid contamination – Make sure control units are truly unaffected by the treatment.
3. Power matters – For GeoLift, run the built‑in market‑selection to avoid under‑powered tests.
4. Robustness checks – Try alternative control sets, add covariates, or use placebo periods.
5. Document everything – Store the exact data version, code, and results in a reproducible notebook or script.

9. Actionable Next Steps

1. Audit your data – Do you have a panel? How many pre‑treatment periods?
2. Pick a method – Start with Diff‑in‑Diff if you have multiple treated units; otherwise, go synthetic control or GeoLift.
3. Run a quick prototype – Use the code snippets above with your own data.
4. Validate assumptions – Visualize trends, check balance, and run placebo tests.
5. Interpret – Translate the statistical output into business terms (e.g., $X lift per 1,000 impressions).
6. Iterate – If results are inconclusive, adjust the control set or gather more data. You can now estimate causal impact without a clean AB test, turning observational variation into actionable insight.