Why standard errors do not apply
With one treated unit there is no sampling distribution in the usual sense. Synthetic control inference asks a design question instead: is the effect we found for Province A unusual compared with the "effects" we would find if we pretended each donor had been treated? That is a permutation test (Module 4), applied across units.
!pip install -q numpy pandas scipy matplotlib
import numpy as np, pandas as pd
from scipy.optimize import minimize
import matplotlib.pyplot as plt
rng = np.random.default_rng(42)
def synth_weights(y_pre, Y_pre):
J = Y_pre.shape[0]
res = minimize(lambda w: np.sum((y_pre - w @ Y_pre) ** 2), np.full(J, 1 / J), method="SLSQP",
bounds=[(0, 1)] * J, constraints={"type": "eq", "fun": lambda w: w.sum() - 1})
return res.x
# Same world as Lesson 12.1, with a larger donor pool
years = np.arange(2005, 2025); T0 = 2018; pre = years < T0; J = 15
factors = np.column_stack([np.ones(len(years)), (years - 2005) / 10, np.sin((years - 2005) / 3)])
donors = rng.uniform([80, 0, -5], [140, 30, 10], size=(J, 3)) @ factors.T + rng.normal(0, 1.5, (J, len(years)))
treated = np.array([0.4, 0.5, 0.1] + [0] * (J - 3)) @ donors + np.where(years >= T0, -15.0, 0) + rng.normal(0, 1, len(years))
units = np.vstack([treated, donors]) # row 0 = Province A
names = ["Province A"] + [f"Donor {j}" for j in range(J)]
In-space placebos
For each unit, treat it as if it were the treated one, build its synthetic control from the other units (excluding the real treated unit from donor pools), and record the gap.
def gap_for(i, all_units, pre_mask):
others = [k for k in range(len(all_units)) if k != i and k != 0] # never use the real treated unit as donor
w = synth_weights(all_units[i, pre_mask], all_units[others][:, pre_mask])
return all_units[i] - w @ all_units[others]
gaps = np.array([gap_for(i, units, pre) for i in range(len(units))])
plt.figure(figsize=(10, 5))
for i in range(1, len(units)):
plt.plot(years, gaps[i], color="grey", alpha=.4)
plt.plot(years, gaps[0], color="red", lw=3, label="Province A")
plt.axvline(T0, color="black", ls=":"); plt.axhline(0, color="black", lw=.5)
plt.title("In-space placebos: gap for every unit"); plt.legend(); plt.show()
RMSPE ratios and a permutation p-value
Some placebo units are poorly fitted before the date, so their gaps are large everywhere. Abadie et al. normalise by fit: for each unit compute
ratio = post-period RMSPE / pre-period RMSPE
A real effect produces a large post-period gap relative to a small pre-period gap. The p-value is the rank of the treated unit's ratio among all units.
rmspe = lambda g: np.sqrt(np.mean(g ** 2))
ratios = pd.Series([rmspe(g[~pre]) / rmspe(g[pre]) for g in gaps], index=names).sort_values(ascending=False)
p_value = (ratios >= ratios["Province A"]).mean()
print(ratios.round(2).head(6))
print(f"Permutation p-value: {p_value:.3f} (smallest possible with {len(units)} units: {1 / len(units):.3f})")
With 16 units, the smallest attainable p-value is 1/16 ≈ 0.06. Small donor pools limit what inference can say, however large the effect. Report the rank and the plot, not just a p-value.
In-time placebos and leave-one-out
In-time placebo: pretend the programme started in 2013, using only data before 2018. If the method finds a large "effect" in 2013–2017, its fit is not trustworthy.
fake_T0 = 2013
fake_pre = years < fake_T0
window = years < T0
w_fake = synth_weights(treated[fake_pre], donors[:, fake_pre])
fake_gap = (treated - w_fake @ donors)[window]
print(f"Average 'effect' of the fake 2013 programme (2013-2017): {fake_gap[years[window] >= fake_T0].mean():.2f}")
Leave-one-out: drop each donor with positive weight in turn and re-estimate. If the effect depends on one particular donor, say so.
w = synth_weights(treated[pre], donors[:, pre])
for j in np.where(w > 0.01)[0]:
keep = [k for k in range(J) if k != j]
w_j = synth_weights(treated[pre], donors[keep][:, pre])
print(f"without Donor {j:>2} (weight {w[j]:.2f}): effect {(treated - w_j @ donors[keep])[~pre].mean():.2f}")
Exercise
Re-run the in-space placebo analysis with a true effect of −3 instead of −15. What happens to Province A's rank and p-value? What does that imply for evaluating modest policies with synthetic control?
Show a solution
treated_small = treated - np.where(years >= T0, -15.0, 0) + np.where(years >= T0, -3.0, 0)
units_small = np.vstack([treated_small, donors])
gaps_small = np.array([gap_for(i, units_small, pre) for i in range(len(units_small))])
r = pd.Series([rmspe(g[~pre]) / rmspe(g[pre]) for g in gaps_small], index=names)
print(f"Rank of Province A: {int((r >= r['Province A']).sum())} of {len(r)} | p = {(r >= r['Province A']).mean():.3f}")
# Small effects are hard to distinguish from placebo noise with few donors: SCM has limited power.
Checklist and recap
- I can run in-space placebos and plot all gaps.
- I compute RMSPE ratios and a permutation p-value, and know its minimum.
- I run in-time placebos and leave-one-out checks.