The question
In 1993 Texas began a massive expansion of its prison capacity, more than doubling the number of beds
within a few years. Did the extra capacity raise incarceration? Cunningham's Causal Inference: The
Mixtape uses synthetic control to answer this for Black male prisoners (bmprison), with the other US
states as donors. Only one state was treated, so this is a textbook SCM setting.
!pip install -q numpy pandas scipy matplotlib causaldata
import numpy as np, pandas as pd
from scipy.optimize import minimize
import matplotlib.pyplot as plt
from causaldata import texas
Data preparation
tx = texas.load_pandas().data
wide = tx.pivot(index="year", columns="state", values="bmprison").sort_index()
wide = wide.dropna(axis=1) # keep states with complete series
years = wide.index.to_numpy(); T0 = 1993; pre = years < T0
treated = wide["Texas"].to_numpy()
donor_names = [s for s in wide.columns if s != "Texas"]
donors = wide[donor_names].to_numpy().T
print(f"{len(donor_names)} donor states, {pre.sum()} pre-treatment years ({years.min()}-{T0 - 1})")
Abadie et al. also match on predictors (income, unemployment, poverty, demographics) averaged over the pre-period, with predictor weights V chosen to minimise the pre-treatment outcome error. A transparent version: standardise predictors, stack them with selected pre-period outcomes, and fit the weights.
preds = ["income", "ur", "poverty", "black", "perc1519", "alcohol"]
X = tx[tx.year < T0].groupby("state")[preds].mean().loc[["Texas"] + donor_names]
X = (X - X.mean()) / X.std() # standardise
def scm(target, pool, pre_mask, x_target=None, x_pool=None, v=0.5):
"""Minimise (1 - v) * pre-period outcome error + v * predictor error (both standardised)."""
scale = target[pre_mask].std()
def loss(w):
out = np.mean(((target[pre_mask] - w @ pool[:, pre_mask]) / scale) ** 2)
pr = 0 if x_target is None else np.mean((x_target - w @ x_pool) ** 2)
return (1 - v) * out + v * pr
J = pool.shape[0]
return minimize(loss, np.full(J, 1 / J), method="SLSQP", bounds=[(0, 1)] * J,
constraints={"type": "eq", "fun": lambda w: w.sum() - 1}, options={"maxiter": 500}).x
Estimation and placebo inference
w = scm(treated, donors, pre, X.loc["Texas"].to_numpy(), X.loc[donor_names].to_numpy(), v=0.2)
synthetic = w @ donors
print(pd.Series(w, index=donor_names).sort_values(ascending=False).head(6).round(3))
gap = treated - synthetic
print(f"Average gap 1993-2000: {gap[~pre].mean():,.0f} prisoners | in 2000: {gap[-1]:,.0f}")
fig, axes = plt.subplots(1, 2, figsize=(14, 4))
axes[0].plot(years, treated, "r", lw=2, label="Texas"); axes[0].plot(years, synthetic, "b--", lw=2, label="Synthetic Texas")
axes[0].axvline(T0, color="k", ls=":"); axes[0].legend(); axes[0].set_title("Black male prisoners")
axes[1].bar(years, gap, color=np.where(pre, "#94a3b8", "#ef4444")); axes[1].set_title("Gap (Texas − synthetic)")
plt.tight_layout(); plt.show()
Now the in-space placebos, using outcomes only to keep things fast. State prison populations differ by orders of magnitude, and with only eight pre-years some small states are fitted almost perfectly, which makes the post/pre RMSPE ratio explode for reasons unrelated to any effect. Two standard adjustments: express gaps relative to each state's pre-period level, and drop placebos with poor pre-fit (here, a pre-period RMSPE above 10% of the state's level), as Abadie et al. do.
all_units = np.vstack([treated, donors]); names = ["Texas"] + donor_names
rows = {}
for i in range(len(all_units)):
level = all_units[i][pre].mean()
if level <= 0:
continue
others = [k for k in range(len(all_units)) if k not in (i, 0)]
g = all_units[i] - scm(all_units[i], all_units[others], pre) @ all_units[others]
rows[names[i]] = {"post_gap_pct": g[~pre].mean() / level, "pre_rmspe_pct": np.sqrt(np.mean(g[pre] ** 2)) / level}
placebo = pd.DataFrame(rows).T
good = placebo[placebo.pre_rmspe_pct <= 0.10].sort_values("post_gap_pct", ascending=False)
print(good.head(5).round(3))
print(f"Texas rank: {list(good.index).index('Texas') + 1} of {len(good)} well-fitted units | "
f"p = {(good.post_gap_pct >= good.loc['Texas', 'post_gap_pct']).mean():.3f}")
The expansion was followed by a rise in Black male incarceration of roughly the size of Texas's entire pre-1993 prison population, larger than the "effect" found for any placebo state. As always, the story requires that nothing else Texas-specific happened in 1993 (sentencing law changes, drug enforcement), which is the SCM analogue of the exclusion restriction.
Limits and extensions
- Convex hull: if the treated unit is extreme (the largest, the richest), no convex combination can match it. Check the fit; do not extrapolate.
- Short pre-periods lead to overfitting: the weights fit noise.
- Interference: donors must not be affected by the treatment (prisoners moved across state lines would violate SUTVA).
- Extensions: synthetic DiD (Arkhangelsky et al., 2021) combines unit weights with time weights and a DiD step; augmented SCM (Ben-Michael, Feller and Rothstein, 2021) corrects poor fit with an outcome model. Both are worth knowing once you have mastered the basic method.
Re-run the Texas analysis using white male prisoners (wmprison) as the outcome. Is there a similar
effect? What would different effects for the two groups suggest?
Show a solution
ww = tx.pivot(index="year", columns="state", values="wmprison").sort_index().loc[1986:].dropna(axis=1)
tw = ww["Texas"].to_numpy(); dn = [s for s in ww.columns if s != "Texas"]; dw = ww[dn].to_numpy().T
synth_w = scm(tw, dw, ww.index.to_numpy() < T0) @ dw
print(f"Average gap 1993-2000 (white male prisoners): {(tw - synth_w)[ww.index >= T0].mean():,.0f}")
Checklist and recap
- I can reshape a panel and run SCM with outcome and predictor matching.
- I can interpret the Texas results with placebo inference.
- I know the main limitations and the modern extensions.