From point estimates to honest ranges
Every point estimate in this course rests on assumptions that cannot be fully tested: unconfoundedness, exclusion, parallel trends. Partial identification (Manski) reverses the logic: start with assumptions so weak that nobody disputes them, and report the range of effects consistent with the data. Then add assumptions one at a time and show how the range narrows.
A wide interval is not a failure. It tells decision-makers exactly how much of the conclusion comes from the data and how much from assumptions.
!pip install -q numpy pandas matplotlib
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
rng = np.random.default_rng(14)
Worst-case bounds with missing outcomes
A randomised job-training programme. The outcome is employment one year later (1 = employed), but 30% of participants could not be found at follow-up. True employment rates: 60% treated, 40% control (ATE = +0.20). Attrition is not random: people who found jobs elsewhere are harder to reach.
N = 2_000
treat = rng.binomial(1, 0.5, N)
employed = rng.binomial(1, np.where(treat == 1, 0.6, 0.4))
p_missing = 0.2 + 0.2 * employed # employed people are harder to reach
missing = rng.binomial(1, p_missing).astype(bool)
y_obs = np.where(missing, np.nan, employed)
df = pd.DataFrame({"treat": treat, "y": y_obs, "missing": missing})
def group(t):
g = df[df.treat == t]
return g.y.mean(), g.missing.mean() # mean among observed, share missing
(m1, q1), (m0, q0) = group(1), group(0)
naive = m1 - m0
# Bounds on each potential-outcome mean: missing outcomes could all be 0 or all be 1
lo1, hi1 = m1 * (1 - q1), m1 * (1 - q1) + q1
lo0, hi0 = m0 * (1 - q0), m0 * (1 - q0) + q0
manski = (lo1 - hi0, hi1 - lo0)
print(f"Missing: treated {q1:.1%}, control {q0:.1%}")
print(f"Naive (complete cases): {naive:+.3f} | truth +0.200")
print(f"Manski bounds on the ATE: [{manski[0]:+.3f}, {manski[1]:+.3f}]")
The worst-case bounds always have width equal to the share missing in the two groups combined (here about 0.6), and they include zero. On its own, the data cannot even sign the effect. That is the honest starting point.
Buying tighter bounds with assumptions
Each assumption narrows the range. Make them explicit so a reader can accept or reject each one.
- Monotone treatment response (MTR): training never reduces employment for anyone, so ATE ≥ 0.
- Missing outcomes are no worse than observed ones (people we lost are at least as likely to be employed as those we found, plausible here because job-finders are hard to reach): replace the "all zero" worst case with the observed mean.
# Assumption: missing people are at least as employed as observed people in the same arm
lo1_b, lo0_b = m1, m0 # lower bounds rise to the observed means
bounds_b = (lo1_b - hi0, hi1 - lo0_b)
bounds_mtr = (max(0, manski[0]), manski[1])
table = pd.DataFrame({
"lower": [manski[0], bounds_mtr[0], max(0, bounds_b[0])],
"upper": [manski[1], bounds_mtr[1], bounds_b[1]],
}, index=["Worst case (no assumptions)", "+ monotone response (ATE ≥ 0)", "+ missing ≥ observed, and MTR"]).round(3)
table["width"] = table.upper - table.lower
print(table)
fig, ax = plt.subplots(figsize=(8, 3))
for i, (name, r) in enumerate(table.iterrows()):
ax.plot([r.lower, r.upper], [i, i], lw=8, solid_capstyle="butt")
ax.axvline(0.2, color="red", ls="--", label="Truth"); ax.axvline(0, color="grey")
ax.set_yticks(range(len(table))); ax.set_yticklabels(table.index); ax.legend(); ax.set_title("What each assumption buys")
plt.show()
Lee bounds for attrition in experiments
When attrition differs between arms, the people we observe in the treatment group are not comparable to those we observe in control. Lee (2009) bounds assume monotone selection: treatment can only make people more (or only less) likely to be observed. Then we trim the arm with more observed people by the excess share, from the top (lower bound) or the bottom (upper bound) of its outcome distribution. Lee bounds are much tighter than Manski bounds and work for continuous outcomes such as wages.
N = 4_000
treat = rng.binomial(1, 0.5, N)
ability = rng.normal(size=N)
works = rng.binomial(1, 1 / (1 + np.exp(-(0.0 + 0.8 * treat + ability)))) # training raises employment
wage = 20 + 2 * treat + 3 * ability + rng.normal(0, 2, N) # true wage effect = 2
wage_obs = np.where(works == 1, wage, np.nan) # wages only seen for workers
d = pd.DataFrame({"treat": treat, "wage": wage_obs})
def lee_bounds(data):
obs1 = data[(data.treat == 1)].wage.notna().mean(); obs0 = data[(data.treat == 0)].wage.notna().mean()
w1 = np.sort(data[(data.treat == 1)].wage.dropna().to_numpy()); w0 = data[(data.treat == 0)].wage.dropna()
p = (obs1 - obs0) / obs1 # share of treated observations to trim
k = int(round(p * len(w1)))
lower = w1[: len(w1) - k].mean() - w0.mean() # drop the top k
upper = w1[k:].mean() - w0.mean() # drop the bottom k
return p, lower, upper
p, lower, upper = lee_bounds(d)
naive_w = d[d.treat == 1].wage.mean() - d[d.treat == 0].wage.mean()
print(f"Observed wage: treated {d[d.treat == 1].wage.notna().mean():.2f}, control {d[d.treat == 0].wage.notna().mean():.2f} -> trim {p:.1%}")
print(f"Naive wage difference (workers only): {naive_w:+.2f}")
print(f"Lee bounds: [{lower:+.2f}, {upper:+.2f}] (true effect for always-workers: +2)")
The naive comparison among workers is biased downward: training pulled lower-ability people into work. Lee bounds bracket the effect for people who would work either way.
Bootstrap the Lee bounds (200 resamples) and report a 95% interval for the identified set (use the 2.5th percentile of the lower bound and the 97.5th percentile of the upper bound). Is zero excluded?
Show a solution
boots = np.array([lee_bounds(d.sample(frac=1, replace=True, random_state=s))[1:] for s in range(200)])
print(f"95% CI for the identified set: [{np.percentile(boots[:, 0], 2.5):+.2f}, {np.percentile(boots[:, 1], 97.5):+.2f}]")
Checklist and recap
- I can compute worst-case Manski bounds and explain their width.
- I can tighten bounds with explicit assumptions and show what each one buys.
- I can compute Lee bounds for differential attrition.