4.1 Potential Outcomes: ATE, ATT, ATU and Selection Bias

Two worlds per person

The potential outcomes framework (Neyman, Rubin) gives every unit two outcomes:

• Y(1): the outcome if the unit is treated;
• Y(0): the outcome if it is not.

The individual treatment effect is Y(1) − Y(0). The fundamental problem of causal inference is that we only ever observe one of the two: a pupil either used a school voucher or did not. The other outcome is the counterfactual, and every method in this course is a strategy for estimating it.

!pip install -q numpy pandas statsmodels matplotlib seaborn
import numpy as np, pandas as pd
import matplotlib.pyplot as plt, seaborn as sns
rng = np.random.default_rng(4)
sns.set_theme(style="whitegrid")

A voucher world where we can see both outcomes

In a simulation we can do what reality forbids: write down both potential outcomes for every pupil. Pupils differ in motivation; motivated families both seek out vouchers and would score higher anyway. The voucher's effect also varies: it helps less-motivated pupils more.

N = 10_000
motivation = rng.normal(0, 1, N)
y0 = 60 + 6 * motivation + rng.normal(0, 5, N)          # score without a voucher
effect = 5 - 2 * motivation                              # heterogeneous effect
y1 = y0 + effect                                         # score with a voucher

# Self-selection: motivated families are more likely to use a voucher
d = rng.binomial(1, 1 / (1 + np.exp(-1.5 * motivation)))
y = np.where(d == 1, y1, y0)                             # what we actually observe

pupils = pd.DataFrame({"motivation": motivation, "y0": y0, "y1": y1, "d": d, "y": y})
ATE = (pupils.y1 - pupils.y0).mean()
ATT = (pupils.y1 - pupils.y0)[pupils.d == 1].mean()
ATU = (pupils.y1 - pupils.y0)[pupils.d == 0].mean()
print(f"ATE = {ATE:.2f} | ATT = {ATT:.2f} | ATU = {ATU:.2f}")
print(pupils.head())

The ATT is smaller than the ATU: users are motivated pupils, who gain less. Which number matters depends on the policy question. Scaling a voucher to non-users is about the ATU, not the ATT.

Naive comparison = effect + selection bias

The naive estimate compares observed outcomes of users and non-users. A little algebra (adding and subtracting E[Y(0) | D = 1]) splits it in two:

E[Y | D=1] − E[Y | D=0] = ATT + ( E[Y(0) | D=1] − E[Y(0) | D=0] )

The second term is selection bias: how different users would have been even without a voucher.

naive = pupils.y[pupils.d == 1].mean() - pupils.y[pupils.d == 0].mean()
selection_bias = pupils.y0[pupils.d == 1].mean() - pupils.y0[pupils.d == 0].mean()
print(f"Naive difference : {naive:6.2f}")
print(f"ATT              : {ATT:6.2f}")
print(f"Selection bias   : {selection_bias:6.2f}")
print(f"ATT + bias       : {ATT + selection_bias:6.2f}")

fig, ax = plt.subplots(figsize=(8, 4))
sns.kdeplot(data=pupils, x="y0", hue="d", common_norm=False, ax=ax)
ax.set(title="Y(0) differs between users and non-users: selection bias", xlabel="Score without voucher")
plt.show()

SUTVA: the assumption hiding in the notation

Writing Y(1) and Y(0) for each pupil already assumes the Stable Unit Treatment Value Assumption:

1. No interference: my outcome depends only on my treatment, not on my classmates'.
2. No hidden versions: "a voucher" means the same thing for everyone.

SUTVA fails with peer effects (vouchers move strong pupils and change the classes left behind), vaccines (my vaccination protects you) and market-wide effects (training many workers lowers wages for all trained workers). Module 13 deals with those cases.

Exercise

y1_const = y0 + 5
y_const = np.where(d == 1, y1_const, y0)
naive_c = y_const[d == 1].mean() - y_const[d == 0].mean()
print(f"ATE=ATT=ATU=5 | naive = {naive_c:.2f} | bias = {y0[d == 1].mean() - y0[d == 0].mean():.2f}")
# With a constant effect ATE = ATT = ATU, but selection bias is unchanged: the naive estimate is
# still wrong, because bias comes from who selects in (Y(0)), not from effect heterogeneity.

Checklist and recap

• I can define Y(1), Y(0) and the fundamental problem of causal inference.
• I can distinguish the ATE, ATT and ATU and say which one a policy question needs.
• I can decompose a naive comparison into ATT + selection bias.
• I can state the two parts of SUTVA.