Lesson 4.2: Randomised Experiments: the Voucher Lottery

Written lesson

Why a lottery solves the problem

When a school voucher programme is oversubscribed, a lottery decides who gets one. Because a coin flip cannot know a family's motivation, winners and losers have the same potential outcomes on average:

E[Y(0) | won] = E[Y(0) | lost], so selection bias is zero and the difference in means estimates the ATE.

Python · Colab cell

!pip install -q numpy pandas statsmodels matplotlib seaborn causaldata
import numpy as np, pandas as pd
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt, seaborn as sns
rng = np.random.default_rng(12)
sns.set_theme(style="whitegrid")

Simulating an oversubscribed voucher lottery

4,000 applicants, 2,000 vouchers. The true average effect on test scores is +4 points.

Python · Colab cell

N = 4_000
applicants = pd.DataFrame({
    "motivation": rng.normal(0, 1, N),
    "baseline_score": rng.normal(60, 10, N),
    "low_income": rng.binomial(1, 0.4, N),
})
applicants["won"] = rng.permutation(np.repeat([0, 1], N // 2))          # the lottery
TRUE_ATE = 4.0
applicants["score"] = (
    10 + 0.8 * applicants.baseline_score + 3 * applicants.motivation
    - 2 * applicants.low_income + TRUE_ATE * applicants.won + rng.normal(0, 6, N)
)

diff = applicants.groupby("won")["score"].mean()
print(f"Difference in means: {diff[1] - diff[0]:.2f} (true {TRUE_ATE})")

ols = smf.ols("score ~ won", data=applicants).fit(cov_type="HC1")
print(ols.summary().tables[1])

Regression of the outcome on a treatment dummy is the difference in means, now with a standard error.

Balance checks

Randomisation balances covariates in expectation. In any given sample, small differences appear by chance. A balance table reports them; the standardised difference (difference in means divided by the pooled standard deviation) is easier to read than a p-value. A common rule flags |std. diff| > 0.1.

Python · Colab cell

def balance_table(df, treat, covariates):
    rows = []
    for c in covariates:
        t, u = df.loc[df[treat] == 1, c], df.loc[df[treat] == 0, c]
        std_diff = (t.mean() - u.mean()) / np.sqrt((t.var() + u.var()) / 2)
        p = smf.ols(f"{c} ~ {treat}", data=df).fit(cov_type="HC1").pvalues[treat]
        rows.append((c, t.mean(), u.mean(), std_diff, p))
    return pd.DataFrame(rows, columns=["covariate", "treated", "control", "std_diff", "p_value"]).round(3)

print(balance_table(applicants, "won", ["motivation", "baseline_score", "low_income"]))

Covariate adjustment for precision

Adding pre-treatment covariates that predict the outcome does not change what we estimate (the lottery already removed bias), but it soaks up noise and shrinks the standard error. A robust recipe (Lin, 2013) interacts treatment with demeaned covariates.

Python · Colab cell

a = applicants.copy()
for c in ["baseline_score", "low_income"]:
    a[c + "_c"] = a[c] - a[c].mean()
adjusted = smf.ols("score ~ won * (baseline_score_c + low_income_c)", data=a).fit(cov_type="HC1")

print(f"Unadjusted: {ols.params['won']:.2f} (SE {ols.bse['won']:.3f})")
print(f"Adjusted:   {adjusted.params['won']:.2f} (SE {adjusted.bse['won']:.3f})")
print(f"Equivalent to a sample {((ols.bse['won'] / adjusted.bse['won'])**2):.1f}x larger")

The why

Only adjust for variables measured before randomisation. A variable affected by the treatment (say, hours of homework after receiving the voucher) is a post-treatment control and can reintroduce bias. Module 5 explains why with DAGs.

Real data: the NSW job-training experiment

The National Supported Work demonstration (LaLonde, 1986) randomly assigned disadvantaged workers to a job-training programme. nsw_mixtape holds the experimental sample; re78 is earnings in 1978.

Python · Colab cell

from causaldata import nsw_mixtape
nsw = nsw_mixtape.load_pandas().data
print(balance_table(nsw, "treat", ["age", "educ", "black", "hisp", "marr", "nodegree", "re74", "re75"]))

nsw_ate = smf.ols("re78 ~ treat", data=nsw).fit(cov_type="HC1")
nsw_adj = smf.ols("re78 ~ treat + age + educ + black + hisp + marr + nodegree + re74 + re75", data=nsw).fit(cov_type="HC1")
print(f"Experimental ATE: ${nsw_ate.params['treat']:,.0f} (SE {nsw_ate.bse['treat']:,.0f})")
print(f"Adjusted:         ${nsw_adj.params['treat']:,.0f} (SE {nsw_adj.bse['treat']:,.0f})")

Keep this number (about $1,800). In Module 6 you will try to recover it without the experiment, from observational survey data, which is exactly LaLonde's famous test of non-experimental methods.

Exercise

Simulate a broken lottery: let families with high motivation be twice as likely to win. Re-run the balance table and the difference in means. Which line of the balance table warns you?

Show a solution

Python · Colab cell

p_win = np.where(applicants.motivation > 0, 0.66, 0.33)
broken = applicants.assign(won=rng.binomial(1, p_win))
broken["score"] = (10 + 0.8 * broken.baseline_score + 3 * broken.motivation - 2 * broken.low_income
                   + TRUE_ATE * broken.won + rng.normal(0, 6, N))
print(balance_table(broken, "won", ["motivation", "baseline_score", "low_income"]))
print(f"Biased difference in means: {smf.ols('score ~ won', data=broken).fit().params['won']:.2f}")

Checklist and recap

I can explain why randomisation makes the difference in means unbiased for the ATE.
I report a balance table with standardised differences.
I use pre-treatment covariates for precision, never post-treatment ones.