Lesson 4.3: Inference for Experiments: Permutation Tests, Clusters and Power

Written lesson

Randomisation inference

Standard errors rely on large-sample approximations. A permutation test relies only on how treatment was assigned. Under the sharp null that the treatment did nothing for anyone, re-shuffling the treatment labels should produce estimates as large as ours fairly often. The p-value is the share of shuffles that do.

Python · Colab cell

!pip install -q numpy pandas statsmodels matplotlib seaborn causaldata
import numpy as np, pandas as pd
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt, seaborn as sns
rng = np.random.default_rng(8)
sns.set_theme(style="whitegrid")

Python · Colab cell

n = 40
d = rng.permutation(np.repeat([0, 1], n // 2))
y = 60 + 4 * d + rng.normal(0, 8, n)
observed = y[d == 1].mean() - y[d == 0].mean()

perm = np.array([
    (lambda s: y[s == 1].mean() - y[s == 0].mean())(rng.permutation(d)) for _ in range(10_000)
])
p_perm = np.mean(np.abs(perm) >= abs(observed))
p_ols = smf.ols("y ~ d", data=pd.DataFrame({"y": y, "d": d})).fit(cov_type="HC1").pvalues["d"]
print(f"Estimate {observed:.2f} | permutation p = {p_perm:.3f} | OLS p = {p_ols:.3f}")

plt.hist(perm, bins=60, alpha=.7); plt.axvline(observed, color="red", lw=2)
plt.title("Distribution of estimates under the sharp null"); plt.show()

The two p-values are close, as they usually are in balanced designs. Notice that with only 40 pupils a real 4-point effect often fails to reach significance: the noise is too large for the sample. That is a power problem, the subject of the last section.

A cluster-randomised field experiment

Thornton (2008) offered randomly assigned cash incentives (got = learned result; any = received any incentive) to people in rural Malawi to collect their HIV test results. Villages differ, and outcomes within a village are correlated, so we cluster standard errors by village (villnum).

Python · Colab cell

from causaldata import thornton_hiv
hiv = thornton_hiv.load_pandas().data.dropna(subset=["got", "any", "tinc", "villnum"])

m_any = smf.ols("got ~ any", data=hiv).fit(cov_type="cluster", cov_kwds={"groups": hiv["villnum"]})
print(f"Share learning status without incentive: {hiv.loc[hiv['any'] == 0, 'got'].mean():.3f}")
print(f"Effect of any incentive: {m_any.params['any']:.3f} (clustered SE {m_any.bse['any']:.3f})")

m_amt = smf.ols("got ~ any + tinc + distvct", data=hiv).fit(cov_type="cluster", cov_kwds={"groups": hiv["villnum"]})
print(m_amt.params.round(4))

Even a token incentive raises the share who learn their result by roughly 45 percentage points, from a base of about 34%. Distance to the testing centre (distvct) lowers it. Small frictions have large effects: a lesson for every programme designer.

Power and the minimum detectable effect

Before running an experiment, ask: if the effect is real, how likely am I to detect it? That probability is power; 80% is the usual target. Power grows with the sample size and the effect, and falls with the noise and with clustering. Simulation handles every design, even complicated ones.

Python · Colab cell

def power(n, effect, sd=10, reps=600, alpha=0.05):
    hits = 0
    for _ in range(reps):
        d = rng.permutation(np.repeat([0, 1], n // 2))
        y = 60 + effect * d + rng.normal(0, sd, n)
        hits += smf.ols("y ~ d", data=pd.DataFrame({"y": y, "d": d})).fit().pvalues["d"] < alpha
    return hits / reps

sizes = [100, 200, 400, 800, 1_600]
curve = [power(n, effect=2.0) for n in sizes]
for n, p in zip(sizes, curve):
    print(f"n = {n:>5}: power = {p:.2f}")

plt.plot(sizes, curve, marker="o"); plt.axhline(0.8, ls="--", color="grey")
plt.xscale("log"); plt.xlabel("Total sample size"); plt.ylabel("Power"); plt.title("Power to detect +2 points"); plt.show()

The analytical shortcut for two equal groups: MDE ≈ 2.8 × σ × √(4 / n), where 2.8 = 1.96 + 0.84.

Python · Colab cell

for n in [400, 1_600]:
    print(f"n = {n:>5}: minimum detectable effect ≈ {2.8 * 10 * np.sqrt(4 / n):.2f} points")

Exercise

A ministry can afford 2,000 pupils but must randomise whole schools of 50 pupils (40 schools), with a school-level shock of SD 4 on top of pupil noise of SD 10. Simulate the power to detect a 2-point effect with clustered standard errors. Compare with 2,000 individually randomised pupils. What would you advise?

Show a solution

Python · Colab cell

def cluster_power(effect=2.0, schools=40, size=50, reps=300):
    hits = 0
    for _ in range(reps):
        s = np.repeat(np.arange(schools), size)
        d = rng.permutation(np.repeat([0, 1], schools // 2))[s]
        y = 60 + effect * d + rng.normal(0, 4, schools)[s] + rng.normal(0, 10, s.size)
        fit = smf.ols("y ~ d", data=pd.DataFrame({"y": y, "d": d, "s": s})).fit(
            cov_type="cluster", cov_kwds={"groups": s}, use_t=True)
        hits += fit.pvalues["d"] < 0.05
    return hits / reps

print(f"Cluster-randomised power: {cluster_power():.2f} | individual power: {power(2_000, 2.0):.2f}")
# Clustering destroys most of the power. Advise more schools with fewer pupils each, or adjusting
# for baseline scores, rather than more pupils per school.

Checklist and recap

I can run a permutation test and explain the sharp null.
I cluster standard errors at the level of randomisation.
I can simulate power and compute a minimum detectable effect before collecting data.