14.3 Omitted-Variable Sensitivity: Robustness Values and Oster's δ

From OVB formula to sensitivity analysis

Lesson 3.2's omitted variable bias formula says how far an estimate moves when a confounder is left out. Cinelli and Hazlett (2020) rewrite it in terms of two partial R² values of a hypothetical confounder U:

• R²(D ~ U | X): how much of the treatment U explains, beyond the controls;
• R²(Y ~ U | D, X): how much of the outcome U explains, beyond treatment and controls.

Their robustness value RV is the strength (the same partial R² with both) that a confounder would need to reduce the estimate to zero. It needs only the t-statistic and the residual degrees of freedom:

f = t / √df, RV = ½ (√(f⁴ + 4f²) − f²).

A confounder weaker than RV on both dimensions cannot explain away the effect.

!pip install -q numpy pandas statsmodels matplotlib causaldata
import numpy as np, pandas as pd
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt
from causaldata import nhefs_complete

Robustness values on the NHEFS smoking data

We return to Module 7's question, the effect of quitting smoking on weight gain, now with a linear regression so we can apply the formulas directly.

nh = nhefs_complete.load_pandas().data
covs = "sex + race + age + I(age**2) + C(education) + smokeintensity + I(smokeintensity**2) + smokeyrs + I(smokeyrs**2) + C(exercise) + C(active) + wt71 + I(wt71**2)"
fit = smf.ols(f"wt82_71 ~ qsmk + {covs}", data=nh).fit()
est, se, t, dof = fit.params["qsmk"], fit.bse["qsmk"], fit.tvalues["qsmk"], fit.df_resid

def robustness_value(t, dof, q=1.0, alpha=None):
    f = q * abs(t) / np.sqrt(dof)
    if alpha is not None:                                  # RV for losing significance at level alpha
        from scipy.stats import t as tdist
        f = max(q * abs(t) / np.sqrt(dof) - tdist.ppf(1 - alpha / 2, dof - 1) / np.sqrt(dof - 1), 0)
    return 0.5 * (np.sqrt(f ** 4 + 4 * f ** 2) - f ** 2)

partial_r2_d = t ** 2 / (t ** 2 + dof)                     # share of outcome variance explained by D itself
print(f"Estimate: {est:.2f} kg (SE {se:.2f}, t = {t:.1f})")
print(f"Partial R² of quitting with weight change: {partial_r2_d:.3f}")
print(f"Robustness value (to bring the estimate to 0):         {robustness_value(t, dof):.3f}")
print(f"Robustness value (to lose significance at 5%):          {robustness_value(t, dof, alpha=0.05):.3f}")

Read: an unobserved confounder would need to explain about RV × 100% of the residual variance of both quitting and weight change to erase the estimate.

Benchmarking against observed covariates

Is that a lot? Compare with the strongest observed confounders. For each covariate group, compute the partial R² it has with the treatment and with the outcome. If the hypothetical U would need to be several times stronger than, say, baseline weight and smoking intensity combined, the result is robust.

def partial_r2(full, var_terms):
    """Partial R² of a group of terms in a fitted model, via the change in R²."""
    reduced_formula = full.model.formula
    for term in var_terms:
        reduced_formula = reduced_formula.replace(f" + {term}", "")
    reduced = smf.ols(reduced_formula, data=nh).fit()
    return (full.rsquared - reduced.rsquared) / (1 - reduced.rsquared)

d_model = smf.ols(f"qsmk ~ {covs}", data=nh).fit()
benchmarks = {
    "age": ["age", "I(age**2)"],
    "smoking intensity": ["smokeintensity", "I(smokeintensity**2)"],
    "baseline weight": ["wt71", "I(wt71**2)"],
}
rows = [(name, partial_r2(d_model, terms), partial_r2(fit, terms)) for name, terms in benchmarks.items()]
bench = pd.DataFrame(rows, columns=["covariate", "R² with quitting | X", "R² with weight change | D, X"]).round(3)
print(bench)
print(f"Robustness value: {robustness_value(t, dof):.3f}")

If the RV is well above every benchmark row, a hidden confounder would have to be considerably stronger than age, smoking intensity or baseline weight, each of which is already among the most important predictors. That is a much more convincing statement than "we controlled for many things".

Coefficient stability: Oster's δ

Oster (2019) formalised a common heuristic: if adding observed controls barely moves the estimate and raises R² substantially, unobserved controls probably would not move it much either. Her δ asks: how strong would selection on unobservables need to be, relative to selection on observables, to drive the effect to zero, if the R² of a hypothetical full model were R_max (often 1.3 × the controlled R²)?

An approximate formula (exact when the treatment's variance is unaffected by controls):

δ ≈ [β̃ · (R̃ − R°)] / [(β° − β̃) · (R_max − R̃)]

where ° denotes the regression without controls and ~ the regression with controls.

short = smf.ols("wt82_71 ~ qsmk", data=nh).fit()
b0, r0 = short.params["qsmk"], short.rsquared
b1, r1 = fit.params["qsmk"], fit.rsquared
r_max = min(1.0, 1.3 * r1)
delta = (b1 * (r1 - r0)) / ((b0 - b1) * (r_max - r1))
print(f"Uncontrolled: {b0:.2f} kg (R² {r0:.3f}) | Controlled: {b1:.2f} kg (R² {r1:.3f}) | R_max {r_max:.3f}")
print(f"Oster's δ for a zero effect: {delta:.2f}")

Here adding controls increases the estimate (β̃ > β°), so a negative δ comes out: unobservables would have to select in the opposite direction to the observables to explain the effect away. |δ| > 1 is the usual threshold for a robust result.

from causaldata import nsw_mixtape, cps_mixtape
nsw = nsw_mixtape.load_pandas().data
lal = pd.concat([nsw[nsw.treat == 1], cps_mixtape.load_pandas().data], ignore_index=True)
f_lal = smf.ols("re78 ~ treat + age + educ + black + hisp + marr + nodegree + re74 + re75", data=lal).fit()
print(f"LaLonde OLS: {f_lal.params['treat']:,.0f} (t = {f_lal.tvalues['treat']:.2f}), "
      f"RV = {robustness_value(f_lal.tvalues['treat'], f_lal.df_resid):.4f}")
# A tiny RV: a very weak hidden confounder could overturn the LaLonde OLS estimate, consistent with
# how far OLS was from the experimental benchmark.

Checklist and recap

• I can compute and interpret a robustness value.
• I benchmark sensitivity against the strongest observed confounders.
• I can compute Oster's δ and write a sensitivity paragraph for decision-makers.