{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "id": "m12-l3-0",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 12.3 · Synthetic Control on Real Data: Texas Prison Expansion\n",
    "\n",
    "*Accelerated Foundation in Applied Causal Inference*\n",
    "\n",
    "**By the end of this lesson you can:**\n",
    "\n",
    "- Prepare a state-year panel for synthetic control\n",
    "- Match on pre-treatment outcomes and predictors, with predictor weights\n",
    "- Estimate the effect of Texas's 1993 prison-capacity expansion on Black male incarceration\n",
    "- Discuss limitations and modern extensions (synthetic DiD, augmented SCM)\n",
    "\n",
    "> Read the full explanations on the course website, then run each cell here in order (**Runtime → Run all** works on a fresh runtime)."
   ]
  },
  {
   "id": "m12-l3-1",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The question"
   ]
  },
  {
   "id": "m12-l3-2",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "!pip install -q numpy pandas scipy matplotlib causaldata\n",
    "import numpy as np, pandas as pd\n",
    "from scipy.optimize import minimize\n",
    "import matplotlib.pyplot as plt\n",
    "from causaldata import texas"
   ]
  },
  {
   "id": "m12-l3-3",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation"
   ]
  },
  {
   "id": "m12-l3-4",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question it answers:** How many Black men would have been in Texas prisons without the 1993 expansion?"
   ]
  },
  {
   "id": "m12-l3-5",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "tx = texas.load_pandas().data\n",
    "wide = tx.pivot(index=\"year\", columns=\"state\", values=\"bmprison\").sort_index()\n",
    "wide = wide.dropna(axis=1)                                   # keep states with complete series\n",
    "years = wide.index.to_numpy(); T0 = 1993; pre = years < T0\n",
    "treated = wide[\"Texas\"].to_numpy()\n",
    "donor_names = [s for s in wide.columns if s != \"Texas\"]\n",
    "donors = wide[donor_names].to_numpy().T\n",
    "print(f\"{len(donor_names)} donor states, {pre.sum()} pre-treatment years ({years.min()}-{T0 - 1})\")"
   ]
  },
  {
   "id": "m12-l3-6",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "preds = [\"income\", \"ur\", \"poverty\", \"black\", \"perc1519\", \"alcohol\"]\n",
    "X = tx[tx.year < T0].groupby(\"state\")[preds].mean().loc[[\"Texas\"] + donor_names]\n",
    "X = (X - X.mean()) / X.std()                                  # standardise\n",
    "\n",
    "def scm(target, pool, pre_mask, x_target=None, x_pool=None, v=0.5):\n",
    "    \"\"\"Minimise (1 - v) * pre-period outcome error + v * predictor error (both standardised).\"\"\"\n",
    "    scale = target[pre_mask].std()\n",
    "    def loss(w):\n",
    "        out = np.mean(((target[pre_mask] - w @ pool[:, pre_mask]) / scale) ** 2)\n",
    "        pr = 0 if x_target is None else np.mean((x_target - w @ x_pool) ** 2)\n",
    "        return (1 - v) * out + v * pr\n",
    "    J = pool.shape[0]\n",
    "    return minimize(loss, np.full(J, 1 / J), method=\"SLSQP\", bounds=[(0, 1)] * J,\n",
    "                    constraints={\"type\": \"eq\", \"fun\": lambda w: w.sum() - 1}, options={\"maxiter\": 500}).x"
   ]
  },
  {
   "id": "m12-l3-7",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimation and placebo inference"
   ]
  },
  {
   "id": "m12-l3-8",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "w = scm(treated, donors, pre, X.loc[\"Texas\"].to_numpy(), X.loc[donor_names].to_numpy(), v=0.2)\n",
    "synthetic = w @ donors\n",
    "print(pd.Series(w, index=donor_names).sort_values(ascending=False).head(6).round(3))\n",
    "\n",
    "gap = treated - synthetic\n",
    "print(f\"Average gap 1993-2000: {gap[~pre].mean():,.0f} prisoners | in 2000: {gap[-1]:,.0f}\")\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 4))\n",
    "axes[0].plot(years, treated, \"r\", lw=2, label=\"Texas\"); axes[0].plot(years, synthetic, \"b--\", lw=2, label=\"Synthetic Texas\")\n",
    "axes[0].axvline(T0, color=\"k\", ls=\":\"); axes[0].legend(); axes[0].set_title(\"Black male prisoners\")\n",
    "axes[1].bar(years, gap, color=np.where(pre, \"#94a3b8\", \"#ef4444\")); axes[1].set_title(\"Gap (Texas − synthetic)\")\n",
    "plt.tight_layout(); plt.show()"
   ]
  },
  {
   "id": "m12-l3-9",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "all_units = np.vstack([treated, donors]); names = [\"Texas\"] + donor_names\n",
    "rows = {}\n",
    "for i in range(len(all_units)):\n",
    "    level = all_units[i][pre].mean()\n",
    "    if level <= 0:\n",
    "        continue\n",
    "    others = [k for k in range(len(all_units)) if k not in (i, 0)]\n",
    "    g = all_units[i] - scm(all_units[i], all_units[others], pre) @ all_units[others]\n",
    "    rows[names[i]] = {\"post_gap_pct\": g[~pre].mean() / level, \"pre_rmspe_pct\": np.sqrt(np.mean(g[pre] ** 2)) / level}\n",
    "placebo = pd.DataFrame(rows).T\n",
    "good = placebo[placebo.pre_rmspe_pct <= 0.10].sort_values(\"post_gap_pct\", ascending=False)\n",
    "print(good.head(5).round(3))\n",
    "print(f\"Texas rank: {list(good.index).index('Texas') + 1} of {len(good)} well-fitted units | \"\n",
    "      f\"p = {(good.post_gap_pct >= good.loc['Texas', 'post_gap_pct']).mean():.3f}\")"
   ]
  },
  {
   "id": "m12-l3-10",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limits and extensions"
   ]
  },
  {
   "id": "m12-l3-11",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✍️ Exercise\n",
    "\n",
    "Re-run the Texas analysis using **white** male prisoners (`wmprison`) as the outcome. Is there a similar\n",
    "effect? What would different effects for the two groups suggest?"
   ]
  },
  {
   "id": "m12-l3-12",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your answer here"
   ]
  },
  {
   "id": "m12-l3-13",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checklist and recap"
   ]
  }
 ]
}