{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "id": "m01-l1-0",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 1.1 · Colab, the Modern Stack and Tidy Data\n",
    "\n",
    "*Accelerated Foundation in Applied Causal Inference*\n",
    "\n",
    "**By the end of this lesson you can:**\n",
    "\n",
    "- Explain why data architecture decides whether a causal estimate can be trusted\n",
    "- Set up a reproducible Google Colab notebook with Polars and DuckDB\n",
    "- Recognise wide, long and tidy layouts and convert between them with Polars\n",
    "- Clean identifiers so that two sources can be matched without silent losses\n",
    "\n",
    "> Read the full explanations on the course website, then run each cell here in order (**Runtime → Run all** works on a fresh runtime)."
   ]
  },
  {
   "id": "m01-l1-1",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why data architecture matters for causal work"
   ]
  },
  {
   "id": "m01-l1-2",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Colab setup and the modern stack"
   ]
  },
  {
   "id": "m01-l1-3",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Module 1 · Environment setup (safe to re-run)\n",
    "!pip install -q polars duckdb pandas pyarrow\n",
    "\n",
    "import polars as pl\n",
    "import duckdb\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "pl.Config.set_tbl_rows(8)          # print at most 8 rows of a table\n",
    "print(\"Polars\", pl.__version__, \"| DuckDB\", duckdb.__version__)"
   ]
  },
  {
   "id": "m01-l1-4",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tidy data: wide vs long"
   ]
  },
  {
   "id": "m01-l1-5",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question it answers:** How do I put two agencies' tables, published in different shapes, on the same footing?"
   ]
  },
  {
   "id": "m01-l1-6",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "dates = pd.date_range(start=\"2020-01-01\", periods=48, freq=\"ME\")   # month-ends, 2020-2023\n",
    "\n",
    "# Source 1: inflation, WIDE (one column per province) – how many agencies publish\n",
    "inflation_wide = pl.DataFrame({\n",
    "    \"date\": dates,\n",
    "    \"inflation_ontario\": np.random.normal(0.02, 0.005, 48).cumsum() + 2.0,\n",
    "    \"inflation_quebec\":  np.random.normal(0.02, 0.006, 48).cumsum() + 1.8,\n",
    "})\n",
    "\n",
    "# Source 2: unemployment, LONG, but the date column has another name\n",
    "unemployment_long = pl.DataFrame({\n",
    "    \"record_date\": np.tile(dates, 2),\n",
    "    \"province\": [\"Ontario\"] * 48 + [\"Quebec\"] * 48,\n",
    "    \"unemp_rate\": np.random.normal(5.5, 0.5, 96),\n",
    "})\n",
    "\n",
    "print(inflation_wide.head(4))\n",
    "print(unemployment_long.head(4))"
   ]
  },
  {
   "id": "m01-l1-7",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Un-pivoting and cleaning keys with Polars"
   ]
  },
  {
   "id": "m01-l1-8",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "inflation_tidy = inflation_wide.unpivot(\n",
    "    index=\"date\",\n",
    "    on=[\"inflation_ontario\", \"inflation_quebec\"],\n",
    "    variable_name=\"province_raw\",\n",
    "    value_name=\"inflation_rate\",\n",
    ")\n",
    "\n",
    "inflation_clean = (\n",
    "    inflation_tidy\n",
    "    .with_columns(\n",
    "        pl.col(\"province_raw\")\n",
    "          .str.replace(\"inflation_\", \"\")\n",
    "          .str.to_titlecase()\n",
    "          .alias(\"province\")\n",
    "    )\n",
    "    .drop(\"province_raw\")\n",
    "    .select(\"date\", \"province\", \"inflation_rate\")\n",
    ")\n",
    "\n",
    "print(inflation_clean.head(4))\n",
    "print(inflation_clean.shape)   # 48 months x 2 provinces = 96 rows"
   ]
  },
  {
   "id": "m01-l1-9",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "dupes = (\n",
    "    inflation_clean\n",
    "    .group_by(\"province\", \"date\")\n",
    "    .agg(pl.len().alias(\"n\"))\n",
    "    .filter(pl.col(\"n\") > 1)\n",
    ")\n",
    "print(\"Duplicated province-month keys:\", dupes.height)\n",
    "assert dupes.height == 0"
   ]
  },
  {
   "id": "m01-l1-10",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: tidy a second messy table"
   ]
  },
  {
   "id": "m01-l1-11",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✍️ Exercise\n",
    "\n",
    "A third source reports the **policy interest rate** with one row per year and one column per month\n",
    "(`jan`, `feb`, …, `dec`). Build it, then reshape it into a tidy table with columns `date` and `policy_rate`.\n",
    "\n",
    "1. Create `rates_wide` with a `year` column (2020–2023) and 12 month columns filled with random rates.\n",
    "2. Un-pivot the month columns.\n",
    "3. Build a proper month-end `date` from `year` and the month name.\n",
    "4. Check there are exactly 48 rows and no duplicated dates."
   ]
  },
  {
   "id": "m01-l1-12",
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your answer here"
   ]
  },
  {
   "id": "m01-l1-13",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checklist and recap"
   ]
  }
 ]
}