{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "153d6f74",
      "metadata": {},
      "source": [
        "# Micrograd — backprop from scratch\n",
        "\n",
        "A tiny automatic-gradient engine in pure Python — **concept -> code -> Your turn** each step.\n",
        "\n",
        "This is notebook **00** of the series and the right place to start: it explains what a\n",
        "*gradient* and *backpropagation* really are, using single numbers you can follow by hand.\n",
        "Every later notebook (the bigram in `01`, the transformer in `06`) calls `loss.backward()`\n",
        "and trusts it. Here we build that machinery ourselves so it is never magic."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aac17ea7",
      "metadata": {},
      "source": [
        "## Prologue — what this notebook does, in plain English\n",
        "\n",
        "We build a small object called a **`Value`**. A `Value` is just a number that also\n",
        "**remembers how it was made** (which other numbers, and which operation: plus, times, ...).\n",
        "\n",
        "Once numbers remember their own history, the computer can answer one very useful question\n",
        "automatically:\n",
        "\n",
        "> *If I nudge this input a tiny bit, how much does the final answer change?*\n",
        "\n",
        "That sensitivity is the **gradient**. Computing all those sensitivities in one efficient\n",
        "backward sweep is **backpropagation** — the single algorithm that trains essentially every\n",
        "neural network, including GPT.\n",
        "\n",
        "This notebook follows Andrej Karpathy's *\"The spelled-out intro to neural networks and\n",
        "backpropagation: building micrograd\"* lecture."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "44dd4b0d",
      "metadata": {},
      "source": [
        "### The whole idea, as a chain of gears\n",
        "\n",
        "Picture a row of **gears** connected together. You turn the first gear a little; the last\n",
        "gear also turns, by an amount that depends on all the gears in between.\n",
        "\n",
        "- The **forward pass** is turning the first gears and reading the last gear (the output).\n",
        "- The **gradient** answers: *if I turn this one gear slightly, how much does the final gear move?*\n",
        "- **Backpropagation** is figuring out that answer for **every** gear at once, by walking\n",
        "  backward from the last gear to the first and multiplying the little ratios along the way\n",
        "  (that multiplication is the **chain rule**).\n",
        "\n",
        "A neural network is just a very big gear train. Training = nudging each gear a hair in the\n",
        "direction that makes the output less wrong."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "84448625",
      "metadata": {},
      "source": [
        "### 0.1 Imports\n",
        "\n",
        "Pure Python plus a little NumPy/Matplotlib for the toy dataset and pictures. No deep-learning\n",
        "library is needed to build the engine itself — that is the whole point.\n",
        "\n",
        "**-> Training:** `math` gives us `exp`/`tanh` for activations; `random` initialises weights;\n",
        "NumPy and Matplotlib are only for the demo dataset and plots."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e31a52ef",
      "metadata": {},
      "outputs": [],
      "source": [
        "import math\n",
        "import random\n",
        "import numpy as np\n",
        "\n",
        "random.seed(1337)\n",
        "np.random.seed(1337)\n",
        "\n",
        "# Plots are optional: the notebook runs fine without matplotlib (it just skips pictures).\n",
        "try:\n",
        "    import matplotlib.pyplot as plt\n",
        "    HAS_PLT = True\n",
        "except Exception:\n",
        "    HAS_PLT = False\n",
        "    print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
        "\n",
        "print(\"ready\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3315510f",
      "metadata": {},
      "source": [
        "### 0.2 A tiny dataset — two interleaving moons\n",
        "\n",
        "Our goal at the end is to train a small network to separate two groups of dots that curl\n",
        "around each other (\"two moons\"). Real-life picture: two handfuls of red and blue beads\n",
        "mixed in a swirl — can the network learn to draw the boundary between them?\n",
        "\n",
        "We try scikit-learn's `make_moons`; if it is not installed we generate the same shape with\n",
        "NumPy so the notebook stays dependency-light."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "507b4df3",
      "metadata": {},
      "outputs": [],
      "source": [
        "def make_moons_fallback(n_samples=100, noise=0.1):\n",
        "    n = n_samples // 2\n",
        "    t = np.linspace(0, np.pi, n)\n",
        "    # outer moon\n",
        "    x1 = np.stack([np.cos(t), np.sin(t)], axis=1)\n",
        "    # inner moon, shifted\n",
        "    x2 = np.stack([1 - np.cos(t), 1 - np.sin(t) - 0.5], axis=1)\n",
        "    X = np.concatenate([x1, x2], axis=0)\n",
        "    X += noise * np.random.randn(*X.shape)\n",
        "    y = np.array([0] * n + [1] * n)\n",
        "    return X, y\n",
        "\n",
        "try:\n",
        "    from sklearn.datasets import make_moons\n",
        "    X, y = make_moons(n_samples=100, noise=0.1, random_state=1337)\n",
        "    print(\"using sklearn make_moons\")\n",
        "except Exception as e:\n",
        "    X, y = make_moons_fallback(100, noise=0.1)\n",
        "    print(\"sklearn not available, using NumPy fallback:\", type(e).__name__)\n",
        "\n",
        "print(\"X shape:\", X.shape, \" y shape:\", y.shape)\n",
        "print(\"first 3 points:\", X[:3].tolist())\n",
        "print(\"first 3 labels:\", y[:3].tolist())\n",
        "\n",
        "if HAS_PLT:\n",
        "    plt.figure(figsize=(5, 4))\n",
        "    plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n",
        "    plt.title(\"two moons — can a tiny net separate the colors?\")\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0d2fe98b",
      "metadata": {},
      "source": [
        "### 1.1 The `Value` object — a number that remembers\n",
        "\n",
        "A normal Python float forgets where it came from: once you compute `3.0`, nobody knows it\n",
        "was `1.0 + 2.0`. Our `Value` keeps that memory: the numbers that made it (`_prev`) and the\n",
        "operation (`_op`).\n",
        "\n",
        "We start with a **minimal** version that can only do the **forward pass** (no gradients yet),\n",
        "just so you can see the \"remembering\" working.\n",
        "\n",
        "**-> Training:** `_prev` is the list of parent numbers; later, gradients flow backward along\n",
        "exactly these links."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eea9bc18",
      "metadata": {},
      "outputs": [],
      "source": [
        "class Value:\n",
        "    def __init__(self, data, _children=(), _op=''):\n",
        "        self.data = data\n",
        "        self._prev = set(_children)\n",
        "        self._op = _op\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f\"Value(data={self.data})\"\n",
        "\n",
        "    def __add__(self, other):\n",
        "        return Value(self.data + other.data, (self, other), '+')\n",
        "\n",
        "    def __mul__(self, other):\n",
        "        return Value(self.data * other.data, (self, other), '*')\n",
        "\n",
        "\n",
        "a = Value(2.0)\n",
        "b = Value(-3.0)\n",
        "c = Value(10.0)\n",
        "e = a * b\n",
        "d = e + c\n",
        "print(\"d =\", d)\n",
        "print(\"d was made by op:\", repr(d._op))\n",
        "print(\"d's parents:\", d._prev)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e65b3200",
      "metadata": {},
      "source": [
        "### 1.2 Reading the expression graph\n",
        "\n",
        "`d = a*b + c` is really a little tree: `d` points back to `e` and `c`; `e` points back to\n",
        "`a` and `b`. Let's print it as an indented tree so the structure is visible. (No graphviz\n",
        "dependency — just recursion.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3a436597",
      "metadata": {},
      "outputs": [],
      "source": [
        "def show(v, indent=0):\n",
        "    print(\"  \" * indent + f\"{v.data:.4f}  (op={v._op or 'leaf'})\")\n",
        "    for child in v._prev:\n",
        "        show(child, indent + 1)\n",
        "\n",
        "show(d)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ee07490",
      "metadata": {},
      "source": [
        "### 2.1 What a gradient means — nudge and measure\n",
        "\n",
        "Before any clever math, here is the *definition* of a gradient, done the dumb way: change an\n",
        "input by a tiny amount `h`, recompute the output, and see how much it moved.\n",
        "\n",
        "slope = (output after nudge - output before) / h\n",
        "\n",
        "Real-life picture: to feel how steep a hill is, take one small step and notice how much your\n",
        "height changed. That ratio is the slope."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fbfbee18",
      "metadata": {},
      "outputs": [],
      "source": [
        "def f_out(a_, b_, c_):\n",
        "    return (a_ * b_) + c_   # plain floats\n",
        "\n",
        "h = 1e-6\n",
        "base = f_out(2.0, -3.0, 10.0)\n",
        "\n",
        "# how sensitive is the output to a?\n",
        "da = (f_out(2.0 + h, -3.0, 10.0) - base) / h\n",
        "# to b?\n",
        "db = (f_out(2.0, -3.0 + h, 10.0) - base) / h\n",
        "# to c?\n",
        "dc = (f_out(2.0, -3.0, 10.0 + h) - base) / h\n",
        "\n",
        "print(f\"output            = {base}\")\n",
        "print(f\"d(out)/da = {da:.4f}   (equals b = -3)\")\n",
        "print(f\"d(out)/db = {db:.4f}   (equals a = 2)\")\n",
        "print(f\"d(out)/dc = {dc:.4f}   (equals 1)\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d3d3c6e2",
      "metadata": {},
      "source": [
        "### 3.1 The chain rule — multiply the little ratios\n",
        "\n",
        "The nudge-and-measure trick is correct but slow: one re-run per input. Real networks have\n",
        "millions of inputs. The **chain rule** gets every sensitivity in *one* backward pass.\n",
        "\n",
        "The rule, in words: *if a affects e, and e affects d, then a's effect on d is the product of\n",
        "the two local effects.*\n",
        "\n",
        "d(d)/d(a) = d(d)/d(e) * d(e)/d(a)\n",
        "\n",
        "Each operation knows its own **local** derivative:\n",
        "\n",
        "- for `+`: output changes 1-for-1 with each input, so the gradient just **passes through**.\n",
        "- for `*`: each input's local derivative is **the other input**.\n",
        "\n",
        "Backprop = start with gradient 1.0 at the output, then walk backward multiplying by each\n",
        "local derivative."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0da2822d",
      "metadata": {},
      "source": [
        "### 4.1 Adding gradients and local backward rules\n",
        "\n",
        "Now the full `Value`. Each operation also defines a `_backward()` that says how to push the\n",
        "output's gradient onto its inputs. Note we **accumulate** (`+=`) gradients, because one value\n",
        "can feed into several places.\n",
        "\n",
        "We add `+`, `*`, `**` (power), `tanh`, and `exp` — enough to build a neural net. We also add\n",
        "convenience operators (`-`, `/`, right-hand versions) so expressions read naturally.\n",
        "\n",
        "**-> Training:** `tanh` is the neuron's \"squashing\" activation; its local derivative is\n",
        "`1 - tanh(x)^2`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9433f6b8",
      "metadata": {},
      "outputs": [],
      "source": [
        "class Value:\n",
        "    def __init__(self, data, _children=(), _op='', label=''):\n",
        "        self.data = data\n",
        "        self.grad = 0.0\n",
        "        self._backward = lambda: None\n",
        "        self._prev = set(_children)\n",
        "        self._op = _op\n",
        "        self.label = label\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f\"Value(data={self.data:.4f}, grad={self.grad:.4f})\"\n",
        "\n",
        "    def __add__(self, other):\n",
        "        other = other if isinstance(other, Value) else Value(other)\n",
        "        out = Value(self.data + other.data, (self, other), '+')\n",
        "        def _backward():\n",
        "            self.grad += 1.0 * out.grad\n",
        "            other.grad += 1.0 * out.grad\n",
        "        out._backward = _backward\n",
        "        return out\n",
        "\n",
        "    def __mul__(self, other):\n",
        "        other = other if isinstance(other, Value) else Value(other)\n",
        "        out = Value(self.data * other.data, (self, other), '*')\n",
        "        def _backward():\n",
        "            self.grad += other.data * out.grad\n",
        "            other.grad += self.data * out.grad\n",
        "        out._backward = _backward\n",
        "        return out\n",
        "\n",
        "    def __pow__(self, other):\n",
        "        assert isinstance(other, (int, float)), \"only int/float powers\"\n",
        "        out = Value(self.data ** other, (self,), f'**{other}')\n",
        "        def _backward():\n",
        "            self.grad += other * (self.data ** (other - 1)) * out.grad\n",
        "        out._backward = _backward\n",
        "        return out\n",
        "\n",
        "    def tanh(self):\n",
        "        x = self.data\n",
        "        t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1)\n",
        "        out = Value(t, (self,), 'tanh')\n",
        "        def _backward():\n",
        "            self.grad += (1 - t ** 2) * out.grad\n",
        "        out._backward = _backward\n",
        "        return out\n",
        "\n",
        "    def exp(self):\n",
        "        out = Value(math.exp(self.data), (self,), 'exp')\n",
        "        def _backward():\n",
        "            self.grad += out.data * out.grad\n",
        "        out._backward = _backward\n",
        "        return out\n",
        "\n",
        "    # convenience\n",
        "    def __neg__(self):\n",
        "        return self * -1\n",
        "\n",
        "    def __sub__(self, other):\n",
        "        return self + (-other if isinstance(other, Value) else Value(-other))\n",
        "\n",
        "    def __radd__(self, other):\n",
        "        return self + other\n",
        "\n",
        "    def __rmul__(self, other):\n",
        "        return self * other\n",
        "\n",
        "    def __truediv__(self, other):\n",
        "        other = other if isinstance(other, Value) else Value(other)\n",
        "        return self * other ** -1\n",
        "\n",
        "    def backward(self):\n",
        "        # build topological order so children come before parents\n",
        "        topo = []\n",
        "        visited = set()\n",
        "        def build_topo(v):\n",
        "            if v not in visited:\n",
        "                visited.add(v)\n",
        "                for child in v._prev:\n",
        "                    build_topo(child)\n",
        "                topo.append(v)\n",
        "        build_topo(self)\n",
        "        # seed the output gradient, then sweep backward\n",
        "        self.grad = 1.0\n",
        "        for node in reversed(topo):\n",
        "            node._backward()\n",
        "\n",
        "\n",
        "print(\"full Value class ready\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7f14c265",
      "metadata": {},
      "source": [
        "### 4.2 One backward pass on the tiny expression\n",
        "\n",
        "Rebuild `d = a*b + c` with the full class, call `d.backward()` once, and read the gradients.\n",
        "They should match the nudge-and-measure numbers from section 2.1 (`a.grad = b = -3`,\n",
        "`b.grad = a = 2`, `c.grad = 1`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9d1e739c",
      "metadata": {},
      "outputs": [],
      "source": [
        "a = Value(2.0, label='a')\n",
        "b = Value(-3.0, label='b')\n",
        "c = Value(10.0, label='c')\n",
        "e = a * b\n",
        "d = e + c\n",
        "\n",
        "d.backward()\n",
        "\n",
        "print(\"d =\", d.data)\n",
        "print(\"a.grad =\", a.grad, \" (expect -3)\")\n",
        "print(\"b.grad =\", b.grad, \" (expect  2)\")\n",
        "print(\"c.grad =\", c.grad, \" (expect  1)\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ac38e12",
      "metadata": {},
      "source": [
        "### 4.3 Trust but verify — numeric gradient check\n",
        "\n",
        "A good habit: confirm the analytic gradient (from `backward()`) matches the slow\n",
        "nudge-and-measure gradient. If they agree, the engine is correct."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fe4b711e",
      "metadata": {},
      "outputs": [],
      "source": [
        "def numeric_grad(f, inputs, i, h=1e-6):\n",
        "    base = f(inputs)\n",
        "    bumped = list(inputs)\n",
        "    bumped[i] += h\n",
        "    return (f(bumped) - base) / h\n",
        "\n",
        "f = lambda v: (v[0] * v[1]) + v[2]\n",
        "inputs = [2.0, -3.0, 10.0]\n",
        "\n",
        "a = Value(2.0); b = Value(-3.0); c = Value(10.0)\n",
        "(a * b + c).backward()\n",
        "\n",
        "for i, (name, val) in enumerate(zip(\"abc\", (a, b, c))):\n",
        "    ng = numeric_grad(f, inputs, i)\n",
        "    print(f\"{name}: analytic={val.grad:+.4f}   numeric={ng:+.4f}   match={abs(val.grad-ng)<1e-4}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6518159c",
      "metadata": {},
      "source": [
        "**Your turn 4** — Build `g = (a*b + c).tanh()` with `a=0.5, b=2.0, c=-1.0`, call\n",
        "`g.backward()`, and check `a.grad` against a numeric nudge. (Hint: reuse `numeric_grad`\n",
        "with `f = lambda v: math.tanh(v[0]*v[1] + v[2])`.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d6bea75d",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Your turn 4 — fill in and run\n",
        "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n",
        "g = (a * b + c).tanh()\n",
        "g.backward()\n",
        "\n",
        "f = lambda v: math.tanh(v[0]*v[1] + v[2])\n",
        "ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0)\n",
        "print(f\"a.grad analytic = {a.grad:+.4f}   numeric = {ng_a:+.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fef7ef8b",
      "metadata": {},
      "source": [
        "### 5.1 Neuron -> Layer -> MLP, all from `Value`\n",
        "\n",
        "A **neuron** computes `tanh(w . x + b)` — a weighted vote of its inputs, squashed to (-1, 1).\n",
        "A **layer** is a list of neurons. An **MLP** (multi-layer perceptron) is a stack of layers.\n",
        "\n",
        "Real-life picture: each neuron is a tiny committee member weighing the evidence; layers stack\n",
        "committees so later ones judge the opinions of earlier ones.\n",
        "\n",
        "**-> Training:** `parameters()` returns every weight and bias — exactly the knobs we nudge."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1d260369",
      "metadata": {},
      "outputs": [],
      "source": [
        "class Neuron:\n",
        "    def __init__(self, nin):\n",
        "        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]\n",
        "        self.b = Value(random.uniform(-1, 1))\n",
        "\n",
        "    def __call__(self, x):\n",
        "        act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)\n",
        "        return act.tanh()\n",
        "\n",
        "    def parameters(self):\n",
        "        return self.w + [self.b]\n",
        "\n",
        "\n",
        "class Layer:\n",
        "    def __init__(self, nin, nout):\n",
        "        self.neurons = [Neuron(nin) for _ in range(nout)]\n",
        "\n",
        "    def __call__(self, x):\n",
        "        outs = [n(x) for n in self.neurons]\n",
        "        return outs[0] if len(outs) == 1 else outs\n",
        "\n",
        "    def parameters(self):\n",
        "        return [p for n in self.neurons for p in n.parameters()]\n",
        "\n",
        "\n",
        "class MLP:\n",
        "    def __init__(self, nin, nouts):\n",
        "        sizes = [nin] + nouts\n",
        "        self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(nouts))]\n",
        "\n",
        "    def __call__(self, x):\n",
        "        for layer in self.layers:\n",
        "            x = layer(x)\n",
        "        return x\n",
        "\n",
        "    def parameters(self):\n",
        "        return [p for layer in self.layers for p in layer.parameters()]\n",
        "\n",
        "\n",
        "net = MLP(2, [8, 8, 1])   # 2 inputs -> 8 -> 8 -> 1 output\n",
        "print(\"parameter count:\", len(net.parameters()))\n",
        "print(\"one prediction:\", net([Value(0.5), Value(-0.2)]))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fc0b5b7b",
      "metadata": {},
      "source": [
        "### 6.1 Train the MLP on the two moons\n",
        "\n",
        "Loop the four familiar steps:\n",
        "\n",
        "1. **forward** — predict on every point\n",
        "2. **loss** — how wrong (here: mean-squared error against labels mapped to -1 / +1)\n",
        "3. **backward** — `loss.backward()` fills every parameter's `.grad`\n",
        "4. **update** — nudge each parameter a little **against** its gradient (downhill)\n",
        "\n",
        "Watch the loss fall. (Pure-Python micrograd is slow, so we keep the net and step count small.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "de95ef11",
      "metadata": {},
      "outputs": [],
      "source": [
        "net = MLP(2, [8, 8, 1])\n",
        "\n",
        "# map labels {0,1} -> {-1,+1} to match tanh output range\n",
        "ys = [1.0 if yi == 1 else -1.0 for yi in y]\n",
        "Xv = [[Value(float(xi[0])), Value(float(xi[1]))] for xi in X]\n",
        "\n",
        "lr = 0.1\n",
        "for step in range(60):\n",
        "    # forward + MSE loss\n",
        "    preds = [net(xrow) for xrow in Xv]\n",
        "    loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0))\n",
        "    loss = loss * (1.0 / len(ys))\n",
        "\n",
        "    # backward (reset grads first)\n",
        "    for p in net.parameters():\n",
        "        p.grad = 0.0\n",
        "    loss.backward()\n",
        "\n",
        "    # update\n",
        "    for p in net.parameters():\n",
        "        p.data -= lr * p.grad\n",
        "\n",
        "    if step % 10 == 0 or step == 59:\n",
        "        # accuracy\n",
        "        acc = sum((1 if (p.data > 0) == (yt > 0) else 0) for p, yt in zip(preds, ys)) / len(ys)\n",
        "        print(f\"step {step:3d}   loss {loss.data:.4f}   acc {acc:.2%}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9cb30391",
      "metadata": {},
      "source": [
        "### 6.2 See the boundary it learned\n",
        "\n",
        "Evaluate the trained net across a grid of points and color the regions. The swirl between the\n",
        "two moons should now be split by a curved boundary — the network \"drew the line.\"\n",
        "\n",
        "(Coarse grid for speed; each grid point is a full forward pass through micrograd.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ee39626a",
      "metadata": {},
      "outputs": [],
      "source": [
        "if HAS_PLT:\n",
        "    xmin, xmax = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n",
        "    ymin, ymax = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n",
        "    xs = np.linspace(xmin, xmax, 30)\n",
        "    ys_grid = np.linspace(ymin, ymax, 30)\n",
        "\n",
        "    Z = np.zeros((len(ys_grid), len(xs)))\n",
        "    for i, gy in enumerate(ys_grid):\n",
        "        for j, gx in enumerate(xs):\n",
        "            out = net([Value(float(gx)), Value(float(gy))])\n",
        "            Z[i, j] = out.data\n",
        "\n",
        "    plt.figure(figsize=(5, 4))\n",
        "    plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3)\n",
        "    plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n",
        "    plt.title(\"decision boundary learned by the micrograd MLP\")\n",
        "    plt.show()\n",
        "else:\n",
        "    print(\"matplotlib missing - skipping the boundary plot (training above still ran)\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b00ac0fd",
      "metadata": {},
      "source": [
        "**Your turn 6** — Try `MLP(2, [16, 16, 1])` or a different learning rate / step count and\n",
        "re-run 6.1 and 6.2. Does a bigger net reach higher accuracy? Does too-large a learning rate\n",
        "make the loss bounce instead of fall?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48623e88",
      "metadata": {},
      "source": [
        "### 7.1 Bridge to PyTorch — same gradients, industrial engine\n",
        "\n",
        "PyTorch is micrograd scaled up to fast tensors. The *idea* is identical: build an expression,\n",
        "call `.backward()`, read `.grad`. Here we redo the tiny `a*b + c` (then a tanh) in PyTorch and\n",
        "confirm the gradients match what our engine produced.\n",
        "\n",
        "**-> Training:** in notebooks `01` and `06` we let PyTorch's autograd do exactly this — now you\n",
        "know what it is doing under the hood."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e6c3140e",
      "metadata": {},
      "outputs": [],
      "source": [
        "# On some Windows setups torch and numpy ship duplicate OpenMP DLLs; this avoids a clash.\n",
        "import os\n",
        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
        "\n",
        "try:\n",
        "    import torch\n",
        "\n",
        "    at = torch.tensor([0.5], requires_grad=True)\n",
        "    bt = torch.tensor([2.0], requires_grad=True)\n",
        "    ct = torch.tensor([-1.0], requires_grad=True)\n",
        "\n",
        "    gt = torch.tanh(at * bt + ct)\n",
        "    gt.backward()\n",
        "    print(\"PyTorch:   a.grad =\", at.grad.item())\n",
        "except OSError as e:\n",
        "    print(\"PyTorch could not load in this environment:\", e)\n",
        "    print(\"(That is fine here - the micrograd result below is the point.)\")\n",
        "\n",
        "# our engine, same expression\n",
        "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n",
        "(a * b + c).tanh().backward()\n",
        "print(\"micrograd: a.grad =\", a.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "49c1ebf8",
      "metadata": {},
      "source": [
        "## What's next\n",
        "\n",
        "You built the engine that powers all of deep learning:\n",
        "\n",
        "- a **number that remembers** its history (`Value`)\n",
        "- **backpropagation** via the chain rule (`backward()`)\n",
        "- a small **MLP** trained by nudging weights downhill\n",
        "\n",
        "Where this leads in the series:\n",
        "\n",
        "- `01_build_gpt.ipynb` — language modeling basics (the bigram), now that backprop is no longer\n",
        "  mysterious.\n",
        "- `02`-`05` — scale these same ideas up with PyTorch on real character data (`names.txt`).\n",
        "- `06_build_gpt_attention.ipynb` — the transformer, where `loss.backward()` is doing exactly\n",
        "  what you built here, just across millions of `Value`-like nodes.\n",
        "\n",
        "**Checklist**\n",
        "- [ ] A `Value` records data, parents, and operation\n",
        "- [ ] Each op defines a local `_backward`\n",
        "- [ ] `backward()` = topological order + chain rule\n",
        "- [ ] Neuron / Layer / MLP built from `Value`\n",
        "- [ ] Trained on two moons; boundary learned\n",
        "- [ ] Confirmed gradients match PyTorch"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".venv (3.12.10)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}