From 1183d108490f87009b7c0ea8df651942624351ee Mon Sep 17 00:00:00 2001 From: betterbeast2023 Date: Wed, 10 Jun 2026 15:43:32 +0800 Subject: [PATCH] c --- 00_micrograd.ipynb | 768 +++++++++++ 01_build_gpt.ipynb | 2982 +++++++++++++++++++++++++++++++++++----- 01_build_gpt.ipynb.bak | 1864 +++++++++++++++++++++++++ merge_transactions.py | 129 -- 4 files changed, 5254 insertions(+), 489 deletions(-) create mode 100644 00_micrograd.ipynb create mode 100644 01_build_gpt.ipynb.bak delete mode 100644 merge_transactions.py diff --git a/00_micrograd.ipynb b/00_micrograd.ipynb new file mode 100644 index 0000000..5db4d43 --- /dev/null +++ b/00_micrograd.ipynb @@ -0,0 +1,768 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "153d6f74", + "metadata": {}, + "source": [ + "# Micrograd — backprop from scratch\n", + "\n", + "A tiny automatic-gradient engine in pure Python — **concept -> code -> Your turn** each step.\n", + "\n", + "This is notebook **00** of the series and the right place to start: it explains what a\n", + "*gradient* and *backpropagation* really are, using single numbers you can follow by hand.\n", + "Every later notebook (the bigram in `01`, the transformer in `06`) calls `loss.backward()`\n", + "and trusts it. Here we build that machinery ourselves so it is never magic." + ] + }, + { + "cell_type": "markdown", + "id": "aac17ea7", + "metadata": {}, + "source": [ + "## Prologue — what this notebook does, in plain English\n", + "\n", + "We build a small object called a **`Value`**. A `Value` is just a number that also\n", + "**remembers how it was made** (which other numbers, and which operation: plus, times, ...).\n", + "\n", + "Once numbers remember their own history, the computer can answer one very useful question\n", + "automatically:\n", + "\n", + "> *If I nudge this input a tiny bit, how much does the final answer change?*\n", + "\n", + "That sensitivity is the **gradient**. Computing all those sensitivities in one efficient\n", + "backward sweep is **backpropagation** — the single algorithm that trains essentially every\n", + "neural network, including GPT.\n", + "\n", + "This notebook follows Andrej Karpathy's *\"The spelled-out intro to neural networks and\n", + "backpropagation: building micrograd\"* lecture." + ] + }, + { + "cell_type": "markdown", + "id": "44dd4b0d", + "metadata": {}, + "source": [ + "### The whole idea, as a chain of gears\n", + "\n", + "Picture a row of **gears** connected together. You turn the first gear a little; the last\n", + "gear also turns, by an amount that depends on all the gears in between.\n", + "\n", + "- The **forward pass** is turning the first gears and reading the last gear (the output).\n", + "- The **gradient** answers: *if I turn this one gear slightly, how much does the final gear move?*\n", + "- **Backpropagation** is figuring out that answer for **every** gear at once, by walking\n", + " backward from the last gear to the first and multiplying the little ratios along the way\n", + " (that multiplication is the **chain rule**).\n", + "\n", + "A neural network is just a very big gear train. Training = nudging each gear a hair in the\n", + "direction that makes the output less wrong." + ] + }, + { + "cell_type": "markdown", + "id": "84448625", + "metadata": {}, + "source": [ + "### 0.1 Imports\n", + "\n", + "Pure Python plus a little NumPy/Matplotlib for the toy dataset and pictures. No deep-learning\n", + "library is needed to build the engine itself — that is the whole point.\n", + "\n", + "**-> Training:** `math` gives us `exp`/`tanh` for activations; `random` initialises weights;\n", + "NumPy and Matplotlib are only for the demo dataset and plots." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e31a52ef", + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "import random\n", + "import numpy as np\n", + "\n", + "random.seed(1337)\n", + "np.random.seed(1337)\n", + "\n", + "# Plots are optional: the notebook runs fine without matplotlib (it just skips pictures).\n", + "try:\n", + " import matplotlib.pyplot as plt\n", + " HAS_PLT = True\n", + "except Exception:\n", + " HAS_PLT = False\n", + " print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n", + "\n", + "print(\"ready\")" + ] + }, + { + "cell_type": "markdown", + "id": "3315510f", + "metadata": {}, + "source": [ + "### 0.2 A tiny dataset — two interleaving moons\n", + "\n", + "Our goal at the end is to train a small network to separate two groups of dots that curl\n", + "around each other (\"two moons\"). Real-life picture: two handfuls of red and blue beads\n", + "mixed in a swirl — can the network learn to draw the boundary between them?\n", + "\n", + "We try scikit-learn's `make_moons`; if it is not installed we generate the same shape with\n", + "NumPy so the notebook stays dependency-light." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "507b4df3", + "metadata": {}, + "outputs": [], + "source": [ + "def make_moons_fallback(n_samples=100, noise=0.1):\n", + " n = n_samples // 2\n", + " t = np.linspace(0, np.pi, n)\n", + " # outer moon\n", + " x1 = np.stack([np.cos(t), np.sin(t)], axis=1)\n", + " # inner moon, shifted\n", + " x2 = np.stack([1 - np.cos(t), 1 - np.sin(t) - 0.5], axis=1)\n", + " X = np.concatenate([x1, x2], axis=0)\n", + " X += noise * np.random.randn(*X.shape)\n", + " y = np.array([0] * n + [1] * n)\n", + " return X, y\n", + "\n", + "try:\n", + " from sklearn.datasets import make_moons\n", + " X, y = make_moons(n_samples=100, noise=0.1, random_state=1337)\n", + " print(\"using sklearn make_moons\")\n", + "except Exception as e:\n", + " X, y = make_moons_fallback(100, noise=0.1)\n", + " print(\"sklearn not available, using NumPy fallback:\", type(e).__name__)\n", + "\n", + "print(\"X shape:\", X.shape, \" y shape:\", y.shape)\n", + "print(\"first 3 points:\", X[:3].tolist())\n", + "print(\"first 3 labels:\", y[:3].tolist())\n", + "\n", + "if HAS_PLT:\n", + " plt.figure(figsize=(5, 4))\n", + " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n", + " plt.title(\"two moons — can a tiny net separate the colors?\")\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "0d2fe98b", + "metadata": {}, + "source": [ + "### 1.1 The `Value` object — a number that remembers\n", + "\n", + "A normal Python float forgets where it came from: once you compute `3.0`, nobody knows it\n", + "was `1.0 + 2.0`. Our `Value` keeps that memory: the numbers that made it (`_prev`) and the\n", + "operation (`_op`).\n", + "\n", + "We start with a **minimal** version that can only do the **forward pass** (no gradients yet),\n", + "just so you can see the \"remembering\" working.\n", + "\n", + "**-> Training:** `_prev` is the list of parent numbers; later, gradients flow backward along\n", + "exactly these links." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eea9bc18", + "metadata": {}, + "outputs": [], + "source": [ + "class Value:\n", + " def __init__(self, data, _children=(), _op=''):\n", + " self.data = data\n", + " self._prev = set(_children)\n", + " self._op = _op\n", + "\n", + " def __repr__(self):\n", + " return f\"Value(data={self.data})\"\n", + "\n", + " def __add__(self, other):\n", + " return Value(self.data + other.data, (self, other), '+')\n", + "\n", + " def __mul__(self, other):\n", + " return Value(self.data * other.data, (self, other), '*')\n", + "\n", + "\n", + "a = Value(2.0)\n", + "b = Value(-3.0)\n", + "c = Value(10.0)\n", + "e = a * b\n", + "d = e + c\n", + "print(\"d =\", d)\n", + "print(\"d was made by op:\", repr(d._op))\n", + "print(\"d's parents:\", d._prev)" + ] + }, + { + "cell_type": "markdown", + "id": "e65b3200", + "metadata": {}, + "source": [ + "### 1.2 Reading the expression graph\n", + "\n", + "`d = a*b + c` is really a little tree: `d` points back to `e` and `c`; `e` points back to\n", + "`a` and `b`. Let's print it as an indented tree so the structure is visible. (No graphviz\n", + "dependency — just recursion.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a436597", + "metadata": {}, + "outputs": [], + "source": [ + "def show(v, indent=0):\n", + " print(\" \" * indent + f\"{v.data:.4f} (op={v._op or 'leaf'})\")\n", + " for child in v._prev:\n", + " show(child, indent + 1)\n", + "\n", + "show(d)" + ] + }, + { + "cell_type": "markdown", + "id": "5ee07490", + "metadata": {}, + "source": [ + "### 2.1 What a gradient means — nudge and measure\n", + "\n", + "Before any clever math, here is the *definition* of a gradient, done the dumb way: change an\n", + "input by a tiny amount `h`, recompute the output, and see how much it moved.\n", + "\n", + "slope = (output after nudge - output before) / h\n", + "\n", + "Real-life picture: to feel how steep a hill is, take one small step and notice how much your\n", + "height changed. That ratio is the slope." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fbfbee18", + "metadata": {}, + "outputs": [], + "source": [ + "def f_out(a_, b_, c_):\n", + " return (a_ * b_) + c_ # plain floats\n", + "\n", + "h = 1e-6\n", + "base = f_out(2.0, -3.0, 10.0)\n", + "\n", + "# how sensitive is the output to a?\n", + "da = (f_out(2.0 + h, -3.0, 10.0) - base) / h\n", + "# to b?\n", + "db = (f_out(2.0, -3.0 + h, 10.0) - base) / h\n", + "# to c?\n", + "dc = (f_out(2.0, -3.0, 10.0 + h) - base) / h\n", + "\n", + "print(f\"output = {base}\")\n", + "print(f\"d(out)/da = {da:.4f} (equals b = -3)\")\n", + "print(f\"d(out)/db = {db:.4f} (equals a = 2)\")\n", + "print(f\"d(out)/dc = {dc:.4f} (equals 1)\")" + ] + }, + { + "cell_type": "markdown", + "id": "d3d3c6e2", + "metadata": {}, + "source": [ + "### 3.1 The chain rule — multiply the little ratios\n", + "\n", + "The nudge-and-measure trick is correct but slow: one re-run per input. Real networks have\n", + "millions of inputs. The **chain rule** gets every sensitivity in *one* backward pass.\n", + "\n", + "The rule, in words: *if a affects e, and e affects d, then a's effect on d is the product of\n", + "the two local effects.*\n", + "\n", + "d(d)/d(a) = d(d)/d(e) * d(e)/d(a)\n", + "\n", + "Each operation knows its own **local** derivative:\n", + "\n", + "- for `+`: output changes 1-for-1 with each input, so the gradient just **passes through**.\n", + "- for `*`: each input's local derivative is **the other input**.\n", + "\n", + "Backprop = start with gradient 1.0 at the output, then walk backward multiplying by each\n", + "local derivative." + ] + }, + { + "cell_type": "markdown", + "id": "0da2822d", + "metadata": {}, + "source": [ + "### 4.1 Adding gradients and local backward rules\n", + "\n", + "Now the full `Value`. Each operation also defines a `_backward()` that says how to push the\n", + "output's gradient onto its inputs. Note we **accumulate** (`+=`) gradients, because one value\n", + "can feed into several places.\n", + "\n", + "We add `+`, `*`, `**` (power), `tanh`, and `exp` — enough to build a neural net. We also add\n", + "convenience operators (`-`, `/`, right-hand versions) so expressions read naturally.\n", + "\n", + "**-> Training:** `tanh` is the neuron's \"squashing\" activation; its local derivative is\n", + "`1 - tanh(x)^2`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9433f6b8", + "metadata": {}, + "outputs": [], + "source": [ + "class Value:\n", + " def __init__(self, data, _children=(), _op='', label=''):\n", + " self.data = data\n", + " self.grad = 0.0\n", + " self._backward = lambda: None\n", + " self._prev = set(_children)\n", + " self._op = _op\n", + " self.label = label\n", + "\n", + " def __repr__(self):\n", + " return f\"Value(data={self.data:.4f}, grad={self.grad:.4f})\"\n", + "\n", + " def __add__(self, other):\n", + " other = other if isinstance(other, Value) else Value(other)\n", + " out = Value(self.data + other.data, (self, other), '+')\n", + " def _backward():\n", + " self.grad += 1.0 * out.grad\n", + " other.grad += 1.0 * out.grad\n", + " out._backward = _backward\n", + " return out\n", + "\n", + " def __mul__(self, other):\n", + " other = other if isinstance(other, Value) else Value(other)\n", + " out = Value(self.data * other.data, (self, other), '*')\n", + " def _backward():\n", + " self.grad += other.data * out.grad\n", + " other.grad += self.data * out.grad\n", + " out._backward = _backward\n", + " return out\n", + "\n", + " def __pow__(self, other):\n", + " assert isinstance(other, (int, float)), \"only int/float powers\"\n", + " out = Value(self.data ** other, (self,), f'**{other}')\n", + " def _backward():\n", + " self.grad += other * (self.data ** (other - 1)) * out.grad\n", + " out._backward = _backward\n", + " return out\n", + "\n", + " def tanh(self):\n", + " x = self.data\n", + " t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1)\n", + " out = Value(t, (self,), 'tanh')\n", + " def _backward():\n", + " self.grad += (1 - t ** 2) * out.grad\n", + " out._backward = _backward\n", + " return out\n", + "\n", + " def exp(self):\n", + " out = Value(math.exp(self.data), (self,), 'exp')\n", + " def _backward():\n", + " self.grad += out.data * out.grad\n", + " out._backward = _backward\n", + " return out\n", + "\n", + " # convenience\n", + " def __neg__(self):\n", + " return self * -1\n", + "\n", + " def __sub__(self, other):\n", + " return self + (-other if isinstance(other, Value) else Value(-other))\n", + "\n", + " def __radd__(self, other):\n", + " return self + other\n", + "\n", + " def __rmul__(self, other):\n", + " return self * other\n", + "\n", + " def __truediv__(self, other):\n", + " other = other if isinstance(other, Value) else Value(other)\n", + " return self * other ** -1\n", + "\n", + " def backward(self):\n", + " # build topological order so children come before parents\n", + " topo = []\n", + " visited = set()\n", + " def build_topo(v):\n", + " if v not in visited:\n", + " visited.add(v)\n", + " for child in v._prev:\n", + " build_topo(child)\n", + " topo.append(v)\n", + " build_topo(self)\n", + " # seed the output gradient, then sweep backward\n", + " self.grad = 1.0\n", + " for node in reversed(topo):\n", + " node._backward()\n", + "\n", + "\n", + "print(\"full Value class ready\")" + ] + }, + { + "cell_type": "markdown", + "id": "7f14c265", + "metadata": {}, + "source": [ + "### 4.2 One backward pass on the tiny expression\n", + "\n", + "Rebuild `d = a*b + c` with the full class, call `d.backward()` once, and read the gradients.\n", + "They should match the nudge-and-measure numbers from section 2.1 (`a.grad = b = -3`,\n", + "`b.grad = a = 2`, `c.grad = 1`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9d1e739c", + "metadata": {}, + "outputs": [], + "source": [ + "a = Value(2.0, label='a')\n", + "b = Value(-3.0, label='b')\n", + "c = Value(10.0, label='c')\n", + "e = a * b\n", + "d = e + c\n", + "\n", + "d.backward()\n", + "\n", + "print(\"d =\", d.data)\n", + "print(\"a.grad =\", a.grad, \" (expect -3)\")\n", + "print(\"b.grad =\", b.grad, \" (expect 2)\")\n", + "print(\"c.grad =\", c.grad, \" (expect 1)\")" + ] + }, + { + "cell_type": "markdown", + "id": "1ac38e12", + "metadata": {}, + "source": [ + "### 4.3 Trust but verify — numeric gradient check\n", + "\n", + "A good habit: confirm the analytic gradient (from `backward()`) matches the slow\n", + "nudge-and-measure gradient. If they agree, the engine is correct." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fe4b711e", + "metadata": {}, + "outputs": [], + "source": [ + "def numeric_grad(f, inputs, i, h=1e-6):\n", + " base = f(inputs)\n", + " bumped = list(inputs)\n", + " bumped[i] += h\n", + " return (f(bumped) - base) / h\n", + "\n", + "f = lambda v: (v[0] * v[1]) + v[2]\n", + "inputs = [2.0, -3.0, 10.0]\n", + "\n", + "a = Value(2.0); b = Value(-3.0); c = Value(10.0)\n", + "(a * b + c).backward()\n", + "\n", + "for i, (name, val) in enumerate(zip(\"abc\", (a, b, c))):\n", + " ng = numeric_grad(f, inputs, i)\n", + " print(f\"{name}: analytic={val.grad:+.4f} numeric={ng:+.4f} match={abs(val.grad-ng)<1e-4}\")" + ] + }, + { + "cell_type": "markdown", + "id": "6518159c", + "metadata": {}, + "source": [ + "**Your turn 4** — Build `g = (a*b + c).tanh()` with `a=0.5, b=2.0, c=-1.0`, call\n", + "`g.backward()`, and check `a.grad` against a numeric nudge. (Hint: reuse `numeric_grad`\n", + "with `f = lambda v: math.tanh(v[0]*v[1] + v[2])`.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6bea75d", + "metadata": {}, + "outputs": [], + "source": [ + "# Your turn 4 — fill in and run\n", + "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n", + "g = (a * b + c).tanh()\n", + "g.backward()\n", + "\n", + "f = lambda v: math.tanh(v[0]*v[1] + v[2])\n", + "ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0)\n", + "print(f\"a.grad analytic = {a.grad:+.4f} numeric = {ng_a:+.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "fef7ef8b", + "metadata": {}, + "source": [ + "### 5.1 Neuron -> Layer -> MLP, all from `Value`\n", + "\n", + "A **neuron** computes `tanh(w . x + b)` — a weighted vote of its inputs, squashed to (-1, 1).\n", + "A **layer** is a list of neurons. An **MLP** (multi-layer perceptron) is a stack of layers.\n", + "\n", + "Real-life picture: each neuron is a tiny committee member weighing the evidence; layers stack\n", + "committees so later ones judge the opinions of earlier ones.\n", + "\n", + "**-> Training:** `parameters()` returns every weight and bias — exactly the knobs we nudge." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1d260369", + "metadata": {}, + "outputs": [], + "source": [ + "class Neuron:\n", + " def __init__(self, nin):\n", + " self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]\n", + " self.b = Value(random.uniform(-1, 1))\n", + "\n", + " def __call__(self, x):\n", + " act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)\n", + " return act.tanh()\n", + "\n", + " def parameters(self):\n", + " return self.w + [self.b]\n", + "\n", + "\n", + "class Layer:\n", + " def __init__(self, nin, nout):\n", + " self.neurons = [Neuron(nin) for _ in range(nout)]\n", + "\n", + " def __call__(self, x):\n", + " outs = [n(x) for n in self.neurons]\n", + " return outs[0] if len(outs) == 1 else outs\n", + "\n", + " def parameters(self):\n", + " return [p for n in self.neurons for p in n.parameters()]\n", + "\n", + "\n", + "class MLP:\n", + " def __init__(self, nin, nouts):\n", + " sizes = [nin] + nouts\n", + " self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(nouts))]\n", + "\n", + " def __call__(self, x):\n", + " for layer in self.layers:\n", + " x = layer(x)\n", + " return x\n", + "\n", + " def parameters(self):\n", + " return [p for layer in self.layers for p in layer.parameters()]\n", + "\n", + "\n", + "net = MLP(2, [8, 8, 1]) # 2 inputs -> 8 -> 8 -> 1 output\n", + "print(\"parameter count:\", len(net.parameters()))\n", + "print(\"one prediction:\", net([Value(0.5), Value(-0.2)]))" + ] + }, + { + "cell_type": "markdown", + "id": "fc0b5b7b", + "metadata": {}, + "source": [ + "### 6.1 Train the MLP on the two moons\n", + "\n", + "Loop the four familiar steps:\n", + "\n", + "1. **forward** — predict on every point\n", + "2. **loss** — how wrong (here: mean-squared error against labels mapped to -1 / +1)\n", + "3. **backward** — `loss.backward()` fills every parameter's `.grad`\n", + "4. **update** — nudge each parameter a little **against** its gradient (downhill)\n", + "\n", + "Watch the loss fall. (Pure-Python micrograd is slow, so we keep the net and step count small.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "de95ef11", + "metadata": {}, + "outputs": [], + "source": [ + "net = MLP(2, [8, 8, 1])\n", + "\n", + "# map labels {0,1} -> {-1,+1} to match tanh output range\n", + "ys = [1.0 if yi == 1 else -1.0 for yi in y]\n", + "Xv = [[Value(float(xi[0])), Value(float(xi[1]))] for xi in X]\n", + "\n", + "lr = 0.1\n", + "for step in range(60):\n", + " # forward + MSE loss\n", + " preds = [net(xrow) for xrow in Xv]\n", + " loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0))\n", + " loss = loss * (1.0 / len(ys))\n", + "\n", + " # backward (reset grads first)\n", + " for p in net.parameters():\n", + " p.grad = 0.0\n", + " loss.backward()\n", + "\n", + " # update\n", + " for p in net.parameters():\n", + " p.data -= lr * p.grad\n", + "\n", + " if step % 10 == 0 or step == 59:\n", + " # accuracy\n", + " acc = sum((1 if (p.data > 0) == (yt > 0) else 0) for p, yt in zip(preds, ys)) / len(ys)\n", + " print(f\"step {step:3d} loss {loss.data:.4f} acc {acc:.2%}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9cb30391", + "metadata": {}, + "source": [ + "### 6.2 See the boundary it learned\n", + "\n", + "Evaluate the trained net across a grid of points and color the regions. The swirl between the\n", + "two moons should now be split by a curved boundary — the network \"drew the line.\"\n", + "\n", + "(Coarse grid for speed; each grid point is a full forward pass through micrograd.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ee39626a", + "metadata": {}, + "outputs": [], + "source": [ + "if HAS_PLT:\n", + " xmin, xmax = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", + " ymin, ymax = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", + " xs = np.linspace(xmin, xmax, 30)\n", + " ys_grid = np.linspace(ymin, ymax, 30)\n", + "\n", + " Z = np.zeros((len(ys_grid), len(xs)))\n", + " for i, gy in enumerate(ys_grid):\n", + " for j, gx in enumerate(xs):\n", + " out = net([Value(float(gx)), Value(float(gy))])\n", + " Z[i, j] = out.data\n", + "\n", + " plt.figure(figsize=(5, 4))\n", + " plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3)\n", + " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n", + " plt.title(\"decision boundary learned by the micrograd MLP\")\n", + " plt.show()\n", + "else:\n", + " print(\"matplotlib missing - skipping the boundary plot (training above still ran)\")" + ] + }, + { + "cell_type": "markdown", + "id": "b00ac0fd", + "metadata": {}, + "source": [ + "**Your turn 6** — Try `MLP(2, [16, 16, 1])` or a different learning rate / step count and\n", + "re-run 6.1 and 6.2. Does a bigger net reach higher accuracy? Does too-large a learning rate\n", + "make the loss bounce instead of fall?" + ] + }, + { + "cell_type": "markdown", + "id": "48623e88", + "metadata": {}, + "source": [ + "### 7.1 Bridge to PyTorch — same gradients, industrial engine\n", + "\n", + "PyTorch is micrograd scaled up to fast tensors. The *idea* is identical: build an expression,\n", + "call `.backward()`, read `.grad`. Here we redo the tiny `a*b + c` (then a tanh) in PyTorch and\n", + "confirm the gradients match what our engine produced.\n", + "\n", + "**-> Training:** in notebooks `01` and `06` we let PyTorch's autograd do exactly this — now you\n", + "know what it is doing under the hood." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e6c3140e", + "metadata": {}, + "outputs": [], + "source": [ + "# On some Windows setups torch and numpy ship duplicate OpenMP DLLs; this avoids a clash.\n", + "import os\n", + "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n", + "\n", + "try:\n", + " import torch\n", + "\n", + " at = torch.tensor([0.5], requires_grad=True)\n", + " bt = torch.tensor([2.0], requires_grad=True)\n", + " ct = torch.tensor([-1.0], requires_grad=True)\n", + "\n", + " gt = torch.tanh(at * bt + ct)\n", + " gt.backward()\n", + " print(\"PyTorch: a.grad =\", at.grad.item())\n", + "except OSError as e:\n", + " print(\"PyTorch could not load in this environment:\", e)\n", + " print(\"(That is fine here - the micrograd result below is the point.)\")\n", + "\n", + "# our engine, same expression\n", + "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n", + "(a * b + c).tanh().backward()\n", + "print(\"micrograd: a.grad =\", a.grad)" + ] + }, + { + "cell_type": "markdown", + "id": "49c1ebf8", + "metadata": {}, + "source": [ + "## What's next\n", + "\n", + "You built the engine that powers all of deep learning:\n", + "\n", + "- a **number that remembers** its history (`Value`)\n", + "- **backpropagation** via the chain rule (`backward()`)\n", + "- a small **MLP** trained by nudging weights downhill\n", + "\n", + "Where this leads in the series:\n", + "\n", + "- `01_build_gpt.ipynb` — language modeling basics (the bigram), now that backprop is no longer\n", + " mysterious.\n", + "- `02`-`05` — scale these same ideas up with PyTorch on real character data (`names.txt`).\n", + "- `06_build_gpt_attention.ipynb` — the transformer, where `loss.backward()` is doing exactly\n", + " what you built here, just across millions of `Value`-like nodes.\n", + "\n", + "**Checklist**\n", + "- [ ] A `Value` records data, parents, and operation\n", + "- [ ] Each op defines a local `_backward`\n", + "- [ ] `backward()` = topological order + chain rule\n", + "- [ ] Neuron / Layer / MLP built from `Value`\n", + "- [ ] Trained on two moons; boundary learned\n", + "- [ ] Confirmed gradients match PyTorch" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.12.10)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/01_build_gpt.ipynb b/01_build_gpt.ipynb index 5490b4c..8b71bc8 100644 --- a/01_build_gpt.ipynb +++ b/01_build_gpt.ipynb @@ -1,363 +1,2625 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Let's build GPT \u2014 from scratch\n", - "\n", - "We are rebuilding a tiny GPT on the *tiny Shakespeare* dataset, following Karpathy's\n", - "\"Let's build GPT\" but with extra explanation at each step.\n", - "\n", - "**Before running:** in VS Code, click the kernel picker (top-right) and choose the\n", - "interpreter at `gpt_from_scratch/.venv`. That venv has CUDA PyTorch installed.\n", - "\n", - "The roadmap inside this notebook:\n", - "1. Data + tokenization\n", - "2. Batching (the input/target shapes)\n", - "3. A Bigram baseline (simplest possible language model) + training loop\n", - "4. *(next session)* Self-attention \u2014 the heart of the Transformer\n", - "" - ] + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Let's build GPT — from scratch\n", + "\n", + "Tiny GPT on *tiny Shakespeare* — **concept -> code -> Your turn** each step.\n" + ], + "id": "ef460528" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prologue — what this project does, in plain English\n", + "\n", + "This notebook builds a **tiny program that writes like Shakespeare**. You give it\n", + "the first few letters of a sentence (\"To be or\") and it tries to guess what comes\n", + "next — letter by letter — producing lines that look like the plays in `input.txt`.\n", + "\n", + "The program is **dumb at first**. It starts with no knowledge of English.\n", + "It only gets better by reading the text file thousands of times and slowly\n", + "learning which letters tend to follow which.\n", + "\n", + "The whole project follows Andrej Karpathy's *\"Let's build GPT from scratch\"*\n", + "YouTube lecture — this is our hands-on version of it.\n" + ], + "id": "8f2d462e" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The whole thing, in six simple steps\n", + "\n", + "Here is everything that happens, top to bottom, before we write a single line of code:\n", + "\n", + "**1. Read the book.** Load `input.txt` — a few Shakespeare plays glued together into one\n", + "long string of text.\n", + "\n", + "**2. Turn letters into numbers.** The computer doesn't read letters. Every unique character\n", + "gets an id number (e.g. `a → 1`, `b → 2`, ...). The whole book becomes one long list of\n", + "integers.\n", + "\n", + "**3. Slice the book into small windows.** Grab a short row of, say, 8 letters at a time\n", + "(\"Before w\"). Shift that same window by one to the right to make the *answer key*\n", + "(\"efore we\"). Each position in the window is a small flashcard: *given the prefix up to\n", + "here, what is the next letter?*\n", + "\n", + "**4. Give the window to the model.** The model looks at the letters it was given and\n", + "outputs a confidence score for **every possible next character** (all ~65 of them). High\n", + "score for `'f'` after \"Be\" would be good; high score for `'z'` after \"Be\" would be bad.\n", + "\n", + "**5. Measure how wrong the model was, and gently fix it.** A single number called *loss*\n", + "says: *on average, how confident was the model about the true next letter?* If the model\n", + "was confidently wrong, loss is big; the bigger the loss, the bigger the nudge we give to\n", + "the model's internal numbers (its *weights*). Then repeat — grab a new random window,\n", + "score again, nudge again.\n", + "\n", + "**6. After thousands of rounds, ask the model to write something new.** Give it one\n", + "starting character, let it predict the next one, then feed that prediction back in as\n", + "input, and repeat. You'll get a short Shakespeare-sounding paragraph.\n" + ], + "id": "0132ba17" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### A single round of training, pictured simply\n", + "\n", + "Steps 3–5 above repeat inside one big loop. Every time the loop runs, this happens:\n", + "\n", + "```\n", + " grab a small chunk of text → model guesses next letter at every position →\n", + " |\n", + " compare guesses to the real letters ←──────────────┘\n", + " |\n", + " compute ONE number (\"how wrong was it?\")\n", + " |\n", + " nudge every weight slightly in the direction that would\n", + " have made the true letter a bit more likely this time\n", + "```\n", + "\n", + "Every loop uses a **different random chunk** of the book so the model can't just\n", + "memorize one passage. After maybe 10,000 rounds, the weights have quietly learned the\n", + "statistical habits of Shakespearean English — `th` is common, `qx` is rare, vowels\n", + "follow consonants, and so on.\n", + "\n", + "**You don't need to understand every formula.** The goal of this notebook is to make\n", + "the *shape* of the machine visible: how text becomes numbers, how one number (loss)\n", + "drives every change, and what the model actually produces when it's done.\n" + ], + "id": "26549fd7" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 0.1 Imports\n", + "\n", + "These three lines appear in almost every PyTorch project.\n", + "\n", + "**→ Training:** `torch` runs the math, `nn` holds the **weight matrices** we train, `F` computes **loss** (how wrong we are) from predictions.\n" + ], + "id": "91321dde" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "from torch.nn import functional as F\n", + "print(\"PyTorch version:\", torch.__version__)\n" + ], + "execution_count": null, + "outputs": [], + "id": "d99d4439" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 0.2 CPU vs GPU (`device`)\n", + "\n", + "Tensors can live on **CPU** (always works) or **CUDA/GPU** (fast). We pick once\n", + "and reuse `device` so every tensor and model lands in the same place.\n", + "\n", + "If `CUDA available` is False, you installed CPU-only PyTorch — training still\n", + "works but will be slow.\n", + "\n", + "**→ Training:** Model **weights** and each **batch** must sit on the same device. Training on GPU is the same logic, just far faster.\n" + ], + "id": "57084056" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(\"CUDA available:\", torch.cuda.is_available())\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(\"using device:\", device)\n", + "if device == \"cuda\":\n", + " print(\"GPU:\", torch.cuda.get_device_name(0))\n" + ], + "execution_count": null, + "outputs": [], + "id": "c3f68853" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 0.3 What is a tensor?\n", + "\n", + "A **tensor** is an n-dimensional array. `shape (5,)` = 5 numbers in a row.\n", + "`shape (2, 3)` = 2 rows × 3 columns. Token ids will be 1-D tensors.\n", + "\n", + "**→ Training:** Training data (`xb`, `yb`), model **weights**, and **gradients** are all tensors. Slicing `train_data` builds each practice batch.\n" + ], + "id": "6126b5a6" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "row = torch.tensor([10, 20, 30])\n", + "grid = torch.tensor([[1, 2], [3, 4]])\n", + "print(\"row shape:\", row.shape, \" values:\", row.tolist())\n", + "print(\"grid shape:\", grid.shape)\n" + ], + "execution_count": null, + "outputs": [], + "id": "2b37eccd" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 0** — Create a 2×2 tensor of zeros on `device`.\n" + ], + "id": "8a1bb533" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "practice = torch.zeros(2, 2, device=device)\n", + "print(practice)\n", + "print(\"on device:\", practice.device.type == device)\n" + ], + "execution_count": null, + "outputs": [], + "id": "25810c05" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Sidebar — where does the word \"tensor\" come from, and why is it everywhere in AI?**\n", + "\n", + "- **The name.** William Rowan Hamilton used \"tensor\" in the 1840s (for the \"norm\" or magnitude of a quaternion). The **modern mathematical** object—the multi-linear map with a transformation law—was formalised by **Gregorio Ricci-Curbastro** and his student **Tullio Levi-Civita** in the 1880s–1890s as \"absolute differential calculus\". **Albert Einstein** then made tensors famous by using them as the language of spacetime curvature in general relativity (1915).\n", + "\n", + "- **Why they were invented.** Physics (and geometry) should not depend on which coordinate system you pick. Tensors give you a single object whose components *transform in a predictable way* when you change coordinates — this is how Einstein could write equations of gravity that look the same to every observer. Informally: a tensor is a \"multi-linear machine\" that eats several vectors and produces a number; scalars, vectors, and matrices are all special cases.\n", + "\n", + "- **Why they are central in AI.** Nearly every neural-network operation, from embedding lookup through attention to softmax, is a **linear or element-wise non-linear operation on rectangular arrays** of numbers — exactly the shape a tensor is designed for. Three consequences:\n", + " 1. it is the **universal container** for data (`xb`, `yb`), model **weights**, and **gradients**;\n", + " 2. the same `Tensor` type abstracts over **CPU and GPU** (and other accelerators) so the training code is device-agnostic;\n", + " 3. PyTorch's `Tensor` tracks a **computation graph**, so `loss.backward()` can differentiate through any composed operation (automatic differentiation).\n", + "\n", + "So: tensors were built to give physics a **coordinate-free, multi-linear language**; modern deep learning happens to need *precisely* that — plus hardware (GPUs) optimised to crunch those rectangular arrays at scale. That is why `torch.Tensor` is the basic unit of everything in this notebook.\n" + ], + "id": "tensor-history-sidebar" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.1 Load the file\n", + "\n", + "Plain Python: read the whole file into one Python string `text`.\n", + "\n", + "**→ Training:** `text` is the raw pool of examples. Nothing trains until we encode it and sample windows from it.\n" + ], + "id": "f7039a6a" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "with open(\"input.txt\", \"r\", encoding=\"utf-8\") as f:\n", + " text = f.read()\n", + "print(\"characters in file:\", len(text))\n", + "print(\"---- first 200 chars ----\")\n", + "print(text[:200])\n" + ], + "execution_count": null, + "outputs": [], + "id": "9e4fa669" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.2 What's inside?\n", + "\n", + "The model sees **every** character: letters, spaces, punctuation, newlines.\n", + "Newlines matter — they mark line breaks in the plays.\n", + "\n", + "**→ Training:** The model learns the **statistics** of these characters (spaces, newlines, letter pairs). It does not store the file verbatim in weights.\n" + ], + "id": "83f3fde7" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(\"newlines:\", text.count(\"\\n\"))\n", + "print(\"unique character count:\", len(set(text)))\n", + "print(\"first line:\", repr(text.split(\"\\n\")[0][:60]))\n" + ], + "execution_count": null, + "outputs": [], + "id": "31f45b4a" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 1** — Print the last 100 characters of `text`.\n" + ], + "id": "9bfde4e7" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(repr(text[-100:]))\n" + ], + "execution_count": null, + "outputs": [], + "id": "dc5dfe07" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.1 Vocabulary\n", + "\n", + "The **vocabulary** = all distinct characters in the training text. `vocab_size`\n", + "is how many different tokens the model can output.\n", + "\n", + "**→ Training:** `vocab_size` fixes the width of the output scores. Each training step: model outputs `vocab_size` logits; truth is **one** of those ids.\n" + ], + "id": "78bd31aa" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "chars = sorted(list(set(text)))\n", + "vocab_size = len(chars)\n", + "print(\"vocab_size:\", vocab_size)\n", + "print(\"all chars:\", repr(\"\".join(chars)))\n" + ], + "execution_count": null, + "outputs": [], + "id": "07af82cf" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.2 Lookup tables: `stoi` and `itos`\n", + "\n", + "- `stoi` (**s**tring **to** **i**nt): character → id\n", + "- `itos` (**i**nt **to** **s**tring): id → character\n", + "\n", + "**→ Training:** `stoi` converts batches of text to `xb`/`yb` tensors. `itos` is for us humans when inspecting generated output.\n" + ], + "id": "9f03ff92" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "stoi = {ch: i for i, ch in enumerate(chars)}\n", + "itos = {i: ch for i, ch in enumerate(chars)}\n", + "print('stoi[h] =', stoi['h'])\n", + "print('itos[0] =', repr(itos[0]))\n" + ], + "execution_count": null, + "outputs": [], + "id": "3e0b4759" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3 `encode` and `decode`\n", + "\n", + "Encode: string → list of ids. Decode: list of ids → string. Round-trip should match.\n", + "\n", + "**→ Training:** `encode(text)` built the master tape `data`. Every training batch is a slice of those ids — never raw strings inside the model.\n" + ], + "id": "8f5a3085" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "encode = lambda s: [stoi[c] for c in s]\n", + "decode = lambda l: \"\".join(itos[i] for i in l)\n", + "demo = \"hello there\"\n", + "ids = encode(demo)\n", + "print(demo, \"->\", ids)\n", + "print(\"back:\", decode(ids))\n" + ], + "execution_count": null, + "outputs": [], + "id": "14c4e3a3" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 2** — Encode a short phrase. Flip one id and decode — see one wrong letter.\n" + ], + "id": "40dd1ac1" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "phrase = \"First Citizen\"\n", + "ids = encode(phrase)\n", + "ids[3] = (ids[3] + 1) % vocab_size\n", + "print(\"corrupted:\", decode(ids))\n" + ], + "execution_count": null, + "outputs": [], + "id": "0b50c865" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "ids_list = encode(\"First\")\n", + "ids_tensor = torch.tensor(ids_list, dtype=torch.long)\n", + "\n", + "print(\"Python list:\", ids_list)\n", + "print(\"tensor: \", ids_tensor)\n", + "print(\"shape:\", ids_tensor.shape)\n", + "print(\"dtype:\", ids_tensor.dtype)\n", + "print()\n", + "print(\"slice [1:3]:\", ids_tensor[1:3])\n", + "print(\"decode slice:\", decode(ids_tensor[1:3].tolist()))\n", + "print(\"back to list:\", ids_tensor.tolist())\n" + ], + "execution_count": null, + "outputs": [], + "id": "21051ca8" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 3.1** — Encode a word of your choice. Build a tensor, print `.shape`\n", + "and `.dtype`. Slice the middle two ids and decode them.\n" + ], + "id": "7b498e68" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "word = \"Citizen\" # edit me\n", + "t = torch.tensor(encode(word), dtype=torch.long)\n", + "print(\"shape:\", t.shape, \" dtype:\", t.dtype)\n", + "mid = t[1:3]\n", + "print(\"middle ids:\", mid.tolist(), \"->\", decode(mid.tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "c40b416b" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2 String → tensor (the whole book)\n", + "\n", + "Now we wrap **every** token id in the dataset into one 1-D tensor `data`.\n", + "Same idea as 3.1, but ~1 million ids long — our **tape**.\n", + "\n", + "`dtype=torch.long` = integers only. Shape `(N,)` = one long row of length `N`.\n", + "\n", + "**→ Training:** `data` / `train_data` never change during training. Only the **model weights** change. The tape is fixed questions; weights are learned answers.\n" + ], + "id": "d1c84cec" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "data = torch.tensor(encode(text), dtype=torch.long)\n", + "print(\"shape:\", data.shape, \" dtype:\", data.dtype)\n", + "print(\"first 10 ids:\", data[:10].tolist())\n", + "print(\"first 10 chars:\", decode(data[:10].tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "d419690c" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.3 Train / validation split\n", + "\n", + "- **train_data** — model learns from this\n", + "- **val_data** — later we check loss on unseen tail of the book\n", + "\n", + "If train loss falls but val loss stays high → **memorising**, not generalising.\n", + "\n", + "**→ Training:** We **only** run `backward()` on train batches. Val loss is read-only: if train improves but val does not, weights are memorising.\n" + ], + "id": "5450cf66" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "n = int(0.9 * len(data))\n", + "train_data = data[:n]\n", + "val_data = data[n:]\n", + "print(\"train:\", len(train_data), \" val:\", len(val_data))\n", + "print(\"sum check:\", len(train_data) + len(val_data) == len(data))\n" + ], + "execution_count": null, + "outputs": [], + "id": "bbc77a95" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.4 The split boundary in plain text\n", + "\n", + "Train ends and val begins at the same place in the original book — no gap, no overlap.\n", + "\n", + "**→ Training:** Confirms train and val are contiguous slices — so val loss is a fair \"unseen tail of the book\" check.\n" + ], + "id": "dcf68150" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(\"...end of train:\", repr(decode(train_data[-40:].tolist())))\n", + "print(\"start of val: \", repr(decode(val_data[:40].tolist())))\n" + ], + "execution_count": null, + "outputs": [], + "id": "e9bc8a68" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 3** — What fraction of the book is validation? *(~10%.)*\n" + ], + "id": "51b55dd8" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "frac_val = len(val_data) / len(data)\n", + "print(f\"validation fraction: {frac_val:.1%}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "70e5f2ac" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.1 The next-token game (one step)\n", + "\n", + "The training question is always the same:\n", + "\n", + "> Given what came before → what is the **single** next character?\n", + "\n", + "There is no separate labels file. The label is hiding **in the text itself**:\n", + "the next character in the sequence.\n", + "\n", + "**Analogy:** a friend starts a sentence and you blurt out the next word. They\n", + "tell you if you were right. That's one training step.\n", + "\n", + "**→ Training:** One row of the loss compares model scores to **one** correct next id. A batch contains `B × T` such comparisons at once.\n" + ], + "id": "e7a8f7c0" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# tiny phrase from our vocabulary — watch one step at a time\n", + "phrase = \"First Citizen\"\n", + "ids = encode(phrase)\n", + "print(\"text: \", phrase)\n", + "print(\"ids: \", ids)\n", + "print()\n", + "for i in range(len(ids) - 1):\n", + " context = decode(ids[: i + 1])\n", + " nxt = itos[ids[i + 1]]\n", + " print(f\" seen {context!r:20s} -> predict {nxt!r}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "6adf5e2f" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 4.1** — Change `phrase` to another short string (letters/spaces only).\n", + "How many training steps does a string of length *L* give? *(Answer: L − 1.)*\n" + ], + "id": "537d244e" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "phrase = \"To be\" # edit me\n", + "ids = encode(phrase)\n", + "steps = len(ids) - 1\n", + "print(f\"length {len(ids)} -> {steps} next-token questions\")\n", + "for i in range(steps):\n", + " print(f\" {decode(ids[:i+1])!r} -> {itos[ids[i+1]]!r}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "49b69d41" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Consolidated walk-through — from text to training (no need to run code)\n", + "\n", + "This cell is a **self-contained explanation** of the chain: `input.txt` → `stoi`/`itos` → `x` and `y` → loss → one training step. You can read it top-to-bottom without executing anything below.\n", + "\n", + "---\n", + "\n", + "**1. The raw file (`input.txt`) looks like:**\n", + "\n", + "```\n", + "First Citizen:\n", + "Before we proceed any further, hear me speak.\n", + "```\n", + "\n", + "Every single character — letters, spaces ` `, newlines `\\n`, punctuation — is a potential token. The model never sees strings inside its computation; everything becomes integers.\n", + "\n", + "**2. The vocabulary (`chars`, `vocab_size`, `stoi`, `itos`)**\n", + "\n", + "`chars = sorted(list(set(text)))` gives every distinct character in the file, sorted. `vocab_size` is its length (about 65 for Shakespeare). Then we build two dictionaries:\n", + "\n", + "```python\n", + "stoi = {ch: i for i, ch in enumerate(chars)} # char -> integer id\n", + "itos = {i: ch for i, ch in enumerate(chars)} # integer id -> char\n", + "```\n", + "\n", + "Sorted order means, for example: `'\\n'` is id 0, `' '` is id 1, `'!'` is id 2, and so on through the uppercase letters, then the lowercase letters, to id 64 (something like `'z'`). The exact ids don't matter conceptually — what matters is that every character has **one** id and every id has **one** character.\n", + "\n", + "**3. `encode` and `decode` — the round-trip**\n", + "\n", + "```python\n", + "encode = lambda s: [stoi[c] for c in s]\n", + "decode = lambda l: \"\".join(itos[i] for i in l)\n", + "```\n", + "\n", + "Concrete example — `\"First Citizen\"` (13 characters):\n", + "\n", + "```\n", + "text: \"First Citizen\"\n", + "encode → [id_of_F, id_of_i, id_of_r, id_of_s, id_of_t, id_of_space, id_of_C, id_of_i, id_of_t, id_of_i, id_of_z, id_of_e, id_of_n]\n", + "decode → \"First Citizen\" (round-trip matches)\n", + "```\n", + "\n", + "**4. The next-token game (section 4.1) — one string → many training questions**\n", + "\n", + "The model's job is: *given everything to the left, predict the single next character.* There is no separate \"answers file\" — the answer is the **next character in the string itself**. A string of length L contains **L − 1** training questions, because at every position from 0 to L−2 we can ask: what comes next?\n", + "\n", + "For `\"First Citizen\"` (13 chars), that is **12 questions**:\n", + "\n", + "| context seen (prefix) | must predict |\n", + "|---|---|\n", + "| `\"F\"` | `'i'` |\n", + "| `\"Fi\"` | `'r'` |\n", + "| `\"Fir\"` | `'s'` |\n", + "| `\"Firs\"` | `'t'` |\n", + "| `\"First\"` | `' '` (space) |\n", + "| `\"First \"` | `'C'` |\n", + "| `\"First C\"` | `'i'` |\n", + "| `\"First Ci\"` | `'t'` |\n", + "| `\"First Cit\"` | `'i'` |\n", + "| `\"First Citi\"` | `'z'` |\n", + "| `\"First Citiz\"` | `'e'` |\n", + "| `\"First Citize\"` | `'n'` |\n", + "\n", + "The model never \"sees\" the target character during its forward pass — it only sees the prefix. It produces scores for **every** character in the vocabulary, and loss measures how little probability mass landed on the one true character.\n", + "\n", + "**5. From the tape to `x` and `y` (sections 4.3–4.4)**\n", + "\n", + "In practice we don't pass in arbitrary short strings. We take the whole training text, convert it to one long 1-D tensor of ids (the \"tape\"), and slice windows of `block_size` (8 here). From a slice we build two equal-length tensors:\n", + "\n", + "```python\n", + "block_size = 8\n", + "x = train_data[1000 : 1000 + block_size] # 8 ids, the context\n", + "y = train_data[1001 : 1001 + block_size] # 8 ids, the answers (shifted by +1)\n", + "```\n", + "\n", + "If at that position on the tape the text reads `\"Before we proceed...\"`, then:\n", + "\n", + "* `x` = ids for `\"Before w\"` (8 chars)\n", + "* `y` = ids for `\"efore we\"` (8 chars, same positions shifted right by one)\n", + "\n", + "Written as a table with growing prefixes:\n", + "\n", + "| position `t` | prefix the model sees (from `x`) | must predict (from `y[t]`) |\n", + "|---|---|---|\n", + "| 0 | `\"B\"` | `'e'` |\n", + "| 1 | `\"Be\"` | `'f'` |\n", + "| 2 | `\"Bef\"` | `'o'` |\n", + "| 3 | `\"Befo\"` | `'r'` |\n", + "| 4 | `\"Befor\"` | `'e'` |\n", + "| 5 | `\"Before\"` | `' '` (space) |\n", + "| 6 | `\"Before \"` | `'w'` |\n", + "| 7 | `\"Before w\"` | `'e'` |\n", + "\n", + "That is **8 supervised questions** from one 8-character row, processed in a single forward pass.\n", + "\n", + "**6. What `logits` and `loss` actually look like at one position**\n", + "\n", + "At each position `t` the model emits `vocab_size` raw scores (`logits`) — one score per possible character in the vocabulary. At `t=1` (context `\"Be\"`, true next char `'f'`), the picture is:\n", + "\n", + "```\n", + "logits at t=1 for all 65 chars:\n", + " 'a' logit = -0.3\n", + " 'c' logit = +2.1 (model's strongest guess — WRONG)\n", + " 'e' logit = +1.5\n", + " 'f' logit = +0.8 (TRUE answer y[1] = 'f')\n", + " 'z' logit = -2.0\n", + " ... (the other 61 chars)\n", + "\n", + "after softmax (squashed into a probability distribution):\n", + " P('c') ≈ 0.52 model wastes over half its confidence on 'c'\n", + " P('e') ≈ 0.28\n", + " P('f') ≈ 0.14 only 14% on the true character\n", + " ... (the rest share what's left)\n", + "\n", + "loss at t=1 = -ln(P('f')) = -ln(0.14) ≈ 1.97\n", + "```\n", + "\n", + "Cross-entropy asks **one question** per position: *how much probability mass landed on the single correct character?* It never compares strings, and it doesn't care which *wrong* characters were ranked highly — only how far `P(true_char)` is from 1.0.\n", + "\n", + "**7. Averaging over the row → one `loss` number**\n", + "\n", + "Loss is averaged across all 8 positions (and across all rows in a batch):\n", + "\n", + "```\n", + "loss = ( loss(\"B\"→'e') + loss(\"Be\"→'f') + loss(\"Bef\"→'o') + ... + loss(\"Before w\"→'e') ) / 8\n", + "```\n", + "\n", + "Then `loss.backward()` computes, for every weight in the model, which direction to nudge it so loss would be slightly smaller. `optimizer.step()` applies that nudge. Repeat this loop thousands of times and the model gradually learns which characters tend to follow which in Shakespearean English.\n", + "\n", + "**8. Generation (the \"spit out words\" loop)**\n", + "\n", + "After training, generation works exactly the same way as the forward pass, but in a loop:\n", + "\n", + "```\n", + "start: [newline_id] # one id, representing \"\\n\"\n", + " |\n", + " v\n", + "model(idx) → logits at the *last* position only\n", + " |\n", + " v\n", + "softmax(logits) → probability distribution over all 65 characters\n", + " |\n", + " v\n", + "sample one id (torch.multinomial) — not always the most-probable one\n", + " |\n", + " v\n", + "append that id to idx (the sequence grows by one)\n", + " |\n", + " v\n", + "repeat (loop back) with the longer sequence\n", + "```\n", + "\n", + "Finally, `decode(sequence_of_ids)` converts the produced list of integers back to a readable string. There is **no stored English, no facts, no rules** inside the model — only floating-point weights encoding co-occurrence statistics.\n", + "\n", + "**9. Shape summary (carry this mental picture into section 5 and beyond)**\n", + "\n", + "| object | shape | meaning |\n", + "|---|---|---|\n", + " `data` | `(N,)` | the whole book, one id per character (a 1-D tensor) |\n", + " `xb` | `(B, T)` | batch of context windows — B rows, T = block_size columns of ids |\n", + " `yb` | `(B, T)` | batch of answer windows — same shape as xb, shifted by +1 on the tape |\n", + " `logits` | `(B, T, vocab_size)` | one score per character, per position, per row |\n", + " `loss` | `(1,)` — a scalar | one number measuring how well the model predicts `yb` from `xb` |\n", + " a generated sequence | `(1, T_generated)` | one row, one id per generated character |\n", + "\n", + "That is the entire training-and-generation pipeline. Every code cell below is a step that makes this pipeline either more visible (printing shapes, inspecting slices) or more powerful (building a better model than the bigram baseline)." + ], + "id": "consolidation-4x" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(\"train_data length:\", len(train_data))\n", + "print(\"first 15 ids: \", train_data[:15].tolist())\n", + "print(\"first 15 chars: \", decode(train_data[:15].tolist()))\n", + "print(\"chars 15-30: \", decode(train_data[15:30].tolist()))\n", + "print()\n", + "print(\"The tape is continuous — char 15 follows char 14 in the real book.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "d217e75d" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.3 Why we use chunks (`block_size`)\n", + "\n", + "We could slide one character at a time, but in practice we grab a **window** of\n", + "`block_size` characters at once. Three reasons:\n", + "\n", + "1. **Memory** — models store state per position; a 1M-wide window would not fit on a GPU.\n", + "2. **Fixed context** — Transformers are built to look at at most `block_size` tokens.\n", + "3. **Efficiency** — one forward pass on 8 positions trains 8 questions in parallel (next part).\n", + "\n", + "`block_size` is the **context window**: how far back the model is *allowed* to look.\n", + "We use 8 here so the numbers fit on screen; GPT-4 uses thousands.\n", + "\n", + "**→ Training:** `block_size` = how many next-token questions per row (`T`). Also caps context length the architecture can use.\n" + ], + "id": "0ecf904b" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "block_size = 8\n", + "start = 1000 # where we cut the tape\n", + "chunk = train_data[start : start + block_size]\n", + "\n", + "print(f\"block_size = {block_size}\")\n", + "print(f\"slice from index {start} to {start + block_size - 1}\")\n", + "print(\"chunk ids: \", chunk.tolist())\n", + "print(\"chunk text: \", repr(decode(chunk.tolist())))\n", + "print()\n", + "print(\"book length:\", len(train_data), \" window:\", block_size,\n", + " \" ratio:\", round(len(train_data) / block_size))\n" + ], + "execution_count": null, + "outputs": [], + "id": "7b330af9" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 4.3** — Change `start`. The chunk should always be exactly `block_size`\n", + "characters (except if you pick a start too close to the end).\n" + ], + "id": "f19d6e14" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "start = 25000 # try different values\n", + "chunk = train_data[start : start + block_size]\n", + "print(\"len(chunk):\", len(chunk), \" text:\", repr(decode(chunk.tolist())))\n" + ], + "execution_count": null, + "outputs": [], + "id": "8528eb17" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.4 Inputs `x` and targets `y` (shift by one)\n", + "\n", + "From one slice we build two equal-length tensors:\n", + "\n", + "- **`x`** = the window `train_data[start : start + block_size]`\n", + "- **`y`** = the same window, shifted **one step forward** in time:\n", + " `train_data[start + 1 : start + block_size + 1]`\n", + "\n", + "So `y` is always \"what comes next\" relative to `x`, position by position:\n", + "\n", + "```\n", + "x: h e l l o _ t h\n", + "y: e l l o _ t h ? ← ? is the 9th char (first char AFTER the window)\n", + "```\n", + "\n", + "The model never sees `y` during the forward pass — only during loss calculation.\n", + "\n", + "#### Tie-in to the training goal\n", + "\n", + "**Ultimate goal:** nudge **weights** so the model scores the true next character highly.\n", + "\n", + "`x` and `y` are how we package **one practice round** for that goal:\n", + "\n", + "| Tensor | Role in training | Analogy |\n", + "|--------|------------------|---------|\n", + "| **`x`** (later `xb`) | **Question** — tokens the model reads to produce logits | exam paper |\n", + "| **`y`** (later `yb`) | **Answer key** — the correct next token at each position | mark scheme |\n", + "\n", + "At column `t`: model reads `x[t]`, predicts a distribution; loss compares it to **`y[t]`** (the char that really followed on the tape). Wrong guess → higher loss → `backward()` pushes weights to do better next time.\n", + "\n", + "**In the chain (why this section exists):**\n", + "```\n", + "GOAL: better weights\n", + " ^ needs backward()\n", + " ^ needs loss (one number)\n", + " ^ needs yb — the CORRECT next tokens ← y is the template for yb\n", + " ^ needs forward on xb — model predictions ← x is the template for xb\n", + "```\n", + "\n", + "Without the shift-by-one split you have text but **no labelled examples** — nothing to score, no loss, no training.\n", + "\n", + "**→ Training:** `get_batch` builds matrices `xb`/`yb` the same way as `x`/`y` here. Every training step is: predict from `xb`, grade against `yb`, update weights.\n", + "\n" + ], + "id": "342c105f" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"x:\", decode(x.tolist()))\n", + "print(\"y:\", decode(y.tolist()))\n", + "print()\n", + "for i in range(block_size):\n", + " print(f\" x[{i}]={repr(itos[x[i].item()])} y[{i}]={repr(itos[y[i].item()])}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "0dec560b" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 4.4** — Confirm `y[i]` is always the id **right after** `x[i]` on the tape.\n" + ], + "id": "70613e82" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "ok = all(y[t].item() == train_data[start + t + 1].item() for t in range(block_size))\n", + "print(\"every y[t] is the next char after x[t] on the tape:\", ok)\n", + "if not ok:\n", + " for t in range(block_size):\n", + " print(t, x[t].item(), y[t].item(), train_data[start + t + 1].item())\n" + ], + "execution_count": null, + "outputs": [], + "id": "5be4764d" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.5 The prefix trick (one chunk → many lessons)\n", + "\n", + "Here is the key idea that confuses many beginners:\n", + "\n", + "An 8-character chunk is **not** one training example. It is **eight** examples\n", + "packed together — one for each prefix length.\n", + "\n", + "| step `t` | context (prefix of `x`) | must predict `y[t]` |\n", + "|----------|---------------------------|---------------------|\n", + "| 0 | 1st char only | 2nd char |\n", + "| 1 | 1st + 2nd | 3rd char |\n", + "| 2 | first 3 chars | 4th char |\n", + "| … | … | … |\n", + "| 7 | all 8 chars | 9th char (just past the window) |\n", + "\n", + "**Analogy:** teaching spelling by asking \"after **h**?\", then \"after **he**?\",\n", + "then \"after **hel**?\" — same word, harder each time.\n", + "\n", + "**→ Training:** One slice yields `block_size` supervised examples per forward pass — more learning signal per batch.\n" + ], + "id": "6fecc0fb" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.5b Worked example — `x='Before w'`, `y='efore we'`\n", + "\n", + "The row above `(x='Before w', y='efore we')` is **not** one training example. It is\n", + "**8** training examples packed together — one for each growing prefix of `x`. Here\n", + "they are, written out:\n", + "\n", + "| Position | Prefix the model sees (from `x`) | Must predict (from `y`) |\n", + "|----------|-----------------------------------|--------------------------|\n", + "| `t=0` | `\"B\"` | `'e'` |\n", + "| `t=1` | `\"Be\"` | `'f'` |\n", + "| `t=2` | `\"Bef\"` | `'o'` |\n", + "| `t=3` | `\"Befo\"` | `'r'` |\n", + "| `t=4` | `\"Befor\"` | `'e'` |\n", + "| `t=5` | `\"Before\"` | `' '` (space) |\n", + "| `t=6` | `\"Before \"` | `'w'` |\n", + "| `t=7` | `\"Before w\"` | `'e'` |\n", + "\n", + "Important: `y[t]` at every position is **one single character**, not a string.\n", + "The fact that `y` reads like a word (`'efore we'`) is a human reading artifact —\n", + "to the training loop it is an array of 8 independent integer ids.\n" + ], + "id": "4cb6a5d2" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What the model outputs per position, and what cross-entropy *actually* compares\n", + "\n", + "At **each** position `t` the model emits `vocab_size` raw logits — one score per\n", + "possible character. `softmax(logits)` then squashes those 65 numbers into a proper\n", + "probability distribution (all positive, sums to 1.0).\n", + "\n", + "Cross-entropy then asks **one question per position**:\n", + "\n", + "> *How much of the 1.0 probability budget landed on the ONE correct character* `y[t]`?\n", + "\n", + "It does **not** compare strings, it does **not** compare predictions to whole chunks\n", + "of text, and it does **not** care which other characters were ranked high or low —\n", + "it only reads off `P(true_char)` at that position and computes `-ln(P(true_char))`.\n", + "\n", + "Concrete scenario at `t=1` (context = `\"Be\"`, true next char = `'f'`):\n", + "\n", + "```\n", + "model's top-3 by logit at t=1:\n", + " 'c' logit = +2.1 (model's highest guess, WRONG)\n", + " 'e' logit = +1.5\n", + " 'f' logit = +0.8 (TRUE answer y[1] = 'f')\n", + " ...\n", + "\n", + "after softmax:\n", + " P('c') ~ 0.52 model wasted over half its confidence on 'c'\n", + " P('e') ~ 0.28\n", + " P('f') ~ 0.14 only 14% on the true character\n", + " ...\n", + "\n", + "loss at t=1 = -ln(0.14) = 1.97\n", + "```\n", + "\n", + "Whether the model's *favorite* was `'c'` or `'q'` or `'Z'` doesn't matter for the\n", + "math — only the probability mass on `'f'` does. The other 64 characters jointly\n", + "soak up probability away from the true character, and that is what makes loss large.\n" + ], + "id": "81710b12" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What softmax does — the pie metaphor\n", + "\n", + "Think of softmax as splitting **one pie** (100% of your confidence) across all 65\n", + "possible next characters.\n", + "\n", + "- Each character starts with a **raw score** (its logit). A higher score means \"I think this letter is more likely.\"\n", + "- Softmax turns those scores into **slice sizes** (probabilities) that are always positive and **add up to exactly 1.0** — the whole pie is always eaten, nothing left over.\n", + "\n", + "If one character's logit is much bigger than the rest, softmax gives it a big slice.\n", + "If every logit is about the same, the pie is cut into roughly equal slices.\n", + "\n", + "**The math (two steps, applied to all 65 logits at once):**\n", + "\n", + "1. **Exponentiate** — raise *e* to the power of each logit: `exp(z_i)`. This makes every value positive and amplifies differences (a logit of +2.1 becomes much larger than one of +0.8).\n", + "2. **Normalise** — divide each result by the **sum** of all 65 exponentiated values:\n", + "\n", + "```\n", + "P(character i) = exp(logit_i) / sum over all j of exp(logit_j)\n", + "```\n", + "\n", + "So softmax is not one exotic trick — it is **exp**, then **divide by the total**.\n", + "That division is what forces the 65 probabilities to sum to 1.0.\n", + "\n", + "**In this notebook (programming language):** the notebook is written in **Python**.\n", + "Softmax is called through **PyTorch** — either `torch.softmax(logits, dim=-1)` or\n", + "`F.softmax(logits, dim=-1)` (same operation; `F` is `torch.nn.functional`). The\n", + "`dim=-1` argument means \"run softmax along the last axis\" — i.e. across the 65\n", + "character scores at each position, independently.\n" + ], + "id": "softmax-pie-metaphor" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### From Python to machine code (summary)\n", + "\n", + "You never implement softmax by hand in a Python loop. You call one function; layers\n", + "below your notebook do the heavy work.\n", + "\n", + "| Layer | What runs it | What it does |\n", + "| --- | --- | --- |\n", + "| Your notebook | Python | Calls `torch.softmax(logits, dim=-1)` |\n", + "| PyTorch engine | C++ (and CUDA on GPU) | Receives the tensor and picks a fast kernel |\n", + "| Math libraries | Vectorised C / CUDA | `exp` on every logit, sum them, divide each by the total |\n", + "| Machine code | x86 / ARM (CPU) or GPU SASS (NVIDIA) | Load floats from memory, multiply/add/divide, write probabilities back |\n", + "\n", + "**What the chip actually does** for one position's 65 logits: load 65 numbers,\n", + "exponentiate each, add them up, divide each exponential by that sum, store 65\n", + "probabilities. On a GPU, many positions run in parallel.\n", + "\n", + "**Take-home:** the *idea* is the pie metaphor (scores -> slices summing to 1.0). The\n", + "*implementation* is compiled numeric code — not Python — optimised for speed on your\n", + "CPU or GPU.\n" + ], + "id": "softmax-machine-code" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Verify the claim above on the exact row: x='Before w', y='efore we'.\n", + "# We run ONE forward pass on this row and print, for every position t:\n", + "# (a) the logit of the true char, (b) its softmax prob, (c) -ln(prob).\n", + "# Then we compare mean(-ln(prob)) to F.cross_entropy to confirm they match.\n", + "\n", + "import math\n", + "\n", + "x_str = \"Before w\"\n", + "y_str = \"efore we\"\n", + "\n", + "x_ids = torch.tensor([stoi[c] for c in x_str], dtype=torch.long, device=device).unsqueeze(0)\n", + "y_ids = torch.tensor([stoi[c] for c in y_str], dtype=torch.long, device=device).unsqueeze(0)\n", + "\n", + "with torch.no_grad():\n", + " logits, _ = model(x_ids) # (1, 8, vocab_size)\n", + "\n", + "probs = torch.softmax(logits, dim=-1) # (1, 8, vocab_size)\n", + "\n", + "print(f\"{'t':>2} {'prefix':>12s} {'true':>5s} logit(true) prob(true) -ln(p)\")\n", + "print(\"-\" * 78)\n", + "\n", + "per_t_loss = []\n", + "for t in range(8):\n", + " ctx = x_str[: t + 1]\n", + " true_ch = y_str[t]\n", + " true_id = stoi[true_ch]\n", + " lg = logits[0, t, true_id].item()\n", + " p = probs[0, t, true_id].item()\n", + " per_t_loss.append(-math.log(p) if p > 0 else float(\"inf\"))\n", + " print(f\"{t:>2} {ctx!r:>14s} {true_ch!r:>5s} {lg:+11.4f} {p:11.6f} {per_t_loss[-1]:7.4f}\")\n", + "\n", + "manual_mean = sum(per_t_loss) / len(per_t_loss)\n", + "pytorch_ce = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n", + "\n", + "print()\n", + "print(f\"mean -ln(p) across 8 positions (hand) = {manual_mean:.6f}\")\n", + "print(f\"F.cross_entropy(logits, y_ids) = {pytorch_ce:.6f}\")\n", + "print(f\"match: {abs(manual_mean - pytorch_ce) < 1e-5}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "8246b96a" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### How the 8 per-position losses combine into one training signal\n", + "\n", + "The 8 per-position penalties are **averaged** into a single scalar:\n", + "\n", + "```\n", + "loss = ( loss(\"B\" → 'e') + loss(\"Be\" → 'f') + loss(\"Bef\" → 'o') + ... + loss(\"Before w\" → 'e') ) / 8\n", + "```\n", + "\n", + "Then `loss.backward()` walks *in reverse* from this one scalar through the\n", + "computation graph. Every weight in the embedding table receives a gradient that\n", + "says: *\"nudge yourself slightly so the true character at every one of these 8\n", + "positions gets a little more probability mass.\"* `optimizer.step()` then applies\n", + "that nudge.\n", + "\n", + "**A useful mental picture:** think of the 8 columns of the row as 8 separate\n", + "flashcards sharing the same model. Each flashcard says \"given this prefix, score\n", + "the correct next char highly\". The model sees all 8 simultaneously, and the loss\n", + "is the average of how it did on each. The gradient nudges the shared weights in\n", + "the direction that helps *all 8* — not just one of them.\n" + ], + "id": "3fdb163c" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What logits actually look like across all 8 positions — and why they are NOT random\n", + "\n", + "The notebook shows logits at `t=1` only. Here we walk through what logits look like at **every** position `t=0` through `t=7` on the same row `(x='Before w', y='efore we')`, and explain why each one follows a recognisable pattern.\n", + "\n", + "---\n", + "\n", + "**Important reminder on how the bigram model works.**\n", + "The bigram model has a single learnable weight table of shape `(vocab_size, vocab_size)`. For each input character id `idx`, the model looks up the corresponding row of that table — that row **is** the `vocab_size` logits for the next character.\n", + "\n", + "```\n", + "logits at position t = token_embedding_table[ x[t] ]\n", + " ^^^ the row for the character at position t\n", + "```\n", + "\n", + "This means: **logits depend only on the single character at position `t`**, not on the whole prefix. That is the *bigram assumption*. (A transformer later removes this assumption — it lets logits depend on the full prefix.)\n", + "\n", + "---\n", + "\n", + "**At every position, the model emits 65 numbers. Here is what they look like and why.**\n", + "\n", + "Below we write logits for a handful of the most interesting characters at each of the 8 positions. These are **illustrative numbers** in the style of what an untrained or barely-trained bigram model would produce — you can get exact numbers by running the verification cell below, but the *shape and pattern* is what matters.\n", + "\n", + "---\n", + "\n", + "**t=0. Context seen by model: `'B'`. Must predict `'e'`.**\n", + "\n", + "The model has seen only the character `'B'`. It looks up one row of the embedding table — the row for `'B'` — and that row becomes its 65 logits. In English text, `'B'` is most commonly followed by `'e'`, `'a'`, `'o'`, `'u'`, `'y'`, or `'r'` (think *But, By, Be, Before, Boy, Bring*). After training, those characters should have the highest logits in this row.\n", + "\n", + "```\n", + "logits at t=0 (row for 'B' in the weight table):\n", + " 'e' logit = +1.8 (TRUE answer y[0] = 'e')\n", + " 'a' logit = +1.2\n", + " 'o' logit = +0.9\n", + " 'u' logit = +0.6\n", + " 'r' logit = +0.4\n", + " 'i' logit = +0.2\n", + " 'B' logit = -0.3 ('B' followed by another 'B' is rare)\n", + " ' ' logit = -0.8 ('B' followed by space is rare — 'B' is capital, usually starts a word)\n", + " 'z' logit = -1.7 ('Bz' essentially never occurs)\n", + " ... (the other 57 characters, mostly negative or near-zero)\n", + "\n", + "softmax at t=0:\n", + " P('e') ≈ 0.28 correct next character gets ~28% of the budget\n", + " P('a') ≈ 0.15\n", + " P('o') ≈ 0.11\n", + " ... the remaining 62 chars share the rest\n", + "\n", + "loss at t=0 = -ln(0.28) ≈ 1.27\n", + "```\n", + "\n", + "Why this pattern? Because `token_embedding_table['B']` has been nudged up and down during training: every time `'B'` appears in the training text followed by `'e'`, the weight for `'B'→'e'` gets a tiny positive gradient. Every time `'B'` is followed by something *other* than `'e'`, the weight for `'B'→'e'` gets a tiny negative gradient (or rather, the correct-next-char weight is pushed up and *all* the wrong ones are pushed down a little by softmax's normalising effect). Over thousands of examples the row for `'B'` reflects the true *next-letter frequency* of `'B'` in Shakespeare.\n", + "\n", + "---\n", + "\n", + "**t=1. Context seen by model: `'Be'`. Must predict `'f'`.**\n", + "\n", + "The bigram model sees *only* `'e'` (the character at index 1), not the full `'Be'` prefix. It looks up the row for `'e'`. In English, `'e'` is often followed by `' '` (space, ending a word), `'r'`, `'s'`, `'t'`, `'d'`, or vowels. In the word `Before`, however, `'e'` is followed by `'f'` — which is not the most common successor of `'e'` in general.\n", + "\n", + "```\n", + "logits at t=1 (row for 'e'):\n", + " ' ' logit = +2.4 (most common — 'e' often ends a word)\n", + " 'r' logit = +1.7\n", + " 's' logit = +1.3\n", + " 'a' logit = +0.9\n", + " 't' logit = +0.7\n", + " 'd' logit = +0.5\n", + " 'f' logit = +0.4 (TRUE answer y[1] = 'f')\n", + " 'n' logit = +0.3\n", + " 'c' logit = +0.1\n", + " ... (most others negative)\n", + "\n", + "softmax at t=1:\n", + " P(' ') ≈ 0.34 model strongly guesses a space here\n", + " P('r') ≈ 0.16\n", + " P('f') ≈ 0.09 only 9% on the true character\n", + "\n", + "loss at t=1 = -ln(0.09) ≈ 2.41\n", + "```\n", + "\n", + "Loss is *higher* here than at t=0 because the model's strongest guess (`' '` or `'r'`) is wrong. The bigram model cannot use the `'B'` before `'e'` to help — it only knows `'e'` was just seen, and `'e'→'f'` is rare in general. This is exactly why bigram models are weak: a bigram loses all context beyond one step back.\n", + "\n", + "---\n", + "\n", + "**t=2. Context seen by model: `'Bef'`. Must predict `'o'`.**\n", + "\n", + "Row `'f'` of the embedding table. In English, `'f'` is followed overwhelmingly by `'o'` (*for, from, of, often, before*), `'r'` (*from, free*), `'e'`, or `'t'` (*first, after*).\n", + "\n", + "```\n", + "logits at t=2 (row for 'f'):\n", + " 'o' logit = +2.1 (TRUE answer y[2] = 'o')\n", + " 'r' logit = +1.4\n", + " 't' logit = +0.8\n", + " 'e' logit = +0.6\n", + " 'f' logit = -0.2 ('ff' is uncommon at word-internal positions in English)\n", + " ...\n", + "\n", + "softmax at t=2:\n", + " P('o') ≈ 0.31 correct next character gets ~31%\n", + "\n", + "loss at t=2 = -ln(0.31) ≈ 1.17\n", + "```\n", + "\n", + "This is a relatively **low loss position** — `'f'→'o'` is genuinely common and the model has learned it.\n", + "\n", + "---\n", + "\n", + "**t=3. Context seen by model: `'Befo'`. Must predict `'r'`.**\n", + "\n", + "Row `'o'` of the embedding table. `'o'` has many common successors: `'u'`, `'r'`, `'n'`, `'t'`, `'f'`, `'m'`, `'s'`, and space.\n", + "\n", + "```\n", + "logits at t=3 (row for 'o'):\n", + " 'u' logit = +1.8 ('you', 'would', 'could', 'out', 'over')\n", + " 'r' logit = +1.5 (TRUE answer y[3] = 'r')\n", + " 'n' logit = +1.3\n", + " 't' logit = +1.1\n", + " 'f' logit = +0.7\n", + " ...\n", + " P('r') ≈ 0.18 (good but not dominant — 'o' has many common follow-ups)\n", + "\n", + "loss at t=3 = -ln(0.18) ≈ 1.71\n", + "```\n", + "\n", + "---\n", + "\n", + "**t=4. Context seen by model: `'Befor'`. Must predict `'e'`.**\n", + "\n", + "Row `'r'` of the embedding table. `'r'` is often followed by `'e'` (*there, were, are, before, more*), `'s'`, `'t'`, `'d'`, or space.\n", + "\n", + "```\n", + "logits at t=4 (row for 'r'):\n", + " 'e' logit = +2.2 (TRUE answer y[4] = 'e')\n", + " 's' logit = +1.3\n", + " 't' logit = +0.9\n", + " ' ' logit = +0.8 ('r' often ends a word: 'or', 'for', 'her', 'our')\n", + " ...\n", + " P('e') ≈ 0.36 strong — 'r'→'e' is very common\n", + "\n", + "loss at t=4 = -ln(0.36) ≈ 1.02\n", + "```\n", + "\n", + "---\n", + "\n", + "**t=5. Context seen by model: `'Before'`. Must predict `' '` (space).**\n", + "\n", + "Row `'e'` of the embedding table, again — same row as at `t=1`. Logits at t=5 and logits at t=1 are **identical** in a bigram model. This is an important point.\n", + "\n", + "```\n", + "logits at t=5 (row for 'e') ← identical numbers to t=1 (same row)\n", + " ' ' logit = +2.4 (TRUE answer y[5] = ' ')\n", + " 'r' logit = +1.7\n", + " 's' logit = +1.3\n", + " ...\n", + " P(' ') ≈ 0.34 correct!\n", + "\n", + "loss at t=5 = -ln(0.34) ≈ 1.08\n", + "```\n", + "\n", + "Notice: at t=1 the model *missed* (`'e'→'f'`), but at t=5 the model *hit* (`'e'→' '`). Both come from the **same row** `token_embedding_table['e']`. One row is asked to predict many different successors in different contexts, and its gradient is the average over all of them.\n", + "\n", + "---\n", + "\n", + "**t=6. Context seen by model: `'Before '`. Must predict `'w'`.**\n", + "\n", + "Row for `' '` (space). This is the single most important row in the whole vocabulary — every word after the first starts with a space, so `' '→?` is essentially asking: *what character starts a new word?* In English, the answer is dominated by the frequent word-starting letters: `'t'` (*the, to, that*), `'o'` (*of, or*), `'a'` (*and, are, at*), `'i'` (*in, is, it*), `'w'` (*we, what, would, will, with*), `'h'` (*he, his, her*), `'b'` (*be, but, by*), `'s'` (*so, shall, she*), and capital letters for sentence starts.\n", + "\n", + "```\n", + "logits at t=6 (row for ' '):\n", + " 't' logit = +2.3\n", + " 'a' logit = +1.8\n", + " 'o' logit = +1.6\n", + " 'i' logit = +1.4\n", + " 'w' logit = +1.2 (TRUE answer y[6] = 'w')\n", + " 'h' logit = +1.0\n", + " 'b' logit = +0.9\n", + " 's' logit = +0.7\n", + " ... (rare letters are negative)\n", + " 'z' logit = -1.8\n", + " 'q' logit = -2.0\n", + " ...\n", + " P('w') ≈ 0.11\n", + "\n", + "loss at t=6 = -ln(0.11) ≈ 2.21\n", + "```\n", + "\n", + "---\n", + "\n", + "**t=7. Context seen by model: `'Before w'`. Must predict `'e'`.**\n", + "\n", + "The bigram model sees only `'w'` (the character at index 7), not the full `'Before w'` prefix. It looks up the row for `'w'` in the embedding table. In English, `'w'` is overwhelmingly followed by `'e'` (*we, were, well, when, what, would, will, with*), `'h'` (*where, why, when, whole*), `'i'` (*will, with, which*), `'o'` (*would, word*), `'a'` (*was, want*).\n", + "\n", + "```\n", + "logits at t=7 (row for 'w'):\n", + " 'e' logit = +2.5 (TRUE answer y[7] = 'e')\n", + " 'h' logit = +1.8\n", + " 'i' logit = +1.5\n", + " 'o' logit = +1.2\n", + " 'a' logit = +1.0\n", + " 'w' logit = -0.1 ('ww' does not occur in English)\n", + " ... (the other 59 characters, mostly negative or near-zero)\n", + "\n", + "softmax at t=7:\n", + " P('e') ≈ 0.40 correct next character gets ~40% of the budget\n", + " P('h') ≈ 0.20\n", + " P('i') ≈ 0.15\n", + " ... the remaining 62 chars share the rest\n", + "\n", + "loss at t=7 = -ln(0.40) ≈ 0.92\n", + "```\n", + "\n", + "This is the model's strongest position on the row — `'w'→'e'` is very common in Shakespeare.\n", + "\n", + "---\n", + "\n", + "**Pulling it together: the average loss over all 8 positions.**\n", + "\n", + "Here is what the model does on this one row, end to end:\n", + "\n", + "**Part A — what you see vs what the bigram uses**\n", + "\n", + "| t | prefix (you read) | target | row used |\n", + "| --- | --- | --- | --- |\n", + "| 0 | B | e | B |\n", + "| 1 | Be | f | e |\n", + "| 2 | Bef | o | f |\n", + "| 3 | Befo | r | o |\n", + "| 4 | Befor | e | r |\n", + "| 5 | Before | space | e |\n", + "| 6 | Before + space | w | space |\n", + "| 7 | Before w | e | w |\n", + "\n", + "The **row used** column is the only input the bigram model cares about — always the single character x[t], not the full prefix.\n", + "\n", + "**Part B — scores and loss at each step**\n", + "\n", + "| t | logit (true char) | P(true) | loss |\n", + "| --- | --- | --- | --- |\n", + "| 0 | +1.8 | 0.28 | 1.27 |\n", + "| 1 | +0.4 | 0.09 | 2.41 |\n", + "| 2 | +2.1 | 0.31 | 1.17 |\n", + "| 3 | +1.5 | 0.18 | 1.71 |\n", + "| 4 | +2.2 | 0.36 | 1.02 |\n", + "| 5 | +2.4 | 0.34 | 1.08 |\n", + "| 6 | +1.2 | 0.11 | 2.21 |\n", + "| 7 | +2.5 | 0.40 | 0.92 |\n", + "\n", + "Averaging the 8 losses: about `(1.27 + 2.41 + 1.17 + 1.71 + 1.02 + 1.08 + 2.21 + 0.92) / 8 ≈ 1.47`. This is one `loss` number, and it drives one `backward()` pass — nudging every row of the embedding table that was looked up (the rows for `'B'`, `'e'`, `'f'`, `'o'`, `'r'`, `' '`, `'w'`) to do slightly better next time.\n", + "\n", + "---\n", + "\n", + "**Are logits random? No — they are *determined* by the weights.**\n", + "\n", + "Logits are **never random**. They are a pure function of two things:\n", + "\n", + "1. **The current input character(s).** `logits at position t = some_function(x[t])`. For the bigram model, `some_function` is just a table lookup: `token_embedding_table[x[t]]`. For a transformer, `some_function` is a neural network that reads the entire prefix `x[0..t]`. Either way, given the same input and the same weights, the same logits always come out.\n", + "\n", + "2. **The model weights.** The weights are what change during training. At the very first forward pass (before any training), the weights are initialised to small random-ish values — logits at that point *look* noisy and nearly uniform. After even a few thousand training steps, the weights have absorbed bigram statistics from the training text: `'f'→'o'` gets a positive weight, `'q'→'x'` gets a strongly negative weight, and so on. The logits become structured.\n", + "\n", + "**The pattern inside logits is therefore exactly the bigram statistics of Shakespeare.** Rows for common letters like `'e'` or `' '` have several high-positive logits (many common successors). Rows for rare letters like `'z'` or `'q'` have most logits near or below zero — almost anything after `'z'` is unusual. Rows for `'q'` have one standout positive logit at `'u'`, because in English `'q'` is almost always followed by `'u'`.\n", + "\n", + "**Take-home:** \"logits at position `t`\" = 65 numbers, one per character in the vocabulary. Positive numbers = the model thinks that character is a plausible next step. Negative = implausible. `softmax` turns them into probabilities. Cross-entropy asks: *what probability did the true character get?* Training nudges the rows so this probability gets higher." + ], + "id": "logits-deep-dive" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"full chunk:\", repr(decode(x.tolist())))\n", + "print()\n", + "for t in range(block_size):\n", + " prefix = decode(x[: t + 1].tolist())\n", + " target = itos[y[t].item()]\n", + " print(f\"t={t} prefix {prefix!r:12s} -> {target!r}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "e4559d0f" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 4.5** — At `t=0`, how many characters does the model see? At `t=7`?\n", + "*(0: one char; 7: all eight.)*\n" + ], + "id": "3be6fbb3" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for t in [0, 3, 7]:\n", + " prefix_len = t + 1\n", + " print(f\"t={t}: prefix length = {prefix_len}, target = {repr(itos[y[t].item()])}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "898927dc" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "for t in range(block_size):\n", + " on_tape = train_data[start + t].item()\n", + " on_tape_next = train_data[start + t + 1].item()\n", + " match = x[t].item() == on_tape and y[t].item() == on_tape_next\n", + " print(f\"t={t} x={repr(itos[x[t].item()])} y={repr(itos[y[t].item()])} matches tape: {match}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "876ad4ce" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "x = train_data[1000 : 1000 + block_size]\n", + "y = train_data[1001 : 1001 + block_size]\n", + "\n", + "print(\"x shape:\", x.shape, \" y shape:\", y.shape)\n", + "print(\"T (time steps) = block_size =\", block_size)\n", + "print()\n", + "# pretend we had batch_size=3 — same T, three independent rows\n", + "B = 3\n", + "print(f\"batched inputs would be shape ({B}, {block_size}) = (B, T)\")\n", + "print(f\"model logits later: ({B}, {block_size}, {vocab_size}) = (B, T, C)\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "8bb27b4f" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.8 Put it together — build your own `x` and `y`\n", + "\n", + "Pick any valid `start` index, build `x` and `y`, decode them, and verify every\n", + "target. This is exactly what `get_batch` does — but with random `start` values.\n", + "\n", + "**→ Training:** Same logic as `get_batch` — you're manually doing what the training loop automates thousands of times.\n" + ], + "id": "f86da52d" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "start = 5000 # change me\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"start:\", start)\n", + "print(\"x:\", decode(x.tolist()))\n", + "print(\"y:\", decode(y.tolist()))\n", + "checks = [y[t].item() == train_data[start + t + 1].item() for t in range(block_size)]\n", + "print(\"all correct:\", all(checks))\n" + ], + "execution_count": null, + "outputs": [], + "id": "c2ee21e5" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.2 Stack two chunks by hand\n", + "\n", + "Before `get_batch`, see how two slices become a 2×8 matrix.\n", + "\n", + "**→ Training:** `get_batch` does this with `batch_size` rows. One `optimizer.step()` updates weights using loss from **all** rows.\n" + ], + "id": "0b02e589" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "torch.manual_seed(0)\n", + "ix = torch.randint(len(train_data) - block_size, (2,))\n", + "chunk_a = train_data[ix[0] : ix[0] + block_size]\n", + "chunk_b = train_data[ix[1] : ix[1] + block_size]\n", + "stacked = torch.stack([chunk_a, chunk_b])\n", + "print(\"stacked shape:\", stacked.shape, \" (= 2 rows, block_size cols)\")\n", + "for row in range(2):\n", + " print(f\" row {row}:\", decode(stacked[row].tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "11fc074b" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.3 `get_batch` — random windows\n", + "\n", + "`xb` = inputs, `yb` = targets (each row shifted +1 on the tape). Same shapes.\n", + "\n", + "**→ Training:** Called every training iteration. Returns the `(xb, yb)` pair that `loss.backward()` will score and correct.\n", + "\n", + "**In the chain:** `get_batch` is **step 0** of every training iteration — supplies `xb` (questions) and `yb` (answers) before `model()` can run.\n" + ], + "id": "c296b808" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "torch.manual_seed(1337)\n", + "batch_size = 4\n", + "\n", + "def get_batch(split):\n", + " d = train_data if split == \"train\" else val_data\n", + " ix = torch.randint(len(d) - block_size, (batch_size,))\n", + " x = torch.stack([d[i : i + block_size] for i in ix])\n", + " y = torch.stack([d[i + 1 : i + block_size + 1] for i in ix])\n", + " return x.to(device), y.to(device)\n", + "\n", + "xb, yb = get_batch(\"train\")\n", + "print(\"xb shape:\", xb.shape, \" yb shape:\", yb.shape)\n", + "print(xb)\n" + ], + "execution_count": null, + "outputs": [], + "id": "67c1ce6f" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.4 Decode a batch row\n", + "\n", + "Row `b`, column `t`: `yb[b,t]` is the next token after `xb[b,t]` on the tape.\n", + "\n", + "**→ Training:** Sanity check that `yb` is the answer key aligned with `xb` before trusting the loss number.\n" + ], + "id": "e72033cb" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "b = 0\n", + "print(\"row\", b, \"input: \", decode(xb[b].tolist()))\n", + "print(\"row\", b, \"target:\", decode(yb[b].tolist()))\n", + "for t in range(block_size):\n", + " print(f\" t={t} x={repr(itos[xb[b,t].item()])} y={repr(itos[yb[b,t].item()])}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "d8be50cf" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 5** — Call `get_batch(\"val\")` and decode row 2.\n" + ], + "id": "3f69fc3b" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "xb_val, yb_val = get_batch(\"val\")\n", + "print(decode(xb_val[2].tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "cd1c51dc" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.1 Logits — raw scores\n", + "\n", + "For each possible next token, the model outputs a **logit** (unnormalised score).\n", + "Higher = more likely. Length = `vocab_size`.\n", + "\n", + "**→ Training:** Logits are not trained directly — **weights** produce logits. Gradients flow back through logits into those weights.\n" + ], + "id": "f6bb711d" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.1b Logits deep-dive — what does a model actually output?\n", + "\n", + "Above we said: **logits are raw, unnormalised scores.** Let's make that\n", + "completely concrete with the text *\"Before we proceed any further, hear me speak.\"*\n", + "We pass one 8-character window through the Bigram model, print the real numbers, and\n", + "show exactly how they become probabilities.\n", + "\n", + "**Intuition: the dice analogy.** Imagine a weighted 6-sided dice. For each possible\n", + "outcome, you express your gut confidence as a raw number (positive = confident this\n", + "will happen, negative = sceptical). Those numbers are *logits*. Then `softmax` squashes\n", + "them into a proper probability distribution (all positive, sums to 1).\n", + "\n", + "| In the dice analogy | In the Bigram model |\n", + "|---|---|\n", + "| 6 sides of the dice | 65 characters in `vocab_size` |\n", + "| Which side will land? | Which character comes next? |\n", + "| Your raw scores (-2, -1, 0, 3, 1, 0) | `logits[b, t, :]` — 65 numbers at each position |\n", + "| softmax → probabilities | `F.softmax(logits, dim=-1)` → 65 probabilities summing to 1 |\n", + "| The side that actually rolled | `yb[b, t]` — the one correct char |\n", + "\n", + "**Why not output probabilities directly?** A neural network's layers naturally produce\n", + "numbers that can be anywhere from -infinity to +infinity. Constraining them to [0, 1] and\n", + "summing to 1 is a *post-processing* step — softmax. Training is also numerically\n", + "more stable when loss is computed on logits (which PyTorch's `F.cross_entropy` does\n", + "for us automatically).\n", + "\n", + "**-> Training:** Only the *relative order* of logits matters for which class wins.\n", + "The absolute scale matters for *how confident* the model thinks it is — and that\n", + "scale is what `backward()` adjusts.\n" + ], + "id": "06417850" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Run the Bigram model on a real 8-character window and see its raw output.\n", + "# We take a short sentence, slice one (x, y) pair, and ask: what logits\n", + "# does the *current* model produce at each position?\n", + "\n", + "sentence = \"Before we proceed any further, hear me speak.\"\n", + "\n", + "import math # for -ln(p) later\n", + "\n", + "# convert to tensor ids (reuse stoi from earlier cells)\n", + "sentence_ids = torch.tensor([stoi[c] for c in sentence], dtype=torch.long, device=device)\n", + "print(\"first 8 chars of sentence:\", repr(sentence[:8]))\n", + "print(\" as ids:\", sentence_ids[:8].tolist())\n", + "\n", + "# run one forward pass *without* targets (generation path — no loss, just logits)\n", + "with torch.no_grad():\n", + " idx = sentence_ids[:8].unsqueeze(0) # shape (1, 8) = one row\n", + " logits, _ = model(idx) # shape (1, 8, vocab_size)\n", + "\n", + "print()\n", + "print(\"logits shape =\", tuple(logits.shape), \" -> (batch=1, time=8, vocab=\" + str(vocab_size) + \")\")\n", + "print(\" That's 8 x\", vocab_size, \"=\", 8 * vocab_size, \"raw scores produced by one small window.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "996e68b2" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Show the logits for ONE position in detail: position t=0 (model saw only 'B').\n", + "# We print the TOP-5 highest-scoring characters and the TRUE next character's score.\n", + "\n", + "with torch.no_grad():\n", + " idx = sentence_ids[:8].unsqueeze(0)\n", + " logits, _ = model(idx)\n", + "\n", + "def report(logits_row, position, context_str, true_next_char):\n", + " \"\"\"Pretty-print the top-k logits + probability for one position.\"\"\"\n", + " scores = logits_row[position, :] # shape (vocab_size,)\n", + " probs = torch.softmax(scores, dim=0)\n", + " top5_vals, top5_ids = torch.topk(scores, k=5)\n", + " print(\"t=\" + str(position) + \" context =\", repr(context_str), \"-> model top-5 guesses for next char:\")\n", + " for v, i in zip(top5_vals, top5_ids):\n", + " ch = itos[i.item()]\n", + " p = probs[i.item()].item()\n", + " print(\" \" + repr(ch) + \" logit=\" + (\"%+7.4f\" % v.item()) + \" prob=\" + (\"%8.4f\" % p))\n", + " true_id = stoi[true_next_char]\n", + " print(\" TRUE next char =\" + repr(true_next_char) + \" logit=\" + (\"%+7.4f\" % scores[true_id].item()) + \" prob=\" + (\"%8.4f\" % probs[true_id].item()))\n", + " print()\n", + "\n", + "for t, context_len in [(0, 1), (3, 4), (7, 8)]:\n", + " ctx_str = \"\".join(itos[sentence_ids[i].item()] for i in range(context_len))\n", + " true_next = itos[sentence_ids[context_len].item()]\n", + " report(logits[0], t, ctx_str, true_next)\n", + "\n", + "print(\"Observations (before training — weights are still random):\")\n", + "print(\" 1. The TRUE next char usually gets a tiny probability (near 1/vocab_size).\")\n", + "print(\" 2. Top-5 guesses are nonsense — the table has not learned anything yet.\")\n", + "print(\" 3. All logits are close to 0; softmax of near-zero inputs is near-uniform.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "d4f0a86d" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Why \"near-uniform\" before training?\n", + "\n", + "`nn.Embedding` initialises every entry from a normal distribution with mean ~ 0 and\n", + "std ~ 1. When every one of 65 logits is a small random number, softmax produces a\n", + "distribution where every character has probability roughly `1/65 ~ 1.5%`. That's\n", + "why the initial loss is always near `-ln(1/vocab_size) ~ 4.17` — a *sanity check*.\n", + "\n", + "As training runs, the table rows for common pairs like `t->h`, `h->e`, `e-> `\n", + "(space) get their logits pushed *up* (more positive), while rare or impossible\n", + "pairs like `t->q` get their logits pushed *down*. That change — per weight, by a\n", + "tiny amount per step — *is* learning.\n" + ], + "id": "a3a7db40" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "toy_logits = torch.tensor([2.0, 0.5, 0.1]) # pretend vocab_size=3\n", + "print(\"logits:\", toy_logits.tolist())\n", + "print(\"argmax (favourite class):\", toy_logits.argmax().item())\n" + ], + "execution_count": null, + "outputs": [], + "id": "e7428034" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.2 Softmax — scores → probabilities\n", + "\n", + "Softmax squashes logits into positive numbers that **sum to 1** — a distribution.\n", + "\n", + "**→ Training:** Used heavily in **generation** (sampling). Training uses cross-entropy on logits directly (softmax is inside that).\n" + ], + "id": "ff289741" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "probs = F.softmax(toy_logits, dim=0)\n", + "print(\"probs:\", [round(p, 3) for p in probs.tolist()])\n", + "print(\"sum:\", probs.sum().item())\n" + ], + "execution_count": null, + "outputs": [], + "id": "71e447c8" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.3 Cross-entropy loss — how wrong were we?\n", + "\n", + "Compare model distribution to the **true** next token id.\n", + "- right answer gets high prob → low loss\n", + "- wrong answer → high loss\n", + "\n", + "Random guessing over `vocab_size` tokens → loss ≈ `-ln(1/vocab_size)`.\n", + "\n", + "**→ Training:** The single number `loss` that `backward()` uses. Lower loss ⇒ nudge weights so true next tokens get higher score.\n", + "\n", + "**In the chain:** cross-entropy turns logits + `yb` into the **single number** `loss`. Section 8 cannot run `backward()` until this exists.\n" + ], + "id": "e60d5dd1" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.3b Cross-entropy deep-dive — what are we actually punishing?\n", + "\n", + "**Shortest summary:** `cross_entropy(logits, true_index)` = a single number saying\n", + "how much probability the model *wasted* on wrong answers, averaged over every\n", + "prediction in the batch.\n", + "\n", + "**Analogy: the weather forecaster.** Every morning a forecaster gives percentages\n", + "for sunny / rain / snow. The next day we see what *actually* happened and score them.\n", + "\n", + "| Day | Forecast (predicted probs) | Reality | Prob assigned to TRUE | `-ln(p)` = penalty |\n", + "|-----|----------------------------|---------|------------------------|-------------------|\n", + "| Mon | sunny 80%, rain 15%, snow 5% | sunny | 0.80 | 0.22 (small — confident & right) |\n", + "| Tue | sunny 30%, rain 60%, snow 10% | rain | 0.60 | 0.51 |\n", + "| Wed | sunny 20%, rain 30%, snow 50% | snow | 0.50 | 0.69 |\n", + "| Thu | sunny 95%, rain 4%, snow 1% | snow | 0.01 | 4.61 (huge — confident & wrong) |\n", + "\n", + "The average of those penalties over a season is exactly **mean cross-entropy**:\n", + "**the lower, the better the forecaster.**\n", + "\n", + "The Bigram model is exactly a 65-outcome forecaster: instead of weather it has\n", + "a-z, spaces, punctuation, newlines as the 65 possible \"next characters\", and it\n", + "must predict \"which character comes next?\" 8 times per row, batch_size rows per batch.\n", + "\n", + "**-> Training:** `loss = F.cross_entropy(logits, yb)` is that single scalar.\n", + "`loss.backward()` then asks, for every weight: *\"if I nudge you up (or down),\n", + "does the penalty get smaller?\"* — and `optimizer.step()` takes that tiny nudge.\n" + ], + "id": "7e1be656" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Compute cross-entropy by hand on one 8-char window, match it to\n", + "# what F.cross_entropy returns, and watch true-char probability climb\n", + "# as training progresses.\n", + "\n", + "with torch.no_grad():\n", + " idx = sentence_ids[:8].unsqueeze(0)\n", + " logits, _ = model(idx) # (1, 8, vocab_size)\n", + "\n", + "y_ids = sentence_ids[1:9].unsqueeze(0) # the 8 true next chars\n", + "\n", + "# --- softmax -> probabilities -> -ln(prob of true char) per position ---\n", + "probs = torch.softmax(logits, dim=-1)\n", + "\n", + "header = \"pos\" + \" \" * 6 + \"ctx\" + \" \" * 11 + \"true\" + \" \" * 7 + \"prob(true)\" + \" \" * 3 + \"-ln(p)\"\n", + "print(header)\n", + "print(\"-\" * 55)\n", + "per_pos_loss = []\n", + "for t in range(8):\n", + " ctx = \"\".join(itos[sentence_ids[i].item()] for i in range(t + 1))\n", + " true_ch = itos[y_ids[0, t].item()]\n", + " p = probs[0, t, y_ids[0, t]].item()\n", + " penalty = -math.log(p) if p > 0 else float(\"inf\")\n", + " per_pos_loss.append(penalty)\n", + " print(\"%3d\" % t + \" \" * 2 + repr(ctx) + \" \" * 5 + repr(true_ch) + \" \" * 5 + (\"%10.6f\" % p) + \" \" * 3 + (\"%8.4f\" % penalty))\n", + "\n", + "manual_loss = sum(per_pos_loss) / len(per_pos_loss)\n", + "pytorch_loss = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n", + "\n", + "print()\n", + "print(\"mean cross-entropy (computed by hand): %.6f\" % manual_loss)\n", + "print(\"mean cross-entropy (F.cross_entropy): %.6f\" % pytorch_loss)\n", + "print(\"match:\", abs(manual_loss - pytorch_loss) < 1e-5)\n", + "print()\n", + "print(\"Notice: every position where the model assigned < 1/vocab to the true\")\n", + "print(\"character contributes a large penalty to the mean. Training shrinks these.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "dda290e1" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Watch a single training step on this same sentence *improve* true-char probability.\n", + "# We run forward -> backward -> step on sentence_ids[:8]/[1:9], then re-print.\n", + "\n", + "def one_step_xent_report(comment):\n", + " with torch.no_grad():\n", + " logits, _ = model(sentence_ids[:8].unsqueeze(0))\n", + " probs = torch.softmax(logits, dim=-1)\n", + " y_ids = sentence_ids[1:9].unsqueeze(0)\n", + " true_probs = [probs[0, t, y_ids[0, t]].item() for t in range(8)]\n", + " L = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n", + " avg_p = sum(true_probs) / len(true_probs)\n", + " print(\" \" + comment.ljust(30) + (\" loss=%.4f\" % L) + (\" avg prob(true char)=%.4f\" % avg_p))\n", + " return L\n", + "\n", + "print(\"=== Nudging the embedding table on this ONE sentence ===\")\n", + "print()\n", + "\n", + "# Slightly bigger LR so the demo shows visible change in a few steps.\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-2)\n", + "one_step_xent_report(\"before 1 step\")\n", + "\n", + "for step in range(20):\n", + " idx = sentence_ids[:8].unsqueeze(0)\n", + " y = sentence_ids[1:9].unsqueeze(0)\n", + " _, loss = model(idx, y)\n", + " optimizer.zero_grad(set_to_none=True)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if (step + 1) % 5 == 0:\n", + " one_step_xent_report(\"after \" + str(step + 1) + \" steps\")\n", + "\n", + "print()\n", + "print(\"Loss dropped on this *specific* window because we deliberately over-trained\")\n", + "print(\"on it. In a real training loop you sample NEW windows each iteration, so the\")\n", + "print(\"model learns general patterns, not just this sentence.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "7744f6f9" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Two rules of thumb for cross-entropy in this notebook\n", + "\n", + "1. **The best a *bigram* model can do is ~2.3.** Seeing only one previous character\n", + "cannot disambiguate `th` from `t ` (space) or `he` from `ha`. Better context\n", + "(self-attention, next notebook) can push loss significantly lower.\n", + "2. **val loss is the honest score.** `train_loss` can be made arbitrarily low by\n", + "training long enough (you're memorising the training set). `val_loss` on unseen\n", + "text is what actually tells you whether the model generalises.\n" + ], + "id": "c0a89db5" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for target in [0, 1, 2]:\n", + " loss = F.cross_entropy(toy_logits.unsqueeze(0), torch.tensor([target]))\n", + " print(f\"true class {target} -> loss {loss.item():.3f}\")\n", + "expected = -torch.log(torch.tensor(1.0 / vocab_size))\n", + "print(f\"random baseline (vocab {vocab_size}): {expected.item():.3f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "c9166861" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 6** — Make class 2 strongest; loss for target 2 should drop.\n" + ], + "id": "7b45f3a7" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "strong = torch.tensor([0.1, 0.5, 2.5])\n", + "for target in [0, 2]:\n", + " loss = F.cross_entropy(strong.unsqueeze(0), torch.tensor([target]))\n", + " print(f\"target {target} -> {loss.item():.3f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "dfe98faf" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.4 Bigram = one lookup table\n", + "\n", + "`nn.Embedding(vocab_size, vocab_size)` stores a matrix: **row i** = logits for\n", + "\"what follows token i?\" No memory of earlier tokens.\n", + "\n", + "**Analogy:** a cheat sheet — \"after `t` usually comes `h` or `e`\" — but only\n", + "looking at the **last** letter.\n", + "\n", + "**→ Training:** The entire learned model is this **one matrix** (~vocab² numbers). Training adjusts every row: \"what follows char i?\"\n" + ], + "id": "7926ed10" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Is the 65×65 table fixed or updated during training?\n", + "\n", + "**The shape is fixed; the numbers inside are constantly updated.**\n", + "\n", + "- **Fixed:** always 65 rows × 65 columns (4,225 numbers) — one row per character, one column per possible next character.\n", + "- **Not fixed:** every number is a **learned weight** that changes during training.\n", + "\n", + "**Before training:** all values start as small random guesses. The model does not yet know that `'q'` is usually followed by `'u'`, or that `'f'` is often followed by `'o'`.\n", + "\n", + "**Each training step:**\n", + "\n", + "1. **Forward** — look up rows, get logits, compute loss.\n", + "2. **Backward** — `loss.backward()` asks each cell: \"how should I change to reduce loss?\"\n", + "3. **Update** — `optimizer.step()` nudges those numbers a tiny bit.\n", + "\n", + "Cells that appear in the current batch get updated; over thousands of steps the whole table absorbs Shakespeare's bigram statistics.\n", + "\n", + "| Stage | What happens to the table |\n", + "| --- | --- |\n", + "| Start | Random values |\n", + "| Each batch | Used cells get small updates |\n", + "| After many steps | Values reflect real patterns (e.g. `'w'→'e'` scores rise) |\n", + "| After training | Table is frozen unless you train again |\n", + "\n", + "**Concrete example:** row `'B'`, column `'e'` might start near +0.02 (meaningless). After seeing many `B…e` pairs in training it might become +1.8 — the model learned that `'B'` often leads to `'e'`.\n", + "\n", + "**Mental picture:** the table is a cheat sheet you fill in while studying. Day 1: blank guesses. Each practice question: adjust a few entries. After thousands of questions: the cheat sheet matches how Shakespeare actually writes.\n", + "\n", + "**Take-home:** the 65×65 **grid size** never changes. The **values** are what training learns — that is exactly what `loss.backward()` and `optimizer.step()` do." + ], + "id": "30a6b989" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "table = nn.Embedding(vocab_size, vocab_size)\n", + "ch = 't'\n", + "tid = stoi[ch]\n", + "row = table.weight[tid]\n", + "print(\"row for\", repr(ch), \"has shape:\", row.shape)\n", + "top_p, top_i = F.softmax(row, dim=0).topk(3)\n", + "print(\"top-3 followers (random weights):\")\n", + "for p, i in zip(top_p, top_i):\n", + " print(f\" {repr(itos[i.item()])}: {p.item():.3f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "f705fa50" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.5 The full `BigramLanguageModel`\n", + "\n", + "Forward: `idx` `(B,T)` → logits `(B,T,C)`. Flatten to `(B*T, C)` for one big\n", + "cross-entropy over every position in the batch.\n", + "\n", + "**→ Training:** `forward(xb, yb)` = predict on all positions, compare to `yb`, return scalar loss. `generate()` uses same weights without `yb`.\n", + "\n", + "**In the chain:** `forward(xb, yb)` is **step 1** of training. Without `yb`, you get logits only (generation path) — no loss, no backward.\n" + ], + "id": "e5208051" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "class BigramLanguageModel(nn.Module):\n", + " def __init__(self, vocab_size):\n", + " super().__init__()\n", + " self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n", + "\n", + " def forward(self, idx, targets=None):\n", + " logits = self.token_embedding_table(idx)\n", + " if targets is None:\n", + " return logits, None\n", + " B, T, C = logits.shape\n", + " loss = F.cross_entropy(logits.view(B * T, C), targets.view(B * T))\n", + " return logits, loss\n", + "\n", + " def generate(self, idx, max_new_tokens):\n", + " for _ in range(max_new_tokens):\n", + " logits, _ = self(idx)\n", + " logits = logits[:, -1, :]\n", + " probs = F.softmax(logits, dim=-1)\n", + " idx_next = torch.multinomial(probs, num_samples=1)\n", + " idx = torch.cat((idx, idx_next), dim=1)\n", + " return idx\n", + "\n", + "model = BigramLanguageModel(vocab_size).to(device)\n", + "xb, yb = get_batch(\"train\")\n", + "logits, loss = model(xb, yb)\n", + "print(\"logits shape (B, T, C):\", logits.shape)\n", + "print(f\"initial loss: {loss.item():.4f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "f7cc9f29" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### At the end — is it just the table, or something more?\n", + "\n", + "For the **bigram model in this notebook**, yes: after training you basically **just have the table**. That table **is** the entire learned model.\n", + "\n", + "There is no second brain, no hidden layers, no memory beyond one letter:\n", + "\n", + "```\n", + "current letter → look up its row → 65 scores → pick next letter\n", + "```\n", + "\n", + "**What is learned vs what is fixed rules**\n", + "\n", + "| Part | Learned? | Role |\n", + "| --- | --- | --- |\n", + "| 65×65 table | Yes — training only updates this | Stores \"after X, how likely is each next letter?\" |\n", + "| Table lookup | No — fixed rule | Given w, read row for w |\n", + "| Softmax | No — fixed math | Turn 65 scores into probabilities summing to 1 |\n", + "| Sampling (multinomial) | No — fixed rule | Roll the dice using those probabilities |\n", + "| Generation loop | No — fixed code | Repeat: predict one letter, append, predict again |\n", + "\n", + "- **Training** teaches only the **table**.\n", + "- **Softmax, sampling, and the loop** are small fixed steps that **use** the table to write text.\n", + "\n", + "**Does it always pick the top letter?** No. The table does not say \"the next letter **is** e.\" It says \"e has 40% chance, h has 20%, ...\" Then sampling **randomly** picks one letter from those chances — that is why generated text varies each run. Always picking the highest score would sound repetitive and robotic.\n", + "\n", + "**Take-home:** end state = one trained cheat sheet (the table). To generate: lookup → softmax → random pick → repeat. Later GPT models add real extra machinery (attention, many layers). The bigram model is intentionally the simplest case: **table only**." + ], + "id": "531060cd" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 6b** — Initial loss should be near `-ln(1/vocab_size)`. Compare.\n" + ], + "id": "1614313d" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n", + "print(f\"loss {loss.item():.4f} vs random baseline {baseline:.4f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "25b7140e" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 7.1 One generation step by hand\n", + "\n", + "Context → forward → logits at **last** position → softmax → sample one id.\n", + "\n", + "**→ Training:** Same forward path as training, but no `yb` and no `backward()`. Shows what one row of trained weights **does** at inference.\n" + ], + "id": "f51c9655" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device) # start: newline\n", + "logits, _ = model(context)\n", + "last_logits = logits[0, -1, :]\n", + "probs = F.softmax(last_logits, dim=0)\n", + "sampled = torch.multinomial(probs, num_samples=1)\n", + "print(\"context:\", repr(itos[context[0,0].item()]))\n", + "print(\"sampled next char:\", repr(itos[sampled.item()]))\n" + ], + "execution_count": null, + "outputs": [], + "id": "f9cdda7f" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 7.2 Full generation (200 chars, untrained)\n", + "\n", + "Start from newline (id 0). Expect nonsense.\n", + "\n", + "**→ Training:** Save mentally (or in output) for contrast with section 9 after weights have been updated.\n" + ], + "id": "f8461a58" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", + "out = model.generate(context, max_new_tokens=200)\n", + "print(decode(out[0].tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "4b5c3e45" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# THE trainable artifact: model parameters (weight matrices)\n", + "n_params = sum(p.numel() for p in model.parameters())\n", + "print(\"trainable parameters:\", n_params)\n", + "print(\"embedding weight shape:\", model.token_embedding_table.weight.shape)\n", + "print(\"dtype:\", model.token_embedding_table.weight.dtype)\n", + "print()\n", + "print(\"These numbers ARE what training rewrites.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "0881b2c7" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# trace the chain on one batch (steps 1 -> 2 -> 3)\n", + "xb, yb = get_batch(\"train\")\n", + "print(\"STEP 1 prerequisite: batch shapes\", xb.shape, yb.shape)\n", + "\n", + "model.zero_grad(set_to_none=True)\n", + "logits, loss = model(xb, yb)\n", + "print(\"STEP 2: forward produced loss =\", round(loss.item(), 4), \"(scalar)\")\n", + "\n", + "loss.backward() # STEP 3\n", + "w = model.token_embedding_table.weight\n", + "print(\"STEP 3: backward filled w.grad with shape\", w.grad.shape)\n", + "print(\" example grad[t,h]:\", w.grad[stoi[\"t\"], stoi[\"h\"]].item())\n", + "print(\"STEP 4: run optimizer.step() to add grad into weights (see 8.1)\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "019a14ab" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8.0 What is a gradient? (and why it matters for AI)\n", + "\n", + "Imagine you are standing on a **hilly field in thick fog**. You cannot see the whole\n", + "landscape, but you can feel which way the ground **slopes upward** under your feet.\n", + "\n", + "The **gradient** is that slope information:\n", + "\n", + "- **Direction** — which way is uphill (steepest climb)\n", + "- **Size** — how steep it is\n", + "\n", + "In machine learning, the “hill” is usually the **loss** (how wrong the model is).\n", + "Your “position” on the hill is all the **weights** in the model. The gradient tells\n", + "each weight: *if you nudge yourself a tiny bit this way, does loss go up or down —\n", + "and by how much?*\n", + "\n", + "#### Simple real-life pictures\n", + "\n", + "**1. Finding the bottom of a valley (blindfolded).**\n", + "You want the **lowest point** (minimum loss). You feel the slope and take a small\n", + "step **downhill**. Repeat. That step direction is the **negative gradient** (opposite\n", + "of uphill). Training is exactly this: start with bad guesses, feel the slope, take\n", + "small downhill steps, thousands of times.\n", + "\n", + "**2. Tuning a radio dial.**\n", + "You turn the dial slightly left — worse static. Turn right — clearer signal. The\n", + "“gradient” is like: *turning right helps; turning left hurts.* Each weight in a\n", + "neural net is a dial; the gradient says which way to turn each one to reduce error.\n", + "\n", + "**3. Baking cookies.**\n", + "Cookies too salty? You try tiny changes — less salt, more sugar, shorter bake — and\n", + "see what improves taste. Gradient descent does the same at massive scale: millions\n", + "of knobs, each nudged based on how much it helped or hurt.\n", + "\n", + "#### Why gradients are central to ML and AI\n", + "\n", + "Modern models have **millions or billions** of weights. You cannot tune them by hand.\n", + "\n", + "Gradients make learning systematic:\n", + "\n", + "1. **Forward** — the model makes predictions (e.g. next letter after `\"Be\"`).\n", + "2. **Loss** — measures how wrong it was (cross-entropy: how little probability\n", + " landed on the true answer).\n", + "3. **Backward** — `loss.backward()` asks every weight: *how much did you contribute\n", + " to the mistake?* (automatic differentiation through the computation graph).\n", + "4. **Update** — `optimizer.step()` moves each weight a tiny step **downhill**.\n", + "\n", + "Without gradients, training would be random guessing. With them, the model improves\n", + "a little on every batch until it fits the data.\n", + "\n", + "| Real life | Machine learning |\n", + "|-----------|------------------|\n", + "| Hilly foggy field | Loss surface over all weights |\n", + "| Where you stand | Current model weights |\n", + "| Slope under your feet | **Gradient** (stored in `.grad`) |\n", + "| Step downhill | **`optimizer.step()`** |\n", + "| Valley floor | **Low loss** — model predicts well |\n", + "\n", + "**Take-home:** a gradient is the model’s way of *feeling* which direction each weight\n", + "should move to make fewer mistakes. Almost all modern AI learns by walking downhill\n", + "on that felt slope." + ], + "id": "7be0163b" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8.1 One training step\n", + "\n", + "Run this once to see loss change after a single update.\n", + "\n", + "**→ Training:** Minimal version of the full loop. If loss changes after `step()`, weights moved — training is working.\n" + ], + "id": "80359cd3" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8.1b The training loop — what *actually* runs each iteration?\n", + "\n", + "The loop in section 8.2 is short. Let's make *every line* explicit on one\n", + "concrete batch. Each iteration performs four operations in order.\n", + "\n", + " 1. xb, yb = get_batch(\"train\") # sample ONE new random batch from the tape\n", + " 2. logits, loss = model(xb, yb) # FORWARD pass — ALL rows/positions in parallel\n", + " 3. optimizer.zero_grad(); loss.backward() # BACKWARD — fills .grad on every weight\n", + " 4. optimizer.step() # NUDGE every weight a tiny bit against its gradient\n", + "\n", + "Crucially: **all batch rows share one forward, one backward, one optimizer step.**\n", + "The loop does NOT iterate over rows. That parallelism is why GPUs matter — each\n", + "training iteration is one big embedding lookup followed by one big gradient update.\n", + "\n", + "**Gradients for each weight are the SUM of contributions from every prediction**\n", + "in the batch. This is why bigger batches are more *stable*: each nudge is averaged\n", + "over more independent evidence, so the direction to nudge is less noisy.\n" + ], + "id": "b5af98ec" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Trace: one iteration, fully explicit, on a fresh batch of 3 rows x 8 positions.\n", + "# We print shapes, the loss scalar, the non-zero-count of the gradient, and\n", + "# the delta on one specific embedding entry (t -> h) before vs after step().\n", + "\n", + "bs = 3\n", + "\n", + "# 1) sample a batch\n", + "ix = torch.randint(len(train_data) - block_size, (bs,))\n", + "xb = torch.stack([train_data[i : i + block_size] for i in ix]).to(device)\n", + "yb = torch.stack([train_data[i + 1 : i + block_size + 1] for i in ix]).to(device)\n", + "\n", + "print(\"xb shape =\", tuple(xb.shape), \" ->\", bs, \"rows x\", block_size, \"positions =\", bs * block_size, \"supervised predictions in a single forward pass\")\n", + "print(\"row text:\", [decode(xb[i].tolist()) for i in range(bs)])\n", + "print()\n", + "\n", + "# 2) forward: produces (bs, block_size, vocab_size) logits AND one scalar loss\n", + "logits, loss = model(xb, yb)\n", + "print(\"forward pass -> logits shape\", tuple(logits.shape), \", loss scalar = %.4f\" % loss.item())\n", + "print()\n", + "\n", + "# 3) zero old gradients, then backprop from the single loss scalar\n", + "optimizer.zero_grad(set_to_none=True)\n", + "loss.backward()\n", + "w = model.token_embedding_table.weight\n", + "grad = w.grad\n", + "nonzero = (grad != 0).sum().item()\n", + "total = grad.numel()\n", + "print(\"backward pass -> gradient tensor has shape\", tuple(grad.shape))\n", + "print(\" entries with a non-zero gradient:\", nonzero, \"/\", total, \"(only char-pairs touched by this batch)\")\n", + "print(\" gradient norm = %.4f\" % grad.norm().item())\n", + "print()\n", + "\n", + "# 4) snapshot one embedding entry, step, and compare\n", + "t_id, h_id = stoi[\"t\"], stoi[\"h\"]\n", + "before = w[t_id, h_id].item()\n", + "optimizer.step()\n", + "after = w[t_id, h_id].item()\n", + "print(\"optimizer.step() -> one weight entry (t -> h): %+.6f -> %+.6f\" % (before, after))\n", + "print(\" delta = %+.8f\" % (after - before))\n", + "print()\n", + "print(\"Run this cell several times. Every iteration:\")\n", + "print(\" - the window text is different (get_batch re-samples),\")\n", + "print(\" - loss starts around 4.2 and slowly drops as training runs,\")\n", + "print(\" - the non-zero count in .grad changes (only chars in the batch count),\")\n", + "print(\" - the t->h entry nudges by ~1e-3 each time — tiny! But 10 000 such\")\n", + "print(\" nudges carve out the bigram statistics of the text.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "99da7331" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Why the model does NOT need to see the whole book at once\n", + "\n", + "The training loop never processes more than `batch_size x block_size` tokens at\n", + "a time. The entire book is never loaded into one matrix. Instead:\n", + "\n", + "- Over 10 000 iterations with `batch_size = 32`, the model sees ~2.5M char-pairs.\n", + "- Different iterations expose different subsets of the vocabulary to gradients.\n", + "- Eventually every (prev_char, next_char) pair that *can* appear *does* appear,\n", + " and the embedding table row for that previous char gets nudged to score the\n", + " common next chars higher than the rare ones.\n", + "\n", + "The fact that the model is *small* (one matrix, ~4 200 parameters) is what makes\n", + "this simple-sampling strategy sufficient: there isn't much capacity to waste, and\n", + "the \"correct\" answer for each row is unambiguous bigram statistics.\n" + ], + "id": "51398f79" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)\n", + "xb, yb = get_batch(\"train\")\n", + "_, loss_before = model(xb, yb)\n", + "optimizer.zero_grad(set_to_none=True)\n", + "_, loss = model(xb, yb)\n", + "loss.backward()\n", + "optimizer.step()\n", + "_, loss_after = model(xb, yb)\n", + "print(f\"before: {loss_before.item():.4f} after: {loss_after.item():.4f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "8949b082" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# watch ONE matrix entry change (run after 8.1 created optimizer)\n", + "w = model.token_embedding_table.weight\n", + "before = w[stoi['t'], stoi['h']].item()\n", + "\n", + "xb, yb = get_batch(\"train\")\n", + "_, loss = model(xb, yb)\n", + "optimizer.zero_grad(set_to_none=True)\n", + "loss.backward()\n", + "optimizer.step()\n", + "\n", + "after = w[stoi['t'], stoi['h']].item()\n", + "print(f\"loss: {loss.item():.4f}\")\n", + "print(f\"weight score for t->h: {before:.6f} -> {after:.6f}\")\n", + "print(\"changed:\", before != after)\n" + ], + "execution_count": null, + "outputs": [], + "id": "8d2cc5da" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "sd = model.state_dict()\n", + "print(\"keys in state_dict:\", list(sd.keys()))\n", + "print(\"weight tensor shape:\", sd[\"token_embedding_table.weight\"].shape)\n", + "\n", + "# optional: save / load (uncomment to use across sessions)\n", + "# torch.save(sd, \"bigram_shakespeare.pt\")\n", + "# model.load_state_dict(torch.load(\"bigram_shakespeare.pt\", map_location=device))\n", + "print(\"trained weights live in memory inside model until you torch.save them.\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "66fd4109" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8.2 Full training loop + val loss\n", + "\n", + "Every 1000 steps we average loss on train **and** val batches. Val should track\n", + "train; if train drops but val does not → overfitting.\n", + "\n", + "**→ Training:** 10,000 updates to the embedding matrix. Printed val loss checks that weights generalise, not only memorise train slices.\n" + ], + "id": "603bd9b0" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "batch_size = 32\n", + "\n", + "@torch.no_grad()\n", + "def estimate_loss(eval_iters=100):\n", + " out = {}\n", + " model.eval()\n", + " for split in [\"train\", \"val\"]:\n", + " losses = torch.zeros(eval_iters)\n", + " for k in range(eval_iters):\n", + " X, Y = get_batch(split)\n", + " _, L = model(X, Y)\n", + " losses[k] = L.item()\n", + " out[split] = losses.mean().item()\n", + " model.train()\n", + " return out\n", + "\n", + "for step in range(10000):\n", + " xb, yb = get_batch(\"train\")\n", + " _, loss = model(xb, yb)\n", + " optimizer.zero_grad(set_to_none=True)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if step % 1000 == 0:\n", + " L = estimate_loss()\n", + " print(f\"step {step:5d} train {L['train']:.4f} val {L['val']:.4f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "b34ee95e" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Your turn 8** — After training, is val loss below the random baseline (~4.17)?\n" + ], + "id": "69ff0dfc" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "L = estimate_loss(50)\n", + "baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n", + "print(f\"train {L['train']:.4f} val {L['val']:.4f} baseline {baseline:.4f}\")\n" + ], + "execution_count": null, + "outputs": [], + "id": "23be596a" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 9. Generation after training + what's next\n", + "\n", + "\n", + "**You are here (section 9):** see the **goal** achieved (partially — bigram is limited).\n", + "\n", + "**Needs:** section 8 (trained weights in `model`).\n", + "**Unlocks:** next notebook — self-attention for longer context.\n", + "Same `generate` as section 7 — but weights learned letter/space patterns.\n", + "Still **not** real Shakespeare: bigram only sees **one** character back.\n", + "\n", + "Next notebook: **self-attention** so each token can use the full prefix.\n", + "\n", + "**Whole notebook checklist**\n", + "- [ ] Text → tokens → 1-D tensor tape\n", + "- [ ] `x`/`y` shift, prefix trick, `(B,T)` batching\n", + "- [ ] Logits, softmax, cross-entropy\n", + "- [ ] Train loop + val loss\n", + "- [ ] Generate before vs after\n", + "\n", + "**→ Training:** Proof that weights changed. Better letter patterns = the matrix learned co-occurrence stats. Weights live in `model` until you `torch.save` them.\n" + ], + "id": "1a49aef3" + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", + "print(decode(model.generate(context, max_new_tokens=400)[0].tolist()))\n" + ], + "execution_count": null, + "outputs": [], + "id": "2ef869fb" + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.12.10)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.10" + } }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 0. Setup check\n", - "\n", - "First confirm PyTorch can actually see your GPU. If `CUDA available` is False,\n", - "stop \u2014 we're accidentally on the CPU build again." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "import torch\n", - "import torch.nn as nn\n", - "from torch.nn import functional as F\n", - "\n", - "print(\"torch:\", torch.__version__)\n", - "print(\"CUDA available:\", torch.cuda.is_available())\n", - "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", - "print(\"device:\", device)\n", - "if device == \"cuda\":\n", - " print(\"GPU:\", torch.cuda.get_device_name(0))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. The data\n", - "\n", - "A language model learns to predict the *next* piece of text given what came before.\n", - "So all we need is a big pile of text. Ours is ~1MB of Shakespeare." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "with open(\"input.txt\", \"r\", encoding=\"utf-8\") as f:\n", - " text = f.read()\n", - "\n", - "print(\"length in characters:\", len(text))\n", - "print(\"----\")\n", - "print(text[:250])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. Tokenization\n", - "\n", - "A neural net only eats **numbers**, not letters. *Tokenization* = deciding how to\n", - "chop text into pieces (\"tokens\") and mapping each piece to an integer.\n", - "\n", - "Real GPTs use *sub-word* tokens (e.g. \"ing\", \"tion\") via an algorithm called BPE.\n", - "We'll start with the simplest possible scheme: **one token = one character**.\n", - "\n", - "- Pro: trivially simple, tiny vocabulary.\n", - "- Con: sequences get long (every letter is a step), so the model has to work harder.\n", - "\n", - "> Analogy: tokenization is choosing your alphabet. Characters = small alphabet, long\n", - "> words. Sub-words = bigger alphabet, shorter words. GPT picks the bigger alphabet." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "# all the unique characters that occur in the text = our vocabulary\n", - "chars = sorted(list(set(text)))\n", - "vocab_size = len(chars)\n", - "print(\"\".join(chars))\n", - "print(\"vocab_size:\", vocab_size)\n", - "\n", - "# build the two lookup tables: char->int (stoi) and int->char (itos)\n", - "stoi = {ch: i for i, ch in enumerate(chars)}\n", - "itos = {i: ch for i, ch in enumerate(chars)}\n", - "encode = lambda s: [stoi[c] for c in s] # string -> list of ints\n", - "decode = lambda l: \"\".join(itos[i] for i in l) # list of ints -> string\n", - "\n", - "print(encode(\"hello there\"))\n", - "print(decode(encode(\"hello there\")))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. Encode the whole dataset into a tensor\n", - "\n", - "We turn the entire text into one long 1-D tensor of integers, then hold out the last\n", - "10% as a validation set (to check the model isn't just memorising)." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "data = torch.tensor(encode(text), dtype=torch.long)\n", - "print(data.shape, data.dtype)\n", - "print(data[:50]) # the first 50 chars, now as numbers\n", - "\n", - "n = int(0.9 * len(data))\n", - "train_data = data[:n]\n", - "val_data = data[n:]\n", - "print(\"train:\", len(train_data), \" val:\", len(val_data))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. Context: what does \"predict the next token\" actually mean?\n", - "\n", - "We never feed the whole 1M-character book at once. We feed **chunks** of a fixed\n", - "length called `block_size` (the context window). Within one chunk of length 8 there\n", - "are actually 8 training examples packed in \u2014 predict char 2 from char 1, char 3 from\n", - "chars 1-2, and so on. This teaches the model to handle contexts of every length from\n", - "1 up to `block_size`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "block_size = 8\n", - "x = train_data[:block_size]\n", - "y = train_data[1:block_size + 1] # y is x shifted left by one\n", - "for t in range(block_size):\n", - " context = x[:t + 1]\n", - " target = y[t]\n", - " print(f\"when input is {context.tolist()} -> target is {target}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5. Batching\n", - "\n", - "GPUs are happiest doing many things in parallel. So instead of one chunk, we grab a\n", - "**batch** of several random chunks and process them at once. Two key numbers:\n", - "\n", - "- `batch_size` = how many independent chunks we stack (parallelism).\n", - "- `block_size` = how long each chunk is (context length).\n", - "\n", - "`get_batch` picks random starting points and returns:\n", - "- `x`: the inputs, shape `(batch_size, block_size)`\n", - "- `y`: the targets (each input shifted by one), same shape.\n", - "\n", - "`.to(device)` moves the tensors onto the GPU." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "torch.manual_seed(1337)\n", - "batch_size = 4\n", - "block_size = 8\n", - "\n", - "def get_batch(split):\n", - " d = train_data if split == \"train\" else val_data\n", - " ix = torch.randint(len(d) - block_size, (batch_size,)) # random start indices\n", - " x = torch.stack([d[i:i + block_size] for i in ix])\n", - " y = torch.stack([d[i + 1:i + block_size + 1] for i in ix])\n", - " return x.to(device), y.to(device)\n", - "\n", - "xb, yb = get_batch(\"train\")\n", - "print(\"inputs \", xb.shape)\n", - "print(xb)\n", - "print(\"targets\", yb.shape)\n", - "print(yb)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6. The Bigram model \u2014 our baseline\n", - "\n", - "Simplest possible language model: to predict the next character, look **only** at the\n", - "current character. We do this with an `nn.Embedding` table of shape\n", - "`(vocab_size, vocab_size)`. Row `i` is directly the scores (\"logits\") for what comes\n", - "after token `i`.\n", - "\n", - "Key concepts in this cell:\n", - "- **Logits**: raw, unnormalised scores for each possible next token.\n", - "- **Cross-entropy loss**: measures how surprised the model is by the true next token.\n", - " Lower = better. With `vocab_size` random guesses we expect loss ~ `-ln(1/vocab_size)`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "class BigramLanguageModel(nn.Module):\n", - " def __init__(self, vocab_size):\n", - " super().__init__()\n", - " # each token directly reads the next-token logits from this lookup table\n", - " self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n", - "\n", - " def forward(self, idx, targets=None):\n", - " logits = self.token_embedding_table(idx) # (B, T, vocab_size)\n", - " if targets is None:\n", - " return logits, None\n", - " # cross_entropy wants (N, C), so flatten the batch & time dims together\n", - " B, T, C = logits.shape\n", - " loss = F.cross_entropy(logits.view(B * T, C), targets.view(B * T))\n", - " return logits, loss\n", - "\n", - " def generate(self, idx, max_new_tokens):\n", - " # idx is (B, T) of current context; extend it one token at a time\n", - " for _ in range(max_new_tokens):\n", - " logits, _ = self(idx)\n", - " logits = logits[:, -1, :] # only the last step -> (B, C)\n", - " probs = F.softmax(logits, dim=-1) # scores -> probabilities\n", - " idx_next = torch.multinomial(probs, num_samples=1) # sample one token\n", - " idx = torch.cat((idx, idx_next), dim=1)\n", - " return idx\n", - "\n", - "model = BigramLanguageModel(vocab_size).to(device)\n", - "logits, loss = model(xb, yb)\n", - "expected = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n", - "print(\"logits shape:\", logits.shape)\n", - "print(f\"initial loss: {loss.item():.4f} (random guessing ~ {expected:.4f})\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 7. Generate *before* training\n", - "\n", - "Right now the weights are random, so this will be pure gibberish. We start generation\n", - "from a single newline character (token 0). This is our \"before\" picture." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", - "print(decode(model.generate(context, max_new_tokens=200)[0].tolist()))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 8. Train it\n", - "\n", - "The training loop is the same four steps forever:\n", - "1. **forward** \u2014 get predictions + loss\n", - "2. **zero_grad** \u2014 clear old gradients\n", - "3. **backward** \u2014 backpropagation computes how each weight affected the loss\n", - "4. **step** \u2014 the optimizer (AdamW) nudges weights to reduce the loss\n", - "\n", - "> Analogy: loss is \"how wrong we are.\" Backprop tells each knob which way to turn.\n", - "> The optimizer turns all the knobs a little. Repeat thousands of times." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)\n", - "batch_size = 32\n", - "\n", - "for step in range(10000):\n", - " xb, yb = get_batch(\"train\")\n", - " logits, loss = model(xb, yb)\n", - " optimizer.zero_grad(set_to_none=True)\n", - " loss.backward()\n", - " optimizer.step()\n", - " if step % 1000 == 0:\n", - " print(f\"step {step:5d}: loss {loss.item():.4f}\")\n", - "\n", - "print(\"final loss:\", round(loss.item(), 4))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 9. Generate *after* training\n", - "\n", - "Still not Shakespeare \u2014 a bigram model only ever looks at one character back, so it\n", - "can't form real words. But it should now produce plausible letter/space patterns\n", - "instead of random noise. That jump is the model *learning*.\n", - "\n", - "The fix for \"only one character back\" is **self-attention** \u2014 letting each token look\n", - "at all previous tokens. That's the heart of the Transformer, and it's our next step." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "execution_count": null, - "outputs": [], - "source": [ - "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", - "print(decode(model.generate(context, max_new_tokens=400)[0].tolist()))" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "name": "python", - "version": "3.12" - } - }, - "nbformat": 4, - "nbformat_minor": 5 + "nbformat": 4, + "nbformat_minor": 5 } \ No newline at end of file diff --git a/01_build_gpt.ipynb.bak b/01_build_gpt.ipynb.bak new file mode 100644 index 0000000..5cbefac --- /dev/null +++ b/01_build_gpt.ipynb.bak @@ -0,0 +1,1864 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "ef460528", + "metadata": {}, + "source": [ + "# Let's build GPT — from scratch\n", + "\n", + "Tiny GPT on *tiny Shakespeare* — **concept -> code -> Your turn** each step.\n", + "\n", + "**Red thread:** every section has **-> Training:** plus a **goal stack**\n", + "so you always know *why* this step exists and what it unlocks next.\n", + "\n", + "**Kernel:** `gpt_from_scratch/.venv`. Run cells top to bottom.\n", + "\n", + "**Ultimate goal:** weight matrices that score the **true** next character\n", + "highly on Shakespeare — so generation looks like language, not noise.\n", + "\n", + "**Read the chain bottom-up.** Each line needs the line below it first:\n", + "```\n", + "GOAL better weights -> better text generation\n", + " ^ step 4 optimizer.step() uses gradients to nudge weights\n", + " ^ step 3 loss.backward() computes gradient per weight\n", + " ^ step 2 loss = cross_entropy(...) one number: how wrong we were\n", + " ^ step 1 forward: model(xb, yb) logits compared to answer key yb\n", + " ^ step 0 get_batch -> xb, yb one practice batch from train_data\n", + " ^ ... tokenize, tensor tape everything in sections 0-5\n", + "```\n", + "\n", + "| Sec | You build | So that later... |\n", + "|-----|-----------|------------------|\n", + "| 0 | PyTorch + GPU | tensors & weights can run fast |\n", + "| 1-2 | text -> ids | model eats numbers |\n", + "| 3 | tensor tape + split | sample train/val batches |\n", + "| 4-5 | x/y windows, batches | one `forward` has questions + answers |\n", + "| 6 | logits, loss, weights | `loss.backward()` has something to fix |\n", + "| 7 | generate (untrained) | baseline before learning |\n", + "| 8 | forward/backward/step | **weights actually change** |\n", + "| 9 | generate (trained) | see the goal — better patterns |\n" + ] + }, + { + "cell_type": "markdown", + "id": "33e0cea0", + "metadata": {}, + "source": [ + "## 0. Setup (overview)\n", + "\n", + "\n", + "**You are here (section 0):** install the math engine (PyTorch + GPU).\n", + "\n", + "**Needs:** nothing in this notebook yet.\n", + "**Unlocks:** tensors on `device` (section 3 batches, section 6 weights all live here).\n", + "We use **PyTorch**: tensors (arrays of numbers) + `torch.nn` (neural nets).\n", + "Training is heavy math on millions of numbers — a **GPU** speeds that up 10–100×.\n", + "\n", + "**→ Training:** Learning = huge amounts of matrix math. Setup ensures PyTorch, tensors, and the GPU are ready to run that math millions of times.\n" + ] + }, + { + "cell_type": "markdown", + "id": "91321dde", + "metadata": {}, + "source": [ + "### 0.1 Imports\n", + "\n", + "These three lines appear in almost every PyTorch project.\n", + "\n", + "**→ Training:** `torch` runs the math, `nn` holds the **weight matrices** we train, `F` computes **loss** (how wrong we are) from predictions.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d99d4439", + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "from torch.nn import functional as F\n", + "print(\"PyTorch version:\", torch.__version__)\n" + ] + }, + { + "cell_type": "markdown", + "id": "57084056", + "metadata": {}, + "source": [ + "### 0.2 CPU vs GPU (`device`)\n", + "\n", + "Tensors can live on **CPU** (always works) or **CUDA/GPU** (fast). We pick once\n", + "and reuse `device` so every tensor and model lands in the same place.\n", + "\n", + "If `CUDA available` is False, you installed CPU-only PyTorch — training still\n", + "works but will be slow.\n", + "\n", + "**→ Training:** Model **weights** and each **batch** must sit on the same device. Training on GPU is the same logic, just far faster.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3f68853", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"CUDA available:\", torch.cuda.is_available())\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(\"using device:\", device)\n", + "if device == \"cuda\":\n", + " print(\"GPU:\", torch.cuda.get_device_name(0))\n" + ] + }, + { + "cell_type": "markdown", + "id": "6126b5a6", + "metadata": {}, + "source": [ + "### 0.3 What is a tensor?\n", + "\n", + "A **tensor** is an n-dimensional array. `shape (5,)` = 5 numbers in a row.\n", + "`shape (2, 3)` = 2 rows × 3 columns. Token ids will be 1-D tensors.\n", + "\n", + "**→ Training:** Training data (`xb`, `yb`), model **weights**, and **gradients** are all tensors. Slicing `train_data` builds each practice batch.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2b37eccd", + "metadata": {}, + "outputs": [], + "source": [ + "row = torch.tensor([10, 20, 30])\n", + "grid = torch.tensor([[1, 2], [3, 4]])\n", + "print(\"row shape:\", row.shape, \" values:\", row.tolist())\n", + "print(\"grid shape:\", grid.shape)\n" + ] + }, + { + "cell_type": "markdown", + "id": "8a1bb533", + "metadata": {}, + "source": [ + "**Your turn 0** — Create a 2×2 tensor of zeros on `device`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25810c05", + "metadata": {}, + "outputs": [], + "source": [ + "practice = torch.zeros(2, 2, device=device)\n", + "print(practice)\n", + "print(\"on device:\", practice.device.type == device)\n" + ] + }, + { + "cell_type": "markdown", + "id": "bad266c2", + "metadata": {}, + "source": [ + "## 1. The data (overview)\n", + "\n", + "\n", + "**You are here (section 1):** load the raw training material.\n", + "\n", + "**Needs:** section 0 (imports).\n", + "**Unlocks:** `text` -> tokenization (section 2) -> the pool of next-token questions.\n", + "A **language model** learns patterns in text by repeatedly asking: *given what\n", + "came before, what comes next?* No separate label file — the text **is** the labels.\n", + "\n", + "We use ~1M characters of Shakespeare in `input.txt`.\n", + "\n", + "**→ Training:** The text is our only ingredient. Each training step asks: \"given chars before, predict the next\" — labels are **inside** the text.\n" + ] + }, + { + "cell_type": "markdown", + "id": "f7039a6a", + "metadata": {}, + "source": [ + "### 1.1 Load the file\n", + "\n", + "Plain Python: read the whole file into one Python string `text`.\n", + "\n", + "**→ Training:** `text` is the raw pool of examples. Nothing trains until we encode it and sample windows from it.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9e4fa669", + "metadata": {}, + "outputs": [], + "source": [ + "with open(\"input.txt\", \"r\", encoding=\"utf-8\") as f:\n", + " text = f.read()\n", + "print(\"characters in file:\", len(text))\n", + "print(\"---- first 200 chars ----\")\n", + "print(text[:200])\n" + ] + }, + { + "cell_type": "markdown", + "id": "83f3fde7", + "metadata": {}, + "source": [ + "### 1.2 What's inside?\n", + "\n", + "The model sees **every** character: letters, spaces, punctuation, newlines.\n", + "Newlines matter — they mark line breaks in the plays.\n", + "\n", + "**→ Training:** The model learns the **statistics** of these characters (spaces, newlines, letter pairs). It does not store the file verbatim in weights.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "31f45b4a", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"newlines:\", text.count(\"\\n\"))\n", + "print(\"unique character count:\", len(set(text)))\n", + "print(\"first line:\", repr(text.split(\"\\n\")[0][:60]))\n" + ] + }, + { + "cell_type": "markdown", + "id": "9bfde4e7", + "metadata": {}, + "source": [ + "**Your turn 1** — Print the last 100 characters of `text`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dc5dfe07", + "metadata": {}, + "outputs": [], + "source": [ + "print(repr(text[-100:]))\n" + ] + }, + { + "cell_type": "markdown", + "id": "f9a73d64", + "metadata": {}, + "source": [ + "## 2. Tokenization (overview)\n", + "\n", + "\n", + "**You are here (section 2):** text -> integer token ids.\n", + "\n", + "**Needs:** section 1 (`text`).\n", + "**Unlocks:** `encode` / `vocab_size` -> tensor tape (section 3) -> model input/output size (section 6).\n", + "Neural nets need **integers**. **Tokenization** = chop text into **tokens** and\n", + "assign each an id.\n", + "\n", + "| Approach | Token | Used by |\n", + "|----------|-------|---------|\n", + "| Character | one letter | us (simple) |\n", + "| Sub-word (BPE) | pieces like \"ing\" | GPT, Llama |\n", + "\n", + "We use **one char = one token** — tiny vocab, longer sequences.\n", + "\n", + "**→ Training:** The network only sees integers. Tokenization turns Shakespeare into the id sequences that every forward pass and loss calculation uses.\n" + ] + }, + { + "cell_type": "markdown", + "id": "78bd31aa", + "metadata": {}, + "source": [ + "### 2.1 Vocabulary\n", + "\n", + "The **vocabulary** = all distinct characters in the training text. `vocab_size`\n", + "is how many different tokens the model can output.\n", + "\n", + "**→ Training:** `vocab_size` fixes the width of the output scores. Each training step: model outputs `vocab_size` logits; truth is **one** of those ids.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "07af82cf", + "metadata": {}, + "outputs": [], + "source": [ + "chars = sorted(list(set(text)))\n", + "vocab_size = len(chars)\n", + "print(\"vocab_size:\", vocab_size)\n", + "print(\"all chars:\", repr(\"\".join(chars)))\n" + ] + }, + { + "cell_type": "markdown", + "id": "9f03ff92", + "metadata": {}, + "source": [ + "### 2.2 Lookup tables: `stoi` and `itos`\n", + "\n", + "- `stoi` (**s**tring **to** **i**nt): character → id\n", + "- `itos` (**i**nt **to** **s**tring): id → character\n", + "\n", + "**→ Training:** `stoi` converts batches of text to `xb`/`yb` tensors. `itos` is for us humans when inspecting generated output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3e0b4759", + "metadata": {}, + "outputs": [], + "source": [ + "stoi = {ch: i for i, ch in enumerate(chars)}\n", + "itos = {i: ch for i, ch in enumerate(chars)}\n", + "print('stoi[h] =', stoi['h'])\n", + "print('itos[0] =', repr(itos[0]))\n" + ] + }, + { + "cell_type": "markdown", + "id": "8f5a3085", + "metadata": {}, + "source": [ + "### 2.3 `encode` and `decode`\n", + "\n", + "Encode: string → list of ids. Decode: list of ids → string. Round-trip should match.\n", + "\n", + "**→ Training:** `encode(text)` built the master tape `data`. Every training batch is a slice of those ids — never raw strings inside the model.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14c4e3a3", + "metadata": {}, + "outputs": [], + "source": [ + "encode = lambda s: [stoi[c] for c in s]\n", + "decode = lambda l: \"\".join(itos[i] for i in l)\n", + "demo = \"hello there\"\n", + "ids = encode(demo)\n", + "print(demo, \"->\", ids)\n", + "print(\"back:\", decode(ids))\n" + ] + }, + { + "cell_type": "markdown", + "id": "40dd1ac1", + "metadata": {}, + "source": [ + "**Your turn 2** — Encode a short phrase. Flip one id and decode — see one wrong letter.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0b50c865", + "metadata": {}, + "outputs": [], + "source": [ + "phrase = \"First Citizen\"\n", + "ids = encode(phrase)\n", + "ids[3] = (ids[3] + 1) % vocab_size\n", + "print(\"corrupted:\", decode(ids))\n" + ] + }, + { + "cell_type": "markdown", + "id": "f90ea698", + "metadata": {}, + "source": [ + "## 3. Tensor tape + train/val split (overview)\n", + "\n", + "\n", + "**You are here (section 3):** whole book as one integer tape + train/val split.\n", + "\n", + "**Needs:** section 2 (`encode`, `vocab_size`).\n", + "**Unlocks:** `train_data` to sample batches (section 5); `val_data` to check generalisation (section 8).\n", + "Section 2 left us with Python lists of ids. Here we wrap the whole book in a\n", + "**tensor** (3.1), then split it into train and validation (3.2–3.4).\n", + "\n", + "**→ Training:** `train_data` is what we sample from when updating weights. `val_data` checks whether those weights work on **held-out** text.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5a1caac1", + "metadata": {}, + "source": [ + "### 3.1 What is a tensor?\n", + "\n", + "In section 2 we had a Python **list** of token ids: `[18, 47, 56, ...]`.\n", + "PyTorch uses a **tensor** for the same idea — a block of numbers the library\n", + "can slice, stack, and run on a GPU very fast.\n", + "\n", + "**Think of it like this:** a list is a handwritten shopping list. A tensor is\n", + "the same list typed into a spreadsheet — same items, but the computer can do\n", + "math on the whole column at once.\n", + "\n", + "#### How many dimensions? (`shape`)\n", + "\n", + "| Example | shape | Meaning |\n", + "|---------|-------|---------|\n", + "| `torch.tensor(5)` | `()` | one number (scalar) |\n", + "| `torch.tensor([1,2,3])` | `(3,)` | one row of 3 ids — **our book tape** |\n", + "| `torch.tensor([[1,2],[3,4]])` | `(2, 2)` | 2 rows × 2 cols — later `(B, T)` batches |\n", + "\n", + "The comma in `(3,)` means \"one dimension of length 3\" — not a tuple pair.\n", + "\n", + "#### `dtype` — what kind of number?\n", + "\n", + "- `torch.long` — **integers** (token ids). No decimals.\n", + "- `torch.float32` — **decimals** (model weights, logits, loss).\n", + "\n", + "Token ids are always integers, so we use `dtype=torch.long`.\n", + "\n", + "#### Why not stay with Python lists?\n", + "\n", + "1. **Speed** — millions of multiply/adds per second on GPU.\n", + "2. **Slicing** — `data[1000:1008]` gives another tensor, no copy of the whole book.\n", + "3. **Gradients** — later, PyTorch tracks how each weight affects loss (training).\n", + "\n", + "Run the cell below: same ids as a list vs as a tensor.\n", + "\n", + "**→ Training:** We store the whole book as one tensor so we can grab random windows fast and (later) move batches to the GPU in one shot.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21051ca8", + "metadata": {}, + "outputs": [], + "source": [ + "ids_list = encode(\"First\")\n", + "ids_tensor = torch.tensor(ids_list, dtype=torch.long)\n", + "\n", + "print(\"Python list:\", ids_list)\n", + "print(\"tensor: \", ids_tensor)\n", + "print(\"shape:\", ids_tensor.shape)\n", + "print(\"dtype:\", ids_tensor.dtype)\n", + "print()\n", + "print(\"slice [1:3]:\", ids_tensor[1:3])\n", + "print(\"decode slice:\", decode(ids_tensor[1:3].tolist()))\n", + "print(\"back to list:\", ids_tensor.tolist())\n" + ] + }, + { + "cell_type": "markdown", + "id": "7b498e68", + "metadata": {}, + "source": [ + "**Your turn 3.1** — Encode a word of your choice. Build a tensor, print `.shape`\n", + "and `.dtype`. Slice the middle two ids and decode them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c40b416b", + "metadata": {}, + "outputs": [], + "source": [ + "word = \"Citizen\" # edit me\n", + "t = torch.tensor(encode(word), dtype=torch.long)\n", + "print(\"shape:\", t.shape, \" dtype:\", t.dtype)\n", + "mid = t[1:3]\n", + "print(\"middle ids:\", mid.tolist(), \"->\", decode(mid.tolist()))\n" + ] + }, + { + "cell_type": "markdown", + "id": "d1c84cec", + "metadata": {}, + "source": [ + "### 3.2 String → tensor (the whole book)\n", + "\n", + "Now we wrap **every** token id in the dataset into one 1-D tensor `data`.\n", + "Same idea as 3.1, but ~1 million ids long — our **tape**.\n", + "\n", + "`dtype=torch.long` = integers only. Shape `(N,)` = one long row of length `N`.\n", + "\n", + "**→ Training:** `data` / `train_data` never change during training. Only the **model weights** change. The tape is fixed questions; weights are learned answers.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d419690c", + "metadata": {}, + "outputs": [], + "source": [ + "data = torch.tensor(encode(text), dtype=torch.long)\n", + "print(\"shape:\", data.shape, \" dtype:\", data.dtype)\n", + "print(\"first 10 ids:\", data[:10].tolist())\n", + "print(\"first 10 chars:\", decode(data[:10].tolist()))\n" + ] + }, + { + "cell_type": "markdown", + "id": "5450cf66", + "metadata": {}, + "source": [ + "### 3.3 Train / validation split\n", + "\n", + "- **train_data** — model learns from this\n", + "- **val_data** — later we check loss on unseen tail of the book\n", + "\n", + "If train loss falls but val loss stays high → **memorising**, not generalising.\n", + "\n", + "**→ Training:** We **only** run `backward()` on train batches. Val loss is read-only: if train improves but val does not, weights are memorising.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bbc77a95", + "metadata": {}, + "outputs": [], + "source": [ + "n = int(0.9 * len(data))\n", + "train_data = data[:n]\n", + "val_data = data[n:]\n", + "print(\"train:\", len(train_data), \" val:\", len(val_data))\n", + "print(\"sum check:\", len(train_data) + len(val_data) == len(data))\n" + ] + }, + { + "cell_type": "markdown", + "id": "dcf68150", + "metadata": {}, + "source": [ + "### 3.4 The split boundary in plain text\n", + "\n", + "Train ends and val begins at the same place in the original book — no gap, no overlap.\n", + "\n", + "**→ Training:** Confirms train and val are contiguous slices — so val loss is a fair \"unseen tail of the book\" check.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e9bc8a68", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"...end of train:\", repr(decode(train_data[-40:].tolist())))\n", + "print(\"start of val: \", repr(decode(val_data[:40].tolist())))\n" + ] + }, + { + "cell_type": "markdown", + "id": "51b55dd8", + "metadata": {}, + "source": [ + "**Your turn 3** — What fraction of the book is validation? *(~10%.)*\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "70e5f2ac", + "metadata": {}, + "outputs": [], + "source": [ + "frac_val = len(val_data) / len(data)\n", + "print(f\"validation fraction: {frac_val:.1%}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "fe71c7d8", + "metadata": {}, + "source": [ + "## 4. Predicting the next token (overview)\n", + "\n", + "\n", + "**You are here (section 4):** define ONE training question (context -> next token).\n", + "\n", + "**Needs:** section 3 (`train_data` tape).\n", + "**Unlocks:** `x` / `y` / `block_size` -> batched training examples (section 5) -> loss targets (section 6).\n", + "A language model is a **next-token guesser**. You show it some text; it scores\n", + "every possible next token; you compare to the real answer and nudge its weights.\n", + "\n", + "Section 4 breaks that idea into small pieces. **Read one part, run the code\n", + "right below it**, then move on. By the end you'll understand chunks, `block_size`,\n", + "`x`/`y`, and the prefix trick — the foundation for batching in section 5.\n", + "\n", + "**→ Training:** The **objective** for the whole notebook: minimise surprise at the true next token. Everything in §4 defines one training example.\n" + ] + }, + { + "cell_type": "markdown", + "id": "e7a8f7c0", + "metadata": {}, + "source": [ + "### 4.1 The next-token game (one step)\n", + "\n", + "The training question is always the same:\n", + "\n", + "> Given what came before → what is the **single** next character?\n", + "\n", + "There is no separate labels file. The label is hiding **in the text itself**:\n", + "the next character in the sequence.\n", + "\n", + "**Analogy:** a friend starts a sentence and you blurt out the next word. They\n", + "tell you if you were right. That's one training step.\n", + "\n", + "**→ Training:** One row of the loss compares model scores to **one** correct next id. A batch contains `B × T` such comparisons at once.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6adf5e2f", + "metadata": {}, + "outputs": [], + "source": [ + "# tiny phrase from our vocabulary — watch one step at a time\n", + "phrase = \"First Citizen\"\n", + "ids = encode(phrase)\n", + "print(\"text: \", phrase)\n", + "print(\"ids: \", ids)\n", + "print()\n", + "for i in range(len(ids) - 1):\n", + " context = decode(ids[: i + 1])\n", + " nxt = itos[ids[i + 1]]\n", + " print(f\" seen {context!r:20s} -> predict {nxt!r}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "537d244e", + "metadata": {}, + "source": [ + "**Your turn 4.1** — Change `phrase` to another short string (letters/spaces only).\n", + "How many training steps does a string of length *L* give? *(Answer: L − 1.)*\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "49b69d41", + "metadata": {}, + "outputs": [], + "source": [ + "phrase = \"To be\" # edit me\n", + "ids = encode(phrase)\n", + "steps = len(ids) - 1\n", + "print(f\"length {len(ids)} -> {steps} next-token questions\")\n", + "for i in range(steps):\n", + " print(f\" {decode(ids[:i+1])!r} -> {itos[ids[i+1]]!r}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "a5039f5f", + "metadata": {}, + "source": [ + "### 4.2 The book is one long tape\n", + "\n", + "After section 3, the whole dataset is a single 1-D tensor `train_data` — a tape\n", + "of token ids with no breaks between chapters:\n", + "\n", + "```\n", + "train_data: [18, 47, 56, 57, ...] ← millions of ids, one after another\n", + "```\n", + "\n", + "Training never asks \"predict the whole book.\" It only ever asks about **local**\n", + "stretches of that tape.\n", + "\n", + "**→ Training:** Every example is a contiguous window on this tape. Random `start` indices = seeing different parts of the book each step.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d217e75d", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"train_data length:\", len(train_data))\n", + "print(\"first 15 ids: \", train_data[:15].tolist())\n", + "print(\"first 15 chars: \", decode(train_data[:15].tolist()))\n", + "print(\"chars 15-30: \", decode(train_data[15:30].tolist()))\n", + "print()\n", + "print(\"The tape is continuous — char 15 follows char 14 in the real book.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "0ecf904b", + "metadata": {}, + "source": [ + "### 4.3 Why we use chunks (`block_size`)\n", + "\n", + "We could slide one character at a time, but in practice we grab a **window** of\n", + "`block_size` characters at once. Three reasons:\n", + "\n", + "1. **Memory** — models store state per position; a 1M-wide window would not fit on a GPU.\n", + "2. **Fixed context** — Transformers are built to look at at most `block_size` tokens.\n", + "3. **Efficiency** — one forward pass on 8 positions trains 8 questions in parallel (next part).\n", + "\n", + "`block_size` is the **context window**: how far back the model is *allowed* to look.\n", + "We use 8 here so the numbers fit on screen; GPT-4 uses thousands.\n", + "\n", + "**→ Training:** `block_size` = how many next-token questions per row (`T`). Also caps context length the architecture can use.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7b330af9", + "metadata": {}, + "outputs": [], + "source": [ + "block_size = 8\n", + "start = 1000 # where we cut the tape\n", + "chunk = train_data[start : start + block_size]\n", + "\n", + "print(f\"block_size = {block_size}\")\n", + "print(f\"slice from index {start} to {start + block_size - 1}\")\n", + "print(\"chunk ids: \", chunk.tolist())\n", + "print(\"chunk text: \", repr(decode(chunk.tolist())))\n", + "print()\n", + "print(\"book length:\", len(train_data), \" window:\", block_size,\n", + " \" ratio:\", round(len(train_data) / block_size))\n" + ] + }, + { + "cell_type": "markdown", + "id": "f19d6e14", + "metadata": {}, + "source": [ + "**Your turn 4.3** — Change `start`. The chunk should always be exactly `block_size`\n", + "characters (except if you pick a start too close to the end).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8528eb17", + "metadata": {}, + "outputs": [], + "source": [ + "start = 25000 # try different values\n", + "chunk = train_data[start : start + block_size]\n", + "print(\"len(chunk):\", len(chunk), \" text:\", repr(decode(chunk.tolist())))\n" + ] + }, + { + "cell_type": "markdown", + "id": "342c105f", + "metadata": {}, + "source": [ + "### 4.4 Inputs `x` and targets `y` (shift by one)\n", + "\n", + "From one slice we build two equal-length tensors:\n", + "\n", + "- **`x`** = the window `train_data[start : start + block_size]`\n", + "- **`y`** = the same window, shifted **one step forward** in time:\n", + " `train_data[start + 1 : start + block_size + 1]`\n", + "\n", + "So `y` is always \"what comes next\" relative to `x`, position by position:\n", + "\n", + "```\n", + "x: h e l l o _ t h\n", + "y: e l l o _ t h ? ← ? is the 9th char (first char AFTER the window)\n", + "```\n", + "\n", + "The model never sees `y` during the forward pass — only during loss calculation.\n", + "\n", + "#### Tie-in to the training goal\n", + "\n", + "**Ultimate goal:** nudge **weights** so the model scores the true next character highly.\n", + "\n", + "`x` and `y` are how we package **one practice round** for that goal:\n", + "\n", + "| Tensor | Role in training | Analogy |\n", + "|--------|------------------|---------|\n", + "| **`x`** (later `xb`) | **Question** — tokens the model reads to produce logits | exam paper |\n", + "| **`y`** (later `yb`) | **Answer key** — the correct next token at each position | mark scheme |\n", + "\n", + "At column `t`: model reads `x[t]`, predicts a distribution; loss compares it to **`y[t]`** (the char that really followed on the tape). Wrong guess → higher loss → `backward()` pushes weights to do better next time.\n", + "\n", + "**In the chain (why this section exists):**\n", + "```\n", + "GOAL: better weights\n", + " ^ needs backward()\n", + " ^ needs loss (one number)\n", + " ^ needs yb — the CORRECT next tokens ← y is the template for yb\n", + " ^ needs forward on xb — model predictions ← x is the template for xb\n", + "```\n", + "\n", + "Without the shift-by-one split you have text but **no labelled examples** — nothing to score, no loss, no training.\n", + "\n", + "**→ Training:** `get_batch` builds matrices `xb`/`yb` the same way as `x`/`y` here. Every training step is: predict from `xb`, grade against `yb`, update weights.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0dec560b", + "metadata": {}, + "outputs": [], + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"x:\", decode(x.tolist()))\n", + "print(\"y:\", decode(y.tolist()))\n", + "print()\n", + "for i in range(block_size):\n", + " print(f\" x[{i}]={repr(itos[x[i].item()])} y[{i}]={repr(itos[y[i].item()])}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "70613e82", + "metadata": {}, + "source": [ + "**Your turn 4.4** — Confirm `y[i]` is always the id **right after** `x[i]` on the tape.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5be4764d", + "metadata": {}, + "outputs": [], + "source": [ + "ok = all(y[t].item() == train_data[start + t + 1].item() for t in range(block_size))\n", + "print(\"every y[t] is the next char after x[t] on the tape:\", ok)\n", + "if not ok:\n", + " for t in range(block_size):\n", + " print(t, x[t].item(), y[t].item(), train_data[start + t + 1].item())\n" + ] + }, + { + "cell_type": "markdown", + "id": "6fecc0fb", + "metadata": {}, + "source": [ + "### 4.5 The prefix trick (one chunk → many lessons)\n", + "\n", + "Here is the key idea that confuses many beginners:\n", + "\n", + "An 8-character chunk is **not** one training example. It is **eight** examples\n", + "packed together — one for each prefix length.\n", + "\n", + "| step `t` | context (prefix of `x`) | must predict `y[t]` |\n", + "|----------|---------------------------|---------------------|\n", + "| 0 | 1st char only | 2nd char |\n", + "| 1 | 1st + 2nd | 3rd char |\n", + "| 2 | first 3 chars | 4th char |\n", + "| … | … | … |\n", + "| 7 | all 8 chars | 9th char (just past the window) |\n", + "\n", + "**Analogy:** teaching spelling by asking \"after **h**?\", then \"after **he**?\",\n", + "then \"after **hel**?\" — same word, harder each time.\n", + "\n", + "**→ Training:** One slice yields `block_size` supervised examples per forward pass — more learning signal per batch.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e4559d0f", + "metadata": {}, + "outputs": [], + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"full chunk:\", repr(decode(x.tolist())))\n", + "print()\n", + "for t in range(block_size):\n", + " prefix = decode(x[: t + 1].tolist())\n", + " target = itos[y[t].item()]\n", + " print(f\"t={t} prefix {prefix!r:12s} -> {target!r}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "3be6fbb3", + "metadata": {}, + "source": [ + "**Your turn 4.5** — At `t=0`, how many characters does the model see? At `t=7`?\n", + "*(0: one char; 7: all eight.)*\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "898927dc", + "metadata": {}, + "outputs": [], + "source": [ + "for t in [0, 3, 7]:\n", + " prefix_len = t + 1\n", + " print(f\"t={t}: prefix length = {prefix_len}, target = {repr(itos[y[t].item()])}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "36cde17a", + "metadata": {}, + "source": [ + "### 4.6 Column alignment (what `x[t]` and `y[t]` mean together)\n", + "\n", + "When we later batch data into a matrix, **column `t`** means:\n", + "\n", + "- input token at that step: `x[t]`\n", + "- correct next token: `y[t]`\n", + "\n", + "Important: `y[t]` is the character that follows **`x[t]`** in the original book,\n", + "not \"the char after the whole prefix.\" (A Transformer uses the full prefix via\n", + "attention; our Bigram model will only look at `x[t]` — more in section 6.)\n", + "\n", + "**→ Training:** At column `t`, loss asks: \"after token `xb[b,t]`, was your distribution close to `yb[b,t]`?\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "876ad4ce", + "metadata": {}, + "outputs": [], + "source": [ + "start = 1000\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "for t in range(block_size):\n", + " on_tape = train_data[start + t].item()\n", + " on_tape_next = train_data[start + t + 1].item()\n", + " match = x[t].item() == on_tape and y[t].item() == on_tape_next\n", + " print(f\"t={t} x={repr(itos[x[t].item()])} y={repr(itos[y[t].item()])} matches tape: {match}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "78bef65a", + "metadata": {}, + "source": [ + "### 4.7 Shape of one chunk\n", + "\n", + "Right now `x` and `y` are **1-D** tensors of shape `(block_size,)` — one chunk,\n", + "one row. In section 5 we stack many chunks:\n", + "\n", + "- `(B, T)` where **B** = batch size, **T** = `block_size`\n", + "- Later the model outputs `(B, T, C)` — **C** = `vocab_size` scores per step\n", + "\n", + "Same idea, just more rows for GPU parallelism.\n", + "\n", + "**→ Training:** `(B, T)` tells us we get **B×T** gradient signals per step. Later `(B, T, C)` is one score vector per position for cross-entropy.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8bb27b4f", + "metadata": {}, + "outputs": [], + "source": [ + "x = train_data[1000 : 1000 + block_size]\n", + "y = train_data[1001 : 1001 + block_size]\n", + "\n", + "print(\"x shape:\", x.shape, \" y shape:\", y.shape)\n", + "print(\"T (time steps) = block_size =\", block_size)\n", + "print()\n", + "# pretend we had batch_size=3 — same T, three independent rows\n", + "B = 3\n", + "print(f\"batched inputs would be shape ({B}, {block_size}) = (B, T)\")\n", + "print(f\"model logits later: ({B}, {block_size}, {vocab_size}) = (B, T, C)\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "f86da52d", + "metadata": {}, + "source": [ + "### 4.8 Put it together — build your own `x` and `y`\n", + "\n", + "Pick any valid `start` index, build `x` and `y`, decode them, and verify every\n", + "target. This is exactly what `get_batch` does — but with random `start` values.\n", + "\n", + "**→ Training:** Same logic as `get_batch` — you're manually doing what the training loop automates thousands of times.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c2ee21e5", + "metadata": {}, + "outputs": [], + "source": [ + "start = 5000 # change me\n", + "x = train_data[start : start + block_size]\n", + "y = train_data[start + 1 : start + block_size + 1]\n", + "\n", + "print(\"start:\", start)\n", + "print(\"x:\", decode(x.tolist()))\n", + "print(\"y:\", decode(y.tolist()))\n", + "checks = [y[t].item() == train_data[start + t + 1].item() for t in range(block_size)]\n", + "print(\"all correct:\", all(checks))\n" + ] + }, + { + "cell_type": "markdown", + "id": "c1d3c8ce", + "metadata": {}, + "source": [ + "### 4.9 Why this matters for the Bigram model\n", + "\n", + "We feed 8 positions per row, and the loss averages over **all** of them — so one\n", + "chunk gives 8 gradients per row. But the Bigram architecture only uses the\n", + "**current** character at each position, not the full prefix. The data is ready for\n", + "long context; the model is not. Fixing that is what **self-attention** does later.\n", + "\n", + "**Section 4 checklist**\n", + "- [ ] Next-token = one step forward on the tape\n", + "- [ ] `block_size` = window length / context limit\n", + "- [ ] `y` is `x` shifted by one\n", + "- [ ] One chunk = `block_size` prefix drills\n", + "- [ ] Shapes: `(T,)` now → `(B, T)` in section 5\n", + "\n", + "**→ Training:** Data supplies full prefixes; Bigram weights only use the current char. Training still works, but capacity is limited until attention.\n" + ] + }, + { + "cell_type": "markdown", + "id": "45dafc47", + "metadata": {}, + "source": [ + "## 5. Batching (overview)\n", + "\n", + "\n", + "**You are here (section 5):** many `x`/`y` windows at once as matrices.\n", + "\n", + "**Needs:** sections 3-4 (`train_data`, `block_size`, shift-by-one).\n", + "**Unlocks:** `get_batch()` -> every `forward` in training (section 8) starts here.\n", + "Section 4 used **one** chunk (shape `(T,)`). GPUs want **many** chunks at once.\n", + "We stack `batch_size` independent windows → shape **`(B, T)`**.\n", + "\n", + "- **B** = `batch_size` (parallel snippets)\n", + "- **T** = `block_size` (time steps / context length)\n", + "\n", + "**→ Training:** Each optimizer step uses one batch `(xb, yb)`. Larger `B` = more examples contributing to the **same** weight update.\n" + ] + }, + { + "cell_type": "markdown", + "id": "ffc77ac6", + "metadata": {}, + "source": [ + "### 5.1 Why batch?\n", + "\n", + "**Analogy:** grading one exam at a time vs a stack of 32 exams — same rules,\n", + "more throughput. Random start indices each step = model sees different parts of\n", + "the book every iteration.\n", + "\n", + "**→ Training:** Random windows each step ≈ shuffled exposure to the book. Parallel rows = faster steps on GPU.\n" + ] + }, + { + "cell_type": "markdown", + "id": "0b02e589", + "metadata": {}, + "source": [ + "### 5.2 Stack two chunks by hand\n", + "\n", + "Before `get_batch`, see how two slices become a 2×8 matrix.\n", + "\n", + "**→ Training:** `get_batch` does this with `batch_size` rows. One `optimizer.step()` updates weights using loss from **all** rows.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11fc074b", + "metadata": {}, + "outputs": [], + "source": [ + "torch.manual_seed(0)\n", + "ix = torch.randint(len(train_data) - block_size, (2,))\n", + "chunk_a = train_data[ix[0] : ix[0] + block_size]\n", + "chunk_b = train_data[ix[1] : ix[1] + block_size]\n", + "stacked = torch.stack([chunk_a, chunk_b])\n", + "print(\"stacked shape:\", stacked.shape, \" (= 2 rows, block_size cols)\")\n", + "for row in range(2):\n", + " print(f\" row {row}:\", decode(stacked[row].tolist()))\n" + ] + }, + { + "cell_type": "markdown", + "id": "c296b808", + "metadata": {}, + "source": [ + "### 5.3 `get_batch` — random windows\n", + "\n", + "`xb` = inputs, `yb` = targets (each row shifted +1 on the tape). Same shapes.\n", + "\n", + "**→ Training:** Called every training iteration. Returns the `(xb, yb)` pair that `loss.backward()` will score and correct.\n", + "\n", + "**In the chain:** `get_batch` is **step 0** of every training iteration — supplies `xb` (questions) and `yb` (answers) before `model()` can run.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "67c1ce6f", + "metadata": {}, + "outputs": [], + "source": [ + "torch.manual_seed(1337)\n", + "batch_size = 4\n", + "\n", + "def get_batch(split):\n", + " d = train_data if split == \"train\" else val_data\n", + " ix = torch.randint(len(d) - block_size, (batch_size,))\n", + " x = torch.stack([d[i : i + block_size] for i in ix])\n", + " y = torch.stack([d[i + 1 : i + block_size + 1] for i in ix])\n", + " return x.to(device), y.to(device)\n", + "\n", + "xb, yb = get_batch(\"train\")\n", + "print(\"xb shape:\", xb.shape, \" yb shape:\", yb.shape)\n", + "print(xb)\n" + ] + }, + { + "cell_type": "markdown", + "id": "e72033cb", + "metadata": {}, + "source": [ + "### 5.4 Decode a batch row\n", + "\n", + "Row `b`, column `t`: `yb[b,t]` is the next token after `xb[b,t]` on the tape.\n", + "\n", + "**→ Training:** Sanity check that `yb` is the answer key aligned with `xb` before trusting the loss number.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d8be50cf", + "metadata": {}, + "outputs": [], + "source": [ + "b = 0\n", + "print(\"row\", b, \"input: \", decode(xb[b].tolist()))\n", + "print(\"row\", b, \"target:\", decode(yb[b].tolist()))\n", + "for t in range(block_size):\n", + " print(f\" t={t} x={repr(itos[xb[b,t].item()])} y={repr(itos[yb[b,t].item()])}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "3f69fc3b", + "metadata": {}, + "source": [ + "**Your turn 5** — Call `get_batch(\"val\")` and decode row 2.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cd1c51dc", + "metadata": {}, + "outputs": [], + "source": [ + "xb_val, yb_val = get_batch(\"val\")\n", + "print(decode(xb_val[2].tolist()))\n" + ] + }, + { + "cell_type": "markdown", + "id": "883c2705", + "metadata": {}, + "source": [ + "### 5.5 `.to(device)`\n", + "\n", + "Moves tensors to GPU so the model (also on GPU) can process them in one place.\n", + "\n", + "**Section 5 checklist**\n", + "- [ ] `(B, T)` = batch of `block_size`-long windows\n", + "- [ ] `yb` is `xb` shifted +1 on the tape\n", + "- [ ] `get_batch` randomises which part of the book each step\n", + "\n", + "**→ Training:** Moves batch tensors next to **weight matrices** on GPU so forward + backward run without device errors.\n" + ] + }, + { + "cell_type": "markdown", + "id": "0e99e2c5", + "metadata": {}, + "source": [ + "## 6. Logits, loss, and the Bigram model (overview)\n", + "\n", + "\n", + "**You are here (section 6):** the **weights** + how we score wrongness.\n", + "\n", + "**Needs:** section 5 (`xb`, `yb` shapes); section 2 (`vocab_size`).\n", + "**Unlocks:** `model(xb,yb)->loss` for `backward()` (section 8); `model(xb)->sample` for generation (section 7).\n", + "The **Bigram** model: next character depends **only** on the current character.\n", + "Before the full class, we unpack **logits**, **softmax**, and **cross-entropy** —\n", + "the same three ideas GPT uses at scale.\n", + "\n", + "**→ Training:** Forward produces **logits**; loss measures wrongness; backward rewrites the **embedding matrix** — the only weights in this model.\n" + ] + }, + { + "cell_type": "markdown", + "id": "f6bb711d", + "metadata": {}, + "source": [ + "### 6.1 Logits — raw scores\n", + "\n", + "For each possible next token, the model outputs a **logit** (unnormalised score).\n", + "Higher = more likely. Length = `vocab_size`.\n", + "\n", + "**→ Training:** Logits are not trained directly — **weights** produce logits. Gradients flow back through logits into those weights.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e7428034", + "metadata": {}, + "outputs": [], + "source": [ + "toy_logits = torch.tensor([2.0, 0.5, 0.1]) # pretend vocab_size=3\n", + "print(\"logits:\", toy_logits.tolist())\n", + "print(\"argmax (favourite class):\", toy_logits.argmax().item())\n" + ] + }, + { + "cell_type": "markdown", + "id": "ff289741", + "metadata": {}, + "source": [ + "### 6.2 Softmax — scores → probabilities\n", + "\n", + "Softmax squashes logits into positive numbers that **sum to 1** — a distribution.\n", + "\n", + "**→ Training:** Used heavily in **generation** (sampling). Training uses cross-entropy on logits directly (softmax is inside that).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "71e447c8", + "metadata": {}, + "outputs": [], + "source": [ + "probs = F.softmax(toy_logits, dim=0)\n", + "print(\"probs:\", [round(p, 3) for p in probs.tolist()])\n", + "print(\"sum:\", probs.sum().item())\n" + ] + }, + { + "cell_type": "markdown", + "id": "e60d5dd1", + "metadata": {}, + "source": [ + "### 6.3 Cross-entropy loss — how wrong were we?\n", + "\n", + "Compare model distribution to the **true** next token id.\n", + "- right answer gets high prob → low loss\n", + "- wrong answer → high loss\n", + "\n", + "Random guessing over `vocab_size` tokens → loss ≈ `-ln(1/vocab_size)`.\n", + "\n", + "**→ Training:** The single number `loss` that `backward()` uses. Lower loss ⇒ nudge weights so true next tokens get higher score.\n", + "\n", + "**In the chain:** cross-entropy turns logits + `yb` into the **single number** `loss`. Section 8 cannot run `backward()` until this exists.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9166861", + "metadata": {}, + "outputs": [], + "source": [ + "for target in [0, 1, 2]:\n", + " loss = F.cross_entropy(toy_logits.unsqueeze(0), torch.tensor([target]))\n", + " print(f\"true class {target} -> loss {loss.item():.3f}\")\n", + "expected = -torch.log(torch.tensor(1.0 / vocab_size))\n", + "print(f\"random baseline (vocab {vocab_size}): {expected.item():.3f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "7b45f3a7", + "metadata": {}, + "source": [ + "**Your turn 6** — Make class 2 strongest; loss for target 2 should drop.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dfe98faf", + "metadata": {}, + "outputs": [], + "source": [ + "strong = torch.tensor([0.1, 0.5, 2.5])\n", + "for target in [0, 2]:\n", + " loss = F.cross_entropy(strong.unsqueeze(0), torch.tensor([target]))\n", + " print(f\"target {target} -> {loss.item():.3f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "7926ed10", + "metadata": {}, + "source": [ + "### 6.4 Bigram = one lookup table\n", + "\n", + "`nn.Embedding(vocab_size, vocab_size)` stores a matrix: **row i** = logits for\n", + "\"what follows token i?\" No memory of earlier tokens.\n", + "\n", + "**Analogy:** a cheat sheet — \"after `t` usually comes `h` or `e`\" — but only\n", + "looking at the **last** letter.\n", + "\n", + "**→ Training:** The entire learned model is this **one matrix** (~vocab² numbers). Training adjusts every row: \"what follows char i?\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f705fa50", + "metadata": {}, + "outputs": [], + "source": [ + "table = nn.Embedding(vocab_size, vocab_size)\n", + "ch = 't'\n", + "tid = stoi[ch]\n", + "row = table.weight[tid]\n", + "print(\"row for\", repr(ch), \"has shape:\", row.shape)\n", + "top_p, top_i = F.softmax(row, dim=0).topk(3)\n", + "print(\"top-3 followers (random weights):\")\n", + "for p, i in zip(top_p, top_i):\n", + " print(f\" {repr(itos[i.item()])}: {p.item():.3f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "e5208051", + "metadata": {}, + "source": [ + "### 6.5 The full `BigramLanguageModel`\n", + "\n", + "Forward: `idx` `(B,T)` → logits `(B,T,C)`. Flatten to `(B*T, C)` for one big\n", + "cross-entropy over every position in the batch.\n", + "\n", + "**→ Training:** `forward(xb, yb)` = predict on all positions, compare to `yb`, return scalar loss. `generate()` uses same weights without `yb`.\n", + "\n", + "**In the chain:** `forward(xb, yb)` is **step 1** of training. Without `yb`, you get logits only (generation path) — no loss, no backward.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f7cc9f29", + "metadata": {}, + "outputs": [], + "source": [ + "class BigramLanguageModel(nn.Module):\n", + " def __init__(self, vocab_size):\n", + " super().__init__()\n", + " self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n", + "\n", + " def forward(self, idx, targets=None):\n", + " logits = self.token_embedding_table(idx)\n", + " if targets is None:\n", + " return logits, None\n", + " B, T, C = logits.shape\n", + " loss = F.cross_entropy(logits.view(B * T, C), targets.view(B * T))\n", + " return logits, loss\n", + "\n", + " def generate(self, idx, max_new_tokens):\n", + " for _ in range(max_new_tokens):\n", + " logits, _ = self(idx)\n", + " logits = logits[:, -1, :]\n", + " probs = F.softmax(logits, dim=-1)\n", + " idx_next = torch.multinomial(probs, num_samples=1)\n", + " idx = torch.cat((idx, idx_next), dim=1)\n", + " return idx\n", + "\n", + "model = BigramLanguageModel(vocab_size).to(device)\n", + "xb, yb = get_batch(\"train\")\n", + "logits, loss = model(xb, yb)\n", + "print(\"logits shape (B, T, C):\", logits.shape)\n", + "print(f\"initial loss: {loss.item():.4f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "1614313d", + "metadata": {}, + "source": [ + "**Your turn 6b** — Initial loss should be near `-ln(1/vocab_size)`. Compare.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25b7140e", + "metadata": {}, + "outputs": [], + "source": [ + "baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n", + "print(f\"loss {loss.item():.4f} vs random baseline {baseline:.4f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "23f37864", + "metadata": {}, + "source": [ + "**Section 6 checklist**\n", + "- [ ] Logits → softmax → probs (generation)\n", + "- [ ] Cross-entropy compares preds to true `yb` (training)\n", + "- [ ] Bigram row `i` = next-token scores after char `i` only\n" + ] + }, + { + "cell_type": "markdown", + "id": "8f6036e2", + "metadata": {}, + "source": [ + "## 7. Generation before training (overview)\n", + "\n", + "\n", + "**You are here (section 7):** use weights **without** training (no `yb`, no `backward`).\n", + "\n", + "**Needs:** section 6 (`model`, random weights).\n", + "**Unlocks:** baseline gibberish to compare after section 9 — proof that training changed weights.\n", + "**Generation** = autoregressive sampling: predict one token, append it, repeat.\n", + "With random weights, output is gibberish — our **before** snapshot.\n", + "\n", + "**→ Training:** With **random** weights, generation is nonsense. This is the baseline before any `optimizer.step()` changes the matrices.\n" + ] + }, + { + "cell_type": "markdown", + "id": "f51c9655", + "metadata": {}, + "source": [ + "### 7.1 One generation step by hand\n", + "\n", + "Context → forward → logits at **last** position → softmax → sample one id.\n", + "\n", + "**→ Training:** Same forward path as training, but no `yb` and no `backward()`. Shows what one row of trained weights **does** at inference.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f9cdda7f", + "metadata": {}, + "outputs": [], + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device) # start: newline\n", + "logits, _ = model(context)\n", + "last_logits = logits[0, -1, :]\n", + "probs = F.softmax(last_logits, dim=0)\n", + "sampled = torch.multinomial(probs, num_samples=1)\n", + "print(\"context:\", repr(itos[context[0,0].item()]))\n", + "print(\"sampled next char:\", repr(itos[sampled.item()]))\n" + ] + }, + { + "cell_type": "markdown", + "id": "f8461a58", + "metadata": {}, + "source": [ + "### 7.2 Full generation (200 chars, untrained)\n", + "\n", + "Start from newline (id 0). Expect nonsense.\n", + "\n", + "**→ Training:** Save mentally (or in output) for contrast with section 9 after weights have been updated.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4b5c3e45", + "metadata": {}, + "outputs": [], + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", + "out = model.generate(context, max_new_tokens=200)\n", + "print(decode(out[0].tolist()))\n" + ] + }, + { + "cell_type": "markdown", + "id": "eb818ac2", + "metadata": {}, + "source": [ + "## 8. Training (overview)\n", + "\n", + "**You are here (section 8):** run the full chain thousands of times.\n", + "\n", + "**Needs:** ALL above — especially `get_batch` (5), `model`+loss (6).\n", + "**Unlocks:** updated weights — the **result** of training.\n", + "\n", + "One iteration — **do not reorder**:\n", + "\n", + "```\n", + "prereq: xb,yb = get_batch() # step 0 — sections 4-5\n", + "step 1: logits, loss = model(xb,yb) # forward — section 6\n", + "step 2: (loss is ready) # scalar surprise\n", + "step 3: loss.backward() # gradients per weight\n", + "step 4: optimizer.step() # nudge weights\n", + "```\n", + "\n", + "Plus `zero_grad` before step 1 each time — wipe last batch's gradients.\n", + "\n", + "**-> Training:** This section is where the ultimate goal happens: weights change.\n" + ] + }, + { + "cell_type": "markdown", + "id": "4ac65ed4", + "metadata": {}, + "source": [ + "### 8.0 What training is — fundamentals\n", + "\n", + "Strip away the jargon and training is four ideas:\n", + "\n", + "1. **Weights** — numbers in matrices (our Bigram: one `vocab_size × vocab_size` table).\n", + "2. **Forward** — multiply/lookup through weights → predictions (logits).\n", + "3. **Loss** — one number: how wrong predictions are vs true next tokens `yb`.\n", + "4. **Update** — nudge every weight a tiny bit to lower loss (`backward` + `step`).\n", + "\n", + "**Analogy:** weights = knobs on a mixer; loss = how bad the song sounds; training turns knobs repeatedly until the mix fits the data.\n", + "\n", + "The model does **not** store the book inside weights. It stores **habits** like \"after `t` often comes `h`.\"\n", + "\n", + "**→ Training:** This is the goal every earlier section prepared: data → batches → loss → changed weights.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0881b2c7", + "metadata": {}, + "outputs": [], + "source": [ + "# THE trainable artifact: model parameters (weight matrices)\n", + "n_params = sum(p.numel() for p in model.parameters())\n", + "print(\"trainable parameters:\", n_params)\n", + "print(\"embedding weight shape:\", model.token_embedding_table.weight.shape)\n", + "print(\"dtype:\", model.token_embedding_table.weight.dtype)\n", + "print()\n", + "print(\"These numbers ARE what training rewrites.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "362158ed", + "metadata": {}, + "source": [ + "### 8.0b One step = forward, loss, backward, update\n", + "\n", + "| Step | What happens | Uses from earlier sections |\n", + "|------|----------------|---------------------------|\n", + "| forward | `model(xb, yb)` → logits, loss | `get_batch`, `xb`/`yb`, embedding |\n", + "| zero_grad | clear old gradients | — |\n", + "| backward | `loss.backward()` computes ∂loss/∂weight | cross-entropy on logits |\n", + "| step | `optimizer.step()` adds small change to weights | AdamW |\n", + "\n", + "One step touches the **same** weight matrix thousands of times indirectly (via every position in the batch), then applies one averaged update.\n", + "\n", + "Run **8.1** next (creates `optimizer`), then the cell below watches one weight change.\n" + ] + }, + { + "cell_type": "markdown", + "id": "eb72fa20", + "metadata": {}, + "source": [ + "### 8.0d Backward (`loss.backward`) — why it exists in the chain\n", + "\n", + "**Goal of backward:** figure out *how each weight should move* to make loss smaller.\n", + "\n", + "You cannot run backward until the steps below exist:\n", + "\n", + "| Prerequisite | Built in | Provides |\n", + "|--------------|----------|----------|\n", + "| **Step 1** `xb`, `yb` | sections 4-5 | input + answer key |\n", + "| **Step 2** `logits`, `loss` | section 6 forward | one scalar \"how wrong\" |\n", + "| **Step 3** `loss.backward()` | **this step** | `.grad` on each weight |\n", + "| **Step 4** `optimizer.step()` | section 8.1+ | weights actually change |\n", + "\n", + "**Analogy:** loss = \"how off-key is the song?\" Backward = \"for each knob,\n", + "which direction fixes pitch?\" Step = \"turn the knobs slightly.\"\n", + "\n", + "Without step 2 (loss), step 3 has nothing to differentiate — PyTorch needs a\n", + "single number that depends on the weights.\n", + "\n", + "**-> Training:** backward is the bridge between \"we were wrong\" (loss) and\n", + "\"here is how to fix each weight\" (gradients).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "019a14ab", + "metadata": {}, + "outputs": [], + "source": [ + "# trace the chain on one batch (steps 1 -> 2 -> 3)\n", + "xb, yb = get_batch(\"train\")\n", + "print(\"STEP 1 prerequisite: batch shapes\", xb.shape, yb.shape)\n", + "\n", + "model.zero_grad(set_to_none=True)\n", + "logits, loss = model(xb, yb)\n", + "print(\"STEP 2: forward produced loss =\", round(loss.item(), 4), \"(scalar)\")\n", + "\n", + "loss.backward() # STEP 3\n", + "w = model.token_embedding_table.weight\n", + "print(\"STEP 3: backward filled w.grad with shape\", w.grad.shape)\n", + "print(\" example grad[t,h]:\", w.grad[stoi[\"t\"], stoi[\"h\"]].item())\n", + "print(\"STEP 4: run optimizer.step() to add grad into weights (see 8.1)\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "843b881d", + "metadata": {}, + "source": [ + "**Your turn 8d** — Run the cell above, then answer:\n", + "1. What shape is `w.grad`? (Same as weights — one partial derivative per entry.)\n", + "2. Why must `loss` be a single number, not a vector?\n", + "3. What section would break if `yb` were missing?\n", + "\n", + "*(2: backward needs one objective to differentiate. 3: section 6 loss — no answer key.)*\n" + ] + }, + { + "cell_type": "markdown", + "id": "80359cd3", + "metadata": {}, + "source": [ + "### 8.1 One training step\n", + "\n", + "Run this once to see loss change after a single update.\n", + "\n", + "**→ Training:** Minimal version of the full loop. If loss changes after `step()`, weights moved — training is working.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8949b082", + "metadata": {}, + "outputs": [], + "source": [ + "optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)\n", + "xb, yb = get_batch(\"train\")\n", + "_, loss_before = model(xb, yb)\n", + "optimizer.zero_grad(set_to_none=True)\n", + "_, loss = model(xb, yb)\n", + "loss.backward()\n", + "optimizer.step()\n", + "_, loss_after = model(xb, yb)\n", + "print(f\"before: {loss_before.item():.4f} after: {loss_after.item():.4f}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8d2cc5da", + "metadata": {}, + "outputs": [], + "source": [ + "# watch ONE matrix entry change (run after 8.1 created optimizer)\n", + "w = model.token_embedding_table.weight\n", + "before = w[stoi['t'], stoi['h']].item()\n", + "\n", + "xb, yb = get_batch(\"train\")\n", + "_, loss = model(xb, yb)\n", + "optimizer.zero_grad(set_to_none=True)\n", + "loss.backward()\n", + "optimizer.step()\n", + "\n", + "after = w[stoi['t'], stoi['h']].item()\n", + "print(f\"loss: {loss.item():.4f}\")\n", + "print(f\"weight score for t->h: {before:.6f} -> {after:.6f}\")\n", + "print(\"changed:\", before != after)\n" + ] + }, + { + "cell_type": "markdown", + "id": "4b41e5fc", + "metadata": {}, + "source": [ + "### 8.0c Where the result of training is stored\n", + "\n", + "| Location | What | Persists? |\n", + "|----------|------|-----------|\n", + "| `model.token_embedding_table.weight` | learned matrix in RAM/VRAM | until kernel restarts |\n", + "| `model.state_dict()` | name → tensor map of all weights | same |\n", + "| disk (`torch.save`) | checkpoint file | yes — **not done by default** |\n", + "\n", + "Re-run the notebook from the top → new random weights → must train again unless you load a saved checkpoint.\n", + "\n", + "**→ Training:** After section 8.2, `state_dict()` holds the **result**. Section 9 reads those same weights to generate text.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "66fd4109", + "metadata": {}, + "outputs": [], + "source": [ + "sd = model.state_dict()\n", + "print(\"keys in state_dict:\", list(sd.keys()))\n", + "print(\"weight tensor shape:\", sd[\"token_embedding_table.weight\"].shape)\n", + "\n", + "# optional: save / load (uncomment to use across sessions)\n", + "# torch.save(sd, \"bigram_shakespeare.pt\")\n", + "# model.load_state_dict(torch.load(\"bigram_shakespeare.pt\", map_location=device))\n", + "print(\"trained weights live in memory inside model until you torch.save them.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "603bd9b0", + "metadata": {}, + "source": [ + "### 8.2 Full training loop + val loss\n", + "\n", + "Every 1000 steps we average loss on train **and** val batches. Val should track\n", + "train; if train drops but val does not → overfitting.\n", + "\n", + "**→ Training:** 10,000 updates to the embedding matrix. Printed val loss checks that weights generalise, not only memorise train slices.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b34ee95e", + "metadata": {}, + "outputs": [], + "source": [ + "batch_size = 32\n", + "\n", + "@torch.no_grad()\n", + "def estimate_loss(eval_iters=100):\n", + " out = {}\n", + " model.eval()\n", + " for split in [\"train\", \"val\"]:\n", + " losses = torch.zeros(eval_iters)\n", + " for k in range(eval_iters):\n", + " X, Y = get_batch(split)\n", + " _, L = model(X, Y)\n", + " losses[k] = L.item()\n", + " out[split] = losses.mean().item()\n", + " model.train()\n", + " return out\n", + "\n", + "for step in range(10000):\n", + " xb, yb = get_batch(\"train\")\n", + " _, loss = model(xb, yb)\n", + " optimizer.zero_grad(set_to_none=True)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if step % 1000 == 0:\n", + " L = estimate_loss()\n", + " print(f\"step {step:5d} train {L['train']:.4f} val {L['val']:.4f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "69ff0dfc", + "metadata": {}, + "source": [ + "**Your turn 8** — After training, is val loss below the random baseline (~4.17)?\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "23be596a", + "metadata": {}, + "outputs": [], + "source": [ + "L = estimate_loss(50)\n", + "baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n", + "print(f\"train {L['train']:.4f} val {L['val']:.4f} baseline {baseline:.4f}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "1a49aef3", + "metadata": {}, + "source": [ + "## 9. Generation after training + what's next\n", + "\n", + "\n", + "**You are here (section 9):** see the **goal** achieved (partially — bigram is limited).\n", + "\n", + "**Needs:** section 8 (trained weights in `model`).\n", + "**Unlocks:** next notebook — self-attention for longer context.\n", + "Same `generate` as section 7 — but weights learned letter/space patterns.\n", + "Still **not** real Shakespeare: bigram only sees **one** character back.\n", + "\n", + "Next notebook: **self-attention** so each token can use the full prefix.\n", + "\n", + "**Whole notebook checklist**\n", + "- [ ] Text → tokens → 1-D tensor tape\n", + "- [ ] `x`/`y` shift, prefix trick, `(B,T)` batching\n", + "- [ ] Logits, softmax, cross-entropy\n", + "- [ ] Train loop + val loss\n", + "- [ ] Generate before vs after\n", + "\n", + "**→ Training:** Proof that weights changed. Better letter patterns = the matrix learned co-occurrence stats. Weights live in `model` until you `torch.save` them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2ef869fb", + "metadata": {}, + "outputs": [], + "source": [ + "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n", + "print(decode(model.generate(context, max_new_tokens=400)[0].tolist()))\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/merge_transactions.py b/merge_transactions.py deleted file mode 100644 index a97986f..0000000 --- a/merge_transactions.py +++ /dev/null @@ -1,129 +0,0 @@ -import os -import pandas as pd -import glob -import sys -from PyQt5.QtWidgets import (QApplication, QWidget, QVBoxLayout, QPushButton, - QLabel, QFileDialog, QMessageBox) -from PyQt5.QtCore import Qt - -class MergeTransactionsApp(QWidget): - def __init__(self): - super().__init__() - self.input_dir = "" - self.output_dir = "" - self.init_ui() - - def init_ui(self): - self.setWindowTitle("Excel Transaction Merger") - self.resize(500, 300) - layout = QVBoxLayout() - - # Input Directory - self.input_label = QLabel(f"Input Directory: {self.input_dir if self.input_dir else 'Not Selected'}") - layout.addWidget(self.input_label) - btn_input = QPushButton("Select Input Directory") - btn_input.clicked.connect(self.select_input_dir) - layout.addWidget(btn_input) - - # Output Directory - self.output_label = QLabel(f"Output Directory: {self.output_dir if self.output_dir else 'Not Selected'}") - layout.addWidget(self.output_label) - btn_output = QPushButton("Select Output Directory") - btn_output.clicked.connect(self.select_output_dir) - layout.addWidget(btn_output) - - # Process Button - self.process_btn = QPushButton("Start Merging") - self.process_btn.setStyleSheet("background-color: #4CAF50; color: white; font-weight: bold;") - self.process_btn.clicked.connect(self.run_merge) - layout.addWidget(self.process_btn) - - self.setLayout(layout) - - def select_input_dir(self): - path = QFileDialog.getExistingDirectory(self, "Select Input Directory") - if path: - self.input_dir = path - self.input_label.setText(f"Input Directory: {path}") - - def select_output_dir(self): - path = QFileDialog.getExistingDirectory(self, "Select Output Directory") - if path: - self.output_dir = path - self.output_label.setText(f"Output Directory: {path}") - - def run_merge(self): - if not self.input_dir or not self.output_dir: - QMessageBox.warning(self, "Error", "Please select both input and output directories.") - return - - success = self.process_files() - if success: - QMessageBox.information(self, "Success", "Merge completed successfully!") - else: - QMessageBox.critical(self, "Error", "An error occurred during the merge process.") - - def process_files(self): - try: - input_path = os.path.abspath(self.input_dir) - if not os.path.isdir(input_path): - print(f"Error: {input_path} is not a valid directory.") - return False - - # Support .xlsx and .xls formats - file_pattern = os.path.join(input_path, "*.xls*") - files = glob.glob(file_pattern) - - all_data = [] - target_sheet = "交易流水" - - for file_path in files: - filename = os.path.basename(file_path) - - # Skip temporary excel files (starting with ~$) and the output summary file - if filename.startswith("~$") or "txn_summary" in filename: - continue - - try: - # Load Excel File - xls = pd.ExcelFile(file_path) - if target_sheet in xls.sheet_names: - # Use dtype=str to ensure all fields are read as strings (prevents scientific notation) - df = pd.read_excel(xls, sheet_name=target_sheet, dtype=str) - - # Record source filename for tracking - df['source_file'] = filename - all_data.append(df) - print(f"Successfully read: {filename}") - else: - print(f"Skipping file {filename}: Sheet '{target_sheet}' not found.") - except Exception as e: - print(f"Error processing {filename}: {e}") - - if all_data: - # Concatenate all DataFrames - merged_df = pd.concat(all_data, ignore_index=True) - - # Define output path in the selected directory - output_path = os.path.join(self.output_dir, "txn_summary.xlsx") - - # Save to Excel - with pd.ExcelWriter(output_path, engine='openpyxl') as writer: - merged_df.to_excel(writer, sheet_name='Summary', index=False) - - print("-" * 30) - print(f"Merge complete! Result saved to: {output_path}") - return True - else: - print("No files containing the '交易流水' sheet were found.") - return False - - except Exception as e: - print(f"Critical error during processing: {e}") - return False - -if __name__ == "__main__": - app = QApplication(sys.argv) - window = MergeTransactionsApp() - window.show() - sys.exit(app.exec_())