2625 lines
115 KiB
Plaintext
2625 lines
115 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Let's build GPT — from scratch\n",
|
||
"\n",
|
||
"Tiny GPT on *tiny Shakespeare* — **concept -> code -> Your turn** each step.\n"
|
||
],
|
||
"id": "ef460528"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Prologue — what this project does, in plain English\n",
|
||
"\n",
|
||
"This notebook builds a **tiny program that writes like Shakespeare**. You give it\n",
|
||
"the first few letters of a sentence (\"To be or\") and it tries to guess what comes\n",
|
||
"next — letter by letter — producing lines that look like the plays in `input.txt`.\n",
|
||
"\n",
|
||
"The program is **dumb at first**. It starts with no knowledge of English.\n",
|
||
"It only gets better by reading the text file thousands of times and slowly\n",
|
||
"learning which letters tend to follow which.\n",
|
||
"\n",
|
||
"The whole project follows Andrej Karpathy's *\"Let's build GPT from scratch\"*\n",
|
||
"YouTube lecture — this is our hands-on version of it.\n"
|
||
],
|
||
"id": "8f2d462e"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### The whole thing, in six simple steps\n",
|
||
"\n",
|
||
"Here is everything that happens, top to bottom, before we write a single line of code:\n",
|
||
"\n",
|
||
"**1. Read the book.** Load `input.txt` — a few Shakespeare plays glued together into one\n",
|
||
"long string of text.\n",
|
||
"\n",
|
||
"**2. Turn letters into numbers.** The computer doesn't read letters. Every unique character\n",
|
||
"gets an id number (e.g. `a → 1`, `b → 2`, ...). The whole book becomes one long list of\n",
|
||
"integers.\n",
|
||
"\n",
|
||
"**3. Slice the book into small windows.** Grab a short row of, say, 8 letters at a time\n",
|
||
"(\"Before w\"). Shift that same window by one to the right to make the *answer key*\n",
|
||
"(\"efore we\"). Each position in the window is a small flashcard: *given the prefix up to\n",
|
||
"here, what is the next letter?*\n",
|
||
"\n",
|
||
"**4. Give the window to the model.** The model looks at the letters it was given and\n",
|
||
"outputs a confidence score for **every possible next character** (all ~65 of them). High\n",
|
||
"score for `'f'` after \"Be\" would be good; high score for `'z'` after \"Be\" would be bad.\n",
|
||
"\n",
|
||
"**5. Measure how wrong the model was, and gently fix it.** A single number called *loss*\n",
|
||
"says: *on average, how confident was the model about the true next letter?* If the model\n",
|
||
"was confidently wrong, loss is big; the bigger the loss, the bigger the nudge we give to\n",
|
||
"the model's internal numbers (its *weights*). Then repeat — grab a new random window,\n",
|
||
"score again, nudge again.\n",
|
||
"\n",
|
||
"**6. After thousands of rounds, ask the model to write something new.** Give it one\n",
|
||
"starting character, let it predict the next one, then feed that prediction back in as\n",
|
||
"input, and repeat. You'll get a short Shakespeare-sounding paragraph.\n"
|
||
],
|
||
"id": "0132ba17"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### A single round of training, pictured simply\n",
|
||
"\n",
|
||
"Steps 3–5 above repeat inside one big loop. Every time the loop runs, this happens:\n",
|
||
"\n",
|
||
"```\n",
|
||
" grab a small chunk of text → model guesses next letter at every position →\n",
|
||
" |\n",
|
||
" compare guesses to the real letters ←──────────────┘\n",
|
||
" |\n",
|
||
" compute ONE number (\"how wrong was it?\")\n",
|
||
" |\n",
|
||
" nudge every weight slightly in the direction that would\n",
|
||
" have made the true letter a bit more likely this time\n",
|
||
"```\n",
|
||
"\n",
|
||
"Every loop uses a **different random chunk** of the book so the model can't just\n",
|
||
"memorize one passage. After maybe 10,000 rounds, the weights have quietly learned the\n",
|
||
"statistical habits of Shakespearean English — `th` is common, `qx` is rare, vowels\n",
|
||
"follow consonants, and so on.\n",
|
||
"\n",
|
||
"**You don't need to understand every formula.** The goal of this notebook is to make\n",
|
||
"the *shape* of the machine visible: how text becomes numbers, how one number (loss)\n",
|
||
"drives every change, and what the model actually produces when it's done.\n"
|
||
],
|
||
"id": "26549fd7"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 0.1 Imports\n",
|
||
"\n",
|
||
"These three lines appear in almost every PyTorch project.\n",
|
||
"\n",
|
||
"**→ Training:** `torch` runs the math, `nn` holds the **weight matrices** we train, `F` computes **loss** (how wrong we are) from predictions.\n"
|
||
],
|
||
"id": "91321dde"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"import torch\n",
|
||
"import torch.nn as nn\n",
|
||
"from torch.nn import functional as F\n",
|
||
"print(\"PyTorch version:\", torch.__version__)\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "d99d4439"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 0.2 CPU vs GPU (`device`)\n",
|
||
"\n",
|
||
"Tensors can live on **CPU** (always works) or **CUDA/GPU** (fast). We pick once\n",
|
||
"and reuse `device` so every tensor and model lands in the same place.\n",
|
||
"\n",
|
||
"If `CUDA available` is False, you installed CPU-only PyTorch — training still\n",
|
||
"works but will be slow.\n",
|
||
"\n",
|
||
"**→ Training:** Model **weights** and each **batch** must sit on the same device. Training on GPU is the same logic, just far faster.\n"
|
||
],
|
||
"id": "57084056"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"CUDA available:\", torch.cuda.is_available())\n",
|
||
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
|
||
"print(\"using device:\", device)\n",
|
||
"if device == \"cuda\":\n",
|
||
" print(\"GPU:\", torch.cuda.get_device_name(0))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "c3f68853"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 0.3 What is a tensor?\n",
|
||
"\n",
|
||
"A **tensor** is an n-dimensional array. `shape (5,)` = 5 numbers in a row.\n",
|
||
"`shape (2, 3)` = 2 rows × 3 columns. Token ids will be 1-D tensors.\n",
|
||
"\n",
|
||
"**→ Training:** Training data (`xb`, `yb`), model **weights**, and **gradients** are all tensors. Slicing `train_data` builds each practice batch.\n"
|
||
],
|
||
"id": "6126b5a6"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"row = torch.tensor([10, 20, 30])\n",
|
||
"grid = torch.tensor([[1, 2], [3, 4]])\n",
|
||
"print(\"row shape:\", row.shape, \" values:\", row.tolist())\n",
|
||
"print(\"grid shape:\", grid.shape)\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "2b37eccd"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 0** — Create a 2×2 tensor of zeros on `device`.\n"
|
||
],
|
||
"id": "8a1bb533"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"practice = torch.zeros(2, 2, device=device)\n",
|
||
"print(practice)\n",
|
||
"print(\"on device:\", practice.device.type == device)\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "25810c05"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Sidebar — where does the word \"tensor\" come from, and why is it everywhere in AI?**\n",
|
||
"\n",
|
||
"- **The name.** William Rowan Hamilton used \"tensor\" in the 1840s (for the \"norm\" or magnitude of a quaternion). The **modern mathematical** object—the multi-linear map with a transformation law—was formalised by **Gregorio Ricci-Curbastro** and his student **Tullio Levi-Civita** in the 1880s–1890s as \"absolute differential calculus\". **Albert Einstein** then made tensors famous by using them as the language of spacetime curvature in general relativity (1915).\n",
|
||
"\n",
|
||
"- **Why they were invented.** Physics (and geometry) should not depend on which coordinate system you pick. Tensors give you a single object whose components *transform in a predictable way* when you change coordinates — this is how Einstein could write equations of gravity that look the same to every observer. Informally: a tensor is a \"multi-linear machine\" that eats several vectors and produces a number; scalars, vectors, and matrices are all special cases.\n",
|
||
"\n",
|
||
"- **Why they are central in AI.** Nearly every neural-network operation, from embedding lookup through attention to softmax, is a **linear or element-wise non-linear operation on rectangular arrays** of numbers — exactly the shape a tensor is designed for. Three consequences:\n",
|
||
" 1. it is the **universal container** for data (`xb`, `yb`), model **weights**, and **gradients**;\n",
|
||
" 2. the same `Tensor` type abstracts over **CPU and GPU** (and other accelerators) so the training code is device-agnostic;\n",
|
||
" 3. PyTorch's `Tensor` tracks a **computation graph**, so `loss.backward()` can differentiate through any composed operation (automatic differentiation).\n",
|
||
"\n",
|
||
"So: tensors were built to give physics a **coordinate-free, multi-linear language**; modern deep learning happens to need *precisely* that — plus hardware (GPUs) optimised to crunch those rectangular arrays at scale. That is why `torch.Tensor` is the basic unit of everything in this notebook.\n"
|
||
],
|
||
"id": "tensor-history-sidebar"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.1 Load the file\n",
|
||
"\n",
|
||
"Plain Python: read the whole file into one Python string `text`.\n",
|
||
"\n",
|
||
"**→ Training:** `text` is the raw pool of examples. Nothing trains until we encode it and sample windows from it.\n"
|
||
],
|
||
"id": "f7039a6a"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"with open(\"input.txt\", \"r\", encoding=\"utf-8\") as f:\n",
|
||
" text = f.read()\n",
|
||
"print(\"characters in file:\", len(text))\n",
|
||
"print(\"---- first 200 chars ----\")\n",
|
||
"print(text[:200])\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "9e4fa669"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.2 What's inside?\n",
|
||
"\n",
|
||
"The model sees **every** character: letters, spaces, punctuation, newlines.\n",
|
||
"Newlines matter — they mark line breaks in the plays.\n",
|
||
"\n",
|
||
"**→ Training:** The model learns the **statistics** of these characters (spaces, newlines, letter pairs). It does not store the file verbatim in weights.\n"
|
||
],
|
||
"id": "83f3fde7"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"newlines:\", text.count(\"\\n\"))\n",
|
||
"print(\"unique character count:\", len(set(text)))\n",
|
||
"print(\"first line:\", repr(text.split(\"\\n\")[0][:60]))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "31f45b4a"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 1** — Print the last 100 characters of `text`.\n"
|
||
],
|
||
"id": "9bfde4e7"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(repr(text[-100:]))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "dc5dfe07"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.1 Vocabulary\n",
|
||
"\n",
|
||
"The **vocabulary** = all distinct characters in the training text. `vocab_size`\n",
|
||
"is how many different tokens the model can output.\n",
|
||
"\n",
|
||
"**→ Training:** `vocab_size` fixes the width of the output scores. Each training step: model outputs `vocab_size` logits; truth is **one** of those ids.\n"
|
||
],
|
||
"id": "78bd31aa"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"chars = sorted(list(set(text)))\n",
|
||
"vocab_size = len(chars)\n",
|
||
"print(\"vocab_size:\", vocab_size)\n",
|
||
"print(\"all chars:\", repr(\"\".join(chars)))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "07af82cf"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.2 Lookup tables: `stoi` and `itos`\n",
|
||
"\n",
|
||
"- `stoi` (**s**tring **to** **i**nt): character → id\n",
|
||
"- `itos` (**i**nt **to** **s**tring): id → character\n",
|
||
"\n",
|
||
"**→ Training:** `stoi` converts batches of text to `xb`/`yb` tensors. `itos` is for us humans when inspecting generated output.\n"
|
||
],
|
||
"id": "9f03ff92"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"stoi = {ch: i for i, ch in enumerate(chars)}\n",
|
||
"itos = {i: ch for i, ch in enumerate(chars)}\n",
|
||
"print('stoi[h] =', stoi['h'])\n",
|
||
"print('itos[0] =', repr(itos[0]))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "3e0b4759"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.3 `encode` and `decode`\n",
|
||
"\n",
|
||
"Encode: string → list of ids. Decode: list of ids → string. Round-trip should match.\n",
|
||
"\n",
|
||
"**→ Training:** `encode(text)` built the master tape `data`. Every training batch is a slice of those ids — never raw strings inside the model.\n"
|
||
],
|
||
"id": "8f5a3085"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"encode = lambda s: [stoi[c] for c in s]\n",
|
||
"decode = lambda l: \"\".join(itos[i] for i in l)\n",
|
||
"demo = \"hello there\"\n",
|
||
"ids = encode(demo)\n",
|
||
"print(demo, \"->\", ids)\n",
|
||
"print(\"back:\", decode(ids))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "14c4e3a3"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 2** — Encode a short phrase. Flip one id and decode — see one wrong letter.\n"
|
||
],
|
||
"id": "40dd1ac1"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"phrase = \"First Citizen\"\n",
|
||
"ids = encode(phrase)\n",
|
||
"ids[3] = (ids[3] + 1) % vocab_size\n",
|
||
"print(\"corrupted:\", decode(ids))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "0b50c865"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"ids_list = encode(\"First\")\n",
|
||
"ids_tensor = torch.tensor(ids_list, dtype=torch.long)\n",
|
||
"\n",
|
||
"print(\"Python list:\", ids_list)\n",
|
||
"print(\"tensor: \", ids_tensor)\n",
|
||
"print(\"shape:\", ids_tensor.shape)\n",
|
||
"print(\"dtype:\", ids_tensor.dtype)\n",
|
||
"print()\n",
|
||
"print(\"slice [1:3]:\", ids_tensor[1:3])\n",
|
||
"print(\"decode slice:\", decode(ids_tensor[1:3].tolist()))\n",
|
||
"print(\"back to list:\", ids_tensor.tolist())\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "21051ca8"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 3.1** — Encode a word of your choice. Build a tensor, print `.shape`\n",
|
||
"and `.dtype`. Slice the middle two ids and decode them.\n"
|
||
],
|
||
"id": "7b498e68"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"word = \"Citizen\" # edit me\n",
|
||
"t = torch.tensor(encode(word), dtype=torch.long)\n",
|
||
"print(\"shape:\", t.shape, \" dtype:\", t.dtype)\n",
|
||
"mid = t[1:3]\n",
|
||
"print(\"middle ids:\", mid.tolist(), \"->\", decode(mid.tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "c40b416b"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.2 String → tensor (the whole book)\n",
|
||
"\n",
|
||
"Now we wrap **every** token id in the dataset into one 1-D tensor `data`.\n",
|
||
"Same idea as 3.1, but ~1 million ids long — our **tape**.\n",
|
||
"\n",
|
||
"`dtype=torch.long` = integers only. Shape `(N,)` = one long row of length `N`.\n",
|
||
"\n",
|
||
"**→ Training:** `data` / `train_data` never change during training. Only the **model weights** change. The tape is fixed questions; weights are learned answers.\n"
|
||
],
|
||
"id": "d1c84cec"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"data = torch.tensor(encode(text), dtype=torch.long)\n",
|
||
"print(\"shape:\", data.shape, \" dtype:\", data.dtype)\n",
|
||
"print(\"first 10 ids:\", data[:10].tolist())\n",
|
||
"print(\"first 10 chars:\", decode(data[:10].tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "d419690c"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.3 Train / validation split\n",
|
||
"\n",
|
||
"- **train_data** — model learns from this\n",
|
||
"- **val_data** — later we check loss on unseen tail of the book\n",
|
||
"\n",
|
||
"If train loss falls but val loss stays high → **memorising**, not generalising.\n",
|
||
"\n",
|
||
"**→ Training:** We **only** run `backward()` on train batches. Val loss is read-only: if train improves but val does not, weights are memorising.\n"
|
||
],
|
||
"id": "5450cf66"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"n = int(0.9 * len(data))\n",
|
||
"train_data = data[:n]\n",
|
||
"val_data = data[n:]\n",
|
||
"print(\"train:\", len(train_data), \" val:\", len(val_data))\n",
|
||
"print(\"sum check:\", len(train_data) + len(val_data) == len(data))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "bbc77a95"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.4 The split boundary in plain text\n",
|
||
"\n",
|
||
"Train ends and val begins at the same place in the original book — no gap, no overlap.\n",
|
||
"\n",
|
||
"**→ Training:** Confirms train and val are contiguous slices — so val loss is a fair \"unseen tail of the book\" check.\n"
|
||
],
|
||
"id": "dcf68150"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"...end of train:\", repr(decode(train_data[-40:].tolist())))\n",
|
||
"print(\"start of val: \", repr(decode(val_data[:40].tolist())))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "e9bc8a68"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 3** — What fraction of the book is validation? *(~10%.)*\n"
|
||
],
|
||
"id": "51b55dd8"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"frac_val = len(val_data) / len(data)\n",
|
||
"print(f\"validation fraction: {frac_val:.1%}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "70e5f2ac"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.1 The next-token game (one step)\n",
|
||
"\n",
|
||
"The training question is always the same:\n",
|
||
"\n",
|
||
"> Given what came before → what is the **single** next character?\n",
|
||
"\n",
|
||
"There is no separate labels file. The label is hiding **in the text itself**:\n",
|
||
"the next character in the sequence.\n",
|
||
"\n",
|
||
"**Analogy:** a friend starts a sentence and you blurt out the next word. They\n",
|
||
"tell you if you were right. That's one training step.\n",
|
||
"\n",
|
||
"**→ Training:** One row of the loss compares model scores to **one** correct next id. A batch contains `B × T` such comparisons at once.\n"
|
||
],
|
||
"id": "e7a8f7c0"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# tiny phrase from our vocabulary — watch one step at a time\n",
|
||
"phrase = \"First Citizen\"\n",
|
||
"ids = encode(phrase)\n",
|
||
"print(\"text: \", phrase)\n",
|
||
"print(\"ids: \", ids)\n",
|
||
"print()\n",
|
||
"for i in range(len(ids) - 1):\n",
|
||
" context = decode(ids[: i + 1])\n",
|
||
" nxt = itos[ids[i + 1]]\n",
|
||
" print(f\" seen {context!r:20s} -> predict {nxt!r}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "6adf5e2f"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 4.1** — Change `phrase` to another short string (letters/spaces only).\n",
|
||
"How many training steps does a string of length *L* give? *(Answer: L − 1.)*\n"
|
||
],
|
||
"id": "537d244e"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"phrase = \"To be\" # edit me\n",
|
||
"ids = encode(phrase)\n",
|
||
"steps = len(ids) - 1\n",
|
||
"print(f\"length {len(ids)} -> {steps} next-token questions\")\n",
|
||
"for i in range(steps):\n",
|
||
" print(f\" {decode(ids[:i+1])!r} -> {itos[ids[i+1]]!r}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "49b69d41"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Consolidated walk-through — from text to training (no need to run code)\n",
|
||
"\n",
|
||
"This cell is a **self-contained explanation** of the chain: `input.txt` → `stoi`/`itos` → `x` and `y` → loss → one training step. You can read it top-to-bottom without executing anything below.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**1. The raw file (`input.txt`) looks like:**\n",
|
||
"\n",
|
||
"```\n",
|
||
"First Citizen:\n",
|
||
"Before we proceed any further, hear me speak.\n",
|
||
"```\n",
|
||
"\n",
|
||
"Every single character — letters, spaces ` `, newlines `\\n`, punctuation — is a potential token. The model never sees strings inside its computation; everything becomes integers.\n",
|
||
"\n",
|
||
"**2. The vocabulary (`chars`, `vocab_size`, `stoi`, `itos`)**\n",
|
||
"\n",
|
||
"`chars = sorted(list(set(text)))` gives every distinct character in the file, sorted. `vocab_size` is its length (about 65 for Shakespeare). Then we build two dictionaries:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"stoi = {ch: i for i, ch in enumerate(chars)} # char -> integer id\n",
|
||
"itos = {i: ch for i, ch in enumerate(chars)} # integer id -> char\n",
|
||
"```\n",
|
||
"\n",
|
||
"Sorted order means, for example: `'\\n'` is id 0, `' '` is id 1, `'!'` is id 2, and so on through the uppercase letters, then the lowercase letters, to id 64 (something like `'z'`). The exact ids don't matter conceptually — what matters is that every character has **one** id and every id has **one** character.\n",
|
||
"\n",
|
||
"**3. `encode` and `decode` — the round-trip**\n",
|
||
"\n",
|
||
"```python\n",
|
||
"encode = lambda s: [stoi[c] for c in s]\n",
|
||
"decode = lambda l: \"\".join(itos[i] for i in l)\n",
|
||
"```\n",
|
||
"\n",
|
||
"Concrete example — `\"First Citizen\"` (13 characters):\n",
|
||
"\n",
|
||
"```\n",
|
||
"text: \"First Citizen\"\n",
|
||
"encode → [id_of_F, id_of_i, id_of_r, id_of_s, id_of_t, id_of_space, id_of_C, id_of_i, id_of_t, id_of_i, id_of_z, id_of_e, id_of_n]\n",
|
||
"decode → \"First Citizen\" (round-trip matches)\n",
|
||
"```\n",
|
||
"\n",
|
||
"**4. The next-token game (section 4.1) — one string → many training questions**\n",
|
||
"\n",
|
||
"The model's job is: *given everything to the left, predict the single next character.* There is no separate \"answers file\" — the answer is the **next character in the string itself**. A string of length L contains **L − 1** training questions, because at every position from 0 to L−2 we can ask: what comes next?\n",
|
||
"\n",
|
||
"For `\"First Citizen\"` (13 chars), that is **12 questions**:\n",
|
||
"\n",
|
||
"| context seen (prefix) | must predict |\n",
|
||
"|---|---|\n",
|
||
"| `\"F\"` | `'i'` |\n",
|
||
"| `\"Fi\"` | `'r'` |\n",
|
||
"| `\"Fir\"` | `'s'` |\n",
|
||
"| `\"Firs\"` | `'t'` |\n",
|
||
"| `\"First\"` | `' '` (space) |\n",
|
||
"| `\"First \"` | `'C'` |\n",
|
||
"| `\"First C\"` | `'i'` |\n",
|
||
"| `\"First Ci\"` | `'t'` |\n",
|
||
"| `\"First Cit\"` | `'i'` |\n",
|
||
"| `\"First Citi\"` | `'z'` |\n",
|
||
"| `\"First Citiz\"` | `'e'` |\n",
|
||
"| `\"First Citize\"` | `'n'` |\n",
|
||
"\n",
|
||
"The model never \"sees\" the target character during its forward pass — it only sees the prefix. It produces scores for **every** character in the vocabulary, and loss measures how little probability mass landed on the one true character.\n",
|
||
"\n",
|
||
"**5. From the tape to `x` and `y` (sections 4.3–4.4)**\n",
|
||
"\n",
|
||
"In practice we don't pass in arbitrary short strings. We take the whole training text, convert it to one long 1-D tensor of ids (the \"tape\"), and slice windows of `block_size` (8 here). From a slice we build two equal-length tensors:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"block_size = 8\n",
|
||
"x = train_data[1000 : 1000 + block_size] # 8 ids, the context\n",
|
||
"y = train_data[1001 : 1001 + block_size] # 8 ids, the answers (shifted by +1)\n",
|
||
"```\n",
|
||
"\n",
|
||
"If at that position on the tape the text reads `\"Before we proceed...\"`, then:\n",
|
||
"\n",
|
||
"* `x` = ids for `\"Before w\"` (8 chars)\n",
|
||
"* `y` = ids for `\"efore we\"` (8 chars, same positions shifted right by one)\n",
|
||
"\n",
|
||
"Written as a table with growing prefixes:\n",
|
||
"\n",
|
||
"| position `t` | prefix the model sees (from `x`) | must predict (from `y[t]`) |\n",
|
||
"|---|---|---|\n",
|
||
"| 0 | `\"B\"` | `'e'` |\n",
|
||
"| 1 | `\"Be\"` | `'f'` |\n",
|
||
"| 2 | `\"Bef\"` | `'o'` |\n",
|
||
"| 3 | `\"Befo\"` | `'r'` |\n",
|
||
"| 4 | `\"Befor\"` | `'e'` |\n",
|
||
"| 5 | `\"Before\"` | `' '` (space) |\n",
|
||
"| 6 | `\"Before \"` | `'w'` |\n",
|
||
"| 7 | `\"Before w\"` | `'e'` |\n",
|
||
"\n",
|
||
"That is **8 supervised questions** from one 8-character row, processed in a single forward pass.\n",
|
||
"\n",
|
||
"**6. What `logits` and `loss` actually look like at one position**\n",
|
||
"\n",
|
||
"At each position `t` the model emits `vocab_size` raw scores (`logits`) — one score per possible character in the vocabulary. At `t=1` (context `\"Be\"`, true next char `'f'`), the picture is:\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=1 for all 65 chars:\n",
|
||
" 'a' logit = -0.3\n",
|
||
" 'c' logit = +2.1 (model's strongest guess — WRONG)\n",
|
||
" 'e' logit = +1.5\n",
|
||
" 'f' logit = +0.8 (TRUE answer y[1] = 'f')\n",
|
||
" 'z' logit = -2.0\n",
|
||
" ... (the other 61 chars)\n",
|
||
"\n",
|
||
"after softmax (squashed into a probability distribution):\n",
|
||
" P('c') ≈ 0.52 model wastes over half its confidence on 'c'\n",
|
||
" P('e') ≈ 0.28\n",
|
||
" P('f') ≈ 0.14 only 14% on the true character\n",
|
||
" ... (the rest share what's left)\n",
|
||
"\n",
|
||
"loss at t=1 = -ln(P('f')) = -ln(0.14) ≈ 1.97\n",
|
||
"```\n",
|
||
"\n",
|
||
"Cross-entropy asks **one question** per position: *how much probability mass landed on the single correct character?* It never compares strings, and it doesn't care which *wrong* characters were ranked highly — only how far `P(true_char)` is from 1.0.\n",
|
||
"\n",
|
||
"**7. Averaging over the row → one `loss` number**\n",
|
||
"\n",
|
||
"Loss is averaged across all 8 positions (and across all rows in a batch):\n",
|
||
"\n",
|
||
"```\n",
|
||
"loss = ( loss(\"B\"→'e') + loss(\"Be\"→'f') + loss(\"Bef\"→'o') + ... + loss(\"Before w\"→'e') ) / 8\n",
|
||
"```\n",
|
||
"\n",
|
||
"Then `loss.backward()` computes, for every weight in the model, which direction to nudge it so loss would be slightly smaller. `optimizer.step()` applies that nudge. Repeat this loop thousands of times and the model gradually learns which characters tend to follow which in Shakespearean English.\n",
|
||
"\n",
|
||
"**8. Generation (the \"spit out words\" loop)**\n",
|
||
"\n",
|
||
"After training, generation works exactly the same way as the forward pass, but in a loop:\n",
|
||
"\n",
|
||
"```\n",
|
||
"start: [newline_id] # one id, representing \"\\n\"\n",
|
||
" |\n",
|
||
" v\n",
|
||
"model(idx) → logits at the *last* position only\n",
|
||
" |\n",
|
||
" v\n",
|
||
"softmax(logits) → probability distribution over all 65 characters\n",
|
||
" |\n",
|
||
" v\n",
|
||
"sample one id (torch.multinomial) — not always the most-probable one\n",
|
||
" |\n",
|
||
" v\n",
|
||
"append that id to idx (the sequence grows by one)\n",
|
||
" |\n",
|
||
" v\n",
|
||
"repeat (loop back) with the longer sequence\n",
|
||
"```\n",
|
||
"\n",
|
||
"Finally, `decode(sequence_of_ids)` converts the produced list of integers back to a readable string. There is **no stored English, no facts, no rules** inside the model — only floating-point weights encoding co-occurrence statistics.\n",
|
||
"\n",
|
||
"**9. Shape summary (carry this mental picture into section 5 and beyond)**\n",
|
||
"\n",
|
||
"| object | shape | meaning |\n",
|
||
"|---|---|---|\n",
|
||
" `data` | `(N,)` | the whole book, one id per character (a 1-D tensor) |\n",
|
||
" `xb` | `(B, T)` | batch of context windows — B rows, T = block_size columns of ids |\n",
|
||
" `yb` | `(B, T)` | batch of answer windows — same shape as xb, shifted by +1 on the tape |\n",
|
||
" `logits` | `(B, T, vocab_size)` | one score per character, per position, per row |\n",
|
||
" `loss` | `(1,)` — a scalar | one number measuring how well the model predicts `yb` from `xb` |\n",
|
||
" a generated sequence | `(1, T_generated)` | one row, one id per generated character |\n",
|
||
"\n",
|
||
"That is the entire training-and-generation pipeline. Every code cell below is a step that makes this pipeline either more visible (printing shapes, inspecting slices) or more powerful (building a better model than the bigram baseline)."
|
||
],
|
||
"id": "consolidation-4x"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"train_data length:\", len(train_data))\n",
|
||
"print(\"first 15 ids: \", train_data[:15].tolist())\n",
|
||
"print(\"first 15 chars: \", decode(train_data[:15].tolist()))\n",
|
||
"print(\"chars 15-30: \", decode(train_data[15:30].tolist()))\n",
|
||
"print()\n",
|
||
"print(\"The tape is continuous — char 15 follows char 14 in the real book.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "d217e75d"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.3 Why we use chunks (`block_size`)\n",
|
||
"\n",
|
||
"We could slide one character at a time, but in practice we grab a **window** of\n",
|
||
"`block_size` characters at once. Three reasons:\n",
|
||
"\n",
|
||
"1. **Memory** — models store state per position; a 1M-wide window would not fit on a GPU.\n",
|
||
"2. **Fixed context** — Transformers are built to look at at most `block_size` tokens.\n",
|
||
"3. **Efficiency** — one forward pass on 8 positions trains 8 questions in parallel (next part).\n",
|
||
"\n",
|
||
"`block_size` is the **context window**: how far back the model is *allowed* to look.\n",
|
||
"We use 8 here so the numbers fit on screen; GPT-4 uses thousands.\n",
|
||
"\n",
|
||
"**→ Training:** `block_size` = how many next-token questions per row (`T`). Also caps context length the architecture can use.\n"
|
||
],
|
||
"id": "0ecf904b"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"block_size = 8\n",
|
||
"start = 1000 # where we cut the tape\n",
|
||
"chunk = train_data[start : start + block_size]\n",
|
||
"\n",
|
||
"print(f\"block_size = {block_size}\")\n",
|
||
"print(f\"slice from index {start} to {start + block_size - 1}\")\n",
|
||
"print(\"chunk ids: \", chunk.tolist())\n",
|
||
"print(\"chunk text: \", repr(decode(chunk.tolist())))\n",
|
||
"print()\n",
|
||
"print(\"book length:\", len(train_data), \" window:\", block_size,\n",
|
||
" \" ratio:\", round(len(train_data) / block_size))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "7b330af9"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 4.3** — Change `start`. The chunk should always be exactly `block_size`\n",
|
||
"characters (except if you pick a start too close to the end).\n"
|
||
],
|
||
"id": "f19d6e14"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"start = 25000 # try different values\n",
|
||
"chunk = train_data[start : start + block_size]\n",
|
||
"print(\"len(chunk):\", len(chunk), \" text:\", repr(decode(chunk.tolist())))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "8528eb17"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.4 Inputs `x` and targets `y` (shift by one)\n",
|
||
"\n",
|
||
"From one slice we build two equal-length tensors:\n",
|
||
"\n",
|
||
"- **`x`** = the window `train_data[start : start + block_size]`\n",
|
||
"- **`y`** = the same window, shifted **one step forward** in time:\n",
|
||
" `train_data[start + 1 : start + block_size + 1]`\n",
|
||
"\n",
|
||
"So `y` is always \"what comes next\" relative to `x`, position by position:\n",
|
||
"\n",
|
||
"```\n",
|
||
"x: h e l l o _ t h\n",
|
||
"y: e l l o _ t h ? ← ? is the 9th char (first char AFTER the window)\n",
|
||
"```\n",
|
||
"\n",
|
||
"The model never sees `y` during the forward pass — only during loss calculation.\n",
|
||
"\n",
|
||
"#### Tie-in to the training goal\n",
|
||
"\n",
|
||
"**Ultimate goal:** nudge **weights** so the model scores the true next character highly.\n",
|
||
"\n",
|
||
"`x` and `y` are how we package **one practice round** for that goal:\n",
|
||
"\n",
|
||
"| Tensor | Role in training | Analogy |\n",
|
||
"|--------|------------------|---------|\n",
|
||
"| **`x`** (later `xb`) | **Question** — tokens the model reads to produce logits | exam paper |\n",
|
||
"| **`y`** (later `yb`) | **Answer key** — the correct next token at each position | mark scheme |\n",
|
||
"\n",
|
||
"At column `t`: model reads `x[t]`, predicts a distribution; loss compares it to **`y[t]`** (the char that really followed on the tape). Wrong guess → higher loss → `backward()` pushes weights to do better next time.\n",
|
||
"\n",
|
||
"**In the chain (why this section exists):**\n",
|
||
"```\n",
|
||
"GOAL: better weights\n",
|
||
" ^ needs backward()\n",
|
||
" ^ needs loss (one number)\n",
|
||
" ^ needs yb — the CORRECT next tokens ← y is the template for yb\n",
|
||
" ^ needs forward on xb — model predictions ← x is the template for xb\n",
|
||
"```\n",
|
||
"\n",
|
||
"Without the shift-by-one split you have text but **no labelled examples** — nothing to score, no loss, no training.\n",
|
||
"\n",
|
||
"**→ Training:** `get_batch` builds matrices `xb`/`yb` the same way as `x`/`y` here. Every training step is: predict from `xb`, grade against `yb`, update weights.\n",
|
||
"\n"
|
||
],
|
||
"id": "342c105f"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"start = 1000\n",
|
||
"x = train_data[start : start + block_size]\n",
|
||
"y = train_data[start + 1 : start + block_size + 1]\n",
|
||
"\n",
|
||
"print(\"x:\", decode(x.tolist()))\n",
|
||
"print(\"y:\", decode(y.tolist()))\n",
|
||
"print()\n",
|
||
"for i in range(block_size):\n",
|
||
" print(f\" x[{i}]={repr(itos[x[i].item()])} y[{i}]={repr(itos[y[i].item()])}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "0dec560b"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 4.4** — Confirm `y[i]` is always the id **right after** `x[i]` on the tape.\n"
|
||
],
|
||
"id": "70613e82"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"ok = all(y[t].item() == train_data[start + t + 1].item() for t in range(block_size))\n",
|
||
"print(\"every y[t] is the next char after x[t] on the tape:\", ok)\n",
|
||
"if not ok:\n",
|
||
" for t in range(block_size):\n",
|
||
" print(t, x[t].item(), y[t].item(), train_data[start + t + 1].item())\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "5be4764d"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.5 The prefix trick (one chunk → many lessons)\n",
|
||
"\n",
|
||
"Here is the key idea that confuses many beginners:\n",
|
||
"\n",
|
||
"An 8-character chunk is **not** one training example. It is **eight** examples\n",
|
||
"packed together — one for each prefix length.\n",
|
||
"\n",
|
||
"| step `t` | context (prefix of `x`) | must predict `y[t]` |\n",
|
||
"|----------|---------------------------|---------------------|\n",
|
||
"| 0 | 1st char only | 2nd char |\n",
|
||
"| 1 | 1st + 2nd | 3rd char |\n",
|
||
"| 2 | first 3 chars | 4th char |\n",
|
||
"| … | … | … |\n",
|
||
"| 7 | all 8 chars | 9th char (just past the window) |\n",
|
||
"\n",
|
||
"**Analogy:** teaching spelling by asking \"after **h**?\", then \"after **he**?\",\n",
|
||
"then \"after **hel**?\" — same word, harder each time.\n",
|
||
"\n",
|
||
"**→ Training:** One slice yields `block_size` supervised examples per forward pass — more learning signal per batch.\n"
|
||
],
|
||
"id": "6fecc0fb"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.5b Worked example — `x='Before w'`, `y='efore we'`\n",
|
||
"\n",
|
||
"The row above `(x='Before w', y='efore we')` is **not** one training example. It is\n",
|
||
"**8** training examples packed together — one for each growing prefix of `x`. Here\n",
|
||
"they are, written out:\n",
|
||
"\n",
|
||
"| Position | Prefix the model sees (from `x`) | Must predict (from `y`) |\n",
|
||
"|----------|-----------------------------------|--------------------------|\n",
|
||
"| `t=0` | `\"B\"` | `'e'` |\n",
|
||
"| `t=1` | `\"Be\"` | `'f'` |\n",
|
||
"| `t=2` | `\"Bef\"` | `'o'` |\n",
|
||
"| `t=3` | `\"Befo\"` | `'r'` |\n",
|
||
"| `t=4` | `\"Befor\"` | `'e'` |\n",
|
||
"| `t=5` | `\"Before\"` | `' '` (space) |\n",
|
||
"| `t=6` | `\"Before \"` | `'w'` |\n",
|
||
"| `t=7` | `\"Before w\"` | `'e'` |\n",
|
||
"\n",
|
||
"Important: `y[t]` at every position is **one single character**, not a string.\n",
|
||
"The fact that `y` reads like a word (`'efore we'`) is a human reading artifact —\n",
|
||
"to the training loop it is an array of 8 independent integer ids.\n"
|
||
],
|
||
"id": "4cb6a5d2"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### What the model outputs per position, and what cross-entropy *actually* compares\n",
|
||
"\n",
|
||
"At **each** position `t` the model emits `vocab_size` raw logits — one score per\n",
|
||
"possible character. `softmax(logits)` then squashes those 65 numbers into a proper\n",
|
||
"probability distribution (all positive, sums to 1.0).\n",
|
||
"\n",
|
||
"Cross-entropy then asks **one question per position**:\n",
|
||
"\n",
|
||
"> *How much of the 1.0 probability budget landed on the ONE correct character* `y[t]`?\n",
|
||
"\n",
|
||
"It does **not** compare strings, it does **not** compare predictions to whole chunks\n",
|
||
"of text, and it does **not** care which other characters were ranked high or low —\n",
|
||
"it only reads off `P(true_char)` at that position and computes `-ln(P(true_char))`.\n",
|
||
"\n",
|
||
"Concrete scenario at `t=1` (context = `\"Be\"`, true next char = `'f'`):\n",
|
||
"\n",
|
||
"```\n",
|
||
"model's top-3 by logit at t=1:\n",
|
||
" 'c' logit = +2.1 (model's highest guess, WRONG)\n",
|
||
" 'e' logit = +1.5\n",
|
||
" 'f' logit = +0.8 (TRUE answer y[1] = 'f')\n",
|
||
" ...\n",
|
||
"\n",
|
||
"after softmax:\n",
|
||
" P('c') ~ 0.52 model wasted over half its confidence on 'c'\n",
|
||
" P('e') ~ 0.28\n",
|
||
" P('f') ~ 0.14 only 14% on the true character\n",
|
||
" ...\n",
|
||
"\n",
|
||
"loss at t=1 = -ln(0.14) = 1.97\n",
|
||
"```\n",
|
||
"\n",
|
||
"Whether the model's *favorite* was `'c'` or `'q'` or `'Z'` doesn't matter for the\n",
|
||
"math — only the probability mass on `'f'` does. The other 64 characters jointly\n",
|
||
"soak up probability away from the true character, and that is what makes loss large.\n"
|
||
],
|
||
"id": "81710b12"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### What softmax does — the pie metaphor\n",
|
||
"\n",
|
||
"Think of softmax as splitting **one pie** (100% of your confidence) across all 65\n",
|
||
"possible next characters.\n",
|
||
"\n",
|
||
"- Each character starts with a **raw score** (its logit). A higher score means \"I think this letter is more likely.\"\n",
|
||
"- Softmax turns those scores into **slice sizes** (probabilities) that are always positive and **add up to exactly 1.0** — the whole pie is always eaten, nothing left over.\n",
|
||
"\n",
|
||
"If one character's logit is much bigger than the rest, softmax gives it a big slice.\n",
|
||
"If every logit is about the same, the pie is cut into roughly equal slices.\n",
|
||
"\n",
|
||
"**The math (two steps, applied to all 65 logits at once):**\n",
|
||
"\n",
|
||
"1. **Exponentiate** — raise *e* to the power of each logit: `exp(z_i)`. This makes every value positive and amplifies differences (a logit of +2.1 becomes much larger than one of +0.8).\n",
|
||
"2. **Normalise** — divide each result by the **sum** of all 65 exponentiated values:\n",
|
||
"\n",
|
||
"```\n",
|
||
"P(character i) = exp(logit_i) / sum over all j of exp(logit_j)\n",
|
||
"```\n",
|
||
"\n",
|
||
"So softmax is not one exotic trick — it is **exp**, then **divide by the total**.\n",
|
||
"That division is what forces the 65 probabilities to sum to 1.0.\n",
|
||
"\n",
|
||
"**In this notebook (programming language):** the notebook is written in **Python**.\n",
|
||
"Softmax is called through **PyTorch** — either `torch.softmax(logits, dim=-1)` or\n",
|
||
"`F.softmax(logits, dim=-1)` (same operation; `F` is `torch.nn.functional`). The\n",
|
||
"`dim=-1` argument means \"run softmax along the last axis\" — i.e. across the 65\n",
|
||
"character scores at each position, independently.\n"
|
||
],
|
||
"id": "softmax-pie-metaphor"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### From Python to machine code (summary)\n",
|
||
"\n",
|
||
"You never implement softmax by hand in a Python loop. You call one function; layers\n",
|
||
"below your notebook do the heavy work.\n",
|
||
"\n",
|
||
"| Layer | What runs it | What it does |\n",
|
||
"| --- | --- | --- |\n",
|
||
"| Your notebook | Python | Calls `torch.softmax(logits, dim=-1)` |\n",
|
||
"| PyTorch engine | C++ (and CUDA on GPU) | Receives the tensor and picks a fast kernel |\n",
|
||
"| Math libraries | Vectorised C / CUDA | `exp` on every logit, sum them, divide each by the total |\n",
|
||
"| Machine code | x86 / ARM (CPU) or GPU SASS (NVIDIA) | Load floats from memory, multiply/add/divide, write probabilities back |\n",
|
||
"\n",
|
||
"**What the chip actually does** for one position's 65 logits: load 65 numbers,\n",
|
||
"exponentiate each, add them up, divide each exponential by that sum, store 65\n",
|
||
"probabilities. On a GPU, many positions run in parallel.\n",
|
||
"\n",
|
||
"**Take-home:** the *idea* is the pie metaphor (scores -> slices summing to 1.0). The\n",
|
||
"*implementation* is compiled numeric code — not Python — optimised for speed on your\n",
|
||
"CPU or GPU.\n"
|
||
],
|
||
"id": "softmax-machine-code"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Verify the claim above on the exact row: x='Before w', y='efore we'.\n",
|
||
"# We run ONE forward pass on this row and print, for every position t:\n",
|
||
"# (a) the logit of the true char, (b) its softmax prob, (c) -ln(prob).\n",
|
||
"# Then we compare mean(-ln(prob)) to F.cross_entropy to confirm they match.\n",
|
||
"\n",
|
||
"import math\n",
|
||
"\n",
|
||
"x_str = \"Before w\"\n",
|
||
"y_str = \"efore we\"\n",
|
||
"\n",
|
||
"x_ids = torch.tensor([stoi[c] for c in x_str], dtype=torch.long, device=device).unsqueeze(0)\n",
|
||
"y_ids = torch.tensor([stoi[c] for c in y_str], dtype=torch.long, device=device).unsqueeze(0)\n",
|
||
"\n",
|
||
"with torch.no_grad():\n",
|
||
" logits, _ = model(x_ids) # (1, 8, vocab_size)\n",
|
||
"\n",
|
||
"probs = torch.softmax(logits, dim=-1) # (1, 8, vocab_size)\n",
|
||
"\n",
|
||
"print(f\"{'t':>2} {'prefix':>12s} {'true':>5s} logit(true) prob(true) -ln(p)\")\n",
|
||
"print(\"-\" * 78)\n",
|
||
"\n",
|
||
"per_t_loss = []\n",
|
||
"for t in range(8):\n",
|
||
" ctx = x_str[: t + 1]\n",
|
||
" true_ch = y_str[t]\n",
|
||
" true_id = stoi[true_ch]\n",
|
||
" lg = logits[0, t, true_id].item()\n",
|
||
" p = probs[0, t, true_id].item()\n",
|
||
" per_t_loss.append(-math.log(p) if p > 0 else float(\"inf\"))\n",
|
||
" print(f\"{t:>2} {ctx!r:>14s} {true_ch!r:>5s} {lg:+11.4f} {p:11.6f} {per_t_loss[-1]:7.4f}\")\n",
|
||
"\n",
|
||
"manual_mean = sum(per_t_loss) / len(per_t_loss)\n",
|
||
"pytorch_ce = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n",
|
||
"\n",
|
||
"print()\n",
|
||
"print(f\"mean -ln(p) across 8 positions (hand) = {manual_mean:.6f}\")\n",
|
||
"print(f\"F.cross_entropy(logits, y_ids) = {pytorch_ce:.6f}\")\n",
|
||
"print(f\"match: {abs(manual_mean - pytorch_ce) < 1e-5}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "8246b96a"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### How the 8 per-position losses combine into one training signal\n",
|
||
"\n",
|
||
"The 8 per-position penalties are **averaged** into a single scalar:\n",
|
||
"\n",
|
||
"```\n",
|
||
"loss = ( loss(\"B\" → 'e') + loss(\"Be\" → 'f') + loss(\"Bef\" → 'o') + ... + loss(\"Before w\" → 'e') ) / 8\n",
|
||
"```\n",
|
||
"\n",
|
||
"Then `loss.backward()` walks *in reverse* from this one scalar through the\n",
|
||
"computation graph. Every weight in the embedding table receives a gradient that\n",
|
||
"says: *\"nudge yourself slightly so the true character at every one of these 8\n",
|
||
"positions gets a little more probability mass.\"* `optimizer.step()` then applies\n",
|
||
"that nudge.\n",
|
||
"\n",
|
||
"**A useful mental picture:** think of the 8 columns of the row as 8 separate\n",
|
||
"flashcards sharing the same model. Each flashcard says \"given this prefix, score\n",
|
||
"the correct next char highly\". The model sees all 8 simultaneously, and the loss\n",
|
||
"is the average of how it did on each. The gradient nudges the shared weights in\n",
|
||
"the direction that helps *all 8* — not just one of them.\n"
|
||
],
|
||
"id": "3fdb163c"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### What logits actually look like across all 8 positions — and why they are NOT random\n",
|
||
"\n",
|
||
"The notebook shows logits at `t=1` only. Here we walk through what logits look like at **every** position `t=0` through `t=7` on the same row `(x='Before w', y='efore we')`, and explain why each one follows a recognisable pattern.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**Important reminder on how the bigram model works.**\n",
|
||
"The bigram model has a single learnable weight table of shape `(vocab_size, vocab_size)`. For each input character id `idx`, the model looks up the corresponding row of that table — that row **is** the `vocab_size` logits for the next character.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at position t = token_embedding_table[ x[t] ]\n",
|
||
" ^^^ the row for the character at position t\n",
|
||
"```\n",
|
||
"\n",
|
||
"This means: **logits depend only on the single character at position `t`**, not on the whole prefix. That is the *bigram assumption*. (A transformer later removes this assumption — it lets logits depend on the full prefix.)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**At every position, the model emits 65 numbers. Here is what they look like and why.**\n",
|
||
"\n",
|
||
"Below we write logits for a handful of the most interesting characters at each of the 8 positions. These are **illustrative numbers** in the style of what an untrained or barely-trained bigram model would produce — you can get exact numbers by running the verification cell below, but the *shape and pattern* is what matters.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=0. Context seen by model: `'B'`. Must predict `'e'`.**\n",
|
||
"\n",
|
||
"The model has seen only the character `'B'`. It looks up one row of the embedding table — the row for `'B'` — and that row becomes its 65 logits. In English text, `'B'` is most commonly followed by `'e'`, `'a'`, `'o'`, `'u'`, `'y'`, or `'r'` (think *But, By, Be, Before, Boy, Bring*). After training, those characters should have the highest logits in this row.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=0 (row for 'B' in the weight table):\n",
|
||
" 'e' logit = +1.8 (TRUE answer y[0] = 'e')\n",
|
||
" 'a' logit = +1.2\n",
|
||
" 'o' logit = +0.9\n",
|
||
" 'u' logit = +0.6\n",
|
||
" 'r' logit = +0.4\n",
|
||
" 'i' logit = +0.2\n",
|
||
" 'B' logit = -0.3 ('B' followed by another 'B' is rare)\n",
|
||
" ' ' logit = -0.8 ('B' followed by space is rare — 'B' is capital, usually starts a word)\n",
|
||
" 'z' logit = -1.7 ('Bz' essentially never occurs)\n",
|
||
" ... (the other 57 characters, mostly negative or near-zero)\n",
|
||
"\n",
|
||
"softmax at t=0:\n",
|
||
" P('e') ≈ 0.28 correct next character gets ~28% of the budget\n",
|
||
" P('a') ≈ 0.15\n",
|
||
" P('o') ≈ 0.11\n",
|
||
" ... the remaining 62 chars share the rest\n",
|
||
"\n",
|
||
"loss at t=0 = -ln(0.28) ≈ 1.27\n",
|
||
"```\n",
|
||
"\n",
|
||
"Why this pattern? Because `token_embedding_table['B']` has been nudged up and down during training: every time `'B'` appears in the training text followed by `'e'`, the weight for `'B'→'e'` gets a tiny positive gradient. Every time `'B'` is followed by something *other* than `'e'`, the weight for `'B'→'e'` gets a tiny negative gradient (or rather, the correct-next-char weight is pushed up and *all* the wrong ones are pushed down a little by softmax's normalising effect). Over thousands of examples the row for `'B'` reflects the true *next-letter frequency* of `'B'` in Shakespeare.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=1. Context seen by model: `'Be'`. Must predict `'f'`.**\n",
|
||
"\n",
|
||
"The bigram model sees *only* `'e'` (the character at index 1), not the full `'Be'` prefix. It looks up the row for `'e'`. In English, `'e'` is often followed by `' '` (space, ending a word), `'r'`, `'s'`, `'t'`, `'d'`, or vowels. In the word `Before`, however, `'e'` is followed by `'f'` — which is not the most common successor of `'e'` in general.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=1 (row for 'e'):\n",
|
||
" ' ' logit = +2.4 (most common — 'e' often ends a word)\n",
|
||
" 'r' logit = +1.7\n",
|
||
" 's' logit = +1.3\n",
|
||
" 'a' logit = +0.9\n",
|
||
" 't' logit = +0.7\n",
|
||
" 'd' logit = +0.5\n",
|
||
" 'f' logit = +0.4 (TRUE answer y[1] = 'f')\n",
|
||
" 'n' logit = +0.3\n",
|
||
" 'c' logit = +0.1\n",
|
||
" ... (most others negative)\n",
|
||
"\n",
|
||
"softmax at t=1:\n",
|
||
" P(' ') ≈ 0.34 model strongly guesses a space here\n",
|
||
" P('r') ≈ 0.16\n",
|
||
" P('f') ≈ 0.09 only 9% on the true character\n",
|
||
"\n",
|
||
"loss at t=1 = -ln(0.09) ≈ 2.41\n",
|
||
"```\n",
|
||
"\n",
|
||
"Loss is *higher* here than at t=0 because the model's strongest guess (`' '` or `'r'`) is wrong. The bigram model cannot use the `'B'` before `'e'` to help — it only knows `'e'` was just seen, and `'e'→'f'` is rare in general. This is exactly why bigram models are weak: a bigram loses all context beyond one step back.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=2. Context seen by model: `'Bef'`. Must predict `'o'`.**\n",
|
||
"\n",
|
||
"Row `'f'` of the embedding table. In English, `'f'` is followed overwhelmingly by `'o'` (*for, from, of, often, before*), `'r'` (*from, free*), `'e'`, or `'t'` (*first, after*).\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=2 (row for 'f'):\n",
|
||
" 'o' logit = +2.1 (TRUE answer y[2] = 'o')\n",
|
||
" 'r' logit = +1.4\n",
|
||
" 't' logit = +0.8\n",
|
||
" 'e' logit = +0.6\n",
|
||
" 'f' logit = -0.2 ('ff' is uncommon at word-internal positions in English)\n",
|
||
" ...\n",
|
||
"\n",
|
||
"softmax at t=2:\n",
|
||
" P('o') ≈ 0.31 correct next character gets ~31%\n",
|
||
"\n",
|
||
"loss at t=2 = -ln(0.31) ≈ 1.17\n",
|
||
"```\n",
|
||
"\n",
|
||
"This is a relatively **low loss position** — `'f'→'o'` is genuinely common and the model has learned it.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=3. Context seen by model: `'Befo'`. Must predict `'r'`.**\n",
|
||
"\n",
|
||
"Row `'o'` of the embedding table. `'o'` has many common successors: `'u'`, `'r'`, `'n'`, `'t'`, `'f'`, `'m'`, `'s'`, and space.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=3 (row for 'o'):\n",
|
||
" 'u' logit = +1.8 ('you', 'would', 'could', 'out', 'over')\n",
|
||
" 'r' logit = +1.5 (TRUE answer y[3] = 'r')\n",
|
||
" 'n' logit = +1.3\n",
|
||
" 't' logit = +1.1\n",
|
||
" 'f' logit = +0.7\n",
|
||
" ...\n",
|
||
" P('r') ≈ 0.18 (good but not dominant — 'o' has many common follow-ups)\n",
|
||
"\n",
|
||
"loss at t=3 = -ln(0.18) ≈ 1.71\n",
|
||
"```\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=4. Context seen by model: `'Befor'`. Must predict `'e'`.**\n",
|
||
"\n",
|
||
"Row `'r'` of the embedding table. `'r'` is often followed by `'e'` (*there, were, are, before, more*), `'s'`, `'t'`, `'d'`, or space.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=4 (row for 'r'):\n",
|
||
" 'e' logit = +2.2 (TRUE answer y[4] = 'e')\n",
|
||
" 's' logit = +1.3\n",
|
||
" 't' logit = +0.9\n",
|
||
" ' ' logit = +0.8 ('r' often ends a word: 'or', 'for', 'her', 'our')\n",
|
||
" ...\n",
|
||
" P('e') ≈ 0.36 strong — 'r'→'e' is very common\n",
|
||
"\n",
|
||
"loss at t=4 = -ln(0.36) ≈ 1.02\n",
|
||
"```\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=5. Context seen by model: `'Before'`. Must predict `' '` (space).**\n",
|
||
"\n",
|
||
"Row `'e'` of the embedding table, again — same row as at `t=1`. Logits at t=5 and logits at t=1 are **identical** in a bigram model. This is an important point.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=5 (row for 'e') ← identical numbers to t=1 (same row)\n",
|
||
" ' ' logit = +2.4 (TRUE answer y[5] = ' ')\n",
|
||
" 'r' logit = +1.7\n",
|
||
" 's' logit = +1.3\n",
|
||
" ...\n",
|
||
" P(' ') ≈ 0.34 correct!\n",
|
||
"\n",
|
||
"loss at t=5 = -ln(0.34) ≈ 1.08\n",
|
||
"```\n",
|
||
"\n",
|
||
"Notice: at t=1 the model *missed* (`'e'→'f'`), but at t=5 the model *hit* (`'e'→' '`). Both come from the **same row** `token_embedding_table['e']`. One row is asked to predict many different successors in different contexts, and its gradient is the average over all of them.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=6. Context seen by model: `'Before '`. Must predict `'w'`.**\n",
|
||
"\n",
|
||
"Row for `' '` (space). This is the single most important row in the whole vocabulary — every word after the first starts with a space, so `' '→?` is essentially asking: *what character starts a new word?* In English, the answer is dominated by the frequent word-starting letters: `'t'` (*the, to, that*), `'o'` (*of, or*), `'a'` (*and, are, at*), `'i'` (*in, is, it*), `'w'` (*we, what, would, will, with*), `'h'` (*he, his, her*), `'b'` (*be, but, by*), `'s'` (*so, shall, she*), and capital letters for sentence starts.\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=6 (row for ' '):\n",
|
||
" 't' logit = +2.3\n",
|
||
" 'a' logit = +1.8\n",
|
||
" 'o' logit = +1.6\n",
|
||
" 'i' logit = +1.4\n",
|
||
" 'w' logit = +1.2 (TRUE answer y[6] = 'w')\n",
|
||
" 'h' logit = +1.0\n",
|
||
" 'b' logit = +0.9\n",
|
||
" 's' logit = +0.7\n",
|
||
" ... (rare letters are negative)\n",
|
||
" 'z' logit = -1.8\n",
|
||
" 'q' logit = -2.0\n",
|
||
" ...\n",
|
||
" P('w') ≈ 0.11\n",
|
||
"\n",
|
||
"loss at t=6 = -ln(0.11) ≈ 2.21\n",
|
||
"```\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**t=7. Context seen by model: `'Before w'`. Must predict `'e'`.**\n",
|
||
"\n",
|
||
"The bigram model sees only `'w'` (the character at index 7), not the full `'Before w'` prefix. It looks up the row for `'w'` in the embedding table. In English, `'w'` is overwhelmingly followed by `'e'` (*we, were, well, when, what, would, will, with*), `'h'` (*where, why, when, whole*), `'i'` (*will, with, which*), `'o'` (*would, word*), `'a'` (*was, want*).\n",
|
||
"\n",
|
||
"```\n",
|
||
"logits at t=7 (row for 'w'):\n",
|
||
" 'e' logit = +2.5 (TRUE answer y[7] = 'e')\n",
|
||
" 'h' logit = +1.8\n",
|
||
" 'i' logit = +1.5\n",
|
||
" 'o' logit = +1.2\n",
|
||
" 'a' logit = +1.0\n",
|
||
" 'w' logit = -0.1 ('ww' does not occur in English)\n",
|
||
" ... (the other 59 characters, mostly negative or near-zero)\n",
|
||
"\n",
|
||
"softmax at t=7:\n",
|
||
" P('e') ≈ 0.40 correct next character gets ~40% of the budget\n",
|
||
" P('h') ≈ 0.20\n",
|
||
" P('i') ≈ 0.15\n",
|
||
" ... the remaining 62 chars share the rest\n",
|
||
"\n",
|
||
"loss at t=7 = -ln(0.40) ≈ 0.92\n",
|
||
"```\n",
|
||
"\n",
|
||
"This is the model's strongest position on the row — `'w'→'e'` is very common in Shakespeare.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**Pulling it together: the average loss over all 8 positions.**\n",
|
||
"\n",
|
||
"Here is what the model does on this one row, end to end:\n",
|
||
"\n",
|
||
"**Part A — what you see vs what the bigram uses**\n",
|
||
"\n",
|
||
"| t | prefix (you read) | target | row used |\n",
|
||
"| --- | --- | --- | --- |\n",
|
||
"| 0 | B | e | B |\n",
|
||
"| 1 | Be | f | e |\n",
|
||
"| 2 | Bef | o | f |\n",
|
||
"| 3 | Befo | r | o |\n",
|
||
"| 4 | Befor | e | r |\n",
|
||
"| 5 | Before | space | e |\n",
|
||
"| 6 | Before + space | w | space |\n",
|
||
"| 7 | Before w | e | w |\n",
|
||
"\n",
|
||
"The **row used** column is the only input the bigram model cares about — always the single character x[t], not the full prefix.\n",
|
||
"\n",
|
||
"**Part B — scores and loss at each step**\n",
|
||
"\n",
|
||
"| t | logit (true char) | P(true) | loss |\n",
|
||
"| --- | --- | --- | --- |\n",
|
||
"| 0 | +1.8 | 0.28 | 1.27 |\n",
|
||
"| 1 | +0.4 | 0.09 | 2.41 |\n",
|
||
"| 2 | +2.1 | 0.31 | 1.17 |\n",
|
||
"| 3 | +1.5 | 0.18 | 1.71 |\n",
|
||
"| 4 | +2.2 | 0.36 | 1.02 |\n",
|
||
"| 5 | +2.4 | 0.34 | 1.08 |\n",
|
||
"| 6 | +1.2 | 0.11 | 2.21 |\n",
|
||
"| 7 | +2.5 | 0.40 | 0.92 |\n",
|
||
"\n",
|
||
"Averaging the 8 losses: about `(1.27 + 2.41 + 1.17 + 1.71 + 1.02 + 1.08 + 2.21 + 0.92) / 8 ≈ 1.47`. This is one `loss` number, and it drives one `backward()` pass — nudging every row of the embedding table that was looked up (the rows for `'B'`, `'e'`, `'f'`, `'o'`, `'r'`, `' '`, `'w'`) to do slightly better next time.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**Are logits random? No — they are *determined* by the weights.**\n",
|
||
"\n",
|
||
"Logits are **never random**. They are a pure function of two things:\n",
|
||
"\n",
|
||
"1. **The current input character(s).** `logits at position t = some_function(x[t])`. For the bigram model, `some_function` is just a table lookup: `token_embedding_table[x[t]]`. For a transformer, `some_function` is a neural network that reads the entire prefix `x[0..t]`. Either way, given the same input and the same weights, the same logits always come out.\n",
|
||
"\n",
|
||
"2. **The model weights.** The weights are what change during training. At the very first forward pass (before any training), the weights are initialised to small random-ish values — logits at that point *look* noisy and nearly uniform. After even a few thousand training steps, the weights have absorbed bigram statistics from the training text: `'f'→'o'` gets a positive weight, `'q'→'x'` gets a strongly negative weight, and so on. The logits become structured.\n",
|
||
"\n",
|
||
"**The pattern inside logits is therefore exactly the bigram statistics of Shakespeare.** Rows for common letters like `'e'` or `' '` have several high-positive logits (many common successors). Rows for rare letters like `'z'` or `'q'` have most logits near or below zero — almost anything after `'z'` is unusual. Rows for `'q'` have one standout positive logit at `'u'`, because in English `'q'` is almost always followed by `'u'`.\n",
|
||
"\n",
|
||
"**Take-home:** \"logits at position `t`\" = 65 numbers, one per character in the vocabulary. Positive numbers = the model thinks that character is a plausible next step. Negative = implausible. `softmax` turns them into probabilities. Cross-entropy asks: *what probability did the true character get?* Training nudges the rows so this probability gets higher."
|
||
],
|
||
"id": "logits-deep-dive"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"start = 1000\n",
|
||
"x = train_data[start : start + block_size]\n",
|
||
"y = train_data[start + 1 : start + block_size + 1]\n",
|
||
"\n",
|
||
"print(\"full chunk:\", repr(decode(x.tolist())))\n",
|
||
"print()\n",
|
||
"for t in range(block_size):\n",
|
||
" prefix = decode(x[: t + 1].tolist())\n",
|
||
" target = itos[y[t].item()]\n",
|
||
" print(f\"t={t} prefix {prefix!r:12s} -> {target!r}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "e4559d0f"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 4.5** — At `t=0`, how many characters does the model see? At `t=7`?\n",
|
||
"*(0: one char; 7: all eight.)*\n"
|
||
],
|
||
"id": "3be6fbb3"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"for t in [0, 3, 7]:\n",
|
||
" prefix_len = t + 1\n",
|
||
" print(f\"t={t}: prefix length = {prefix_len}, target = {repr(itos[y[t].item()])}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "898927dc"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"start = 1000\n",
|
||
"x = train_data[start : start + block_size]\n",
|
||
"y = train_data[start + 1 : start + block_size + 1]\n",
|
||
"\n",
|
||
"for t in range(block_size):\n",
|
||
" on_tape = train_data[start + t].item()\n",
|
||
" on_tape_next = train_data[start + t + 1].item()\n",
|
||
" match = x[t].item() == on_tape and y[t].item() == on_tape_next\n",
|
||
" print(f\"t={t} x={repr(itos[x[t].item()])} y={repr(itos[y[t].item()])} matches tape: {match}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "876ad4ce"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"x = train_data[1000 : 1000 + block_size]\n",
|
||
"y = train_data[1001 : 1001 + block_size]\n",
|
||
"\n",
|
||
"print(\"x shape:\", x.shape, \" y shape:\", y.shape)\n",
|
||
"print(\"T (time steps) = block_size =\", block_size)\n",
|
||
"print()\n",
|
||
"# pretend we had batch_size=3 — same T, three independent rows\n",
|
||
"B = 3\n",
|
||
"print(f\"batched inputs would be shape ({B}, {block_size}) = (B, T)\")\n",
|
||
"print(f\"model logits later: ({B}, {block_size}, {vocab_size}) = (B, T, C)\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "8bb27b4f"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.8 Put it together — build your own `x` and `y`\n",
|
||
"\n",
|
||
"Pick any valid `start` index, build `x` and `y`, decode them, and verify every\n",
|
||
"target. This is exactly what `get_batch` does — but with random `start` values.\n",
|
||
"\n",
|
||
"**→ Training:** Same logic as `get_batch` — you're manually doing what the training loop automates thousands of times.\n"
|
||
],
|
||
"id": "f86da52d"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"start = 5000 # change me\n",
|
||
"x = train_data[start : start + block_size]\n",
|
||
"y = train_data[start + 1 : start + block_size + 1]\n",
|
||
"\n",
|
||
"print(\"start:\", start)\n",
|
||
"print(\"x:\", decode(x.tolist()))\n",
|
||
"print(\"y:\", decode(y.tolist()))\n",
|
||
"checks = [y[t].item() == train_data[start + t + 1].item() for t in range(block_size)]\n",
|
||
"print(\"all correct:\", all(checks))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "c2ee21e5"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 5.2 Stack two chunks by hand\n",
|
||
"\n",
|
||
"Before `get_batch`, see how two slices become a 2×8 matrix.\n",
|
||
"\n",
|
||
"**→ Training:** `get_batch` does this with `batch_size` rows. One `optimizer.step()` updates weights using loss from **all** rows.\n"
|
||
],
|
||
"id": "0b02e589"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"torch.manual_seed(0)\n",
|
||
"ix = torch.randint(len(train_data) - block_size, (2,))\n",
|
||
"chunk_a = train_data[ix[0] : ix[0] + block_size]\n",
|
||
"chunk_b = train_data[ix[1] : ix[1] + block_size]\n",
|
||
"stacked = torch.stack([chunk_a, chunk_b])\n",
|
||
"print(\"stacked shape:\", stacked.shape, \" (= 2 rows, block_size cols)\")\n",
|
||
"for row in range(2):\n",
|
||
" print(f\" row {row}:\", decode(stacked[row].tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "11fc074b"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 5.3 `get_batch` — random windows\n",
|
||
"\n",
|
||
"`xb` = inputs, `yb` = targets (each row shifted +1 on the tape). Same shapes.\n",
|
||
"\n",
|
||
"**→ Training:** Called every training iteration. Returns the `(xb, yb)` pair that `loss.backward()` will score and correct.\n",
|
||
"\n",
|
||
"**In the chain:** `get_batch` is **step 0** of every training iteration — supplies `xb` (questions) and `yb` (answers) before `model()` can run.\n"
|
||
],
|
||
"id": "c296b808"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"torch.manual_seed(1337)\n",
|
||
"batch_size = 4\n",
|
||
"\n",
|
||
"def get_batch(split):\n",
|
||
" d = train_data if split == \"train\" else val_data\n",
|
||
" ix = torch.randint(len(d) - block_size, (batch_size,))\n",
|
||
" x = torch.stack([d[i : i + block_size] for i in ix])\n",
|
||
" y = torch.stack([d[i + 1 : i + block_size + 1] for i in ix])\n",
|
||
" return x.to(device), y.to(device)\n",
|
||
"\n",
|
||
"xb, yb = get_batch(\"train\")\n",
|
||
"print(\"xb shape:\", xb.shape, \" yb shape:\", yb.shape)\n",
|
||
"print(xb)\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "67c1ce6f"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 5.4 Decode a batch row\n",
|
||
"\n",
|
||
"Row `b`, column `t`: `yb[b,t]` is the next token after `xb[b,t]` on the tape.\n",
|
||
"\n",
|
||
"**→ Training:** Sanity check that `yb` is the answer key aligned with `xb` before trusting the loss number.\n"
|
||
],
|
||
"id": "e72033cb"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"b = 0\n",
|
||
"print(\"row\", b, \"input: \", decode(xb[b].tolist()))\n",
|
||
"print(\"row\", b, \"target:\", decode(yb[b].tolist()))\n",
|
||
"for t in range(block_size):\n",
|
||
" print(f\" t={t} x={repr(itos[xb[b,t].item()])} y={repr(itos[yb[b,t].item()])}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "d8be50cf"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 5** — Call `get_batch(\"val\")` and decode row 2.\n"
|
||
],
|
||
"id": "3f69fc3b"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"xb_val, yb_val = get_batch(\"val\")\n",
|
||
"print(decode(xb_val[2].tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "cd1c51dc"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.1 Logits — raw scores\n",
|
||
"\n",
|
||
"For each possible next token, the model outputs a **logit** (unnormalised score).\n",
|
||
"Higher = more likely. Length = `vocab_size`.\n",
|
||
"\n",
|
||
"**→ Training:** Logits are not trained directly — **weights** produce logits. Gradients flow back through logits into those weights.\n"
|
||
],
|
||
"id": "f6bb711d"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.1b Logits deep-dive — what does a model actually output?\n",
|
||
"\n",
|
||
"Above we said: **logits are raw, unnormalised scores.** Let's make that\n",
|
||
"completely concrete with the text *\"Before we proceed any further, hear me speak.\"*\n",
|
||
"We pass one 8-character window through the Bigram model, print the real numbers, and\n",
|
||
"show exactly how they become probabilities.\n",
|
||
"\n",
|
||
"**Intuition: the dice analogy.** Imagine a weighted 6-sided dice. For each possible\n",
|
||
"outcome, you express your gut confidence as a raw number (positive = confident this\n",
|
||
"will happen, negative = sceptical). Those numbers are *logits*. Then `softmax` squashes\n",
|
||
"them into a proper probability distribution (all positive, sums to 1).\n",
|
||
"\n",
|
||
"| In the dice analogy | In the Bigram model |\n",
|
||
"|---|---|\n",
|
||
"| 6 sides of the dice | 65 characters in `vocab_size` |\n",
|
||
"| Which side will land? | Which character comes next? |\n",
|
||
"| Your raw scores (-2, -1, 0, 3, 1, 0) | `logits[b, t, :]` — 65 numbers at each position |\n",
|
||
"| softmax → probabilities | `F.softmax(logits, dim=-1)` → 65 probabilities summing to 1 |\n",
|
||
"| The side that actually rolled | `yb[b, t]` — the one correct char |\n",
|
||
"\n",
|
||
"**Why not output probabilities directly?** A neural network's layers naturally produce\n",
|
||
"numbers that can be anywhere from -infinity to +infinity. Constraining them to [0, 1] and\n",
|
||
"summing to 1 is a *post-processing* step — softmax. Training is also numerically\n",
|
||
"more stable when loss is computed on logits (which PyTorch's `F.cross_entropy` does\n",
|
||
"for us automatically).\n",
|
||
"\n",
|
||
"**-> Training:** Only the *relative order* of logits matters for which class wins.\n",
|
||
"The absolute scale matters for *how confident* the model thinks it is — and that\n",
|
||
"scale is what `backward()` adjusts.\n"
|
||
],
|
||
"id": "06417850"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Run the Bigram model on a real 8-character window and see its raw output.\n",
|
||
"# We take a short sentence, slice one (x, y) pair, and ask: what logits\n",
|
||
"# does the *current* model produce at each position?\n",
|
||
"\n",
|
||
"sentence = \"Before we proceed any further, hear me speak.\"\n",
|
||
"\n",
|
||
"import math # for -ln(p) later\n",
|
||
"\n",
|
||
"# convert to tensor ids (reuse stoi from earlier cells)\n",
|
||
"sentence_ids = torch.tensor([stoi[c] for c in sentence], dtype=torch.long, device=device)\n",
|
||
"print(\"first 8 chars of sentence:\", repr(sentence[:8]))\n",
|
||
"print(\" as ids:\", sentence_ids[:8].tolist())\n",
|
||
"\n",
|
||
"# run one forward pass *without* targets (generation path — no loss, just logits)\n",
|
||
"with torch.no_grad():\n",
|
||
" idx = sentence_ids[:8].unsqueeze(0) # shape (1, 8) = one row\n",
|
||
" logits, _ = model(idx) # shape (1, 8, vocab_size)\n",
|
||
"\n",
|
||
"print()\n",
|
||
"print(\"logits shape =\", tuple(logits.shape), \" -> (batch=1, time=8, vocab=\" + str(vocab_size) + \")\")\n",
|
||
"print(\" That's 8 x\", vocab_size, \"=\", 8 * vocab_size, \"raw scores produced by one small window.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "996e68b2"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Show the logits for ONE position in detail: position t=0 (model saw only 'B').\n",
|
||
"# We print the TOP-5 highest-scoring characters and the TRUE next character's score.\n",
|
||
"\n",
|
||
"with torch.no_grad():\n",
|
||
" idx = sentence_ids[:8].unsqueeze(0)\n",
|
||
" logits, _ = model(idx)\n",
|
||
"\n",
|
||
"def report(logits_row, position, context_str, true_next_char):\n",
|
||
" \"\"\"Pretty-print the top-k logits + probability for one position.\"\"\"\n",
|
||
" scores = logits_row[position, :] # shape (vocab_size,)\n",
|
||
" probs = torch.softmax(scores, dim=0)\n",
|
||
" top5_vals, top5_ids = torch.topk(scores, k=5)\n",
|
||
" print(\"t=\" + str(position) + \" context =\", repr(context_str), \"-> model top-5 guesses for next char:\")\n",
|
||
" for v, i in zip(top5_vals, top5_ids):\n",
|
||
" ch = itos[i.item()]\n",
|
||
" p = probs[i.item()].item()\n",
|
||
" print(\" \" + repr(ch) + \" logit=\" + (\"%+7.4f\" % v.item()) + \" prob=\" + (\"%8.4f\" % p))\n",
|
||
" true_id = stoi[true_next_char]\n",
|
||
" print(\" TRUE next char =\" + repr(true_next_char) + \" logit=\" + (\"%+7.4f\" % scores[true_id].item()) + \" prob=\" + (\"%8.4f\" % probs[true_id].item()))\n",
|
||
" print()\n",
|
||
"\n",
|
||
"for t, context_len in [(0, 1), (3, 4), (7, 8)]:\n",
|
||
" ctx_str = \"\".join(itos[sentence_ids[i].item()] for i in range(context_len))\n",
|
||
" true_next = itos[sentence_ids[context_len].item()]\n",
|
||
" report(logits[0], t, ctx_str, true_next)\n",
|
||
"\n",
|
||
"print(\"Observations (before training — weights are still random):\")\n",
|
||
"print(\" 1. The TRUE next char usually gets a tiny probability (near 1/vocab_size).\")\n",
|
||
"print(\" 2. Top-5 guesses are nonsense — the table has not learned anything yet.\")\n",
|
||
"print(\" 3. All logits are close to 0; softmax of near-zero inputs is near-uniform.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "d4f0a86d"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Why \"near-uniform\" before training?\n",
|
||
"\n",
|
||
"`nn.Embedding` initialises every entry from a normal distribution with mean ~ 0 and\n",
|
||
"std ~ 1. When every one of 65 logits is a small random number, softmax produces a\n",
|
||
"distribution where every character has probability roughly `1/65 ~ 1.5%`. That's\n",
|
||
"why the initial loss is always near `-ln(1/vocab_size) ~ 4.17` — a *sanity check*.\n",
|
||
"\n",
|
||
"As training runs, the table rows for common pairs like `t->h`, `h->e`, `e-> `\n",
|
||
"(space) get their logits pushed *up* (more positive), while rare or impossible\n",
|
||
"pairs like `t->q` get their logits pushed *down*. That change — per weight, by a\n",
|
||
"tiny amount per step — *is* learning.\n"
|
||
],
|
||
"id": "a3a7db40"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"toy_logits = torch.tensor([2.0, 0.5, 0.1]) # pretend vocab_size=3\n",
|
||
"print(\"logits:\", toy_logits.tolist())\n",
|
||
"print(\"argmax (favourite class):\", toy_logits.argmax().item())\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "e7428034"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.2 Softmax — scores → probabilities\n",
|
||
"\n",
|
||
"Softmax squashes logits into positive numbers that **sum to 1** — a distribution.\n",
|
||
"\n",
|
||
"**→ Training:** Used heavily in **generation** (sampling). Training uses cross-entropy on logits directly (softmax is inside that).\n"
|
||
],
|
||
"id": "ff289741"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"probs = F.softmax(toy_logits, dim=0)\n",
|
||
"print(\"probs:\", [round(p, 3) for p in probs.tolist()])\n",
|
||
"print(\"sum:\", probs.sum().item())\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "71e447c8"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.3 Cross-entropy loss — how wrong were we?\n",
|
||
"\n",
|
||
"Compare model distribution to the **true** next token id.\n",
|
||
"- right answer gets high prob → low loss\n",
|
||
"- wrong answer → high loss\n",
|
||
"\n",
|
||
"Random guessing over `vocab_size` tokens → loss ≈ `-ln(1/vocab_size)`.\n",
|
||
"\n",
|
||
"**→ Training:** The single number `loss` that `backward()` uses. Lower loss ⇒ nudge weights so true next tokens get higher score.\n",
|
||
"\n",
|
||
"**In the chain:** cross-entropy turns logits + `yb` into the **single number** `loss`. Section 8 cannot run `backward()` until this exists.\n"
|
||
],
|
||
"id": "e60d5dd1"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.3b Cross-entropy deep-dive — what are we actually punishing?\n",
|
||
"\n",
|
||
"**Shortest summary:** `cross_entropy(logits, true_index)` = a single number saying\n",
|
||
"how much probability the model *wasted* on wrong answers, averaged over every\n",
|
||
"prediction in the batch.\n",
|
||
"\n",
|
||
"**Analogy: the weather forecaster.** Every morning a forecaster gives percentages\n",
|
||
"for sunny / rain / snow. The next day we see what *actually* happened and score them.\n",
|
||
"\n",
|
||
"| Day | Forecast (predicted probs) | Reality | Prob assigned to TRUE | `-ln(p)` = penalty |\n",
|
||
"|-----|----------------------------|---------|------------------------|-------------------|\n",
|
||
"| Mon | sunny 80%, rain 15%, snow 5% | sunny | 0.80 | 0.22 (small — confident & right) |\n",
|
||
"| Tue | sunny 30%, rain 60%, snow 10% | rain | 0.60 | 0.51 |\n",
|
||
"| Wed | sunny 20%, rain 30%, snow 50% | snow | 0.50 | 0.69 |\n",
|
||
"| Thu | sunny 95%, rain 4%, snow 1% | snow | 0.01 | 4.61 (huge — confident & wrong) |\n",
|
||
"\n",
|
||
"The average of those penalties over a season is exactly **mean cross-entropy**:\n",
|
||
"**the lower, the better the forecaster.**\n",
|
||
"\n",
|
||
"The Bigram model is exactly a 65-outcome forecaster: instead of weather it has\n",
|
||
"a-z, spaces, punctuation, newlines as the 65 possible \"next characters\", and it\n",
|
||
"must predict \"which character comes next?\" 8 times per row, batch_size rows per batch.\n",
|
||
"\n",
|
||
"**-> Training:** `loss = F.cross_entropy(logits, yb)` is that single scalar.\n",
|
||
"`loss.backward()` then asks, for every weight: *\"if I nudge you up (or down),\n",
|
||
"does the penalty get smaller?\"* — and `optimizer.step()` takes that tiny nudge.\n"
|
||
],
|
||
"id": "7e1be656"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Compute cross-entropy by hand on one 8-char window, match it to\n",
|
||
"# what F.cross_entropy returns, and watch true-char probability climb\n",
|
||
"# as training progresses.\n",
|
||
"\n",
|
||
"with torch.no_grad():\n",
|
||
" idx = sentence_ids[:8].unsqueeze(0)\n",
|
||
" logits, _ = model(idx) # (1, 8, vocab_size)\n",
|
||
"\n",
|
||
"y_ids = sentence_ids[1:9].unsqueeze(0) # the 8 true next chars\n",
|
||
"\n",
|
||
"# --- softmax -> probabilities -> -ln(prob of true char) per position ---\n",
|
||
"probs = torch.softmax(logits, dim=-1)\n",
|
||
"\n",
|
||
"header = \"pos\" + \" \" * 6 + \"ctx\" + \" \" * 11 + \"true\" + \" \" * 7 + \"prob(true)\" + \" \" * 3 + \"-ln(p)\"\n",
|
||
"print(header)\n",
|
||
"print(\"-\" * 55)\n",
|
||
"per_pos_loss = []\n",
|
||
"for t in range(8):\n",
|
||
" ctx = \"\".join(itos[sentence_ids[i].item()] for i in range(t + 1))\n",
|
||
" true_ch = itos[y_ids[0, t].item()]\n",
|
||
" p = probs[0, t, y_ids[0, t]].item()\n",
|
||
" penalty = -math.log(p) if p > 0 else float(\"inf\")\n",
|
||
" per_pos_loss.append(penalty)\n",
|
||
" print(\"%3d\" % t + \" \" * 2 + repr(ctx) + \" \" * 5 + repr(true_ch) + \" \" * 5 + (\"%10.6f\" % p) + \" \" * 3 + (\"%8.4f\" % penalty))\n",
|
||
"\n",
|
||
"manual_loss = sum(per_pos_loss) / len(per_pos_loss)\n",
|
||
"pytorch_loss = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n",
|
||
"\n",
|
||
"print()\n",
|
||
"print(\"mean cross-entropy (computed by hand): %.6f\" % manual_loss)\n",
|
||
"print(\"mean cross-entropy (F.cross_entropy): %.6f\" % pytorch_loss)\n",
|
||
"print(\"match:\", abs(manual_loss - pytorch_loss) < 1e-5)\n",
|
||
"print()\n",
|
||
"print(\"Notice: every position where the model assigned < 1/vocab to the true\")\n",
|
||
"print(\"character contributes a large penalty to the mean. Training shrinks these.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "dda290e1"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Watch a single training step on this same sentence *improve* true-char probability.\n",
|
||
"# We run forward -> backward -> step on sentence_ids[:8]/[1:9], then re-print.\n",
|
||
"\n",
|
||
"def one_step_xent_report(comment):\n",
|
||
" with torch.no_grad():\n",
|
||
" logits, _ = model(sentence_ids[:8].unsqueeze(0))\n",
|
||
" probs = torch.softmax(logits, dim=-1)\n",
|
||
" y_ids = sentence_ids[1:9].unsqueeze(0)\n",
|
||
" true_probs = [probs[0, t, y_ids[0, t]].item() for t in range(8)]\n",
|
||
" L = F.cross_entropy(logits.view(-1, vocab_size), y_ids.view(-1)).item()\n",
|
||
" avg_p = sum(true_probs) / len(true_probs)\n",
|
||
" print(\" \" + comment.ljust(30) + (\" loss=%.4f\" % L) + (\" avg prob(true char)=%.4f\" % avg_p))\n",
|
||
" return L\n",
|
||
"\n",
|
||
"print(\"=== Nudging the embedding table on this ONE sentence ===\")\n",
|
||
"print()\n",
|
||
"\n",
|
||
"# Slightly bigger LR so the demo shows visible change in a few steps.\n",
|
||
"optimizer = torch.optim.AdamW(model.parameters(), lr=5e-2)\n",
|
||
"one_step_xent_report(\"before 1 step\")\n",
|
||
"\n",
|
||
"for step in range(20):\n",
|
||
" idx = sentence_ids[:8].unsqueeze(0)\n",
|
||
" y = sentence_ids[1:9].unsqueeze(0)\n",
|
||
" _, loss = model(idx, y)\n",
|
||
" optimizer.zero_grad(set_to_none=True)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
" if (step + 1) % 5 == 0:\n",
|
||
" one_step_xent_report(\"after \" + str(step + 1) + \" steps\")\n",
|
||
"\n",
|
||
"print()\n",
|
||
"print(\"Loss dropped on this *specific* window because we deliberately over-trained\")\n",
|
||
"print(\"on it. In a real training loop you sample NEW windows each iteration, so the\")\n",
|
||
"print(\"model learns general patterns, not just this sentence.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "7744f6f9"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Two rules of thumb for cross-entropy in this notebook\n",
|
||
"\n",
|
||
"1. **The best a *bigram* model can do is ~2.3.** Seeing only one previous character\n",
|
||
"cannot disambiguate `th` from `t ` (space) or `he` from `ha`. Better context\n",
|
||
"(self-attention, next notebook) can push loss significantly lower.\n",
|
||
"2. **val loss is the honest score.** `train_loss` can be made arbitrarily low by\n",
|
||
"training long enough (you're memorising the training set). `val_loss` on unseen\n",
|
||
"text is what actually tells you whether the model generalises.\n"
|
||
],
|
||
"id": "c0a89db5"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"for target in [0, 1, 2]:\n",
|
||
" loss = F.cross_entropy(toy_logits.unsqueeze(0), torch.tensor([target]))\n",
|
||
" print(f\"true class {target} -> loss {loss.item():.3f}\")\n",
|
||
"expected = -torch.log(torch.tensor(1.0 / vocab_size))\n",
|
||
"print(f\"random baseline (vocab {vocab_size}): {expected.item():.3f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "c9166861"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 6** — Make class 2 strongest; loss for target 2 should drop.\n"
|
||
],
|
||
"id": "7b45f3a7"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"strong = torch.tensor([0.1, 0.5, 2.5])\n",
|
||
"for target in [0, 2]:\n",
|
||
" loss = F.cross_entropy(strong.unsqueeze(0), torch.tensor([target]))\n",
|
||
" print(f\"target {target} -> {loss.item():.3f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "dfe98faf"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.4 Bigram = one lookup table\n",
|
||
"\n",
|
||
"`nn.Embedding(vocab_size, vocab_size)` stores a matrix: **row i** = logits for\n",
|
||
"\"what follows token i?\" No memory of earlier tokens.\n",
|
||
"\n",
|
||
"**Analogy:** a cheat sheet — \"after `t` usually comes `h` or `e`\" — but only\n",
|
||
"looking at the **last** letter.\n",
|
||
"\n",
|
||
"**→ Training:** The entire learned model is this **one matrix** (~vocab² numbers). Training adjusts every row: \"what follows char i?\"\n"
|
||
],
|
||
"id": "7926ed10"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Is the 65×65 table fixed or updated during training?\n",
|
||
"\n",
|
||
"**The shape is fixed; the numbers inside are constantly updated.**\n",
|
||
"\n",
|
||
"- **Fixed:** always 65 rows × 65 columns (4,225 numbers) — one row per character, one column per possible next character.\n",
|
||
"- **Not fixed:** every number is a **learned weight** that changes during training.\n",
|
||
"\n",
|
||
"**Before training:** all values start as small random guesses. The model does not yet know that `'q'` is usually followed by `'u'`, or that `'f'` is often followed by `'o'`.\n",
|
||
"\n",
|
||
"**Each training step:**\n",
|
||
"\n",
|
||
"1. **Forward** — look up rows, get logits, compute loss.\n",
|
||
"2. **Backward** — `loss.backward()` asks each cell: \"how should I change to reduce loss?\"\n",
|
||
"3. **Update** — `optimizer.step()` nudges those numbers a tiny bit.\n",
|
||
"\n",
|
||
"Cells that appear in the current batch get updated; over thousands of steps the whole table absorbs Shakespeare's bigram statistics.\n",
|
||
"\n",
|
||
"| Stage | What happens to the table |\n",
|
||
"| --- | --- |\n",
|
||
"| Start | Random values |\n",
|
||
"| Each batch | Used cells get small updates |\n",
|
||
"| After many steps | Values reflect real patterns (e.g. `'w'→'e'` scores rise) |\n",
|
||
"| After training | Table is frozen unless you train again |\n",
|
||
"\n",
|
||
"**Concrete example:** row `'B'`, column `'e'` might start near +0.02 (meaningless). After seeing many `B…e` pairs in training it might become +1.8 — the model learned that `'B'` often leads to `'e'`.\n",
|
||
"\n",
|
||
"**Mental picture:** the table is a cheat sheet you fill in while studying. Day 1: blank guesses. Each practice question: adjust a few entries. After thousands of questions: the cheat sheet matches how Shakespeare actually writes.\n",
|
||
"\n",
|
||
"**Take-home:** the 65×65 **grid size** never changes. The **values** are what training learns — that is exactly what `loss.backward()` and `optimizer.step()` do."
|
||
],
|
||
"id": "30a6b989"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"table = nn.Embedding(vocab_size, vocab_size)\n",
|
||
"ch = 't'\n",
|
||
"tid = stoi[ch]\n",
|
||
"row = table.weight[tid]\n",
|
||
"print(\"row for\", repr(ch), \"has shape:\", row.shape)\n",
|
||
"top_p, top_i = F.softmax(row, dim=0).topk(3)\n",
|
||
"print(\"top-3 followers (random weights):\")\n",
|
||
"for p, i in zip(top_p, top_i):\n",
|
||
" print(f\" {repr(itos[i.item()])}: {p.item():.3f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "f705fa50"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 6.5 The full `BigramLanguageModel`\n",
|
||
"\n",
|
||
"Forward: `idx` `(B,T)` → logits `(B,T,C)`. Flatten to `(B*T, C)` for one big\n",
|
||
"cross-entropy over every position in the batch.\n",
|
||
"\n",
|
||
"**→ Training:** `forward(xb, yb)` = predict on all positions, compare to `yb`, return scalar loss. `generate()` uses same weights without `yb`.\n",
|
||
"\n",
|
||
"**In the chain:** `forward(xb, yb)` is **step 1** of training. Without `yb`, you get logits only (generation path) — no loss, no backward.\n"
|
||
],
|
||
"id": "e5208051"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"class BigramLanguageModel(nn.Module):\n",
|
||
" def __init__(self, vocab_size):\n",
|
||
" super().__init__()\n",
|
||
" self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n",
|
||
"\n",
|
||
" def forward(self, idx, targets=None):\n",
|
||
" logits = self.token_embedding_table(idx)\n",
|
||
" if targets is None:\n",
|
||
" return logits, None\n",
|
||
" B, T, C = logits.shape\n",
|
||
" loss = F.cross_entropy(logits.view(B * T, C), targets.view(B * T))\n",
|
||
" return logits, loss\n",
|
||
"\n",
|
||
" def generate(self, idx, max_new_tokens):\n",
|
||
" for _ in range(max_new_tokens):\n",
|
||
" logits, _ = self(idx)\n",
|
||
" logits = logits[:, -1, :]\n",
|
||
" probs = F.softmax(logits, dim=-1)\n",
|
||
" idx_next = torch.multinomial(probs, num_samples=1)\n",
|
||
" idx = torch.cat((idx, idx_next), dim=1)\n",
|
||
" return idx\n",
|
||
"\n",
|
||
"model = BigramLanguageModel(vocab_size).to(device)\n",
|
||
"xb, yb = get_batch(\"train\")\n",
|
||
"logits, loss = model(xb, yb)\n",
|
||
"print(\"logits shape (B, T, C):\", logits.shape)\n",
|
||
"print(f\"initial loss: {loss.item():.4f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "f7cc9f29"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### At the end — is it just the table, or something more?\n",
|
||
"\n",
|
||
"For the **bigram model in this notebook**, yes: after training you basically **just have the table**. That table **is** the entire learned model.\n",
|
||
"\n",
|
||
"There is no second brain, no hidden layers, no memory beyond one letter:\n",
|
||
"\n",
|
||
"```\n",
|
||
"current letter → look up its row → 65 scores → pick next letter\n",
|
||
"```\n",
|
||
"\n",
|
||
"**What is learned vs what is fixed rules**\n",
|
||
"\n",
|
||
"| Part | Learned? | Role |\n",
|
||
"| --- | --- | --- |\n",
|
||
"| 65×65 table | Yes — training only updates this | Stores \"after X, how likely is each next letter?\" |\n",
|
||
"| Table lookup | No — fixed rule | Given w, read row for w |\n",
|
||
"| Softmax | No — fixed math | Turn 65 scores into probabilities summing to 1 |\n",
|
||
"| Sampling (multinomial) | No — fixed rule | Roll the dice using those probabilities |\n",
|
||
"| Generation loop | No — fixed code | Repeat: predict one letter, append, predict again |\n",
|
||
"\n",
|
||
"- **Training** teaches only the **table**.\n",
|
||
"- **Softmax, sampling, and the loop** are small fixed steps that **use** the table to write text.\n",
|
||
"\n",
|
||
"**Does it always pick the top letter?** No. The table does not say \"the next letter **is** e.\" It says \"e has 40% chance, h has 20%, ...\" Then sampling **randomly** picks one letter from those chances — that is why generated text varies each run. Always picking the highest score would sound repetitive and robotic.\n",
|
||
"\n",
|
||
"**Take-home:** end state = one trained cheat sheet (the table). To generate: lookup → softmax → random pick → repeat. Later GPT models add real extra machinery (attention, many layers). The bigram model is intentionally the simplest case: **table only**."
|
||
],
|
||
"id": "531060cd"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 6b** — Initial loss should be near `-ln(1/vocab_size)`. Compare.\n"
|
||
],
|
||
"id": "1614313d"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n",
|
||
"print(f\"loss {loss.item():.4f} vs random baseline {baseline:.4f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "25b7140e"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 7.1 One generation step by hand\n",
|
||
"\n",
|
||
"Context → forward → logits at **last** position → softmax → sample one id.\n",
|
||
"\n",
|
||
"**→ Training:** Same forward path as training, but no `yb` and no `backward()`. Shows what one row of trained weights **does** at inference.\n"
|
||
],
|
||
"id": "f51c9655"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"context = torch.zeros((1, 1), dtype=torch.long, device=device) # start: newline\n",
|
||
"logits, _ = model(context)\n",
|
||
"last_logits = logits[0, -1, :]\n",
|
||
"probs = F.softmax(last_logits, dim=0)\n",
|
||
"sampled = torch.multinomial(probs, num_samples=1)\n",
|
||
"print(\"context:\", repr(itos[context[0,0].item()]))\n",
|
||
"print(\"sampled next char:\", repr(itos[sampled.item()]))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "f9cdda7f"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 7.2 Full generation (200 chars, untrained)\n",
|
||
"\n",
|
||
"Start from newline (id 0). Expect nonsense.\n",
|
||
"\n",
|
||
"**→ Training:** Save mentally (or in output) for contrast with section 9 after weights have been updated.\n"
|
||
],
|
||
"id": "f8461a58"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
|
||
"out = model.generate(context, max_new_tokens=200)\n",
|
||
"print(decode(out[0].tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "4b5c3e45"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# THE trainable artifact: model parameters (weight matrices)\n",
|
||
"n_params = sum(p.numel() for p in model.parameters())\n",
|
||
"print(\"trainable parameters:\", n_params)\n",
|
||
"print(\"embedding weight shape:\", model.token_embedding_table.weight.shape)\n",
|
||
"print(\"dtype:\", model.token_embedding_table.weight.dtype)\n",
|
||
"print()\n",
|
||
"print(\"These numbers ARE what training rewrites.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "0881b2c7"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# trace the chain on one batch (steps 1 -> 2 -> 3)\n",
|
||
"xb, yb = get_batch(\"train\")\n",
|
||
"print(\"STEP 1 prerequisite: batch shapes\", xb.shape, yb.shape)\n",
|
||
"\n",
|
||
"model.zero_grad(set_to_none=True)\n",
|
||
"logits, loss = model(xb, yb)\n",
|
||
"print(\"STEP 2: forward produced loss =\", round(loss.item(), 4), \"(scalar)\")\n",
|
||
"\n",
|
||
"loss.backward() # STEP 3\n",
|
||
"w = model.token_embedding_table.weight\n",
|
||
"print(\"STEP 3: backward filled w.grad with shape\", w.grad.shape)\n",
|
||
"print(\" example grad[t,h]:\", w.grad[stoi[\"t\"], stoi[\"h\"]].item())\n",
|
||
"print(\"STEP 4: run optimizer.step() to add grad into weights (see 8.1)\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "019a14ab"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 8.0 What is a gradient? (and why it matters for AI)\n",
|
||
"\n",
|
||
"Imagine you are standing on a **hilly field in thick fog**. You cannot see the whole\n",
|
||
"landscape, but you can feel which way the ground **slopes upward** under your feet.\n",
|
||
"\n",
|
||
"The **gradient** is that slope information:\n",
|
||
"\n",
|
||
"- **Direction** — which way is uphill (steepest climb)\n",
|
||
"- **Size** — how steep it is\n",
|
||
"\n",
|
||
"In machine learning, the “hill” is usually the **loss** (how wrong the model is).\n",
|
||
"Your “position” on the hill is all the **weights** in the model. The gradient tells\n",
|
||
"each weight: *if you nudge yourself a tiny bit this way, does loss go up or down —\n",
|
||
"and by how much?*\n",
|
||
"\n",
|
||
"#### Simple real-life pictures\n",
|
||
"\n",
|
||
"**1. Finding the bottom of a valley (blindfolded).**\n",
|
||
"You want the **lowest point** (minimum loss). You feel the slope and take a small\n",
|
||
"step **downhill**. Repeat. That step direction is the **negative gradient** (opposite\n",
|
||
"of uphill). Training is exactly this: start with bad guesses, feel the slope, take\n",
|
||
"small downhill steps, thousands of times.\n",
|
||
"\n",
|
||
"**2. Tuning a radio dial.**\n",
|
||
"You turn the dial slightly left — worse static. Turn right — clearer signal. The\n",
|
||
"“gradient” is like: *turning right helps; turning left hurts.* Each weight in a\n",
|
||
"neural net is a dial; the gradient says which way to turn each one to reduce error.\n",
|
||
"\n",
|
||
"**3. Baking cookies.**\n",
|
||
"Cookies too salty? You try tiny changes — less salt, more sugar, shorter bake — and\n",
|
||
"see what improves taste. Gradient descent does the same at massive scale: millions\n",
|
||
"of knobs, each nudged based on how much it helped or hurt.\n",
|
||
"\n",
|
||
"#### Why gradients are central to ML and AI\n",
|
||
"\n",
|
||
"Modern models have **millions or billions** of weights. You cannot tune them by hand.\n",
|
||
"\n",
|
||
"Gradients make learning systematic:\n",
|
||
"\n",
|
||
"1. **Forward** — the model makes predictions (e.g. next letter after `\"Be\"`).\n",
|
||
"2. **Loss** — measures how wrong it was (cross-entropy: how little probability\n",
|
||
" landed on the true answer).\n",
|
||
"3. **Backward** — `loss.backward()` asks every weight: *how much did you contribute\n",
|
||
" to the mistake?* (automatic differentiation through the computation graph).\n",
|
||
"4. **Update** — `optimizer.step()` moves each weight a tiny step **downhill**.\n",
|
||
"\n",
|
||
"Without gradients, training would be random guessing. With them, the model improves\n",
|
||
"a little on every batch until it fits the data.\n",
|
||
"\n",
|
||
"| Real life | Machine learning |\n",
|
||
"|-----------|------------------|\n",
|
||
"| Hilly foggy field | Loss surface over all weights |\n",
|
||
"| Where you stand | Current model weights |\n",
|
||
"| Slope under your feet | **Gradient** (stored in `.grad`) |\n",
|
||
"| Step downhill | **`optimizer.step()`** |\n",
|
||
"| Valley floor | **Low loss** — model predicts well |\n",
|
||
"\n",
|
||
"**Take-home:** a gradient is the model’s way of *feeling* which direction each weight\n",
|
||
"should move to make fewer mistakes. Almost all modern AI learns by walking downhill\n",
|
||
"on that felt slope."
|
||
],
|
||
"id": "7be0163b"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 8.1 One training step\n",
|
||
"\n",
|
||
"Run this once to see loss change after a single update.\n",
|
||
"\n",
|
||
"**→ Training:** Minimal version of the full loop. If loss changes after `step()`, weights moved — training is working.\n"
|
||
],
|
||
"id": "80359cd3"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 8.1b The training loop — what *actually* runs each iteration?\n",
|
||
"\n",
|
||
"The loop in section 8.2 is short. Let's make *every line* explicit on one\n",
|
||
"concrete batch. Each iteration performs four operations in order.\n",
|
||
"\n",
|
||
" 1. xb, yb = get_batch(\"train\") # sample ONE new random batch from the tape\n",
|
||
" 2. logits, loss = model(xb, yb) # FORWARD pass — ALL rows/positions in parallel\n",
|
||
" 3. optimizer.zero_grad(); loss.backward() # BACKWARD — fills .grad on every weight\n",
|
||
" 4. optimizer.step() # NUDGE every weight a tiny bit against its gradient\n",
|
||
"\n",
|
||
"Crucially: **all batch rows share one forward, one backward, one optimizer step.**\n",
|
||
"The loop does NOT iterate over rows. That parallelism is why GPUs matter — each\n",
|
||
"training iteration is one big embedding lookup followed by one big gradient update.\n",
|
||
"\n",
|
||
"**Gradients for each weight are the SUM of contributions from every prediction**\n",
|
||
"in the batch. This is why bigger batches are more *stable*: each nudge is averaged\n",
|
||
"over more independent evidence, so the direction to nudge is less noisy.\n"
|
||
],
|
||
"id": "b5af98ec"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Trace: one iteration, fully explicit, on a fresh batch of 3 rows x 8 positions.\n",
|
||
"# We print shapes, the loss scalar, the non-zero-count of the gradient, and\n",
|
||
"# the delta on one specific embedding entry (t -> h) before vs after step().\n",
|
||
"\n",
|
||
"bs = 3\n",
|
||
"\n",
|
||
"# 1) sample a batch\n",
|
||
"ix = torch.randint(len(train_data) - block_size, (bs,))\n",
|
||
"xb = torch.stack([train_data[i : i + block_size] for i in ix]).to(device)\n",
|
||
"yb = torch.stack([train_data[i + 1 : i + block_size + 1] for i in ix]).to(device)\n",
|
||
"\n",
|
||
"print(\"xb shape =\", tuple(xb.shape), \" ->\", bs, \"rows x\", block_size, \"positions =\", bs * block_size, \"supervised predictions in a single forward pass\")\n",
|
||
"print(\"row text:\", [decode(xb[i].tolist()) for i in range(bs)])\n",
|
||
"print()\n",
|
||
"\n",
|
||
"# 2) forward: produces (bs, block_size, vocab_size) logits AND one scalar loss\n",
|
||
"logits, loss = model(xb, yb)\n",
|
||
"print(\"forward pass -> logits shape\", tuple(logits.shape), \", loss scalar = %.4f\" % loss.item())\n",
|
||
"print()\n",
|
||
"\n",
|
||
"# 3) zero old gradients, then backprop from the single loss scalar\n",
|
||
"optimizer.zero_grad(set_to_none=True)\n",
|
||
"loss.backward()\n",
|
||
"w = model.token_embedding_table.weight\n",
|
||
"grad = w.grad\n",
|
||
"nonzero = (grad != 0).sum().item()\n",
|
||
"total = grad.numel()\n",
|
||
"print(\"backward pass -> gradient tensor has shape\", tuple(grad.shape))\n",
|
||
"print(\" entries with a non-zero gradient:\", nonzero, \"/\", total, \"(only char-pairs touched by this batch)\")\n",
|
||
"print(\" gradient norm = %.4f\" % grad.norm().item())\n",
|
||
"print()\n",
|
||
"\n",
|
||
"# 4) snapshot one embedding entry, step, and compare\n",
|
||
"t_id, h_id = stoi[\"t\"], stoi[\"h\"]\n",
|
||
"before = w[t_id, h_id].item()\n",
|
||
"optimizer.step()\n",
|
||
"after = w[t_id, h_id].item()\n",
|
||
"print(\"optimizer.step() -> one weight entry (t -> h): %+.6f -> %+.6f\" % (before, after))\n",
|
||
"print(\" delta = %+.8f\" % (after - before))\n",
|
||
"print()\n",
|
||
"print(\"Run this cell several times. Every iteration:\")\n",
|
||
"print(\" - the window text is different (get_batch re-samples),\")\n",
|
||
"print(\" - loss starts around 4.2 and slowly drops as training runs,\")\n",
|
||
"print(\" - the non-zero count in .grad changes (only chars in the batch count),\")\n",
|
||
"print(\" - the t->h entry nudges by ~1e-3 each time — tiny! But 10 000 such\")\n",
|
||
"print(\" nudges carve out the bigram statistics of the text.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "99da7331"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Why the model does NOT need to see the whole book at once\n",
|
||
"\n",
|
||
"The training loop never processes more than `batch_size x block_size` tokens at\n",
|
||
"a time. The entire book is never loaded into one matrix. Instead:\n",
|
||
"\n",
|
||
"- Over 10 000 iterations with `batch_size = 32`, the model sees ~2.5M char-pairs.\n",
|
||
"- Different iterations expose different subsets of the vocabulary to gradients.\n",
|
||
"- Eventually every (prev_char, next_char) pair that *can* appear *does* appear,\n",
|
||
" and the embedding table row for that previous char gets nudged to score the\n",
|
||
" common next chars higher than the rare ones.\n",
|
||
"\n",
|
||
"The fact that the model is *small* (one matrix, ~4 200 parameters) is what makes\n",
|
||
"this simple-sampling strategy sufficient: there isn't much capacity to waste, and\n",
|
||
"the \"correct\" answer for each row is unambiguous bigram statistics.\n"
|
||
],
|
||
"id": "51398f79"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)\n",
|
||
"xb, yb = get_batch(\"train\")\n",
|
||
"_, loss_before = model(xb, yb)\n",
|
||
"optimizer.zero_grad(set_to_none=True)\n",
|
||
"_, loss = model(xb, yb)\n",
|
||
"loss.backward()\n",
|
||
"optimizer.step()\n",
|
||
"_, loss_after = model(xb, yb)\n",
|
||
"print(f\"before: {loss_before.item():.4f} after: {loss_after.item():.4f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "8949b082"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"# watch ONE matrix entry change (run after 8.1 created optimizer)\n",
|
||
"w = model.token_embedding_table.weight\n",
|
||
"before = w[stoi['t'], stoi['h']].item()\n",
|
||
"\n",
|
||
"xb, yb = get_batch(\"train\")\n",
|
||
"_, loss = model(xb, yb)\n",
|
||
"optimizer.zero_grad(set_to_none=True)\n",
|
||
"loss.backward()\n",
|
||
"optimizer.step()\n",
|
||
"\n",
|
||
"after = w[stoi['t'], stoi['h']].item()\n",
|
||
"print(f\"loss: {loss.item():.4f}\")\n",
|
||
"print(f\"weight score for t->h: {before:.6f} -> {after:.6f}\")\n",
|
||
"print(\"changed:\", before != after)\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "8d2cc5da"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"sd = model.state_dict()\n",
|
||
"print(\"keys in state_dict:\", list(sd.keys()))\n",
|
||
"print(\"weight tensor shape:\", sd[\"token_embedding_table.weight\"].shape)\n",
|
||
"\n",
|
||
"# optional: save / load (uncomment to use across sessions)\n",
|
||
"# torch.save(sd, \"bigram_shakespeare.pt\")\n",
|
||
"# model.load_state_dict(torch.load(\"bigram_shakespeare.pt\", map_location=device))\n",
|
||
"print(\"trained weights live in memory inside model until you torch.save them.\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "66fd4109"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 8.2 Full training loop + val loss\n",
|
||
"\n",
|
||
"Every 1000 steps we average loss on train **and** val batches. Val should track\n",
|
||
"train; if train drops but val does not → overfitting.\n",
|
||
"\n",
|
||
"**→ Training:** 10,000 updates to the embedding matrix. Printed val loss checks that weights generalise, not only memorise train slices.\n"
|
||
],
|
||
"id": "603bd9b0"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"batch_size = 32\n",
|
||
"\n",
|
||
"@torch.no_grad()\n",
|
||
"def estimate_loss(eval_iters=100):\n",
|
||
" out = {}\n",
|
||
" model.eval()\n",
|
||
" for split in [\"train\", \"val\"]:\n",
|
||
" losses = torch.zeros(eval_iters)\n",
|
||
" for k in range(eval_iters):\n",
|
||
" X, Y = get_batch(split)\n",
|
||
" _, L = model(X, Y)\n",
|
||
" losses[k] = L.item()\n",
|
||
" out[split] = losses.mean().item()\n",
|
||
" model.train()\n",
|
||
" return out\n",
|
||
"\n",
|
||
"for step in range(10000):\n",
|
||
" xb, yb = get_batch(\"train\")\n",
|
||
" _, loss = model(xb, yb)\n",
|
||
" optimizer.zero_grad(set_to_none=True)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
" if step % 1000 == 0:\n",
|
||
" L = estimate_loss()\n",
|
||
" print(f\"step {step:5d} train {L['train']:.4f} val {L['val']:.4f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "b34ee95e"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Your turn 8** — After training, is val loss below the random baseline (~4.17)?\n"
|
||
],
|
||
"id": "69ff0dfc"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"L = estimate_loss(50)\n",
|
||
"baseline = -torch.log(torch.tensor(1.0 / vocab_size)).item()\n",
|
||
"print(f\"train {L['train']:.4f} val {L['val']:.4f} baseline {baseline:.4f}\")\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "23be596a"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 9. Generation after training + what's next\n",
|
||
"\n",
|
||
"\n",
|
||
"**You are here (section 9):** see the **goal** achieved (partially — bigram is limited).\n",
|
||
"\n",
|
||
"**Needs:** section 8 (trained weights in `model`).\n",
|
||
"**Unlocks:** next notebook — self-attention for longer context.\n",
|
||
"Same `generate` as section 7 — but weights learned letter/space patterns.\n",
|
||
"Still **not** real Shakespeare: bigram only sees **one** character back.\n",
|
||
"\n",
|
||
"Next notebook: **self-attention** so each token can use the full prefix.\n",
|
||
"\n",
|
||
"**Whole notebook checklist**\n",
|
||
"- [ ] Text → tokens → 1-D tensor tape\n",
|
||
"- [ ] `x`/`y` shift, prefix trick, `(B,T)` batching\n",
|
||
"- [ ] Logits, softmax, cross-entropy\n",
|
||
"- [ ] Train loop + val loss\n",
|
||
"- [ ] Generate before vs after\n",
|
||
"\n",
|
||
"**→ Training:** Proof that weights changed. Better letter patterns = the matrix learned co-occurrence stats. Weights live in `model` until you `torch.save` them.\n"
|
||
],
|
||
"id": "1a49aef3"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"metadata": {},
|
||
"source": [
|
||
"context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
|
||
"print(decode(model.generate(context, max_new_tokens=400)[0].tolist()))\n"
|
||
],
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"id": "2ef869fb"
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv (3.12.10)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python",
|
||
"version": "3.12.10"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
} |