beastgitea2026/gpt_from_scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

c

Browse Source
This commit is contained in:
betterbeast2023
2026-06-10 19:45:12 +08:00
parent 1183d10849
commit a3ebb28ce0
12 changed files with 35469 additions and 491 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02_makemore_mlp.ipynb
+581
View File
@@ -0,0 +1,581 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b9c2fa10",
"metadata": {},
"source": [
"# makemore — the MLP character model\n",
"\n",
"Predict the next letter from the **previous few letters** with a small neural net — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **02** of the series. Notebook `00` built backprop by hand; the bigram in `01`\n",
"only ever looked at **one** character back. Here we take a real step up: a multi-layer\n",
"perceptron (MLP) that looks at a small **window** of previous characters, following Bengio et\n",
"al. 2003 and Karpathy's *\"makemore part 2: MLP\"* lecture.\n",
"\n",
"Dataset: `names.txt` — 32,033 real first names. Goal: **make more** name-like words."
]
},
{
"cell_type": "markdown",
"id": "69f5f6bc",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"A bigram only remembers the last letter, so it cannot tell that `bri` should probably be\n",
"followed by `a` (Brianna) rather than a random vowel. We fix that by letting the model read a\n",
"**block** of the last few letters at once.\n",
"\n",
"The trick that makes this work is the **embedding**: instead of treating each letter as an\n",
"isolated symbol, we give every letter a short list of numbers (a vector) that the network\n",
"learns. Letters that behave similarly drift close together.\n",
"\n",
"Real-life picture: imagine each letter is a person, and the embedding is their personality\n",
"profile. The network learns these profiles so that \"similar\" letters (say, the vowels) end up\n",
"with similar profiles, and it can generalise: a context it never saw can still be handled\n",
"because it resembles ones it did see."
]
},
{
"cell_type": "markdown",
"id": "2d5dee6c",
"metadata": {},
"source": [
"### 0.1 Imports\n",
"\n",
"We import **torch first** (before matplotlib) to avoid a Windows OpenMP DLL clash, and make\n",
"plotting optional so the notebook runs even without matplotlib.\n",
"\n",
"**-> Training:** `torch` holds the weight tensors and does autograd; `F.cross_entropy` turns\n",
"logits + the true next letter into the single `loss` number we minimise."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07c035c6",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\") # avoid duplicate-OpenMP crash on Windows\n",
"\n",
"import torch\n",
"import torch.nn.functional as F\n",
"import random\n",
"\n",
"try:\n",
" import matplotlib.pyplot as plt\n",
" HAS_PLT = True\n",
"except Exception:\n",
" HAS_PLT = False\n",
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
"\n",
"print(\"PyTorch version:\", torch.__version__)"
]
},
{
"cell_type": "markdown",
"id": "dc58a8fc",
"metadata": {},
"source": [
"### 0.2 Load the names and build the alphabet\n",
"\n",
"Each name becomes letters plus a special **`.`** marker (id 0) that means *start* and *end*.\n",
"So `emma` is really `. e m m a .` — the dots tell the model where words begin and stop."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b7e8198",
"metadata": {},
"outputs": [],
"source": [
"words = open(\"names.txt\").read().splitlines()\n",
"print(\"number of names:\", len(words))\n",
"print(\"examples:\", words[:8])\n",
"print(\"shortest/longest:\", min(len(w) for w in words), \"/\", max(len(w) for w in words))\n",
"\n",
"chars = sorted(list(set(\"\".join(words))))\n",
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
"stoi[\".\"] = 0\n",
"itos = {i: ch for ch, i in stoi.items()}\n",
"vocab_size = len(itos)\n",
"print(\"vocab_size:\", vocab_size)\n",
"print(\"stoi:\", stoi)"
]
},
{
"cell_type": "markdown",
"id": "d7a9d8ab",
"metadata": {},
"source": [
"### 1.1 The context window (`block_size`)\n",
"\n",
"`block_size` = how many previous letters the model sees before guessing the next one. With\n",
"`block_size = 3`, predicting the letters of `emma` looks like this:\n",
"\n",
"```\n",
"... -> e (start: three dots, predict 'e')\n",
"..e -> m\n",
".em -> m\n",
"emm -> a\n",
"mma -> . (predict the end marker)\n",
"```\n",
"\n",
"Each arrow is one training example. Real-life picture: a sliding window of 3 letters moves\n",
"left to right across the word; at each stop the model must guess what comes next."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4c0464a0",
"metadata": {},
"outputs": [],
"source": [
"block_size = 3\n",
"\n",
"def build_dataset(word_list):\n",
" X, Y = [], []\n",
" for w in word_list:\n",
" context = [0] * block_size # start with '...'\n",
" for ch in w + \".\":\n",
" ix = stoi[ch]\n",
" X.append(context)\n",
" Y.append(ix)\n",
" context = context[1:] + [ix] # slide the window\n",
" return torch.tensor(X), torch.tensor(Y)\n",
"\n",
"# show the examples for one name\n",
"print(\"examples generated from 'emma':\")\n",
"Xtmp, Ytmp = build_dataset([\"emma\"])\n",
"for ctx, target in zip(Xtmp.tolist(), Ytmp.tolist()):\n",
" print(\"\".join(itos[i] for i in ctx), \"->\", itos[target])"
]
},
{
"cell_type": "markdown",
"id": "5f607e40",
"metadata": {},
"source": [
"### 1.2 Train / dev / test split\n",
"\n",
"We shuffle the names and cut three piles:\n",
"\n",
"- **train (80%)** — the model learns from these.\n",
"- **dev (10%)** — we tune choices (size, learning rate) against these.\n",
"- **test (10%)** — touched only at the very end, to get an honest score.\n",
"\n",
"Real-life picture: train = homework you study, dev = practice quiz you check yourself against,\n",
"test = the final exam you only sit once."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5c8e4f0c",
"metadata": {},
"outputs": [],
"source": [
"random.seed(42)\n",
"random.shuffle(words)\n",
"n1 = int(0.8 * len(words))\n",
"n2 = int(0.9 * len(words))\n",
"\n",
"Xtr, Ytr = build_dataset(words[:n1])\n",
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
"Xte, Yte = build_dataset(words[n2:])\n",
"\n",
"print(\"train:\", tuple(Xtr.shape), \"dev:\", tuple(Xdev.shape), \"test:\", tuple(Xte.shape))"
]
},
{
"cell_type": "markdown",
"id": "b10d4c19",
"metadata": {},
"source": [
"### 2.1 The model — embedding, hidden layer, output\n",
"\n",
"Three learnable pieces:\n",
"\n",
"1. **`C`** — the embedding table: each of the 27 symbols gets an `n_embd`-dim vector.\n",
"2. **`W1, b1`** — a hidden layer with `tanh`: it mixes the 3 letter-vectors into `n_hidden`\n",
" features. Analogy: a committee that reads the 3 personalities and forms opinions.\n",
"3. **`W2, b2`** — the output layer: turns those opinions into `vocab_size` scores (logits) for\n",
" the next letter.\n",
"\n",
"**-> Training:** `parameters` is the full list of knobs; every one gets a gradient and a nudge."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a0bcc2f",
"metadata": {},
"outputs": [],
"source": [
"g = torch.Generator().manual_seed(2147483647)\n",
"\n",
"n_embd = 10 # size of each letter's vector\n",
"n_hidden = 200 # neurons in the hidden layer\n",
"\n",
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * 0.2\n",
"b1 = torch.randn(n_hidden, generator=g) * 0.01\n",
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
"b2 = torch.randn(vocab_size, generator=g) * 0.0\n",
"\n",
"parameters = [C, W1, b1, W2, b2]\n",
"for p in parameters:\n",
" p.requires_grad = True\n",
"\n",
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
]
},
{
"cell_type": "markdown",
"id": "77778ae3",
"metadata": {},
"source": [
"### 2.2 One forward pass, step by step\n",
"\n",
"Watch the shapes flow. The key move is `emb.view(-1, n_embd*block_size)`: it **flattens** the\n",
"3 letter-vectors of each example into one long row so the hidden layer can read them together.\n",
"\n",
"**-> Training:** `F.cross_entropy` = softmax + negative-log-likelihood in one numerically-stable\n",
"call; lower means more probability on the true next letter."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13123079",
"metadata": {},
"outputs": [],
"source": [
"ix = torch.randint(0, Xtr.shape[0], (4,), generator=g) # 4 random examples\n",
"Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
"emb = C[Xb] # (4, 3, n_embd)\n",
"flat = emb.view(-1, n_embd * block_size) # (4, 30)\n",
"h = torch.tanh(flat @ W1 + b1) # (4, n_hidden)\n",
"logits = h @ W2 + b2 # (4, vocab_size)\n",
"loss = F.cross_entropy(logits, Yb)\n",
"\n",
"print(\"emb :\", tuple(emb.shape))\n",
"print(\"flat :\", tuple(flat.shape))\n",
"print(\"h :\", tuple(h.shape))\n",
"print(\"logits:\", tuple(logits.shape))\n",
"print(\"loss :\", loss.item())"
]
},
{
"cell_type": "markdown",
"id": "b90331ba",
"metadata": {},
"source": [
"### 3.1 Finding a good learning rate\n",
"\n",
"The learning rate (how big each nudge is) matters a lot: too small and training crawls, too\n",
"big and the loss bounces or explodes. A cheap way to find a good one: sweep the rate from tiny\n",
"to large over a few hundred steps and watch where the loss drops fastest.\n",
"\n",
"Real-life picture: testing how big a step to take downhill in fog — tiny steps are safe but\n",
"slow; huge steps overshoot the valley.\n",
"\n",
"We use a **fresh** set of weights here so the search does not disturb the real model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acae02ad",
"metadata": {},
"outputs": [],
"source": [
"# fresh throwaway weights just for the search\n",
"gs = torch.Generator().manual_seed(2147483647)\n",
"Cs = torch.randn((vocab_size, n_embd), generator=gs)\n",
"W1s = torch.randn((n_embd * block_size, n_hidden), generator=gs) * 0.2\n",
"b1s = torch.randn(n_hidden, generator=gs) * 0.01\n",
"W2s = torch.randn((n_hidden, vocab_size), generator=gs) * 0.01\n",
"b2s = torch.randn(vocab_size, generator=gs) * 0.0\n",
"params_s = [Cs, W1s, b1s, W2s, b2s]\n",
"for p in params_s:\n",
" p.requires_grad = True\n",
"\n",
"# candidate learning rates spaced from 10^-3 up to 10^0\n",
"lre = torch.linspace(-3, 0, 1000)\n",
"lrs = 10 ** lre\n",
"lr_used, losses = [], []\n",
"\n",
"for i in range(1000):\n",
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=gs)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
" emb = Cs[Xb]\n",
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1s + b1s)\n",
" loss = F.cross_entropy(h @ W2s + b2s, Yb)\n",
" for p in params_s:\n",
" p.grad = None\n",
" loss.backward()\n",
" lr = lrs[i].item()\n",
" for p in params_s:\n",
" p.data += -lr * p.grad\n",
" lr_used.append(lre[i].item())\n",
" losses.append(loss.item())\n",
"\n",
"best = lre[min(range(len(losses)), key=lambda k: losses[k])].item()\n",
"print(f\"loss dropped fastest around 10^{best:.2f} ~= {10**best:.3f}\")\n",
"print(\"good practical choice: lr = 0.1\")\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(5, 4))\n",
" plt.plot(lr_used, losses)\n",
" plt.xlabel(\"log10(learning rate)\")\n",
" plt.ylabel(\"minibatch loss\")\n",
" plt.title(\"learning-rate sweep\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "13569252",
"metadata": {},
"source": [
"### 4.1 Train the model\n",
"\n",
"Now the real run on the actual `parameters`. Each step:\n",
"\n",
"1. grab a random **minibatch** of 32 examples (fast, noisy, good enough)\n",
"2. forward -> loss\n",
"3. `loss.backward()` -> gradients\n",
"4. nudge every parameter downhill\n",
"\n",
"We start at `lr = 0.1` and **decay** to `0.01` near the end to settle into the valley."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c5ec35d5",
"metadata": {},
"outputs": [],
"source": [
"max_steps = 20000\n",
"batch_size = 32\n",
"lossi = []\n",
"\n",
"for i in range(max_steps):\n",
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
" emb = C[Xb]\n",
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
" logits = h @ W2 + b2\n",
" loss = F.cross_entropy(logits, Yb)\n",
"\n",
" for p in parameters:\n",
" p.grad = None\n",
" loss.backward()\n",
"\n",
" lr = 0.1 if i < 15000 else 0.01\n",
" for p in parameters:\n",
" p.data += -lr * p.grad\n",
"\n",
" lossi.append(loss.log10().item())\n",
" if i % 4000 == 0 or i == max_steps - 1:\n",
" print(f\"step {i:5d}/{max_steps} minibatch loss {loss.item():.4f}\")\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(5, 4))\n",
" plt.plot(lossi)\n",
" plt.xlabel(\"step\")\n",
" plt.ylabel(\"log10(loss)\")\n",
" plt.title(\"training loss\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "e6baab73",
"metadata": {},
"source": [
"### 4.2 Honest scores on train and dev\n",
"\n",
"Minibatch loss is noisy. Evaluate on the **whole** train and dev sets for a stable number.\n",
"If dev loss is much worse than train, the model is memorising; if they are close, it is\n",
"generalising. (A plain bigram on names scores around 2.45 — we should beat that.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "579fe1c8",
"metadata": {},
"outputs": [],
"source": [
"@torch.no_grad()\n",
"def split_loss(split):\n",
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
" emb = C[x]\n",
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
" logits = h @ W2 + b2\n",
" return F.cross_entropy(logits, y).item()\n",
"\n",
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
]
},
{
"cell_type": "markdown",
"id": "054682f1",
"metadata": {},
"source": [
"### 5.1 Generate new names\n",
"\n",
"Sampling, letter by letter: start from `...`, predict a distribution, **draw** one letter,\n",
"slide the window, repeat until the model emits the end marker `.`.\n",
"\n",
"These are brand-new names the model invented — not copied from the file."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06bf121c",
"metadata": {},
"outputs": [],
"source": [
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
"\n",
"for _ in range(20):\n",
" out = []\n",
" context = [0] * block_size\n",
" while True:\n",
" emb = C[torch.tensor([context])]\n",
" h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
" logits = h @ W2 + b2\n",
" probs = F.softmax(logits, dim=1)\n",
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
" context = context[1:] + [ix]\n",
" if ix == 0:\n",
" break\n",
" out.append(itos[ix])\n",
" print(\"\".join(out))"
]
},
{
"cell_type": "markdown",
"id": "0a0de526",
"metadata": {},
"source": [
"**Your turn 5** — Re-run section 5.1 a few times (different random draws) and read the\n",
"names. Then try raising `n_hidden` (e.g. 300) or `block_size` (e.g. 4 or 5) back in section 2,\n",
"retrain, and see whether dev loss improves and the names look more realistic."
]
},
{
"cell_type": "markdown",
"id": "bc2003e7",
"metadata": {},
"source": [
"### 6.1 Bonus — visualise letters in 2-D\n",
"\n",
"When the embedding is 2-D we can literally plot it. We quickly train a tiny `n_embd = 2`\n",
"version and scatter the 27 symbols. Watch the **vowels** drift into their own neighbourhood —\n",
"the network discovered that they play a similar role, without ever being told what a vowel is.\n",
"\n",
"(Optional: needs matplotlib. Smaller/shorter than the main model, so the clustering is rough.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7786f75a",
"metadata": {},
"outputs": [],
"source": [
"g3 = torch.Generator().manual_seed(2147483647)\n",
"C2 = torch.randn((vocab_size, 2), generator=g3)\n",
"W1b = torch.randn((2 * block_size, n_hidden), generator=g3) * 0.2\n",
"b1b = torch.randn(n_hidden, generator=g3) * 0.01\n",
"W2b = torch.randn((n_hidden, vocab_size), generator=g3) * 0.01\n",
"b2b = torch.randn(vocab_size, generator=g3) * 0.0\n",
"params2 = [C2, W1b, b1b, W2b, b2b]\n",
"for p in params2:\n",
" p.requires_grad = True\n",
"\n",
"for i in range(8000):\n",
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=g3)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
" emb = C2[Xb]\n",
" h = torch.tanh(emb.view(-1, 2 * block_size) @ W1b + b1b)\n",
" loss = F.cross_entropy(h @ W2b + b2b, Yb)\n",
" for p in params2:\n",
" p.grad = None\n",
" loss.backward()\n",
" for p in params2:\n",
" p.data += -0.1 * p.grad\n",
"\n",
"print(\"2-D model dev-ish minibatch loss:\", round(loss.item(), 4))\n",
"\n",
"if HAS_PLT:\n",
" pts = C2.detach()\n",
" plt.figure(figsize=(6, 6))\n",
" plt.scatter(pts[:, 0], pts[:, 1], s=300)\n",
" for i in range(vocab_size):\n",
" plt.text(pts[i, 0].item(), pts[i, 1].item(), itos[i],\n",
" ha=\"center\", va=\"center\", color=\"white\")\n",
" plt.title(\"learned 2-D letter embedding (look for the vowels clustering)\")\n",
" plt.grid(True)\n",
" plt.show()\n",
"else:\n",
" print(\"matplotlib missing - skipping the embedding scatter\")"
]
},
{
"cell_type": "markdown",
"id": "c5e5b7a7",
"metadata": {},
"source": [
"## What's next\n",
"\n",
"You went from \"one letter back\" (bigram) to a real neural net that reads a **window** of\n",
"letters through a learned **embedding**, and it makes noticeably better names.\n",
"\n",
"Where this leads in the series:\n",
"\n",
"- `03_batchnorm_activations.ipynb` — why deep nets are fragile, and how BatchNorm keeps the\n",
" signal healthy so we can train bigger models.\n",
"- `04_backprop_ninja.ipynb` — compute all these gradients **by hand**, cementing notebook `00`.\n",
"- `06_build_gpt_attention.ipynb` — replace the fixed window with **attention**, so the model\n",
" can look back as far as it needs.\n",
"\n",
"**Checklist**\n",
"- [ ] Names -> context windows (`block_size`) -> (X, Y)\n",
"- [ ] train / dev / test split\n",
"- [ ] embedding `C` + hidden `tanh` + output logits\n",
"- [ ] cross-entropy loss, minibatch training\n",
"- [ ] learning-rate search\n",
"- [ ] sampled new names; visualised the 2-D embedding"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03_batchnorm_activations.ipynb
+496
View File
@@ -0,0 +1,496 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "aedd6523",
"metadata": {},
"source": [
"# makemore — activations, gradients, and BatchNorm\n",
"\n",
"Make the MLP **trainable and healthy** — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **03**. In `02` the MLP worked, but we got a bit lucky with how the weights\n",
"started. Here we look *inside* the network at the numbers flowing through it, see two common\n",
"ways training goes wrong, and fix them — first by hand (good initialisation), then with the\n",
"robust tool everyone uses: **Batch Normalization**. Follows Karpathy's *\"makemore part 3\"*."
]
},
{
"cell_type": "markdown",
"id": "2980029a",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"A neural net is a stack of layers. At each layer numbers flow forward (activations) and\n",
"gradients flow backward. If those numbers get **too big** or **too small**, training stalls.\n",
"\n",
"Two failures we will see and fix:\n",
"\n",
"1. **Confidently wrong at the start.** If the output layer starts with large random weights,\n",
" the model makes wild over-confident guesses, so the first loss is huge and the first many\n",
" steps are wasted just calming it down.\n",
"2. **Saturated `tanh`.** If the hidden numbers are too spread out, `tanh` slams them to -1 or\n",
" +1. There the curve is flat, so its gradient is ~0 and those neurons stop learning — they\n",
" are \"dead.\"\n",
"\n",
"Real-life picture: think of each layer as an audio stage. Too quiet and you hear nothing; too\n",
"loud and everything **clips** into distortion. We want every stage at a healthy volume.\n",
"**BatchNorm** is the auto-leveler that keeps the volume right at every stage."
]
},
{
"cell_type": "markdown",
"id": "18a296f3",
"metadata": {},
"source": [
"### 0.1 Imports\n",
"\n",
"`torch` first (avoids a Windows OpenMP DLL clash), matplotlib optional for the diagnostics."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac69da81",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
"\n",
"import math\n",
"import random\n",
"import torch\n",
"import torch.nn.functional as F\n",
"\n",
"try:\n",
" import matplotlib.pyplot as plt\n",
" HAS_PLT = True\n",
"except Exception:\n",
" HAS_PLT = False\n",
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
"\n",
"print(\"PyTorch version:\", torch.__version__)"
]
},
{
"cell_type": "markdown",
"id": "92143322",
"metadata": {},
"source": [
"### 0.2 Data — same names, same context windows as notebook 02"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d99c17df",
"metadata": {},
"outputs": [],
"source": [
"words = open(\"names.txt\").read().splitlines()\n",
"chars = sorted(list(set(\"\".join(words))))\n",
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
"stoi[\".\"] = 0\n",
"itos = {i: ch for ch, i in stoi.items()}\n",
"vocab_size = len(itos)\n",
"block_size = 3\n",
"\n",
"def build_dataset(word_list):\n",
" X, Y = [], []\n",
" for w in word_list:\n",
" context = [0] * block_size\n",
" for ch in w + \".\":\n",
" ix = stoi[ch]\n",
" X.append(context)\n",
" Y.append(ix)\n",
" context = context[1:] + [ix]\n",
" return torch.tensor(X), torch.tensor(Y)\n",
"\n",
"random.seed(42)\n",
"random.shuffle(words)\n",
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
"Xtr, Ytr = build_dataset(words[:n1])\n",
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
"Xte, Yte = build_dataset(words[n2:])\n",
"print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
]
},
{
"cell_type": "markdown",
"id": "36b836bc",
"metadata": {},
"source": [
"### 1.1 Problem 1 — confidently wrong at the start\n",
"\n",
"At step 0 the model knows nothing, so the *best* it can do is spread probability evenly over\n",
"all 27 symbols. That ideal first loss is `-ln(1/27) = ln(27) ~= 3.30`.\n",
"\n",
"But with large random output weights the logits are all over the place, so the first loss is\n",
"much higher. Let's measure it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1219a887",
"metadata": {},
"outputs": [],
"source": [
"g0 = torch.Generator().manual_seed(2147483647)\n",
"n_embd, n_hidden = 10, 200\n",
"\n",
"# deliberately naive (large) initialisation\n",
"Cn = torch.randn((vocab_size, n_embd), generator=g0)\n",
"W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
"b1n = torch.randn(n_hidden, generator=g0)\n",
"W2n = torch.randn((n_hidden, vocab_size), generator=g0)\n",
"b2n = torch.randn(vocab_size, generator=g0)\n",
"\n",
"emb = Cn[Xtr[:32]]\n",
"h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
"logits = h @ W2n + b2n\n",
"print(\"naive initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
"print(\"ideal (ln 27) :\", math.log(27))"
]
},
{
"cell_type": "markdown",
"id": "4c418501",
"metadata": {},
"source": [
"### 1.2 Fix 1 — make the output layer humble at the start\n",
"\n",
"We want the starting logits near zero (no opinion yet). So scale `W2` **down** and set `b2`\n",
"to zero. Now the first loss sits right near the ideal `~3.30`, and no steps are wasted."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1b9bb4a",
"metadata": {},
"outputs": [],
"source": [
"g0 = torch.Generator().manual_seed(2147483647)\n",
"Cn = torch.randn((vocab_size, n_embd), generator=g0)\n",
"W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
"b1n = torch.randn(n_hidden, generator=g0)\n",
"W2n = torch.randn((n_hidden, vocab_size), generator=g0) * 0.01 # tiny\n",
"b2n = torch.randn(vocab_size, generator=g0) * 0.0 # zero\n",
"\n",
"emb = Cn[Xtr[:32]]\n",
"h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
"logits = h @ W2n + b2n\n",
"print(\"fixed initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
"print(\"ideal (ln 27) :\", math.log(27))"
]
},
{
"cell_type": "markdown",
"id": "2ab522f7",
"metadata": {},
"source": [
"### 2.1 Problem 2 — saturated `tanh` (dead neurons)\n",
"\n",
"Now look at the hidden activations `h = tanh(...)`. With large `W1`, the inputs to `tanh` are\n",
"very spread out, so most outputs are pinned at -1 or +1. That is the flat part of `tanh`,\n",
"where the gradient is ~0 — those neurons learn nothing.\n",
"\n",
"We measure the fraction of \"saturated\" activations (|h| > 0.99) and, if matplotlib is present,\n",
"show the tell-tale picture: bright cells are saturated; a fully bright column = a dead neuron."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8fd25db6",
"metadata": {},
"outputs": [],
"source": [
"emb = Cn[Xtr[:1000]]\n",
"hpre = emb.view(-1, n_embd * block_size) @ W1n + b1n # pre-activation\n",
"h = torch.tanh(hpre)\n",
"sat = (h.abs() > 0.99).float().mean().item()\n",
"print(f\"fraction of saturated activations: {sat:.1%} (high = bad)\")\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(10, 3))\n",
" plt.imshow(h.abs() > 0.99, cmap=\"gray\", aspect=\"auto\")\n",
" plt.title(\"white = saturated (|tanh| > 0.99); a full white column = a dead neuron\")\n",
" plt.xlabel(\"neuron\")\n",
" plt.ylabel(\"example\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "e11aee42",
"metadata": {},
"source": [
"### 2.2 Fix 2 — scale the hidden layer (Kaiming init)\n",
"\n",
"The cure is to make `W1` smaller so the pre-activations have a sensible spread (roughly unit\n",
"standard deviation). The principled recipe is **Kaiming init**: multiply random weights by\n",
"`gain / sqrt(fan_in)`, where `fan_in` is the number of inputs and the `gain` for `tanh` is\n",
"`5/3`.\n",
"\n",
"Real-life picture: setting the input volume so the signal fills the range without clipping."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "71f808d3",
"metadata": {},
"outputs": [],
"source": [
"fan_in = n_embd * block_size\n",
"W1k = torch.randn((fan_in, n_hidden), generator=torch.Generator().manual_seed(1)) * ((5/3) / fan_in**0.5)\n",
"b1k = torch.zeros(n_hidden)\n",
"\n",
"emb = Cn[Xtr[:1000]]\n",
"h = torch.tanh(emb.view(-1, fan_in) @ W1k + b1k)\n",
"sat = (h.abs() > 0.99).float().mean().item()\n",
"print(f\"saturated now: {sat:.1%} (much lower)\")\n",
"print(f\"activation std: {h.std().item():.3f} (we want it near ~0.6-0.7 for tanh)\")\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(5, 3))\n",
" plt.hist(h.view(-1).tolist(), bins=50)\n",
" plt.title(\"healthy tanh activations (not all piled at +/-1)\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "50d6f683",
"metadata": {},
"source": [
"### 3.1 BatchNorm — the auto-leveler\n",
"\n",
"Hand-tuning `W1` works, but in deep nets it is fiddly. **Batch Normalization** does it\n",
"automatically: before `tanh`, it **normalises** the pre-activations so that, across the\n",
"current batch, each neuron has mean 0 and std 1. Then it lets the network rescale/shift them\n",
"with two learnable knobs, `gamma` (gain) and `beta` (bias), in case 0/1 is not ideal.\n",
"\n",
"Real-life picture: an audio compressor that auto-levels every stage to a healthy volume, with\n",
"a manual trim (`gamma`, `beta`) if the engineer wants to nudge it.\n",
"\n",
"One catch: at test time we have no batch to average over, so during training we also keep a\n",
"**running** mean/std to use later for single predictions."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5aeb780e",
"metadata": {},
"outputs": [],
"source": [
"g = torch.Generator().manual_seed(2147483647)\n",
"n_embd, n_hidden = 10, 200\n",
"fan_in = n_embd * block_size\n",
"\n",
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
"W1 = torch.randn((fan_in, n_hidden), generator=g) * ((5/3) / fan_in**0.5)\n",
"# no b1: BatchNorm's beta plays that role\n",
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
"b2 = torch.randn(vocab_size, generator=g) * 0.0\n",
"\n",
"bngain = torch.ones((1, n_hidden))\n",
"bnbias = torch.zeros((1, n_hidden))\n",
"bnmean_running = torch.zeros((1, n_hidden))\n",
"bnstd_running = torch.ones((1, n_hidden))\n",
"\n",
"parameters = [C, W1, W2, b2, bngain, bnbias]\n",
"for p in parameters:\n",
" p.requires_grad = True\n",
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
]
},
{
"cell_type": "markdown",
"id": "2b07d16a",
"metadata": {},
"source": [
"### 3.2 Train with BatchNorm\n",
"\n",
"Each step normalises the pre-activations using the **current batch**, updates the running\n",
"stats (for later), then continues as usual. Watch the loss fall, starting near the ideal 3.3."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2410e7c4",
"metadata": {},
"outputs": [],
"source": [
"max_steps = 20000\n",
"batch_size = 32\n",
"lossi = []\n",
"\n",
"for i in range(max_steps):\n",
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
" emb = C[Xb]\n",
" hpreact = emb.view(-1, fan_in) @ W1\n",
"\n",
" # --- BatchNorm ---\n",
" bnmeani = hpreact.mean(0, keepdim=True)\n",
" bnstdi = hpreact.std(0, keepdim=True)\n",
" hpreact = bngain * (hpreact - bnmeani) / bnstdi + bnbias\n",
" with torch.no_grad():\n",
" bnmean_running = 0.999 * bnmean_running + 0.001 * bnmeani\n",
" bnstd_running = 0.999 * bnstd_running + 0.001 * bnstdi\n",
" # -----------------\n",
"\n",
" h = torch.tanh(hpreact)\n",
" logits = h @ W2 + b2\n",
" loss = F.cross_entropy(logits, Yb)\n",
"\n",
" for p in parameters:\n",
" p.grad = None\n",
" loss.backward()\n",
" lr = 0.1 if i < 15000 else 0.01\n",
" for p in parameters:\n",
" p.data += -lr * p.grad\n",
"\n",
" lossi.append(loss.log10().item())\n",
" if i % 4000 == 0 or i == max_steps - 1:\n",
" print(f\"step {i:5d}/{max_steps} loss {loss.item():.4f}\")\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(5, 4))\n",
" plt.plot(lossi)\n",
" plt.xlabel(\"step\"); plt.ylabel(\"log10(loss)\"); plt.title(\"training loss (with BatchNorm)\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "d80dccf6",
"metadata": {},
"source": [
"### 3.3 Evaluate — using the running stats\n",
"\n",
"At evaluation we have no batch, so we normalise with the **running** mean/std collected during\n",
"training. Train and dev loss should be close (good generalisation) and beat the bigram (~2.45)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "30ae8013",
"metadata": {},
"outputs": [],
"source": [
"@torch.no_grad()\n",
"def split_loss(split):\n",
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
" emb = C[x]\n",
" hpreact = emb.view(-1, fan_in) @ W1\n",
" hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
" h = torch.tanh(hpreact)\n",
" logits = h @ W2 + b2\n",
" return F.cross_entropy(logits, y).item()\n",
"\n",
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
]
},
{
"cell_type": "markdown",
"id": "269c8943",
"metadata": {},
"source": [
"### 4.1 Generate names with the BatchNorm model\n",
"\n",
"Same letter-by-letter sampling as before, but normalising with the running stats."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "63711cd1",
"metadata": {},
"outputs": [],
"source": [
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
"\n",
"for _ in range(20):\n",
" out = []\n",
" context = [0] * block_size\n",
" while True:\n",
" emb = C[torch.tensor([context])]\n",
" hpreact = emb.view(1, -1) @ W1\n",
" hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
" h = torch.tanh(hpreact)\n",
" logits = h @ W2 + b2\n",
" probs = F.softmax(logits, dim=1)\n",
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
" context = context[1:] + [ix]\n",
" if ix == 0:\n",
" break\n",
" out.append(itos[ix])\n",
" print(\"\".join(out))"
]
},
{
"cell_type": "markdown",
"id": "365be7ef",
"metadata": {},
"source": [
"**Your turn 4** — In section 2.1, change `W1n`'s scale (e.g. `* 0.1` vs `* 3.0`) and re-run\n",
"the saturation check. Bigger scale -> more dead neurons. Then confirm BatchNorm (section 3.2)\n",
"trains fine even from a less careful init, because it re-levels the signal every step."
]
},
{
"cell_type": "markdown",
"id": "65b60eec",
"metadata": {},
"source": [
"## What's next\n",
"\n",
"You learned to look **inside** the network and keep it healthy:\n",
"\n",
"- start the output layer humble (good first loss)\n",
"- keep `tanh` out of saturation (Kaiming init)\n",
"- or just use **BatchNorm** to auto-level every layer\n",
"\n",
"Where this leads in the series:\n",
"\n",
"- `04_backprop_ninja.ipynb` — compute every gradient here **by hand**, including through\n",
" BatchNorm, to truly own backprop.\n",
"- `05_wavenet.ipynb` — stack these healthy layers into a deeper, hierarchical model.\n",
"- `06_build_gpt_attention.ipynb` — the transformer, which uses LayerNorm (a cousin of\n",
" BatchNorm) for the same \"keep the signal healthy\" reason.\n",
"\n",
"**Checklist**\n",
"- [ ] Ideal initial loss = ln(vocab_size)\n",
"- [ ] Humble output init (small W2, zero b2)\n",
"- [ ] Saturation / dead neurons diagnosed\n",
"- [ ] Kaiming init (gain / sqrt(fan_in))\n",
"- [ ] BatchNorm: per-batch normalise + gamma/beta + running stats\n",
"- [ ] Trained, evaluated, sampled names"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04_backprop_ninja.ipynb
+529
View File
@@ -0,0 +1,529 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "6caa1908",
"metadata": {},
"source": [
"# Becoming a backprop ninja\n",
"\n",
"Compute every gradient **by hand** — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **04**. So far we have leaned on `loss.backward()` to do the calculus for us.\n",
"Here we turn it off and backpropagate **manually** through the whole MLP + BatchNorm +\n",
"cross-entropy, checking each result against PyTorch. After this, autograd is never a mystery.\n",
"Follows Karpathy's *\"makemore part 4: becoming a backprop ninja\"*."
]
},
{
"cell_type": "markdown",
"id": "9ed68a60",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"`loss.backward()` answers one question for every weight: *if I nudge you a little, how does\n",
"the loss change?* That answer is the gradient. We have trusted PyTorch to compute it.\n",
"\n",
"Now we earn it. We will:\n",
"\n",
"1. Run the forward pass but **save every intermediate value**.\n",
"2. Walk backward, one operation at a time, applying the chain rule by hand (exactly the\n",
" `00_micrograd.ipynb` idea, now on whole tensors).\n",
"3. Compare each hand-derived gradient to PyTorch's — they should match.\n",
"\n",
"Real-life picture: notebook `00` showed one gear; here we backprop through the whole gearbox,\n",
"and check our work against the manufacturer's spec (PyTorch)."
]
},
{
"cell_type": "markdown",
"id": "95e96905",
"metadata": {},
"source": [
"### 0.1 Imports and data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b29d3f0",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
"\n",
"import random\n",
"import torch\n",
"import torch.nn.functional as F\n",
"\n",
"words = open(\"names.txt\").read().splitlines()\n",
"chars = sorted(list(set(\"\".join(words))))\n",
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
"stoi[\".\"] = 0\n",
"itos = {i: ch for ch, i in stoi.items()}\n",
"vocab_size = len(itos)\n",
"block_size = 3\n",
"\n",
"def build_dataset(word_list):\n",
" X, Y = [], []\n",
" for w in word_list:\n",
" context = [0] * block_size\n",
" for ch in w + \".\":\n",
" ix = stoi[ch]\n",
" X.append(context); Y.append(ix)\n",
" context = context[1:] + [ix]\n",
" return torch.tensor(X), torch.tensor(Y)\n",
"\n",
"random.seed(42)\n",
"random.shuffle(words)\n",
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
"Xtr, Ytr = build_dataset(words[:n1])\n",
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
"print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
]
},
{
"cell_type": "markdown",
"id": "19612a96",
"metadata": {},
"source": [
"### 0.2 The gradient-check helper\n",
"\n",
"`cmp` compares our hand-computed gradient `dt` to PyTorch's `t.grad`:\n",
"\n",
"- **exact** = bit-for-bit identical\n",
"- **approx** = equal within tiny floating-point tolerance (this is the one that matters)\n",
"- **maxdiff** = largest single difference"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c6c9ccf",
"metadata": {},
"outputs": [],
"source": [
"def cmp(name, dt, t):\n",
" ex = torch.all(dt == t.grad).item()\n",
" app = torch.allclose(dt, t.grad, rtol=1e-4, atol=1e-7)\n",
" maxdiff = (dt - t.grad).abs().max().item()\n",
" print(f\"{name:12s} | exact: {str(ex):5s} | approx: {str(app):5s} | maxdiff: {maxdiff:.2e}\")"
]
},
{
"cell_type": "markdown",
"id": "ed7b6dec",
"metadata": {},
"source": [
"### 1.1 Parameters\n",
"\n",
"We initialise the same MLP + BatchNorm as notebook `03`. Biases are made small but non-zero\n",
"so every gradient path is actually exercised (a zero would hide bugs)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "70e88d72",
"metadata": {},
"outputs": [],
"source": [
"n_embd, n_hidden = 10, 64\n",
"g = torch.Generator().manual_seed(2147483647)\n",
"\n",
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
"b1 = torch.randn(n_hidden, generator=g) * 0.1\n",
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.1\n",
"b2 = torch.randn(vocab_size, generator=g) * 0.1\n",
"bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
"bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
"\n",
"parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
"for p in parameters:\n",
" p.requires_grad = True\n",
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
]
},
{
"cell_type": "markdown",
"id": "04fa6a1e",
"metadata": {},
"source": [
"### 1.2 Forward pass, broken into tiny steps\n",
"\n",
"Cross-entropy is written out by hand (max-subtract, exp, sum, log) so we can backprop through\n",
"each piece. Every intermediate calls `retain_grad()` so PyTorch will store its gradient for us\n",
"to compare against."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ad34b3b",
"metadata": {},
"outputs": [],
"source": [
"n = 32 # fixed batch size for the manual math\n",
"ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
"Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
"emb = C[Xb] # embed the characters\n",
"embcat = emb.view(emb.shape[0], -1) # flatten\n",
"# linear 1\n",
"hprebn = embcat @ W1 + b1\n",
"# batchnorm\n",
"bnmeani = 1 / n * hprebn.sum(0, keepdim=True)\n",
"bndiff = hprebn - bnmeani\n",
"bndiff2 = bndiff ** 2\n",
"bnvar = 1 / (n - 1) * bndiff2.sum(0, keepdim=True) # Bessel's correction (n-1)\n",
"bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
"bnraw = bndiff * bnvar_inv\n",
"hpreact = bngain * bnraw + bnbias\n",
"# nonlinearity\n",
"h = torch.tanh(hpreact)\n",
"# linear 2\n",
"logits = h @ W2 + b2\n",
"# cross-entropy (manual)\n",
"logit_maxes = logits.max(1, keepdim=True).values\n",
"norm_logits = logits - logit_maxes\n",
"counts = norm_logits.exp()\n",
"counts_sum = counts.sum(1, keepdim=True)\n",
"counts_sum_inv = counts_sum ** -1\n",
"probs = counts * counts_sum_inv\n",
"logprobs = probs.log()\n",
"loss = -logprobs[range(n), Yb].mean()\n",
"\n",
"for t in [logprobs, probs, counts, counts_sum, counts_sum_inv, norm_logits, logit_maxes,\n",
" logits, h, hpreact, bnraw, bnvar_inv, bnvar, bndiff2, bndiff, bnmeani, hprebn,\n",
" embcat, emb]:\n",
" t.retain_grad()\n",
"for p in parameters:\n",
" p.retain_grad()\n",
"loss.backward()\n",
"print(\"loss:\", loss.item())"
]
},
{
"cell_type": "markdown",
"id": "6fcbe6af",
"metadata": {},
"source": [
"### 2.1 Backprop through cross-entropy\n",
"\n",
"Walk backward from `loss`. For each line of the forward pass we write the local derivative and\n",
"multiply by the gradient coming from above (the chain rule). Read each line as: *how does this\n",
"intermediate affect the loss?*"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c7bc68e2",
"metadata": {},
"outputs": [],
"source": [
"dlogprobs = torch.zeros_like(logprobs)\n",
"dlogprobs[range(n), Yb] = -1.0 / n # loss = -mean of the chosen logprobs\n",
"dprobs = (1.0 / probs) * dlogprobs # logprobs = log(probs)\n",
"dcounts_sum_inv = (counts * dprobs).sum(1, keepdim=True)\n",
"dcounts = counts_sum_inv * dprobs # probs = counts * counts_sum_inv\n",
"dcounts_sum = (-counts_sum ** -2) * dcounts_sum_inv\n",
"dcounts += torch.ones_like(counts) * dcounts_sum\n",
"dnorm_logits = counts * dcounts # counts = exp(norm_logits)\n",
"dlogits = dnorm_logits.clone()\n",
"dlogit_maxes = (-dnorm_logits).sum(1, keepdim=True)\n",
"dlogits += F.one_hot(logits.max(1).indices, num_classes=logits.shape[1]) * dlogit_maxes\n",
"\n",
"cmp(\"logprobs\", dlogprobs, logprobs)\n",
"cmp(\"probs\", dprobs, probs)\n",
"cmp(\"counts_sum_inv\", dcounts_sum_inv, counts_sum_inv)\n",
"cmp(\"counts_sum\", dcounts_sum, counts_sum)\n",
"cmp(\"counts\", dcounts, counts)\n",
"cmp(\"norm_logits\", dnorm_logits, norm_logits)\n",
"cmp(\"logit_maxes\", dlogit_maxes, logit_maxes)\n",
"cmp(\"logits\", dlogits, logits)"
]
},
{
"cell_type": "markdown",
"id": "51a6e8d7",
"metadata": {},
"source": [
"### 2.2 Backprop through the second linear layer and tanh\n",
"\n",
"`logits = h @ W2 + b2`. Matrix-multiply gradients follow the standard transpose rules; `tanh`\n",
"contributes its local derivative `1 - tanh(x)^2`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39f0e645",
"metadata": {},
"outputs": [],
"source": [
"dh = dlogits @ W2.T\n",
"dW2 = h.T @ dlogits\n",
"db2 = dlogits.sum(0)\n",
"dhpreact = (1.0 - h ** 2) * dh # tanh local derivative\n",
"\n",
"cmp(\"h\", dh, h)\n",
"cmp(\"W2\", dW2, W2)\n",
"cmp(\"b2\", db2, b2)\n",
"cmp(\"hpreact\", dhpreact, hpreact)"
]
},
{
"cell_type": "markdown",
"id": "a7ecc492",
"metadata": {},
"source": [
"### 2.3 Backprop through BatchNorm\n",
"\n",
"This is the famously fiddly part. We go through each step: the scale/shift, the normalisation,\n",
"the variance, and the mean. Note how the **mean and variance both depend on every example**, so\n",
"gradients fan out across the batch."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6f4f5d49",
"metadata": {},
"outputs": [],
"source": [
"dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
"dbnbias = dhpreact.sum(0, keepdim=True)\n",
"dbnraw = bngain * dhpreact\n",
"dbndiff = bnvar_inv * dbnraw\n",
"dbnvar_inv = (bndiff * dbnraw).sum(0, keepdim=True)\n",
"dbnvar = (-0.5 * (bnvar + 1e-5) ** -1.5) * dbnvar_inv\n",
"dbndiff2 = (1.0 / (n - 1)) * torch.ones_like(bndiff2) * dbnvar\n",
"dbndiff += (2 * bndiff) * dbndiff2\n",
"dhprebn = dbndiff.clone()\n",
"dbnmeani = (-dbndiff).sum(0)\n",
"dhprebn += 1.0 / n * (torch.ones_like(hprebn) * dbnmeani)\n",
"\n",
"cmp(\"bngain\", dbngain, bngain)\n",
"cmp(\"bnbias\", dbnbias, bnbias)\n",
"cmp(\"bnraw\", dbnraw, bnraw)\n",
"cmp(\"bnvar_inv\", dbnvar_inv, bnvar_inv)\n",
"cmp(\"bnvar\", dbnvar, bnvar)\n",
"cmp(\"bndiff2\", dbndiff2, bndiff2)\n",
"cmp(\"bndiff\", dbndiff, bndiff)\n",
"cmp(\"bnmeani\", dbnmeani, bnmeani)\n",
"cmp(\"hprebn\", dhprebn, hprebn)"
]
},
{
"cell_type": "markdown",
"id": "3be0bf67",
"metadata": {},
"source": [
"### 2.4 Backprop through the first linear layer and the embedding\n",
"\n",
"`hprebn = embcat @ W1 + b1`, then un-flatten back to `emb`, then scatter each row's gradient to\n",
"the right row of the embedding table `C` (an embedding lookup is just \"pick rows\", so its\n",
"backward is \"add the gradient back into those rows\")."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b1317637",
"metadata": {},
"outputs": [],
"source": [
"dembcat = dhprebn @ W1.T\n",
"dW1 = embcat.T @ dhprebn\n",
"db1 = dhprebn.sum(0)\n",
"demb = dembcat.view(emb.shape)\n",
"\n",
"dC = torch.zeros_like(C)\n",
"for k in range(Xb.shape[0]):\n",
" for j in range(Xb.shape[1]):\n",
" dC[Xb[k, j]] += demb[k, j]\n",
"\n",
"cmp(\"embcat\", dembcat, embcat)\n",
"cmp(\"W1\", dW1, W1)\n",
"cmp(\"b1\", db1, b1)\n",
"cmp(\"emb\", demb, emb)\n",
"cmp(\"C\", dC, C)"
]
},
{
"cell_type": "markdown",
"id": "3f6d3bff",
"metadata": {},
"source": [
"### 3.1 The elegant shortcuts\n",
"\n",
"Doing it step-by-step proves we understand it. But two of those backward passes collapse into\n",
"beautiful one-liners once you do the algebra:\n",
"\n",
"- **Cross-entropy:** `dlogits = softmax(logits)`, then subtract 1 from the true class, divide by n.\n",
" Intuition: push down whatever probability the model gave, pull up the correct class.\n",
"- **BatchNorm:** one expression for `dhprebn` (no need for all the intermediate d's).\n",
"\n",
"We verify both against PyTorch."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "91248dc5",
"metadata": {},
"outputs": [],
"source": [
"# elegant cross-entropy backward\n",
"dlogits_fast = F.softmax(logits, 1)\n",
"dlogits_fast[range(n), Yb] -= 1\n",
"dlogits_fast /= n\n",
"print(\"dlogits | approx:\", torch.allclose(dlogits_fast, logits.grad, rtol=1e-4, atol=1e-7),\n",
" \" | maxdiff:\", (dlogits_fast - logits.grad).abs().max().item())\n",
"\n",
"# elegant batchnorm backward\n",
"dhprebn_fast = bngain * bnvar_inv / n * (\n",
" n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
")\n",
"print(\"dhprebn | approx:\", torch.allclose(dhprebn_fast, hprebn.grad, rtol=1e-4, atol=1e-7),\n",
" \" | maxdiff:\", (dhprebn_fast - hprebn.grad).abs().max().item())"
]
},
{
"cell_type": "markdown",
"id": "09834b6c",
"metadata": {},
"source": [
"### 3.2 Train with 100% hand-written gradients\n",
"\n",
"The real proof: turn off autograd and train using only our manual gradients (the elegant\n",
"versions, plus the manual `W2/b2/W1/b1/C/bngain/bnbias`). If the loss falls, our calculus is\n",
"correct."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9bfb8004",
"metadata": {},
"outputs": [],
"source": [
"# fresh params\n",
"g = torch.Generator().manual_seed(2147483647)\n",
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
"b1 = torch.randn(n_hidden, generator=g) * 0.1\n",
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.1\n",
"b2 = torch.randn(vocab_size, generator=g) * 0.1\n",
"bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
"bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
"parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
"for p in parameters:\n",
" p.requires_grad = True\n",
"\n",
"n = 32\n",
"for i in range(8000):\n",
" ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
" # forward (keep what the manual backward needs)\n",
" emb = C[Xb]\n",
" embcat = emb.view(emb.shape[0], -1)\n",
" hprebn = embcat @ W1 + b1\n",
" bnmeani = hprebn.mean(0, keepdim=True)\n",
" bnvar = hprebn.var(0, keepdim=True, unbiased=True)\n",
" bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
" bnraw = (hprebn - bnmeani) * bnvar_inv\n",
" hpreact = bngain * bnraw + bnbias\n",
" h = torch.tanh(hpreact)\n",
" logits = h @ W2 + b2\n",
" loss = F.cross_entropy(logits, Yb)\n",
"\n",
" # manual backward (no loss.backward())\n",
" dlogits = F.softmax(logits, 1)\n",
" dlogits[range(n), Yb] -= 1\n",
" dlogits /= n\n",
" dh = dlogits @ W2.T\n",
" dW2 = h.T @ dlogits\n",
" db2 = dlogits.sum(0)\n",
" dhpreact = (1.0 - h ** 2) * dh\n",
" dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
" dbnbias = dhpreact.sum(0, keepdim=True)\n",
" dhprebn = bngain * bnvar_inv / n * (\n",
" n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
" )\n",
" dembcat = dhprebn @ W1.T\n",
" dW1 = embcat.T @ dhprebn\n",
" db1 = dhprebn.sum(0)\n",
" demb = dembcat.view(emb.shape)\n",
" dC = torch.zeros_like(C)\n",
" dC.index_add_(0, Xb.view(-1), demb.view(-1, n_embd))\n",
"\n",
" grads = [dC, dW1, db1, dW2, db2, dbngain, dbnbias]\n",
" lr = 0.1 if i < 6000 else 0.01\n",
" with torch.no_grad():\n",
" for p, gr in zip(parameters, grads):\n",
" p += -lr * gr\n",
"\n",
" if i % 2000 == 0 or i == 7999:\n",
" print(f\"step {i:5d} loss {loss.item():.4f}\")\n",
"\n",
"print(\"trained using only hand-written gradients.\")"
]
},
{
"cell_type": "markdown",
"id": "422e7804",
"metadata": {},
"source": [
"**Your turn 4** — In section 2.3, deliberately introduce a bug (e.g. drop the\n",
"`n/(n-1)` Bessel factor in `dbnvar`) and re-run. Watch `cmp` flip to `approx: False` and the\n",
"`maxdiff` jump. Then fix it. This is exactly how you debug a real backward pass."
]
},
{
"cell_type": "markdown",
"id": "36c4bc1d",
"metadata": {},
"source": [
"## What's next\n",
"\n",
"You backpropagated by hand through cross-entropy, a linear layer, `tanh`, BatchNorm, and an\n",
"embedding, matched PyTorch exactly, and trained a net with zero autograd. Backprop is now\n",
"yours.\n",
"\n",
"Where this leads in the series:\n",
"\n",
"- `05_wavenet.ipynb` — stack layers into a deeper, hierarchical model using `torch.nn`-style\n",
" building blocks.\n",
"- `06_build_gpt_attention.ipynb` — the transformer; `loss.backward()` there is doing exactly\n",
" what you just did, across far more nodes.\n",
"\n",
"**Checklist**\n",
"- [ ] Forward pass saved every intermediate\n",
"- [ ] Manual backward matched PyTorch (cross-entropy, linear, tanh)\n",
"- [ ] Manual backward through BatchNorm matched\n",
"- [ ] Embedding backward = scatter-add into rows\n",
"- [ ] Elegant one-line cross-entropy and BatchNorm gradients\n",
"- [ ] Trained a net with only hand-written gradients"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
05_wavenet.ipynb
+424
View File
@@ -0,0 +1,424 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d24a5a8b",
"metadata": {},
"source": [
"# makemore — a WaveNet\n",
"\n",
"Go **deeper and hierarchical** — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **05**. The MLP in `02`/`03` squashed the whole context into **one** layer.\n",
"Here we widen the context to 8 letters and fuse them **gradually**, two at a time, in a\n",
"tree — the structure behind DeepMind's WaveNet (2016). We also wrap everything in tidy\n",
"`torch.nn`-style building blocks. Follows Karpathy's *\"makemore part 5: WaveNet\"*."
]
},
{
"cell_type": "markdown",
"id": "187db6ed",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"Reading 8 letters by mashing them into one layer throws away structure. WaveNet's idea: fuse\n",
"information **slowly**.\n",
"\n",
"Real-life picture: a knockout tournament. 8 players (letters) play in pairs -> 4 winners ->\n",
"play in pairs -> 2 -> 1 champion. Each round combines neighbours into a richer summary. Our\n",
"network does the same: combine letters in pairs, then pairs-of-pairs, then again, until a\n",
"single vector represents the whole context.\n",
"\n",
"We also build small reusable layers (`Linear`, `BatchNorm1d`, `Tanh`, `Embedding`,\n",
"`FlattenConsecutive`, `Sequential`) that behave like PyTorch's own — so the model reads like a\n",
"clean list of steps."
]
},
{
"cell_type": "markdown",
"id": "6bf209d0",
"metadata": {},
"source": [
"### 0.1 Imports and data (context grows to 8)\n",
"\n",
"The only data change from earlier notebooks: `block_size = 8`, so the model sees up to 8\n",
"previous letters."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88825091",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
"\n",
"import random\n",
"import torch\n",
"import torch.nn.functional as F\n",
"\n",
"try:\n",
" import matplotlib.pyplot as plt\n",
" HAS_PLT = True\n",
"except Exception:\n",
" HAS_PLT = False\n",
"\n",
"words = open(\"names.txt\").read().splitlines()\n",
"chars = sorted(list(set(\"\".join(words))))\n",
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
"stoi[\".\"] = 0\n",
"itos = {i: ch for ch, i in stoi.items()}\n",
"vocab_size = len(itos)\n",
"block_size = 8\n",
"\n",
"def build_dataset(word_list):\n",
" X, Y = [], []\n",
" for w in word_list:\n",
" context = [0] * block_size\n",
" for ch in w + \".\":\n",
" ix = stoi[ch]\n",
" X.append(context); Y.append(ix)\n",
" context = context[1:] + [ix]\n",
" return torch.tensor(X), torch.tensor(Y)\n",
"\n",
"random.seed(42)\n",
"random.shuffle(words)\n",
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
"Xtr, Ytr = build_dataset(words[:n1])\n",
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
"Xte, Yte = build_dataset(words[n2:])\n",
"print(\"train:\", tuple(Xtr.shape), \" (context length =\", block_size, \")\")"
]
},
{
"cell_type": "markdown",
"id": "4c8200dd",
"metadata": {},
"source": [
"### 1.1 PyTorch-style building blocks\n",
"\n",
"Each layer is a tiny class with a `__call__` (forward) and `parameters()`. The interesting one\n",
"is **`FlattenConsecutive(n)`**: it groups every `n` neighbouring positions together — that is\n",
"the \"combine in pairs\" tournament step. `BatchNorm1d` is written to also handle the 3-D tensors\n",
"that appear between fusion steps (a subtle but important detail)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7695c863",
"metadata": {},
"outputs": [],
"source": [
"class Linear:\n",
" def __init__(self, fan_in, fan_out, bias=True):\n",
" self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in ** 0.5\n",
" self.bias = torch.zeros(fan_out) if bias else None\n",
" def __call__(self, x):\n",
" self.out = x @ self.weight\n",
" if self.bias is not None:\n",
" self.out = self.out + self.bias\n",
" return self.out\n",
" def parameters(self):\n",
" return [self.weight] + ([] if self.bias is None else [self.bias])\n",
"\n",
"\n",
"class BatchNorm1d:\n",
" def __init__(self, dim, eps=1e-5, momentum=0.1):\n",
" self.eps = eps; self.momentum = momentum; self.training = True\n",
" self.gamma = torch.ones(dim); self.beta = torch.zeros(dim)\n",
" self.running_mean = torch.zeros(dim); self.running_var = torch.ones(dim)\n",
" def __call__(self, x):\n",
" if self.training:\n",
" dim = 0 if x.ndim == 2 else (0, 1) # 3-D inputs: average over batch AND time\n",
" xmean = x.mean(dim, keepdim=True)\n",
" xvar = x.var(dim, keepdim=True)\n",
" else:\n",
" xmean = self.running_mean\n",
" xvar = self.running_var\n",
" xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
" self.out = self.gamma * xhat + self.beta\n",
" if self.training:\n",
" with torch.no_grad():\n",
" self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean\n",
" self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar\n",
" return self.out\n",
" def parameters(self):\n",
" return [self.gamma, self.beta]\n",
"\n",
"\n",
"class Tanh:\n",
" def __call__(self, x):\n",
" self.out = torch.tanh(x); return self.out\n",
" def parameters(self):\n",
" return []\n",
"\n",
"\n",
"class Embedding:\n",
" def __init__(self, num_embeddings, embedding_dim):\n",
" self.weight = torch.randn((num_embeddings, embedding_dim), generator=g)\n",
" def __call__(self, IX):\n",
" self.out = self.weight[IX]; return self.out\n",
" def parameters(self):\n",
" return [self.weight]\n",
"\n",
"\n",
"class FlattenConsecutive:\n",
" def __init__(self, n):\n",
" self.n = n\n",
" def __call__(self, x):\n",
" B, T, C = x.shape\n",
" x = x.view(B, T // self.n, C * self.n) # fuse every n neighbours\n",
" if x.shape[1] == 1:\n",
" x = x.squeeze(1)\n",
" self.out = x; return self.out\n",
" def parameters(self):\n",
" return []\n",
"\n",
"\n",
"class Sequential:\n",
" def __init__(self, layers):\n",
" self.layers = layers\n",
" def __call__(self, x):\n",
" for layer in self.layers:\n",
" x = layer(x)\n",
" self.out = x; return self.out\n",
" def parameters(self):\n",
" return [p for layer in self.layers for p in layer.parameters()]\n",
"\n",
"print(\"layers defined\")"
]
},
{
"cell_type": "markdown",
"id": "ab8cd26d",
"metadata": {},
"source": [
"### 1.2 Build the hierarchical (WaveNet) model\n",
"\n",
"`block_size = 8` fuses in three rounds: 8 -> 4 -> 2 -> 1. Each round is\n",
"`FlattenConsecutive(2) -> Linear -> BatchNorm1d -> Tanh`. Finally a `Linear` maps to the 27\n",
"next-letter scores. We shrink the last layer's weights so the first loss starts near the ideal\n",
"`ln(27)`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1dcd57ee",
"metadata": {},
"outputs": [],
"source": [
"g = torch.Generator().manual_seed(42)\n",
"n_embd = 24\n",
"n_hidden = 128\n",
"\n",
"model = Sequential([\n",
" Embedding(vocab_size, n_embd),\n",
" FlattenConsecutive(2), Linear(n_embd * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
" FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
" FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
" Linear(n_hidden, vocab_size),\n",
"])\n",
"\n",
"with torch.no_grad():\n",
" model.layers[-1].weight *= 0.1 # humble output\n",
"\n",
"parameters = model.parameters()\n",
"for p in parameters:\n",
" p.requires_grad = True\n",
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
]
},
{
"cell_type": "markdown",
"id": "dfe5653b",
"metadata": {},
"source": [
"### 1.3 Watch the tournament shrink the sequence\n",
"\n",
"Run one batch and print each layer's output shape. See the time dimension halve 8 -> 4 -> 2 -> 1\n",
"as neighbours fuse, while the feature dimension does the work."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5a37517c",
"metadata": {},
"outputs": [],
"source": [
"ix = torch.randint(0, Xtr.shape[0], (4,), generator=g)\n",
"_ = model(Xtr[ix])\n",
"for layer in model.layers:\n",
" print(f\"{layer.__class__.__name__:18s} : {tuple(layer.out.shape)}\")"
]
},
{
"cell_type": "markdown",
"id": "43949ea9",
"metadata": {},
"source": [
"### 2.1 Train\n",
"\n",
"Standard loop: minibatch -> forward -> `loss.backward()` -> nudge. (We use autograd again now;\n",
"notebook `04` already proved we could do it by hand.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f0800ce0",
"metadata": {},
"outputs": [],
"source": [
"max_steps = 20000\n",
"batch_size = 32\n",
"lossi = []\n",
"\n",
"for i in range(max_steps):\n",
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
"\n",
" logits = model(Xb)\n",
" loss = F.cross_entropy(logits, Yb)\n",
"\n",
" for p in parameters:\n",
" p.grad = None\n",
" loss.backward()\n",
" lr = 0.1 if i < 15000 else 0.01\n",
" for p in parameters:\n",
" p.data += -lr * p.grad\n",
"\n",
" lossi.append(loss.log10().item())\n",
" if i % 4000 == 0 or i == max_steps - 1:\n",
" print(f\"step {i:5d}/{max_steps} loss {loss.item():.4f}\")\n",
"\n",
"if HAS_PLT:\n",
" # smooth the noisy per-step loss by averaging in chunks of 200\n",
" lt = torch.tensor(lossi)\n",
" plt.figure(figsize=(5, 4))\n",
" plt.plot(lt.view(-1, 200).mean(1))\n",
" plt.xlabel(\"step / 200\"); plt.ylabel(\"log10(loss)\"); plt.title(\"WaveNet training loss\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "053ba8a7",
"metadata": {},
"source": [
"### 2.2 Evaluate\n",
"\n",
"Switch BatchNorm layers to eval mode (use running stats), then score train and dev. The deeper\n",
"hierarchical model should edge below the single-layer MLP's ~2.1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a5d6228",
"metadata": {},
"outputs": [],
"source": [
"for layer in model.layers:\n",
" if isinstance(layer, BatchNorm1d):\n",
" layer.training = False\n",
"\n",
"@torch.no_grad()\n",
"def split_loss(split):\n",
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
" logits = model(x)\n",
" return F.cross_entropy(logits, y).item()\n",
"\n",
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
]
},
{
"cell_type": "markdown",
"id": "b78eaf70",
"metadata": {},
"source": [
"### 3.1 Generate names\n",
"\n",
"Sample letter by letter as before. With 8 letters of context and the hierarchical fusion, the\n",
"names tend to look a little more name-like."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33833066",
"metadata": {},
"outputs": [],
"source": [
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
"\n",
"for _ in range(20):\n",
" out = []\n",
" context = [0] * block_size\n",
" while True:\n",
" logits = model(torch.tensor([context]))\n",
" probs = F.softmax(logits, dim=1)\n",
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
" context = context[1:] + [ix]\n",
" if ix == 0:\n",
" break\n",
" out.append(itos[ix])\n",
" print(\"\".join(out))"
]
},
{
"cell_type": "markdown",
"id": "31d52503",
"metadata": {},
"source": [
"**Your turn 5** — The model is just a list. Add another fusion round (so it handles\n",
"`block_size = 16`), or raise `n_hidden`, retrain, and compare dev loss. Because everything is a\n",
"clean `Sequential`, experimenting is just editing the list."
]
},
{
"cell_type": "markdown",
"id": "894cc92e",
"metadata": {},
"source": [
"## What's next\n",
"\n",
"You built a deeper, hierarchical model out of reusable PyTorch-style blocks — the same shape of\n",
"code real libraries use.\n",
"\n",
"Where this leads in the series:\n",
"\n",
"- `06_build_gpt_attention.ipynb` — the transformer. Instead of a fixed tournament that always\n",
" fuses neighbours, **attention** lets every position decide *which* earlier positions to listen\n",
" to. That flexibility is what makes GPT powerful.\n",
"- `07_gpt_tokenizer.ipynb` — how real models turn text into tokens in the first place.\n",
"\n",
"**Checklist**\n",
"- [ ] Context widened to 8 letters\n",
"- [ ] PyTorch-style layers (Linear, BatchNorm1d, Tanh, Embedding, FlattenConsecutive, Sequential)\n",
"- [ ] Hierarchical fusion 8 -> 4 -> 2 -> 1\n",
"- [ ] BatchNorm1d handles 3-D tensors\n",
"- [ ] Trained, evaluated, sampled names"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
06_build_gpt_attention.ipynb
+480
View File
@@ -0,0 +1,480 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "3d5936ca",
"metadata": {},
"source": [
"# Let's build GPT — self-attention to a full transformer\n",
"\n",
"Give every token the power to **look back** — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **06**, the payoff. Notebook `01` built a bigram that only sees one character\n",
"back. Here we add **self-attention** so each position can gather information from all earlier\n",
"positions, then stack it into a real (small) **GPT** and train it on tiny Shakespeare.\n",
"Follows Karpathy's *\"Let's build GPT from scratch\"*."
]
},
{
"cell_type": "markdown",
"id": "8b6c4fb3",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"The bigram's weakness: predicting the next letter from only the previous one. WaveNet (`05`)\n",
"widened the view but fused neighbours in a fixed pattern. **Attention** is smarter: every\n",
"token looks at all earlier tokens and decides, on its own, *which* ones matter right now.\n",
"\n",
"Real-life picture: you are reading a sentence and hit the word \"it.\" To know what \"it\" refers\n",
"to, your eyes jump back to the relevant earlier noun — not to a fixed position, but to whichever\n",
"word is relevant. Self-attention lets each token make that kind of targeted look-back.\n",
"\n",
"We build it up in small pieces: the averaging trick, then queries/keys/values, then multiple\n",
"heads, then the full transformer block (attention + feed-forward + residual + LayerNorm)."
]
},
{
"cell_type": "markdown",
"id": "028dce18",
"metadata": {},
"source": [
"### 0.1 Setup and data (standalone)\n",
"\n",
"Self-contained: load `input.txt`, build the character vocabulary, and make `get_batch`. Uses\n",
"GPU if available, else CPU. The model is intentionally **small** so it trains on a laptop CPU;\n",
"the comments show how to scale it up if you have a GPU."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36f4f6e6",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
"\n",
"import torch\n",
"import torch.nn as nn\n",
"from torch.nn import functional as F\n",
"\n",
"torch.manual_seed(1337)\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(\"device:\", device, \"| torch\", torch.__version__)\n",
"\n",
"# --- hyperparameters (small for CPU; bump these up on a GPU) ---\n",
"batch_size = 16\n",
"block_size = 32 # max context length\n",
"n_embd = 64\n",
"n_head = 4\n",
"n_layer = 3\n",
"dropout = 0.0\n",
"learning_rate = 1e-3\n",
"max_iters = 3000\n",
"eval_interval = 1000\n",
"eval_iters = 50\n",
"\n",
"# --- data ---\n",
"text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()\n",
"chars = sorted(list(set(text)))\n",
"vocab_size = len(chars)\n",
"stoi = {ch: i for i, ch in enumerate(chars)}\n",
"itos = {i: ch for i, ch in enumerate(chars)}\n",
"encode = lambda s: [stoi[c] for c in s]\n",
"decode = lambda l: \"\".join(itos[i] for i in l)\n",
"\n",
"data = torch.tensor(encode(text), dtype=torch.long)\n",
"n = int(0.9 * len(data))\n",
"train_data, val_data = data[:n], data[n:]\n",
"\n",
"def get_batch(split):\n",
" d = train_data if split == \"train\" else val_data\n",
" ix = torch.randint(len(d) - block_size, (batch_size,))\n",
" x = torch.stack([d[i:i + block_size] for i in ix])\n",
" y = torch.stack([d[i + 1:i + block_size + 1] for i in ix])\n",
" return x.to(device), y.to(device)\n",
"\n",
"print(\"vocab_size:\", vocab_size, \"| chars in dataset:\", len(text))"
]
},
{
"cell_type": "markdown",
"id": "d2b12520",
"metadata": {},
"source": [
"### 1.1 The math trick: a token can average its past\n",
"\n",
"Before real attention, here is the key move in its simplest form: let each position become the\n",
"**average of itself and all positions before it** (never the future — that would be cheating).\n",
"We show three ways that give the *same* result, from slow to elegant."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "54c69615",
"metadata": {},
"outputs": [],
"source": [
"torch.manual_seed(1337)\n",
"B, T, C = 4, 8, 2\n",
"x = torch.randn(B, T, C)\n",
"\n",
"# v1: explicit loop — average of all previous tokens\n",
"xbow = torch.zeros((B, T, C))\n",
"for b in range(B):\n",
" for t in range(T):\n",
" xbow[b, t] = x[b, :t + 1].mean(0)\n",
"\n",
"# v2: same thing as a matrix multiply with a lower-triangular average matrix\n",
"wei = torch.tril(torch.ones(T, T))\n",
"wei = wei / wei.sum(1, keepdim=True)\n",
"xbow2 = wei @ x\n",
"\n",
"# v3: same again, but the weights come from softmax over a masked matrix\n",
"tril = torch.tril(torch.ones(T, T))\n",
"wei = torch.zeros((T, T)).masked_fill(tril == 0, float(\"-inf\"))\n",
"wei = F.softmax(wei, dim=-1)\n",
"xbow3 = wei @ x\n",
"\n",
"print(\"v1 == v2 :\", torch.allclose(xbow, xbow2, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow2).abs().max())\n",
"print(\"v1 == v3 :\", torch.allclose(xbow, xbow3, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow3).abs().max())\n",
"print(\"\\nthe softmax weight matrix (row t can only attend to columns <= t):\")\n",
"print(wei)"
]
},
{
"cell_type": "markdown",
"id": "aa46672a",
"metadata": {},
"source": [
"Why the `-inf` + softmax version matters: those weights do **not** have to be a flat\n",
"average. If the numbers in `wei` are *learned*, each token can decide how much to listen to each\n",
"earlier token. That is exactly what attention does next.\n",
"\n",
"Analogy: a meeting where you may only hear people who already spoke (the mask), and you choose\n",
"how much weight to give each speaker (the learned weights)."
]
},
{
"cell_type": "markdown",
"id": "b522ef98",
"metadata": {},
"source": [
"### 2.1 A single self-attention head — query, key, value\n",
"\n",
"Each token emits three vectors:\n",
"\n",
"- **query**: \"what am I looking for?\"\n",
"- **key**: \"what do I offer?\"\n",
"- **value**: \"what will I actually pass on if chosen?\"\n",
"\n",
"A token's attention to another = how well its **query** matches the other's **key**\n",
"(dot product). We scale by `1/sqrt(head_size)` to keep the numbers tame, mask the future, softmax\n",
"into weights, then take the weighted sum of **values**."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9adfb9ea",
"metadata": {},
"outputs": [],
"source": [
"torch.manual_seed(1337)\n",
"B, T, C = 4, 8, 32\n",
"x = torch.randn(B, T, C)\n",
"head_size = 16\n",
"\n",
"key = nn.Linear(C, head_size, bias=False)\n",
"query = nn.Linear(C, head_size, bias=False)\n",
"value = nn.Linear(C, head_size, bias=False)\n",
"\n",
"k = key(x) # (B, T, head_size)\n",
"q = query(x) # (B, T, head_size)\n",
"wei = q @ k.transpose(-2, -1) * head_size ** -0.5 # (B, T, T) affinities\n",
"\n",
"tril = torch.tril(torch.ones(T, T))\n",
"wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
"wei = F.softmax(wei, dim=-1)\n",
"\n",
"v = value(x)\n",
"out = wei @ v # (B, T, head_size)\n",
"print(\"attention output shape:\", tuple(out.shape))\n",
"print(\"\\nrow 0 of the learned attention weights (note they are NOT uniform):\")\n",
"print(wei[0, -1])"
]
},
{
"cell_type": "markdown",
"id": "ff4eea4c",
"metadata": {},
"source": [
"### 3.1 The full GPT, assembled from clean modules\n",
"\n",
"Now we package it. The pieces:\n",
"\n",
"- **Head** — one self-attention head (above), with dropout and a causal mask buffer.\n",
"- **MultiHeadAttention** — several heads in parallel, concatenated, then projected. Analogy:\n",
" several readers each look back for a different kind of clue, then pool notes.\n",
"- **FeedForward** — a small per-token MLP: \"time to think about what I gathered.\"\n",
"- **Block** — attention + feed-forward, each wrapped with a **residual** connection and\n",
" **LayerNorm**. Residual = keep the original and add the new insight (so deep stacks still\n",
" train). LayerNorm = re-level each token's vector (the `03` lesson, per-token).\n",
"- **GPTLanguageModel** — token + position embeddings -> a stack of Blocks -> final LayerNorm ->\n",
" a linear head to vocab logits."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0a4b08d",
"metadata": {},
"outputs": [],
"source": [
"class Head(nn.Module):\n",
" def __init__(self, head_size):\n",
" super().__init__()\n",
" self.key = nn.Linear(n_embd, head_size, bias=False)\n",
" self.query = nn.Linear(n_embd, head_size, bias=False)\n",
" self.value = nn.Linear(n_embd, head_size, bias=False)\n",
" self.register_buffer(\"tril\", torch.tril(torch.ones(block_size, block_size)))\n",
" self.dropout = nn.Dropout(dropout)\n",
"\n",
" def forward(self, x):\n",
" B, T, C = x.shape\n",
" k = self.key(x)\n",
" q = self.query(x)\n",
" wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5\n",
" wei = wei.masked_fill(self.tril[:T, :T] == 0, float(\"-inf\"))\n",
" wei = F.softmax(wei, dim=-1)\n",
" wei = self.dropout(wei)\n",
" v = self.value(x)\n",
" return wei @ v\n",
"\n",
"\n",
"class MultiHeadAttention(nn.Module):\n",
" def __init__(self, num_heads, head_size):\n",
" super().__init__()\n",
" self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
" self.proj = nn.Linear(head_size * num_heads, n_embd)\n",
" self.dropout = nn.Dropout(dropout)\n",
"\n",
" def forward(self, x):\n",
" out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
" return self.dropout(self.proj(out))\n",
"\n",
"\n",
"class FeedForward(nn.Module):\n",
" def __init__(self, n_embd):\n",
" super().__init__()\n",
" self.net = nn.Sequential(\n",
" nn.Linear(n_embd, 4 * n_embd),\n",
" nn.ReLU(),\n",
" nn.Linear(4 * n_embd, n_embd),\n",
" nn.Dropout(dropout),\n",
" )\n",
"\n",
" def forward(self, x):\n",
" return self.net(x)\n",
"\n",
"\n",
"class Block(nn.Module):\n",
" def __init__(self, n_embd, n_head):\n",
" super().__init__()\n",
" head_size = n_embd // n_head\n",
" self.sa = MultiHeadAttention(n_head, head_size)\n",
" self.ffwd = FeedForward(n_embd)\n",
" self.ln1 = nn.LayerNorm(n_embd)\n",
" self.ln2 = nn.LayerNorm(n_embd)\n",
"\n",
" def forward(self, x):\n",
" x = x + self.sa(self.ln1(x)) # residual around attention\n",
" x = x + self.ffwd(self.ln2(x)) # residual around feed-forward\n",
" return x\n",
"\n",
"\n",
"class GPTLanguageModel(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
" self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
" self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)])\n",
" self.ln_f = nn.LayerNorm(n_embd)\n",
" self.lm_head = nn.Linear(n_embd, vocab_size)\n",
"\n",
" def forward(self, idx, targets=None):\n",
" B, T = idx.shape\n",
" tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
" pos_emb = self.position_embedding_table(torch.arange(T, device=device))\n",
" x = tok_emb + pos_emb\n",
" x = self.blocks(x)\n",
" x = self.ln_f(x)\n",
" logits = self.lm_head(x) # (B,T,vocab)\n",
" if targets is None:\n",
" return logits, None\n",
" B, T, Cc = logits.shape\n",
" loss = F.cross_entropy(logits.view(B * T, Cc), targets.view(B * T))\n",
" return logits, loss\n",
"\n",
" def generate(self, idx, max_new_tokens):\n",
" for _ in range(max_new_tokens):\n",
" idx_cond = idx[:, -block_size:] # never exceed the context window\n",
" logits, _ = self(idx_cond)\n",
" logits = logits[:, -1, :] # last position only\n",
" probs = F.softmax(logits, dim=-1)\n",
" idx_next = torch.multinomial(probs, num_samples=1)\n",
" idx = torch.cat((idx, idx_next), dim=1)\n",
" return idx\n",
"\n",
"\n",
"model = GPTLanguageModel().to(device)\n",
"print(\"parameters (millions):\", sum(p.numel() for p in model.parameters()) / 1e6)"
]
},
{
"cell_type": "markdown",
"id": "dbb781c5",
"metadata": {},
"source": [
"### 3.2 Positional embeddings — why we need them\n",
"\n",
"Attention by itself is **order-blind**: it treats the context as a bag of tokens. But \"dog\n",
"bites man\" and \"man bites dog\" must differ. So besides each token's meaning\n",
"(`token_embedding_table`), we add a vector for its **position**\n",
"(`position_embedding_table`). Now a token knows both *what* it is and *where* it is.\n",
"\n",
"(That addition already happens inside `forward`: `x = tok_emb + pos_emb`.)"
]
},
{
"cell_type": "markdown",
"id": "0385a0ef",
"metadata": {},
"source": [
"### 4.1 Train the GPT\n",
"\n",
"Same training shape as always, with the AdamW optimizer. Watch train and val loss fall well\n",
"below the bigram's ~2.5. (Small model + CPU, so this is a few minutes; scale up on a GPU for\n",
"real Shakespeare.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "395e6922",
"metadata": {},
"outputs": [],
"source": [
"@torch.no_grad()\n",
"def estimate_loss():\n",
" out = {}\n",
" model.eval()\n",
" for split in [\"train\", \"val\"]:\n",
" losses = torch.zeros(eval_iters)\n",
" for k in range(eval_iters):\n",
" X, Y = get_batch(split)\n",
" _, loss = model(X, Y)\n",
" losses[k] = loss.item()\n",
" out[split] = losses.mean().item()\n",
" model.train()\n",
" return out\n",
"\n",
"optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
"\n",
"for it in range(max_iters):\n",
" if it % eval_interval == 0 or it == max_iters - 1:\n",
" losses = estimate_loss()\n",
" print(f\"step {it:4d}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
" xb, yb = get_batch(\"train\")\n",
" _, loss = model(xb, yb)\n",
" optimizer.zero_grad(set_to_none=True)\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
"print(\"done training\")"
]
},
{
"cell_type": "markdown",
"id": "0ae5a608",
"metadata": {},
"source": [
"### 5.1 Generate Shakespeare-ish text\n",
"\n",
"Start from a single newline and let the model write. It will not be perfect Shakespeare (tiny\n",
"model, short training), but compared to the bigram it now produces real-looking words, spacing,\n",
"and line breaks — because each character can attend to the ones before it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61db808c",
"metadata": {},
"outputs": [],
"source": [
"context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
"print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))"
]
},
{
"cell_type": "markdown",
"id": "6b43261f",
"metadata": {},
"source": [
"**Your turn 6** — If you have a GPU, scale up (e.g. `n_embd=384`, `n_head=6`,\n",
"`n_layer=6`, `block_size=256`, `dropout=0.2`, `max_iters=5000`) and retrain — the samples get\n",
"dramatically better. On CPU, try just raising `n_layer` to 4 or `max_iters` to 5000 and compare\n",
"the val loss."
]
},
{
"cell_type": "markdown",
"id": "1e1142ab",
"metadata": {},
"source": [
"## What's next — and the connection to ChatGPT\n",
"\n",
"You built a real transformer: token + position embeddings, multi-head causal self-attention,\n",
"feed-forward layers, residuals, and LayerNorm — the exact architecture behind GPT.\n",
"\n",
"What you trained here is a **pretrained** model in miniature: it only learns to *continue text*\n",
"(predict the next token) from a big pile of text. ChatGPT starts the same way, then adds steps\n",
"this series does not cover:\n",
"\n",
"- **Supervised finetuning** on example conversations (teach it to answer, not just continue).\n",
"- **Reinforcement learning from human feedback (RLHF)** to align responses with what people\n",
" prefer.\n",
"\n",
"The remaining notebook:\n",
"\n",
"- `07_gpt_tokenizer.ipynb` — real models do not use one-character-per-token; they use **Byte\n",
" Pair Encoding**. That is the final piece of the pipeline.\n",
"\n",
"**Checklist**\n",
"- [ ] The averaging trick (loop = tril matmul = masked softmax)\n",
"- [ ] Query / key / value self-attention head\n",
"- [ ] Multi-head attention + projection\n",
"- [ ] Feed-forward, residuals, LayerNorm -> Block\n",
"- [ ] Token + position embeddings -> full GPT\n",
"- [ ] Trained below the bigram baseline; generated text"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
07_gpt_tokenizer.ipynb
+375
View File
@@ -0,0 +1,375 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "67a422d8",
"metadata": {},
"source": [
"# The GPT tokenizer — Byte Pair Encoding from scratch\n",
"\n",
"Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model\n",
"either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**\n",
"tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then\n",
"see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's\n",
"*\"Let's build the GPT Tokenizer\"*."
]
},
{
"cell_type": "markdown",
"id": "2037ef52",
"metadata": {},
"source": [
"## Prologue — in plain English\n",
"\n",
"A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece\n",
"mapped to an id. Two extremes:\n",
"\n",
"- **One token per character** (what we did): tiny vocabulary, but sequences are very long and\n",
" the model must relearn common spellings everywhere.\n",
"- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.\n",
"\n",
"**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common\n",
"adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single\n",
"tokens; rare text still survives as smaller pieces.\n",
"\n",
"Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep\n",
"writing \"t-h-e\", so you invent one squiggle for \"the.\" Then you notice \" a-n-d\" and give it a\n",
"squiggle too. BPE does this automatically, keeping the most useful shorthands."
]
},
{
"cell_type": "markdown",
"id": "75d40ed6",
"metadata": {},
"source": [
"### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)\n",
"\n",
"Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts\n",
"with exactly 256 tokens — every possible byte. Plain English letters are one byte each;\n",
"accented or non-Latin characters take several bytes (which is why they cost more tokens later)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1037d36e",
"metadata": {},
"outputs": [],
"source": [
"sample = \"Hello world! tokenization is fun.\"\n",
"the_bytes = sample.encode(\"utf-8\")\n",
"print(\"text :\", sample)\n",
"print(\"bytes :\", list(the_bytes))\n",
"print(\"length :\", len(the_bytes), \"bytes ->\", len(the_bytes), \"starting tokens\")\n",
"\n",
"multi = \"café naïve\" # accented letters take >1 byte each\n",
"print(\"\\nnon-ASCII example:\", multi)\n",
"print(\"bytes:\", list(multi.encode(\"utf-8\")), \"(note: more bytes than characters)\")"
]
},
{
"cell_type": "markdown",
"id": "5db66687",
"metadata": {},
"source": [
"### 2.1 Two tiny helpers: count pairs, and merge a pair\n",
"\n",
"- `get_stats` counts how often each adjacent pair appears.\n",
"- `merge` replaces every occurrence of a chosen pair with a single new token id.\n",
"\n",
"That is the entire mechanism of BPE."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6bc93ad8",
"metadata": {},
"outputs": [],
"source": [
"def get_stats(ids):\n",
" counts = {}\n",
" for pair in zip(ids, ids[1:]):\n",
" counts[pair] = counts.get(pair, 0) + 1\n",
" return counts\n",
"\n",
"def merge(ids, pair, idx):\n",
" new_ids = []\n",
" i = 0\n",
" while i < len(ids):\n",
" if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:\n",
" new_ids.append(idx)\n",
" i += 2\n",
" else:\n",
" new_ids.append(ids[i])\n",
" i += 1\n",
" return new_ids\n",
"\n",
"# demo on a toy list: merge the most common pair\n",
"demo = [1, 2, 3, 1, 2, 3, 1, 2]\n",
"st = get_stats(demo)\n",
"top = max(st, key=st.get)\n",
"print(\"counts:\", st)\n",
"print(\"most common pair:\", top, \"->\", st[top], \"times\")\n",
"print(\"after merging\", top, \"into 99:\", merge(demo, top, 99))"
]
},
{
"cell_type": "markdown",
"id": "cfb961e0",
"metadata": {},
"source": [
"### 2.2 Train BPE — learn the merges\n",
"\n",
"We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find\n",
"the most common pair and merge it, recording each merge. After `num_merges` rounds we have a\n",
"vocabulary of `256 + num_merges` tokens."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f62fbf5d",
"metadata": {},
"outputs": [],
"source": [
"train_text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()[:20000]\n",
"ids = list(train_text.encode(\"utf-8\"))\n",
"print(\"training bytes:\", len(ids))\n",
"\n",
"num_merges = 60\n",
"merges = {} # (a, b) -> new_id\n",
"work = list(ids)\n",
"for i in range(num_merges):\n",
" stats = get_stats(work)\n",
" pair = max(stats, key=stats.get)\n",
" idx = 256 + i\n",
" work = merge(work, pair, idx)\n",
" merges[pair] = idx\n",
"\n",
"print(\"merges learned:\", len(merges))\n",
"print(\"tokens after training:\", len(work), \"(was\", len(ids), \"bytes)\")\n",
"print(f\"compression: {len(ids) / len(work):.2f}x\")"
]
},
{
"cell_type": "markdown",
"id": "34b2bc27",
"metadata": {},
"source": [
"### 3.1 encode and decode\n",
"\n",
"- **decode**: turn token ids back into text. We build each token's byte string by stitching the\n",
" merges back together.\n",
"- **encode**: turn new text into token ids by applying the learned merges, always doing the\n",
" *earliest-learned* applicable merge first."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c4f1dcab",
"metadata": {},
"outputs": [],
"source": [
"# build the id -> bytes table\n",
"vocab = {idx: bytes([idx]) for idx in range(256)}\n",
"for (p0, p1), idx in merges.items():\n",
" vocab[idx] = vocab[p0] + vocab[p1]\n",
"\n",
"def decode(ids):\n",
" data = b\"\".join(vocab[idx] for idx in ids)\n",
" return data.decode(\"utf-8\", errors=\"replace\")\n",
"\n",
"def encode(text):\n",
" tokens = list(text.encode(\"utf-8\"))\n",
" while len(tokens) >= 2:\n",
" stats = get_stats(tokens)\n",
" # pick the pair whose merge was learned earliest\n",
" pair = min(stats, key=lambda p: merges.get(p, float(\"inf\")))\n",
" if pair not in merges:\n",
" break\n",
" tokens = merge(tokens, pair, merges[pair])\n",
" return tokens\n",
"\n",
"trial = \"the king and the queen\"\n",
"enc = encode(trial)\n",
"print(\"text :\", trial)\n",
"print(\"tokens:\", enc)\n",
"print(\"count :\", len(enc), \"tokens for\", len(trial), \"characters\")\n",
"print(\"decode round-trip ok:\", decode(enc) == trial)"
]
},
{
"cell_type": "markdown",
"id": "a41ee40c",
"metadata": {},
"source": [
"### 3.2 See what the learned tokens are\n",
"\n",
"The merges discovered the common chunks of English on their own. Let's print a few learned\n",
"tokens (ids >= 256) as the text they stand for."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dadcc751",
"metadata": {},
"outputs": [],
"source": [
"print(\"a few learned tokens:\")\n",
"for idx in range(256, 256 + min(20, num_merges)):\n",
" piece = vocab[idx].decode(\"utf-8\", errors=\"replace\")\n",
" print(f\" id {idx}: {piece!r}\")"
]
},
{
"cell_type": "markdown",
"id": "9193a4cd",
"metadata": {},
"source": [
"### 4.1 Regex pre-splitting (what real GPTs add)\n",
"\n",
"A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`\n",
"into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,\n",
"runs of spaces, punctuation), then runs BPE **inside** each chunk only.\n",
"\n",
"Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the\n",
"standard `re` module to show the idea."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "96cf29fc",
"metadata": {},
"outputs": [],
"source": [
"import re\n",
"\n",
"# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})\n",
"pat = re.compile(r\"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+\")\n",
"\n",
"demo_text = \"Hello, world! It's 2026 already.\"\n",
"chunks = re.findall(pat, demo_text)\n",
"print(\"split into chunks (BPE then runs inside each one):\")\n",
"print(chunks)"
]
},
{
"cell_type": "markdown",
"id": "e825ed0b",
"metadata": {},
"source": [
"### 4.2 Special tokens\n",
"\n",
"Real tokenizers also reserve **special tokens** that are not learned from text but inserted to\n",
"mark structure, for example `<|endoftext|>` between documents, or chat markers like\n",
"`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.\n",
"\n",
"Real-life picture: punctuation the model never \"spells\" — single reserved symbols that mean\n",
"\"new document starts here\" or \"the user is speaking now.\"\n",
"\n",
"```\n",
"<|endoftext|> -> id 50256 in GPT-2 (separates documents)\n",
"<|im_start|> -> chat role marker in chat models\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "638d25d8",
"metadata": {},
"source": [
"### 5.1 Why tokenization causes LLM quirks\n",
"\n",
"Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.\n",
"\n",
"- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does\n",
" not \"see\" its letters.\n",
"- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.\n",
"- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more\n",
" tokens (and more money / context)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f051e82",
"metadata": {},
"outputs": [],
"source": [
"def n_tokens(s):\n",
" return len(encode(s))\n",
"\n",
"print(\"spelling: 'extraordinary' ->\", n_tokens(\"extraordinary\"), \"token(s)\")\n",
"print(\" the model sees a chunk, not the 13 letters\\n\")\n",
"\n",
"print(\"numbers : '1234567' ->\", n_tokens(\"1234567\"), \"tokens (split oddly, not per-digit)\")\n",
"print(\" '127 + 677' ->\", n_tokens(\"127 + 677\"), \"tokens\\n\")\n",
"\n",
"eng = \"hello how are you\"\n",
"non = \"你好你好你好\" # non-English of similar visible length\n",
"print(\"non-English costs more tokens for similar content:\")\n",
"print(\" english :\", repr(eng), \"->\", n_tokens(eng), \"tokens\")\n",
"print(\" non-latin :\", repr(non), \"->\", n_tokens(non), \"tokens\")"
]
},
{
"cell_type": "markdown",
"id": "cda06321",
"metadata": {},
"source": [
"**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the\n",
"compression ratio and the token counts in section 5.1. More merges = better compression, but a\n",
"bigger vocabulary. This trade-off is exactly what real tokenizer designers tune."
]
},
{
"cell_type": "markdown",
"id": "35388d46",
"metadata": {},
"source": [
"## Series wrap-up\n",
"\n",
"You built the whole stack, from the ground up:\n",
"\n",
"- `00_micrograd.ipynb` — backprop from scratch (the engine)\n",
"- `01_build_gpt.ipynb` — the bigram baseline\n",
"- `02_makemore_mlp.ipynb` — an MLP with embeddings\n",
"- `03_batchnorm_activations.ipynb` — keeping deep nets healthy\n",
"- `04_backprop_ninja.ipynb` — gradients by hand\n",
"- `05_wavenet.ipynb` — a deeper, hierarchical model\n",
"- `06_build_gpt_attention.ipynb` — the transformer (GPT)\n",
"- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)\n",
"\n",
"Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,\n",
"with runnable code at every step.\n",
"\n",
"**Checklist**\n",
"- [ ] Text -> UTF-8 bytes (256 base tokens)\n",
"- [ ] `get_stats` + `merge` = the BPE mechanism\n",
"- [ ] Trained merges; measured compression\n",
"- [ ] encode / decode round-trips\n",
"- [ ] Regex pre-splitting and special tokens\n",
"- [ ] Explained tokenization-caused LLM quirks"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
_build_07.py
+276
View File
@@ -0,0 +1,276 @@
import json
import uuid
from pathlib import Path
cells = []
def _src(text):
lines = text.split("\n")
out = [line + "\n" if i < len(lines) - 1 else line for i, line in enumerate(lines)]
if out and out[-1] == "":
out.pop()
return out
def md(text):
cells.append({"cell_type": "markdown", "id": uuid.uuid4().hex[:8],
"metadata": {}, "source": _src(text)})
def code(text):
cells.append({"cell_type": "code", "execution_count": None, "id": uuid.uuid4().hex[:8],
"metadata": {}, "outputs": [], "source": _src(text)})
# ----------------------------------------------------------------------------
md("""# The GPT tokenizer — Byte Pair Encoding from scratch
Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.
This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model
either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**
tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then
see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's
*"Let's build the GPT Tokenizer"*.""")
md("""## Prologue — in plain English
A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece
mapped to an id. Two extremes:
- **One token per character** (what we did): tiny vocabulary, but sequences are very long and
the model must relearn common spellings everywhere.
- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.
**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common
adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single
tokens; rare text still survives as smaller pieces.
Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep
writing "t-h-e", so you invent one squiggle for "the." Then you notice " a-n-d" and give it a
squiggle too. BPE does this automatically, keeping the most useful shorthands.""")
# ----------------------------------------------------------------------------
md("""### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)
Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts
with exactly 256 tokens — every possible byte. Plain English letters are one byte each;
accented or non-Latin characters take several bytes (which is why they cost more tokens later).""")
code("""sample = "Hello world! tokenization is fun."
the_bytes = sample.encode("utf-8")
print("text :", sample)
print("bytes :", list(the_bytes))
print("length :", len(the_bytes), "bytes ->", len(the_bytes), "starting tokens")
multi = "café na\u00efve" # accented letters take >1 byte each
print("\\nnon-ASCII example:", multi)
print("bytes:", list(multi.encode("utf-8")), "(note: more bytes than characters)")""")
# ----------------------------------------------------------------------------
md("""### 2.1 Two tiny helpers: count pairs, and merge a pair
- `get_stats` counts how often each adjacent pair appears.
- `merge` replaces every occurrence of a chosen pair with a single new token id.
That is the entire mechanism of BPE.""")
code("""def get_stats(ids):
counts = {}
for pair in zip(ids, ids[1:]):
counts[pair] = counts.get(pair, 0) + 1
return counts
def merge(ids, pair, idx):
new_ids = []
i = 0
while i < len(ids):
if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:
new_ids.append(idx)
i += 2
else:
new_ids.append(ids[i])
i += 1
return new_ids
# demo on a toy list: merge the most common pair
demo = [1, 2, 3, 1, 2, 3, 1, 2]
st = get_stats(demo)
top = max(st, key=st.get)
print("counts:", st)
print("most common pair:", top, "->", st[top], "times")
print("after merging", top, "into 99:", merge(demo, top, 99))""")
md("""### 2.2 Train BPE — learn the merges
We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find
the most common pair and merge it, recording each merge. After `num_merges` rounds we have a
vocabulary of `256 + num_merges` tokens.""")
code("""train_text = open("input.txt", "r", encoding="utf-8").read()[:20000]
ids = list(train_text.encode("utf-8"))
print("training bytes:", len(ids))
num_merges = 60
merges = {} # (a, b) -> new_id
work = list(ids)
for i in range(num_merges):
stats = get_stats(work)
pair = max(stats, key=stats.get)
idx = 256 + i
work = merge(work, pair, idx)
merges[pair] = idx
print("merges learned:", len(merges))
print("tokens after training:", len(work), "(was", len(ids), "bytes)")
print(f"compression: {len(ids) / len(work):.2f}x")""")
# ----------------------------------------------------------------------------
md("""### 3.1 encode and decode
- **decode**: turn token ids back into text. We build each token's byte string by stitching the
merges back together.
- **encode**: turn new text into token ids by applying the learned merges, always doing the
*earliest-learned* applicable merge first.""")
code("""# build the id -> bytes table
vocab = {idx: bytes([idx]) for idx in range(256)}
for (p0, p1), idx in merges.items():
vocab[idx] = vocab[p0] + vocab[p1]
def decode(ids):
data = b"".join(vocab[idx] for idx in ids)
return data.decode("utf-8", errors="replace")
def encode(text):
tokens = list(text.encode("utf-8"))
while len(tokens) >= 2:
stats = get_stats(tokens)
# pick the pair whose merge was learned earliest
pair = min(stats, key=lambda p: merges.get(p, float("inf")))
if pair not in merges:
break
tokens = merge(tokens, pair, merges[pair])
return tokens
trial = "the king and the queen"
enc = encode(trial)
print("text :", trial)
print("tokens:", enc)
print("count :", len(enc), "tokens for", len(trial), "characters")
print("decode round-trip ok:", decode(enc) == trial)""")
md("""### 3.2 See what the learned tokens are
The merges discovered the common chunks of English on their own. Let's print a few learned
tokens (ids >= 256) as the text they stand for.""")
code("""print("a few learned tokens:")
for idx in range(256, 256 + min(20, num_merges)):
piece = vocab[idx].decode("utf-8", errors="replace")
print(f" id {idx}: {piece!r}")""")
# ----------------------------------------------------------------------------
md("""### 4.1 Regex pre-splitting (what real GPTs add)
A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`
into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,
runs of spaces, punctuation), then runs BPE **inside** each chunk only.
Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the
standard `re` module to show the idea.""")
code("""import re
# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})
pat = re.compile(r"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+")
demo_text = "Hello, world! It's 2026 already."
chunks = re.findall(pat, demo_text)
print("split into chunks (BPE then runs inside each one):")
print(chunks)""")
md("""### 4.2 Special tokens
Real tokenizers also reserve **special tokens** that are not learned from text but inserted to
mark structure, for example `<|endoftext|>` between documents, or chat markers like
`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.
Real-life picture: punctuation the model never "spells" — single reserved symbols that mean
"new document starts here" or "the user is speaking now."
```
<|endoftext|> -> id 50256 in GPT-2 (separates documents)
<|im_start|> -> chat role marker in chat models
```""")
# ----------------------------------------------------------------------------
md("""### 5.1 Why tokenization causes LLM quirks
Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.
- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does
not "see" its letters.
- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.
- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more
tokens (and more money / context).""")
code("""def n_tokens(s):
return len(encode(s))
print("spelling: 'extraordinary' ->", n_tokens("extraordinary"), "token(s)")
print(" the model sees a chunk, not the 13 letters\\n")
print("numbers : '1234567' ->", n_tokens("1234567"), "tokens (split oddly, not per-digit)")
print(" '127 + 677' ->", n_tokens("127 + 677"), "tokens\\n")
eng = "hello how are you"
non = "\u4f60\u597d\u4f60\u597d\u4f60\u597d" # non-English of similar visible length
print("non-English costs more tokens for similar content:")
print(" english :", repr(eng), "->", n_tokens(eng), "tokens")
print(" non-latin :", repr(non), "->", n_tokens(non), "tokens")""")
md("""**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the
compression ratio and the token counts in section 5.1. More merges = better compression, but a
bigger vocabulary. This trade-off is exactly what real tokenizer designers tune.""")
# ----------------------------------------------------------------------------
md("""## Series wrap-up
You built the whole stack, from the ground up:
- `00_micrograd.ipynb` — backprop from scratch (the engine)
- `01_build_gpt.ipynb` — the bigram baseline
- `02_makemore_mlp.ipynb` — an MLP with embeddings
- `03_batchnorm_activations.ipynb` — keeping deep nets healthy
- `04_backprop_ninja.ipynb` — gradients by hand
- `05_wavenet.ipynb` — a deeper, hierarchical model
- `06_build_gpt_attention.ipynb` — the transformer (GPT)
- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)
Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,
with runnable code at every step.
**Checklist**
- [ ] Text -> UTF-8 bytes (256 base tokens)
- [ ] `get_stats` + `merge` = the BPE mechanism
- [ ] Trained merges; measured compression
- [ ] encode / decode round-trips
- [ ] Regex pre-splitting and special tokens
- [ ] Explained tokenization-caused LLM quirks""")
# ----------------------------------------------------------------------------
nb = {
"cells": cells,
"metadata": {
"kernelspec": {"display_name": ".venv (3.12.10)", "language": "python", "name": "python3"},
"language_info": {"name": "python", "version": "3.12.10"},
},
"nbformat": 4,
"nbformat_minor": 5,
}
out_path = Path(r"d:\BeastProjects\gpt_from_scratch\07_gpt_tokenizer.ipynb")
out_path.write_text(json.dumps(nb, ensure_ascii=False, indent=2), encoding="utf-8")
print("wrote", out_path, "with", len(cells), "cells")
_validate_nb.py
+25
View File
@@ -0,0 +1,25 @@
import json, sys, time
path = sys.argv[1]
nb = json.load(open(path, encoding="utf-8"))
g = {"__name__": "__main__"}
n = 0
ok = True
for c in nb["cells"]:
if c["cell_type"] != "code":
continue
n += 1
src = "".join(c["source"])
t0 = time.time()
try:
exec(src, g)
if "plt" in g and getattr(g["plt"], "show", None):
g["plt"].show = lambda *a, **k: None
except Exception as e:
print("CELL", n, "FAILED:", type(e).__name__, e)
ok = False
break
print(f" (cell {n} ok, {time.time()-t0:.1f}s)")
if ok:
print("ALL", n, "CODE CELLS RAN OK")
local_ai_setup.md
-149
View File
@@ -1,149 +0,0 @@
# Local AI Coding Setup — Windows 11 (Aider + Ollama)
## Hardware
- GPU: RTX 5070 Ti, 12GB VRAM
- RAM: 32GB
- OS: Windows 11
---
## Prerequisites
### 1. Install Ollama
Download from https://ollama.com and install.
### 2. Pull a model
```powershell
ollama pull qwen3-coder:30b # 18GB, strong coding model (MoE, VRAM+RAM split)
ollama pull gemma4:12b # 12GB, responds in English more reliably
```
Confirm models are available:
```powershell
ollama list
```
Confirm Ollama is serving:
```powershell
curl http://localhost:11434/api/tags
```
---
## Install Aider
```powershell
pip install aider-chat
aider --version # should print version number
```
---
## Project Config
Create `.aider.conf.yml` in your project root:
```yaml
model: ollama/gemma4:12b
openai-api-base: http://localhost:11434/v1
openai-api-key: ollama
edit-format: whole
map-tokens: 2048
no-auto-commits: true
system-prompt: "Always respond in English only. Never use Chinese or any other language."
```
**Config notes:**
- `edit-format: whole` — more reliable file writes for local models (avoids code showing in chat only)
- `map-tokens: 2048` — limits repo map size, important for smaller context windows
- `no-auto-commits: true` — review changes before they hit git
- `system-prompt` — forces English responses (important for Qwen3 which defaults to Chinese)
Switch models without editing the file:
```
/model ollama/qwen3-coder:30b
```
---
## Optional: Performance Tuning (if inference is slow)
Create a `Modelfile` in the project root:
```
FROM qwen3-coder:30b
PARAMETER num_ctx 8192
PARAMETER num_gpu 99
```
Register it:
```powershell
ollama create qwen3-coder-tuned -f Modelfile
```
Update `.aider.conf.yml`:
```yaml
model: ollama/qwen3-coder-tuned
```
**Why:** `num_gpu 99` maximizes VRAM usage. `num_ctx 8192` prevents the KV cache from
spilling too much into RAM. Recommended context range: 4096–16384.
---
## Daily Aider Workflow
Start Aider (reads config automatically):
```powershell
aider
```
| Command | What it does |
|---------|-------------|
| `/add src/foo.py` | Add file to context |
| `/drop src/foo.py` | Remove file from context |
| `/diff` | See pending changes |
| `/undo` | Revert last change |
| `/run pytest` | Run command, share output with model |
| `/run git rm --cached file.py` | Run git commands from inside Aider |
| `/clear` | Clear chat history (keep files) |
| `/settings` | Show loaded config (verify auto_commits, system_prompt) |
| `/model ollama/gemma4:12b` | Switch model on the fly |
| `/exit` | Quit |
---
## Known Issues & Fixes
### Model responds in Chinese
Cause: Qwen3 is a Chinese model and defaults to Chinese.
Fix: `system-prompt` in `.aider.conf.yml` (already included above). Restart Aider after config change.
### Aider shows code in chat but doesn't write files
Cause: Model didn't follow Aider's edit format.
Fix 1: `/add target_file.py` before asking — hints the target file to the model.
Fix 2: `edit-format: whole` in config (already included above).
### "File not found" / repo-map warnings for deleted files
Cause: File deleted from disk but still tracked in git index.
Fix:
```powershell
git rm --cached filename.py
```
Or from inside Aider:
```
/run git rm --cached filename.py
```
### Config changes not taking effect
Fix: Exit Aider (`/exit`) and restart. Config is only read at startup.
---
## Verify Everything Works
1. `ollama list` — model appears
2. `aider --version` — prints version
3. Start Aider, run `/settings` — confirm `auto_commits: False` and `system_prompt` present
4. Ask Aider to create a simple file — it should write to disk, not just chat
5. Response is in English
local_llm.md
-342
View File
@@ -1,342 +0,0 @@
# Local AI Coding Environment Implementation Plan
## Objective
Build a local-first AI coding environment on a Windows 11 workstation with the following requirements:
### Hardware
* NVIDIA RTX 5070 Ti 12GB VRAM
* 64GB+ system RAM (if available)
* Windows 11
* WSL2 Ubuntu
* VS Code
### Existing Software
* Ollama installed and operational
* Qwen3-Coder 30B installed in Ollama
* Claude Code CLI installed
* Git installed
### Constraints
* Source code is confidential.
* All code must remain local by default.
* Cloud models may be used only as an optional fallback.
* Minimize API costs.
* Support large codebases.
---
# Target Architecture
```text
VS Code
|
+---- Continue Extension
|
+---- Claude Code CLI
|
+---- Aider
|
v
Ollama
|
v
Qwen3-Coder 30B
```
---
# Project Goals
Implement and validate the following workflows.
## Workflow 1: Local Chat
Developer can:
* Ask coding questions
* Explain code
* Generate code
* Review code
using:
```text
VS Code
+
Continue
+
Ollama
+
Qwen3-Coder 30B
```
---
## Workflow 2: Local Agent
Developer can:
* Refactor code
* Create files
* Modify files
* Run git-aware edits
using:
```text
Aider
+
Ollama
+
Qwen3-Coder 30B
```
---
## Workflow 3: Claude Code Optional
Developer can:
* Use Claude Code against local Ollama models
* Compare behavior against Aider
* Determine whether Claude Code provides additional value
using:
```text
Claude Code
+
Ollama
+
Qwen3-Coder 30B
```
---
# Deliverables
Produce the following:
## Deliverable 1
Environment verification script.
Verify:
* Ollama installed
* NVIDIA GPU visible
* CUDA available
* WSL functioning
* Qwen3-Coder model available
Expected output:
```text
PASS: Ollama
PASS: GPU
PASS: CUDA
PASS: Qwen3-Coder
```
---
## Deliverable 2
Aider installation guide.
Include:
### Linux / WSL installation
Commands:
```bash
pipx install aider-chat
```
or preferred installation method.
### Ollama integration
Configuration examples.
### Verification steps
Simple repository test.
---
## Deliverable 3
Continue configuration.
Create:
### VS Code setup instructions
Install Continue extension.
### Model configuration
Configure Continue to use:
```text
Ollama
Qwen3-Coder 30B
```
### Example config files
Include complete examples.
---
## Deliverable 4
Claude Code local model integration.
Research and implement the best available approach for:
```text
Claude Code
->
OpenAI-compatible endpoint
->
Ollama
```
Requirements:
* Use officially supported methods where possible.
* Avoid unsupported hacks.
* Document limitations.
* Provide rollback procedure.
---
## Deliverable 5
Performance optimization.
Analyze:
### VRAM usage
Qwen3-Coder 30B on RTX 5070 Ti 12GB.
### Recommended quantization
Evaluate:
* Q4
* Q5
* IQ3
Recommend the best balance between:
* Quality
* Speed
* Memory usage
---
## Deliverable 6
Benchmark suite.
Create repeatable tests:
### Test 1
Generate a REST API.
### Test 2
Refactor a medium-sized module.
### Test 3
Write unit tests.
### Test 4
Debug a failing application.
Measure:
* Completion time
* Accuracy
* Token throughput
* User effort
Compare:
```text
Aider + Qwen3-Coder 30B
Claude Code + Qwen3-Coder 30B
Continue + Qwen3-Coder 30B
```
---
# Preferred Outcome
Primary development workflow:
```text
VS Code
+
Continue
+
Qwen3-Coder 30B
```
Agent workflow:
```text
Aider
+
Qwen3-Coder 30B
```
Optional advanced workflow:
```text
Claude Code
+
Qwen3-Coder 30B
```
---
# Success Criteria
The project is successful if:
1. All code remains local.
2. No cloud services are required.
3. Developer can perform daily coding tasks locally.
4. Aider successfully edits repositories.
5. Continue provides a productive IDE experience.
6. Claude Code local integration is evaluated and documented.
7. Setup can be reproduced on a fresh machine.
---
# Final Report
Produce a final report containing:
* Architecture diagram
* Installation steps
* Configuration files
* Benchmark results
* Known limitations
* Recommended workflow
* Future upgrade path
The report should be suitable for long-term maintenance and onboarding of additional developers.
names.txt
+32033
View File
File diff suppressed because it is too large Load Diff
opencode_lesson.md
+250
View File
@@ -0,0 +1,250 @@
# Local AI Coding Environment — Reference & Lessons Learned
> Consolidates `local_ai_setup.md` and `local_llm.md`. Those files are superseded by this one.
---
## Hardware
| Component | Spec |
|-----------|------|
| GPU | NVIDIA RTX 5070 Ti, 12 GB VRAM |
| RAM | 32 GB system RAM |
| OS | Windows 11 |
| WSL | WSL2 Ubuntu |
---
## Software Stack
| Tool | Purpose | Notes |
|------|---------|-------|
| Ollama | Local model server | Serves OpenAI-compatible API at `localhost:11434/v1` |
| OpenCode | Primary AI coding agent | Installed via npm (`opencode-ai`) |
| Aider | Alternative AI coding agent | pip: `aider-chat` |
| Claude Code | Cloud AI coding agent | Optional; can point at Ollama |
| VS Code | Editor | With Continue extension for inline chat |
---
## Architecture
```
VS Code
├── Continue Extension ──────────────┐
├── Claude Code CLI ────────────────┤
└── OpenCode / Aider ───────────────┤
▼
Ollama
│
▼
Qwen3-Coder 30B (primary)
```
All code stays local. Cloud models are optional fallback only.
---
## Ollama — Available Models
| Model | Size | Use |
|-------|------|-----|
| `qwen3-coder:30b-fixed` | 18 GB | **Primary** — OpenCode agent (fixed config, see below) |
| `qwen3-coder:30b` | 18 GB | Original pull (unmodified) |
| `gemma4:latest` | 9.6 GB | English-reliable alternative |
| `gemma4:12b` | 7.6 GB | Lighter English-reliable option |
| `qwen2.5-coder:14b` | 9.0 GB | Mid-weight coding model |
| `qwen2.5-coder:7b` | 4.7 GB | Fast, lightweight coding |
| `qwen2.5-coder:1.5b` | 986 MB | Minimal footprint |
| `qwen3:latest` | 5.2 GB | General chat |
| `qwen3:1.7b` | 1.4 GB | Fast general chat |
| `bge-m3:latest` | 1.2 GB | Embeddings |
---
## OpenCode Configuration
**Config file:** `~/.config/opencode/opencode.jsonc`
```jsonc
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ollama": {
"options": {
"baseURL": "http://localhost:11434/v1"
},
"models": {
"qwen3-coder:30b-fixed": { "name": "Qwen3 Coder 30B" },
"qwen2.5-coder:14b": { "name": "Qwen 2.5 Coder 14B" },
"qwen2.5-coder:7b": { "name": "Qwen 2.5 Coder 7B" },
"gemma4:latest": { "name": "Gemma 4 8B (latest)" },
"qwen3:latest": { "name": "Qwen3 8B" }
}
}
},
"model": "ollama/qwen3-coder:30b-fixed"
}
```
---
## Active Model — qwen3-coder:30b-fixed Modelfile
```
FROM <local blob: sha256-9124a16eb1a7...>
TEMPLATE {{ .Prompt }}
PARAMETER repeat_penalty 1.05
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
PARAMETER stop <|endoftext|>
PARAMETER temperature 0.7
PARAMETER top_k 20
PARAMETER top_p 0.8
PARAMETER num_ctx 32768 ← fixed (was missing)
PARAMETER num_predict 8192 ← fixed (was missing)
```
Model specs: 30.5B params, MoE architecture (qwen3moe), Q4_K_M quantization, supports tools + completion.
---
## Issue & Fix — Tool Call JSON Truncation
### Error
```
llama-server returned invalid tool call arguments for "edit":
unexpected end of ... [retrying in 51s attempt N]
```
### Root Cause
Two Ollama parameters were missing from the model's Modelfile:
| Parameter | Default when missing | Required for tool calls |
|-----------|---------------------|------------------------|
| `num_ctx` | **2048 tokens** | Minimum ~8192; 32768 recommended |
| `num_predict` | **128 tokens** | Minimum ~1024; 8192 recommended |
With only 128 output tokens allowed, the model began generating valid JSON for the `edit` tool call but ran out of budget before closing all braces, producing malformed JSON. The 2048-token context also filled up quickly during long sessions, compounding the problem.
### Fix Applied
Rebuilt `qwen3-coder:30b-fixed` with the two parameters added:
```bash
# Export current Modelfile, append fixes, rebuild
ollama show qwen3-coder:30b-fixed --modelfile > Modelfile_fixed
# (add num_ctx 32768 and num_predict 8192)
ollama create qwen3-coder:30b-fixed -f Modelfile_fixed
```
No changes needed to the OpenCode config — same model name, same config file.
### Verification
```bash
ollama show qwen3-coder:30b-fixed
# Should show: num_ctx 32768 and num_predict 8192 in Parameters section
```
---
## Aider Configuration
**Config file:** `.aider.conf.yml` in project root
```yaml
model: ollama/qwen3-coder:30b
openai-api-base: http://localhost:11434/v1
openai-api-key: ollama
edit-format: whole
map-tokens: 2048
no-auto-commits: true
system-prompt: "Always respond in English only. Never use Chinese or any other language."
```
| Option | Why |
|--------|-----|
| `edit-format: whole` | More reliable file writes for local models |
| `map-tokens: 2048` | Limits repo map size for smaller context windows |
| `no-auto-commits: true` | Review changes before they hit git |
| `system-prompt` | Qwen3 defaults to Chinese; forces English |
### Aider Quick Reference
| Command | What it does |
|---------|-------------|
| `/add src/foo.py` | Add file to context |
| `/drop src/foo.py` | Remove file from context |
| `/diff` | See pending changes |
| `/undo` | Revert last change |
| `/run pytest` | Run command, share output with model |
| `/clear` | Clear chat history (keep files) |
| `/settings` | Show loaded config |
| `/model ollama/gemma4:12b` | Switch model on the fly |
---
## Known Issues & Fixes
### Tool call JSON truncated (OpenCode / llama-server)
**Error:** `llama-server returned invalid tool call arguments for "edit": unexpected end of...`
**Cause:** `num_predict` too low (default 128) cuts off JSON mid-generation.
**Fix:** See "Issue & Fix" section above — add `num_ctx` and `num_predict` to Modelfile.
### Model responds in Chinese
**Cause:** Qwen3 is a Chinese model and defaults to Chinese.
**Fix (Aider):** `system-prompt` in `.aider.conf.yml`.
**Fix (OpenCode):** Add a system prompt in OpenCode's project config if supported, or use `gemma4` models which default to English.
### Aider writes code to chat but not to disk
**Cause:** Model didn't follow Aider's edit format.
**Fix 1:** `/add target_file.py` before the request — hints the target.
**Fix 2:** `edit-format: whole` in config (already included above).
### Config changes not taking effect (Aider)
**Fix:** Exit (`/exit`) and restart. Config is read only at startup.
### "File not found" / repo-map warnings for deleted files
**Cause:** File deleted from disk but still tracked in git index.
**Fix:**
```bash
git rm --cached filename.py
# or from inside Aider:
/run git rm --cached filename.py
```
### Model loops with filler text, never executes tool calls
**Symptom:** Model repeatedly says "I'll continue..." but never reads/edits files. Context counter in OpenCode bottom-left shows a number near your `num_ctx` limit.
**Cause:** Context window is full. No tokens left for tool call JSON, so the model outputs short filler and stalls.
**Fix (immediate):** Start a new OpenCode session. Break large tasks (e.g. "translate all files") into one file per session.
**Fix (permanent):** Increase `num_ctx` in the Modelfile (e.g. 65536), rebuild, restart Ollama. Watch RAM usage — each doubling of `num_ctx` roughly doubles KV cache memory.
### Context fills up, model performance degrades
**Cause:** Long session accumulates tokens beyond `num_ctx`.
**Fix (Aider):** `/clear` to reset chat history without losing file context.
**Fix (Ollama model):** Increase `num_ctx` in Modelfile (current: 32768).
---
## Performance Notes
- qwen3-coder:30b is a MoE model — fits in VRAM+RAM split on RTX 5070 Ti (12 GB VRAM).
- `num_gpu 99` in Modelfile maximizes layers on VRAM; remainder spills to RAM.
- Safe `num_ctx` range for this hardware: **8192–32768**. Above 32768 risks RAM pressure.
- Q4_K_M quantization gives best quality/speed/memory balance for this size class.
---
## Environment Verification Checklist
```bash
ollama list # models appear
ollama show qwen3-coder:30b-fixed # num_ctx and num_predict present
curl http://localhost:11434/api/tags # Ollama serving
aider --version # Aider installed
opencode --version # OpenCode installed
nvidia-smi # GPU visible
```