581 lines
22 KiB
Plaintext
581 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b9c2fa10",
|
|
"metadata": {},
|
|
"source": [
|
|
"# makemore — the MLP character model\n",
|
|
"\n",
|
|
"Predict the next letter from the **previous few letters** with a small neural net — **concept -> code -> Your turn** each step.\n",
|
|
"\n",
|
|
"This is notebook **02** of the series. Notebook `00` built backprop by hand; the bigram in `01`\n",
|
|
"only ever looked at **one** character back. Here we take a real step up: a multi-layer\n",
|
|
"perceptron (MLP) that looks at a small **window** of previous characters, following Bengio et\n",
|
|
"al. 2003 and Karpathy's *\"makemore part 2: MLP\"* lecture.\n",
|
|
"\n",
|
|
"Dataset: `names.txt` — 32,033 real first names. Goal: **make more** name-like words."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "69f5f6bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prologue — in plain English\n",
|
|
"\n",
|
|
"A bigram only remembers the last letter, so it cannot tell that `bri` should probably be\n",
|
|
"followed by `a` (Brianna) rather than a random vowel. We fix that by letting the model read a\n",
|
|
"**block** of the last few letters at once.\n",
|
|
"\n",
|
|
"The trick that makes this work is the **embedding**: instead of treating each letter as an\n",
|
|
"isolated symbol, we give every letter a short list of numbers (a vector) that the network\n",
|
|
"learns. Letters that behave similarly drift close together.\n",
|
|
"\n",
|
|
"Real-life picture: imagine each letter is a person, and the embedding is their personality\n",
|
|
"profile. The network learns these profiles so that \"similar\" letters (say, the vowels) end up\n",
|
|
"with similar profiles, and it can generalise: a context it never saw can still be handled\n",
|
|
"because it resembles ones it did see."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2d5dee6c",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 0.1 Imports\n",
|
|
"\n",
|
|
"We import **torch first** (before matplotlib) to avoid a Windows OpenMP DLL clash, and make\n",
|
|
"plotting optional so the notebook runs even without matplotlib.\n",
|
|
"\n",
|
|
"**-> Training:** `torch` holds the weight tensors and does autograd; `F.cross_entropy` turns\n",
|
|
"logits + the true next letter into the single `loss` number we minimise."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "07c035c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\") # avoid duplicate-OpenMP crash on Windows\n",
|
|
"\n",
|
|
"import torch\n",
|
|
"import torch.nn.functional as F\n",
|
|
"import random\n",
|
|
"\n",
|
|
"try:\n",
|
|
" import matplotlib.pyplot as plt\n",
|
|
" HAS_PLT = True\n",
|
|
"except Exception:\n",
|
|
" HAS_PLT = False\n",
|
|
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
|
|
"\n",
|
|
"print(\"PyTorch version:\", torch.__version__)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dc58a8fc",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 0.2 Load the names and build the alphabet\n",
|
|
"\n",
|
|
"Each name becomes letters plus a special **`.`** marker (id 0) that means *start* and *end*.\n",
|
|
"So `emma` is really `. e m m a .` — the dots tell the model where words begin and stop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6b7e8198",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"words = open(\"names.txt\").read().splitlines()\n",
|
|
"print(\"number of names:\", len(words))\n",
|
|
"print(\"examples:\", words[:8])\n",
|
|
"print(\"shortest/longest:\", min(len(w) for w in words), \"/\", max(len(w) for w in words))\n",
|
|
"\n",
|
|
"chars = sorted(list(set(\"\".join(words))))\n",
|
|
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
|
|
"stoi[\".\"] = 0\n",
|
|
"itos = {i: ch for ch, i in stoi.items()}\n",
|
|
"vocab_size = len(itos)\n",
|
|
"print(\"vocab_size:\", vocab_size)\n",
|
|
"print(\"stoi:\", stoi)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d7a9d8ab",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 1.1 The context window (`block_size`)\n",
|
|
"\n",
|
|
"`block_size` = how many previous letters the model sees before guessing the next one. With\n",
|
|
"`block_size = 3`, predicting the letters of `emma` looks like this:\n",
|
|
"\n",
|
|
"```\n",
|
|
"... -> e (start: three dots, predict 'e')\n",
|
|
"..e -> m\n",
|
|
".em -> m\n",
|
|
"emm -> a\n",
|
|
"mma -> . (predict the end marker)\n",
|
|
"```\n",
|
|
"\n",
|
|
"Each arrow is one training example. Real-life picture: a sliding window of 3 letters moves\n",
|
|
"left to right across the word; at each stop the model must guess what comes next."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4c0464a0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"block_size = 3\n",
|
|
"\n",
|
|
"def build_dataset(word_list):\n",
|
|
" X, Y = [], []\n",
|
|
" for w in word_list:\n",
|
|
" context = [0] * block_size # start with '...'\n",
|
|
" for ch in w + \".\":\n",
|
|
" ix = stoi[ch]\n",
|
|
" X.append(context)\n",
|
|
" Y.append(ix)\n",
|
|
" context = context[1:] + [ix] # slide the window\n",
|
|
" return torch.tensor(X), torch.tensor(Y)\n",
|
|
"\n",
|
|
"# show the examples for one name\n",
|
|
"print(\"examples generated from 'emma':\")\n",
|
|
"Xtmp, Ytmp = build_dataset([\"emma\"])\n",
|
|
"for ctx, target in zip(Xtmp.tolist(), Ytmp.tolist()):\n",
|
|
" print(\"\".join(itos[i] for i in ctx), \"->\", itos[target])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5f607e40",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 1.2 Train / dev / test split\n",
|
|
"\n",
|
|
"We shuffle the names and cut three piles:\n",
|
|
"\n",
|
|
"- **train (80%)** — the model learns from these.\n",
|
|
"- **dev (10%)** — we tune choices (size, learning rate) against these.\n",
|
|
"- **test (10%)** — touched only at the very end, to get an honest score.\n",
|
|
"\n",
|
|
"Real-life picture: train = homework you study, dev = practice quiz you check yourself against,\n",
|
|
"test = the final exam you only sit once."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5c8e4f0c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"random.seed(42)\n",
|
|
"random.shuffle(words)\n",
|
|
"n1 = int(0.8 * len(words))\n",
|
|
"n2 = int(0.9 * len(words))\n",
|
|
"\n",
|
|
"Xtr, Ytr = build_dataset(words[:n1])\n",
|
|
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
|
|
"Xte, Yte = build_dataset(words[n2:])\n",
|
|
"\n",
|
|
"print(\"train:\", tuple(Xtr.shape), \"dev:\", tuple(Xdev.shape), \"test:\", tuple(Xte.shape))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b10d4c19",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 2.1 The model — embedding, hidden layer, output\n",
|
|
"\n",
|
|
"Three learnable pieces:\n",
|
|
"\n",
|
|
"1. **`C`** — the embedding table: each of the 27 symbols gets an `n_embd`-dim vector.\n",
|
|
"2. **`W1, b1`** — a hidden layer with `tanh`: it mixes the 3 letter-vectors into `n_hidden`\n",
|
|
" features. Analogy: a committee that reads the 3 personalities and forms opinions.\n",
|
|
"3. **`W2, b2`** — the output layer: turns those opinions into `vocab_size` scores (logits) for\n",
|
|
" the next letter.\n",
|
|
"\n",
|
|
"**-> Training:** `parameters` is the full list of knobs; every one gets a gradient and a nudge."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4a0bcc2f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"g = torch.Generator().manual_seed(2147483647)\n",
|
|
"\n",
|
|
"n_embd = 10 # size of each letter's vector\n",
|
|
"n_hidden = 200 # neurons in the hidden layer\n",
|
|
"\n",
|
|
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
|
|
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * 0.2\n",
|
|
"b1 = torch.randn(n_hidden, generator=g) * 0.01\n",
|
|
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
|
|
"b2 = torch.randn(vocab_size, generator=g) * 0.0\n",
|
|
"\n",
|
|
"parameters = [C, W1, b1, W2, b2]\n",
|
|
"for p in parameters:\n",
|
|
" p.requires_grad = True\n",
|
|
"\n",
|
|
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "77778ae3",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 2.2 One forward pass, step by step\n",
|
|
"\n",
|
|
"Watch the shapes flow. The key move is `emb.view(-1, n_embd*block_size)`: it **flattens** the\n",
|
|
"3 letter-vectors of each example into one long row so the hidden layer can read them together.\n",
|
|
"\n",
|
|
"**-> Training:** `F.cross_entropy` = softmax + negative-log-likelihood in one numerically-stable\n",
|
|
"call; lower means more probability on the true next letter."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "13123079",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"ix = torch.randint(0, Xtr.shape[0], (4,), generator=g) # 4 random examples\n",
|
|
"Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
|
"\n",
|
|
"emb = C[Xb] # (4, 3, n_embd)\n",
|
|
"flat = emb.view(-1, n_embd * block_size) # (4, 30)\n",
|
|
"h = torch.tanh(flat @ W1 + b1) # (4, n_hidden)\n",
|
|
"logits = h @ W2 + b2 # (4, vocab_size)\n",
|
|
"loss = F.cross_entropy(logits, Yb)\n",
|
|
"\n",
|
|
"print(\"emb :\", tuple(emb.shape))\n",
|
|
"print(\"flat :\", tuple(flat.shape))\n",
|
|
"print(\"h :\", tuple(h.shape))\n",
|
|
"print(\"logits:\", tuple(logits.shape))\n",
|
|
"print(\"loss :\", loss.item())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b90331ba",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 3.1 Finding a good learning rate\n",
|
|
"\n",
|
|
"The learning rate (how big each nudge is) matters a lot: too small and training crawls, too\n",
|
|
"big and the loss bounces or explodes. A cheap way to find a good one: sweep the rate from tiny\n",
|
|
"to large over a few hundred steps and watch where the loss drops fastest.\n",
|
|
"\n",
|
|
"Real-life picture: testing how big a step to take downhill in fog — tiny steps are safe but\n",
|
|
"slow; huge steps overshoot the valley.\n",
|
|
"\n",
|
|
"We use a **fresh** set of weights here so the search does not disturb the real model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "acae02ad",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fresh throwaway weights just for the search\n",
|
|
"gs = torch.Generator().manual_seed(2147483647)\n",
|
|
"Cs = torch.randn((vocab_size, n_embd), generator=gs)\n",
|
|
"W1s = torch.randn((n_embd * block_size, n_hidden), generator=gs) * 0.2\n",
|
|
"b1s = torch.randn(n_hidden, generator=gs) * 0.01\n",
|
|
"W2s = torch.randn((n_hidden, vocab_size), generator=gs) * 0.01\n",
|
|
"b2s = torch.randn(vocab_size, generator=gs) * 0.0\n",
|
|
"params_s = [Cs, W1s, b1s, W2s, b2s]\n",
|
|
"for p in params_s:\n",
|
|
" p.requires_grad = True\n",
|
|
"\n",
|
|
"# candidate learning rates spaced from 10^-3 up to 10^0\n",
|
|
"lre = torch.linspace(-3, 0, 1000)\n",
|
|
"lrs = 10 ** lre\n",
|
|
"lr_used, losses = [], []\n",
|
|
"\n",
|
|
"for i in range(1000):\n",
|
|
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=gs)\n",
|
|
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
|
" emb = Cs[Xb]\n",
|
|
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1s + b1s)\n",
|
|
" loss = F.cross_entropy(h @ W2s + b2s, Yb)\n",
|
|
" for p in params_s:\n",
|
|
" p.grad = None\n",
|
|
" loss.backward()\n",
|
|
" lr = lrs[i].item()\n",
|
|
" for p in params_s:\n",
|
|
" p.data += -lr * p.grad\n",
|
|
" lr_used.append(lre[i].item())\n",
|
|
" losses.append(loss.item())\n",
|
|
"\n",
|
|
"best = lre[min(range(len(losses)), key=lambda k: losses[k])].item()\n",
|
|
"print(f\"loss dropped fastest around 10^{best:.2f} ~= {10**best:.3f}\")\n",
|
|
"print(\"good practical choice: lr = 0.1\")\n",
|
|
"\n",
|
|
"if HAS_PLT:\n",
|
|
" plt.figure(figsize=(5, 4))\n",
|
|
" plt.plot(lr_used, losses)\n",
|
|
" plt.xlabel(\"log10(learning rate)\")\n",
|
|
" plt.ylabel(\"minibatch loss\")\n",
|
|
" plt.title(\"learning-rate sweep\")\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "13569252",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 4.1 Train the model\n",
|
|
"\n",
|
|
"Now the real run on the actual `parameters`. Each step:\n",
|
|
"\n",
|
|
"1. grab a random **minibatch** of 32 examples (fast, noisy, good enough)\n",
|
|
"2. forward -> loss\n",
|
|
"3. `loss.backward()` -> gradients\n",
|
|
"4. nudge every parameter downhill\n",
|
|
"\n",
|
|
"We start at `lr = 0.1` and **decay** to `0.01` near the end to settle into the valley."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c5ec35d5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"max_steps = 20000\n",
|
|
"batch_size = 32\n",
|
|
"lossi = []\n",
|
|
"\n",
|
|
"for i in range(max_steps):\n",
|
|
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
|
|
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
|
"\n",
|
|
" emb = C[Xb]\n",
|
|
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
|
|
" logits = h @ W2 + b2\n",
|
|
" loss = F.cross_entropy(logits, Yb)\n",
|
|
"\n",
|
|
" for p in parameters:\n",
|
|
" p.grad = None\n",
|
|
" loss.backward()\n",
|
|
"\n",
|
|
" lr = 0.1 if i < 15000 else 0.01\n",
|
|
" for p in parameters:\n",
|
|
" p.data += -lr * p.grad\n",
|
|
"\n",
|
|
" lossi.append(loss.log10().item())\n",
|
|
" if i % 4000 == 0 or i == max_steps - 1:\n",
|
|
" print(f\"step {i:5d}/{max_steps} minibatch loss {loss.item():.4f}\")\n",
|
|
"\n",
|
|
"if HAS_PLT:\n",
|
|
" plt.figure(figsize=(5, 4))\n",
|
|
" plt.plot(lossi)\n",
|
|
" plt.xlabel(\"step\")\n",
|
|
" plt.ylabel(\"log10(loss)\")\n",
|
|
" plt.title(\"training loss\")\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e6baab73",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 4.2 Honest scores on train and dev\n",
|
|
"\n",
|
|
"Minibatch loss is noisy. Evaluate on the **whole** train and dev sets for a stable number.\n",
|
|
"If dev loss is much worse than train, the model is memorising; if they are close, it is\n",
|
|
"generalising. (A plain bigram on names scores around 2.45 — we should beat that.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "579fe1c8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"@torch.no_grad()\n",
|
|
"def split_loss(split):\n",
|
|
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
|
|
" emb = C[x]\n",
|
|
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
|
|
" logits = h @ W2 + b2\n",
|
|
" return F.cross_entropy(logits, y).item()\n",
|
|
"\n",
|
|
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
|
|
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "054682f1",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 5.1 Generate new names\n",
|
|
"\n",
|
|
"Sampling, letter by letter: start from `...`, predict a distribution, **draw** one letter,\n",
|
|
"slide the window, repeat until the model emits the end marker `.`.\n",
|
|
"\n",
|
|
"These are brand-new names the model invented — not copied from the file."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "06bf121c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
|
|
"\n",
|
|
"for _ in range(20):\n",
|
|
" out = []\n",
|
|
" context = [0] * block_size\n",
|
|
" while True:\n",
|
|
" emb = C[torch.tensor([context])]\n",
|
|
" h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
|
|
" logits = h @ W2 + b2\n",
|
|
" probs = F.softmax(logits, dim=1)\n",
|
|
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
|
|
" context = context[1:] + [ix]\n",
|
|
" if ix == 0:\n",
|
|
" break\n",
|
|
" out.append(itos[ix])\n",
|
|
" print(\"\".join(out))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0a0de526",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Your turn 5** — Re-run section 5.1 a few times (different random draws) and read the\n",
|
|
"names. Then try raising `n_hidden` (e.g. 300) or `block_size` (e.g. 4 or 5) back in section 2,\n",
|
|
"retrain, and see whether dev loss improves and the names look more realistic."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc2003e7",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 6.1 Bonus — visualise letters in 2-D\n",
|
|
"\n",
|
|
"When the embedding is 2-D we can literally plot it. We quickly train a tiny `n_embd = 2`\n",
|
|
"version and scatter the 27 symbols. Watch the **vowels** drift into their own neighbourhood —\n",
|
|
"the network discovered that they play a similar role, without ever being told what a vowel is.\n",
|
|
"\n",
|
|
"(Optional: needs matplotlib. Smaller/shorter than the main model, so the clustering is rough.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7786f75a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"g3 = torch.Generator().manual_seed(2147483647)\n",
|
|
"C2 = torch.randn((vocab_size, 2), generator=g3)\n",
|
|
"W1b = torch.randn((2 * block_size, n_hidden), generator=g3) * 0.2\n",
|
|
"b1b = torch.randn(n_hidden, generator=g3) * 0.01\n",
|
|
"W2b = torch.randn((n_hidden, vocab_size), generator=g3) * 0.01\n",
|
|
"b2b = torch.randn(vocab_size, generator=g3) * 0.0\n",
|
|
"params2 = [C2, W1b, b1b, W2b, b2b]\n",
|
|
"for p in params2:\n",
|
|
" p.requires_grad = True\n",
|
|
"\n",
|
|
"for i in range(8000):\n",
|
|
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=g3)\n",
|
|
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
|
" emb = C2[Xb]\n",
|
|
" h = torch.tanh(emb.view(-1, 2 * block_size) @ W1b + b1b)\n",
|
|
" loss = F.cross_entropy(h @ W2b + b2b, Yb)\n",
|
|
" for p in params2:\n",
|
|
" p.grad = None\n",
|
|
" loss.backward()\n",
|
|
" for p in params2:\n",
|
|
" p.data += -0.1 * p.grad\n",
|
|
"\n",
|
|
"print(\"2-D model dev-ish minibatch loss:\", round(loss.item(), 4))\n",
|
|
"\n",
|
|
"if HAS_PLT:\n",
|
|
" pts = C2.detach()\n",
|
|
" plt.figure(figsize=(6, 6))\n",
|
|
" plt.scatter(pts[:, 0], pts[:, 1], s=300)\n",
|
|
" for i in range(vocab_size):\n",
|
|
" plt.text(pts[i, 0].item(), pts[i, 1].item(), itos[i],\n",
|
|
" ha=\"center\", va=\"center\", color=\"white\")\n",
|
|
" plt.title(\"learned 2-D letter embedding (look for the vowels clustering)\")\n",
|
|
" plt.grid(True)\n",
|
|
" plt.show()\n",
|
|
"else:\n",
|
|
" print(\"matplotlib missing - skipping the embedding scatter\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c5e5b7a7",
|
|
"metadata": {},
|
|
"source": [
|
|
"## What's next\n",
|
|
"\n",
|
|
"You went from \"one letter back\" (bigram) to a real neural net that reads a **window** of\n",
|
|
"letters through a learned **embedding**, and it makes noticeably better names.\n",
|
|
"\n",
|
|
"Where this leads in the series:\n",
|
|
"\n",
|
|
"- `03_batchnorm_activations.ipynb` — why deep nets are fragile, and how BatchNorm keeps the\n",
|
|
" signal healthy so we can train bigger models.\n",
|
|
"- `04_backprop_ninja.ipynb` — compute all these gradients **by hand**, cementing notebook `00`.\n",
|
|
"- `06_build_gpt_attention.ipynb` — replace the fixed window with **attention**, so the model\n",
|
|
" can look back as far as it needs.\n",
|
|
"\n",
|
|
"**Checklist**\n",
|
|
"- [ ] Names -> context windows (`block_size`) -> (X, Y)\n",
|
|
"- [ ] train / dev / test split\n",
|
|
"- [ ] embedding `C` + hidden `tanh` + output logits\n",
|
|
"- [ ] cross-entropy loss, minibatch training\n",
|
|
"- [ ] learning-rate search\n",
|
|
"- [ ] sampled new names; visualised the 2-D embedding"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": ".venv (3.12.10)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python",
|
|
"version": "3.12.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |