c

Co-authored-by: aider (ollama/gemma4:12b) <aider@aider.chat>

02_makemore_mlp.ipynb

@@ -0,0 +1,581 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "b9c2fa10",
+      "metadata": {},
+      "source": [
+        "# makemore — the MLP character model\n",
+        "\n",
+        "Predict the next letter from the **previous few letters** with a small neural net — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **02** of the series. Notebook `00` built backprop by hand; the bigram in `01`\n",
+        "only ever looked at **one** character back. Here we take a real step up: a multi-layer\n",
+        "perceptron (MLP) that looks at a small **window** of previous characters, following Bengio et\n",
+        "al. 2003 and Karpathy's *\"makemore part 2: MLP\"* lecture.\n",
+        "\n",
+        "Dataset: `names.txt` — 32,033 real first names. Goal: **make more** name-like words."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "69f5f6bc",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "A bigram only remembers the last letter, so it cannot tell that `bri` should probably be\n",
+        "followed by `a` (Brianna) rather than a random vowel. We fix that by letting the model read a\n",
+        "**block** of the last few letters at once.\n",
+        "\n",
+        "The trick that makes this work is the **embedding**: instead of treating each letter as an\n",
+        "isolated symbol, we give every letter a short list of numbers (a vector) that the network\n",
+        "learns. Letters that behave similarly drift close together.\n",
+        "\n",
+        "Real-life picture: imagine each letter is a person, and the embedding is their personality\n",
+        "profile. The network learns these profiles so that \"similar\" letters (say, the vowels) end up\n",
+        "with similar profiles, and it can generalise: a context it never saw can still be handled\n",
+        "because it resembles ones it did see."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2d5dee6c",
+      "metadata": {},
+      "source": [
+        "### 0.1 Imports\n",
+        "\n",
+        "We import **torch first** (before matplotlib) to avoid a Windows OpenMP DLL clash, and make\n",
+        "plotting optional so the notebook runs even without matplotlib.\n",
+        "\n",
+        "**-> Training:** `torch` holds the weight tensors and does autograd; `F.cross_entropy` turns\n",
+        "logits + the true next letter into the single `loss` number we minimise."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "07c035c6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")  # avoid duplicate-OpenMP crash on Windows\n",
+        "\n",
+        "import torch\n",
+        "import torch.nn.functional as F\n",
+        "import random\n",
+        "\n",
+        "try:\n",
+        "    import matplotlib.pyplot as plt\n",
+        "    HAS_PLT = True\n",
+        "except Exception:\n",
+        "    HAS_PLT = False\n",
+        "    print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
+        "\n",
+        "print(\"PyTorch version:\", torch.__version__)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dc58a8fc",
+      "metadata": {},
+      "source": [
+        "### 0.2 Load the names and build the alphabet\n",
+        "\n",
+        "Each name becomes letters plus a special **`.`** marker (id 0) that means *start* and *end*.\n",
+        "So `emma` is really `. e m m a .` — the dots tell the model where words begin and stop."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "6b7e8198",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "words = open(\"names.txt\").read().splitlines()\n",
+        "print(\"number of names:\", len(words))\n",
+        "print(\"examples:\", words[:8])\n",
+        "print(\"shortest/longest:\", min(len(w) for w in words), \"/\", max(len(w) for w in words))\n",
+        "\n",
+        "chars = sorted(list(set(\"\".join(words))))\n",
+        "stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
+        "stoi[\".\"] = 0\n",
+        "itos = {i: ch for ch, i in stoi.items()}\n",
+        "vocab_size = len(itos)\n",
+        "print(\"vocab_size:\", vocab_size)\n",
+        "print(\"stoi:\", stoi)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d7a9d8ab",
+      "metadata": {},
+      "source": [
+        "### 1.1 The context window (`block_size`)\n",
+        "\n",
+        "`block_size` = how many previous letters the model sees before guessing the next one. With\n",
+        "`block_size = 3`, predicting the letters of `emma` looks like this:\n",
+        "\n",
+        "```\n",
+        "...  -> e      (start: three dots, predict 'e')\n",
+        "..e  -> m\n",
+        ".em  -> m\n",
+        "emm  -> a\n",
+        "mma  -> .      (predict the end marker)\n",
+        "```\n",
+        "\n",
+        "Each arrow is one training example. Real-life picture: a sliding window of 3 letters moves\n",
+        "left to right across the word; at each stop the model must guess what comes next."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "4c0464a0",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "block_size = 3\n",
+        "\n",
+        "def build_dataset(word_list):\n",
+        "    X, Y = [], []\n",
+        "    for w in word_list:\n",
+        "        context = [0] * block_size           # start with '...'\n",
+        "        for ch in w + \".\":\n",
+        "            ix = stoi[ch]\n",
+        "            X.append(context)\n",
+        "            Y.append(ix)\n",
+        "            context = context[1:] + [ix]      # slide the window\n",
+        "    return torch.tensor(X), torch.tensor(Y)\n",
+        "\n",
+        "# show the examples for one name\n",
+        "print(\"examples generated from 'emma':\")\n",
+        "Xtmp, Ytmp = build_dataset([\"emma\"])\n",
+        "for ctx, target in zip(Xtmp.tolist(), Ytmp.tolist()):\n",
+        "    print(\"\".join(itos[i] for i in ctx), \"->\", itos[target])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5f607e40",
+      "metadata": {},
+      "source": [
+        "### 1.2 Train / dev / test split\n",
+        "\n",
+        "We shuffle the names and cut three piles:\n",
+        "\n",
+        "- **train (80%)** — the model learns from these.\n",
+        "- **dev (10%)** — we tune choices (size, learning rate) against these.\n",
+        "- **test (10%)** — touched only at the very end, to get an honest score.\n",
+        "\n",
+        "Real-life picture: train = homework you study, dev = practice quiz you check yourself against,\n",
+        "test = the final exam you only sit once."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5c8e4f0c",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "random.seed(42)\n",
+        "random.shuffle(words)\n",
+        "n1 = int(0.8 * len(words))\n",
+        "n2 = int(0.9 * len(words))\n",
+        "\n",
+        "Xtr, Ytr = build_dataset(words[:n1])\n",
+        "Xdev, Ydev = build_dataset(words[n1:n2])\n",
+        "Xte, Yte = build_dataset(words[n2:])\n",
+        "\n",
+        "print(\"train:\", tuple(Xtr.shape), \"dev:\", tuple(Xdev.shape), \"test:\", tuple(Xte.shape))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b10d4c19",
+      "metadata": {},
+      "source": [
+        "### 2.1 The model — embedding, hidden layer, output\n",
+        "\n",
+        "Three learnable pieces:\n",
+        "\n",
+        "1. **`C`** — the embedding table: each of the 27 symbols gets an `n_embd`-dim vector.\n",
+        "2. **`W1, b1`** — a hidden layer with `tanh`: it mixes the 3 letter-vectors into `n_hidden`\n",
+        "   features. Analogy: a committee that reads the 3 personalities and forms opinions.\n",
+        "3. **`W2, b2`** — the output layer: turns those opinions into `vocab_size` scores (logits) for\n",
+        "   the next letter.\n",
+        "\n",
+        "**-> Training:** `parameters` is the full list of knobs; every one gets a gradient and a nudge."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "4a0bcc2f",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g = torch.Generator().manual_seed(2147483647)\n",
+        "\n",
+        "n_embd = 10        # size of each letter's vector\n",
+        "n_hidden = 200     # neurons in the hidden layer\n",
+        "\n",
+        "C  = torch.randn((vocab_size, n_embd),            generator=g)\n",
+        "W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * 0.2\n",
+        "b1 = torch.randn(n_hidden,                         generator=g) * 0.01\n",
+        "W2 = torch.randn((n_hidden, vocab_size),          generator=g) * 0.01\n",
+        "b2 = torch.randn(vocab_size,                       generator=g) * 0.0\n",
+        "\n",
+        "parameters = [C, W1, b1, W2, b2]\n",
+        "for p in parameters:\n",
+        "    p.requires_grad = True\n",
+        "\n",
+        "print(\"total parameters:\", sum(p.nelement() for p in parameters))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "77778ae3",
+      "metadata": {},
+      "source": [
+        "### 2.2 One forward pass, step by step\n",
+        "\n",
+        "Watch the shapes flow. The key move is `emb.view(-1, n_embd*block_size)`: it **flattens** the\n",
+        "3 letter-vectors of each example into one long row so the hidden layer can read them together.\n",
+        "\n",
+        "**-> Training:** `F.cross_entropy` = softmax + negative-log-likelihood in one numerically-stable\n",
+        "call; lower means more probability on the true next letter."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "13123079",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ix = torch.randint(0, Xtr.shape[0], (4,), generator=g)   # 4 random examples\n",
+        "Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "emb = C[Xb]                                          # (4, 3, n_embd)\n",
+        "flat = emb.view(-1, n_embd * block_size)             # (4, 30)\n",
+        "h = torch.tanh(flat @ W1 + b1)                       # (4, n_hidden)\n",
+        "logits = h @ W2 + b2                                 # (4, vocab_size)\n",
+        "loss = F.cross_entropy(logits, Yb)\n",
+        "\n",
+        "print(\"emb   :\", tuple(emb.shape))\n",
+        "print(\"flat  :\", tuple(flat.shape))\n",
+        "print(\"h     :\", tuple(h.shape))\n",
+        "print(\"logits:\", tuple(logits.shape))\n",
+        "print(\"loss  :\", loss.item())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b90331ba",
+      "metadata": {},
+      "source": [
+        "### 3.1 Finding a good learning rate\n",
+        "\n",
+        "The learning rate (how big each nudge is) matters a lot: too small and training crawls, too\n",
+        "big and the loss bounces or explodes. A cheap way to find a good one: sweep the rate from tiny\n",
+        "to large over a few hundred steps and watch where the loss drops fastest.\n",
+        "\n",
+        "Real-life picture: testing how big a step to take downhill in fog — tiny steps are safe but\n",
+        "slow; huge steps overshoot the valley.\n",
+        "\n",
+        "We use a **fresh** set of weights here so the search does not disturb the real model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "acae02ad",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# fresh throwaway weights just for the search\n",
+        "gs = torch.Generator().manual_seed(2147483647)\n",
+        "Cs  = torch.randn((vocab_size, n_embd),            generator=gs)\n",
+        "W1s = torch.randn((n_embd * block_size, n_hidden), generator=gs) * 0.2\n",
+        "b1s = torch.randn(n_hidden,                         generator=gs) * 0.01\n",
+        "W2s = torch.randn((n_hidden, vocab_size),          generator=gs) * 0.01\n",
+        "b2s = torch.randn(vocab_size,                       generator=gs) * 0.0\n",
+        "params_s = [Cs, W1s, b1s, W2s, b2s]\n",
+        "for p in params_s:\n",
+        "    p.requires_grad = True\n",
+        "\n",
+        "# candidate learning rates spaced from 10^-3 up to 10^0\n",
+        "lre = torch.linspace(-3, 0, 1000)\n",
+        "lrs = 10 ** lre\n",
+        "lr_used, losses = [], []\n",
+        "\n",
+        "for i in range(1000):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (32,), generator=gs)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "    emb = Cs[Xb]\n",
+        "    h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1s + b1s)\n",
+        "    loss = F.cross_entropy(h @ W2s + b2s, Yb)\n",
+        "    for p in params_s:\n",
+        "        p.grad = None\n",
+        "    loss.backward()\n",
+        "    lr = lrs[i].item()\n",
+        "    for p in params_s:\n",
+        "        p.data += -lr * p.grad\n",
+        "    lr_used.append(lre[i].item())\n",
+        "    losses.append(loss.item())\n",
+        "\n",
+        "best = lre[min(range(len(losses)), key=lambda k: losses[k])].item()\n",
+        "print(f\"loss dropped fastest around 10^{best:.2f} ~= {10**best:.3f}\")\n",
+        "print(\"good practical choice: lr = 0.1\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    plt.figure(figsize=(5, 4))\n",
+        "    plt.plot(lr_used, losses)\n",
+        "    plt.xlabel(\"log10(learning rate)\")\n",
+        "    plt.ylabel(\"minibatch loss\")\n",
+        "    plt.title(\"learning-rate sweep\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "13569252",
+      "metadata": {},
+      "source": [
+        "### 4.1 Train the model\n",
+        "\n",
+        "Now the real run on the actual `parameters`. Each step:\n",
+        "\n",
+        "1. grab a random **minibatch** of 32 examples (fast, noisy, good enough)\n",
+        "2. forward -> loss\n",
+        "3. `loss.backward()` -> gradients\n",
+        "4. nudge every parameter downhill\n",
+        "\n",
+        "We start at `lr = 0.1` and **decay** to `0.01` near the end to settle into the valley."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c5ec35d5",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "max_steps = 20000\n",
+        "batch_size = 32\n",
+        "lossi = []\n",
+        "\n",
+        "for i in range(max_steps):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "    emb = C[Xb]\n",
+        "    h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
+        "    logits = h @ W2 + b2\n",
+        "    loss = F.cross_entropy(logits, Yb)\n",
+        "\n",
+        "    for p in parameters:\n",
+        "        p.grad = None\n",
+        "    loss.backward()\n",
+        "\n",
+        "    lr = 0.1 if i < 15000 else 0.01\n",
+        "    for p in parameters:\n",
+        "        p.data += -lr * p.grad\n",
+        "\n",
+        "    lossi.append(loss.log10().item())\n",
+        "    if i % 4000 == 0 or i == max_steps - 1:\n",
+        "        print(f\"step {i:5d}/{max_steps}   minibatch loss {loss.item():.4f}\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    plt.figure(figsize=(5, 4))\n",
+        "    plt.plot(lossi)\n",
+        "    plt.xlabel(\"step\")\n",
+        "    plt.ylabel(\"log10(loss)\")\n",
+        "    plt.title(\"training loss\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "e6baab73",
+      "metadata": {},
+      "source": [
+        "### 4.2 Honest scores on train and dev\n",
+        "\n",
+        "Minibatch loss is noisy. Evaluate on the **whole** train and dev sets for a stable number.\n",
+        "If dev loss is much worse than train, the model is memorising; if they are close, it is\n",
+        "generalising. (A plain bigram on names scores around 2.45 — we should beat that.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "579fe1c8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "@torch.no_grad()\n",
+        "def split_loss(split):\n",
+        "    x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
+        "    emb = C[x]\n",
+        "    h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
+        "    logits = h @ W2 + b2\n",
+        "    return F.cross_entropy(logits, y).item()\n",
+        "\n",
+        "print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
+        "print(\"dev   loss:\", round(split_loss(\"dev\"), 4))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "054682f1",
+      "metadata": {},
+      "source": [
+        "### 5.1 Generate new names\n",
+        "\n",
+        "Sampling, letter by letter: start from `...`, predict a distribution, **draw** one letter,\n",
+        "slide the window, repeat until the model emits the end marker `.`.\n",
+        "\n",
+        "These are brand-new names the model invented — not copied from the file."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "06bf121c",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
+        "\n",
+        "for _ in range(20):\n",
+        "    out = []\n",
+        "    context = [0] * block_size\n",
+        "    while True:\n",
+        "        emb = C[torch.tensor([context])]\n",
+        "        h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
+        "        logits = h @ W2 + b2\n",
+        "        probs = F.softmax(logits, dim=1)\n",
+        "        ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
+        "        context = context[1:] + [ix]\n",
+        "        if ix == 0:\n",
+        "            break\n",
+        "        out.append(itos[ix])\n",
+        "    print(\"\".join(out))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0a0de526",
+      "metadata": {},
+      "source": [
+        "**Your turn 5** — Re-run section 5.1 a few times (different random draws) and read the\n",
+        "names. Then try raising `n_hidden` (e.g. 300) or `block_size` (e.g. 4 or 5) back in section 2,\n",
+        "retrain, and see whether dev loss improves and the names look more realistic."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "bc2003e7",
+      "metadata": {},
+      "source": [
+        "### 6.1 Bonus — visualise letters in 2-D\n",
+        "\n",
+        "When the embedding is 2-D we can literally plot it. We quickly train a tiny `n_embd = 2`\n",
+        "version and scatter the 27 symbols. Watch the **vowels** drift into their own neighbourhood —\n",
+        "the network discovered that they play a similar role, without ever being told what a vowel is.\n",
+        "\n",
+        "(Optional: needs matplotlib. Smaller/shorter than the main model, so the clustering is rough.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "7786f75a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g3 = torch.Generator().manual_seed(2147483647)\n",
+        "C2  = torch.randn((vocab_size, 2),                generator=g3)\n",
+        "W1b = torch.randn((2 * block_size, n_hidden),     generator=g3) * 0.2\n",
+        "b1b = torch.randn(n_hidden,                        generator=g3) * 0.01\n",
+        "W2b = torch.randn((n_hidden, vocab_size),         generator=g3) * 0.01\n",
+        "b2b = torch.randn(vocab_size,                      generator=g3) * 0.0\n",
+        "params2 = [C2, W1b, b1b, W2b, b2b]\n",
+        "for p in params2:\n",
+        "    p.requires_grad = True\n",
+        "\n",
+        "for i in range(8000):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (32,), generator=g3)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "    emb = C2[Xb]\n",
+        "    h = torch.tanh(emb.view(-1, 2 * block_size) @ W1b + b1b)\n",
+        "    loss = F.cross_entropy(h @ W2b + b2b, Yb)\n",
+        "    for p in params2:\n",
+        "        p.grad = None\n",
+        "    loss.backward()\n",
+        "    for p in params2:\n",
+        "        p.data += -0.1 * p.grad\n",
+        "\n",
+        "print(\"2-D model dev-ish minibatch loss:\", round(loss.item(), 4))\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    pts = C2.detach()\n",
+        "    plt.figure(figsize=(6, 6))\n",
+        "    plt.scatter(pts[:, 0], pts[:, 1], s=300)\n",
+        "    for i in range(vocab_size):\n",
+        "        plt.text(pts[i, 0].item(), pts[i, 1].item(), itos[i],\n",
+        "                 ha=\"center\", va=\"center\", color=\"white\")\n",
+        "    plt.title(\"learned 2-D letter embedding (look for the vowels clustering)\")\n",
+        "    plt.grid(True)\n",
+        "    plt.show()\n",
+        "else:\n",
+        "    print(\"matplotlib missing - skipping the embedding scatter\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "c5e5b7a7",
+      "metadata": {},
+      "source": [
+        "## What's next\n",
+        "\n",
+        "You went from \"one letter back\" (bigram) to a real neural net that reads a **window** of\n",
+        "letters through a learned **embedding**, and it makes noticeably better names.\n",
+        "\n",
+        "Where this leads in the series:\n",
+        "\n",
+        "- `03_batchnorm_activations.ipynb` — why deep nets are fragile, and how BatchNorm keeps the\n",
+        "  signal healthy so we can train bigger models.\n",
+        "- `04_backprop_ninja.ipynb` — compute all these gradients **by hand**, cementing notebook `00`.\n",
+        "- `06_build_gpt_attention.ipynb` — replace the fixed window with **attention**, so the model\n",
+        "  can look back as far as it needs.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] Names -> context windows (`block_size`) -> (X, Y)\n",
+        "- [ ] train / dev / test split\n",
+        "- [ ] embedding `C` + hidden `tanh` + output logits\n",
+        "- [ ] cross-entropy loss, minibatch training\n",
+        "- [ ] learning-rate search\n",
+        "- [ ] sampled new names; visualised the 2-D embedding"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

03_batchnorm_activations.ipynb

@@ -0,0 +1,496 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "aedd6523",
+      "metadata": {},
+      "source": [
+        "# makemore — activations, gradients, and BatchNorm\n",
+        "\n",
+        "Make the MLP **trainable and healthy** — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **03**. In `02` the MLP worked, but we got a bit lucky with how the weights\n",
+        "started. Here we look *inside* the network at the numbers flowing through it, see two common\n",
+        "ways training goes wrong, and fix them — first by hand (good initialisation), then with the\n",
+        "robust tool everyone uses: **Batch Normalization**. Follows Karpathy's *\"makemore part 3\"*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2980029a",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "A neural net is a stack of layers. At each layer numbers flow forward (activations) and\n",
+        "gradients flow backward. If those numbers get **too big** or **too small**, training stalls.\n",
+        "\n",
+        "Two failures we will see and fix:\n",
+        "\n",
+        "1. **Confidently wrong at the start.** If the output layer starts with large random weights,\n",
+        "   the model makes wild over-confident guesses, so the first loss is huge and the first many\n",
+        "   steps are wasted just calming it down.\n",
+        "2. **Saturated `tanh`.** If the hidden numbers are too spread out, `tanh` slams them to -1 or\n",
+        "   +1. There the curve is flat, so its gradient is ~0 and those neurons stop learning — they\n",
+        "   are \"dead.\"\n",
+        "\n",
+        "Real-life picture: think of each layer as an audio stage. Too quiet and you hear nothing; too\n",
+        "loud and everything **clips** into distortion. We want every stage at a healthy volume.\n",
+        "**BatchNorm** is the auto-leveler that keeps the volume right at every stage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "18a296f3",
+      "metadata": {},
+      "source": [
+        "### 0.1 Imports\n",
+        "\n",
+        "`torch` first (avoids a Windows OpenMP DLL clash), matplotlib optional for the diagnostics."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ac69da81",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
+        "\n",
+        "import math\n",
+        "import random\n",
+        "import torch\n",
+        "import torch.nn.functional as F\n",
+        "\n",
+        "try:\n",
+        "    import matplotlib.pyplot as plt\n",
+        "    HAS_PLT = True\n",
+        "except Exception:\n",
+        "    HAS_PLT = False\n",
+        "    print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
+        "\n",
+        "print(\"PyTorch version:\", torch.__version__)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "92143322",
+      "metadata": {},
+      "source": [
+        "### 0.2 Data — same names, same context windows as notebook 02"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "d99c17df",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "words = open(\"names.txt\").read().splitlines()\n",
+        "chars = sorted(list(set(\"\".join(words))))\n",
+        "stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
+        "stoi[\".\"] = 0\n",
+        "itos = {i: ch for ch, i in stoi.items()}\n",
+        "vocab_size = len(itos)\n",
+        "block_size = 3\n",
+        "\n",
+        "def build_dataset(word_list):\n",
+        "    X, Y = [], []\n",
+        "    for w in word_list:\n",
+        "        context = [0] * block_size\n",
+        "        for ch in w + \".\":\n",
+        "            ix = stoi[ch]\n",
+        "            X.append(context)\n",
+        "            Y.append(ix)\n",
+        "            context = context[1:] + [ix]\n",
+        "    return torch.tensor(X), torch.tensor(Y)\n",
+        "\n",
+        "random.seed(42)\n",
+        "random.shuffle(words)\n",
+        "n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
+        "Xtr, Ytr = build_dataset(words[:n1])\n",
+        "Xdev, Ydev = build_dataset(words[n1:n2])\n",
+        "Xte, Yte = build_dataset(words[n2:])\n",
+        "print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "36b836bc",
+      "metadata": {},
+      "source": [
+        "### 1.1 Problem 1 — confidently wrong at the start\n",
+        "\n",
+        "At step 0 the model knows nothing, so the *best* it can do is spread probability evenly over\n",
+        "all 27 symbols. That ideal first loss is `-ln(1/27) = ln(27) ~= 3.30`.\n",
+        "\n",
+        "But with large random output weights the logits are all over the place, so the first loss is\n",
+        "much higher. Let's measure it."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "1219a887",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g0 = torch.Generator().manual_seed(2147483647)\n",
+        "n_embd, n_hidden = 10, 200\n",
+        "\n",
+        "# deliberately naive (large) initialisation\n",
+        "Cn  = torch.randn((vocab_size, n_embd),            generator=g0)\n",
+        "W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
+        "b1n = torch.randn(n_hidden,                         generator=g0)\n",
+        "W2n = torch.randn((n_hidden, vocab_size),          generator=g0)\n",
+        "b2n = torch.randn(vocab_size,                       generator=g0)\n",
+        "\n",
+        "emb = Cn[Xtr[:32]]\n",
+        "h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
+        "logits = h @ W2n + b2n\n",
+        "print(\"naive initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
+        "print(\"ideal  (ln 27)     :\", math.log(27))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "4c418501",
+      "metadata": {},
+      "source": [
+        "### 1.2 Fix 1 — make the output layer humble at the start\n",
+        "\n",
+        "We want the starting logits near zero (no opinion yet). So scale `W2` **down** and set `b2`\n",
+        "to zero. Now the first loss sits right near the ideal `~3.30`, and no steps are wasted."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "e1b9bb4a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g0 = torch.Generator().manual_seed(2147483647)\n",
+        "Cn  = torch.randn((vocab_size, n_embd),            generator=g0)\n",
+        "W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
+        "b1n = torch.randn(n_hidden,                         generator=g0)\n",
+        "W2n = torch.randn((n_hidden, vocab_size),          generator=g0) * 0.01   # tiny\n",
+        "b2n = torch.randn(vocab_size,                       generator=g0) * 0.0    # zero\n",
+        "\n",
+        "emb = Cn[Xtr[:32]]\n",
+        "h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
+        "logits = h @ W2n + b2n\n",
+        "print(\"fixed initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
+        "print(\"ideal  (ln 27)     :\", math.log(27))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2ab522f7",
+      "metadata": {},
+      "source": [
+        "### 2.1 Problem 2 — saturated `tanh` (dead neurons)\n",
+        "\n",
+        "Now look at the hidden activations `h = tanh(...)`. With large `W1`, the inputs to `tanh` are\n",
+        "very spread out, so most outputs are pinned at -1 or +1. That is the flat part of `tanh`,\n",
+        "where the gradient is ~0 — those neurons learn nothing.\n",
+        "\n",
+        "We measure the fraction of \"saturated\" activations (|h| > 0.99) and, if matplotlib is present,\n",
+        "show the tell-tale picture: bright cells are saturated; a fully bright column = a dead neuron."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "8fd25db6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "emb = Cn[Xtr[:1000]]\n",
+        "hpre = emb.view(-1, n_embd * block_size) @ W1n + b1n   # pre-activation\n",
+        "h = torch.tanh(hpre)\n",
+        "sat = (h.abs() > 0.99).float().mean().item()\n",
+        "print(f\"fraction of saturated activations: {sat:.1%}  (high = bad)\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    plt.figure(figsize=(10, 3))\n",
+        "    plt.imshow(h.abs() > 0.99, cmap=\"gray\", aspect=\"auto\")\n",
+        "    plt.title(\"white = saturated (|tanh| > 0.99); a full white column = a dead neuron\")\n",
+        "    plt.xlabel(\"neuron\")\n",
+        "    plt.ylabel(\"example\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "e11aee42",
+      "metadata": {},
+      "source": [
+        "### 2.2 Fix 2 — scale the hidden layer (Kaiming init)\n",
+        "\n",
+        "The cure is to make `W1` smaller so the pre-activations have a sensible spread (roughly unit\n",
+        "standard deviation). The principled recipe is **Kaiming init**: multiply random weights by\n",
+        "`gain / sqrt(fan_in)`, where `fan_in` is the number of inputs and the `gain` for `tanh` is\n",
+        "`5/3`.\n",
+        "\n",
+        "Real-life picture: setting the input volume so the signal fills the range without clipping."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "71f808d3",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "fan_in = n_embd * block_size\n",
+        "W1k = torch.randn((fan_in, n_hidden), generator=torch.Generator().manual_seed(1)) * ((5/3) / fan_in**0.5)\n",
+        "b1k = torch.zeros(n_hidden)\n",
+        "\n",
+        "emb = Cn[Xtr[:1000]]\n",
+        "h = torch.tanh(emb.view(-1, fan_in) @ W1k + b1k)\n",
+        "sat = (h.abs() > 0.99).float().mean().item()\n",
+        "print(f\"saturated now: {sat:.1%}  (much lower)\")\n",
+        "print(f\"activation std: {h.std().item():.3f}  (we want it near ~0.6-0.7 for tanh)\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    plt.figure(figsize=(5, 3))\n",
+        "    plt.hist(h.view(-1).tolist(), bins=50)\n",
+        "    plt.title(\"healthy tanh activations (not all piled at +/-1)\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "50d6f683",
+      "metadata": {},
+      "source": [
+        "### 3.1 BatchNorm — the auto-leveler\n",
+        "\n",
+        "Hand-tuning `W1` works, but in deep nets it is fiddly. **Batch Normalization** does it\n",
+        "automatically: before `tanh`, it **normalises** the pre-activations so that, across the\n",
+        "current batch, each neuron has mean 0 and std 1. Then it lets the network rescale/shift them\n",
+        "with two learnable knobs, `gamma` (gain) and `beta` (bias), in case 0/1 is not ideal.\n",
+        "\n",
+        "Real-life picture: an audio compressor that auto-levels every stage to a healthy volume, with\n",
+        "a manual trim (`gamma`, `beta`) if the engineer wants to nudge it.\n",
+        "\n",
+        "One catch: at test time we have no batch to average over, so during training we also keep a\n",
+        "**running** mean/std to use later for single predictions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5aeb780e",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g = torch.Generator().manual_seed(2147483647)\n",
+        "n_embd, n_hidden = 10, 200\n",
+        "fan_in = n_embd * block_size\n",
+        "\n",
+        "C  = torch.randn((vocab_size, n_embd),   generator=g)\n",
+        "W1 = torch.randn((fan_in, n_hidden),     generator=g) * ((5/3) / fan_in**0.5)\n",
+        "# no b1: BatchNorm's beta plays that role\n",
+        "W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
+        "b2 = torch.randn(vocab_size,             generator=g) * 0.0\n",
+        "\n",
+        "bngain = torch.ones((1, n_hidden))\n",
+        "bnbias = torch.zeros((1, n_hidden))\n",
+        "bnmean_running = torch.zeros((1, n_hidden))\n",
+        "bnstd_running  = torch.ones((1, n_hidden))\n",
+        "\n",
+        "parameters = [C, W1, W2, b2, bngain, bnbias]\n",
+        "for p in parameters:\n",
+        "    p.requires_grad = True\n",
+        "print(\"total parameters:\", sum(p.nelement() for p in parameters))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2b07d16a",
+      "metadata": {},
+      "source": [
+        "### 3.2 Train with BatchNorm\n",
+        "\n",
+        "Each step normalises the pre-activations using the **current batch**, updates the running\n",
+        "stats (for later), then continues as usual. Watch the loss fall, starting near the ideal 3.3."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "2410e7c4",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "max_steps = 20000\n",
+        "batch_size = 32\n",
+        "lossi = []\n",
+        "\n",
+        "for i in range(max_steps):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "    emb = C[Xb]\n",
+        "    hpreact = emb.view(-1, fan_in) @ W1\n",
+        "\n",
+        "    # --- BatchNorm ---\n",
+        "    bnmeani = hpreact.mean(0, keepdim=True)\n",
+        "    bnstdi = hpreact.std(0, keepdim=True)\n",
+        "    hpreact = bngain * (hpreact - bnmeani) / bnstdi + bnbias\n",
+        "    with torch.no_grad():\n",
+        "        bnmean_running = 0.999 * bnmean_running + 0.001 * bnmeani\n",
+        "        bnstd_running  = 0.999 * bnstd_running + 0.001 * bnstdi\n",
+        "    # -----------------\n",
+        "\n",
+        "    h = torch.tanh(hpreact)\n",
+        "    logits = h @ W2 + b2\n",
+        "    loss = F.cross_entropy(logits, Yb)\n",
+        "\n",
+        "    for p in parameters:\n",
+        "        p.grad = None\n",
+        "    loss.backward()\n",
+        "    lr = 0.1 if i < 15000 else 0.01\n",
+        "    for p in parameters:\n",
+        "        p.data += -lr * p.grad\n",
+        "\n",
+        "    lossi.append(loss.log10().item())\n",
+        "    if i % 4000 == 0 or i == max_steps - 1:\n",
+        "        print(f\"step {i:5d}/{max_steps}   loss {loss.item():.4f}\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    plt.figure(figsize=(5, 4))\n",
+        "    plt.plot(lossi)\n",
+        "    plt.xlabel(\"step\"); plt.ylabel(\"log10(loss)\"); plt.title(\"training loss (with BatchNorm)\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d80dccf6",
+      "metadata": {},
+      "source": [
+        "### 3.3 Evaluate — using the running stats\n",
+        "\n",
+        "At evaluation we have no batch, so we normalise with the **running** mean/std collected during\n",
+        "training. Train and dev loss should be close (good generalisation) and beat the bigram (~2.45)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "30ae8013",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "@torch.no_grad()\n",
+        "def split_loss(split):\n",
+        "    x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
+        "    emb = C[x]\n",
+        "    hpreact = emb.view(-1, fan_in) @ W1\n",
+        "    hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
+        "    h = torch.tanh(hpreact)\n",
+        "    logits = h @ W2 + b2\n",
+        "    return F.cross_entropy(logits, y).item()\n",
+        "\n",
+        "print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
+        "print(\"dev   loss:\", round(split_loss(\"dev\"), 4))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "269c8943",
+      "metadata": {},
+      "source": [
+        "### 4.1 Generate names with the BatchNorm model\n",
+        "\n",
+        "Same letter-by-letter sampling as before, but normalising with the running stats."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "63711cd1",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
+        "\n",
+        "for _ in range(20):\n",
+        "    out = []\n",
+        "    context = [0] * block_size\n",
+        "    while True:\n",
+        "        emb = C[torch.tensor([context])]\n",
+        "        hpreact = emb.view(1, -1) @ W1\n",
+        "        hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
+        "        h = torch.tanh(hpreact)\n",
+        "        logits = h @ W2 + b2\n",
+        "        probs = F.softmax(logits, dim=1)\n",
+        "        ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
+        "        context = context[1:] + [ix]\n",
+        "        if ix == 0:\n",
+        "            break\n",
+        "        out.append(itos[ix])\n",
+        "    print(\"\".join(out))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "365be7ef",
+      "metadata": {},
+      "source": [
+        "**Your turn 4** — In section 2.1, change `W1n`'s scale (e.g. `* 0.1` vs `* 3.0`) and re-run\n",
+        "the saturation check. Bigger scale -> more dead neurons. Then confirm BatchNorm (section 3.2)\n",
+        "trains fine even from a less careful init, because it re-levels the signal every step."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "65b60eec",
+      "metadata": {},
+      "source": [
+        "## What's next\n",
+        "\n",
+        "You learned to look **inside** the network and keep it healthy:\n",
+        "\n",
+        "- start the output layer humble (good first loss)\n",
+        "- keep `tanh` out of saturation (Kaiming init)\n",
+        "- or just use **BatchNorm** to auto-level every layer\n",
+        "\n",
+        "Where this leads in the series:\n",
+        "\n",
+        "- `04_backprop_ninja.ipynb` — compute every gradient here **by hand**, including through\n",
+        "  BatchNorm, to truly own backprop.\n",
+        "- `05_wavenet.ipynb` — stack these healthy layers into a deeper, hierarchical model.\n",
+        "- `06_build_gpt_attention.ipynb` — the transformer, which uses LayerNorm (a cousin of\n",
+        "  BatchNorm) for the same \"keep the signal healthy\" reason.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] Ideal initial loss = ln(vocab_size)\n",
+        "- [ ] Humble output init (small W2, zero b2)\n",
+        "- [ ] Saturation / dead neurons diagnosed\n",
+        "- [ ] Kaiming init (gain / sqrt(fan_in))\n",
+        "- [ ] BatchNorm: per-batch normalise + gamma/beta + running stats\n",
+        "- [ ] Trained, evaluated, sampled names"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

04_backprop_ninja.ipynb

@@ -0,0 +1,529 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "6caa1908",
+      "metadata": {},
+      "source": [
+        "# Becoming a backprop ninja\n",
+        "\n",
+        "Compute every gradient **by hand** — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **04**. So far we have leaned on `loss.backward()` to do the calculus for us.\n",
+        "Here we turn it off and backpropagate **manually** through the whole MLP + BatchNorm +\n",
+        "cross-entropy, checking each result against PyTorch. After this, autograd is never a mystery.\n",
+        "Follows Karpathy's *\"makemore part 4: becoming a backprop ninja\"*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "9ed68a60",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "`loss.backward()` answers one question for every weight: *if I nudge you a little, how does\n",
+        "the loss change?* That answer is the gradient. We have trusted PyTorch to compute it.\n",
+        "\n",
+        "Now we earn it. We will:\n",
+        "\n",
+        "1. Run the forward pass but **save every intermediate value**.\n",
+        "2. Walk backward, one operation at a time, applying the chain rule by hand (exactly the\n",
+        "   `00_micrograd.ipynb` idea, now on whole tensors).\n",
+        "3. Compare each hand-derived gradient to PyTorch's — they should match.\n",
+        "\n",
+        "Real-life picture: notebook `00` showed one gear; here we backprop through the whole gearbox,\n",
+        "and check our work against the manufacturer's spec (PyTorch)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "95e96905",
+      "metadata": {},
+      "source": [
+        "### 0.1 Imports and data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "3b29d3f0",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
+        "\n",
+        "import random\n",
+        "import torch\n",
+        "import torch.nn.functional as F\n",
+        "\n",
+        "words = open(\"names.txt\").read().splitlines()\n",
+        "chars = sorted(list(set(\"\".join(words))))\n",
+        "stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
+        "stoi[\".\"] = 0\n",
+        "itos = {i: ch for ch, i in stoi.items()}\n",
+        "vocab_size = len(itos)\n",
+        "block_size = 3\n",
+        "\n",
+        "def build_dataset(word_list):\n",
+        "    X, Y = [], []\n",
+        "    for w in word_list:\n",
+        "        context = [0] * block_size\n",
+        "        for ch in w + \".\":\n",
+        "            ix = stoi[ch]\n",
+        "            X.append(context); Y.append(ix)\n",
+        "            context = context[1:] + [ix]\n",
+        "    return torch.tensor(X), torch.tensor(Y)\n",
+        "\n",
+        "random.seed(42)\n",
+        "random.shuffle(words)\n",
+        "n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
+        "Xtr, Ytr = build_dataset(words[:n1])\n",
+        "Xdev, Ydev = build_dataset(words[n1:n2])\n",
+        "print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "19612a96",
+      "metadata": {},
+      "source": [
+        "### 0.2 The gradient-check helper\n",
+        "\n",
+        "`cmp` compares our hand-computed gradient `dt` to PyTorch's `t.grad`:\n",
+        "\n",
+        "- **exact** = bit-for-bit identical\n",
+        "- **approx** = equal within tiny floating-point tolerance (this is the one that matters)\n",
+        "- **maxdiff** = largest single difference"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "2c6c9ccf",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "def cmp(name, dt, t):\n",
+        "    ex = torch.all(dt == t.grad).item()\n",
+        "    app = torch.allclose(dt, t.grad, rtol=1e-4, atol=1e-7)\n",
+        "    maxdiff = (dt - t.grad).abs().max().item()\n",
+        "    print(f\"{name:12s} | exact: {str(ex):5s} | approx: {str(app):5s} | maxdiff: {maxdiff:.2e}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ed7b6dec",
+      "metadata": {},
+      "source": [
+        "### 1.1 Parameters\n",
+        "\n",
+        "We initialise the same MLP + BatchNorm as notebook `03`. Biases are made small but non-zero\n",
+        "so every gradient path is actually exercised (a zero would hide bugs)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "70e88d72",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "n_embd, n_hidden = 10, 64\n",
+        "g = torch.Generator().manual_seed(2147483647)\n",
+        "\n",
+        "C  = torch.randn((vocab_size, n_embd),            generator=g)\n",
+        "W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
+        "b1 = torch.randn(n_hidden,                         generator=g) * 0.1\n",
+        "W2 = torch.randn((n_hidden, vocab_size),          generator=g) * 0.1\n",
+        "b2 = torch.randn(vocab_size,                       generator=g) * 0.1\n",
+        "bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
+        "bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
+        "\n",
+        "parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
+        "for p in parameters:\n",
+        "    p.requires_grad = True\n",
+        "print(\"total parameters:\", sum(p.nelement() for p in parameters))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "04fa6a1e",
+      "metadata": {},
+      "source": [
+        "### 1.2 Forward pass, broken into tiny steps\n",
+        "\n",
+        "Cross-entropy is written out by hand (max-subtract, exp, sum, log) so we can backprop through\n",
+        "each piece. Every intermediate calls `retain_grad()` so PyTorch will store its gradient for us\n",
+        "to compare against."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "6ad34b3b",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "n = 32   # fixed batch size for the manual math\n",
+        "ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
+        "Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "emb = C[Xb]                                  # embed the characters\n",
+        "embcat = emb.view(emb.shape[0], -1)          # flatten\n",
+        "# linear 1\n",
+        "hprebn = embcat @ W1 + b1\n",
+        "# batchnorm\n",
+        "bnmeani = 1 / n * hprebn.sum(0, keepdim=True)\n",
+        "bndiff = hprebn - bnmeani\n",
+        "bndiff2 = bndiff ** 2\n",
+        "bnvar = 1 / (n - 1) * bndiff2.sum(0, keepdim=True)   # Bessel's correction (n-1)\n",
+        "bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
+        "bnraw = bndiff * bnvar_inv\n",
+        "hpreact = bngain * bnraw + bnbias\n",
+        "# nonlinearity\n",
+        "h = torch.tanh(hpreact)\n",
+        "# linear 2\n",
+        "logits = h @ W2 + b2\n",
+        "# cross-entropy (manual)\n",
+        "logit_maxes = logits.max(1, keepdim=True).values\n",
+        "norm_logits = logits - logit_maxes\n",
+        "counts = norm_logits.exp()\n",
+        "counts_sum = counts.sum(1, keepdim=True)\n",
+        "counts_sum_inv = counts_sum ** -1\n",
+        "probs = counts * counts_sum_inv\n",
+        "logprobs = probs.log()\n",
+        "loss = -logprobs[range(n), Yb].mean()\n",
+        "\n",
+        "for t in [logprobs, probs, counts, counts_sum, counts_sum_inv, norm_logits, logit_maxes,\n",
+        "          logits, h, hpreact, bnraw, bnvar_inv, bnvar, bndiff2, bndiff, bnmeani, hprebn,\n",
+        "          embcat, emb]:\n",
+        "    t.retain_grad()\n",
+        "for p in parameters:\n",
+        "    p.retain_grad()\n",
+        "loss.backward()\n",
+        "print(\"loss:\", loss.item())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6fcbe6af",
+      "metadata": {},
+      "source": [
+        "### 2.1 Backprop through cross-entropy\n",
+        "\n",
+        "Walk backward from `loss`. For each line of the forward pass we write the local derivative and\n",
+        "multiply by the gradient coming from above (the chain rule). Read each line as: *how does this\n",
+        "intermediate affect the loss?*"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c7bc68e2",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dlogprobs = torch.zeros_like(logprobs)\n",
+        "dlogprobs[range(n), Yb] = -1.0 / n              # loss = -mean of the chosen logprobs\n",
+        "dprobs = (1.0 / probs) * dlogprobs              # logprobs = log(probs)\n",
+        "dcounts_sum_inv = (counts * dprobs).sum(1, keepdim=True)\n",
+        "dcounts = counts_sum_inv * dprobs               # probs = counts * counts_sum_inv\n",
+        "dcounts_sum = (-counts_sum ** -2) * dcounts_sum_inv\n",
+        "dcounts += torch.ones_like(counts) * dcounts_sum\n",
+        "dnorm_logits = counts * dcounts                 # counts = exp(norm_logits)\n",
+        "dlogits = dnorm_logits.clone()\n",
+        "dlogit_maxes = (-dnorm_logits).sum(1, keepdim=True)\n",
+        "dlogits += F.one_hot(logits.max(1).indices, num_classes=logits.shape[1]) * dlogit_maxes\n",
+        "\n",
+        "cmp(\"logprobs\", dlogprobs, logprobs)\n",
+        "cmp(\"probs\", dprobs, probs)\n",
+        "cmp(\"counts_sum_inv\", dcounts_sum_inv, counts_sum_inv)\n",
+        "cmp(\"counts_sum\", dcounts_sum, counts_sum)\n",
+        "cmp(\"counts\", dcounts, counts)\n",
+        "cmp(\"norm_logits\", dnorm_logits, norm_logits)\n",
+        "cmp(\"logit_maxes\", dlogit_maxes, logit_maxes)\n",
+        "cmp(\"logits\", dlogits, logits)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "51a6e8d7",
+      "metadata": {},
+      "source": [
+        "### 2.2 Backprop through the second linear layer and tanh\n",
+        "\n",
+        "`logits = h @ W2 + b2`. Matrix-multiply gradients follow the standard transpose rules; `tanh`\n",
+        "contributes its local derivative `1 - tanh(x)^2`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "39f0e645",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dh = dlogits @ W2.T\n",
+        "dW2 = h.T @ dlogits\n",
+        "db2 = dlogits.sum(0)\n",
+        "dhpreact = (1.0 - h ** 2) * dh                  # tanh local derivative\n",
+        "\n",
+        "cmp(\"h\", dh, h)\n",
+        "cmp(\"W2\", dW2, W2)\n",
+        "cmp(\"b2\", db2, b2)\n",
+        "cmp(\"hpreact\", dhpreact, hpreact)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a7ecc492",
+      "metadata": {},
+      "source": [
+        "### 2.3 Backprop through BatchNorm\n",
+        "\n",
+        "This is the famously fiddly part. We go through each step: the scale/shift, the normalisation,\n",
+        "the variance, and the mean. Note how the **mean and variance both depend on every example**, so\n",
+        "gradients fan out across the batch."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "6f4f5d49",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
+        "dbnbias = dhpreact.sum(0, keepdim=True)\n",
+        "dbnraw = bngain * dhpreact\n",
+        "dbndiff = bnvar_inv * dbnraw\n",
+        "dbnvar_inv = (bndiff * dbnraw).sum(0, keepdim=True)\n",
+        "dbnvar = (-0.5 * (bnvar + 1e-5) ** -1.5) * dbnvar_inv\n",
+        "dbndiff2 = (1.0 / (n - 1)) * torch.ones_like(bndiff2) * dbnvar\n",
+        "dbndiff += (2 * bndiff) * dbndiff2\n",
+        "dhprebn = dbndiff.clone()\n",
+        "dbnmeani = (-dbndiff).sum(0)\n",
+        "dhprebn += 1.0 / n * (torch.ones_like(hprebn) * dbnmeani)\n",
+        "\n",
+        "cmp(\"bngain\", dbngain, bngain)\n",
+        "cmp(\"bnbias\", dbnbias, bnbias)\n",
+        "cmp(\"bnraw\", dbnraw, bnraw)\n",
+        "cmp(\"bnvar_inv\", dbnvar_inv, bnvar_inv)\n",
+        "cmp(\"bnvar\", dbnvar, bnvar)\n",
+        "cmp(\"bndiff2\", dbndiff2, bndiff2)\n",
+        "cmp(\"bndiff\", dbndiff, bndiff)\n",
+        "cmp(\"bnmeani\", dbnmeani, bnmeani)\n",
+        "cmp(\"hprebn\", dhprebn, hprebn)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "3be0bf67",
+      "metadata": {},
+      "source": [
+        "### 2.4 Backprop through the first linear layer and the embedding\n",
+        "\n",
+        "`hprebn = embcat @ W1 + b1`, then un-flatten back to `emb`, then scatter each row's gradient to\n",
+        "the right row of the embedding table `C` (an embedding lookup is just \"pick rows\", so its\n",
+        "backward is \"add the gradient back into those rows\")."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "b1317637",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dembcat = dhprebn @ W1.T\n",
+        "dW1 = embcat.T @ dhprebn\n",
+        "db1 = dhprebn.sum(0)\n",
+        "demb = dembcat.view(emb.shape)\n",
+        "\n",
+        "dC = torch.zeros_like(C)\n",
+        "for k in range(Xb.shape[0]):\n",
+        "    for j in range(Xb.shape[1]):\n",
+        "        dC[Xb[k, j]] += demb[k, j]\n",
+        "\n",
+        "cmp(\"embcat\", dembcat, embcat)\n",
+        "cmp(\"W1\", dW1, W1)\n",
+        "cmp(\"b1\", db1, b1)\n",
+        "cmp(\"emb\", demb, emb)\n",
+        "cmp(\"C\", dC, C)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "3f6d3bff",
+      "metadata": {},
+      "source": [
+        "### 3.1 The elegant shortcuts\n",
+        "\n",
+        "Doing it step-by-step proves we understand it. But two of those backward passes collapse into\n",
+        "beautiful one-liners once you do the algebra:\n",
+        "\n",
+        "- **Cross-entropy:** `dlogits = softmax(logits)`, then subtract 1 from the true class, divide by n.\n",
+        "  Intuition: push down whatever probability the model gave, pull up the correct class.\n",
+        "- **BatchNorm:** one expression for `dhprebn` (no need for all the intermediate d's).\n",
+        "\n",
+        "We verify both against PyTorch."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "91248dc5",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# elegant cross-entropy backward\n",
+        "dlogits_fast = F.softmax(logits, 1)\n",
+        "dlogits_fast[range(n), Yb] -= 1\n",
+        "dlogits_fast /= n\n",
+        "print(\"dlogits  | approx:\", torch.allclose(dlogits_fast, logits.grad, rtol=1e-4, atol=1e-7),\n",
+        "      \" | maxdiff:\", (dlogits_fast - logits.grad).abs().max().item())\n",
+        "\n",
+        "# elegant batchnorm backward\n",
+        "dhprebn_fast = bngain * bnvar_inv / n * (\n",
+        "    n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
+        ")\n",
+        "print(\"dhprebn  | approx:\", torch.allclose(dhprebn_fast, hprebn.grad, rtol=1e-4, atol=1e-7),\n",
+        "      \" | maxdiff:\", (dhprebn_fast - hprebn.grad).abs().max().item())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "09834b6c",
+      "metadata": {},
+      "source": [
+        "### 3.2 Train with 100% hand-written gradients\n",
+        "\n",
+        "The real proof: turn off autograd and train using only our manual gradients (the elegant\n",
+        "versions, plus the manual `W2/b2/W1/b1/C/bngain/bnbias`). If the loss falls, our calculus is\n",
+        "correct."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "9bfb8004",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# fresh params\n",
+        "g = torch.Generator().manual_seed(2147483647)\n",
+        "C  = torch.randn((vocab_size, n_embd),            generator=g)\n",
+        "W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
+        "b1 = torch.randn(n_hidden,                         generator=g) * 0.1\n",
+        "W2 = torch.randn((n_hidden, vocab_size),          generator=g) * 0.1\n",
+        "b2 = torch.randn(vocab_size,                       generator=g) * 0.1\n",
+        "bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
+        "bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
+        "parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
+        "for p in parameters:\n",
+        "    p.requires_grad = True\n",
+        "\n",
+        "n = 32\n",
+        "for i in range(8000):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "    # forward (keep what the manual backward needs)\n",
+        "    emb = C[Xb]\n",
+        "    embcat = emb.view(emb.shape[0], -1)\n",
+        "    hprebn = embcat @ W1 + b1\n",
+        "    bnmeani = hprebn.mean(0, keepdim=True)\n",
+        "    bnvar = hprebn.var(0, keepdim=True, unbiased=True)\n",
+        "    bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
+        "    bnraw = (hprebn - bnmeani) * bnvar_inv\n",
+        "    hpreact = bngain * bnraw + bnbias\n",
+        "    h = torch.tanh(hpreact)\n",
+        "    logits = h @ W2 + b2\n",
+        "    loss = F.cross_entropy(logits, Yb)\n",
+        "\n",
+        "    # manual backward (no loss.backward())\n",
+        "    dlogits = F.softmax(logits, 1)\n",
+        "    dlogits[range(n), Yb] -= 1\n",
+        "    dlogits /= n\n",
+        "    dh = dlogits @ W2.T\n",
+        "    dW2 = h.T @ dlogits\n",
+        "    db2 = dlogits.sum(0)\n",
+        "    dhpreact = (1.0 - h ** 2) * dh\n",
+        "    dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
+        "    dbnbias = dhpreact.sum(0, keepdim=True)\n",
+        "    dhprebn = bngain * bnvar_inv / n * (\n",
+        "        n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
+        "    )\n",
+        "    dembcat = dhprebn @ W1.T\n",
+        "    dW1 = embcat.T @ dhprebn\n",
+        "    db1 = dhprebn.sum(0)\n",
+        "    demb = dembcat.view(emb.shape)\n",
+        "    dC = torch.zeros_like(C)\n",
+        "    dC.index_add_(0, Xb.view(-1), demb.view(-1, n_embd))\n",
+        "\n",
+        "    grads = [dC, dW1, db1, dW2, db2, dbngain, dbnbias]\n",
+        "    lr = 0.1 if i < 6000 else 0.01\n",
+        "    with torch.no_grad():\n",
+        "        for p, gr in zip(parameters, grads):\n",
+        "            p += -lr * gr\n",
+        "\n",
+        "    if i % 2000 == 0 or i == 7999:\n",
+        "        print(f\"step {i:5d}   loss {loss.item():.4f}\")\n",
+        "\n",
+        "print(\"trained using only hand-written gradients.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "422e7804",
+      "metadata": {},
+      "source": [
+        "**Your turn 4** — In section 2.3, deliberately introduce a bug (e.g. drop the\n",
+        "`n/(n-1)` Bessel factor in `dbnvar`) and re-run. Watch `cmp` flip to `approx: False` and the\n",
+        "`maxdiff` jump. Then fix it. This is exactly how you debug a real backward pass."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "36c4bc1d",
+      "metadata": {},
+      "source": [
+        "## What's next\n",
+        "\n",
+        "You backpropagated by hand through cross-entropy, a linear layer, `tanh`, BatchNorm, and an\n",
+        "embedding, matched PyTorch exactly, and trained a net with zero autograd. Backprop is now\n",
+        "yours.\n",
+        "\n",
+        "Where this leads in the series:\n",
+        "\n",
+        "- `05_wavenet.ipynb` — stack layers into a deeper, hierarchical model using `torch.nn`-style\n",
+        "  building blocks.\n",
+        "- `06_build_gpt_attention.ipynb` — the transformer; `loss.backward()` there is doing exactly\n",
+        "  what you just did, across far more nodes.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] Forward pass saved every intermediate\n",
+        "- [ ] Manual backward matched PyTorch (cross-entropy, linear, tanh)\n",
+        "- [ ] Manual backward through BatchNorm matched\n",
+        "- [ ] Embedding backward = scatter-add into rows\n",
+        "- [ ] Elegant one-line cross-entropy and BatchNorm gradients\n",
+        "- [ ] Trained a net with only hand-written gradients"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

05_wavenet.ipynb

@@ -0,0 +1,424 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "d24a5a8b",
+      "metadata": {},
+      "source": [
+        "# makemore — a WaveNet\n",
+        "\n",
+        "Go **deeper and hierarchical** — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **05**. The MLP in `02`/`03` squashed the whole context into **one** layer.\n",
+        "Here we widen the context to 8 letters and fuse them **gradually**, two at a time, in a\n",
+        "tree — the structure behind DeepMind's WaveNet (2016). We also wrap everything in tidy\n",
+        "`torch.nn`-style building blocks. Follows Karpathy's *\"makemore part 5: WaveNet\"*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "187db6ed",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "Reading 8 letters by mashing them into one layer throws away structure. WaveNet's idea: fuse\n",
+        "information **slowly**.\n",
+        "\n",
+        "Real-life picture: a knockout tournament. 8 players (letters) play in pairs -> 4 winners ->\n",
+        "play in pairs -> 2 -> 1 champion. Each round combines neighbours into a richer summary. Our\n",
+        "network does the same: combine letters in pairs, then pairs-of-pairs, then again, until a\n",
+        "single vector represents the whole context.\n",
+        "\n",
+        "We also build small reusable layers (`Linear`, `BatchNorm1d`, `Tanh`, `Embedding`,\n",
+        "`FlattenConsecutive`, `Sequential`) that behave like PyTorch's own — so the model reads like a\n",
+        "clean list of steps."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6bf209d0",
+      "metadata": {},
+      "source": [
+        "### 0.1 Imports and data (context grows to 8)\n",
+        "\n",
+        "The only data change from earlier notebooks: `block_size = 8`, so the model sees up to 8\n",
+        "previous letters."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "88825091",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
+        "\n",
+        "import random\n",
+        "import torch\n",
+        "import torch.nn.functional as F\n",
+        "\n",
+        "try:\n",
+        "    import matplotlib.pyplot as plt\n",
+        "    HAS_PLT = True\n",
+        "except Exception:\n",
+        "    HAS_PLT = False\n",
+        "\n",
+        "words = open(\"names.txt\").read().splitlines()\n",
+        "chars = sorted(list(set(\"\".join(words))))\n",
+        "stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
+        "stoi[\".\"] = 0\n",
+        "itos = {i: ch for ch, i in stoi.items()}\n",
+        "vocab_size = len(itos)\n",
+        "block_size = 8\n",
+        "\n",
+        "def build_dataset(word_list):\n",
+        "    X, Y = [], []\n",
+        "    for w in word_list:\n",
+        "        context = [0] * block_size\n",
+        "        for ch in w + \".\":\n",
+        "            ix = stoi[ch]\n",
+        "            X.append(context); Y.append(ix)\n",
+        "            context = context[1:] + [ix]\n",
+        "    return torch.tensor(X), torch.tensor(Y)\n",
+        "\n",
+        "random.seed(42)\n",
+        "random.shuffle(words)\n",
+        "n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
+        "Xtr, Ytr = build_dataset(words[:n1])\n",
+        "Xdev, Ydev = build_dataset(words[n1:n2])\n",
+        "Xte, Yte = build_dataset(words[n2:])\n",
+        "print(\"train:\", tuple(Xtr.shape), \" (context length =\", block_size, \")\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "4c8200dd",
+      "metadata": {},
+      "source": [
+        "### 1.1 PyTorch-style building blocks\n",
+        "\n",
+        "Each layer is a tiny class with a `__call__` (forward) and `parameters()`. The interesting one\n",
+        "is **`FlattenConsecutive(n)`**: it groups every `n` neighbouring positions together — that is\n",
+        "the \"combine in pairs\" tournament step. `BatchNorm1d` is written to also handle the 3-D tensors\n",
+        "that appear between fusion steps (a subtle but important detail)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "7695c863",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "class Linear:\n",
+        "    def __init__(self, fan_in, fan_out, bias=True):\n",
+        "        self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in ** 0.5\n",
+        "        self.bias = torch.zeros(fan_out) if bias else None\n",
+        "    def __call__(self, x):\n",
+        "        self.out = x @ self.weight\n",
+        "        if self.bias is not None:\n",
+        "            self.out = self.out + self.bias\n",
+        "        return self.out\n",
+        "    def parameters(self):\n",
+        "        return [self.weight] + ([] if self.bias is None else [self.bias])\n",
+        "\n",
+        "\n",
+        "class BatchNorm1d:\n",
+        "    def __init__(self, dim, eps=1e-5, momentum=0.1):\n",
+        "        self.eps = eps; self.momentum = momentum; self.training = True\n",
+        "        self.gamma = torch.ones(dim); self.beta = torch.zeros(dim)\n",
+        "        self.running_mean = torch.zeros(dim); self.running_var = torch.ones(dim)\n",
+        "    def __call__(self, x):\n",
+        "        if self.training:\n",
+        "            dim = 0 if x.ndim == 2 else (0, 1)     # 3-D inputs: average over batch AND time\n",
+        "            xmean = x.mean(dim, keepdim=True)\n",
+        "            xvar = x.var(dim, keepdim=True)\n",
+        "        else:\n",
+        "            xmean = self.running_mean\n",
+        "            xvar = self.running_var\n",
+        "        xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
+        "        self.out = self.gamma * xhat + self.beta\n",
+        "        if self.training:\n",
+        "            with torch.no_grad():\n",
+        "                self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean\n",
+        "                self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar\n",
+        "        return self.out\n",
+        "    def parameters(self):\n",
+        "        return [self.gamma, self.beta]\n",
+        "\n",
+        "\n",
+        "class Tanh:\n",
+        "    def __call__(self, x):\n",
+        "        self.out = torch.tanh(x); return self.out\n",
+        "    def parameters(self):\n",
+        "        return []\n",
+        "\n",
+        "\n",
+        "class Embedding:\n",
+        "    def __init__(self, num_embeddings, embedding_dim):\n",
+        "        self.weight = torch.randn((num_embeddings, embedding_dim), generator=g)\n",
+        "    def __call__(self, IX):\n",
+        "        self.out = self.weight[IX]; return self.out\n",
+        "    def parameters(self):\n",
+        "        return [self.weight]\n",
+        "\n",
+        "\n",
+        "class FlattenConsecutive:\n",
+        "    def __init__(self, n):\n",
+        "        self.n = n\n",
+        "    def __call__(self, x):\n",
+        "        B, T, C = x.shape\n",
+        "        x = x.view(B, T // self.n, C * self.n)   # fuse every n neighbours\n",
+        "        if x.shape[1] == 1:\n",
+        "            x = x.squeeze(1)\n",
+        "        self.out = x; return self.out\n",
+        "    def parameters(self):\n",
+        "        return []\n",
+        "\n",
+        "\n",
+        "class Sequential:\n",
+        "    def __init__(self, layers):\n",
+        "        self.layers = layers\n",
+        "    def __call__(self, x):\n",
+        "        for layer in self.layers:\n",
+        "            x = layer(x)\n",
+        "        self.out = x; return self.out\n",
+        "    def parameters(self):\n",
+        "        return [p for layer in self.layers for p in layer.parameters()]\n",
+        "\n",
+        "print(\"layers defined\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ab8cd26d",
+      "metadata": {},
+      "source": [
+        "### 1.2 Build the hierarchical (WaveNet) model\n",
+        "\n",
+        "`block_size = 8` fuses in three rounds: 8 -> 4 -> 2 -> 1. Each round is\n",
+        "`FlattenConsecutive(2) -> Linear -> BatchNorm1d -> Tanh`. Finally a `Linear` maps to the 27\n",
+        "next-letter scores. We shrink the last layer's weights so the first loss starts near the ideal\n",
+        "`ln(27)`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "1dcd57ee",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g = torch.Generator().manual_seed(42)\n",
+        "n_embd = 24\n",
+        "n_hidden = 128\n",
+        "\n",
+        "model = Sequential([\n",
+        "    Embedding(vocab_size, n_embd),\n",
+        "    FlattenConsecutive(2), Linear(n_embd * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
+        "    FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
+        "    FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
+        "    Linear(n_hidden, vocab_size),\n",
+        "])\n",
+        "\n",
+        "with torch.no_grad():\n",
+        "    model.layers[-1].weight *= 0.1   # humble output\n",
+        "\n",
+        "parameters = model.parameters()\n",
+        "for p in parameters:\n",
+        "    p.requires_grad = True\n",
+        "print(\"total parameters:\", sum(p.nelement() for p in parameters))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dfe5653b",
+      "metadata": {},
+      "source": [
+        "### 1.3 Watch the tournament shrink the sequence\n",
+        "\n",
+        "Run one batch and print each layer's output shape. See the time dimension halve 8 -> 4 -> 2 -> 1\n",
+        "as neighbours fuse, while the feature dimension does the work."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5a37517c",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ix = torch.randint(0, Xtr.shape[0], (4,), generator=g)\n",
+        "_ = model(Xtr[ix])\n",
+        "for layer in model.layers:\n",
+        "    print(f\"{layer.__class__.__name__:18s} : {tuple(layer.out.shape)}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "43949ea9",
+      "metadata": {},
+      "source": [
+        "### 2.1 Train\n",
+        "\n",
+        "Standard loop: minibatch -> forward -> `loss.backward()` -> nudge. (We use autograd again now;\n",
+        "notebook `04` already proved we could do it by hand.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f0800ce0",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "max_steps = 20000\n",
+        "batch_size = 32\n",
+        "lossi = []\n",
+        "\n",
+        "for i in range(max_steps):\n",
+        "    ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
+        "    Xb, Yb = Xtr[ix], Ytr[ix]\n",
+        "\n",
+        "    logits = model(Xb)\n",
+        "    loss = F.cross_entropy(logits, Yb)\n",
+        "\n",
+        "    for p in parameters:\n",
+        "        p.grad = None\n",
+        "    loss.backward()\n",
+        "    lr = 0.1 if i < 15000 else 0.01\n",
+        "    for p in parameters:\n",
+        "        p.data += -lr * p.grad\n",
+        "\n",
+        "    lossi.append(loss.log10().item())\n",
+        "    if i % 4000 == 0 or i == max_steps - 1:\n",
+        "        print(f\"step {i:5d}/{max_steps}   loss {loss.item():.4f}\")\n",
+        "\n",
+        "if HAS_PLT:\n",
+        "    # smooth the noisy per-step loss by averaging in chunks of 200\n",
+        "    lt = torch.tensor(lossi)\n",
+        "    plt.figure(figsize=(5, 4))\n",
+        "    plt.plot(lt.view(-1, 200).mean(1))\n",
+        "    plt.xlabel(\"step / 200\"); plt.ylabel(\"log10(loss)\"); plt.title(\"WaveNet training loss\")\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "053ba8a7",
+      "metadata": {},
+      "source": [
+        "### 2.2 Evaluate\n",
+        "\n",
+        "Switch BatchNorm layers to eval mode (use running stats), then score train and dev. The deeper\n",
+        "hierarchical model should edge below the single-layer MLP's ~2.1."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "3a5d6228",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for layer in model.layers:\n",
+        "    if isinstance(layer, BatchNorm1d):\n",
+        "        layer.training = False\n",
+        "\n",
+        "@torch.no_grad()\n",
+        "def split_loss(split):\n",
+        "    x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
+        "    logits = model(x)\n",
+        "    return F.cross_entropy(logits, y).item()\n",
+        "\n",
+        "print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
+        "print(\"dev   loss:\", round(split_loss(\"dev\"), 4))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b78eaf70",
+      "metadata": {},
+      "source": [
+        "### 3.1 Generate names\n",
+        "\n",
+        "Sample letter by letter as before. With 8 letters of context and the hierarchical fusion, the\n",
+        "names tend to look a little more name-like."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "33833066",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
+        "\n",
+        "for _ in range(20):\n",
+        "    out = []\n",
+        "    context = [0] * block_size\n",
+        "    while True:\n",
+        "        logits = model(torch.tensor([context]))\n",
+        "        probs = F.softmax(logits, dim=1)\n",
+        "        ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
+        "        context = context[1:] + [ix]\n",
+        "        if ix == 0:\n",
+        "            break\n",
+        "        out.append(itos[ix])\n",
+        "    print(\"\".join(out))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "31d52503",
+      "metadata": {},
+      "source": [
+        "**Your turn 5** — The model is just a list. Add another fusion round (so it handles\n",
+        "`block_size = 16`), or raise `n_hidden`, retrain, and compare dev loss. Because everything is a\n",
+        "clean `Sequential`, experimenting is just editing the list."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "894cc92e",
+      "metadata": {},
+      "source": [
+        "## What's next\n",
+        "\n",
+        "You built a deeper, hierarchical model out of reusable PyTorch-style blocks — the same shape of\n",
+        "code real libraries use.\n",
+        "\n",
+        "Where this leads in the series:\n",
+        "\n",
+        "- `06_build_gpt_attention.ipynb` — the transformer. Instead of a fixed tournament that always\n",
+        "  fuses neighbours, **attention** lets every position decide *which* earlier positions to listen\n",
+        "  to. That flexibility is what makes GPT powerful.\n",
+        "- `07_gpt_tokenizer.ipynb` — how real models turn text into tokens in the first place.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] Context widened to 8 letters\n",
+        "- [ ] PyTorch-style layers (Linear, BatchNorm1d, Tanh, Embedding, FlattenConsecutive, Sequential)\n",
+        "- [ ] Hierarchical fusion 8 -> 4 -> 2 -> 1\n",
+        "- [ ] BatchNorm1d handles 3-D tensors\n",
+        "- [ ] Trained, evaluated, sampled names"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

06_build_gpt_attention.ipynb

@@ -0,0 +1,480 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "3d5936ca",
+      "metadata": {},
+      "source": [
+        "# Let's build GPT — self-attention to a full transformer\n",
+        "\n",
+        "Give every token the power to **look back** — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **06**, the payoff. Notebook `01` built a bigram that only sees one character\n",
+        "back. Here we add **self-attention** so each position can gather information from all earlier\n",
+        "positions, then stack it into a real (small) **GPT** and train it on tiny Shakespeare.\n",
+        "Follows Karpathy's *\"Let's build GPT from scratch\"*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "8b6c4fb3",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "The bigram's weakness: predicting the next letter from only the previous one. WaveNet (`05`)\n",
+        "widened the view but fused neighbours in a fixed pattern. **Attention** is smarter: every\n",
+        "token looks at all earlier tokens and decides, on its own, *which* ones matter right now.\n",
+        "\n",
+        "Real-life picture: you are reading a sentence and hit the word \"it.\" To know what \"it\" refers\n",
+        "to, your eyes jump back to the relevant earlier noun — not to a fixed position, but to whichever\n",
+        "word is relevant. Self-attention lets each token make that kind of targeted look-back.\n",
+        "\n",
+        "We build it up in small pieces: the averaging trick, then queries/keys/values, then multiple\n",
+        "heads, then the full transformer block (attention + feed-forward + residual + LayerNorm)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "028dce18",
+      "metadata": {},
+      "source": [
+        "### 0.1 Setup and data (standalone)\n",
+        "\n",
+        "Self-contained: load `input.txt`, build the character vocabulary, and make `get_batch`. Uses\n",
+        "GPU if available, else CPU. The model is intentionally **small** so it trains on a laptop CPU;\n",
+        "the comments show how to scale it up if you have a GPU."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "36f4f6e6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
+        "\n",
+        "import torch\n",
+        "import torch.nn as nn\n",
+        "from torch.nn import functional as F\n",
+        "\n",
+        "torch.manual_seed(1337)\n",
+        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+        "print(\"device:\", device, \"| torch\", torch.__version__)\n",
+        "\n",
+        "# --- hyperparameters (small for CPU; bump these up on a GPU) ---\n",
+        "batch_size = 16\n",
+        "block_size = 32       # max context length\n",
+        "n_embd = 64\n",
+        "n_head = 4\n",
+        "n_layer = 3\n",
+        "dropout = 0.0\n",
+        "learning_rate = 1e-3\n",
+        "max_iters = 3000\n",
+        "eval_interval = 1000\n",
+        "eval_iters = 50\n",
+        "\n",
+        "# --- data ---\n",
+        "text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()\n",
+        "chars = sorted(list(set(text)))\n",
+        "vocab_size = len(chars)\n",
+        "stoi = {ch: i for i, ch in enumerate(chars)}\n",
+        "itos = {i: ch for i, ch in enumerate(chars)}\n",
+        "encode = lambda s: [stoi[c] for c in s]\n",
+        "decode = lambda l: \"\".join(itos[i] for i in l)\n",
+        "\n",
+        "data = torch.tensor(encode(text), dtype=torch.long)\n",
+        "n = int(0.9 * len(data))\n",
+        "train_data, val_data = data[:n], data[n:]\n",
+        "\n",
+        "def get_batch(split):\n",
+        "    d = train_data if split == \"train\" else val_data\n",
+        "    ix = torch.randint(len(d) - block_size, (batch_size,))\n",
+        "    x = torch.stack([d[i:i + block_size] for i in ix])\n",
+        "    y = torch.stack([d[i + 1:i + block_size + 1] for i in ix])\n",
+        "    return x.to(device), y.to(device)\n",
+        "\n",
+        "print(\"vocab_size:\", vocab_size, \"| chars in dataset:\", len(text))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d2b12520",
+      "metadata": {},
+      "source": [
+        "### 1.1 The math trick: a token can average its past\n",
+        "\n",
+        "Before real attention, here is the key move in its simplest form: let each position become the\n",
+        "**average of itself and all positions before it** (never the future — that would be cheating).\n",
+        "We show three ways that give the *same* result, from slow to elegant."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "54c69615",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "torch.manual_seed(1337)\n",
+        "B, T, C = 4, 8, 2\n",
+        "x = torch.randn(B, T, C)\n",
+        "\n",
+        "# v1: explicit loop — average of all previous tokens\n",
+        "xbow = torch.zeros((B, T, C))\n",
+        "for b in range(B):\n",
+        "    for t in range(T):\n",
+        "        xbow[b, t] = x[b, :t + 1].mean(0)\n",
+        "\n",
+        "# v2: same thing as a matrix multiply with a lower-triangular average matrix\n",
+        "wei = torch.tril(torch.ones(T, T))\n",
+        "wei = wei / wei.sum(1, keepdim=True)\n",
+        "xbow2 = wei @ x\n",
+        "\n",
+        "# v3: same again, but the weights come from softmax over a masked matrix\n",
+        "tril = torch.tril(torch.ones(T, T))\n",
+        "wei = torch.zeros((T, T)).masked_fill(tril == 0, float(\"-inf\"))\n",
+        "wei = F.softmax(wei, dim=-1)\n",
+        "xbow3 = wei @ x\n",
+        "\n",
+        "print(\"v1 == v2 :\", torch.allclose(xbow, xbow2, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow2).abs().max())\n",
+        "print(\"v1 == v3 :\", torch.allclose(xbow, xbow3, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow3).abs().max())\n",
+        "print(\"\\nthe softmax weight matrix (row t can only attend to columns <= t):\")\n",
+        "print(wei)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "aa46672a",
+      "metadata": {},
+      "source": [
+        "Why the `-inf` + softmax version matters: those weights do **not** have to be a flat\n",
+        "average. If the numbers in `wei` are *learned*, each token can decide how much to listen to each\n",
+        "earlier token. That is exactly what attention does next.\n",
+        "\n",
+        "Analogy: a meeting where you may only hear people who already spoke (the mask), and you choose\n",
+        "how much weight to give each speaker (the learned weights)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b522ef98",
+      "metadata": {},
+      "source": [
+        "### 2.1 A single self-attention head — query, key, value\n",
+        "\n",
+        "Each token emits three vectors:\n",
+        "\n",
+        "- **query**: \"what am I looking for?\"\n",
+        "- **key**: \"what do I offer?\"\n",
+        "- **value**: \"what will I actually pass on if chosen?\"\n",
+        "\n",
+        "A token's attention to another = how well its **query** matches the other's **key**\n",
+        "(dot product). We scale by `1/sqrt(head_size)` to keep the numbers tame, mask the future, softmax\n",
+        "into weights, then take the weighted sum of **values**."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "9adfb9ea",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "torch.manual_seed(1337)\n",
+        "B, T, C = 4, 8, 32\n",
+        "x = torch.randn(B, T, C)\n",
+        "head_size = 16\n",
+        "\n",
+        "key   = nn.Linear(C, head_size, bias=False)\n",
+        "query = nn.Linear(C, head_size, bias=False)\n",
+        "value = nn.Linear(C, head_size, bias=False)\n",
+        "\n",
+        "k = key(x)        # (B, T, head_size)\n",
+        "q = query(x)      # (B, T, head_size)\n",
+        "wei = q @ k.transpose(-2, -1) * head_size ** -0.5   # (B, T, T) affinities\n",
+        "\n",
+        "tril = torch.tril(torch.ones(T, T))\n",
+        "wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
+        "wei = F.softmax(wei, dim=-1)\n",
+        "\n",
+        "v = value(x)\n",
+        "out = wei @ v     # (B, T, head_size)\n",
+        "print(\"attention output shape:\", tuple(out.shape))\n",
+        "print(\"\\nrow 0 of the learned attention weights (note they are NOT uniform):\")\n",
+        "print(wei[0, -1])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ff4eea4c",
+      "metadata": {},
+      "source": [
+        "### 3.1 The full GPT, assembled from clean modules\n",
+        "\n",
+        "Now we package it. The pieces:\n",
+        "\n",
+        "- **Head** — one self-attention head (above), with dropout and a causal mask buffer.\n",
+        "- **MultiHeadAttention** — several heads in parallel, concatenated, then projected. Analogy:\n",
+        "  several readers each look back for a different kind of clue, then pool notes.\n",
+        "- **FeedForward** — a small per-token MLP: \"time to think about what I gathered.\"\n",
+        "- **Block** — attention + feed-forward, each wrapped with a **residual** connection and\n",
+        "  **LayerNorm**. Residual = keep the original and add the new insight (so deep stacks still\n",
+        "  train). LayerNorm = re-level each token's vector (the `03` lesson, per-token).\n",
+        "- **GPTLanguageModel** — token + position embeddings -> a stack of Blocks -> final LayerNorm ->\n",
+        "  a linear head to vocab logits."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "b0a4b08d",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "class Head(nn.Module):\n",
+        "    def __init__(self, head_size):\n",
+        "        super().__init__()\n",
+        "        self.key = nn.Linear(n_embd, head_size, bias=False)\n",
+        "        self.query = nn.Linear(n_embd, head_size, bias=False)\n",
+        "        self.value = nn.Linear(n_embd, head_size, bias=False)\n",
+        "        self.register_buffer(\"tril\", torch.tril(torch.ones(block_size, block_size)))\n",
+        "        self.dropout = nn.Dropout(dropout)\n",
+        "\n",
+        "    def forward(self, x):\n",
+        "        B, T, C = x.shape\n",
+        "        k = self.key(x)\n",
+        "        q = self.query(x)\n",
+        "        wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5\n",
+        "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float(\"-inf\"))\n",
+        "        wei = F.softmax(wei, dim=-1)\n",
+        "        wei = self.dropout(wei)\n",
+        "        v = self.value(x)\n",
+        "        return wei @ v\n",
+        "\n",
+        "\n",
+        "class MultiHeadAttention(nn.Module):\n",
+        "    def __init__(self, num_heads, head_size):\n",
+        "        super().__init__()\n",
+        "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
+        "        self.proj = nn.Linear(head_size * num_heads, n_embd)\n",
+        "        self.dropout = nn.Dropout(dropout)\n",
+        "\n",
+        "    def forward(self, x):\n",
+        "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
+        "        return self.dropout(self.proj(out))\n",
+        "\n",
+        "\n",
+        "class FeedForward(nn.Module):\n",
+        "    def __init__(self, n_embd):\n",
+        "        super().__init__()\n",
+        "        self.net = nn.Sequential(\n",
+        "            nn.Linear(n_embd, 4 * n_embd),\n",
+        "            nn.ReLU(),\n",
+        "            nn.Linear(4 * n_embd, n_embd),\n",
+        "            nn.Dropout(dropout),\n",
+        "        )\n",
+        "\n",
+        "    def forward(self, x):\n",
+        "        return self.net(x)\n",
+        "\n",
+        "\n",
+        "class Block(nn.Module):\n",
+        "    def __init__(self, n_embd, n_head):\n",
+        "        super().__init__()\n",
+        "        head_size = n_embd // n_head\n",
+        "        self.sa = MultiHeadAttention(n_head, head_size)\n",
+        "        self.ffwd = FeedForward(n_embd)\n",
+        "        self.ln1 = nn.LayerNorm(n_embd)\n",
+        "        self.ln2 = nn.LayerNorm(n_embd)\n",
+        "\n",
+        "    def forward(self, x):\n",
+        "        x = x + self.sa(self.ln1(x))     # residual around attention\n",
+        "        x = x + self.ffwd(self.ln2(x))   # residual around feed-forward\n",
+        "        return x\n",
+        "\n",
+        "\n",
+        "class GPTLanguageModel(nn.Module):\n",
+        "    def __init__(self):\n",
+        "        super().__init__()\n",
+        "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
+        "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
+        "        self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)])\n",
+        "        self.ln_f = nn.LayerNorm(n_embd)\n",
+        "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
+        "\n",
+        "    def forward(self, idx, targets=None):\n",
+        "        B, T = idx.shape\n",
+        "        tok_emb = self.token_embedding_table(idx)                       # (B,T,C)\n",
+        "        pos_emb = self.position_embedding_table(torch.arange(T, device=device))\n",
+        "        x = tok_emb + pos_emb\n",
+        "        x = self.blocks(x)\n",
+        "        x = self.ln_f(x)\n",
+        "        logits = self.lm_head(x)                                        # (B,T,vocab)\n",
+        "        if targets is None:\n",
+        "            return logits, None\n",
+        "        B, T, Cc = logits.shape\n",
+        "        loss = F.cross_entropy(logits.view(B * T, Cc), targets.view(B * T))\n",
+        "        return logits, loss\n",
+        "\n",
+        "    def generate(self, idx, max_new_tokens):\n",
+        "        for _ in range(max_new_tokens):\n",
+        "            idx_cond = idx[:, -block_size:]          # never exceed the context window\n",
+        "            logits, _ = self(idx_cond)\n",
+        "            logits = logits[:, -1, :]                # last position only\n",
+        "            probs = F.softmax(logits, dim=-1)\n",
+        "            idx_next = torch.multinomial(probs, num_samples=1)\n",
+        "            idx = torch.cat((idx, idx_next), dim=1)\n",
+        "        return idx\n",
+        "\n",
+        "\n",
+        "model = GPTLanguageModel().to(device)\n",
+        "print(\"parameters (millions):\", sum(p.numel() for p in model.parameters()) / 1e6)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dbb781c5",
+      "metadata": {},
+      "source": [
+        "### 3.2 Positional embeddings — why we need them\n",
+        "\n",
+        "Attention by itself is **order-blind**: it treats the context as a bag of tokens. But \"dog\n",
+        "bites man\" and \"man bites dog\" must differ. So besides each token's meaning\n",
+        "(`token_embedding_table`), we add a vector for its **position**\n",
+        "(`position_embedding_table`). Now a token knows both *what* it is and *where* it is.\n",
+        "\n",
+        "(That addition already happens inside `forward`: `x = tok_emb + pos_emb`.)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0385a0ef",
+      "metadata": {},
+      "source": [
+        "### 4.1 Train the GPT\n",
+        "\n",
+        "Same training shape as always, with the AdamW optimizer. Watch train and val loss fall well\n",
+        "below the bigram's ~2.5. (Small model + CPU, so this is a few minutes; scale up on a GPU for\n",
+        "real Shakespeare.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "395e6922",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "@torch.no_grad()\n",
+        "def estimate_loss():\n",
+        "    out = {}\n",
+        "    model.eval()\n",
+        "    for split in [\"train\", \"val\"]:\n",
+        "        losses = torch.zeros(eval_iters)\n",
+        "        for k in range(eval_iters):\n",
+        "            X, Y = get_batch(split)\n",
+        "            _, loss = model(X, Y)\n",
+        "            losses[k] = loss.item()\n",
+        "        out[split] = losses.mean().item()\n",
+        "    model.train()\n",
+        "    return out\n",
+        "\n",
+        "optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
+        "\n",
+        "for it in range(max_iters):\n",
+        "    if it % eval_interval == 0 or it == max_iters - 1:\n",
+        "        losses = estimate_loss()\n",
+        "        print(f\"step {it:4d}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
+        "    xb, yb = get_batch(\"train\")\n",
+        "    _, loss = model(xb, yb)\n",
+        "    optimizer.zero_grad(set_to_none=True)\n",
+        "    loss.backward()\n",
+        "    optimizer.step()\n",
+        "\n",
+        "print(\"done training\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0ae5a608",
+      "metadata": {},
+      "source": [
+        "### 5.1 Generate Shakespeare-ish text\n",
+        "\n",
+        "Start from a single newline and let the model write. It will not be perfect Shakespeare (tiny\n",
+        "model, short training), but compared to the bigram it now produces real-looking words, spacing,\n",
+        "and line breaks — because each character can attend to the ones before it."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "61db808c",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
+        "print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6b43261f",
+      "metadata": {},
+      "source": [
+        "**Your turn 6** — If you have a GPU, scale up (e.g. `n_embd=384`, `n_head=6`,\n",
+        "`n_layer=6`, `block_size=256`, `dropout=0.2`, `max_iters=5000`) and retrain — the samples get\n",
+        "dramatically better. On CPU, try just raising `n_layer` to 4 or `max_iters` to 5000 and compare\n",
+        "the val loss."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "1e1142ab",
+      "metadata": {},
+      "source": [
+        "## What's next — and the connection to ChatGPT\n",
+        "\n",
+        "You built a real transformer: token + position embeddings, multi-head causal self-attention,\n",
+        "feed-forward layers, residuals, and LayerNorm — the exact architecture behind GPT.\n",
+        "\n",
+        "What you trained here is a **pretrained** model in miniature: it only learns to *continue text*\n",
+        "(predict the next token) from a big pile of text. ChatGPT starts the same way, then adds steps\n",
+        "this series does not cover:\n",
+        "\n",
+        "- **Supervised finetuning** on example conversations (teach it to answer, not just continue).\n",
+        "- **Reinforcement learning from human feedback (RLHF)** to align responses with what people\n",
+        "  prefer.\n",
+        "\n",
+        "The remaining notebook:\n",
+        "\n",
+        "- `07_gpt_tokenizer.ipynb` — real models do not use one-character-per-token; they use **Byte\n",
+        "  Pair Encoding**. That is the final piece of the pipeline.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] The averaging trick (loop = tril matmul = masked softmax)\n",
+        "- [ ] Query / key / value self-attention head\n",
+        "- [ ] Multi-head attention + projection\n",
+        "- [ ] Feed-forward, residuals, LayerNorm -> Block\n",
+        "- [ ] Token + position embeddings -> full GPT\n",
+        "- [ ] Trained below the bigram baseline; generated text"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

07_gpt_tokenizer.ipynb

@@ -0,0 +1,375 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "67a422d8",
+      "metadata": {},
+      "source": [
+        "# The GPT tokenizer — Byte Pair Encoding from scratch\n",
+        "\n",
+        "Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.\n",
+        "\n",
+        "This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model\n",
+        "either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**\n",
+        "tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then\n",
+        "see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's\n",
+        "*\"Let's build the GPT Tokenizer\"*."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2037ef52",
+      "metadata": {},
+      "source": [
+        "## Prologue — in plain English\n",
+        "\n",
+        "A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece\n",
+        "mapped to an id. Two extremes:\n",
+        "\n",
+        "- **One token per character** (what we did): tiny vocabulary, but sequences are very long and\n",
+        "  the model must relearn common spellings everywhere.\n",
+        "- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.\n",
+        "\n",
+        "**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common\n",
+        "adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single\n",
+        "tokens; rare text still survives as smaller pieces.\n",
+        "\n",
+        "Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep\n",
+        "writing \"t-h-e\", so you invent one squiggle for \"the.\" Then you notice \" a-n-d\" and give it a\n",
+        "squiggle too. BPE does this automatically, keeping the most useful shorthands."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "75d40ed6",
+      "metadata": {},
+      "source": [
+        "### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)\n",
+        "\n",
+        "Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts\n",
+        "with exactly 256 tokens — every possible byte. Plain English letters are one byte each;\n",
+        "accented or non-Latin characters take several bytes (which is why they cost more tokens later)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "1037d36e",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sample = \"Hello world! tokenization is fun.\"\n",
+        "the_bytes = sample.encode(\"utf-8\")\n",
+        "print(\"text   :\", sample)\n",
+        "print(\"bytes  :\", list(the_bytes))\n",
+        "print(\"length :\", len(the_bytes), \"bytes ->\", len(the_bytes), \"starting tokens\")\n",
+        "\n",
+        "multi = \"café  naïve\"   # accented letters take >1 byte each\n",
+        "print(\"\\nnon-ASCII example:\", multi)\n",
+        "print(\"bytes:\", list(multi.encode(\"utf-8\")), \"(note: more bytes than characters)\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5db66687",
+      "metadata": {},
+      "source": [
+        "### 2.1 Two tiny helpers: count pairs, and merge a pair\n",
+        "\n",
+        "- `get_stats` counts how often each adjacent pair appears.\n",
+        "- `merge` replaces every occurrence of a chosen pair with a single new token id.\n",
+        "\n",
+        "That is the entire mechanism of BPE."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "6bc93ad8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "def get_stats(ids):\n",
+        "    counts = {}\n",
+        "    for pair in zip(ids, ids[1:]):\n",
+        "        counts[pair] = counts.get(pair, 0) + 1\n",
+        "    return counts\n",
+        "\n",
+        "def merge(ids, pair, idx):\n",
+        "    new_ids = []\n",
+        "    i = 0\n",
+        "    while i < len(ids):\n",
+        "        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:\n",
+        "            new_ids.append(idx)\n",
+        "            i += 2\n",
+        "        else:\n",
+        "            new_ids.append(ids[i])\n",
+        "            i += 1\n",
+        "    return new_ids\n",
+        "\n",
+        "# demo on a toy list: merge the most common pair\n",
+        "demo = [1, 2, 3, 1, 2, 3, 1, 2]\n",
+        "st = get_stats(demo)\n",
+        "top = max(st, key=st.get)\n",
+        "print(\"counts:\", st)\n",
+        "print(\"most common pair:\", top, \"->\", st[top], \"times\")\n",
+        "print(\"after merging\", top, \"into 99:\", merge(demo, top, 99))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "cfb961e0",
+      "metadata": {},
+      "source": [
+        "### 2.2 Train BPE — learn the merges\n",
+        "\n",
+        "We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find\n",
+        "the most common pair and merge it, recording each merge. After `num_merges` rounds we have a\n",
+        "vocabulary of `256 + num_merges` tokens."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f62fbf5d",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "train_text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()[:20000]\n",
+        "ids = list(train_text.encode(\"utf-8\"))\n",
+        "print(\"training bytes:\", len(ids))\n",
+        "\n",
+        "num_merges = 60\n",
+        "merges = {}                      # (a, b) -> new_id\n",
+        "work = list(ids)\n",
+        "for i in range(num_merges):\n",
+        "    stats = get_stats(work)\n",
+        "    pair = max(stats, key=stats.get)\n",
+        "    idx = 256 + i\n",
+        "    work = merge(work, pair, idx)\n",
+        "    merges[pair] = idx\n",
+        "\n",
+        "print(\"merges learned:\", len(merges))\n",
+        "print(\"tokens after training:\", len(work), \"(was\", len(ids), \"bytes)\")\n",
+        "print(f\"compression: {len(ids) / len(work):.2f}x\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "34b2bc27",
+      "metadata": {},
+      "source": [
+        "### 3.1 encode and decode\n",
+        "\n",
+        "- **decode**: turn token ids back into text. We build each token's byte string by stitching the\n",
+        "  merges back together.\n",
+        "- **encode**: turn new text into token ids by applying the learned merges, always doing the\n",
+        "  *earliest-learned* applicable merge first."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c4f1dcab",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# build the id -> bytes table\n",
+        "vocab = {idx: bytes([idx]) for idx in range(256)}\n",
+        "for (p0, p1), idx in merges.items():\n",
+        "    vocab[idx] = vocab[p0] + vocab[p1]\n",
+        "\n",
+        "def decode(ids):\n",
+        "    data = b\"\".join(vocab[idx] for idx in ids)\n",
+        "    return data.decode(\"utf-8\", errors=\"replace\")\n",
+        "\n",
+        "def encode(text):\n",
+        "    tokens = list(text.encode(\"utf-8\"))\n",
+        "    while len(tokens) >= 2:\n",
+        "        stats = get_stats(tokens)\n",
+        "        # pick the pair whose merge was learned earliest\n",
+        "        pair = min(stats, key=lambda p: merges.get(p, float(\"inf\")))\n",
+        "        if pair not in merges:\n",
+        "            break\n",
+        "        tokens = merge(tokens, pair, merges[pair])\n",
+        "    return tokens\n",
+        "\n",
+        "trial = \"the king and the queen\"\n",
+        "enc = encode(trial)\n",
+        "print(\"text  :\", trial)\n",
+        "print(\"tokens:\", enc)\n",
+        "print(\"count :\", len(enc), \"tokens for\", len(trial), \"characters\")\n",
+        "print(\"decode round-trip ok:\", decode(enc) == trial)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a41ee40c",
+      "metadata": {},
+      "source": [
+        "### 3.2 See what the learned tokens are\n",
+        "\n",
+        "The merges discovered the common chunks of English on their own. Let's print a few learned\n",
+        "tokens (ids >= 256) as the text they stand for."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "dadcc751",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(\"a few learned tokens:\")\n",
+        "for idx in range(256, 256 + min(20, num_merges)):\n",
+        "    piece = vocab[idx].decode(\"utf-8\", errors=\"replace\")\n",
+        "    print(f\"  id {idx}: {piece!r}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "9193a4cd",
+      "metadata": {},
+      "source": [
+        "### 4.1 Regex pre-splitting (what real GPTs add)\n",
+        "\n",
+        "A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`\n",
+        "into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,\n",
+        "runs of spaces, punctuation), then runs BPE **inside** each chunk only.\n",
+        "\n",
+        "Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the\n",
+        "standard `re` module to show the idea."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "96cf29fc",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import re\n",
+        "\n",
+        "# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})\n",
+        "pat = re.compile(r\"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+\")\n",
+        "\n",
+        "demo_text = \"Hello, world! It's 2026 already.\"\n",
+        "chunks = re.findall(pat, demo_text)\n",
+        "print(\"split into chunks (BPE then runs inside each one):\")\n",
+        "print(chunks)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "e825ed0b",
+      "metadata": {},
+      "source": [
+        "### 4.2 Special tokens\n",
+        "\n",
+        "Real tokenizers also reserve **special tokens** that are not learned from text but inserted to\n",
+        "mark structure, for example `<|endoftext|>` between documents, or chat markers like\n",
+        "`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.\n",
+        "\n",
+        "Real-life picture: punctuation the model never \"spells\" — single reserved symbols that mean\n",
+        "\"new document starts here\" or \"the user is speaking now.\"\n",
+        "\n",
+        "```\n",
+        "<|endoftext|>   -> id 50256 in GPT-2 (separates documents)\n",
+        "<|im_start|>    -> chat role marker in chat models\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "638d25d8",
+      "metadata": {},
+      "source": [
+        "### 5.1 Why tokenization causes LLM quirks\n",
+        "\n",
+        "Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.\n",
+        "\n",
+        "- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does\n",
+        "  not \"see\" its letters.\n",
+        "- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.\n",
+        "- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more\n",
+        "  tokens (and more money / context)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "4f051e82",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "def n_tokens(s):\n",
+        "    return len(encode(s))\n",
+        "\n",
+        "print(\"spelling: 'extraordinary' ->\", n_tokens(\"extraordinary\"), \"token(s)\")\n",
+        "print(\"          the model sees a chunk, not the 13 letters\\n\")\n",
+        "\n",
+        "print(\"numbers : '1234567'      ->\", n_tokens(\"1234567\"), \"tokens (split oddly, not per-digit)\")\n",
+        "print(\"          '127 + 677'    ->\", n_tokens(\"127 + 677\"), \"tokens\\n\")\n",
+        "\n",
+        "eng = \"hello how are you\"\n",
+        "non = \"你好你好你好\"   # non-English of similar visible length\n",
+        "print(\"non-English costs more tokens for similar content:\")\n",
+        "print(\"  english   :\", repr(eng), \"->\", n_tokens(eng), \"tokens\")\n",
+        "print(\"  non-latin :\", repr(non), \"->\", n_tokens(non), \"tokens\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "cda06321",
+      "metadata": {},
+      "source": [
+        "**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the\n",
+        "compression ratio and the token counts in section 5.1. More merges = better compression, but a\n",
+        "bigger vocabulary. This trade-off is exactly what real tokenizer designers tune."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "35388d46",
+      "metadata": {},
+      "source": [
+        "## Series wrap-up\n",
+        "\n",
+        "You built the whole stack, from the ground up:\n",
+        "\n",
+        "- `00_micrograd.ipynb` — backprop from scratch (the engine)\n",
+        "- `01_build_gpt.ipynb` — the bigram baseline\n",
+        "- `02_makemore_mlp.ipynb` — an MLP with embeddings\n",
+        "- `03_batchnorm_activations.ipynb` — keeping deep nets healthy\n",
+        "- `04_backprop_ninja.ipynb` — gradients by hand\n",
+        "- `05_wavenet.ipynb` — a deeper, hierarchical model\n",
+        "- `06_build_gpt_attention.ipynb` — the transformer (GPT)\n",
+        "- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)\n",
+        "\n",
+        "Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,\n",
+        "with runnable code at every step.\n",
+        "\n",
+        "**Checklist**\n",
+        "- [ ] Text -> UTF-8 bytes (256 base tokens)\n",
+        "- [ ] `get_stats` + `merge` = the BPE mechanism\n",
+        "- [ ] Trained merges; measured compression\n",
+        "- [ ] encode / decode round-trips\n",
+        "- [ ] Regex pre-splitting and special tokens\n",
+        "- [ ] Explained tokenization-caused LLM quirks"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv (3.12.10)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.12.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

_validate_nb.py

@@ -0,0 +1,25 @@
+import json, sys, time
+
+path = sys.argv[1]
+nb = json.load(open(path, encoding="utf-8"))
+g = {"__name__": "__main__"}
+
+n = 0
+ok = True
+for c in nb["cells"]:
+    if c["cell_type"] != "code":
+        continue
+    n += 1
+    src = "".join(c["source"])
+    t0 = time.time()
+    try:
+        exec(src, g)
+        if "plt" in g and getattr(g["plt"], "show", None):
+            g["plt"].show = lambda *a, **k: None
+    except Exception as e:
+        print("CELL", n, "FAILED:", type(e).__name__, e)
+        ok = False
+        break
+    print(f"  (cell {n} ok, {time.time()-t0:.1f}s)")
+if ok:
+    print("ALL", n, "CODE CELLS RAN OK")

kylin_remote_setup.md

@@ -0,0 +1,475 @@
+# Kylin Remote Control & Monitoring Setup
+
+Complete reference for controlling and monitoring a Kylin Linux machine from a
+Windows 11 PC over a local phone-hotspot network. No VPN is used for the SSH
+connection itself — only the local Wi-Fi/hotspot.
+
+---
+
+## 0. Bootstrap — START HERE in a new session
+
+**The connection is already fully set up** (SSH key login + passwordless sudo).
+A new session does **NOT** need to repeat the setup in §2–§3 — those sections are
+records of what was already done. Just connect and go.
+
+**Verify everything with one command** (run from the Windows machine, e.g. Claude Code's Bash tool):
+
+```bash
+ssh -o BatchMode=yes -o ConnectTimeout=10 kylin 'echo SSH_OK; whoami; hostname; sudo -n true && echo SUDO_OK'
+```
+
+Expected: `SSH_OK` / `caichl` / `caichl-HUAWEIQINGYUNL420` / `SUDO_OK`.
+
+- ✅ **`SSH_OK` + `SUDO_OK`** → full control. Run anything (incl. `sudo`) via
+  `ssh -o BatchMode=yes kylin "<command>"`. No password needed.
+- ❌ **asks for password / times out** → most likely the **hotspot IP changed**
+  (`192.168.43.248` is DHCP-assigned and can change between sessions). Ask the user
+  to re-check the Kylin IP (`hostname -I` on that machine), update
+  `C:\Users\adusu\.ssh\config`, then re-verify. If it asks for a password, the key
+  may have been removed — see §2.3 / §2.4 to re-establish.
+- ℹ️ SSH prints a harmless **post-quantum warning** to stderr — ignore it (the
+  examples in this doc filter it out).
+
+**Connect manually:** `ssh kylin` (alias in `C:\Users\adusu\.ssh\config` → `caichl@192.168.43.248`,
+with the pinned cipher/MAC already configured — see §2).
+
+> A fresh Claude Code session also auto-loads memory pointers (`MEMORY.md` →
+> `kylin-remote-machine`, `kylin-command-logging`, `kylin-mihomo-vpn`) that link
+> back here. This file is the authoritative, self-contained reference.
+
+---
+
+## 1. The two machines
+
+| Role | Details |
+|------|---------|
+| **Controller** | Windows 11 (`C:\Users\adusu`), where Claude Code + VPN run |
+| **Target** | Kylin V10 SP1, Huawei QINGYUN L420, **ARM64 / aarch64** |
+| **Network** | Shared phone hotspot, subnet `192.168.43.x` |
+| **Target IP** | `192.168.43.248` |
+| **Target user** | `caichl` |
+
+> **Key idea:** Claude Code needs the VPN to reach Anthropic's servers, but the
+> SSH hop from Windows → Kylin is **local LAN traffic** and needs no VPN. The two
+> connections are independent.
+
+---
+
+## 2. SSH connection
+
+### 2.1 Required cipher/MAC (important)
+
+The default cipher fails over this hotspot link with
+`Corrupted MAC on input`. The connection must pin a specific cipher and MAC:
+
+```
+-c aes256-ctr -m hmac-sha2-256
+```
+
+### 2.2 Windows SSH config
+
+File: `C:\Users\adusu\.ssh\config`
+
+```sshconfig
+Host kylin kyln
+    HostName 192.168.43.248
+    User caichl
+    Ciphers aes256-ctr
+    MACs hmac-sha2-256
+```
+
+With this in place, simply run:
+
+```powershell
+ssh kylin       # or: ssh kyln  (both aliases work)
+```
+
+The equivalent explicit command (no config) is:
+
+```powershell
+ssh -m hmac-sha2-256 -c aes256-ctr caichl@192.168.43.248
+```
+
+### 2.3 Passwordless login (SSH key)
+
+The Windows key `~/.ssh/id_ed25519.pub` was copied to the Kylin machine:
+
+```powershell
+Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub | `
+  ssh kylin "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
+```
+
+Result: `ssh kylin` logs in with **no password**.
+
+### 2.4 Passwordless sudo
+
+So admin commands can run non-interactively from Windows:
+
+```bash
+# Run once on the Kylin machine (asks for password the one time)
+sudo bash -c 'echo "caichl ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/caichl-nopasswd; \
+              chmod 440 /etc/sudoers.d/caichl-nopasswd; \
+              visudo -cf /etc/sudoers.d/caichl-nopasswd'
+```
+
+> **Security note:** This makes `caichl` a passwordless-sudo account — acceptable
+> for a disposable machine only. To revert: `sudo rm /etc/sudoers.d/caichl-nopasswd`.
+
+After this, from Windows you can run, e.g.:
+
+```powershell
+ssh -o BatchMode=yes kylin "sudo systemctl status ssh"
+```
+
+### 2.5 Account credentials (recovery only)
+
+> ⚠️ **Plaintext password below — keep this file private; do NOT commit or share it.**
+
+| Item | Value |
+|------|-------|
+| User | `caichl` |
+| Password | `Caicai@2026` |
+
+**Normally not needed** — SSH login uses the key (§2.3) and sudo is passwordless
+(§2.4). Use this only to **recover** if the key auth or sudoers drop-in is ever lost
+(e.g. to log in interactively and re-copy the key, or to re-create
+`/etc/sudoers.d/caichl-nopasswd`). To change it: run `passwd` on the Kylin machine.
+
+---
+
+## 3. Initial SSH server setup (done on the Kylin machine)
+
+```bash
+sudo apt update
+sudo apt install -y openssh-server
+sudo systemctl enable --now ssh
+hostname -I        # find the IP (192.168.43.248)
+whoami             # confirm the username (caichl)
+```
+
+---
+
+## 4. Monitoring
+
+### 4.1 SSH logins & sessions (works out of the box)
+
+```bash
+who                                  # who is logged in + source
+w                                    # sessions + current command
+last                                 # login history with IPs
+lastb                                # FAILED login attempts (sudo)
+sudo grep "Accepted" /var/log/auth.log   # successful SSH logins
+sudo grep "Failed"   /var/log/auth.log   # failed/suspicious attempts
+sudo tail -f /var/log/auth.log           # live SSH activity
+```
+
+### 4.2 Command logging — no reboot (WORKING)
+
+Logs every command typed in an interactive shell, with user, tty, source IP,
+and the command text.
+
+**`/etc/profile.d/cmdlog.sh`**
+```bash
+# Log interactive shell commands to syslog (facility local6) -> /var/log/cmdlog.log
+if [ -n "$BASH" ] && [ -n "$PS1" ]; then
+  export PROMPT_COMMAND='logger -p local6.notice -t cmdlog "user=$USER tty=$(tty 2>/dev/null) from=$(who am i 2>/dev/null | sed -n "s/.*(\(.*\)).*/\1/p") cmd=$(history 1 | sed "s/^ *[0-9]* *//")"'
+fi
+```
+
+**`/etc/rsyslog.d/30-cmdlog.conf`**
+```
+local6.*    /var/log/cmdlog.log
+```
+
+**Setup commands**
+```bash
+sudo touch /var/log/cmdlog.log
+sudo chown syslog:adm /var/log/cmdlog.log   # MUST be syslog-owned (rsyslog drops privs)
+sudo chmod 640 /var/log/cmdlog.log
+sudo systemctl restart rsyslog
+```
+
+**View the logs**
+```bash
+sudo cat /var/log/cmdlog.log          # all logged commands
+sudo tail -f /var/log/cmdlog.log      # live
+```
+
+Example line:
+```
+Jun 11 10:54:22 caichl-HUAWEIQINGYUNL420 cmdlog: user=caichl tty=/dev/pts/5 from=192.168.43.180 cmd=echo hello
+```
+
+> **Gotcha:** rsyslog runs as the `syslog` user. If `/var/log/cmdlog.log` is owned
+> by `root:root`, nothing is written. It must be `chown syslog:adm`.
+
+### 4.3 System-wide auditing — auditd (QUEUED, UNVERIFIED)
+
+`auditd` is installed and enabled, with a rule logging all `execve` for real
+users:
+
+**`/etc/audit/rules.d/cmdlog.rules`**
+```
+-a exit,always -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
+```
+
+It cannot run yet because the kernel booted with `audit=0`. This was changed to
+`audit=1`:
+
+```bash
+sudo sed -i 's/audit=0/audit=1/' /etc/default/grub
+# /boot is read-only on this device — remount to rebuild grub
+sudo mount -o remount,rw /boot
+sudo update-grub
+sudo mount -o remount,ro /boot
+```
+
+> **Caveat:** This device appears to boot via **Huawei firmware, not standard
+> GRUB** (running kernel `-27` is not the one in `grub.cfg` which lists `-11`, and
+> the kernel cmdline has Huawei-specific params). So the `audit=1` change **may
+> not take effect** on reboot. The no-reboot logging in §4.2 already covers
+> interactive command logging, so auditd is a bonus.
+
+**Verify after any reboot**
+```bash
+cat /proc/sys/kernel/audit_enabled     # want: 1
+sudo systemctl is-active auditd        # want: active
+sudo ausearch -k cmdlog -i | tail -40  # human-readable command audit
+sudo aureport -x --summary             # summary of executables run
+```
+
+---
+
+## 5. Device-specific quirks (Huawei Kylin ARM64)
+
+- **Architecture is ARM64 (aarch64).** Software must have an ARM64 build; x86-only
+  proprietary apps will not install/run.
+- **`/boot` is a separate ext4 partition mounted read-only.** Remount rw to change
+  GRUB, then back to ro (see §4.3).
+- **Boot likely controlled by Huawei firmware**, limiting effectiveness of GRUB
+  cmdline edits.
+- **Hotspot link needs `aes256-ctr` / `hmac-sha2-256`** or SSH fails with
+  "Corrupted MAC on input".
+
+---
+
+## 6. Quick reference
+
+```powershell
+# Connect
+ssh kylin
+
+# Run a one-off command from Windows
+ssh kylin "uptime"
+
+# Run an admin command from Windows
+ssh kylin "sudo systemctl status ssh"
+
+# See logged commands
+ssh kylin "sudo tail -50 /var/log/cmdlog.log"
+
+# Watch live
+ssh kylin "sudo tail -f /var/log/cmdlog.log"
+
+# SSH login history
+ssh kylin "last"
+```
+
+---
+
+## 7. Troubleshooting
+
+| Symptom | Cause / Fix |
+|---------|-------------|
+| `Corrupted MAC on input` | Use `-c aes256-ctr -m hmac-sha2-256` (already in config) |
+| `Connection timed out` | Both devices not on same hotspot, or AP/client isolation on the phone — disable isolation |
+| `ssh kyln` "could not resolve" | Alias spelling — config now accepts both `kylin` and `kyln` |
+| Still asks for password | SSH key not copied (§2.3) |
+| `sudo` asks for password | sudoers drop-in empty/missing (§2.4) |
+| `audit support not in kernel` | Kernel booted `audit=0`; needs reboot with `audit=1` (§4.3) |
+| `cmdlog.log` empty | File not owned by `syslog:adm`; `sudo chown syslog:adm /var/log/cmdlog.log` then restart rsyslog |
+| `grub.cfg: Read-only file system` | `/boot` is read-only; `sudo mount -o remount,rw /boot` first |
+| VPN GUI app won't run (`GLIBC_2.34 not found`) | Kylin has glibc 2.31; modern Flutter/Electron clients need 2.34+. Use the Go-based mihomo core instead (see §8) |
+
+---
+
+## 8. VPN / proxy: mihomo + web dashboard
+
+**Why not FlClash / GUI clients:** Kylin V10 SP1 has **glibc 2.31**, but FlClash 0.8.93
+and similar modern GUI clients require **glibc 2.34+** and fail to run. Upgrading
+glibc would risk breaking the OS. mihomo is a **statically-linked Go binary** with
+no glibc dependency, so it runs fine — controlled via a browser dashboard.
+
+### Current state
+> **VPN is currently OFF.** The `mihomo` service is **stopped** and **disabled**
+> (auto-start removed), so it stays off across reboots until re-enabled. Ports
+> `7890`/`9090` are closed. The full configuration below is intact and ready to
+> resume.
+
+### Turn the VPN on / off (run on Kylin, or via `ssh kylin`)
+```bash
+# TURN ON (and restore auto-start on boot)
+sudo systemctl enable --now mihomo
+
+# TURN OFF for this session only (comes back on next reboot)
+sudo systemctl stop mihomo
+
+# TURN OFF and keep it off across reboots  <-- current state
+sudo systemctl stop mihomo && sudo systemctl disable mihomo
+
+# Check state
+systemctl is-active mihomo    # active / inactive
+systemctl is-enabled mihomo   # enabled / disabled
+```
+> Reminder: if you had set the Kylin **system proxy** to `127.0.0.1:7890`, switch
+> it back to "None" while the VPN is off, or apps will fail to reach the network.
+
+### Desktop access
+A launcher **"Mihomo VPN Dashboard"** was added to the app menu (Network category)
+at `/usr/share/applications/mihomo-dashboard.desktop`; it opens
+`http://127.0.0.1:9090/ui/`. (Only works while the service is running.)
+
+### Installed components
+| Item | Location / value |
+|------|------------------|
+| mihomo core | `/usr/local/bin/mihomo` (Clash.Meta v1.19.27, arm64) |
+| Config | `/etc/mihomo/config.yaml` |
+| Web dashboard (zashboard) | `/etc/mihomo/ui/` |
+| systemd service | `/etc/systemd/system/mihomo.service` (auto-starts on boot) |
+| Proxy port (HTTP+SOCKS) | `7890` |
+| Dashboard API | `0.0.0.0:9090` |
+| Dashboard secret | `428968aebf7c301565c6b6d3a11826c4` |
+
+### Install steps (done; for reference / rebuild)
+```bash
+# binary (downloaded on Windows through VPN, scp'd over to dodge chicken-and-egg)
+gunzip mihomo-linux-arm64-vX.gz
+sudo install -m 755 mihomo-linux-arm64 /usr/local/bin/mihomo
+# dashboard
+sudo mkdir -p /etc/mihomo/ui && sudo unzip zashboard-dist.zip -d /etc/mihomo/ui
+# config.yaml: mixed-port 7890, external-controller 0.0.0.0:9090, secret, external-ui: ui,
+#   proxy-providers (subscription URL), PROXY (select) + AUTO (url-test) groups
+sudo systemctl enable --now mihomo
+```
+> The subscription URL lives in `/etc/mihomo/config.yaml` under `proxy-providers:`.
+> `external-ui-name` must NOT be set (it makes mihomo look in a `ui/<name>` subdir
+> and auto-download); serve `/etc/mihomo/ui` directly.
+
+### Open the dashboard
+- **From Windows browser:** `http://192.168.43.248:9090/ui/`
+  - backend/API: `http://192.168.43.248:9090`  •  secret: `428968aebf7c301565c6b6d3a11826c4`
+- **From Kylin desktop browser:** `http://127.0.0.1:9090/ui/`
+
+### Use the proxy
+Point apps at `127.0.0.1:7890` (HTTP/SOCKS), or set the Kylin system proxy
+(Settings → Network → Proxy → Manual → `127.0.0.1:7890`). For transparent
+whole-system routing, enable mihomo **TUN mode** (`/dev/net/tun` is present).
+
+### Manage via API (examples)
+```bash
+S=428968aebf7c301565c6b6d3a11826c4
+# switch PROXY group to a node / AUTO
+curl -X PUT -H "Authorization: Bearer $S" http://127.0.0.1:9090/proxies/PROXY -d '{"name":"AUTO"}'
+# verify exit IP goes through the proxy
+curl -x http://127.0.0.1:7890 -s https://api.ip.sb/geoip
+# update subscription / reload
+sudo systemctl restart mihomo
+```
+
+**Verified working:** exit IP resolved to Singapore and `google.com/generate_204`
+returned HTTP 204 through the proxy.
+
+---
+
+## 9. Browsers
+
+| Browser | App-menu name | Type / location | Notes |
+|---------|---------------|-----------------|-------|
+| **Chromium** | "Chromium" | snap `chromium` v149 (`/snap/bin/chromium`) | Installed as the Chrome/Edge equivalent — same engine, Chrome Web Store extensions work. **Set as the default browser.** |
+
+- **Chrome & Edge cannot be installed** — neither has an official ARM64 Linux build
+  (same architecture wall as the VPN GUI clients). Chromium is the substitute.
+- Installing the chromium snap first required **updating snapd** (Kylin shipped 2.54,
+  too old → error `assumes unsupported features: snapd2.55`). Fix:
+  `sudo snap install snapd` (brought it to 2.75.2), then `sudo snap install chromium`.
+- Default browser is recorded in `~/.config/mimeapps.list`
+  (`text/html`, `x-scheme-handler/http(s)` → `chromium_chromium.desktop`). If a
+  double-click opens the wrong app, right-click the file → **Open with → Chromium →
+  Set as default**.
+
+### Desktop instruction files (for the user at the machine)
+- `~/桌面/VPN_Guide.html` + `~/桌面/VPN_Guide.md` (copies also in `~/Desktop/`):
+  a user-facing "how to turn the VPN on/off" guide. Double-click the **`.html`** to
+  read it rendered in a browser; the **`.md`** opens in **VS Code** (`code`, installed)
+  with `Ctrl+Shift+V` preview.
+- App-menu launcher **"Mihomo VPN Dashboard"** → opens `http://127.0.0.1:9090/ui/`
+  (only works while the mihomo service is running).
+
+---
+
+## 10. Inventory — everything set up on the Kylin machine
+
+| Area | What | State |
+|------|------|-------|
+| Access | SSH alias `kylin`/`kyln`, key login, passwordless sudo | ✅ active |
+| Monitoring | SSH login logs (`auth.log`); command logging → `/var/log/cmdlog.log` | ✅ active |
+| Monitoring | auditd execve rule | ⏳ queued (needs reboot + `audit=1`; may not take on Huawei firmware) |
+| VPN | mihomo core + zashboard dashboard, subscription configured | ⛔ installed, **OFF** (stopped + disabled) |
+| Browser | Chromium v149 (default), 360/CNOOC browser | ✅ installed |
+| Editors | VS Code (`code`), pluma, WPS Office | ✅ pre-installed |
+| IDE | Cursor 3.7.27 (`/opt/cursor`) via `env -i` wrapper | ✅ installed (see §11) |
+| Desktop docs | `VPN_Guide.html` / `.md`, dashboard launcher | ✅ on desktop |
+
+---
+
+## 11. Cursor IDE (Electron) — install + crash fixes
+
+Cursor (AI code editor, Electron-based) installed as an **extracted AppImage** at
+`/opt/cursor` (currently **v3.7.27**, ARM64). It will NOT run with a normal launcher
+on this Huawei/Kylin box — it needs a special wrapper.
+
+### Why the special launcher (three Kylin/Huawei ARM64 problems)
+1. **EFAULT machine-ID probe crash** — Cursor runs a command to generate a unique
+   machine ID; the Kylin kernel blocks that memory `write` as suspicious →
+   `Error: EFAULT: bad address in system call argument, write` → instant crash.
+   **Fix:** `env -i` strips the environment (especially D-Bus) so Cursor skips the probe.
+2. **Mali GPU crash (段错误 / segfault)** — hardware rendering conflicts with Huawei's
+   Mali GPU driver (`dmesg`: `mali gpu: kctx ... create/destroyed` at crash time).
+   **Fix:** `--disable-gpu` (software rendering).
+3. **Blocked sandbox** — `--no-sandbox`.
+   Also installed `xdg-desktop-portal` + `xdg-desktop-portal-gtk` (1.6.0) for the
+   Wayland file picker (necessary but not sufficient alone).
+
+### The launcher (already installed)
+- **Wrapper:** `/usr/local/bin/cursor-launch`
+- **Start-menu entry:** `/usr/share/applications/cursor.desktop` → `Exec=/usr/local/bin/cursor-launch %F`
+
+```sh
+#!/bin/sh
+exec env -i \
+  HOME="$HOME" DISPLAY="${DISPLAY:-:0}" PATH="$PATH" \
+  /opt/cursor/usr/share/cursor/cursor --no-sandbox --disable-gpu "$@"
+```
+
+### Updating Cursor to a newer version
+1. Get the latest ARM64 AppImage URL (downloads on Windows through the VPN):
+   ```bash
+   curl -sL -A "Mozilla/5.0" -H "Accept: application/json" \
+     "https://www.cursor.com/api/download?platform=linux-arm64&releaseTrack=stable"
+   # -> JSON with "downloadUrl" (…/Cursor-X.Y.Z-aarch64.AppImage) and "version"
+   curl -L -C - -o Cursor-new.AppImage "<downloadUrl>"
+   ```
+2. `scp` it to Kylin, then:
+   ```bash
+   chmod +x Cursor-new.AppImage
+   cd /tmp && ./Cursor-new.AppImage --appimage-extract        # -> /tmp/squashfs-root
+   sudo rm -rf /opt/cursor && sudo cp -r /tmp/squashfs-root /opt/cursor
+   sudo chown -R root:root /opt/cursor && rm -rf /tmp/squashfs-root
+   ```
+3. **No change needed** to the wrapper or menu entry — paths stay the same.
+   Verify glibc first (must need ≤ 2.31): `objdump -T /opt/cursor/usr/share/cursor/cursor | grep -oE 'GLIBC_[0-9.]+' | sort -V | tail`.
+
+### Cleanup note
+Old AppImages in `~/software/` (`Cursor-3.2.11`, `Cursor-3.7.27`) and the user's
+manual extract `~/software/squashfs-root` (3.2.11) are redundant now that `/opt/cursor`
+is the install — safe to delete to reclaim space.

local_ai_setup.md

@@ -1,149 +0,0 @@
-# Local AI Coding Setup — Windows 11 (Aider + Ollama)
-
-## Hardware
-- GPU: RTX 5070 Ti, 12GB VRAM
-- RAM: 32GB
-- OS: Windows 11
-
----
-
-## Prerequisites
-
-### 1. Install Ollama
-Download from https://ollama.com and install.
-
-### 2. Pull a model
-```powershell
-ollama pull qwen3-coder:30b   # 18GB, strong coding model (MoE, VRAM+RAM split)
-ollama pull gemma4:12b        # 12GB, responds in English more reliably
-```
-
-Confirm models are available:
-```powershell
-ollama list
-```
-
-Confirm Ollama is serving:
-```powershell
-curl http://localhost:11434/api/tags
-```
-
----
-
-## Install Aider
-
-```powershell
-pip install aider-chat
-aider --version   # should print version number
-```
-
----
-
-## Project Config
-
-Create `.aider.conf.yml` in your project root:
-
-```yaml
-model: ollama/gemma4:12b
-openai-api-base: http://localhost:11434/v1
-openai-api-key: ollama
-edit-format: whole
-map-tokens: 2048
-no-auto-commits: true
-system-prompt: "Always respond in English only. Never use Chinese or any other language."
-```
-
-**Config notes:**
-- `edit-format: whole` — more reliable file writes for local models (avoids code showing in chat only)
-- `map-tokens: 2048` — limits repo map size, important for smaller context windows
-- `no-auto-commits: true` — review changes before they hit git
-- `system-prompt` — forces English responses (important for Qwen3 which defaults to Chinese)
-
-Switch models without editing the file:
-```
-/model ollama/qwen3-coder:30b
-```
-
----
-
-## Optional: Performance Tuning (if inference is slow)
-
-Create a `Modelfile` in the project root:
-
-```
-FROM qwen3-coder:30b
-PARAMETER num_ctx 8192
-PARAMETER num_gpu 99
-```
-
-Register it:
-```powershell
-ollama create qwen3-coder-tuned -f Modelfile
-```
-
-Update `.aider.conf.yml`:
-```yaml
-model: ollama/qwen3-coder-tuned
-```
-
-**Why:** `num_gpu 99` maximizes VRAM usage. `num_ctx 8192` prevents the KV cache from
-spilling too much into RAM. Recommended context range: 4096–16384.
-
----
-
-## Daily Aider Workflow
-
-Start Aider (reads config automatically):
-```powershell
-aider
-```
-
-| Command | What it does |
-|---------|-------------|
-| `/add src/foo.py` | Add file to context |
-| `/drop src/foo.py` | Remove file from context |
-| `/diff` | See pending changes |
-| `/undo` | Revert last change |
-| `/run pytest` | Run command, share output with model |
-| `/run git rm --cached file.py` | Run git commands from inside Aider |
-| `/clear` | Clear chat history (keep files) |
-| `/settings` | Show loaded config (verify auto_commits, system_prompt) |
-| `/model ollama/gemma4:12b` | Switch model on the fly |
-| `/exit` | Quit |
-
----
-
-## Known Issues & Fixes
-
-### Model responds in Chinese
-Cause: Qwen3 is a Chinese model and defaults to Chinese.
-Fix: `system-prompt` in `.aider.conf.yml` (already included above). Restart Aider after config change.
-
-### Aider shows code in chat but doesn't write files
-Cause: Model didn't follow Aider's edit format.
-Fix 1: `/add target_file.py` before asking — hints the target file to the model.
-Fix 2: `edit-format: whole` in config (already included above).
-
-### "File not found" / repo-map warnings for deleted files
-Cause: File deleted from disk but still tracked in git index.
-Fix:
-```powershell
-git rm --cached filename.py
-```
-Or from inside Aider:
-```
-/run git rm --cached filename.py
-```
-
-### Config changes not taking effect
-Fix: Exit Aider (`/exit`) and restart. Config is only read at startup.
-
----
-
-## Verify Everything Works
-
-1. `ollama list` — model appears
-2. `aider --version` — prints version
-3. Start Aider, run `/settings` — confirm `auto_commits: False` and `system_prompt` present
-4. Ask Aider to create a simple file — it should write to disk, not just chat
-5. Response is in English

local_llm.md

@@ -1,342 +0,0 @@
-# Local AI Coding Environment Implementation Plan
-
-## Objective
-
-Build a local-first AI coding environment on a Windows 11 workstation with the following requirements:
-
-### Hardware
-
-* NVIDIA RTX 5070 Ti 12GB VRAM
-* 64GB+ system RAM (if available)
-* Windows 11
-* WSL2 Ubuntu
-* VS Code
-
-### Existing Software
-
-* Ollama installed and operational
-* Qwen3-Coder 30B installed in Ollama
-* Claude Code CLI installed
-* Git installed
-
-### Constraints
-
-* Source code is confidential.
-* All code must remain local by default.
-* Cloud models may be used only as an optional fallback.
-* Minimize API costs.
-* Support large codebases.
-
----
-
-# Target Architecture
-
-```text
-VS Code
-    |
-    +---- Continue Extension
-    |
-    +---- Claude Code CLI
-    |
-    +---- Aider
-              |
-              v
-           Ollama
-              |
-              v
-      Qwen3-Coder 30B
-```
-
----
-
-# Project Goals
-
-Implement and validate the following workflows.
-
-## Workflow 1: Local Chat
-
-Developer can:
-
-* Ask coding questions
-* Explain code
-* Generate code
-* Review code
-
-using:
-
-```text
-VS Code
-    +
-Continue
-    +
-Ollama
-    +
-Qwen3-Coder 30B
-```
-
----
-
-## Workflow 2: Local Agent
-
-Developer can:
-
-* Refactor code
-* Create files
-* Modify files
-* Run git-aware edits
-
-using:
-
-```text
-Aider
-    +
-Ollama
-    +
-Qwen3-Coder 30B
-```
-
----
-
-## Workflow 3: Claude Code Optional
-
-Developer can:
-
-* Use Claude Code against local Ollama models
-* Compare behavior against Aider
-* Determine whether Claude Code provides additional value
-
-using:
-
-```text
-Claude Code
-    +
-Ollama
-    +
-Qwen3-Coder 30B
-```
-
----
-
-# Deliverables
-
-Produce the following:
-
-## Deliverable 1
-
-Environment verification script.
-
-Verify:
-
-* Ollama installed
-* NVIDIA GPU visible
-* CUDA available
-* WSL functioning
-* Qwen3-Coder model available
-
-Expected output:
-
-```text
-PASS: Ollama
-PASS: GPU
-PASS: CUDA
-PASS: Qwen3-Coder
-```
-
----
-
-## Deliverable 2
-
-Aider installation guide.
-
-Include:
-
-### Linux / WSL installation
-
-Commands:
-
-```bash
-pipx install aider-chat
-```
-
-or preferred installation method.
-
-### Ollama integration
-
-Configuration examples.
-
-### Verification steps
-
-Simple repository test.
-
----
-
-## Deliverable 3
-
-Continue configuration.
-
-Create:
-
-### VS Code setup instructions
-
-Install Continue extension.
-
-### Model configuration
-
-Configure Continue to use:
-
-```text
-Ollama
-Qwen3-Coder 30B
-```
-
-### Example config files
-
-Include complete examples.
-
----
-
-## Deliverable 4
-
-Claude Code local model integration.
-
-Research and implement the best available approach for:
-
-```text
-Claude Code
-        ->
-OpenAI-compatible endpoint
-        ->
-Ollama
-```
-
-Requirements:
-
-* Use officially supported methods where possible.
-* Avoid unsupported hacks.
-* Document limitations.
-* Provide rollback procedure.
-
----
-
-## Deliverable 5
-
-Performance optimization.
-
-Analyze:
-
-### VRAM usage
-
-Qwen3-Coder 30B on RTX 5070 Ti 12GB.
-
-### Recommended quantization
-
-Evaluate:
-
-* Q4
-* Q5
-* IQ3
-
-Recommend the best balance between:
-
-* Quality
-* Speed
-* Memory usage
-
----
-
-## Deliverable 6
-
-Benchmark suite.
-
-Create repeatable tests:
-
-### Test 1
-
-Generate a REST API.
-
-### Test 2
-
-Refactor a medium-sized module.
-
-### Test 3
-
-Write unit tests.
-
-### Test 4
-
-Debug a failing application.
-
-Measure:
-
-* Completion time
-* Accuracy
-* Token throughput
-* User effort
-
-Compare:
-
-```text
-Aider + Qwen3-Coder 30B
-Claude Code + Qwen3-Coder 30B
-Continue + Qwen3-Coder 30B
-```
-
----
-
-# Preferred Outcome
-
-Primary development workflow:
-
-```text
-VS Code
-    +
-Continue
-    +
-Qwen3-Coder 30B
-```
-
-Agent workflow:
-
-```text
-Aider
-    +
-Qwen3-Coder 30B
-```
-
-Optional advanced workflow:
-
-```text
-Claude Code
-    +
-Qwen3-Coder 30B
-```
-
----
-
-# Success Criteria
-
-The project is successful if:
-
-1. All code remains local.
-2. No cloud services are required.
-3. Developer can perform daily coding tasks locally.
-4. Aider successfully edits repositories.
-5. Continue provides a productive IDE experience.
-6. Claude Code local integration is evaluated and documented.
-7. Setup can be reproduced on a fresh machine.
-
----
-
-# Final Report
-
-Produce a final report containing:
-
-* Architecture diagram
-* Installation steps
-* Configuration files
-* Benchmark results
-* Known limitations
-* Recommended workflow
-* Future upgrade path
-
-The report should be suitable for long-term maintenance and onboarding of additional developers.

merge_sheet.py

@@ -0,0 +1,27 @@
+import os
+import pandas as pdc
+from openpyxl import load_workbook
+
+input_folder = r"d:\input"
+
+merged_dfs = []
+total_rows = 0
+
+excel_files = [f for f in os.listdir(input_folder) if f.endswith(".xlsx")]
+
+for excel_file in sorted(excel_files):    
+    try:
+        wb = load_workbook(excel_file, data_only=False)
+        sheet_name = wb.sheetnames[0]  # Get first sheet name
+        df = pdc.read_excel(excel_file, sheet_name=sheet_name)
+        merged_dfs.append(df)
+        total_rows += len(df)
+    except Exception as e:
+        print(f"Error processing {excel_file}: " + str(e))
+        continue
+
+if merged_dfs:
+    merged_output = pd.concat(merged_dfs, ignore_index=True)
+    output_file = r"d:\output\merged_file.xlsx"
+    merged_output.to_excel(output_file, index=False)
+    print(f"Successfully saved {total_rows} rows to output file")

mlp_playground.html

@@ -0,0 +1,403 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8" />
+<meta name="viewport" content="width=device-width, initial-scale=1.0" />
+<title>MLP Playground — Neuron → Layer → MLP (section 5.1)</title>
+<script src="https://cdn.jsdelivr.net/npm/d3@7"></script>
+<style>
+  :root {
+    --bg: #0f1420;
+    --panel: #161d2e;
+    --panel-2: #1d2940;
+    --ink: #e8edf6;
+    --muted: #94a3b8;
+    --line: #2a3550;
+    --pos: #4ade80;   /* positive weight / activation */
+    --neg: #f87171;   /* negative weight / activation */
+    --accent: #60a5fa;
+    --accent-2: #fbbf24;
+  }
+  * { box-sizing: border-box; }
+  html, body { margin: 0; padding: 0; }
+  body {
+    background: radial-gradient(1200px 600px at 70% -10%, #1b2740 0%, var(--bg) 55%);
+    color: var(--ink);
+    font-family: "Segoe UI", system-ui, -apple-system, Roboto, sans-serif;
+    line-height: 1.5;
+    padding: 22px clamp(14px, 3vw, 40px) 60px;
+  }
+  h1 { font-size: clamp(20px, 2.6vw, 28px); margin: 0 0 4px; }
+  .sub { color: var(--muted); margin: 0 0 18px; max-width: 70ch; }
+  .sub b { color: var(--ink); }
+  .layout { display: grid; grid-template-columns: 300px 1fr; gap: 18px; align-items: start; }
+  @media (max-width: 900px) { .layout { grid-template-columns: 1fr; } }
+
+  .panel {
+    background: linear-gradient(180deg, var(--panel) 0%, var(--panel-2) 100%);
+    border: 1px solid var(--line);
+    border-radius: 14px;
+    padding: 16px 16px 18px;
+    box-shadow: 0 10px 30px rgba(0,0,0,.35);
+  }
+  .panel h2 { font-size: 14px; text-transform: uppercase; letter-spacing: .08em; color: var(--muted); margin: 0 0 12px; }
+
+  .field { margin-bottom: 16px; }
+  .field label { display: flex; justify-content: space-between; font-size: 13px; margin-bottom: 6px; }
+  .field label .val { font-variant-numeric: tabular-nums; color: var(--accent); font-weight: 600; }
+  input[type=range] { width: 100%; accent-color: var(--accent); }
+
+  .btn {
+    appearance: none; border: 1px solid var(--line); cursor: pointer;
+    background: var(--panel-2); color: var(--ink);
+    padding: 9px 12px; border-radius: 9px; font-size: 13px; font-weight: 600;
+    transition: .15s; width: 100%; margin-bottom: 9px;
+  }
+  .btn:hover { border-color: var(--accent); color: #fff; }
+  .btn.primary { background: linear-gradient(180deg, #3b82f6, #2563eb); border-color: #2563eb; }
+  .btn.primary:hover { background: linear-gradient(180deg, #4f93ff, #2f6fe0); }
+
+  .legend { display: flex; flex-direction: column; gap: 7px; font-size: 12.5px; color: var(--muted); }
+  .legend .row { display: flex; align-items: center; gap: 8px; }
+  .swatch { width: 22px; height: 4px; border-radius: 3px; }
+
+  .stage { background: rgba(0,0,0,.18); border-radius: 14px; }
+  svg { display: block; width: 100%; height: auto; }
+
+  .out-card {
+    margin-top: 14px; padding: 14px 16px; border-radius: 12px;
+    border: 1px solid var(--line); background: rgba(0,0,0,.2);
+    display: flex; align-items: center; justify-content: space-between; gap: 14px;
+  }
+  .out-card .label { color: var(--muted); font-size: 13px; }
+  .out-card .verdict { font-size: 26px; font-weight: 800; font-variant-numeric: tabular-nums; }
+
+  /* tooltip */
+  .tip {
+    position: fixed; pointer-events: none; z-index: 50; opacity: 0;
+    background: #0b1322; border: 1px solid var(--accent); color: var(--ink);
+    padding: 8px 10px; border-radius: 9px; font-size: 12.5px; max-width: 240px;
+    box-shadow: 0 8px 24px rgba(0,0,0,.5); transition: opacity .12s;
+  }
+  .tip b { color: var(--accent); }
+  .tip .mono { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; }
+
+  /* node + edge base styles */
+  .edge { transition: stroke-opacity .2s; }
+  .node circle { stroke: #0b1322; stroke-width: 1.5px; cursor: pointer; }
+  .node-label { font-size: 11px; fill: var(--muted); }
+  .layer-title { font-size: 12px; fill: var(--ink); font-weight: 700; text-anchor: middle; }
+  .layer-cap { font-size: 11px; fill: var(--muted); text-anchor: middle; }
+  .hint { color: var(--muted); font-size: 12px; margin-top: 10px; }
+  .pill { display:inline-block; padding:2px 7px; border:1px solid var(--line); border-radius:999px; font-size:11px; color:var(--muted); }
+</style>
+</head>
+<body>
+  <h1>MLP Playground — <span style="color:var(--accent)">Neuron → Layer → MLP</span></h1>
+  <p class="sub">
+    The reviewer-panel story from <b>section 5.1</b>, made live. Two inputs
+    (<b>income</b>, <b>credit</b>) flow through two panels of 8 reviewers into one final judge:
+    the <span class="pill">2 → 8 → 8 → 1</span> network. Every line is a weight, every dot is a
+    <span class="mono">tanh(w·x + b)</span> opinion. Drag the inputs, hover anything, or press play.
+  </p>
+
+  <div class="layout">
+    <div class="panel">
+      <h2>Inputs (the application)</h2>
+      <div class="field">
+        <label>Income (x₁) <span class="val" id="x1v">0.50</span></label>
+        <input type="range" id="x1" min="-1" max="1" step="0.01" value="0.5" />
+      </div>
+      <div class="field">
+        <label>Credit (x₂) <span class="val" id="x2v">-0.30</span></label>
+        <input type="range" id="x2" min="-1" max="1" step="0.01" value="-0.3" />
+      </div>
+
+      <h2 style="margin-top:6px">Controls</h2>
+      <button class="btn primary" id="play">▶ Animate forward pass</button>
+      <button class="btn" id="reroll">🎲 New random weights</button>
+      <button class="btn" id="reset">↺ Reset inputs</button>
+
+      <h2 style="margin-top:18px">Read the colors</h2>
+      <div class="legend">
+        <div class="row"><span class="swatch" style="background:var(--pos)"></span> positive weight / activation</div>
+        <div class="row"><span class="swatch" style="background:var(--neg)"></span> negative weight / activation</div>
+        <div class="row"><span class="swatch" style="background:var(--muted);height:2px"></span> thin line = weak, thick = strong</div>
+      </div>
+
+      <div class="out-card">
+        <span class="label">Final judge<br>score (tanh)</span>
+        <span class="verdict" id="verdict">—</span>
+      </div>
+      <p class="hint" id="verdict-text">Closer to <b style="color:var(--pos)">+1</b> = likely good tenant, closer to <b style="color:var(--neg)">−1</b> = likely risky.</p>
+    </div>
+
+    <div class="panel stage">
+      <svg id="net" viewBox="0 0 980 560" preserveAspectRatio="xMidYMid meet" aria-label="MLP diagram"></svg>
+      <p class="hint" style="padding:0 6px 4px">
+        Hover a <b>line</b> to see its weight, or a <b>dot</b> to see that reviewer's
+        <span class="mono">w·x + b</span> and squashed opinion. Layers left→right: form → junior panel → senior panel → final judge.
+      </p>
+    </div>
+  </div>
+
+  <div class="tip" id="tip"></div>
+
+<script>
+// ----------------------------------------------------------------------------
+// Tiny MLP, faithful to section 5.1: Neuron = tanh(w·x + b), Layer = neurons in
+// parallel, MLP = layers in sequence. Shape: 2 -> 8 -> 8 -> 1.
+// ----------------------------------------------------------------------------
+const SIZES = [2, 8, 8, 1];
+const LAYER_TITLES = ["Inputs", "Hidden panel 1", "Hidden panel 2", "Final judge"];
+const LAYER_CAPS   = ["income, credit", "8 junior reviewers", "8 senior reviewers", "1 score"];
+
+const rand = () => Math.random() * 2 - 1;          // uniform(-1, 1), like micrograd
+const tanh = (z) => Math.tanh(z);
+
+// Build network: weights[l] is matrix [nout x nin], biases[l] is [nout].
+let weights, biases;
+function buildNetwork() {
+  weights = []; biases = [];
+  for (let l = 0; l < SIZES.length - 1; l++) {
+    const nin = SIZES[l], nout = SIZES[l + 1];
+    weights.push(Array.from({length: nout}, () => Array.from({length: nin}, rand)));
+    biases.push(Array.from({length: nout}, rand));
+  }
+}
+buildNetwork();
+
+// Forward pass. Returns activations per layer (layer 0 = raw inputs) and the
+// pre-activation (w·x+b) for each neuron so the tooltip can show both.
+function forward(x) {
+  const acts = [x.slice()];
+  const preacts = [x.slice()];   // inputs have no pre-activation; mirror for indexing
+  let cur = x.slice();
+  for (let l = 0; l < weights.length; l++) {
+    const z = [], a = [];
+    for (let j = 0; j < weights[l].length; j++) {
+      let s = biases[l][j];
+      for (let i = 0; i < cur.length; i++) s += weights[l][j][i] * cur[i];
+      z.push(s); a.push(tanh(s));
+    }
+    preacts.push(z); acts.push(a); cur = a;
+  }
+  return { acts, preacts };
+}
+
+// ----------------------------------------------------------------------------
+// Layout geometry
+// ----------------------------------------------------------------------------
+const W = 980, H = 560, PAD_X = 90, PAD_TOP = 70, PAD_BOT = 50;
+const svg = d3.select("#net");
+const tip = d3.select("#tip");
+
+function layerX(l) { return PAD_X + (W - 2 * PAD_X) * (l / (SIZES.length - 1)); }
+function nodeY(l, j) {
+  const n = SIZES[l];
+  const top = PAD_TOP, bot = H - PAD_BOT, span = bot - top;
+  if (n === 1) return (top + bot) / 2;
+  return top + span * (j / (n - 1));
+}
+
+// color scales: weights & activations mapped to red(-)/green(+)
+function weightColor(w) { return w >= 0 ? "var(--pos)" : "var(--neg)"; }
+function actColor(a)    { return a >= 0 ? "var(--pos)" : "var(--neg)"; }
+function weightWidth(w) { return 0.6 + Math.min(5.5, Math.abs(w) * 3.2); }
+
+const gEdges  = svg.append("g").attr("class", "edges");
+const gNodes  = svg.append("g").attr("class", "nodes");
+const gTitles = svg.append("g").attr("class", "titles");
+
+// layer titles + captions
+function drawTitles() {
+  gTitles.selectAll("*").remove();
+  SIZES.forEach((n, l) => {
+    gTitles.append("text").attr("class", "layer-title")
+      .attr("x", layerX(l)).attr("y", 28).text(LAYER_TITLES[l]);
+    gTitles.append("text").attr("class", "layer-cap")
+      .attr("x", layerX(l)).attr("y", 46).text(LAYER_CAPS[l]);
+  });
+}
+
+// build edges + nodes once (geometry is static; only color/width/anim update)
+let edgeSel, nodeSel;
+function drawStructure() {
+  // edges
+  const edges = [];
+  for (let l = 0; l < weights.length; l++) {
+    for (let j = 0; j < SIZES[l + 1]; j++) {
+      for (let i = 0; i < SIZES[l]; i++) {
+        edges.push({ l, i, j,
+          x1: layerX(l),     y1: nodeY(l, i),
+          x2: layerX(l + 1), y2: nodeY(l + 1, j) });
+      }
+    }
+  }
+  edgeSel = gEdges.selectAll("line").data(edges).join("line")
+    .attr("class", "edge")
+    .attr("x1", d => d.x1).attr("y1", d => d.y1)
+    .attr("x2", d => d.x2).attr("y2", d => d.y2)
+    .on("mousemove", (e, d) => {
+      const w = weights[d.l][d.j][d.i];
+      showTip(e, `<b>Weight</b> · panel line<br>
+        <span class="mono">layer ${d.l} → ${d.l + 1}</span><br>
+        from neuron ${d.i} → neuron ${d.j}<br>
+        <span class="mono">w = ${w.toFixed(3)}</span><br>
+        <span style="color:${w>=0?'var(--pos)':'var(--neg)'}">${w>=0?'boosts':'damps'} this signal</span>`);
+    })
+    .on("mouseleave", hideTip);
+
+  // nodes
+  const nodes = [];
+  SIZES.forEach((n, l) => {
+    for (let j = 0; j < n; j++) nodes.push({ l, j, x: layerX(l), y: nodeY(l, j) });
+  });
+  const g = gNodes.selectAll("g.node").data(nodes).join("g").attr("class", "node")
+    .attr("transform", d => `translate(${d.x},${d.y})`);
+  g.selectAll("circle").data(d => [d]).join("circle").attr("r", d => d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11);
+  nodeSel = g;
+
+  // input row labels
+  gNodes.selectAll("text.in-lab").remove();
+  ["x₁ income", "x₂ credit"].forEach((t, i) => {
+    gNodes.append("text").attr("class", "node-label in-lab")
+      .attr("x", layerX(0) - 22).attr("y", nodeY(0, i) + 4)
+      .attr("text-anchor", "end").text(t);
+  });
+
+  g.on("mousemove", (e, d) => {
+    if (d.l === 0) {
+      showTip(e, `<b>Input</b><br><span class="mono">x${d.j+1} = ${state.acts[0][d.j].toFixed(3)}</span><br>${d.j===0?'monthly income':'credit score'} (raw form field)`);
+    } else {
+      const z = state.preacts[d.l][d.j], a = state.acts[d.l][d.j];
+      const role = d.l === SIZES.length - 1 ? "final judge" : `reviewer ${d.j} in panel ${d.l}`;
+      showTip(e, `<b>Neuron</b> · ${role}<br>
+        <span class="mono">w·x + b = ${z.toFixed(3)}</span><br>
+        <span class="mono">tanh(...) = <b style="color:${a>=0?'var(--pos)':'var(--neg)'}">${a.toFixed(3)}</b></span><br>
+        ${a>=0?'leaning yes':'leaning no'}`);
+    }
+  }).on("mouseleave", hideTip);
+}
+
+// ----------------------------------------------------------------------------
+// Paint current state (colors + widths) onto edges and nodes
+// ----------------------------------------------------------------------------
+let state = { acts: [], preacts: [] };
+
+function recompute() {
+  const x1 = +d3.select("#x1").property("value");
+  const x2 = +d3.select("#x2").property("value");
+  state = forward([x1, x2]);
+  paint();
+  updateVerdict();
+}
+
+function paint() {
+  edgeSel
+    .attr("stroke", d => weightColor(weights[d.l][d.j][d.i]))
+    .attr("stroke-width", d => weightWidth(weights[d.l][d.j][d.i]))
+    .attr("stroke-opacity", 0.45);
+
+  nodeSel.select("circle")
+    .attr("fill", d => {
+      const a = state.acts[d.l] ? state.acts[d.l][d.j] : 0;
+      const mag = Math.min(1, Math.abs(a));
+      const base = a >= 0 ? [74,222,128] : [248,113,113];
+      // blend toward dark panel for low magnitude
+      const mix = (c) => Math.round(40 + (c - 40) * (0.25 + 0.75 * mag));
+      return `rgb(${mix(base[0])},${mix(base[1])},${mix(base[2])})`;
+    });
+}
+
+function updateVerdict() {
+  const out = state.acts[state.acts.length - 1][0];
+  const el = d3.select("#verdict");
+  el.text((out >= 0 ? "+" : "") + out.toFixed(3))
+    .style("color", out >= 0 ? "var(--pos)" : "var(--neg)");
+  d3.select("#verdict-text").html(
+    out >= 0
+      ? `Score <b style="color:var(--pos)">${out.toFixed(2)}</b> → leaning <b>good tenant</b>.`
+      : `Score <b style="color:var(--neg)">${out.toFixed(2)}</b> → leaning <b>risky</b>.`
+  );
+}
+
+// ----------------------------------------------------------------------------
+// Animated forward pass: light up layer by layer, pulse edges between them
+// ----------------------------------------------------------------------------
+let playing = false;
+function animate() {
+  if (playing) return;
+  playing = true;
+  // dim everything first
+  edgeSel.attr("stroke-opacity", 0.08);
+  nodeSel.select("circle").attr("opacity", 0.25);
+  d3.select("#play").attr("disabled", true).text("▶ playing…");
+
+  const STEP = 620;
+  // light input layer
+  lightLayer(0, 0);
+  for (let l = 0; l < weights.length; l++) {
+    pulseEdges(l, STEP * (l + 0.15));
+    lightLayer(l + 1, STEP * (l + 1));
+  }
+  setTimeout(() => {
+    edgeSel.transition().duration(300).attr("stroke-opacity", 0.45);
+    nodeSel.select("circle").transition().duration(300).attr("opacity", 1);
+    playing = false;
+    d3.select("#play").attr("disabled", null).text("▶ Animate forward pass");
+  }, STEP * (weights.length + 1) + 250);
+}
+
+function lightLayer(l, delay) {
+  nodeSel.filter(d => d.l === l).select("circle")
+    .transition().delay(delay).duration(280)
+    .attr("opacity", 1)
+    .attr("r", d => (d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11) + 4)
+    .transition().duration(220)
+    .attr("r", d => d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11);
+}
+
+function pulseEdges(l, delay) {
+  edgeSel.filter(d => d.l === l)
+    .transition().delay(delay).duration(300).attr("stroke-opacity", 0.9)
+    .transition().duration(320).attr("stroke-opacity", 0.18);
+}
+
+// ----------------------------------------------------------------------------
+// Tooltip helpers
+// ----------------------------------------------------------------------------
+function showTip(e, html) {
+  tip.html(html).style("opacity", 1)
+    .style("left", (e.clientX + 14) + "px")
+    .style("top",  (e.clientY + 14) + "px");
+}
+function hideTip() { tip.style("opacity", 0); }
+
+// ----------------------------------------------------------------------------
+// Wire up controls
+// ----------------------------------------------------------------------------
+function syncLabels() {
+  d3.select("#x1v").text((+d3.select("#x1").property("value")).toFixed(2));
+  d3.select("#x2v").text((+d3.select("#x2").property("value")).toFixed(2));
+}
+d3.select("#x1").on("input", () => { syncLabels(); recompute(); });
+d3.select("#x2").on("input", () => { syncLabels(); recompute(); });
+d3.select("#play").on("click", animate);
+d3.select("#reroll").on("click", () => { buildNetwork(); drawStructure(); recompute(); });
+d3.select("#reset").on("click", () => {
+  d3.select("#x1").property("value", 0.5);
+  d3.select("#x2").property("value", -0.3);
+  syncLabels(); recompute();
+});
+
+// ----------------------------------------------------------------------------
+// Init
+// ----------------------------------------------------------------------------
+drawTitles();
+drawStructure();
+syncLabels();
+recompute();
+</script>
+</body>
+</html>

opencode_lesson.md

@@ -0,0 +1,250 @@
+# Local AI Coding Environment — Reference & Lessons Learned
+
+> Consolidates `local_ai_setup.md` and `local_llm.md`. Those files are superseded by this one.
+
+---
+
+## Hardware
+
+| Component | Spec |
+|-----------|------|
+| GPU | NVIDIA RTX 5070 Ti, 12 GB VRAM |
+| RAM | 32 GB system RAM |
+| OS | Windows 11 |
+| WSL | WSL2 Ubuntu |
+
+---
+
+## Software Stack
+
+| Tool | Purpose | Notes |
+|------|---------|-------|
+| Ollama | Local model server | Serves OpenAI-compatible API at `localhost:11434/v1` |
+| OpenCode | Primary AI coding agent | Installed via npm (`opencode-ai`) |
+| Aider | Alternative AI coding agent | pip: `aider-chat` |
+| Claude Code | Cloud AI coding agent | Optional; can point at Ollama |
+| VS Code | Editor | With Continue extension for inline chat |
+
+---
+
+## Architecture
+
+```
+VS Code
+  ├── Continue Extension  ──────────────┐
+  ├── Claude Code CLI  ────────────────┤
+  └── OpenCode / Aider  ───────────────┤
+                                        ▼
+                                     Ollama
+                                        │
+                                        ▼
+                               Qwen3-Coder 30B (primary)
+```
+
+All code stays local. Cloud models are optional fallback only.
+
+---
+
+## Ollama — Available Models
+
+| Model | Size | Use |
+|-------|------|-----|
+| `qwen3-coder:30b-fixed` | 18 GB | **Primary** — OpenCode agent (fixed config, see below) |
+| `qwen3-coder:30b` | 18 GB | Original pull (unmodified) |
+| `gemma4:latest` | 9.6 GB | English-reliable alternative |
+| `gemma4:12b` | 7.6 GB | Lighter English-reliable option |
+| `qwen2.5-coder:14b` | 9.0 GB | Mid-weight coding model |
+| `qwen2.5-coder:7b` | 4.7 GB | Fast, lightweight coding |
+| `qwen2.5-coder:1.5b` | 986 MB | Minimal footprint |
+| `qwen3:latest` | 5.2 GB | General chat |
+| `qwen3:1.7b` | 1.4 GB | Fast general chat |
+| `bge-m3:latest` | 1.2 GB | Embeddings |
+
+---
+
+## OpenCode Configuration
+
+**Config file:** `~/.config/opencode/opencode.jsonc`
+
+```jsonc
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "ollama": {
+      "options": {
+        "baseURL": "http://localhost:11434/v1"
+      },
+      "models": {
+        "qwen3-coder:30b-fixed": { "name": "Qwen3 Coder 30B" },
+        "qwen2.5-coder:14b":     { "name": "Qwen 2.5 Coder 14B" },
+        "qwen2.5-coder:7b":      { "name": "Qwen 2.5 Coder 7B" },
+        "gemma4:latest":         { "name": "Gemma 4 8B (latest)" },
+        "qwen3:latest":          { "name": "Qwen3 8B" }
+      }
+    }
+  },
+  "model": "ollama/qwen3-coder:30b-fixed"
+}
+```
+
+---
+
+## Active Model — qwen3-coder:30b-fixed Modelfile
+
+```
+FROM <local blob: sha256-9124a16eb1a7...>
+TEMPLATE {{ .Prompt }}
+PARAMETER repeat_penalty  1.05
+PARAMETER stop            <|im_start|>
+PARAMETER stop            <|im_end|>
+PARAMETER stop            <|endoftext|>
+PARAMETER temperature     0.7
+PARAMETER top_k           20
+PARAMETER top_p           0.8
+PARAMETER num_ctx         32768      ← fixed (was missing)
+PARAMETER num_predict     8192       ← fixed (was missing)
+```
+
+Model specs: 30.5B params, MoE architecture (qwen3moe), Q4_K_M quantization, supports tools + completion.
+
+---
+
+## Issue & Fix — Tool Call JSON Truncation
+
+### Error
+
+```
+llama-server returned invalid tool call arguments for "edit":
+unexpected end of ... [retrying in 51s attempt N]
+```
+
+### Root Cause
+
+Two Ollama parameters were missing from the model's Modelfile:
+
+| Parameter | Default when missing | Required for tool calls |
+|-----------|---------------------|------------------------|
+| `num_ctx` | **2048 tokens** | Minimum ~8192; 32768 recommended |
+| `num_predict` | **128 tokens** | Minimum ~1024; 8192 recommended |
+
+With only 128 output tokens allowed, the model began generating valid JSON for the `edit` tool call but ran out of budget before closing all braces, producing malformed JSON. The 2048-token context also filled up quickly during long sessions, compounding the problem.
+
+### Fix Applied
+
+Rebuilt `qwen3-coder:30b-fixed` with the two parameters added:
+
+```bash
+# Export current Modelfile, append fixes, rebuild
+ollama show qwen3-coder:30b-fixed --modelfile > Modelfile_fixed
+# (add num_ctx 32768 and num_predict 8192)
+ollama create qwen3-coder:30b-fixed -f Modelfile_fixed
+```
+
+No changes needed to the OpenCode config — same model name, same config file.
+
+### Verification
+
+```bash
+ollama show qwen3-coder:30b-fixed
+# Should show: num_ctx 32768 and num_predict 8192 in Parameters section
+```
+
+---
+
+## Aider Configuration
+
+**Config file:** `.aider.conf.yml` in project root
+
+```yaml
+model: ollama/qwen3-coder:30b
+openai-api-base: http://localhost:11434/v1
+openai-api-key: ollama
+edit-format: whole
+map-tokens: 2048
+no-auto-commits: true
+system-prompt: "Always respond in English only. Never use Chinese or any other language."
+```
+
+| Option | Why |
+|--------|-----|
+| `edit-format: whole` | More reliable file writes for local models |
+| `map-tokens: 2048` | Limits repo map size for smaller context windows |
+| `no-auto-commits: true` | Review changes before they hit git |
+| `system-prompt` | Qwen3 defaults to Chinese; forces English |
+
+### Aider Quick Reference
+
+| Command | What it does |
+|---------|-------------|
+| `/add src/foo.py` | Add file to context |
+| `/drop src/foo.py` | Remove file from context |
+| `/diff` | See pending changes |
+| `/undo` | Revert last change |
+| `/run pytest` | Run command, share output with model |
+| `/clear` | Clear chat history (keep files) |
+| `/settings` | Show loaded config |
+| `/model ollama/gemma4:12b` | Switch model on the fly |
+
+---
+
+## Known Issues & Fixes
+
+### Tool call JSON truncated (OpenCode / llama-server)
+**Error:** `llama-server returned invalid tool call arguments for "edit": unexpected end of...`
+**Cause:** `num_predict` too low (default 128) cuts off JSON mid-generation.
+**Fix:** See "Issue & Fix" section above — add `num_ctx` and `num_predict` to Modelfile.
+
+### Model responds in Chinese
+**Cause:** Qwen3 is a Chinese model and defaults to Chinese.
+**Fix (Aider):** `system-prompt` in `.aider.conf.yml`.
+**Fix (OpenCode):** Add a system prompt in OpenCode's project config if supported, or use `gemma4` models which default to English.
+
+### Aider writes code to chat but not to disk
+**Cause:** Model didn't follow Aider's edit format.
+**Fix 1:** `/add target_file.py` before the request — hints the target.
+**Fix 2:** `edit-format: whole` in config (already included above).
+
+### Config changes not taking effect (Aider)
+**Fix:** Exit (`/exit`) and restart. Config is read only at startup.
+
+### "File not found" / repo-map warnings for deleted files
+**Cause:** File deleted from disk but still tracked in git index.
+**Fix:**
+```bash
+git rm --cached filename.py
+# or from inside Aider:
+/run git rm --cached filename.py
+```
+
+### Model loops with filler text, never executes tool calls
+**Symptom:** Model repeatedly says "I'll continue..." but never reads/edits files. Context counter in OpenCode bottom-left shows a number near your `num_ctx` limit.
+**Cause:** Context window is full. No tokens left for tool call JSON, so the model outputs short filler and stalls.
+**Fix (immediate):** Start a new OpenCode session. Break large tasks (e.g. "translate all files") into one file per session.
+**Fix (permanent):** Increase `num_ctx` in the Modelfile (e.g. 65536), rebuild, restart Ollama. Watch RAM usage — each doubling of `num_ctx` roughly doubles KV cache memory.
+
+### Context fills up, model performance degrades
+**Cause:** Long session accumulates tokens beyond `num_ctx`.
+**Fix (Aider):** `/clear` to reset chat history without losing file context.
+**Fix (Ollama model):** Increase `num_ctx` in Modelfile (current: 32768).
+
+---
+
+## Performance Notes
+
+- qwen3-coder:30b is a MoE model — fits in VRAM+RAM split on RTX 5070 Ti (12 GB VRAM).
+- `num_gpu 99` in Modelfile maximizes layers on VRAM; remainder spills to RAM.
+- Safe `num_ctx` range for this hardware: **8192–32768**. Above 32768 risks RAM pressure.
+- Q4_K_M quantization gives best quality/speed/memory balance for this size class.
+
+---
+
+## Environment Verification Checklist
+
+```bash
+ollama list                          # models appear
+ollama show qwen3-coder:30b-fixed    # num_ctx and num_predict present
+curl http://localhost:11434/api/tags # Ollama serving
+aider --version                      # Aider installed
+opencode --version                   # OpenCode installed
+nvidia-smi                           # GPU visible
+```

requirements.txt

@@ -0,0 +1,4 @@
+torch>=2.0.0
+numpy
+ipython
+ipykernel

website/CLAUDE.md

@@ -0,0 +1,86 @@
+# Server Information
+
+**SSH Access:** `ssh root@39.102.124.161` (Alibaba Cloud)
+**OS:** CentOS with Baota panel pre-configured
+**Website:** https://www.thereisnospoonadu.com/
+**Web Server:** Tengine (Alibaba's fork of Nginx) — use `tengine` commands, NOT `nginx`
+
+## Gitea (self-hosted Git server)
+
+**Web UI:** https://git.thereisnospoonadu.com
+**Admin username:** beastgitea2026
+**API token:** paste fresh when needed (not stored)
+**SSH remote:** `git@git.thereisnospoonadu.com:beastgitea2026/<repo>.git`
+**SSH port:** 222 — configured in `~/.ssh/config` on dev machine
+**Gitea data:** `/var/lib/gitea/` | **Docker container name:** `gitea`
+**Tengine config:** `/etc/tengine/conf.d/gitea.conf`
+
+## Critical File Paths
+
+**Web Root:** `/www/wwwroot/soft_download/`
+**Tengine Config (ACTUAL):** `/etc/tengine/conf.d/thereisnospoonadu.conf`
+  - Site uses HTTPS (port 443), HTTP redirects to HTTPS
+  - Always test changes with: `tengine -t && tengine -s reload`
+
+**Auth File:** `/etc/tengine/.htpasswd`
+**Logs:** Check tengine log paths inside `/etc/tengine/conf.d/thereisnospoonadu.conf`
+
+## User Management
+
+Add a new end user (can login and download files):
+```bash
+# Add user (file already exists — do NOT use -c or it will overwrite all users)
+htpasswd /etc/tengine/.htpasswd username
+
+# Verify a user's password
+htpasswd -v /etc/tengine/.htpasswd username
+
+# Delete a user
+htpasswd -D /etc/tengine/.htpasswd username
+
+# List current users
+cat /etc/tengine/.htpasswd
+```
+Note: `htpasswd` requires `httpd-tools` package (`yum install -y httpd-tools`).
+
+## Architecture
+
+**Download Portal System:**
+- Login page: `/login.html` (public)
+- File listing: `/list/` (requires HTTP Basic Auth via Authorization header)
+- File downloads: Direct links to `.exe`, `.md`, `.pdf`, etc. (NO auth required)
+- Security model: Must login to see file list, but downloads are public once you know the filename
+
+**Key Files:**
+- `/www/wwwroot/soft_download/index.html` - Main file browser
+- `/www/wwwroot/soft_download/login.html` - Login page
+- `/www/wwwroot/soft_download/api/validate` - Auth validation endpoint
+
+## Important Notes
+
+- When making Tengine changes, ALWAYS edit `/etc/tengine/conf.d/thereisnospoonadu.conf`
+- Test against the actual domain: `curl -I https://www.thereisnospoonadu.com/filename`
+- PHP is NOT configured/working on this server - use Tengine-only solutions
+- Large files (170MB+ exe) need native browser downloads, not blob/XHR approach
+
+## Verification Checklist (Before Making Changes)
+
+```bash
+# 1. Verify which Tengine config is active
+tengine -T | grep -A 30 'server_name.*thereisnospoonadu'
+
+# 2. Test current behavior against actual domain
+curl -I https://www.thereisnospoonadu.com/版本升级说明（用记事本打开）.md
+
+# 3. After changes: test and reload
+tengine -t && tengine -s reload
+```
+
+## Troubleshooting (Only When Issues Occur)
+
+```bash
+# Find log paths from config
+grep 'access_log\|error_log' /etc/tengine/conf.d/thereisnospoonadu.conf
+```
+
+**Note:** Logs are automatically rotated daily, keeping 7 days of history (compressed).

website/admin_server.py

@@ -0,0 +1,276 @@
+#!/usr/bin/env python3
+import re
+import os
+from http.server import HTTPServer, BaseHTTPRequestHandler
+from datetime import datetime, timedelta
+from collections import defaultdict
+
+LOG_FILE = "/usr/local/nginx/logs/download_access.log"
+
+LOG_RE = re.compile(
+    r'(?P<ip>\S+) - (?P<user>\S+) \[(?P<time>[^\]]+)\] "(?P<request>[^"]*)" '
+    r'(?P<status>\d+) (?P<bytes>\d+) "(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
+)
+
+DOWNLOAD_EXTS = re.compile(
+    r'\.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)(\?|$)',
+    re.IGNORECASE
+)
+
+def fmt_bytes(b):
+    b = int(b)
+    for unit in ['B', 'KB', 'MB', 'GB']:
+        if b < 1024:
+            return "{:.1f} {}".format(b, unit)
+        b /= 1024
+    return "{:.1f} TB".format(b)
+
+def parse_time(s):
+    try:
+        return datetime.strptime(s, "%d/%b/%Y:%H:%M:%S %z")
+    except Exception:
+        return None
+
+def parse_logs():
+    entries = []
+    try:
+        with open(LOG_FILE, 'r', errors='replace') as f:
+            for line in f:
+                m = LOG_RE.match(line)
+                if not m:
+                    continue
+                user = m.group('user')
+                entries.append({
+                    'ip':      m.group('ip'),
+                    'user':    user,
+                    'time':    parse_time(m.group('time')),
+                    'request': m.group('request'),
+                    'status':  int(m.group('status')),
+                    'bytes':   int(m.group('bytes')),
+                })
+    except FileNotFoundError:
+        pass
+    return entries
+
+def build_stats(entries):
+    users = defaultdict(lambda: {
+        'logins': [], 'downloads': [], 'total_bytes': 0, 'ips': set()
+    })
+    all_downloads = []
+
+    for e in entries:
+        u = e['user']
+        t = e['time']
+        req = e['request']
+        parts = req.split(' ')
+        path = parts[1] if len(parts) >= 2 else req
+
+        # For anonymous download entries, extract username from ?u= query param
+        if u == '-' and DOWNLOAD_EXTS.search(path) and e['status'] in (200, 206):
+            m = re.search(r'[?&]u=([^&\s]+)', path)
+            if m:
+                try:
+                    from urllib.parse import unquote
+                    u = unquote(m.group(1))
+                except Exception:
+                    u = m.group(1)
+            else:
+                continue
+
+        if u == '-':
+            continue
+
+        users[u]['ips'].add(e['ip'])
+
+        # Count login: either /api/validate or /list/ success
+        if e['status'] == 200 and t:
+            if '/api/validate' in path or path.rstrip('/') == '/list':
+                users[u]['logins'].append(t)
+
+        if DOWNLOAD_EXTS.search(path) and e['status'] in (200, 206):
+            filename = path.split('/')[-1].split('?')[0]
+            try:
+                from urllib.parse import unquote
+                filename = unquote(filename)
+                # Tengine logs non-ASCII bytes as \xNN sequences; decode them as UTF-8
+                def _decode_nginx_escapes(s):
+                    parts = re.split(r'((?:\\x[0-9A-Fa-f]{2})+)', s)
+                    out = []
+                    for part in parts:
+                        if part.startswith('\\x'):
+                            raw = bytes(int(h, 16) for h in re.findall(r'\\x([0-9A-Fa-f]{2})', part))
+                            out.append(raw.decode('utf-8', errors='replace'))
+                        else:
+                            out.append(part)
+                    return ''.join(out)
+                filename = _decode_nginx_escapes(filename)
+            except Exception:
+                pass
+            users[u]['downloads'].append({'time': t, 'file': filename, 'bytes': e['bytes']})
+            users[u]['total_bytes'] += e['bytes']
+            all_downloads.append({'user': u, 'time': t, 'file': filename, 'bytes': e['bytes']})
+
+    def sort_key(x):
+        t = x['time']
+        if t is None:
+            return datetime.min.replace(tzinfo=None)
+        try:
+            return t.replace(tzinfo=None)
+        except Exception:
+            return datetime.min.replace(tzinfo=None)
+
+    all_downloads.sort(key=sort_key, reverse=True)
+    return users, all_downloads
+
+def render_html(users, all_downloads):
+    now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+    user_rows = ""
+    for uname in sorted(users.keys()):
+        data = users[uname]
+        logins = sorted(data['logins'], reverse=True)
+        last_login = logins[0].strftime("%Y-%m-%d %H:%M") if logins else "&mdash;"
+        login_count = len(logins)
+        dl_count = len(data['downloads'])
+        total_data = fmt_bytes(data['total_bytes'])
+        ips = ", ".join(sorted(data['ips']))
+
+        if logins:
+            delta = datetime.now(logins[0].tzinfo) - logins[0]
+            is_active = delta < timedelta(hours=24)
+        else:
+            is_active = False
+
+        status_class = "active" if is_active else "inactive"
+        status_label = "Active 24h" if is_active else "Inactive"
+
+        user_rows += (
+            "<tr>"
+            "<td><span class='username'>{}</span></td>"
+            "<td><span class='badge {}'>{}</span></td>"
+            "<td>{}</td>"
+            "<td class='num'>{}</td>"
+            "<td class='num'>{}</td>"
+            "<td class='num'>{}</td>"
+            "<td class='ip'>{}</td>"
+            "</tr>"
+        ).format(uname, status_class, status_label, last_login,
+                 login_count, dl_count, total_data, ips)
+
+    dl_rows = ""
+    for d in all_downloads[:100]:
+        t = d['time'].strftime("%Y-%m-%d %H:%M") if d['time'] else "&mdash;"
+        dl_rows += (
+            "<tr>"
+            "<td>{}</td>"
+            "<td><span class='username'>{}</span></td>"
+            "<td class='filename'>{}</td>"
+            "<td class='num'>{}</td>"
+            "</tr>"
+        ).format(t, d['user'], d['file'], fmt_bytes(d['bytes']))
+
+    total_users = len(users)
+    total_dls = sum(len(v['downloads']) for v in users.values())
+    total_data_all = fmt_bytes(sum(v['total_bytes'] for v in users.values()))
+
+    no_users = "<tr><td colspan='7' class='empty'>No authenticated activity yet</td></tr>"
+    no_dls = "<tr><td colspan='4' class='empty'>No downloads recorded yet</td></tr>"
+
+    html = """<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta http-equiv="refresh" content="60">
+<title>Admin Dashboard</title>
+<style>
+  * {{ box-sizing: border-box; margin: 0; padding: 0; }}
+  body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #f1f5f9; color: #1e293b; min-height: 100vh; }}
+  header {{ background: #ffffff; border-bottom: 1px solid #e2e8f0; padding: 20px 32px; display: flex; align-items: center; justify-content: space-between; }}
+  header h1 {{ font-size: 1.4rem; font-weight: 600; color: #7c3aed; }}
+  .refresh-note {{ font-size: 0.75rem; color: #94a3b8; }}
+  .stats-bar {{ display: flex; gap: 16px; padding: 24px 32px; }}
+  .stat-card {{ background: #ffffff; border: 1px solid #e2e8f0; border-radius: 10px; padding: 18px 24px; flex: 1; }}
+  .stat-card .label {{ font-size: 0.75rem; color: #94a3b8; text-transform: uppercase; letter-spacing: 0.05em; }}
+  .stat-card .value {{ font-size: 2rem; font-weight: 700; color: #7c3aed; margin-top: 4px; }}
+  .section {{ padding: 0 32px 32px; }}
+  .section h2 {{ font-size: 1rem; font-weight: 600; color: #64748b; margin-bottom: 12px; text-transform: uppercase; letter-spacing: 0.05em; }}
+  table {{ width: 100%; border-collapse: collapse; background: #ffffff; border: 1px solid #e2e8f0; border-radius: 10px; overflow: hidden; }}
+  th {{ background: #f8fafc; padding: 12px 16px; text-align: left; font-size: 0.75rem; color: #94a3b8; text-transform: uppercase; letter-spacing: 0.05em; font-weight: 600; }}
+  td {{ padding: 12px 16px; border-top: 1px solid #f1f5f9; font-size: 0.875rem; }}
+  tr:hover td {{ background: #f8fafc; }}
+  .username {{ color: #2563eb; font-weight: 500; }}
+  .filename {{ color: #059669; font-family: monospace; font-size: 0.8rem; }}
+  .ip {{ color: #64748b; font-size: 0.8rem; font-family: monospace; }}
+  .num {{ text-align: right; color: #1e293b; }}
+  .badge {{ padding: 2px 10px; border-radius: 999px; font-size: 0.7rem; font-weight: 600; }}
+  .badge.active {{ background: #d1fae5; color: #065f46; }}
+  .badge.inactive {{ background: #f1f5f9; color: #94a3b8; }}
+  .empty {{ text-align: center; color: #94a3b8; padding: 32px; }}
+</style>
+</head>
+<body>
+<header>
+  <h1>Admin Dashboard &mdash; thereisnospoonadu.com</h1>
+  <span class="refresh-note">Last updated: {now} &nbsp;|&nbsp; Auto-refresh every 60s</span>
+</header>
+
+<div class="stats-bar">
+  <div class="stat-card"><div class="label">Total Users</div><div class="value">{total_users}</div></div>
+  <div class="stat-card"><div class="label">Total Downloads</div><div class="value">{total_dls}</div></div>
+  <div class="stat-card"><div class="label">Total Data Served</div><div class="value">{total_data_all}</div></div>
+</div>
+
+<div class="section">
+  <h2>User Summary</h2>
+  <table>
+    <thead><tr>
+      <th>Username</th><th>Status</th><th>Last Login</th>
+      <th style="text-align:right">Logins</th>
+      <th style="text-align:right">Downloads</th>
+      <th style="text-align:right">Data Used</th>
+      <th>IP Address(es)</th>
+    </tr></thead>
+    <tbody>{user_rows}</tbody>
+  </table>
+</div>
+
+<div class="section">
+  <h2>Download History (last 100)</h2>
+  <table>
+    <thead><tr><th>Time</th><th>User</th><th>File</th><th style="text-align:right">Size</th></tr></thead>
+    <tbody>{dl_rows}</tbody>
+  </table>
+</div>
+</body>
+</html>""".format(
+        now=now,
+        total_users=total_users,
+        total_dls=total_dls,
+        total_data_all=total_data_all,
+        user_rows=user_rows if user_rows else no_users,
+        dl_rows=dl_rows if dl_rows else no_dls,
+    )
+    return html
+
+
+class Handler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        entries = parse_logs()
+        users, all_downloads = build_stats(entries)
+        html = render_html(users, all_downloads)
+        body = html.encode('utf-8')
+        self.send_response(200)
+        self.send_header('Content-Type', 'text/html; charset=utf-8')
+        self.send_header('Content-Length', str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, format, *args):
+        pass
+
+
+if __name__ == '__main__':
+    server = HTTPServer(('127.0.0.1', 8765), Handler)
+    print("Admin dashboard running on 127.0.0.1:8765")
+    server.serve_forever()

website/auth_check.php

@@ -0,0 +1,14 @@
+<?php
+/**
+ * Used by Nginx auth_request. Returns 200 if session valid, 401 otherwise.
+ * Also returns X-Auth-User header so Tengine can log the authenticated username.
+ * Deploy to /www/wwwroot/soft_download/auth_check.php
+ */
+session_start();
+if (!empty($_SESSION['soft_download_user'])) {
+    header('X-Auth-User: ' . $_SESSION['soft_download_user']);
+    http_response_code(200);
+    exit;
+}
+http_response_code(401);
+exit;

website/index.html

@@ -0,0 +1,197 @@
+<!DOCTYPE html>
+<html lang="zh">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>软件下载页面</title>
+<style>
+body{font-size:18px!important;line-height:2;font-family:sans-serif;margin:.5em 1.5em 2em}
+h1{font-size:24px!important;margin-top:0;margin-bottom:.5em;border-bottom:2px solid #ccc;padding-bottom:.3em}
+.breadcrumb{font-size:18px;margin-bottom:.75em}
+.breadcrumb a{color:#0066cc;cursor:pointer}
+.tools-nav{margin-bottom:.75em}
+.tools-link{text-decoration:underline;font-style:italic}
+.loading{font-size:20px;color:#666}
+.error{color:#c00;font-size:18px}
+.user-bar{margin-bottom:.4em;padding:.25em 0;display:flex;align-items:center;justify-content:flex-end;gap:1em}
+.user-bar .logged-in{color:green}
+.user-bar .logout-btn{padding:.3em .8em;font-size:16px;cursor:pointer;background:#0066cc;color:#fff;border:none;border-radius:4px}
+.user-bar .logout-btn:hover{background:#0052a3}
+.file-list{margin-top:.25em;overflow-x:auto}
+.file-table{border-collapse:collapse;width:75%;table-layout:fixed}
+.file-table th,.file-table td{padding:.45em .65em;border-bottom:1px solid #ddd;text-align:left;vertical-align:middle}
+.file-table thead th{font-size:17px;font-weight:600;color:#333;background:#f5f5f5;border-bottom:2px solid #ccc}
+.file-table .col-name{width:70%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
+.file-table .col-size{width:10%;white-space:nowrap;font-size:16px;color:#555}
+.file-table .col-mtime{width:20%;white-space:nowrap;font-size:16px;color:#555}
+.file-table a{font-size:19px!important;color:#0066cc;text-decoration:none}
+.file-table a:hover{text-decoration:underline}
+.file-table .dir{font-weight:bold;cursor:pointer}
+.file-table .empty-msg{text-align:center;color:#666;padding:1em}
+</style>
+</head>
+<body>
+<div class="user-bar">
+<span class="logged-in">当前已登录: <span id="username-display"></span></span>
+<button class="logout-btn" id="logout-btn">注销</button>
+</div>
+<h1>文件列表 <span id="path-display"></span></h1>
+<div class="breadcrumb" id="breadcrumb"></div>
+<div id="content">
+<div class="loading" id="loading">正在加载...</div>
+<div class="file-list" id="file-list" style="display:none"></div>
+<div class="error" id="error" style="display:none"></div>
+</div>
+<script>
+(function(){
+var authHeader=sessionStorage.getItem("auth");
+var username=sessionStorage.getItem("username");
+if(!authHeader){
+window.location.href="/login.html?return="+encodeURIComponent(window.location.pathname);
+return;
+}
+document.getElementById("username-display").textContent=username||"";
+loadFileList(getPath());
+
+function getPath(){
+var p=window.location.pathname;
+if(p==="/"||p==="")return"";
+if(p.endsWith("/"))p=p.slice(0,-1);
+return p.replace(/^\//,"");
+}
+
+function shouldHideFile(filename){
+var hiddenPatterns=[/\.php$/i,/\.html$/i,/\.backup$/i,/^api$/i,/^auth_check/i,/^login/i,/^logout/i,/^index\./i];
+return hiddenPatterns.some(function(pattern){return pattern.test(filename)});
+}
+
+function escapeHtml(s){
+if(s==null||s==="")return"";
+return String(s).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;");
+}
+
+function formatSize(bytes){
+if(!bytes)return"-";
+if(bytes<1024)return bytes+" B";
+if(bytes<1048576)return(bytes/1024).toFixed(1)+" K";
+if(bytes<1073741824)return(bytes/1048576).toFixed(1)+" M";
+return(bytes/1073741824).toFixed(1)+" G";
+}
+
+function formatMtime(mtime){
+if(!mtime)return"";
+try{
+var d=new Date(mtime);
+return isNaN(d)?mtime:d.toLocaleString();
+}catch(e){
+return mtime;
+}
+}
+
+function normalizeName(name){
+return (name||"").replace(/\/$/,"");
+}
+
+function isDirectory(item){
+return item.type==="directory"||/\/$/.test(item.name||"");
+}
+
+function renderBreadcrumb(path){
+var parts=path?path.split("/").filter(Boolean):[];
+if(parts.length===0){
+document.getElementById("breadcrumb").innerHTML="";
+return;
+}
+var inTools=parts[0]==="tools";
+var html='<a'+(inTools?' class="tools-link"':'')+' onclick="navigateTo(\'/\')">根目录</a>';
+var acc="";
+for(var i=0;i<parts.length;i++){
+acc+=(acc?"/":"")+parts[i];
+var linkClass=parts[i]==="tools"?' class="tools-link"':'';
+html+=' / <a'+linkClass+' onclick="navigateTo(\'/'+acc+'/\')">'+escapeHtml(parts[i])+"</a>";
+}
+document.getElementById("breadcrumb").innerHTML=html;
+}
+
+function buildToolsNavHtml(path,items){
+if(path)return "";
+var hasTools=items.some(function(item){return isDirectory(item)&&normalizeName(item.name)==="tools"});
+if(!hasTools)return "";
+return '<div class="breadcrumb tools-nav"><a class="tools-link" onclick="navigateTo(\'/tools/\')">tools</a></div>';
+}
+
+window.navigateTo=function(path){
+window.location.href=path;
+};
+
+function loadFileList(path){
+var loadingEl=document.getElementById("loading");
+var errorEl=document.getElementById("error");
+var listEl=document.getElementById("file-list");
+loadingEl.style.display="block";
+errorEl.style.display="none";
+listEl.style.display="none";
+
+fetch("/list/"+(path?path+"/":""),{
+headers:{Authorization:authHeader}
+})
+.then(function(r){
+if(r.status===401){
+sessionStorage.clear();
+window.location.href="/login.html";
+return;
+}
+if(!r.ok)throw new Error("加载失败");
+return r.json();
+})
+.then(function(data){
+if(!data)return;
+loadingEl.style.display="none";
+document.getElementById("path-display").textContent=path?"/"+path:"";
+renderBreadcrumb(path);
+
+var rows="";
+var items=Array.isArray(data)?data:[];
+var toolsHtml=buildToolsNavHtml(path,items);
+items.forEach(function(item){
+if(shouldHideFile(item.name))return;
+var name=normalizeName(item.name);
+if(!path&&isDirectory(item)&&name==="tools")return;
+var fullPath=(path?path+"/":"")+name;
+var isDir=isDirectory(item);
+var size=formatSize(item.size);
+var mtime=formatMtime(item.mtime);
+var nameEsc=escapeHtml(name)+(isDir?"/":"");
+
+if(isDir){
+var navPath="/"+fullPath+"/";
+rows+="<tr><td class=\"col-name\"><a onclick='navigateTo("+JSON.stringify(navPath)+")' class=\"dir\" href=\"javascript:void(0)\">"+nameEsc+"</a></td><td class=\"col-size\">"+escapeHtml(size)+"</td><td class=\"col-mtime\">"+escapeHtml(mtime)+"</td></tr>";
+}else{
+rows+="<tr><td class=\"col-name\"><a href=\"/"+fullPath+"?u="+encodeURIComponent(username)+"\" download>"+nameEsc+"</a></td><td class=\"col-size\">"+escapeHtml(size)+"</td><td class=\"col-mtime\">"+escapeHtml(mtime)+"</td></tr>";
+}
+});
+
+var tableStart='<table class="file-table"><thead><tr><th class="col-name">文件名称</th><th class="col-size">文件大小</th><th class="col-mtime">上传时间</th></tr></thead><tbody>';
+var tableEnd="</tbody></table>";
+if(!rows){
+listEl.innerHTML=toolsHtml+tableStart+'<tr><td class="empty-msg" colspan="3">目录为空</td></tr>'+tableEnd;
+}else{
+listEl.innerHTML=toolsHtml+tableStart+rows+tableEnd;
+}
+listEl.style.display="block";
+})
+.catch(function(err){
+loadingEl.style.display="none";
+errorEl.textContent="加载失败: "+(err.message||"未知错误");
+errorEl.style.display="block";
+});
+}
+
+document.getElementById("logout-btn").addEventListener("click",function(){
+sessionStorage.clear();
+window.location.href="/login.html";
+});
+})();
+</script>
+</body>
+</html>

website/login.html

@@ -0,0 +1,103 @@
+<!DOCTYPE html>
+<!-- Deploy: copy to /www/wwwroot/soft_download/login.html on the server. Must be excluded from auth_basic in Nginx. -->
+<html lang="zh">
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>登录 - 软件下载页面</title>
+    <style>
+        * { box-sizing: border-box; }
+        body { font-size: 18px; line-height: 1.6; font-family: sans-serif; margin: 0; padding: 2em; min-height: 100vh; display: flex; align-items: center; justify-content: center; background: #f5f5f5; }
+        .login-box { background: #fff; padding: 2em 2.5em; border-radius: 8px; box-shadow: 0 2px 10px rgba(0,0,0,0.1); max-width: 380px; width: 100%; }
+        h1 { font-size: 22px; margin: 0 0 1.2em 0; color: #333; border-bottom: 2px solid #ccc; padding-bottom: 0.4em; }
+        .form-group { margin-bottom: 1.2em; }
+        .form-group label { display: block; margin-bottom: 0.3em; color: #555; font-size: 16px; }
+        .form-group input { width: 100%; padding: 0.5em 0.8em; font-size: 16px; border: 1px solid #ccc; border-radius: 4px; }
+        .form-group input:focus { outline: none; border-color: #0066cc; }
+        .login-btn { width: 100%; padding: 0.6em 1em; font-size: 18px; background: #0066cc; color: #fff; border: none; border-radius: 4px; cursor: pointer; }
+        .login-btn:hover { background: #0052a3; }
+        .error-msg { color: #c00; font-size: 14px; margin-top: 0.5em; display: none; }
+    </style>
+</head>
+<body>
+    <div class="login-box">
+        <h1>登录</h1>
+        <form id="login-form">
+            <div class="form-group">
+                <label for="username">用户名</label>
+                <input type="text" id="username" name="username" required autocomplete="username">
+            </div>
+            <div class="form-group">
+                <label for="password">密码</label>
+                <input type="password" id="password" name="password" required autocomplete="current-password">
+            </div>
+            <div class="error-msg" id="error-msg"></div>
+            <button type="submit" class="login-btn">登录</button>
+        </form>
+    </div>
+    <script>
+        (function() {
+            var form = document.getElementById('login-form');
+            var errorEl = document.getElementById('error-msg');
+
+            function getReturnPath() {
+                var params = new URLSearchParams(window.location.search);
+                var ret = params.get('return');
+                return ret && ret.length > 0 ? ret : '/';
+            }
+
+            form.addEventListener('submit', function(e) {
+                e.preventDefault();
+                var user = document.getElementById('username').value.trim();
+                var pass = document.getElementById('password').value;
+                if (!user || !pass) {
+                    errorEl.textContent = '请输入用户名和密码';
+                    errorEl.style.display = 'block';
+                    return;
+                }
+                var returnPath = getReturnPath();
+                if (!returnPath.startsWith('/')) returnPath = '/' + returnPath;
+
+                // Submit to login.php via POST
+                var formData = new FormData();
+                formData.append('username', user);
+                formData.append('password', pass);
+                formData.append('return', returnPath);
+
+                fetch('/login.php', {
+                    method: 'POST',
+                    body: formData,
+                    credentials: 'same-origin'
+                })
+                .then(function(response) {
+                    if (response.redirected) {
+                        window.location.href = response.url;
+                    } else if (response.ok) {
+                        window.location.href = returnPath;
+                    } else {
+                        return response.text();
+                    }
+                })
+                .then(function(html) {
+                    if (html) {
+                        // Login failed, extract error message from response
+                        var parser = new DOMParser();
+                        var doc = parser.parseFromString(html, 'text/html');
+                        var errorMsg = doc.querySelector('.error-msg');
+                        if (errorMsg) {
+                            errorEl.textContent = errorMsg.textContent;
+                        } else {
+                            errorEl.textContent = '登录失败，请重试';
+                        }
+                        errorEl.style.display = 'block';
+                    }
+                })
+                .catch(function(err) {
+                    errorEl.textContent = '网络错误，请重试';
+                    errorEl.style.display = 'block';
+                });
+            });
+        })();
+    </script>
+</body>
+</html>

website/login.php

@@ -0,0 +1,91 @@
+<?php
+/**
+ * Session-based login. POST: validate against .htpasswd and set session. GET: show form.
+ * Deploy to /www/wwwroot/soft_download/login.php
+ * Requires: PHP with session support, htpasswd in PATH for validation.
+ */
+session_start();
+
+$HTPASSWD_FILE = '/usr/local/nginx/conf/.htpasswd';
+$HTPASSWD_CMD = '/usr/bin/htpasswd'; // or 'htpasswd' if in PATH
+
+if ($_SERVER['REQUEST_METHOD'] === 'POST') {
+    $user = isset($_POST['username']) ? trim((string) $_POST['username']) : '';
+    $pass = isset($_POST['password']) ? (string) $_POST['password'] : '';
+    $return_path = isset($_POST['return']) ? trim((string) $_POST['return']) : '/';
+    if (!strlen($return_path) || $return_path[0] !== '/') {
+        $return_path = '/';
+    }
+
+    $error = '';
+    if ($user !== '' && $pass !== '' && is_readable($HTPASSWD_FILE)) {
+        $user_esc = escapeshellarg($user);
+        $pass_esc = escapeshellarg($pass);
+        $file_esc = escapeshellarg($HTPASSWD_FILE);
+        $cmd = sprintf('%s -vb %s %s %s 2>/dev/null', $HTPASSWD_CMD, $file_esc, $user_esc, $pass_esc);
+        exec($cmd, $out, $code);
+        if ($code === 0) {
+            $_SESSION['soft_download_user'] = $user;
+            header('Location: ' . $return_path);
+            exit;
+        }
+        $error = '用户名或密码错误';
+    } else {
+        if ($user === '' && $pass === '') {
+            $error = '请输入用户名和密码';
+        } else {
+            $error = '用户名或密码错误';
+        }
+    }
+} else {
+    $error = '';
+    $return_path = isset($_GET['return']) ? trim((string) $_GET['return']) : '/';
+    if (!strlen($return_path) || $return_path[0] !== '/') {
+        $return_path = '/';
+    }
+}
+
+$return_esc = htmlspecialchars($return_path, ENT_QUOTES, 'UTF-8');
+$error_esc = htmlspecialchars($error ?? '', ENT_QUOTES, 'UTF-8');
+?>
+<!DOCTYPE html>
+<html lang="zh">
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>登录 - 软件下载页面</title>
+    <style>
+        * { box-sizing: border-box; }
+        body { font-size: 18px; line-height: 1.6; font-family: sans-serif; margin: 0; padding: 2em; min-height: 100vh; display: flex; align-items: center; justify-content: center; background: #f5f5f5; }
+        .login-box { background: #fff; padding: 2em 2.5em; border-radius: 8px; box-shadow: 0 2px 10px rgba(0,0,0,0.1); max-width: 380px; width: 100%; }
+        h1 { font-size: 22px; margin: 0 0 1.2em 0; color: #333; border-bottom: 2px solid #ccc; padding-bottom: 0.4em; }
+        .form-group { margin-bottom: 1.2em; }
+        .form-group label { display: block; margin-bottom: 0.3em; color: #555; font-size: 16px; }
+        .form-group input { width: 100%; padding: 0.5em 0.8em; font-size: 16px; border: 1px solid #ccc; border-radius: 4px; }
+        .form-group input:focus { outline: none; border-color: #0066cc; }
+        .login-btn { width: 100%; padding: 0.6em 1em; font-size: 18px; background: #0066cc; color: #fff; border: none; border-radius: 4px; cursor: pointer; }
+        .login-btn:hover { background: #0052a3; }
+        .error-msg { color: #c00; font-size: 14px; margin-top: 0.5em; }
+    </style>
+</head>
+<body>
+    <div class="login-box">
+        <h1>登录</h1>
+        <form method="post" action="login.php">
+            <input type="hidden" name="return" value="<?php echo $return_esc; ?>">
+            <div class="form-group">
+                <label for="username">用户名</label>
+                <input type="text" id="username" name="username" required autocomplete="username" value="<?php echo htmlspecialchars($_POST['username'] ?? '', ENT_QUOTES, 'UTF-8'); ?>">
+            </div>
+            <div class="form-group">
+                <label for="password">密码</label>
+                <input type="password" id="password" name="password" required autocomplete="current-password">
+            </div>
+            <?php if ($error_esc !== ''): ?>
+            <div class="error-msg"><?php echo $error_esc; ?></div>
+            <?php endif; ?>
+            <button type="submit" class="login-btn">登录</button>
+        </form>
+    </div>
+</body>
+</html>

website/logout.php

@@ -0,0 +1,14 @@
+<?php
+/**
+ * Destroy session and redirect to login.
+ * Deploy to /www/wwwroot/soft_download/logout.php
+ */
+session_start();
+$_SESSION = [];
+if (ini_get('session.use_cookies')) {
+    $p = session_get_cookie_params();
+    setcookie(session_name(), '', time() - 3600, $p['path'], $p['domain'], $p['secure'], $p['httponly']);
+}
+session_destroy();
+header('Location: login.php');
+exit;

website/send_user_emails.py

@@ -0,0 +1,168 @@
+#!/usr/bin/env python3
+"""
+Email sender for user credentials
+Sends username and password to all users in user_data.csv
+"""
+
+import csv
+import smtplib
+import time
+import os
+from email.mime.text import MIMEText
+from email.mime.multipart import MIMEMultipart
+from email.header import Header
+from datetime import datetime
+
+# ============ CONFIGURATION ============
+# TODO: Fill in your NetEase email credentials before running
+SENDER_EMAIL = "sofomoupc@163.com"  # Your NetEase email address
+SMTP_PASSWORD = "FERFPaJFJ89ckKw5"  # SMTP authorization code (NOT login password)
+SMTP_SERVER = "smtp.163.com"  # Use smtp.126.com for @126.com, smtp.yeah.net for @yeah.net
+SMTP_PORT = 465  # SSL port
+
+# Email settings
+EMAIL_SUBJECT = "“海察”软件下载网址用户名、密码"
+# Use absolute path to CSV file
+SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+CSV_FILE = os.path.join(SCRIPT_DIR, "user_data.csv")
+
+# Delay between emails (seconds) to avoid being flagged as spam
+DELAY_BETWEEN_EMAILS = 2
+# =======================================
+
+
+def create_email_body(username, password):
+    """Create email body with user credentials"""
+    return f"""用户名：{username}
+密码：{password}"""
+
+
+def send_email(sender, password, recipient, subject, body):
+    """Send email via NetEase SMTP server"""
+    try:
+        # Create message
+        msg = MIMEMultipart()
+        msg['From'] = Header(sender)
+        msg['To'] = Header(recipient)
+        msg['Subject'] = Header(subject, 'utf-8')
+
+        # Attach body
+        msg.attach(MIMEText(body, 'plain', 'utf-8'))
+
+        # Connect to SMTP server with SSL
+        server = smtplib.SMTP_SSL(SMTP_SERVER, SMTP_PORT)
+        server.login(sender, password)
+
+        # Send email
+        server.sendmail(sender, recipient, msg.as_string())
+        server.quit()
+
+        return True, None
+    except Exception as e:
+        return False, str(e)
+
+
+def read_users_from_csv(filename):
+    """Read user data from CSV file"""
+    users = []
+    try:
+        with open(filename, 'r', encoding='utf-8') as f:
+            reader = csv.DictReader(f)
+            for row in reader:
+                users.append({
+                    'username': row['username'],
+                    'passwd': row['passwd'],
+                    'email': row['email']
+                })
+        return users
+    except Exception as e:
+        print(f"Error reading CSV file: {e}")
+        return None
+
+
+def main():
+    """Main function to send emails to all users"""
+    print("=" * 60)
+    print("User Credentials Email Sender")
+    print("=" * 60)
+
+    # Validate configuration
+    if SENDER_EMAIL == "your_email@163.com" or SMTP_PASSWORD == "your_smtp_auth_code":
+        print("\n❌ ERROR: Please configure your email credentials first!")
+        print("Edit the CONFIGURATION section at the top of this script:")
+        print("  - SENDER_EMAIL: Your NetEase email address")
+        print("  - SMTP_PASSWORD: Your SMTP authorization code")
+        print("  - SMTP_SERVER: smtp.163.com (or smtp.126.com, smtp.yeah.net)")
+        return
+
+    # Read users from CSV
+    print(f"\n📂 Reading users from {CSV_FILE}...")
+    users = read_users_from_csv(CSV_FILE)
+
+    if not users:
+        print("❌ Failed to read users from CSV file")
+        return
+
+    print(f"✓ Found {len(users)} users")
+
+    # Confirm before sending
+    print(f"\n📧 Ready to send {len(users)} emails")
+    print(f"   From: {SENDER_EMAIL}")
+    print(f"   Subject: {EMAIL_SUBJECT}")
+    response = input("\nProceed? (yes/no): ")
+
+    if response.lower() not in ['yes', 'y']:
+        print("Cancelled.")
+        return
+
+    # Send emails
+    print(f"\n🚀 Sending emails...\n")
+    success_count = 0
+    failed_users = []
+
+    for i, user in enumerate(users, 1):
+        username = user['username']
+        passwd = user['passwd']
+        email = user['email']
+
+        print(f"[{i}/{len(users)}] Sending to {username} ({email})...", end=" ")
+
+        # Create email body
+        body = create_email_body(username, passwd)
+
+        # Send email
+        success, error = send_email(SENDER_EMAIL, SMTP_PASSWORD, email, EMAIL_SUBJECT, body)
+
+        if success:
+            print("✓")
+            success_count += 1
+        else:
+            print(f"✗ Failed: {error}")
+            failed_users.append({'user': username, 'email': email, 'error': error})
+
+        # Delay between emails
+        if i < len(users):
+            time.sleep(DELAY_BETWEEN_EMAILS)
+
+    # Summary
+    print("\n" + "=" * 60)
+    print(f"✓ Successfully sent: {success_count}/{len(users)}")
+
+    if failed_users:
+        print(f"✗ Failed: {len(failed_users)}")
+        print("\nFailed users:")
+        for failed in failed_users:
+            print(f"  - {failed['user']} ({failed['email']}): {failed['error']}")
+
+        # Save failed users to log
+        log_file = f"failed_emails_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
+        with open(log_file, 'w', encoding='utf-8') as f:
+            for failed in failed_users:
+                f.write(f"{failed['user']},{failed['email']},{failed['error']}\n")
+        print(f"\n📝 Failed users saved to: {log_file}")
+
+    print("=" * 60)
+
+
+if __name__ == "__main__":
+    main()

website/ssh_setup.py

@@ -0,0 +1,158 @@
+import paramiko
+import sys
+
+HOST = "39.102.124.161"
+USER = "root"
+PASS = "Beast2026"
+
+VHOST_CONFIG = r"""# Redirect HTTP to HTTPS
+server {
+    listen 80;
+    server_name www.thereisnospoonadu.com thereisnospoonadu.com;
+
+    location /.well-known/acme-challenge/ {
+        root /var/www/certbot;
+        allow all;
+    }
+
+    location / {
+        return 301 https://$host$request_uri;
+    }
+}
+
+# HTTPS Server
+server {
+    listen 443 ssl;
+    server_name www.thereisnospoonadu.com thereisnospoonadu.com;
+
+    ssl_certificate /etc/letsencrypt/live/www.thereisnospoonadu.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/www.thereisnospoonadu.com/privkey.pem;
+
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
+    ssl_prefer_server_ciphers on;
+    ssl_session_cache shared:SSL:10m;
+    ssl_session_timeout 10m;
+
+    root /www/wwwroot/soft_download;
+    index index.html index.htm;
+    charset utf-8;
+
+    client_max_body_size 500M;
+    client_body_timeout 300s;
+    send_timeout 300s;
+    keepalive_timeout 300s;
+    proxy_read_timeout 300s;
+
+    location = / {
+        try_files /index.html =404;
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    location = /index.html {
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    location = /login.html {
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    location = /api/validate {
+        auth_basic "App Download Portal";
+        auth_basic_user_file /etc/tengine/.htpasswd;
+        add_header Content-Type application/json;
+        add_header Access-Control-Allow-Origin *;
+        try_files /api/validate =404;
+    }
+
+    location ^~ /list/ {
+        index nothing_will_match;
+        auth_basic "App Download Portal";
+        auth_basic_user_file /etc/tengine/.htpasswd;
+        alias /www/wwwroot/soft_download/;
+        autoindex on;
+        autoindex_exact_size off;
+        autoindex_localtime on;
+        autoindex_format json;
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    location ~ \.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)$ {
+        add_header Content-Disposition "attachment";
+        add_header X-Content-Type-Options "nosniff";
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    location / {
+        auth_basic "App Download Portal";
+        auth_basic_user_file /etc/tengine/.htpasswd;
+        try_files $uri $uri/ =404;
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    access_log /var/log/tengine/download_access.log;
+    error_log /var/log/tengine/download_error.log warn;
+    server_tokens off;
+}
+"""
+
+def run_cmd(ssh, cmd, desc=""):
+    if desc:
+        print(f"\n{'='*60}")
+        print(f">>> {desc}")
+        print(f"CMD: {cmd}")
+        print('='*60)
+    stdin, stdout, stderr = ssh.exec_command(cmd, timeout=30)
+    out = stdout.read().decode('utf-8', errors='replace')
+    err = stderr.read().decode('utf-8', errors='replace')
+    rc = stdout.channel.recv_exit_status()
+    if out:
+        print("STDOUT:", out)
+    if err:
+        print("STDERR:", err)
+    print(f"EXIT CODE: {rc}")
+    return rc, out, err
+
+def main():
+    print(f"Connecting to {HOST}...")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(HOST, username=USER, password=PASS, timeout=15)
+    print("Connected.\n")
+
+    # Step 1: Write the vhost config
+    print("="*60)
+    print(">>> STEP 1: Write vhost config")
+    print("="*60)
+    run_cmd(ssh, "mkdir -p /etc/tengine/conf.d", "Ensure conf.d dir exists")
+
+    sftp = ssh.open_sftp()
+    with sftp.open("/etc/tengine/conf.d/thereisnospoonadu.conf", "w") as f:
+        f.write(VHOST_CONFIG)
+    sftp.close()
+    print("File written: /etc/tengine/conf.d/thereisnospoonadu.conf")
+
+    # Verify write
+    run_cmd(ssh, "cat /etc/tengine/conf.d/thereisnospoonadu.conf", "Verify written file")
+
+    # Step 2: Test the config
+    run_cmd(ssh, "tengine -t", "STEP 2: tengine -t (config test)")
+
+    # Step 3: Enable and start tengine
+    run_cmd(ssh, "systemctl enable tengine && systemctl start tengine", "STEP 3: Enable and start tengine")
+
+    # Step 4: Verify running and listening
+    run_cmd(ssh, "systemctl status tengine --no-pager", "STEP 4a: systemctl status tengine")
+    run_cmd(ssh, "ss -tlnp | grep -E '80|443'", "STEP 4b: ss -tlnp ports 80/443")
+
+    # Step 5: Quick local test
+    run_cmd(ssh,
+        'curl -sk https://localhost/ -o /dev/null -w "%{http_code}" --resolve www.thereisnospoonadu.com:443:127.0.0.1 -H "Host: www.thereisnospoonadu.com" 2>/dev/null || echo "curl test done"',
+        "STEP 5: Quick local curl test"
+    )
+
+    ssh.close()
+    print("\nAll steps complete.")
+
+if __name__ == "__main__":
+    main()

website/thereisnospoonadu.conf

@@ -0,0 +1,91 @@
+# Redirect all HTTP traffic to HTTPS (forces secure connections)
+server {
+    listen 80;
+    server_name www.thereisnospoonadu.com thereisnospoonadu.com;
+
+    # Certbot ACME challenge - must be served over HTTP for Let's Encrypt validation
+    location /.well-known/acme-challenge/ {
+        root /var/www/certbot;
+        allow all;
+    }
+
+    location / {
+        return 301 https://$host$request_uri;
+    }
+}
+
+# Custom log format: $remote_user is populated by auth_basic, captures real username
+log_format download '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" "$http_user_agent"';
+
+# HTTPS Server Configuration (core SSL setup + Basic Auth download portal)
+server {
+    listen 443 ssl;
+    server_name www.thereisnospoonadu.com thereisnospoonadu.com;
+
+    # Path to your Certbot SSL certificate (unchanged—valid path)
+    ssl_certificate /etc/letsencrypt/live/www.thereisnospoonadu.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/www.thereisnospoonadu.com/privkey.pem;
+
+    # Modern SSL security settings (A+ grade)
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
+    ssl_prefer_server_ciphers on;
+    ssl_session_cache shared:SSL:10m;
+    ssl_session_timeout 10m;
+
+    root /www/wwwroot/soft_download;
+    index index.html index.htm;
+    charset utf-8;
+
+    # Entry points: no auth required
+    location = / {
+        try_files /index.html =404;
+        add_header Access-Control-Allow-Origin *;
+    }
+    location = /index.html {
+        add_header Access-Control-Allow-Origin *;
+    }
+    location = /login.html {
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    # API validation endpoint: HTTP Basic Auth
+    location = /api/validate {
+        auth_basic "App Download Portal";
+        auth_basic_user_file /etc/tengine/.htpasswd;
+        default_type text/plain;
+        alias /www/wwwroot/soft_download/api/validate.txt;
+    }
+
+    # JSON file listing: HTTP Basic Auth
+    location /list/ {
+        auth_basic "App Download Portal";
+        auth_basic_user_file /etc/tengine/.htpasswd;
+        alias /www/wwwroot/soft_download/;
+        index nothing_will_match;
+        autoindex on;
+        autoindex_exact_size off;
+        autoindex_localtime on;
+        autoindex_format json;
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    # File downloads: no auth required (username tracked via ?u= query param)
+    location ~ \.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)$ {
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    # All other paths: serve static files (no auth)
+    location / {
+        try_files $uri $uri/ =404;
+        error_page 403 = /index.html;
+        add_header Access-Control-Allow-Origin *;
+    }
+
+    access_log /usr/local/nginx/logs/download_access.log download;
+    error_log /usr/local/nginx/logs/download_error.log warn;
+
+    # Security: Hide Nginx version
+    server_tokens off;
+}

website/user_data copy.csv

@@ -0,0 +1,28 @@
+username,passwd,email
+huangjing,1090,179864000@qq.com
+douzhn,1111,29867370@qq.com
+zhenglu,0872,18622306979@163.com
+zhangshh,1081,nkuzsh@sina.com
+qiangdq,1165,eacone@sina.com
+zhangwd,1381,zhangwendi0820@163.com
+chengqi,1481,531740421@qq.com
+lizj,1231,tjlizejun@163.com
+sunyl,1270,39204996@qq.com
+caichl,1296,cwtjwd@126.com
+lijy,0818,lijiayan222@qq.com
+chzhl,1295,46113395@qq.com
+qinglu,1230,luqingsmiles@foxmail.com
+renzw,1653,2734310562@qq.com
+songht,1215,luck_0076@163.com
+wangwx,1280,wenxingone@qq.com
+yangbo,1837,yangbo_6@126.com
+wangyi,0870,691511321@qq.com
+wangzhr,1407,396400640@qq.com
+weihq,1816,hailife@163.com
+yushj,1126,122758312@qq.com
+zhbl,1468,sdzbxiaolang@163.com
+shxj,1120,13811022092@139.com
+zhangl,1665,452838289@qq.com
+heyb,1019,121867346@qq.com
+dingpc,1164,pchding@163.com
+chengyh,1026,359957622@qq.com

website/user_data.csv

@@ -0,0 +1,26 @@
+username,passwd,email
+douzhn,1111,29867370@qq.com
+zhenglu,0872,18622306979@163.com
+zhangshh,1081,nkuzsh@sina.com
+qiangdq,1165,eacone@sina.com
+zhangwd,1381,zhangwendi0820@163.com
+chengqi,1481,531740421@qq.com
+lizj,1231,tjlizejun@163.com
+sunyl,1270,39204996@qq.com
+caichl,1296,cwtjwd@126.com
+lijy,0818,lijiayan222@qq.com
+chzhl,1295,46113395@qq.com
+qinglu,1230,luqingsmiles@foxmail.com
+renzw,1653,2734310562@qq.com
+songht,1215,luck_0076@163.com
+wangwx,1280,wenxingone@qq.com
+yangbo,1837,yangbo_6@126.com
+wangyi,0870,691511321@qq.com
+weihq,1816,hailife@163.com
+yushj,1126,122758312@qq.com
+zhbl,1468,sdzbxiaolang@163.com
+shxj,1120,13811022092@139.com
+zhangl,1665,452838289@qq.com
+heyb,1019,121867346@qq.com
+dingpc,1164,pchding@163.com
+chengyh,1026,359957622@qq.com

website/软件下载页面.mhtml

@@ -0,0 +1,105 @@
+From: <Saved by Blink>
+Snapshot-Content-Location: https://www.thereisnospoonadu.com/
+Subject: =?utf-8?Q?=E8=BD=AF=E4=BB=B6=E4=B8=8B=E8=BD=BD=E9=A1=B5=E9=9D=A2?=
+Date: Fri, 3 Apr 2026 14:22:37 +0800
+MIME-Version: 1.0
+Content-Type: multipart/related;
+	type="text/html";
+	boundary="----MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----"
+
+
+------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----
+Content-Type: text/html
+Content-ID: <frame-86EA5B71750C5F07310888D40FAFED31@mhtml.blink>
+Content-Transfer-Encoding: quoted-printable
+Content-Location: https://www.thereisnospoonadu.com/
+
+<!DOCTYPE html><html lang=3D"zh"><head><meta http-equiv=3D"Content-Type" co=
+ntent=3D"text/html; charset=3DUTF-8"><link rel=3D"stylesheet" type=3D"text/=
+css" href=3D"cid:css-37d7331d-d968-49c2-b3bd-7d9250c044e7@mhtml.blink" />
+
+<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=3D1.=
+0">
+<title>=E8=BD=AF=E4=BB=B6=E4=B8=8B=E8=BD=BD=E9=A1=B5=E9=9D=A2</title>
+
+</head>
+<body>
+<div class=3D"user-bar">
+<span class=3D"logged-in">=E5=BD=93=E5=89=8D=E5=B7=B2=E7=99=BB=E5=BD=95: <s=
+pan id=3D"username-display">haoxp</span></span>
+<button class=3D"logout-btn" id=3D"logout-btn">=E6=B3=A8=E9=94=80</button>
+</div>
+<h1>=E6=96=87=E4=BB=B6=E5=88=97=E8=A1=A8 <span id=3D"path-display"></span><=
+/h1>
+<div class=3D"breadcrumb" id=3D"breadcrumb"></div>
+<div id=3D"content">
+<div class=3D"loading" id=3D"loading" style=3D"display: none;">=E6=AD=A3=E5=
+=9C=A8=E5=8A=A0=E8=BD=BD...</div>
+<div class=3D"file-list" id=3D"file-list" style=3D"display: block;"><a href=
+=3D"https://www.thereisnospoonadu.com/%E7%89%88%E6%9C%AC%E5%8D%87%E7%BA%A7%=
+E8%AF%B4%E6%98%8E%EF%BC%88%E7%94%A8%E8%AE%B0%E4%BA%8B%E6%9C%AC%E6%89%93%E5%=
+BC%80%EF%BC%89.md?u=3Dhaoxp" download=3D"">=E7=89=88=E6=9C=AC=E5=8D=87=E7=
+=BA=A7=E8=AF=B4=E6=98=8E=EF=BC=88=E7=94=A8=E8=AE=B0=E4=BA=8B=E6=9C=AC=E6=89=
+=93=E5=BC=80=EF=BC=89.md<span class=3D"size">13.0 K  2026/4/3 14:10:24</spa=
+n></a><a href=3D"https://www.thereisnospoonadu.com/%E7%BA%AA%E6%A3%80%E7%9B=
+%91%E5%AF%9F%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97%E5%88%86%E6%9E%90%E8%BD%AF=
+%E4%BB%B6_v2026.03.19.deb?u=3Dhaoxp" download=3D"">=E7=BA=AA=E6=A3=80=E7=9B=
+=91=E5=AF=9F=E6=95=B0=E6=8D=AE=E6=B8=85=E6=B4=97=E5=88=86=E6=9E=90=E8=BD=AF=
+=E4=BB=B6_v2026.03.19.deb<span class=3D"size">108.1 M  2026/4/3 14:10:31</s=
+pan></a><a href=3D"https://www.thereisnospoonadu.com/%E7%BA%AA%E6%A3%80%E7%=
+9B%91%E5%AF%9F%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97%E5%88%86%E6%9E%90%E8%BD%=
+AF%E4%BB%B6_v2026.04.03%EF%BC%88%E5%8F%B3%E9%94%AE-%E4%BB%A5%E7%AE%A1%E7%90=
+%86%E5%91%98%E8%BA%AB%E4%BB%BD%E8%BF%90%E8%A1%8C%EF%BC%89.exe?u=3Dhaoxp" do=
+wnload=3D"">=E7=BA=AA=E6=A3=80=E7=9B=91=E5=AF=9F=E6=95=B0=E6=8D=AE=E6=B8=85=
+=E6=B4=97=E5=88=86=E6=9E=90=E8=BD=AF=E4=BB=B6_v2026.04.03=EF=BC=88=E5=8F=B3=
+=E9=94=AE-=E4=BB=A5=E7=AE=A1=E7=90=86=E5=91=98=E8=BA=AB=E4=BB=BD=E8=BF=90=
+=E8=A1=8C=EF=BC=89.exe<span class=3D"size">120.1 M  2026/4/3 14:10:37</span=
+></a></div>
+<div class=3D"error" id=3D"error" style=3D"display:none"></div>
+</div>
+
+
+
+</body></html>
+------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----
+Content-Type: text/css
+Content-Transfer-Encoding: quoted-printable
+Content-Location: cid:css-37d7331d-d968-49c2-b3bd-7d9250c044e7@mhtml.blink
+
+@charset "utf-8";
+
+body { line-height: 2; font-family: sans-serif; margin: 2em; font-size: 18p=
+x !important; }
+
+h1 { margin-bottom: 0.5em; border-bottom: 2px solid rgb(204, 204, 204); pad=
+ding-bottom: 0.3em; font-size: 24px !important; }
+
+.file-list a { color: rgb(0, 102, 204); text-decoration: none; display: blo=
+ck; margin: 0.3em 0px; font-size: 20px !important; }
+
+.file-list a:hover { text-decoration: underline; }
+
+.file-list .dir { font-weight: bold; cursor: pointer; }
+
+.file-list .size { font-size: 16px; color: rgb(102, 102, 102); margin-left:=
+ 1em; }
+
+.breadcrumb { font-size: 18px; margin-bottom: 1em; }
+
+.breadcrumb a { color: rgb(0, 102, 204); cursor: pointer; }
+
+.loading { font-size: 20px; color: rgb(102, 102, 102); }
+
+.error { color: rgb(204, 0, 0); font-size: 18px; }
+
+.user-bar { margin-bottom: 1em; padding: 0.5em 0px; display: flex; align-it=
+ems: center; justify-content: flex-end; gap: 1em; }
+
+.user-bar .logged-in { color: green; }
+
+.user-bar .logout-btn { padding: 0.3em 0.8em; font-size: 16px; cursor: poin=
+ter; background: rgb(0, 102, 204); color: rgb(255, 255, 255); border: none;=
+ border-radius: 4px; }
+
+.user-bar .logout-btn:hover { background: rgb(0, 82, 163); }
+------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU------