Compare commits
12 Commits
9ba8306b8d
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
 beastgitea2026
|
407b840b9e | ||
 betterbeast2023
|
51d73095b7 | ||
|
beastgitea2026
|
8d2e18c5a4 | ||
|
beastgitea2026
|
a3ebb28ce0 | ||
|
beastgitea2026
|
1183d10849 | ||
|
beastgitea2026
|
ebf38476a8 | ||
|
beastgitea2026
|
1dbfaebd3a | ||
|
beastgitea2026
|
51386e87e9 | ||
|
beastgitea2026
|
d9c6f79a36 | ||
|
beastgitea2026
|
f13e3c66a5 | ||
|
beastgitea2026
|
223000c4f1 | ||
|
beastgitea2026
|
4a43fbc895 |
+1404
File diff suppressed because one or more lines are too long
+2420
-158
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,581 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b9c2fa10",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# makemore — the MLP character model\n",
|
||||
"\n",
|
||||
"Predict the next letter from the **previous few letters** with a small neural net — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **02** of the series. Notebook `00` built backprop by hand; the bigram in `01`\n",
|
||||
"only ever looked at **one** character back. Here we take a real step up: a multi-layer\n",
|
||||
"perceptron (MLP) that looks at a small **window** of previous characters, following Bengio et\n",
|
||||
"al. 2003 and Karpathy's *\"makemore part 2: MLP\"* lecture.\n",
|
||||
"\n",
|
||||
"Dataset: `names.txt` — 32,033 real first names. Goal: **make more** name-like words."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "69f5f6bc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"A bigram only remembers the last letter, so it cannot tell that `bri` should probably be\n",
|
||||
"followed by `a` (Brianna) rather than a random vowel. We fix that by letting the model read a\n",
|
||||
"**block** of the last few letters at once.\n",
|
||||
"\n",
|
||||
"The trick that makes this work is the **embedding**: instead of treating each letter as an\n",
|
||||
"isolated symbol, we give every letter a short list of numbers (a vector) that the network\n",
|
||||
"learns. Letters that behave similarly drift close together.\n",
|
||||
"\n",
|
||||
"Real-life picture: imagine each letter is a person, and the embedding is their personality\n",
|
||||
"profile. The network learns these profiles so that \"similar\" letters (say, the vowels) end up\n",
|
||||
"with similar profiles, and it can generalise: a context it never saw can still be handled\n",
|
||||
"because it resembles ones it did see."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2d5dee6c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Imports\n",
|
||||
"\n",
|
||||
"We import **torch first** (before matplotlib) to avoid a Windows OpenMP DLL clash, and make\n",
|
||||
"plotting optional so the notebook runs even without matplotlib.\n",
|
||||
"\n",
|
||||
"**-> Training:** `torch` holds the weight tensors and does autograd; `F.cross_entropy` turns\n",
|
||||
"logits + the true next letter into the single `loss` number we minimise."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "07c035c6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\") # avoid duplicate-OpenMP crash on Windows\n",
|
||||
"\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import random\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import matplotlib.pyplot as plt\n",
|
||||
" HAS_PLT = True\n",
|
||||
"except Exception:\n",
|
||||
" HAS_PLT = False\n",
|
||||
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
|
||||
"\n",
|
||||
"print(\"PyTorch version:\", torch.__version__)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc58a8fc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.2 Load the names and build the alphabet\n",
|
||||
"\n",
|
||||
"Each name becomes letters plus a special **`.`** marker (id 0) that means *start* and *end*.\n",
|
||||
"So `emma` is really `. e m m a .` — the dots tell the model where words begin and stop."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6b7e8198",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"words = open(\"names.txt\").read().splitlines()\n",
|
||||
"print(\"number of names:\", len(words))\n",
|
||||
"print(\"examples:\", words[:8])\n",
|
||||
"print(\"shortest/longest:\", min(len(w) for w in words), \"/\", max(len(w) for w in words))\n",
|
||||
"\n",
|
||||
"chars = sorted(list(set(\"\".join(words))))\n",
|
||||
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
|
||||
"stoi[\".\"] = 0\n",
|
||||
"itos = {i: ch for ch, i in stoi.items()}\n",
|
||||
"vocab_size = len(itos)\n",
|
||||
"print(\"vocab_size:\", vocab_size)\n",
|
||||
"print(\"stoi:\", stoi)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d7a9d8ab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 The context window (`block_size`)\n",
|
||||
"\n",
|
||||
"`block_size` = how many previous letters the model sees before guessing the next one. With\n",
|
||||
"`block_size = 3`, predicting the letters of `emma` looks like this:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"... -> e (start: three dots, predict 'e')\n",
|
||||
"..e -> m\n",
|
||||
".em -> m\n",
|
||||
"emm -> a\n",
|
||||
"mma -> . (predict the end marker)\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Each arrow is one training example. Real-life picture: a sliding window of 3 letters moves\n",
|
||||
"left to right across the word; at each stop the model must guess what comes next."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4c0464a0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"block_size = 3\n",
|
||||
"\n",
|
||||
"def build_dataset(word_list):\n",
|
||||
" X, Y = [], []\n",
|
||||
" for w in word_list:\n",
|
||||
" context = [0] * block_size # start with '...'\n",
|
||||
" for ch in w + \".\":\n",
|
||||
" ix = stoi[ch]\n",
|
||||
" X.append(context)\n",
|
||||
" Y.append(ix)\n",
|
||||
" context = context[1:] + [ix] # slide the window\n",
|
||||
" return torch.tensor(X), torch.tensor(Y)\n",
|
||||
"\n",
|
||||
"# show the examples for one name\n",
|
||||
"print(\"examples generated from 'emma':\")\n",
|
||||
"Xtmp, Ytmp = build_dataset([\"emma\"])\n",
|
||||
"for ctx, target in zip(Xtmp.tolist(), Ytmp.tolist()):\n",
|
||||
" print(\"\".join(itos[i] for i in ctx), \"->\", itos[target])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5f607e40",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.2 Train / dev / test split\n",
|
||||
"\n",
|
||||
"We shuffle the names and cut three piles:\n",
|
||||
"\n",
|
||||
"- **train (80%)** — the model learns from these.\n",
|
||||
"- **dev (10%)** — we tune choices (size, learning rate) against these.\n",
|
||||
"- **test (10%)** — touched only at the very end, to get an honest score.\n",
|
||||
"\n",
|
||||
"Real-life picture: train = homework you study, dev = practice quiz you check yourself against,\n",
|
||||
"test = the final exam you only sit once."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5c8e4f0c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"random.seed(42)\n",
|
||||
"random.shuffle(words)\n",
|
||||
"n1 = int(0.8 * len(words))\n",
|
||||
"n2 = int(0.9 * len(words))\n",
|
||||
"\n",
|
||||
"Xtr, Ytr = build_dataset(words[:n1])\n",
|
||||
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
|
||||
"Xte, Yte = build_dataset(words[n2:])\n",
|
||||
"\n",
|
||||
"print(\"train:\", tuple(Xtr.shape), \"dev:\", tuple(Xdev.shape), \"test:\", tuple(Xte.shape))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b10d4c19",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 The model — embedding, hidden layer, output\n",
|
||||
"\n",
|
||||
"Three learnable pieces:\n",
|
||||
"\n",
|
||||
"1. **`C`** — the embedding table: each of the 27 symbols gets an `n_embd`-dim vector.\n",
|
||||
"2. **`W1, b1`** — a hidden layer with `tanh`: it mixes the 3 letter-vectors into `n_hidden`\n",
|
||||
" features. Analogy: a committee that reads the 3 personalities and forms opinions.\n",
|
||||
"3. **`W2, b2`** — the output layer: turns those opinions into `vocab_size` scores (logits) for\n",
|
||||
" the next letter.\n",
|
||||
"\n",
|
||||
"**-> Training:** `parameters` is the full list of knobs; every one gets a gradient and a nudge."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4a0bcc2f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g = torch.Generator().manual_seed(2147483647)\n",
|
||||
"\n",
|
||||
"n_embd = 10 # size of each letter's vector\n",
|
||||
"n_hidden = 200 # neurons in the hidden layer\n",
|
||||
"\n",
|
||||
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
|
||||
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * 0.2\n",
|
||||
"b1 = torch.randn(n_hidden, generator=g) * 0.01\n",
|
||||
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
|
||||
"b2 = torch.randn(vocab_size, generator=g) * 0.0\n",
|
||||
"\n",
|
||||
"parameters = [C, W1, b1, W2, b2]\n",
|
||||
"for p in parameters:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"\n",
|
||||
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "77778ae3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.2 One forward pass, step by step\n",
|
||||
"\n",
|
||||
"Watch the shapes flow. The key move is `emb.view(-1, n_embd*block_size)`: it **flattens** the\n",
|
||||
"3 letter-vectors of each example into one long row so the hidden layer can read them together.\n",
|
||||
"\n",
|
||||
"**-> Training:** `F.cross_entropy` = softmax + negative-log-likelihood in one numerically-stable\n",
|
||||
"call; lower means more probability on the true next letter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "13123079",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ix = torch.randint(0, Xtr.shape[0], (4,), generator=g) # 4 random examples\n",
|
||||
"Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
"emb = C[Xb] # (4, 3, n_embd)\n",
|
||||
"flat = emb.view(-1, n_embd * block_size) # (4, 30)\n",
|
||||
"h = torch.tanh(flat @ W1 + b1) # (4, n_hidden)\n",
|
||||
"logits = h @ W2 + b2 # (4, vocab_size)\n",
|
||||
"loss = F.cross_entropy(logits, Yb)\n",
|
||||
"\n",
|
||||
"print(\"emb :\", tuple(emb.shape))\n",
|
||||
"print(\"flat :\", tuple(flat.shape))\n",
|
||||
"print(\"h :\", tuple(h.shape))\n",
|
||||
"print(\"logits:\", tuple(logits.shape))\n",
|
||||
"print(\"loss :\", loss.item())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b90331ba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 Finding a good learning rate\n",
|
||||
"\n",
|
||||
"The learning rate (how big each nudge is) matters a lot: too small and training crawls, too\n",
|
||||
"big and the loss bounces or explodes. A cheap way to find a good one: sweep the rate from tiny\n",
|
||||
"to large over a few hundred steps and watch where the loss drops fastest.\n",
|
||||
"\n",
|
||||
"Real-life picture: testing how big a step to take downhill in fog — tiny steps are safe but\n",
|
||||
"slow; huge steps overshoot the valley.\n",
|
||||
"\n",
|
||||
"We use a **fresh** set of weights here so the search does not disturb the real model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "acae02ad",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# fresh throwaway weights just for the search\n",
|
||||
"gs = torch.Generator().manual_seed(2147483647)\n",
|
||||
"Cs = torch.randn((vocab_size, n_embd), generator=gs)\n",
|
||||
"W1s = torch.randn((n_embd * block_size, n_hidden), generator=gs) * 0.2\n",
|
||||
"b1s = torch.randn(n_hidden, generator=gs) * 0.01\n",
|
||||
"W2s = torch.randn((n_hidden, vocab_size), generator=gs) * 0.01\n",
|
||||
"b2s = torch.randn(vocab_size, generator=gs) * 0.0\n",
|
||||
"params_s = [Cs, W1s, b1s, W2s, b2s]\n",
|
||||
"for p in params_s:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"\n",
|
||||
"# candidate learning rates spaced from 10^-3 up to 10^0\n",
|
||||
"lre = torch.linspace(-3, 0, 1000)\n",
|
||||
"lrs = 10 ** lre\n",
|
||||
"lr_used, losses = [], []\n",
|
||||
"\n",
|
||||
"for i in range(1000):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=gs)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
" emb = Cs[Xb]\n",
|
||||
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1s + b1s)\n",
|
||||
" loss = F.cross_entropy(h @ W2s + b2s, Yb)\n",
|
||||
" for p in params_s:\n",
|
||||
" p.grad = None\n",
|
||||
" loss.backward()\n",
|
||||
" lr = lrs[i].item()\n",
|
||||
" for p in params_s:\n",
|
||||
" p.data += -lr * p.grad\n",
|
||||
" lr_used.append(lre[i].item())\n",
|
||||
" losses.append(loss.item())\n",
|
||||
"\n",
|
||||
"best = lre[min(range(len(losses)), key=lambda k: losses[k])].item()\n",
|
||||
"print(f\"loss dropped fastest around 10^{best:.2f} ~= {10**best:.3f}\")\n",
|
||||
"print(\"good practical choice: lr = 0.1\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" plt.figure(figsize=(5, 4))\n",
|
||||
" plt.plot(lr_used, losses)\n",
|
||||
" plt.xlabel(\"log10(learning rate)\")\n",
|
||||
" plt.ylabel(\"minibatch loss\")\n",
|
||||
" plt.title(\"learning-rate sweep\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13569252",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.1 Train the model\n",
|
||||
"\n",
|
||||
"Now the real run on the actual `parameters`. Each step:\n",
|
||||
"\n",
|
||||
"1. grab a random **minibatch** of 32 examples (fast, noisy, good enough)\n",
|
||||
"2. forward -> loss\n",
|
||||
"3. `loss.backward()` -> gradients\n",
|
||||
"4. nudge every parameter downhill\n",
|
||||
"\n",
|
||||
"We start at `lr = 0.1` and **decay** to `0.01` near the end to settle into the valley."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c5ec35d5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"max_steps = 20000\n",
|
||||
"batch_size = 32\n",
|
||||
"lossi = []\n",
|
||||
"\n",
|
||||
"for i in range(max_steps):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
" emb = C[Xb]\n",
|
||||
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" loss = F.cross_entropy(logits, Yb)\n",
|
||||
"\n",
|
||||
" for p in parameters:\n",
|
||||
" p.grad = None\n",
|
||||
" loss.backward()\n",
|
||||
"\n",
|
||||
" lr = 0.1 if i < 15000 else 0.01\n",
|
||||
" for p in parameters:\n",
|
||||
" p.data += -lr * p.grad\n",
|
||||
"\n",
|
||||
" lossi.append(loss.log10().item())\n",
|
||||
" if i % 4000 == 0 or i == max_steps - 1:\n",
|
||||
" print(f\"step {i:5d}/{max_steps} minibatch loss {loss.item():.4f}\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" plt.figure(figsize=(5, 4))\n",
|
||||
" plt.plot(lossi)\n",
|
||||
" plt.xlabel(\"step\")\n",
|
||||
" plt.ylabel(\"log10(loss)\")\n",
|
||||
" plt.title(\"training loss\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6baab73",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.2 Honest scores on train and dev\n",
|
||||
"\n",
|
||||
"Minibatch loss is noisy. Evaluate on the **whole** train and dev sets for a stable number.\n",
|
||||
"If dev loss is much worse than train, the model is memorising; if they are close, it is\n",
|
||||
"generalising. (A plain bigram on names scores around 2.45 — we should beat that.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "579fe1c8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@torch.no_grad()\n",
|
||||
"def split_loss(split):\n",
|
||||
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
|
||||
" emb = C[x]\n",
|
||||
" h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1 + b1)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" return F.cross_entropy(logits, y).item()\n",
|
||||
"\n",
|
||||
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
|
||||
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "054682f1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 5.1 Generate new names\n",
|
||||
"\n",
|
||||
"Sampling, letter by letter: start from `...`, predict a distribution, **draw** one letter,\n",
|
||||
"slide the window, repeat until the model emits the end marker `.`.\n",
|
||||
"\n",
|
||||
"These are brand-new names the model invented — not copied from the file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "06bf121c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
|
||||
"\n",
|
||||
"for _ in range(20):\n",
|
||||
" out = []\n",
|
||||
" context = [0] * block_size\n",
|
||||
" while True:\n",
|
||||
" emb = C[torch.tensor([context])]\n",
|
||||
" h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" probs = F.softmax(logits, dim=1)\n",
|
||||
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" if ix == 0:\n",
|
||||
" break\n",
|
||||
" out.append(itos[ix])\n",
|
||||
" print(\"\".join(out))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0a0de526",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 5** — Re-run section 5.1 a few times (different random draws) and read the\n",
|
||||
"names. Then try raising `n_hidden` (e.g. 300) or `block_size` (e.g. 4 or 5) back in section 2,\n",
|
||||
"retrain, and see whether dev loss improves and the names look more realistic."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bc2003e7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 6.1 Bonus — visualise letters in 2-D\n",
|
||||
"\n",
|
||||
"When the embedding is 2-D we can literally plot it. We quickly train a tiny `n_embd = 2`\n",
|
||||
"version and scatter the 27 symbols. Watch the **vowels** drift into their own neighbourhood —\n",
|
||||
"the network discovered that they play a similar role, without ever being told what a vowel is.\n",
|
||||
"\n",
|
||||
"(Optional: needs matplotlib. Smaller/shorter than the main model, so the clustering is rough.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7786f75a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g3 = torch.Generator().manual_seed(2147483647)\n",
|
||||
"C2 = torch.randn((vocab_size, 2), generator=g3)\n",
|
||||
"W1b = torch.randn((2 * block_size, n_hidden), generator=g3) * 0.2\n",
|
||||
"b1b = torch.randn(n_hidden, generator=g3) * 0.01\n",
|
||||
"W2b = torch.randn((n_hidden, vocab_size), generator=g3) * 0.01\n",
|
||||
"b2b = torch.randn(vocab_size, generator=g3) * 0.0\n",
|
||||
"params2 = [C2, W1b, b1b, W2b, b2b]\n",
|
||||
"for p in params2:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"\n",
|
||||
"for i in range(8000):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (32,), generator=g3)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
" emb = C2[Xb]\n",
|
||||
" h = torch.tanh(emb.view(-1, 2 * block_size) @ W1b + b1b)\n",
|
||||
" loss = F.cross_entropy(h @ W2b + b2b, Yb)\n",
|
||||
" for p in params2:\n",
|
||||
" p.grad = None\n",
|
||||
" loss.backward()\n",
|
||||
" for p in params2:\n",
|
||||
" p.data += -0.1 * p.grad\n",
|
||||
"\n",
|
||||
"print(\"2-D model dev-ish minibatch loss:\", round(loss.item(), 4))\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" pts = C2.detach()\n",
|
||||
" plt.figure(figsize=(6, 6))\n",
|
||||
" plt.scatter(pts[:, 0], pts[:, 1], s=300)\n",
|
||||
" for i in range(vocab_size):\n",
|
||||
" plt.text(pts[i, 0].item(), pts[i, 1].item(), itos[i],\n",
|
||||
" ha=\"center\", va=\"center\", color=\"white\")\n",
|
||||
" plt.title(\"learned 2-D letter embedding (look for the vowels clustering)\")\n",
|
||||
" plt.grid(True)\n",
|
||||
" plt.show()\n",
|
||||
"else:\n",
|
||||
" print(\"matplotlib missing - skipping the embedding scatter\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c5e5b7a7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What's next\n",
|
||||
"\n",
|
||||
"You went from \"one letter back\" (bigram) to a real neural net that reads a **window** of\n",
|
||||
"letters through a learned **embedding**, and it makes noticeably better names.\n",
|
||||
"\n",
|
||||
"Where this leads in the series:\n",
|
||||
"\n",
|
||||
"- `03_batchnorm_activations.ipynb` — why deep nets are fragile, and how BatchNorm keeps the\n",
|
||||
" signal healthy so we can train bigger models.\n",
|
||||
"- `04_backprop_ninja.ipynb` — compute all these gradients **by hand**, cementing notebook `00`.\n",
|
||||
"- `06_build_gpt_attention.ipynb` — replace the fixed window with **attention**, so the model\n",
|
||||
" can look back as far as it needs.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] Names -> context windows (`block_size`) -> (X, Y)\n",
|
||||
"- [ ] train / dev / test split\n",
|
||||
"- [ ] embedding `C` + hidden `tanh` + output logits\n",
|
||||
"- [ ] cross-entropy loss, minibatch training\n",
|
||||
"- [ ] learning-rate search\n",
|
||||
"- [ ] sampled new names; visualised the 2-D embedding"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,496 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "aedd6523",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# makemore — activations, gradients, and BatchNorm\n",
|
||||
"\n",
|
||||
"Make the MLP **trainable and healthy** — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **03**. In `02` the MLP worked, but we got a bit lucky with how the weights\n",
|
||||
"started. Here we look *inside* the network at the numbers flowing through it, see two common\n",
|
||||
"ways training goes wrong, and fix them — first by hand (good initialisation), then with the\n",
|
||||
"robust tool everyone uses: **Batch Normalization**. Follows Karpathy's *\"makemore part 3\"*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2980029a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"A neural net is a stack of layers. At each layer numbers flow forward (activations) and\n",
|
||||
"gradients flow backward. If those numbers get **too big** or **too small**, training stalls.\n",
|
||||
"\n",
|
||||
"Two failures we will see and fix:\n",
|
||||
"\n",
|
||||
"1. **Confidently wrong at the start.** If the output layer starts with large random weights,\n",
|
||||
" the model makes wild over-confident guesses, so the first loss is huge and the first many\n",
|
||||
" steps are wasted just calming it down.\n",
|
||||
"2. **Saturated `tanh`.** If the hidden numbers are too spread out, `tanh` slams them to -1 or\n",
|
||||
" +1. There the curve is flat, so its gradient is ~0 and those neurons stop learning — they\n",
|
||||
" are \"dead.\"\n",
|
||||
"\n",
|
||||
"Real-life picture: think of each layer as an audio stage. Too quiet and you hear nothing; too\n",
|
||||
"loud and everything **clips** into distortion. We want every stage at a healthy volume.\n",
|
||||
"**BatchNorm** is the auto-leveler that keeps the volume right at every stage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "18a296f3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Imports\n",
|
||||
"\n",
|
||||
"`torch` first (avoids a Windows OpenMP DLL clash), matplotlib optional for the diagnostics."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ac69da81",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
|
||||
"\n",
|
||||
"import math\n",
|
||||
"import random\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import matplotlib.pyplot as plt\n",
|
||||
" HAS_PLT = True\n",
|
||||
"except Exception:\n",
|
||||
" HAS_PLT = False\n",
|
||||
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
|
||||
"\n",
|
||||
"print(\"PyTorch version:\", torch.__version__)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "92143322",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.2 Data — same names, same context windows as notebook 02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d99c17df",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"words = open(\"names.txt\").read().splitlines()\n",
|
||||
"chars = sorted(list(set(\"\".join(words))))\n",
|
||||
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
|
||||
"stoi[\".\"] = 0\n",
|
||||
"itos = {i: ch for ch, i in stoi.items()}\n",
|
||||
"vocab_size = len(itos)\n",
|
||||
"block_size = 3\n",
|
||||
"\n",
|
||||
"def build_dataset(word_list):\n",
|
||||
" X, Y = [], []\n",
|
||||
" for w in word_list:\n",
|
||||
" context = [0] * block_size\n",
|
||||
" for ch in w + \".\":\n",
|
||||
" ix = stoi[ch]\n",
|
||||
" X.append(context)\n",
|
||||
" Y.append(ix)\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" return torch.tensor(X), torch.tensor(Y)\n",
|
||||
"\n",
|
||||
"random.seed(42)\n",
|
||||
"random.shuffle(words)\n",
|
||||
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
|
||||
"Xtr, Ytr = build_dataset(words[:n1])\n",
|
||||
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
|
||||
"Xte, Yte = build_dataset(words[n2:])\n",
|
||||
"print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "36b836bc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 Problem 1 — confidently wrong at the start\n",
|
||||
"\n",
|
||||
"At step 0 the model knows nothing, so the *best* it can do is spread probability evenly over\n",
|
||||
"all 27 symbols. That ideal first loss is `-ln(1/27) = ln(27) ~= 3.30`.\n",
|
||||
"\n",
|
||||
"But with large random output weights the logits are all over the place, so the first loss is\n",
|
||||
"much higher. Let's measure it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1219a887",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g0 = torch.Generator().manual_seed(2147483647)\n",
|
||||
"n_embd, n_hidden = 10, 200\n",
|
||||
"\n",
|
||||
"# deliberately naive (large) initialisation\n",
|
||||
"Cn = torch.randn((vocab_size, n_embd), generator=g0)\n",
|
||||
"W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
|
||||
"b1n = torch.randn(n_hidden, generator=g0)\n",
|
||||
"W2n = torch.randn((n_hidden, vocab_size), generator=g0)\n",
|
||||
"b2n = torch.randn(vocab_size, generator=g0)\n",
|
||||
"\n",
|
||||
"emb = Cn[Xtr[:32]]\n",
|
||||
"h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
|
||||
"logits = h @ W2n + b2n\n",
|
||||
"print(\"naive initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
|
||||
"print(\"ideal (ln 27) :\", math.log(27))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4c418501",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.2 Fix 1 — make the output layer humble at the start\n",
|
||||
"\n",
|
||||
"We want the starting logits near zero (no opinion yet). So scale `W2` **down** and set `b2`\n",
|
||||
"to zero. Now the first loss sits right near the ideal `~3.30`, and no steps are wasted."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e1b9bb4a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g0 = torch.Generator().manual_seed(2147483647)\n",
|
||||
"Cn = torch.randn((vocab_size, n_embd), generator=g0)\n",
|
||||
"W1n = torch.randn((n_embd * block_size, n_hidden), generator=g0)\n",
|
||||
"b1n = torch.randn(n_hidden, generator=g0)\n",
|
||||
"W2n = torch.randn((n_hidden, vocab_size), generator=g0) * 0.01 # tiny\n",
|
||||
"b2n = torch.randn(vocab_size, generator=g0) * 0.0 # zero\n",
|
||||
"\n",
|
||||
"emb = Cn[Xtr[:32]]\n",
|
||||
"h = torch.tanh(emb.view(-1, n_embd * block_size) @ W1n + b1n)\n",
|
||||
"logits = h @ W2n + b2n\n",
|
||||
"print(\"fixed initial loss :\", F.cross_entropy(logits, Ytr[:32]).item())\n",
|
||||
"print(\"ideal (ln 27) :\", math.log(27))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ab522f7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 Problem 2 — saturated `tanh` (dead neurons)\n",
|
||||
"\n",
|
||||
"Now look at the hidden activations `h = tanh(...)`. With large `W1`, the inputs to `tanh` are\n",
|
||||
"very spread out, so most outputs are pinned at -1 or +1. That is the flat part of `tanh`,\n",
|
||||
"where the gradient is ~0 — those neurons learn nothing.\n",
|
||||
"\n",
|
||||
"We measure the fraction of \"saturated\" activations (|h| > 0.99) and, if matplotlib is present,\n",
|
||||
"show the tell-tale picture: bright cells are saturated; a fully bright column = a dead neuron."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "8fd25db6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"emb = Cn[Xtr[:1000]]\n",
|
||||
"hpre = emb.view(-1, n_embd * block_size) @ W1n + b1n # pre-activation\n",
|
||||
"h = torch.tanh(hpre)\n",
|
||||
"sat = (h.abs() > 0.99).float().mean().item()\n",
|
||||
"print(f\"fraction of saturated activations: {sat:.1%} (high = bad)\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" plt.figure(figsize=(10, 3))\n",
|
||||
" plt.imshow(h.abs() > 0.99, cmap=\"gray\", aspect=\"auto\")\n",
|
||||
" plt.title(\"white = saturated (|tanh| > 0.99); a full white column = a dead neuron\")\n",
|
||||
" plt.xlabel(\"neuron\")\n",
|
||||
" plt.ylabel(\"example\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e11aee42",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.2 Fix 2 — scale the hidden layer (Kaiming init)\n",
|
||||
"\n",
|
||||
"The cure is to make `W1` smaller so the pre-activations have a sensible spread (roughly unit\n",
|
||||
"standard deviation). The principled recipe is **Kaiming init**: multiply random weights by\n",
|
||||
"`gain / sqrt(fan_in)`, where `fan_in` is the number of inputs and the `gain` for `tanh` is\n",
|
||||
"`5/3`.\n",
|
||||
"\n",
|
||||
"Real-life picture: setting the input volume so the signal fills the range without clipping."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "71f808d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fan_in = n_embd * block_size\n",
|
||||
"W1k = torch.randn((fan_in, n_hidden), generator=torch.Generator().manual_seed(1)) * ((5/3) / fan_in**0.5)\n",
|
||||
"b1k = torch.zeros(n_hidden)\n",
|
||||
"\n",
|
||||
"emb = Cn[Xtr[:1000]]\n",
|
||||
"h = torch.tanh(emb.view(-1, fan_in) @ W1k + b1k)\n",
|
||||
"sat = (h.abs() > 0.99).float().mean().item()\n",
|
||||
"print(f\"saturated now: {sat:.1%} (much lower)\")\n",
|
||||
"print(f\"activation std: {h.std().item():.3f} (we want it near ~0.6-0.7 for tanh)\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" plt.figure(figsize=(5, 3))\n",
|
||||
" plt.hist(h.view(-1).tolist(), bins=50)\n",
|
||||
" plt.title(\"healthy tanh activations (not all piled at +/-1)\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "50d6f683",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 BatchNorm — the auto-leveler\n",
|
||||
"\n",
|
||||
"Hand-tuning `W1` works, but in deep nets it is fiddly. **Batch Normalization** does it\n",
|
||||
"automatically: before `tanh`, it **normalises** the pre-activations so that, across the\n",
|
||||
"current batch, each neuron has mean 0 and std 1. Then it lets the network rescale/shift them\n",
|
||||
"with two learnable knobs, `gamma` (gain) and `beta` (bias), in case 0/1 is not ideal.\n",
|
||||
"\n",
|
||||
"Real-life picture: an audio compressor that auto-levels every stage to a healthy volume, with\n",
|
||||
"a manual trim (`gamma`, `beta`) if the engineer wants to nudge it.\n",
|
||||
"\n",
|
||||
"One catch: at test time we have no batch to average over, so during training we also keep a\n",
|
||||
"**running** mean/std to use later for single predictions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5aeb780e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g = torch.Generator().manual_seed(2147483647)\n",
|
||||
"n_embd, n_hidden = 10, 200\n",
|
||||
"fan_in = n_embd * block_size\n",
|
||||
"\n",
|
||||
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
|
||||
"W1 = torch.randn((fan_in, n_hidden), generator=g) * ((5/3) / fan_in**0.5)\n",
|
||||
"# no b1: BatchNorm's beta plays that role\n",
|
||||
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.01\n",
|
||||
"b2 = torch.randn(vocab_size, generator=g) * 0.0\n",
|
||||
"\n",
|
||||
"bngain = torch.ones((1, n_hidden))\n",
|
||||
"bnbias = torch.zeros((1, n_hidden))\n",
|
||||
"bnmean_running = torch.zeros((1, n_hidden))\n",
|
||||
"bnstd_running = torch.ones((1, n_hidden))\n",
|
||||
"\n",
|
||||
"parameters = [C, W1, W2, b2, bngain, bnbias]\n",
|
||||
"for p in parameters:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2b07d16a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.2 Train with BatchNorm\n",
|
||||
"\n",
|
||||
"Each step normalises the pre-activations using the **current batch**, updates the running\n",
|
||||
"stats (for later), then continues as usual. Watch the loss fall, starting near the ideal 3.3."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2410e7c4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"max_steps = 20000\n",
|
||||
"batch_size = 32\n",
|
||||
"lossi = []\n",
|
||||
"\n",
|
||||
"for i in range(max_steps):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
" emb = C[Xb]\n",
|
||||
" hpreact = emb.view(-1, fan_in) @ W1\n",
|
||||
"\n",
|
||||
" # --- BatchNorm ---\n",
|
||||
" bnmeani = hpreact.mean(0, keepdim=True)\n",
|
||||
" bnstdi = hpreact.std(0, keepdim=True)\n",
|
||||
" hpreact = bngain * (hpreact - bnmeani) / bnstdi + bnbias\n",
|
||||
" with torch.no_grad():\n",
|
||||
" bnmean_running = 0.999 * bnmean_running + 0.001 * bnmeani\n",
|
||||
" bnstd_running = 0.999 * bnstd_running + 0.001 * bnstdi\n",
|
||||
" # -----------------\n",
|
||||
"\n",
|
||||
" h = torch.tanh(hpreact)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" loss = F.cross_entropy(logits, Yb)\n",
|
||||
"\n",
|
||||
" for p in parameters:\n",
|
||||
" p.grad = None\n",
|
||||
" loss.backward()\n",
|
||||
" lr = 0.1 if i < 15000 else 0.01\n",
|
||||
" for p in parameters:\n",
|
||||
" p.data += -lr * p.grad\n",
|
||||
"\n",
|
||||
" lossi.append(loss.log10().item())\n",
|
||||
" if i % 4000 == 0 or i == max_steps - 1:\n",
|
||||
" print(f\"step {i:5d}/{max_steps} loss {loss.item():.4f}\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" plt.figure(figsize=(5, 4))\n",
|
||||
" plt.plot(lossi)\n",
|
||||
" plt.xlabel(\"step\"); plt.ylabel(\"log10(loss)\"); plt.title(\"training loss (with BatchNorm)\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d80dccf6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.3 Evaluate — using the running stats\n",
|
||||
"\n",
|
||||
"At evaluation we have no batch, so we normalise with the **running** mean/std collected during\n",
|
||||
"training. Train and dev loss should be close (good generalisation) and beat the bigram (~2.45)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "30ae8013",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@torch.no_grad()\n",
|
||||
"def split_loss(split):\n",
|
||||
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
|
||||
" emb = C[x]\n",
|
||||
" hpreact = emb.view(-1, fan_in) @ W1\n",
|
||||
" hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
|
||||
" h = torch.tanh(hpreact)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" return F.cross_entropy(logits, y).item()\n",
|
||||
"\n",
|
||||
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
|
||||
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "269c8943",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.1 Generate names with the BatchNorm model\n",
|
||||
"\n",
|
||||
"Same letter-by-letter sampling as before, but normalising with the running stats."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "63711cd1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
|
||||
"\n",
|
||||
"for _ in range(20):\n",
|
||||
" out = []\n",
|
||||
" context = [0] * block_size\n",
|
||||
" while True:\n",
|
||||
" emb = C[torch.tensor([context])]\n",
|
||||
" hpreact = emb.view(1, -1) @ W1\n",
|
||||
" hpreact = bngain * (hpreact - bnmean_running) / bnstd_running + bnbias\n",
|
||||
" h = torch.tanh(hpreact)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" probs = F.softmax(logits, dim=1)\n",
|
||||
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" if ix == 0:\n",
|
||||
" break\n",
|
||||
" out.append(itos[ix])\n",
|
||||
" print(\"\".join(out))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "365be7ef",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 4** — In section 2.1, change `W1n`'s scale (e.g. `* 0.1` vs `* 3.0`) and re-run\n",
|
||||
"the saturation check. Bigger scale -> more dead neurons. Then confirm BatchNorm (section 3.2)\n",
|
||||
"trains fine even from a less careful init, because it re-levels the signal every step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65b60eec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What's next\n",
|
||||
"\n",
|
||||
"You learned to look **inside** the network and keep it healthy:\n",
|
||||
"\n",
|
||||
"- start the output layer humble (good first loss)\n",
|
||||
"- keep `tanh` out of saturation (Kaiming init)\n",
|
||||
"- or just use **BatchNorm** to auto-level every layer\n",
|
||||
"\n",
|
||||
"Where this leads in the series:\n",
|
||||
"\n",
|
||||
"- `04_backprop_ninja.ipynb` — compute every gradient here **by hand**, including through\n",
|
||||
" BatchNorm, to truly own backprop.\n",
|
||||
"- `05_wavenet.ipynb` — stack these healthy layers into a deeper, hierarchical model.\n",
|
||||
"- `06_build_gpt_attention.ipynb` — the transformer, which uses LayerNorm (a cousin of\n",
|
||||
" BatchNorm) for the same \"keep the signal healthy\" reason.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] Ideal initial loss = ln(vocab_size)\n",
|
||||
"- [ ] Humble output init (small W2, zero b2)\n",
|
||||
"- [ ] Saturation / dead neurons diagnosed\n",
|
||||
"- [ ] Kaiming init (gain / sqrt(fan_in))\n",
|
||||
"- [ ] BatchNorm: per-batch normalise + gamma/beta + running stats\n",
|
||||
"- [ ] Trained, evaluated, sampled names"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,529 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6caa1908",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Becoming a backprop ninja\n",
|
||||
"\n",
|
||||
"Compute every gradient **by hand** — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **04**. So far we have leaned on `loss.backward()` to do the calculus for us.\n",
|
||||
"Here we turn it off and backpropagate **manually** through the whole MLP + BatchNorm +\n",
|
||||
"cross-entropy, checking each result against PyTorch. After this, autograd is never a mystery.\n",
|
||||
"Follows Karpathy's *\"makemore part 4: becoming a backprop ninja\"*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9ed68a60",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"`loss.backward()` answers one question for every weight: *if I nudge you a little, how does\n",
|
||||
"the loss change?* That answer is the gradient. We have trusted PyTorch to compute it.\n",
|
||||
"\n",
|
||||
"Now we earn it. We will:\n",
|
||||
"\n",
|
||||
"1. Run the forward pass but **save every intermediate value**.\n",
|
||||
"2. Walk backward, one operation at a time, applying the chain rule by hand (exactly the\n",
|
||||
" `00_micrograd.ipynb` idea, now on whole tensors).\n",
|
||||
"3. Compare each hand-derived gradient to PyTorch's — they should match.\n",
|
||||
"\n",
|
||||
"Real-life picture: notebook `00` showed one gear; here we backprop through the whole gearbox,\n",
|
||||
"and check our work against the manufacturer's spec (PyTorch)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "95e96905",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Imports and data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3b29d3f0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
|
||||
"\n",
|
||||
"import random\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"words = open(\"names.txt\").read().splitlines()\n",
|
||||
"chars = sorted(list(set(\"\".join(words))))\n",
|
||||
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
|
||||
"stoi[\".\"] = 0\n",
|
||||
"itos = {i: ch for ch, i in stoi.items()}\n",
|
||||
"vocab_size = len(itos)\n",
|
||||
"block_size = 3\n",
|
||||
"\n",
|
||||
"def build_dataset(word_list):\n",
|
||||
" X, Y = [], []\n",
|
||||
" for w in word_list:\n",
|
||||
" context = [0] * block_size\n",
|
||||
" for ch in w + \".\":\n",
|
||||
" ix = stoi[ch]\n",
|
||||
" X.append(context); Y.append(ix)\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" return torch.tensor(X), torch.tensor(Y)\n",
|
||||
"\n",
|
||||
"random.seed(42)\n",
|
||||
"random.shuffle(words)\n",
|
||||
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
|
||||
"Xtr, Ytr = build_dataset(words[:n1])\n",
|
||||
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
|
||||
"print(\"train:\", tuple(Xtr.shape), \" vocab:\", vocab_size)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "19612a96",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.2 The gradient-check helper\n",
|
||||
"\n",
|
||||
"`cmp` compares our hand-computed gradient `dt` to PyTorch's `t.grad`:\n",
|
||||
"\n",
|
||||
"- **exact** = bit-for-bit identical\n",
|
||||
"- **approx** = equal within tiny floating-point tolerance (this is the one that matters)\n",
|
||||
"- **maxdiff** = largest single difference"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2c6c9ccf",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def cmp(name, dt, t):\n",
|
||||
" ex = torch.all(dt == t.grad).item()\n",
|
||||
" app = torch.allclose(dt, t.grad, rtol=1e-4, atol=1e-7)\n",
|
||||
" maxdiff = (dt - t.grad).abs().max().item()\n",
|
||||
" print(f\"{name:12s} | exact: {str(ex):5s} | approx: {str(app):5s} | maxdiff: {maxdiff:.2e}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ed7b6dec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 Parameters\n",
|
||||
"\n",
|
||||
"We initialise the same MLP + BatchNorm as notebook `03`. Biases are made small but non-zero\n",
|
||||
"so every gradient path is actually exercised (a zero would hide bugs)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "70e88d72",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n_embd, n_hidden = 10, 64\n",
|
||||
"g = torch.Generator().manual_seed(2147483647)\n",
|
||||
"\n",
|
||||
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
|
||||
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
|
||||
"b1 = torch.randn(n_hidden, generator=g) * 0.1\n",
|
||||
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.1\n",
|
||||
"b2 = torch.randn(vocab_size, generator=g) * 0.1\n",
|
||||
"bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
|
||||
"bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
|
||||
"\n",
|
||||
"parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
|
||||
"for p in parameters:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "04fa6a1e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.2 Forward pass, broken into tiny steps\n",
|
||||
"\n",
|
||||
"Cross-entropy is written out by hand (max-subtract, exp, sum, log) so we can backprop through\n",
|
||||
"each piece. Every intermediate calls `retain_grad()` so PyTorch will store its gradient for us\n",
|
||||
"to compare against."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6ad34b3b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 32 # fixed batch size for the manual math\n",
|
||||
"ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
|
||||
"Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
"emb = C[Xb] # embed the characters\n",
|
||||
"embcat = emb.view(emb.shape[0], -1) # flatten\n",
|
||||
"# linear 1\n",
|
||||
"hprebn = embcat @ W1 + b1\n",
|
||||
"# batchnorm\n",
|
||||
"bnmeani = 1 / n * hprebn.sum(0, keepdim=True)\n",
|
||||
"bndiff = hprebn - bnmeani\n",
|
||||
"bndiff2 = bndiff ** 2\n",
|
||||
"bnvar = 1 / (n - 1) * bndiff2.sum(0, keepdim=True) # Bessel's correction (n-1)\n",
|
||||
"bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
|
||||
"bnraw = bndiff * bnvar_inv\n",
|
||||
"hpreact = bngain * bnraw + bnbias\n",
|
||||
"# nonlinearity\n",
|
||||
"h = torch.tanh(hpreact)\n",
|
||||
"# linear 2\n",
|
||||
"logits = h @ W2 + b2\n",
|
||||
"# cross-entropy (manual)\n",
|
||||
"logit_maxes = logits.max(1, keepdim=True).values\n",
|
||||
"norm_logits = logits - logit_maxes\n",
|
||||
"counts = norm_logits.exp()\n",
|
||||
"counts_sum = counts.sum(1, keepdim=True)\n",
|
||||
"counts_sum_inv = counts_sum ** -1\n",
|
||||
"probs = counts * counts_sum_inv\n",
|
||||
"logprobs = probs.log()\n",
|
||||
"loss = -logprobs[range(n), Yb].mean()\n",
|
||||
"\n",
|
||||
"for t in [logprobs, probs, counts, counts_sum, counts_sum_inv, norm_logits, logit_maxes,\n",
|
||||
" logits, h, hpreact, bnraw, bnvar_inv, bnvar, bndiff2, bndiff, bnmeani, hprebn,\n",
|
||||
" embcat, emb]:\n",
|
||||
" t.retain_grad()\n",
|
||||
"for p in parameters:\n",
|
||||
" p.retain_grad()\n",
|
||||
"loss.backward()\n",
|
||||
"print(\"loss:\", loss.item())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6fcbe6af",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 Backprop through cross-entropy\n",
|
||||
"\n",
|
||||
"Walk backward from `loss`. For each line of the forward pass we write the local derivative and\n",
|
||||
"multiply by the gradient coming from above (the chain rule). Read each line as: *how does this\n",
|
||||
"intermediate affect the loss?*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c7bc68e2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dlogprobs = torch.zeros_like(logprobs)\n",
|
||||
"dlogprobs[range(n), Yb] = -1.0 / n # loss = -mean of the chosen logprobs\n",
|
||||
"dprobs = (1.0 / probs) * dlogprobs # logprobs = log(probs)\n",
|
||||
"dcounts_sum_inv = (counts * dprobs).sum(1, keepdim=True)\n",
|
||||
"dcounts = counts_sum_inv * dprobs # probs = counts * counts_sum_inv\n",
|
||||
"dcounts_sum = (-counts_sum ** -2) * dcounts_sum_inv\n",
|
||||
"dcounts += torch.ones_like(counts) * dcounts_sum\n",
|
||||
"dnorm_logits = counts * dcounts # counts = exp(norm_logits)\n",
|
||||
"dlogits = dnorm_logits.clone()\n",
|
||||
"dlogit_maxes = (-dnorm_logits).sum(1, keepdim=True)\n",
|
||||
"dlogits += F.one_hot(logits.max(1).indices, num_classes=logits.shape[1]) * dlogit_maxes\n",
|
||||
"\n",
|
||||
"cmp(\"logprobs\", dlogprobs, logprobs)\n",
|
||||
"cmp(\"probs\", dprobs, probs)\n",
|
||||
"cmp(\"counts_sum_inv\", dcounts_sum_inv, counts_sum_inv)\n",
|
||||
"cmp(\"counts_sum\", dcounts_sum, counts_sum)\n",
|
||||
"cmp(\"counts\", dcounts, counts)\n",
|
||||
"cmp(\"norm_logits\", dnorm_logits, norm_logits)\n",
|
||||
"cmp(\"logit_maxes\", dlogit_maxes, logit_maxes)\n",
|
||||
"cmp(\"logits\", dlogits, logits)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "51a6e8d7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.2 Backprop through the second linear layer and tanh\n",
|
||||
"\n",
|
||||
"`logits = h @ W2 + b2`. Matrix-multiply gradients follow the standard transpose rules; `tanh`\n",
|
||||
"contributes its local derivative `1 - tanh(x)^2`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "39f0e645",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dh = dlogits @ W2.T\n",
|
||||
"dW2 = h.T @ dlogits\n",
|
||||
"db2 = dlogits.sum(0)\n",
|
||||
"dhpreact = (1.0 - h ** 2) * dh # tanh local derivative\n",
|
||||
"\n",
|
||||
"cmp(\"h\", dh, h)\n",
|
||||
"cmp(\"W2\", dW2, W2)\n",
|
||||
"cmp(\"b2\", db2, b2)\n",
|
||||
"cmp(\"hpreact\", dhpreact, hpreact)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a7ecc492",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.3 Backprop through BatchNorm\n",
|
||||
"\n",
|
||||
"This is the famously fiddly part. We go through each step: the scale/shift, the normalisation,\n",
|
||||
"the variance, and the mean. Note how the **mean and variance both depend on every example**, so\n",
|
||||
"gradients fan out across the batch."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6f4f5d49",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
|
||||
"dbnbias = dhpreact.sum(0, keepdim=True)\n",
|
||||
"dbnraw = bngain * dhpreact\n",
|
||||
"dbndiff = bnvar_inv * dbnraw\n",
|
||||
"dbnvar_inv = (bndiff * dbnraw).sum(0, keepdim=True)\n",
|
||||
"dbnvar = (-0.5 * (bnvar + 1e-5) ** -1.5) * dbnvar_inv\n",
|
||||
"dbndiff2 = (1.0 / (n - 1)) * torch.ones_like(bndiff2) * dbnvar\n",
|
||||
"dbndiff += (2 * bndiff) * dbndiff2\n",
|
||||
"dhprebn = dbndiff.clone()\n",
|
||||
"dbnmeani = (-dbndiff).sum(0)\n",
|
||||
"dhprebn += 1.0 / n * (torch.ones_like(hprebn) * dbnmeani)\n",
|
||||
"\n",
|
||||
"cmp(\"bngain\", dbngain, bngain)\n",
|
||||
"cmp(\"bnbias\", dbnbias, bnbias)\n",
|
||||
"cmp(\"bnraw\", dbnraw, bnraw)\n",
|
||||
"cmp(\"bnvar_inv\", dbnvar_inv, bnvar_inv)\n",
|
||||
"cmp(\"bnvar\", dbnvar, bnvar)\n",
|
||||
"cmp(\"bndiff2\", dbndiff2, bndiff2)\n",
|
||||
"cmp(\"bndiff\", dbndiff, bndiff)\n",
|
||||
"cmp(\"bnmeani\", dbnmeani, bnmeani)\n",
|
||||
"cmp(\"hprebn\", dhprebn, hprebn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3be0bf67",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.4 Backprop through the first linear layer and the embedding\n",
|
||||
"\n",
|
||||
"`hprebn = embcat @ W1 + b1`, then un-flatten back to `emb`, then scatter each row's gradient to\n",
|
||||
"the right row of the embedding table `C` (an embedding lookup is just \"pick rows\", so its\n",
|
||||
"backward is \"add the gradient back into those rows\")."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b1317637",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dembcat = dhprebn @ W1.T\n",
|
||||
"dW1 = embcat.T @ dhprebn\n",
|
||||
"db1 = dhprebn.sum(0)\n",
|
||||
"demb = dembcat.view(emb.shape)\n",
|
||||
"\n",
|
||||
"dC = torch.zeros_like(C)\n",
|
||||
"for k in range(Xb.shape[0]):\n",
|
||||
" for j in range(Xb.shape[1]):\n",
|
||||
" dC[Xb[k, j]] += demb[k, j]\n",
|
||||
"\n",
|
||||
"cmp(\"embcat\", dembcat, embcat)\n",
|
||||
"cmp(\"W1\", dW1, W1)\n",
|
||||
"cmp(\"b1\", db1, b1)\n",
|
||||
"cmp(\"emb\", demb, emb)\n",
|
||||
"cmp(\"C\", dC, C)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3f6d3bff",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 The elegant shortcuts\n",
|
||||
"\n",
|
||||
"Doing it step-by-step proves we understand it. But two of those backward passes collapse into\n",
|
||||
"beautiful one-liners once you do the algebra:\n",
|
||||
"\n",
|
||||
"- **Cross-entropy:** `dlogits = softmax(logits)`, then subtract 1 from the true class, divide by n.\n",
|
||||
" Intuition: push down whatever probability the model gave, pull up the correct class.\n",
|
||||
"- **BatchNorm:** one expression for `dhprebn` (no need for all the intermediate d's).\n",
|
||||
"\n",
|
||||
"We verify both against PyTorch."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "91248dc5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# elegant cross-entropy backward\n",
|
||||
"dlogits_fast = F.softmax(logits, 1)\n",
|
||||
"dlogits_fast[range(n), Yb] -= 1\n",
|
||||
"dlogits_fast /= n\n",
|
||||
"print(\"dlogits | approx:\", torch.allclose(dlogits_fast, logits.grad, rtol=1e-4, atol=1e-7),\n",
|
||||
" \" | maxdiff:\", (dlogits_fast - logits.grad).abs().max().item())\n",
|
||||
"\n",
|
||||
"# elegant batchnorm backward\n",
|
||||
"dhprebn_fast = bngain * bnvar_inv / n * (\n",
|
||||
" n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
|
||||
")\n",
|
||||
"print(\"dhprebn | approx:\", torch.allclose(dhprebn_fast, hprebn.grad, rtol=1e-4, atol=1e-7),\n",
|
||||
" \" | maxdiff:\", (dhprebn_fast - hprebn.grad).abs().max().item())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "09834b6c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.2 Train with 100% hand-written gradients\n",
|
||||
"\n",
|
||||
"The real proof: turn off autograd and train using only our manual gradients (the elegant\n",
|
||||
"versions, plus the manual `W2/b2/W1/b1/C/bngain/bnbias`). If the loss falls, our calculus is\n",
|
||||
"correct."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9bfb8004",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# fresh params\n",
|
||||
"g = torch.Generator().manual_seed(2147483647)\n",
|
||||
"C = torch.randn((vocab_size, n_embd), generator=g)\n",
|
||||
"W1 = torch.randn((n_embd * block_size, n_hidden), generator=g) * (5/3) / ((n_embd * block_size) ** 0.5)\n",
|
||||
"b1 = torch.randn(n_hidden, generator=g) * 0.1\n",
|
||||
"W2 = torch.randn((n_hidden, vocab_size), generator=g) * 0.1\n",
|
||||
"b2 = torch.randn(vocab_size, generator=g) * 0.1\n",
|
||||
"bngain = torch.randn((1, n_hidden), generator=g) * 0.1 + 1.0\n",
|
||||
"bnbias = torch.randn((1, n_hidden), generator=g) * 0.1\n",
|
||||
"parameters = [C, W1, b1, W2, b2, bngain, bnbias]\n",
|
||||
"for p in parameters:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"\n",
|
||||
"n = 32\n",
|
||||
"for i in range(8000):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (n,), generator=g)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
" # forward (keep what the manual backward needs)\n",
|
||||
" emb = C[Xb]\n",
|
||||
" embcat = emb.view(emb.shape[0], -1)\n",
|
||||
" hprebn = embcat @ W1 + b1\n",
|
||||
" bnmeani = hprebn.mean(0, keepdim=True)\n",
|
||||
" bnvar = hprebn.var(0, keepdim=True, unbiased=True)\n",
|
||||
" bnvar_inv = (bnvar + 1e-5) ** -0.5\n",
|
||||
" bnraw = (hprebn - bnmeani) * bnvar_inv\n",
|
||||
" hpreact = bngain * bnraw + bnbias\n",
|
||||
" h = torch.tanh(hpreact)\n",
|
||||
" logits = h @ W2 + b2\n",
|
||||
" loss = F.cross_entropy(logits, Yb)\n",
|
||||
"\n",
|
||||
" # manual backward (no loss.backward())\n",
|
||||
" dlogits = F.softmax(logits, 1)\n",
|
||||
" dlogits[range(n), Yb] -= 1\n",
|
||||
" dlogits /= n\n",
|
||||
" dh = dlogits @ W2.T\n",
|
||||
" dW2 = h.T @ dlogits\n",
|
||||
" db2 = dlogits.sum(0)\n",
|
||||
" dhpreact = (1.0 - h ** 2) * dh\n",
|
||||
" dbngain = (bnraw * dhpreact).sum(0, keepdim=True)\n",
|
||||
" dbnbias = dhpreact.sum(0, keepdim=True)\n",
|
||||
" dhprebn = bngain * bnvar_inv / n * (\n",
|
||||
" n * dhpreact - dhpreact.sum(0) - n / (n - 1) * bnraw * (dhpreact * bnraw).sum(0)\n",
|
||||
" )\n",
|
||||
" dembcat = dhprebn @ W1.T\n",
|
||||
" dW1 = embcat.T @ dhprebn\n",
|
||||
" db1 = dhprebn.sum(0)\n",
|
||||
" demb = dembcat.view(emb.shape)\n",
|
||||
" dC = torch.zeros_like(C)\n",
|
||||
" dC.index_add_(0, Xb.view(-1), demb.view(-1, n_embd))\n",
|
||||
"\n",
|
||||
" grads = [dC, dW1, db1, dW2, db2, dbngain, dbnbias]\n",
|
||||
" lr = 0.1 if i < 6000 else 0.01\n",
|
||||
" with torch.no_grad():\n",
|
||||
" for p, gr in zip(parameters, grads):\n",
|
||||
" p += -lr * gr\n",
|
||||
"\n",
|
||||
" if i % 2000 == 0 or i == 7999:\n",
|
||||
" print(f\"step {i:5d} loss {loss.item():.4f}\")\n",
|
||||
"\n",
|
||||
"print(\"trained using only hand-written gradients.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "422e7804",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 4** — In section 2.3, deliberately introduce a bug (e.g. drop the\n",
|
||||
"`n/(n-1)` Bessel factor in `dbnvar`) and re-run. Watch `cmp` flip to `approx: False` and the\n",
|
||||
"`maxdiff` jump. Then fix it. This is exactly how you debug a real backward pass."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "36c4bc1d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What's next\n",
|
||||
"\n",
|
||||
"You backpropagated by hand through cross-entropy, a linear layer, `tanh`, BatchNorm, and an\n",
|
||||
"embedding, matched PyTorch exactly, and trained a net with zero autograd. Backprop is now\n",
|
||||
"yours.\n",
|
||||
"\n",
|
||||
"Where this leads in the series:\n",
|
||||
"\n",
|
||||
"- `05_wavenet.ipynb` — stack layers into a deeper, hierarchical model using `torch.nn`-style\n",
|
||||
" building blocks.\n",
|
||||
"- `06_build_gpt_attention.ipynb` — the transformer; `loss.backward()` there is doing exactly\n",
|
||||
" what you just did, across far more nodes.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] Forward pass saved every intermediate\n",
|
||||
"- [ ] Manual backward matched PyTorch (cross-entropy, linear, tanh)\n",
|
||||
"- [ ] Manual backward through BatchNorm matched\n",
|
||||
"- [ ] Embedding backward = scatter-add into rows\n",
|
||||
"- [ ] Elegant one-line cross-entropy and BatchNorm gradients\n",
|
||||
"- [ ] Trained a net with only hand-written gradients"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d24a5a8b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# makemore — a WaveNet\n",
|
||||
"\n",
|
||||
"Go **deeper and hierarchical** — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **05**. The MLP in `02`/`03` squashed the whole context into **one** layer.\n",
|
||||
"Here we widen the context to 8 letters and fuse them **gradually**, two at a time, in a\n",
|
||||
"tree — the structure behind DeepMind's WaveNet (2016). We also wrap everything in tidy\n",
|
||||
"`torch.nn`-style building blocks. Follows Karpathy's *\"makemore part 5: WaveNet\"*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "187db6ed",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"Reading 8 letters by mashing them into one layer throws away structure. WaveNet's idea: fuse\n",
|
||||
"information **slowly**.\n",
|
||||
"\n",
|
||||
"Real-life picture: a knockout tournament. 8 players (letters) play in pairs -> 4 winners ->\n",
|
||||
"play in pairs -> 2 -> 1 champion. Each round combines neighbours into a richer summary. Our\n",
|
||||
"network does the same: combine letters in pairs, then pairs-of-pairs, then again, until a\n",
|
||||
"single vector represents the whole context.\n",
|
||||
"\n",
|
||||
"We also build small reusable layers (`Linear`, `BatchNorm1d`, `Tanh`, `Embedding`,\n",
|
||||
"`FlattenConsecutive`, `Sequential`) that behave like PyTorch's own — so the model reads like a\n",
|
||||
"clean list of steps."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6bf209d0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Imports and data (context grows to 8)\n",
|
||||
"\n",
|
||||
"The only data change from earlier notebooks: `block_size = 8`, so the model sees up to 8\n",
|
||||
"previous letters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "88825091",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
|
||||
"\n",
|
||||
"import random\n",
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import matplotlib.pyplot as plt\n",
|
||||
" HAS_PLT = True\n",
|
||||
"except Exception:\n",
|
||||
" HAS_PLT = False\n",
|
||||
"\n",
|
||||
"words = open(\"names.txt\").read().splitlines()\n",
|
||||
"chars = sorted(list(set(\"\".join(words))))\n",
|
||||
"stoi = {ch: i + 1 for i, ch in enumerate(chars)}\n",
|
||||
"stoi[\".\"] = 0\n",
|
||||
"itos = {i: ch for ch, i in stoi.items()}\n",
|
||||
"vocab_size = len(itos)\n",
|
||||
"block_size = 8\n",
|
||||
"\n",
|
||||
"def build_dataset(word_list):\n",
|
||||
" X, Y = [], []\n",
|
||||
" for w in word_list:\n",
|
||||
" context = [0] * block_size\n",
|
||||
" for ch in w + \".\":\n",
|
||||
" ix = stoi[ch]\n",
|
||||
" X.append(context); Y.append(ix)\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" return torch.tensor(X), torch.tensor(Y)\n",
|
||||
"\n",
|
||||
"random.seed(42)\n",
|
||||
"random.shuffle(words)\n",
|
||||
"n1, n2 = int(0.8 * len(words)), int(0.9 * len(words))\n",
|
||||
"Xtr, Ytr = build_dataset(words[:n1])\n",
|
||||
"Xdev, Ydev = build_dataset(words[n1:n2])\n",
|
||||
"Xte, Yte = build_dataset(words[n2:])\n",
|
||||
"print(\"train:\", tuple(Xtr.shape), \" (context length =\", block_size, \")\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4c8200dd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 PyTorch-style building blocks\n",
|
||||
"\n",
|
||||
"Each layer is a tiny class with a `__call__` (forward) and `parameters()`. The interesting one\n",
|
||||
"is **`FlattenConsecutive(n)`**: it groups every `n` neighbouring positions together — that is\n",
|
||||
"the \"combine in pairs\" tournament step. `BatchNorm1d` is written to also handle the 3-D tensors\n",
|
||||
"that appear between fusion steps (a subtle but important detail)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7695c863",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Linear:\n",
|
||||
" def __init__(self, fan_in, fan_out, bias=True):\n",
|
||||
" self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in ** 0.5\n",
|
||||
" self.bias = torch.zeros(fan_out) if bias else None\n",
|
||||
" def __call__(self, x):\n",
|
||||
" self.out = x @ self.weight\n",
|
||||
" if self.bias is not None:\n",
|
||||
" self.out = self.out + self.bias\n",
|
||||
" return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return [self.weight] + ([] if self.bias is None else [self.bias])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class BatchNorm1d:\n",
|
||||
" def __init__(self, dim, eps=1e-5, momentum=0.1):\n",
|
||||
" self.eps = eps; self.momentum = momentum; self.training = True\n",
|
||||
" self.gamma = torch.ones(dim); self.beta = torch.zeros(dim)\n",
|
||||
" self.running_mean = torch.zeros(dim); self.running_var = torch.ones(dim)\n",
|
||||
" def __call__(self, x):\n",
|
||||
" if self.training:\n",
|
||||
" dim = 0 if x.ndim == 2 else (0, 1) # 3-D inputs: average over batch AND time\n",
|
||||
" xmean = x.mean(dim, keepdim=True)\n",
|
||||
" xvar = x.var(dim, keepdim=True)\n",
|
||||
" else:\n",
|
||||
" xmean = self.running_mean\n",
|
||||
" xvar = self.running_var\n",
|
||||
" xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
|
||||
" self.out = self.gamma * xhat + self.beta\n",
|
||||
" if self.training:\n",
|
||||
" with torch.no_grad():\n",
|
||||
" self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean\n",
|
||||
" self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar\n",
|
||||
" return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return [self.gamma, self.beta]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Tanh:\n",
|
||||
" def __call__(self, x):\n",
|
||||
" self.out = torch.tanh(x); return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return []\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Embedding:\n",
|
||||
" def __init__(self, num_embeddings, embedding_dim):\n",
|
||||
" self.weight = torch.randn((num_embeddings, embedding_dim), generator=g)\n",
|
||||
" def __call__(self, IX):\n",
|
||||
" self.out = self.weight[IX]; return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return [self.weight]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class FlattenConsecutive:\n",
|
||||
" def __init__(self, n):\n",
|
||||
" self.n = n\n",
|
||||
" def __call__(self, x):\n",
|
||||
" B, T, C = x.shape\n",
|
||||
" x = x.view(B, T // self.n, C * self.n) # fuse every n neighbours\n",
|
||||
" if x.shape[1] == 1:\n",
|
||||
" x = x.squeeze(1)\n",
|
||||
" self.out = x; return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return []\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Sequential:\n",
|
||||
" def __init__(self, layers):\n",
|
||||
" self.layers = layers\n",
|
||||
" def __call__(self, x):\n",
|
||||
" for layer in self.layers:\n",
|
||||
" x = layer(x)\n",
|
||||
" self.out = x; return self.out\n",
|
||||
" def parameters(self):\n",
|
||||
" return [p for layer in self.layers for p in layer.parameters()]\n",
|
||||
"\n",
|
||||
"print(\"layers defined\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ab8cd26d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.2 Build the hierarchical (WaveNet) model\n",
|
||||
"\n",
|
||||
"`block_size = 8` fuses in three rounds: 8 -> 4 -> 2 -> 1. Each round is\n",
|
||||
"`FlattenConsecutive(2) -> Linear -> BatchNorm1d -> Tanh`. Finally a `Linear` maps to the 27\n",
|
||||
"next-letter scores. We shrink the last layer's weights so the first loss starts near the ideal\n",
|
||||
"`ln(27)`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1dcd57ee",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g = torch.Generator().manual_seed(42)\n",
|
||||
"n_embd = 24\n",
|
||||
"n_hidden = 128\n",
|
||||
"\n",
|
||||
"model = Sequential([\n",
|
||||
" Embedding(vocab_size, n_embd),\n",
|
||||
" FlattenConsecutive(2), Linear(n_embd * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
|
||||
" FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
|
||||
" FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(),\n",
|
||||
" Linear(n_hidden, vocab_size),\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"with torch.no_grad():\n",
|
||||
" model.layers[-1].weight *= 0.1 # humble output\n",
|
||||
"\n",
|
||||
"parameters = model.parameters()\n",
|
||||
"for p in parameters:\n",
|
||||
" p.requires_grad = True\n",
|
||||
"print(\"total parameters:\", sum(p.nelement() for p in parameters))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dfe5653b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.3 Watch the tournament shrink the sequence\n",
|
||||
"\n",
|
||||
"Run one batch and print each layer's output shape. See the time dimension halve 8 -> 4 -> 2 -> 1\n",
|
||||
"as neighbours fuse, while the feature dimension does the work."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5a37517c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ix = torch.randint(0, Xtr.shape[0], (4,), generator=g)\n",
|
||||
"_ = model(Xtr[ix])\n",
|
||||
"for layer in model.layers:\n",
|
||||
" print(f\"{layer.__class__.__name__:18s} : {tuple(layer.out.shape)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "43949ea9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 Train\n",
|
||||
"\n",
|
||||
"Standard loop: minibatch -> forward -> `loss.backward()` -> nudge. (We use autograd again now;\n",
|
||||
"notebook `04` already proved we could do it by hand.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f0800ce0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"max_steps = 20000\n",
|
||||
"batch_size = 32\n",
|
||||
"lossi = []\n",
|
||||
"\n",
|
||||
"for i in range(max_steps):\n",
|
||||
" ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)\n",
|
||||
" Xb, Yb = Xtr[ix], Ytr[ix]\n",
|
||||
"\n",
|
||||
" logits = model(Xb)\n",
|
||||
" loss = F.cross_entropy(logits, Yb)\n",
|
||||
"\n",
|
||||
" for p in parameters:\n",
|
||||
" p.grad = None\n",
|
||||
" loss.backward()\n",
|
||||
" lr = 0.1 if i < 15000 else 0.01\n",
|
||||
" for p in parameters:\n",
|
||||
" p.data += -lr * p.grad\n",
|
||||
"\n",
|
||||
" lossi.append(loss.log10().item())\n",
|
||||
" if i % 4000 == 0 or i == max_steps - 1:\n",
|
||||
" print(f\"step {i:5d}/{max_steps} loss {loss.item():.4f}\")\n",
|
||||
"\n",
|
||||
"if HAS_PLT:\n",
|
||||
" # smooth the noisy per-step loss by averaging in chunks of 200\n",
|
||||
" lt = torch.tensor(lossi)\n",
|
||||
" plt.figure(figsize=(5, 4))\n",
|
||||
" plt.plot(lt.view(-1, 200).mean(1))\n",
|
||||
" plt.xlabel(\"step / 200\"); plt.ylabel(\"log10(loss)\"); plt.title(\"WaveNet training loss\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "053ba8a7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.2 Evaluate\n",
|
||||
"\n",
|
||||
"Switch BatchNorm layers to eval mode (use running stats), then score train and dev. The deeper\n",
|
||||
"hierarchical model should edge below the single-layer MLP's ~2.1."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3a5d6228",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for layer in model.layers:\n",
|
||||
" if isinstance(layer, BatchNorm1d):\n",
|
||||
" layer.training = False\n",
|
||||
"\n",
|
||||
"@torch.no_grad()\n",
|
||||
"def split_loss(split):\n",
|
||||
" x, y = {\"train\": (Xtr, Ytr), \"dev\": (Xdev, Ydev), \"test\": (Xte, Yte)}[split]\n",
|
||||
" logits = model(x)\n",
|
||||
" return F.cross_entropy(logits, y).item()\n",
|
||||
"\n",
|
||||
"print(\"train loss:\", round(split_loss(\"train\"), 4))\n",
|
||||
"print(\"dev loss:\", round(split_loss(\"dev\"), 4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b78eaf70",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 Generate names\n",
|
||||
"\n",
|
||||
"Sample letter by letter as before. With 8 letters of context and the hierarchical fusion, the\n",
|
||||
"names tend to look a little more name-like."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "33833066",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"g2 = torch.Generator().manual_seed(2147483647 + 10)\n",
|
||||
"\n",
|
||||
"for _ in range(20):\n",
|
||||
" out = []\n",
|
||||
" context = [0] * block_size\n",
|
||||
" while True:\n",
|
||||
" logits = model(torch.tensor([context]))\n",
|
||||
" probs = F.softmax(logits, dim=1)\n",
|
||||
" ix = torch.multinomial(probs, num_samples=1, generator=g2).item()\n",
|
||||
" context = context[1:] + [ix]\n",
|
||||
" if ix == 0:\n",
|
||||
" break\n",
|
||||
" out.append(itos[ix])\n",
|
||||
" print(\"\".join(out))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "31d52503",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 5** — The model is just a list. Add another fusion round (so it handles\n",
|
||||
"`block_size = 16`), or raise `n_hidden`, retrain, and compare dev loss. Because everything is a\n",
|
||||
"clean `Sequential`, experimenting is just editing the list."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "894cc92e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What's next\n",
|
||||
"\n",
|
||||
"You built a deeper, hierarchical model out of reusable PyTorch-style blocks — the same shape of\n",
|
||||
"code real libraries use.\n",
|
||||
"\n",
|
||||
"Where this leads in the series:\n",
|
||||
"\n",
|
||||
"- `06_build_gpt_attention.ipynb` — the transformer. Instead of a fixed tournament that always\n",
|
||||
" fuses neighbours, **attention** lets every position decide *which* earlier positions to listen\n",
|
||||
" to. That flexibility is what makes GPT powerful.\n",
|
||||
"- `07_gpt_tokenizer.ipynb` — how real models turn text into tokens in the first place.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] Context widened to 8 letters\n",
|
||||
"- [ ] PyTorch-style layers (Linear, BatchNorm1d, Tanh, Embedding, FlattenConsecutive, Sequential)\n",
|
||||
"- [ ] Hierarchical fusion 8 -> 4 -> 2 -> 1\n",
|
||||
"- [ ] BatchNorm1d handles 3-D tensors\n",
|
||||
"- [ ] Trained, evaluated, sampled names"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,480 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3d5936ca",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Let's build GPT — self-attention to a full transformer\n",
|
||||
"\n",
|
||||
"Give every token the power to **look back** — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **06**, the payoff. Notebook `01` built a bigram that only sees one character\n",
|
||||
"back. Here we add **self-attention** so each position can gather information from all earlier\n",
|
||||
"positions, then stack it into a real (small) **GPT** and train it on tiny Shakespeare.\n",
|
||||
"Follows Karpathy's *\"Let's build GPT from scratch\"*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8b6c4fb3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"The bigram's weakness: predicting the next letter from only the previous one. WaveNet (`05`)\n",
|
||||
"widened the view but fused neighbours in a fixed pattern. **Attention** is smarter: every\n",
|
||||
"token looks at all earlier tokens and decides, on its own, *which* ones matter right now.\n",
|
||||
"\n",
|
||||
"Real-life picture: you are reading a sentence and hit the word \"it.\" To know what \"it\" refers\n",
|
||||
"to, your eyes jump back to the relevant earlier noun — not to a fixed position, but to whichever\n",
|
||||
"word is relevant. Self-attention lets each token make that kind of targeted look-back.\n",
|
||||
"\n",
|
||||
"We build it up in small pieces: the averaging trick, then queries/keys/values, then multiple\n",
|
||||
"heads, then the full transformer block (attention + feed-forward + residual + LayerNorm)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "028dce18",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Setup and data (standalone)\n",
|
||||
"\n",
|
||||
"Self-contained: load `input.txt`, build the character vocabulary, and make `get_batch`. Uses\n",
|
||||
"GPU if available, else CPU. The model is intentionally **small** so it trains on a laptop CPU;\n",
|
||||
"the comments show how to scale it up if you have a GPU."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "36f4f6e6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
|
||||
"\n",
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"from torch.nn import functional as F\n",
|
||||
"\n",
|
||||
"torch.manual_seed(1337)\n",
|
||||
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
|
||||
"print(\"device:\", device, \"| torch\", torch.__version__)\n",
|
||||
"\n",
|
||||
"# --- hyperparameters (small for CPU; bump these up on a GPU) ---\n",
|
||||
"batch_size = 16\n",
|
||||
"block_size = 32 # max context length\n",
|
||||
"n_embd = 64\n",
|
||||
"n_head = 4\n",
|
||||
"n_layer = 3\n",
|
||||
"dropout = 0.0\n",
|
||||
"learning_rate = 1e-3\n",
|
||||
"max_iters = 3000\n",
|
||||
"eval_interval = 1000\n",
|
||||
"eval_iters = 50\n",
|
||||
"\n",
|
||||
"# --- data ---\n",
|
||||
"text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()\n",
|
||||
"chars = sorted(list(set(text)))\n",
|
||||
"vocab_size = len(chars)\n",
|
||||
"stoi = {ch: i for i, ch in enumerate(chars)}\n",
|
||||
"itos = {i: ch for i, ch in enumerate(chars)}\n",
|
||||
"encode = lambda s: [stoi[c] for c in s]\n",
|
||||
"decode = lambda l: \"\".join(itos[i] for i in l)\n",
|
||||
"\n",
|
||||
"data = torch.tensor(encode(text), dtype=torch.long)\n",
|
||||
"n = int(0.9 * len(data))\n",
|
||||
"train_data, val_data = data[:n], data[n:]\n",
|
||||
"\n",
|
||||
"def get_batch(split):\n",
|
||||
" d = train_data if split == \"train\" else val_data\n",
|
||||
" ix = torch.randint(len(d) - block_size, (batch_size,))\n",
|
||||
" x = torch.stack([d[i:i + block_size] for i in ix])\n",
|
||||
" y = torch.stack([d[i + 1:i + block_size + 1] for i in ix])\n",
|
||||
" return x.to(device), y.to(device)\n",
|
||||
"\n",
|
||||
"print(\"vocab_size:\", vocab_size, \"| chars in dataset:\", len(text))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d2b12520",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 The math trick: a token can average its past\n",
|
||||
"\n",
|
||||
"Before real attention, here is the key move in its simplest form: let each position become the\n",
|
||||
"**average of itself and all positions before it** (never the future — that would be cheating).\n",
|
||||
"We show three ways that give the *same* result, from slow to elegant."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "54c69615",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"torch.manual_seed(1337)\n",
|
||||
"B, T, C = 4, 8, 2\n",
|
||||
"x = torch.randn(B, T, C)\n",
|
||||
"\n",
|
||||
"# v1: explicit loop — average of all previous tokens\n",
|
||||
"xbow = torch.zeros((B, T, C))\n",
|
||||
"for b in range(B):\n",
|
||||
" for t in range(T):\n",
|
||||
" xbow[b, t] = x[b, :t + 1].mean(0)\n",
|
||||
"\n",
|
||||
"# v2: same thing as a matrix multiply with a lower-triangular average matrix\n",
|
||||
"wei = torch.tril(torch.ones(T, T))\n",
|
||||
"wei = wei / wei.sum(1, keepdim=True)\n",
|
||||
"xbow2 = wei @ x\n",
|
||||
"\n",
|
||||
"# v3: same again, but the weights come from softmax over a masked matrix\n",
|
||||
"tril = torch.tril(torch.ones(T, T))\n",
|
||||
"wei = torch.zeros((T, T)).masked_fill(tril == 0, float(\"-inf\"))\n",
|
||||
"wei = F.softmax(wei, dim=-1)\n",
|
||||
"xbow3 = wei @ x\n",
|
||||
"\n",
|
||||
"print(\"v1 == v2 :\", torch.allclose(xbow, xbow2, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow2).abs().max())\n",
|
||||
"print(\"v1 == v3 :\", torch.allclose(xbow, xbow3, atol=1e-6), \" (max diff %.1e - just float rounding)\" % (xbow - xbow3).abs().max())\n",
|
||||
"print(\"\\nthe softmax weight matrix (row t can only attend to columns <= t):\")\n",
|
||||
"print(wei)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "aa46672a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Why the `-inf` + softmax version matters: those weights do **not** have to be a flat\n",
|
||||
"average. If the numbers in `wei` are *learned*, each token can decide how much to listen to each\n",
|
||||
"earlier token. That is exactly what attention does next.\n",
|
||||
"\n",
|
||||
"Analogy: a meeting where you may only hear people who already spoke (the mask), and you choose\n",
|
||||
"how much weight to give each speaker (the learned weights)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b522ef98",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 A single self-attention head — query, key, value\n",
|
||||
"\n",
|
||||
"Each token emits three vectors:\n",
|
||||
"\n",
|
||||
"- **query**: \"what am I looking for?\"\n",
|
||||
"- **key**: \"what do I offer?\"\n",
|
||||
"- **value**: \"what will I actually pass on if chosen?\"\n",
|
||||
"\n",
|
||||
"A token's attention to another = how well its **query** matches the other's **key**\n",
|
||||
"(dot product). We scale by `1/sqrt(head_size)` to keep the numbers tame, mask the future, softmax\n",
|
||||
"into weights, then take the weighted sum of **values**."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9adfb9ea",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"torch.manual_seed(1337)\n",
|
||||
"B, T, C = 4, 8, 32\n",
|
||||
"x = torch.randn(B, T, C)\n",
|
||||
"head_size = 16\n",
|
||||
"\n",
|
||||
"key = nn.Linear(C, head_size, bias=False)\n",
|
||||
"query = nn.Linear(C, head_size, bias=False)\n",
|
||||
"value = nn.Linear(C, head_size, bias=False)\n",
|
||||
"\n",
|
||||
"k = key(x) # (B, T, head_size)\n",
|
||||
"q = query(x) # (B, T, head_size)\n",
|
||||
"wei = q @ k.transpose(-2, -1) * head_size ** -0.5 # (B, T, T) affinities\n",
|
||||
"\n",
|
||||
"tril = torch.tril(torch.ones(T, T))\n",
|
||||
"wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
|
||||
"wei = F.softmax(wei, dim=-1)\n",
|
||||
"\n",
|
||||
"v = value(x)\n",
|
||||
"out = wei @ v # (B, T, head_size)\n",
|
||||
"print(\"attention output shape:\", tuple(out.shape))\n",
|
||||
"print(\"\\nrow 0 of the learned attention weights (note they are NOT uniform):\")\n",
|
||||
"print(wei[0, -1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ff4eea4c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 The full GPT, assembled from clean modules\n",
|
||||
"\n",
|
||||
"Now we package it. The pieces:\n",
|
||||
"\n",
|
||||
"- **Head** — one self-attention head (above), with dropout and a causal mask buffer.\n",
|
||||
"- **MultiHeadAttention** — several heads in parallel, concatenated, then projected. Analogy:\n",
|
||||
" several readers each look back for a different kind of clue, then pool notes.\n",
|
||||
"- **FeedForward** — a small per-token MLP: \"time to think about what I gathered.\"\n",
|
||||
"- **Block** — attention + feed-forward, each wrapped with a **residual** connection and\n",
|
||||
" **LayerNorm**. Residual = keep the original and add the new insight (so deep stacks still\n",
|
||||
" train). LayerNorm = re-level each token's vector (the `03` lesson, per-token).\n",
|
||||
"- **GPTLanguageModel** — token + position embeddings -> a stack of Blocks -> final LayerNorm ->\n",
|
||||
" a linear head to vocab logits."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b0a4b08d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Head(nn.Module):\n",
|
||||
" def __init__(self, head_size):\n",
|
||||
" super().__init__()\n",
|
||||
" self.key = nn.Linear(n_embd, head_size, bias=False)\n",
|
||||
" self.query = nn.Linear(n_embd, head_size, bias=False)\n",
|
||||
" self.value = nn.Linear(n_embd, head_size, bias=False)\n",
|
||||
" self.register_buffer(\"tril\", torch.tril(torch.ones(block_size, block_size)))\n",
|
||||
" self.dropout = nn.Dropout(dropout)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" B, T, C = x.shape\n",
|
||||
" k = self.key(x)\n",
|
||||
" q = self.query(x)\n",
|
||||
" wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5\n",
|
||||
" wei = wei.masked_fill(self.tril[:T, :T] == 0, float(\"-inf\"))\n",
|
||||
" wei = F.softmax(wei, dim=-1)\n",
|
||||
" wei = self.dropout(wei)\n",
|
||||
" v = self.value(x)\n",
|
||||
" return wei @ v\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class MultiHeadAttention(nn.Module):\n",
|
||||
" def __init__(self, num_heads, head_size):\n",
|
||||
" super().__init__()\n",
|
||||
" self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
|
||||
" self.proj = nn.Linear(head_size * num_heads, n_embd)\n",
|
||||
" self.dropout = nn.Dropout(dropout)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
|
||||
" return self.dropout(self.proj(out))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class FeedForward(nn.Module):\n",
|
||||
" def __init__(self, n_embd):\n",
|
||||
" super().__init__()\n",
|
||||
" self.net = nn.Sequential(\n",
|
||||
" nn.Linear(n_embd, 4 * n_embd),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(4 * n_embd, n_embd),\n",
|
||||
" nn.Dropout(dropout),\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" return self.net(x)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class Block(nn.Module):\n",
|
||||
" def __init__(self, n_embd, n_head):\n",
|
||||
" super().__init__()\n",
|
||||
" head_size = n_embd // n_head\n",
|
||||
" self.sa = MultiHeadAttention(n_head, head_size)\n",
|
||||
" self.ffwd = FeedForward(n_embd)\n",
|
||||
" self.ln1 = nn.LayerNorm(n_embd)\n",
|
||||
" self.ln2 = nn.LayerNorm(n_embd)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = x + self.sa(self.ln1(x)) # residual around attention\n",
|
||||
" x = x + self.ffwd(self.ln2(x)) # residual around feed-forward\n",
|
||||
" return x\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class GPTLanguageModel(nn.Module):\n",
|
||||
" def __init__(self):\n",
|
||||
" super().__init__()\n",
|
||||
" self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
|
||||
" self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
|
||||
" self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)])\n",
|
||||
" self.ln_f = nn.LayerNorm(n_embd)\n",
|
||||
" self.lm_head = nn.Linear(n_embd, vocab_size)\n",
|
||||
"\n",
|
||||
" def forward(self, idx, targets=None):\n",
|
||||
" B, T = idx.shape\n",
|
||||
" tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
|
||||
" pos_emb = self.position_embedding_table(torch.arange(T, device=device))\n",
|
||||
" x = tok_emb + pos_emb\n",
|
||||
" x = self.blocks(x)\n",
|
||||
" x = self.ln_f(x)\n",
|
||||
" logits = self.lm_head(x) # (B,T,vocab)\n",
|
||||
" if targets is None:\n",
|
||||
" return logits, None\n",
|
||||
" B, T, Cc = logits.shape\n",
|
||||
" loss = F.cross_entropy(logits.view(B * T, Cc), targets.view(B * T))\n",
|
||||
" return logits, loss\n",
|
||||
"\n",
|
||||
" def generate(self, idx, max_new_tokens):\n",
|
||||
" for _ in range(max_new_tokens):\n",
|
||||
" idx_cond = idx[:, -block_size:] # never exceed the context window\n",
|
||||
" logits, _ = self(idx_cond)\n",
|
||||
" logits = logits[:, -1, :] # last position only\n",
|
||||
" probs = F.softmax(logits, dim=-1)\n",
|
||||
" idx_next = torch.multinomial(probs, num_samples=1)\n",
|
||||
" idx = torch.cat((idx, idx_next), dim=1)\n",
|
||||
" return idx\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"model = GPTLanguageModel().to(device)\n",
|
||||
"print(\"parameters (millions):\", sum(p.numel() for p in model.parameters()) / 1e6)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dbb781c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.2 Positional embeddings — why we need them\n",
|
||||
"\n",
|
||||
"Attention by itself is **order-blind**: it treats the context as a bag of tokens. But \"dog\n",
|
||||
"bites man\" and \"man bites dog\" must differ. So besides each token's meaning\n",
|
||||
"(`token_embedding_table`), we add a vector for its **position**\n",
|
||||
"(`position_embedding_table`). Now a token knows both *what* it is and *where* it is.\n",
|
||||
"\n",
|
||||
"(That addition already happens inside `forward`: `x = tok_emb + pos_emb`.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0385a0ef",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.1 Train the GPT\n",
|
||||
"\n",
|
||||
"Same training shape as always, with the AdamW optimizer. Watch train and val loss fall well\n",
|
||||
"below the bigram's ~2.5. (Small model + CPU, so this is a few minutes; scale up on a GPU for\n",
|
||||
"real Shakespeare.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "395e6922",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@torch.no_grad()\n",
|
||||
"def estimate_loss():\n",
|
||||
" out = {}\n",
|
||||
" model.eval()\n",
|
||||
" for split in [\"train\", \"val\"]:\n",
|
||||
" losses = torch.zeros(eval_iters)\n",
|
||||
" for k in range(eval_iters):\n",
|
||||
" X, Y = get_batch(split)\n",
|
||||
" _, loss = model(X, Y)\n",
|
||||
" losses[k] = loss.item()\n",
|
||||
" out[split] = losses.mean().item()\n",
|
||||
" model.train()\n",
|
||||
" return out\n",
|
||||
"\n",
|
||||
"optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
|
||||
"\n",
|
||||
"for it in range(max_iters):\n",
|
||||
" if it % eval_interval == 0 or it == max_iters - 1:\n",
|
||||
" losses = estimate_loss()\n",
|
||||
" print(f\"step {it:4d}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
|
||||
" xb, yb = get_batch(\"train\")\n",
|
||||
" _, loss = model(xb, yb)\n",
|
||||
" optimizer.zero_grad(set_to_none=True)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
"print(\"done training\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ae5a608",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 5.1 Generate Shakespeare-ish text\n",
|
||||
"\n",
|
||||
"Start from a single newline and let the model write. It will not be perfect Shakespeare (tiny\n",
|
||||
"model, short training), but compared to the bigram it now produces real-looking words, spacing,\n",
|
||||
"and line breaks — because each character can attend to the ones before it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "61db808c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
|
||||
"print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6b43261f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 6** — If you have a GPU, scale up (e.g. `n_embd=384`, `n_head=6`,\n",
|
||||
"`n_layer=6`, `block_size=256`, `dropout=0.2`, `max_iters=5000`) and retrain — the samples get\n",
|
||||
"dramatically better. On CPU, try just raising `n_layer` to 4 or `max_iters` to 5000 and compare\n",
|
||||
"the val loss."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1e1142ab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What's next — and the connection to ChatGPT\n",
|
||||
"\n",
|
||||
"You built a real transformer: token + position embeddings, multi-head causal self-attention,\n",
|
||||
"feed-forward layers, residuals, and LayerNorm — the exact architecture behind GPT.\n",
|
||||
"\n",
|
||||
"What you trained here is a **pretrained** model in miniature: it only learns to *continue text*\n",
|
||||
"(predict the next token) from a big pile of text. ChatGPT starts the same way, then adds steps\n",
|
||||
"this series does not cover:\n",
|
||||
"\n",
|
||||
"- **Supervised finetuning** on example conversations (teach it to answer, not just continue).\n",
|
||||
"- **Reinforcement learning from human feedback (RLHF)** to align responses with what people\n",
|
||||
" prefer.\n",
|
||||
"\n",
|
||||
"The remaining notebook:\n",
|
||||
"\n",
|
||||
"- `07_gpt_tokenizer.ipynb` — real models do not use one-character-per-token; they use **Byte\n",
|
||||
" Pair Encoding**. That is the final piece of the pipeline.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] The averaging trick (loop = tril matmul = masked softmax)\n",
|
||||
"- [ ] Query / key / value self-attention head\n",
|
||||
"- [ ] Multi-head attention + projection\n",
|
||||
"- [ ] Feed-forward, residuals, LayerNorm -> Block\n",
|
||||
"- [ ] Token + position embeddings -> full GPT\n",
|
||||
"- [ ] Trained below the bigram baseline; generated text"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,375 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "67a422d8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# The GPT tokenizer — Byte Pair Encoding from scratch\n",
|
||||
"\n",
|
||||
"Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.\n",
|
||||
"\n",
|
||||
"This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model\n",
|
||||
"either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**\n",
|
||||
"tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then\n",
|
||||
"see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's\n",
|
||||
"*\"Let's build the GPT Tokenizer\"*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2037ef52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prologue — in plain English\n",
|
||||
"\n",
|
||||
"A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece\n",
|
||||
"mapped to an id. Two extremes:\n",
|
||||
"\n",
|
||||
"- **One token per character** (what we did): tiny vocabulary, but sequences are very long and\n",
|
||||
" the model must relearn common spellings everywhere.\n",
|
||||
"- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.\n",
|
||||
"\n",
|
||||
"**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common\n",
|
||||
"adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single\n",
|
||||
"tokens; rare text still survives as smaller pieces.\n",
|
||||
"\n",
|
||||
"Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep\n",
|
||||
"writing \"t-h-e\", so you invent one squiggle for \"the.\" Then you notice \" a-n-d\" and give it a\n",
|
||||
"squiggle too. BPE does this automatically, keeping the most useful shorthands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "75d40ed6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)\n",
|
||||
"\n",
|
||||
"Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts\n",
|
||||
"with exactly 256 tokens — every possible byte. Plain English letters are one byte each;\n",
|
||||
"accented or non-Latin characters take several bytes (which is why they cost more tokens later)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1037d36e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sample = \"Hello world! tokenization is fun.\"\n",
|
||||
"the_bytes = sample.encode(\"utf-8\")\n",
|
||||
"print(\"text :\", sample)\n",
|
||||
"print(\"bytes :\", list(the_bytes))\n",
|
||||
"print(\"length :\", len(the_bytes), \"bytes ->\", len(the_bytes), \"starting tokens\")\n",
|
||||
"\n",
|
||||
"multi = \"café naïve\" # accented letters take >1 byte each\n",
|
||||
"print(\"\\nnon-ASCII example:\", multi)\n",
|
||||
"print(\"bytes:\", list(multi.encode(\"utf-8\")), \"(note: more bytes than characters)\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5db66687",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.1 Two tiny helpers: count pairs, and merge a pair\n",
|
||||
"\n",
|
||||
"- `get_stats` counts how often each adjacent pair appears.\n",
|
||||
"- `merge` replaces every occurrence of a chosen pair with a single new token id.\n",
|
||||
"\n",
|
||||
"That is the entire mechanism of BPE."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6bc93ad8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_stats(ids):\n",
|
||||
" counts = {}\n",
|
||||
" for pair in zip(ids, ids[1:]):\n",
|
||||
" counts[pair] = counts.get(pair, 0) + 1\n",
|
||||
" return counts\n",
|
||||
"\n",
|
||||
"def merge(ids, pair, idx):\n",
|
||||
" new_ids = []\n",
|
||||
" i = 0\n",
|
||||
" while i < len(ids):\n",
|
||||
" if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:\n",
|
||||
" new_ids.append(idx)\n",
|
||||
" i += 2\n",
|
||||
" else:\n",
|
||||
" new_ids.append(ids[i])\n",
|
||||
" i += 1\n",
|
||||
" return new_ids\n",
|
||||
"\n",
|
||||
"# demo on a toy list: merge the most common pair\n",
|
||||
"demo = [1, 2, 3, 1, 2, 3, 1, 2]\n",
|
||||
"st = get_stats(demo)\n",
|
||||
"top = max(st, key=st.get)\n",
|
||||
"print(\"counts:\", st)\n",
|
||||
"print(\"most common pair:\", top, \"->\", st[top], \"times\")\n",
|
||||
"print(\"after merging\", top, \"into 99:\", merge(demo, top, 99))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cfb961e0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 2.2 Train BPE — learn the merges\n",
|
||||
"\n",
|
||||
"We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find\n",
|
||||
"the most common pair and merge it, recording each merge. After `num_merges` rounds we have a\n",
|
||||
"vocabulary of `256 + num_merges` tokens."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f62fbf5d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()[:20000]\n",
|
||||
"ids = list(train_text.encode(\"utf-8\"))\n",
|
||||
"print(\"training bytes:\", len(ids))\n",
|
||||
"\n",
|
||||
"num_merges = 60\n",
|
||||
"merges = {} # (a, b) -> new_id\n",
|
||||
"work = list(ids)\n",
|
||||
"for i in range(num_merges):\n",
|
||||
" stats = get_stats(work)\n",
|
||||
" pair = max(stats, key=stats.get)\n",
|
||||
" idx = 256 + i\n",
|
||||
" work = merge(work, pair, idx)\n",
|
||||
" merges[pair] = idx\n",
|
||||
"\n",
|
||||
"print(\"merges learned:\", len(merges))\n",
|
||||
"print(\"tokens after training:\", len(work), \"(was\", len(ids), \"bytes)\")\n",
|
||||
"print(f\"compression: {len(ids) / len(work):.2f}x\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "34b2bc27",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.1 encode and decode\n",
|
||||
"\n",
|
||||
"- **decode**: turn token ids back into text. We build each token's byte string by stitching the\n",
|
||||
" merges back together.\n",
|
||||
"- **encode**: turn new text into token ids by applying the learned merges, always doing the\n",
|
||||
" *earliest-learned* applicable merge first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c4f1dcab",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# build the id -> bytes table\n",
|
||||
"vocab = {idx: bytes([idx]) for idx in range(256)}\n",
|
||||
"for (p0, p1), idx in merges.items():\n",
|
||||
" vocab[idx] = vocab[p0] + vocab[p1]\n",
|
||||
"\n",
|
||||
"def decode(ids):\n",
|
||||
" data = b\"\".join(vocab[idx] for idx in ids)\n",
|
||||
" return data.decode(\"utf-8\", errors=\"replace\")\n",
|
||||
"\n",
|
||||
"def encode(text):\n",
|
||||
" tokens = list(text.encode(\"utf-8\"))\n",
|
||||
" while len(tokens) >= 2:\n",
|
||||
" stats = get_stats(tokens)\n",
|
||||
" # pick the pair whose merge was learned earliest\n",
|
||||
" pair = min(stats, key=lambda p: merges.get(p, float(\"inf\")))\n",
|
||||
" if pair not in merges:\n",
|
||||
" break\n",
|
||||
" tokens = merge(tokens, pair, merges[pair])\n",
|
||||
" return tokens\n",
|
||||
"\n",
|
||||
"trial = \"the king and the queen\"\n",
|
||||
"enc = encode(trial)\n",
|
||||
"print(\"text :\", trial)\n",
|
||||
"print(\"tokens:\", enc)\n",
|
||||
"print(\"count :\", len(enc), \"tokens for\", len(trial), \"characters\")\n",
|
||||
"print(\"decode round-trip ok:\", decode(enc) == trial)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a41ee40c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3.2 See what the learned tokens are\n",
|
||||
"\n",
|
||||
"The merges discovered the common chunks of English on their own. Let's print a few learned\n",
|
||||
"tokens (ids >= 256) as the text they stand for."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dadcc751",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"a few learned tokens:\")\n",
|
||||
"for idx in range(256, 256 + min(20, num_merges)):\n",
|
||||
" piece = vocab[idx].decode(\"utf-8\", errors=\"replace\")\n",
|
||||
" print(f\" id {idx}: {piece!r}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9193a4cd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.1 Regex pre-splitting (what real GPTs add)\n",
|
||||
"\n",
|
||||
"A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`\n",
|
||||
"into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,\n",
|
||||
"runs of spaces, punctuation), then runs BPE **inside** each chunk only.\n",
|
||||
"\n",
|
||||
"Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the\n",
|
||||
"standard `re` module to show the idea."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "96cf29fc",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import re\n",
|
||||
"\n",
|
||||
"# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})\n",
|
||||
"pat = re.compile(r\"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+\")\n",
|
||||
"\n",
|
||||
"demo_text = \"Hello, world! It's 2026 already.\"\n",
|
||||
"chunks = re.findall(pat, demo_text)\n",
|
||||
"print(\"split into chunks (BPE then runs inside each one):\")\n",
|
||||
"print(chunks)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e825ed0b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.2 Special tokens\n",
|
||||
"\n",
|
||||
"Real tokenizers also reserve **special tokens** that are not learned from text but inserted to\n",
|
||||
"mark structure, for example `<|endoftext|>` between documents, or chat markers like\n",
|
||||
"`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.\n",
|
||||
"\n",
|
||||
"Real-life picture: punctuation the model never \"spells\" — single reserved symbols that mean\n",
|
||||
"\"new document starts here\" or \"the user is speaking now.\"\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"<|endoftext|> -> id 50256 in GPT-2 (separates documents)\n",
|
||||
"<|im_start|> -> chat role marker in chat models\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "638d25d8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 5.1 Why tokenization causes LLM quirks\n",
|
||||
"\n",
|
||||
"Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.\n",
|
||||
"\n",
|
||||
"- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does\n",
|
||||
" not \"see\" its letters.\n",
|
||||
"- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.\n",
|
||||
"- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more\n",
|
||||
" tokens (and more money / context)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4f051e82",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def n_tokens(s):\n",
|
||||
" return len(encode(s))\n",
|
||||
"\n",
|
||||
"print(\"spelling: 'extraordinary' ->\", n_tokens(\"extraordinary\"), \"token(s)\")\n",
|
||||
"print(\" the model sees a chunk, not the 13 letters\\n\")\n",
|
||||
"\n",
|
||||
"print(\"numbers : '1234567' ->\", n_tokens(\"1234567\"), \"tokens (split oddly, not per-digit)\")\n",
|
||||
"print(\" '127 + 677' ->\", n_tokens(\"127 + 677\"), \"tokens\\n\")\n",
|
||||
"\n",
|
||||
"eng = \"hello how are you\"\n",
|
||||
"non = \"你好你好你好\" # non-English of similar visible length\n",
|
||||
"print(\"non-English costs more tokens for similar content:\")\n",
|
||||
"print(\" english :\", repr(eng), \"->\", n_tokens(eng), \"tokens\")\n",
|
||||
"print(\" non-latin :\", repr(non), \"->\", n_tokens(non), \"tokens\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cda06321",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the\n",
|
||||
"compression ratio and the token counts in section 5.1. More merges = better compression, but a\n",
|
||||
"bigger vocabulary. This trade-off is exactly what real tokenizer designers tune."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "35388d46",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Series wrap-up\n",
|
||||
"\n",
|
||||
"You built the whole stack, from the ground up:\n",
|
||||
"\n",
|
||||
"- `00_micrograd.ipynb` — backprop from scratch (the engine)\n",
|
||||
"- `01_build_gpt.ipynb` — the bigram baseline\n",
|
||||
"- `02_makemore_mlp.ipynb` — an MLP with embeddings\n",
|
||||
"- `03_batchnorm_activations.ipynb` — keeping deep nets healthy\n",
|
||||
"- `04_backprop_ninja.ipynb` — gradients by hand\n",
|
||||
"- `05_wavenet.ipynb` — a deeper, hierarchical model\n",
|
||||
"- `06_build_gpt_attention.ipynb` — the transformer (GPT)\n",
|
||||
"- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)\n",
|
||||
"\n",
|
||||
"Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,\n",
|
||||
"with runnable code at every step.\n",
|
||||
"\n",
|
||||
"**Checklist**\n",
|
||||
"- [ ] Text -> UTF-8 bytes (256 base tokens)\n",
|
||||
"- [ ] `get_stats` + `merge` = the BPE mechanism\n",
|
||||
"- [ ] Trained merges; measured compression\n",
|
||||
"- [ ] encode / decode round-trips\n",
|
||||
"- [ ] Regex pre-splitting and special tokens\n",
|
||||
"- [ ] Explained tokenization-caused LLM quirks"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv (3.12.10)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.12.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import json, sys, time
|
||||
|
||||
path = sys.argv[1]
|
||||
nb = json.load(open(path, encoding="utf-8"))
|
||||
g = {"__name__": "__main__"}
|
||||
|
||||
n = 0
|
||||
ok = True
|
||||
for c in nb["cells"]:
|
||||
if c["cell_type"] != "code":
|
||||
continue
|
||||
n += 1
|
||||
src = "".join(c["source"])
|
||||
t0 = time.time()
|
||||
try:
|
||||
exec(src, g)
|
||||
if "plt" in g and getattr(g["plt"], "show", None):
|
||||
g["plt"].show = lambda *a, **k: None
|
||||
except Exception as e:
|
||||
print("CELL", n, "FAILED:", type(e).__name__, e)
|
||||
ok = False
|
||||
break
|
||||
print(f" (cell {n} ok, {time.time()-t0:.1f}s)")
|
||||
if ok:
|
||||
print("ALL", n, "CODE CELLS RAN OK")
|
||||
@@ -0,0 +1,475 @@
|
||||
# Kylin Remote Control & Monitoring Setup
|
||||
|
||||
Complete reference for controlling and monitoring a Kylin Linux machine from a
|
||||
Windows 11 PC over a local phone-hotspot network. No VPN is used for the SSH
|
||||
connection itself — only the local Wi-Fi/hotspot.
|
||||
|
||||
---
|
||||
|
||||
## 0. Bootstrap — START HERE in a new session
|
||||
|
||||
**The connection is already fully set up** (SSH key login + passwordless sudo).
|
||||
A new session does **NOT** need to repeat the setup in §2–§3 — those sections are
|
||||
records of what was already done. Just connect and go.
|
||||
|
||||
**Verify everything with one command** (run from the Windows machine, e.g. Claude Code's Bash tool):
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 kylin 'echo SSH_OK; whoami; hostname; sudo -n true && echo SUDO_OK'
|
||||
```
|
||||
|
||||
Expected: `SSH_OK` / `caichl` / `caichl-HUAWEIQINGYUNL420` / `SUDO_OK`.
|
||||
|
||||
- ✅ **`SSH_OK` + `SUDO_OK`** → full control. Run anything (incl. `sudo`) via
|
||||
`ssh -o BatchMode=yes kylin "<command>"`. No password needed.
|
||||
- ❌ **asks for password / times out** → most likely the **hotspot IP changed**
|
||||
(`192.168.43.248` is DHCP-assigned and can change between sessions). Ask the user
|
||||
to re-check the Kylin IP (`hostname -I` on that machine), update
|
||||
`C:\Users\adusu\.ssh\config`, then re-verify. If it asks for a password, the key
|
||||
may have been removed — see §2.3 / §2.4 to re-establish.
|
||||
- ℹ️ SSH prints a harmless **post-quantum warning** to stderr — ignore it (the
|
||||
examples in this doc filter it out).
|
||||
|
||||
**Connect manually:** `ssh kylin` (alias in `C:\Users\adusu\.ssh\config` → `caichl@192.168.43.248`,
|
||||
with the pinned cipher/MAC already configured — see §2).
|
||||
|
||||
> A fresh Claude Code session also auto-loads memory pointers (`MEMORY.md` →
|
||||
> `kylin-remote-machine`, `kylin-command-logging`, `kylin-mihomo-vpn`) that link
|
||||
> back here. This file is the authoritative, self-contained reference.
|
||||
|
||||
---
|
||||
|
||||
## 1. The two machines
|
||||
|
||||
| Role | Details |
|
||||
|------|---------|
|
||||
| **Controller** | Windows 11 (`C:\Users\adusu`), where Claude Code + VPN run |
|
||||
| **Target** | Kylin V10 SP1, Huawei QINGYUN L420, **ARM64 / aarch64** |
|
||||
| **Network** | Shared phone hotspot, subnet `192.168.43.x` |
|
||||
| **Target IP** | `192.168.43.248` |
|
||||
| **Target user** | `caichl` |
|
||||
|
||||
> **Key idea:** Claude Code needs the VPN to reach Anthropic's servers, but the
|
||||
> SSH hop from Windows → Kylin is **local LAN traffic** and needs no VPN. The two
|
||||
> connections are independent.
|
||||
|
||||
---
|
||||
|
||||
## 2. SSH connection
|
||||
|
||||
### 2.1 Required cipher/MAC (important)
|
||||
|
||||
The default cipher fails over this hotspot link with
|
||||
`Corrupted MAC on input`. The connection must pin a specific cipher and MAC:
|
||||
|
||||
```
|
||||
-c aes256-ctr -m hmac-sha2-256
|
||||
```
|
||||
|
||||
### 2.2 Windows SSH config
|
||||
|
||||
File: `C:\Users\adusu\.ssh\config`
|
||||
|
||||
```sshconfig
|
||||
Host kylin kyln
|
||||
HostName 192.168.43.248
|
||||
User caichl
|
||||
Ciphers aes256-ctr
|
||||
MACs hmac-sha2-256
|
||||
```
|
||||
|
||||
With this in place, simply run:
|
||||
|
||||
```powershell
|
||||
ssh kylin # or: ssh kyln (both aliases work)
|
||||
```
|
||||
|
||||
The equivalent explicit command (no config) is:
|
||||
|
||||
```powershell
|
||||
ssh -m hmac-sha2-256 -c aes256-ctr caichl@192.168.43.248
|
||||
```
|
||||
|
||||
### 2.3 Passwordless login (SSH key)
|
||||
|
||||
The Windows key `~/.ssh/id_ed25519.pub` was copied to the Kylin machine:
|
||||
|
||||
```powershell
|
||||
Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub | `
|
||||
ssh kylin "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
|
||||
```
|
||||
|
||||
Result: `ssh kylin` logs in with **no password**.
|
||||
|
||||
### 2.4 Passwordless sudo
|
||||
|
||||
So admin commands can run non-interactively from Windows:
|
||||
|
||||
```bash
|
||||
# Run once on the Kylin machine (asks for password the one time)
|
||||
sudo bash -c 'echo "caichl ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/caichl-nopasswd; \
|
||||
chmod 440 /etc/sudoers.d/caichl-nopasswd; \
|
||||
visudo -cf /etc/sudoers.d/caichl-nopasswd'
|
||||
```
|
||||
|
||||
> **Security note:** This makes `caichl` a passwordless-sudo account — acceptable
|
||||
> for a disposable machine only. To revert: `sudo rm /etc/sudoers.d/caichl-nopasswd`.
|
||||
|
||||
After this, from Windows you can run, e.g.:
|
||||
|
||||
```powershell
|
||||
ssh -o BatchMode=yes kylin "sudo systemctl status ssh"
|
||||
```
|
||||
|
||||
### 2.5 Account credentials (recovery only)
|
||||
|
||||
> ⚠️ **Plaintext password below — keep this file private; do NOT commit or share it.**
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| User | `caichl` |
|
||||
| Password | `Caicai@2026` |
|
||||
|
||||
**Normally not needed** — SSH login uses the key (§2.3) and sudo is passwordless
|
||||
(§2.4). Use this only to **recover** if the key auth or sudoers drop-in is ever lost
|
||||
(e.g. to log in interactively and re-copy the key, or to re-create
|
||||
`/etc/sudoers.d/caichl-nopasswd`). To change it: run `passwd` on the Kylin machine.
|
||||
|
||||
---
|
||||
|
||||
## 3. Initial SSH server setup (done on the Kylin machine)
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y openssh-server
|
||||
sudo systemctl enable --now ssh
|
||||
hostname -I # find the IP (192.168.43.248)
|
||||
whoami # confirm the username (caichl)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Monitoring
|
||||
|
||||
### 4.1 SSH logins & sessions (works out of the box)
|
||||
|
||||
```bash
|
||||
who # who is logged in + source
|
||||
w # sessions + current command
|
||||
last # login history with IPs
|
||||
lastb # FAILED login attempts (sudo)
|
||||
sudo grep "Accepted" /var/log/auth.log # successful SSH logins
|
||||
sudo grep "Failed" /var/log/auth.log # failed/suspicious attempts
|
||||
sudo tail -f /var/log/auth.log # live SSH activity
|
||||
```
|
||||
|
||||
### 4.2 Command logging — no reboot (WORKING)
|
||||
|
||||
Logs every command typed in an interactive shell, with user, tty, source IP,
|
||||
and the command text.
|
||||
|
||||
**`/etc/profile.d/cmdlog.sh`**
|
||||
```bash
|
||||
# Log interactive shell commands to syslog (facility local6) -> /var/log/cmdlog.log
|
||||
if [ -n "$BASH" ] && [ -n "$PS1" ]; then
|
||||
export PROMPT_COMMAND='logger -p local6.notice -t cmdlog "user=$USER tty=$(tty 2>/dev/null) from=$(who am i 2>/dev/null | sed -n "s/.*(\(.*\)).*/\1/p") cmd=$(history 1 | sed "s/^ *[0-9]* *//")"'
|
||||
fi
|
||||
```
|
||||
|
||||
**`/etc/rsyslog.d/30-cmdlog.conf`**
|
||||
```
|
||||
local6.* /var/log/cmdlog.log
|
||||
```
|
||||
|
||||
**Setup commands**
|
||||
```bash
|
||||
sudo touch /var/log/cmdlog.log
|
||||
sudo chown syslog:adm /var/log/cmdlog.log # MUST be syslog-owned (rsyslog drops privs)
|
||||
sudo chmod 640 /var/log/cmdlog.log
|
||||
sudo systemctl restart rsyslog
|
||||
```
|
||||
|
||||
**View the logs**
|
||||
```bash
|
||||
sudo cat /var/log/cmdlog.log # all logged commands
|
||||
sudo tail -f /var/log/cmdlog.log # live
|
||||
```
|
||||
|
||||
Example line:
|
||||
```
|
||||
Jun 11 10:54:22 caichl-HUAWEIQINGYUNL420 cmdlog: user=caichl tty=/dev/pts/5 from=192.168.43.180 cmd=echo hello
|
||||
```
|
||||
|
||||
> **Gotcha:** rsyslog runs as the `syslog` user. If `/var/log/cmdlog.log` is owned
|
||||
> by `root:root`, nothing is written. It must be `chown syslog:adm`.
|
||||
|
||||
### 4.3 System-wide auditing — auditd (QUEUED, UNVERIFIED)
|
||||
|
||||
`auditd` is installed and enabled, with a rule logging all `execve` for real
|
||||
users:
|
||||
|
||||
**`/etc/audit/rules.d/cmdlog.rules`**
|
||||
```
|
||||
-a exit,always -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
|
||||
```
|
||||
|
||||
It cannot run yet because the kernel booted with `audit=0`. This was changed to
|
||||
`audit=1`:
|
||||
|
||||
```bash
|
||||
sudo sed -i 's/audit=0/audit=1/' /etc/default/grub
|
||||
# /boot is read-only on this device — remount to rebuild grub
|
||||
sudo mount -o remount,rw /boot
|
||||
sudo update-grub
|
||||
sudo mount -o remount,ro /boot
|
||||
```
|
||||
|
||||
> **Caveat:** This device appears to boot via **Huawei firmware, not standard
|
||||
> GRUB** (running kernel `-27` is not the one in `grub.cfg` which lists `-11`, and
|
||||
> the kernel cmdline has Huawei-specific params). So the `audit=1` change **may
|
||||
> not take effect** on reboot. The no-reboot logging in §4.2 already covers
|
||||
> interactive command logging, so auditd is a bonus.
|
||||
|
||||
**Verify after any reboot**
|
||||
```bash
|
||||
cat /proc/sys/kernel/audit_enabled # want: 1
|
||||
sudo systemctl is-active auditd # want: active
|
||||
sudo ausearch -k cmdlog -i | tail -40 # human-readable command audit
|
||||
sudo aureport -x --summary # summary of executables run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Device-specific quirks (Huawei Kylin ARM64)
|
||||
|
||||
- **Architecture is ARM64 (aarch64).** Software must have an ARM64 build; x86-only
|
||||
proprietary apps will not install/run.
|
||||
- **`/boot` is a separate ext4 partition mounted read-only.** Remount rw to change
|
||||
GRUB, then back to ro (see §4.3).
|
||||
- **Boot likely controlled by Huawei firmware**, limiting effectiveness of GRUB
|
||||
cmdline edits.
|
||||
- **Hotspot link needs `aes256-ctr` / `hmac-sha2-256`** or SSH fails with
|
||||
"Corrupted MAC on input".
|
||||
|
||||
---
|
||||
|
||||
## 6. Quick reference
|
||||
|
||||
```powershell
|
||||
# Connect
|
||||
ssh kylin
|
||||
|
||||
# Run a one-off command from Windows
|
||||
ssh kylin "uptime"
|
||||
|
||||
# Run an admin command from Windows
|
||||
ssh kylin "sudo systemctl status ssh"
|
||||
|
||||
# See logged commands
|
||||
ssh kylin "sudo tail -50 /var/log/cmdlog.log"
|
||||
|
||||
# Watch live
|
||||
ssh kylin "sudo tail -f /var/log/cmdlog.log"
|
||||
|
||||
# SSH login history
|
||||
ssh kylin "last"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Cause / Fix |
|
||||
|---------|-------------|
|
||||
| `Corrupted MAC on input` | Use `-c aes256-ctr -m hmac-sha2-256` (already in config) |
|
||||
| `Connection timed out` | Both devices not on same hotspot, or AP/client isolation on the phone — disable isolation |
|
||||
| `ssh kyln` "could not resolve" | Alias spelling — config now accepts both `kylin` and `kyln` |
|
||||
| Still asks for password | SSH key not copied (§2.3) |
|
||||
| `sudo` asks for password | sudoers drop-in empty/missing (§2.4) |
|
||||
| `audit support not in kernel` | Kernel booted `audit=0`; needs reboot with `audit=1` (§4.3) |
|
||||
| `cmdlog.log` empty | File not owned by `syslog:adm`; `sudo chown syslog:adm /var/log/cmdlog.log` then restart rsyslog |
|
||||
| `grub.cfg: Read-only file system` | `/boot` is read-only; `sudo mount -o remount,rw /boot` first |
|
||||
| VPN GUI app won't run (`GLIBC_2.34 not found`) | Kylin has glibc 2.31; modern Flutter/Electron clients need 2.34+. Use the Go-based mihomo core instead (see §8) |
|
||||
|
||||
---
|
||||
|
||||
## 8. VPN / proxy: mihomo + web dashboard
|
||||
|
||||
**Why not FlClash / GUI clients:** Kylin V10 SP1 has **glibc 2.31**, but FlClash 0.8.93
|
||||
and similar modern GUI clients require **glibc 2.34+** and fail to run. Upgrading
|
||||
glibc would risk breaking the OS. mihomo is a **statically-linked Go binary** with
|
||||
no glibc dependency, so it runs fine — controlled via a browser dashboard.
|
||||
|
||||
### Current state
|
||||
> **VPN is currently OFF.** The `mihomo` service is **stopped** and **disabled**
|
||||
> (auto-start removed), so it stays off across reboots until re-enabled. Ports
|
||||
> `7890`/`9090` are closed. The full configuration below is intact and ready to
|
||||
> resume.
|
||||
|
||||
### Turn the VPN on / off (run on Kylin, or via `ssh kylin`)
|
||||
```bash
|
||||
# TURN ON (and restore auto-start on boot)
|
||||
sudo systemctl enable --now mihomo
|
||||
|
||||
# TURN OFF for this session only (comes back on next reboot)
|
||||
sudo systemctl stop mihomo
|
||||
|
||||
# TURN OFF and keep it off across reboots <-- current state
|
||||
sudo systemctl stop mihomo && sudo systemctl disable mihomo
|
||||
|
||||
# Check state
|
||||
systemctl is-active mihomo # active / inactive
|
||||
systemctl is-enabled mihomo # enabled / disabled
|
||||
```
|
||||
> Reminder: if you had set the Kylin **system proxy** to `127.0.0.1:7890`, switch
|
||||
> it back to "None" while the VPN is off, or apps will fail to reach the network.
|
||||
|
||||
### Desktop access
|
||||
A launcher **"Mihomo VPN Dashboard"** was added to the app menu (Network category)
|
||||
at `/usr/share/applications/mihomo-dashboard.desktop`; it opens
|
||||
`http://127.0.0.1:9090/ui/`. (Only works while the service is running.)
|
||||
|
||||
### Installed components
|
||||
| Item | Location / value |
|
||||
|------|------------------|
|
||||
| mihomo core | `/usr/local/bin/mihomo` (Clash.Meta v1.19.27, arm64) |
|
||||
| Config | `/etc/mihomo/config.yaml` |
|
||||
| Web dashboard (zashboard) | `/etc/mihomo/ui/` |
|
||||
| systemd service | `/etc/systemd/system/mihomo.service` (auto-starts on boot) |
|
||||
| Proxy port (HTTP+SOCKS) | `7890` |
|
||||
| Dashboard API | `0.0.0.0:9090` |
|
||||
| Dashboard secret | `428968aebf7c301565c6b6d3a11826c4` |
|
||||
|
||||
### Install steps (done; for reference / rebuild)
|
||||
```bash
|
||||
# binary (downloaded on Windows through VPN, scp'd over to dodge chicken-and-egg)
|
||||
gunzip mihomo-linux-arm64-vX.gz
|
||||
sudo install -m 755 mihomo-linux-arm64 /usr/local/bin/mihomo
|
||||
# dashboard
|
||||
sudo mkdir -p /etc/mihomo/ui && sudo unzip zashboard-dist.zip -d /etc/mihomo/ui
|
||||
# config.yaml: mixed-port 7890, external-controller 0.0.0.0:9090, secret, external-ui: ui,
|
||||
# proxy-providers (subscription URL), PROXY (select) + AUTO (url-test) groups
|
||||
sudo systemctl enable --now mihomo
|
||||
```
|
||||
> The subscription URL lives in `/etc/mihomo/config.yaml` under `proxy-providers:`.
|
||||
> `external-ui-name` must NOT be set (it makes mihomo look in a `ui/<name>` subdir
|
||||
> and auto-download); serve `/etc/mihomo/ui` directly.
|
||||
|
||||
### Open the dashboard
|
||||
- **From Windows browser:** `http://192.168.43.248:9090/ui/`
|
||||
- backend/API: `http://192.168.43.248:9090` • secret: `428968aebf7c301565c6b6d3a11826c4`
|
||||
- **From Kylin desktop browser:** `http://127.0.0.1:9090/ui/`
|
||||
|
||||
### Use the proxy
|
||||
Point apps at `127.0.0.1:7890` (HTTP/SOCKS), or set the Kylin system proxy
|
||||
(Settings → Network → Proxy → Manual → `127.0.0.1:7890`). For transparent
|
||||
whole-system routing, enable mihomo **TUN mode** (`/dev/net/tun` is present).
|
||||
|
||||
### Manage via API (examples)
|
||||
```bash
|
||||
S=428968aebf7c301565c6b6d3a11826c4
|
||||
# switch PROXY group to a node / AUTO
|
||||
curl -X PUT -H "Authorization: Bearer $S" http://127.0.0.1:9090/proxies/PROXY -d '{"name":"AUTO"}'
|
||||
# verify exit IP goes through the proxy
|
||||
curl -x http://127.0.0.1:7890 -s https://api.ip.sb/geoip
|
||||
# update subscription / reload
|
||||
sudo systemctl restart mihomo
|
||||
```
|
||||
|
||||
**Verified working:** exit IP resolved to Singapore and `google.com/generate_204`
|
||||
returned HTTP 204 through the proxy.
|
||||
|
||||
---
|
||||
|
||||
## 9. Browsers
|
||||
|
||||
| Browser | App-menu name | Type / location | Notes |
|
||||
|---------|---------------|-----------------|-------|
|
||||
| **Chromium** | "Chromium" | snap `chromium` v149 (`/snap/bin/chromium`) | Installed as the Chrome/Edge equivalent — same engine, Chrome Web Store extensions work. **Set as the default browser.** |
|
||||
|
||||
- **Chrome & Edge cannot be installed** — neither has an official ARM64 Linux build
|
||||
(same architecture wall as the VPN GUI clients). Chromium is the substitute.
|
||||
- Installing the chromium snap first required **updating snapd** (Kylin shipped 2.54,
|
||||
too old → error `assumes unsupported features: snapd2.55`). Fix:
|
||||
`sudo snap install snapd` (brought it to 2.75.2), then `sudo snap install chromium`.
|
||||
- Default browser is recorded in `~/.config/mimeapps.list`
|
||||
(`text/html`, `x-scheme-handler/http(s)` → `chromium_chromium.desktop`). If a
|
||||
double-click opens the wrong app, right-click the file → **Open with → Chromium →
|
||||
Set as default**.
|
||||
|
||||
### Desktop instruction files (for the user at the machine)
|
||||
- `~/桌面/VPN_Guide.html` + `~/桌面/VPN_Guide.md` (copies also in `~/Desktop/`):
|
||||
a user-facing "how to turn the VPN on/off" guide. Double-click the **`.html`** to
|
||||
read it rendered in a browser; the **`.md`** opens in **VS Code** (`code`, installed)
|
||||
with `Ctrl+Shift+V` preview.
|
||||
- App-menu launcher **"Mihomo VPN Dashboard"** → opens `http://127.0.0.1:9090/ui/`
|
||||
(only works while the mihomo service is running).
|
||||
|
||||
---
|
||||
|
||||
## 10. Inventory — everything set up on the Kylin machine
|
||||
|
||||
| Area | What | State |
|
||||
|------|------|-------|
|
||||
| Access | SSH alias `kylin`/`kyln`, key login, passwordless sudo | ✅ active |
|
||||
| Monitoring | SSH login logs (`auth.log`); command logging → `/var/log/cmdlog.log` | ✅ active |
|
||||
| Monitoring | auditd execve rule | ⏳ queued (needs reboot + `audit=1`; may not take on Huawei firmware) |
|
||||
| VPN | mihomo core + zashboard dashboard, subscription configured | ⛔ installed, **OFF** (stopped + disabled) |
|
||||
| Browser | Chromium v149 (default), 360/CNOOC browser | ✅ installed |
|
||||
| Editors | VS Code (`code`), pluma, WPS Office | ✅ pre-installed |
|
||||
| IDE | Cursor 3.7.27 (`/opt/cursor`) via `env -i` wrapper | ✅ installed (see §11) |
|
||||
| Desktop docs | `VPN_Guide.html` / `.md`, dashboard launcher | ✅ on desktop |
|
||||
|
||||
---
|
||||
|
||||
## 11. Cursor IDE (Electron) — install + crash fixes
|
||||
|
||||
Cursor (AI code editor, Electron-based) installed as an **extracted AppImage** at
|
||||
`/opt/cursor` (currently **v3.7.27**, ARM64). It will NOT run with a normal launcher
|
||||
on this Huawei/Kylin box — it needs a special wrapper.
|
||||
|
||||
### Why the special launcher (three Kylin/Huawei ARM64 problems)
|
||||
1. **EFAULT machine-ID probe crash** — Cursor runs a command to generate a unique
|
||||
machine ID; the Kylin kernel blocks that memory `write` as suspicious →
|
||||
`Error: EFAULT: bad address in system call argument, write` → instant crash.
|
||||
**Fix:** `env -i` strips the environment (especially D-Bus) so Cursor skips the probe.
|
||||
2. **Mali GPU crash (段错误 / segfault)** — hardware rendering conflicts with Huawei's
|
||||
Mali GPU driver (`dmesg`: `mali gpu: kctx ... create/destroyed` at crash time).
|
||||
**Fix:** `--disable-gpu` (software rendering).
|
||||
3. **Blocked sandbox** — `--no-sandbox`.
|
||||
Also installed `xdg-desktop-portal` + `xdg-desktop-portal-gtk` (1.6.0) for the
|
||||
Wayland file picker (necessary but not sufficient alone).
|
||||
|
||||
### The launcher (already installed)
|
||||
- **Wrapper:** `/usr/local/bin/cursor-launch`
|
||||
- **Start-menu entry:** `/usr/share/applications/cursor.desktop` → `Exec=/usr/local/bin/cursor-launch %F`
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
exec env -i \
|
||||
HOME="$HOME" DISPLAY="${DISPLAY:-:0}" PATH="$PATH" \
|
||||
/opt/cursor/usr/share/cursor/cursor --no-sandbox --disable-gpu "$@"
|
||||
```
|
||||
|
||||
### Updating Cursor to a newer version
|
||||
1. Get the latest ARM64 AppImage URL (downloads on Windows through the VPN):
|
||||
```bash
|
||||
curl -sL -A "Mozilla/5.0" -H "Accept: application/json" \
|
||||
"https://www.cursor.com/api/download?platform=linux-arm64&releaseTrack=stable"
|
||||
# -> JSON with "downloadUrl" (…/Cursor-X.Y.Z-aarch64.AppImage) and "version"
|
||||
curl -L -C - -o Cursor-new.AppImage "<downloadUrl>"
|
||||
```
|
||||
2. `scp` it to Kylin, then:
|
||||
```bash
|
||||
chmod +x Cursor-new.AppImage
|
||||
cd /tmp && ./Cursor-new.AppImage --appimage-extract # -> /tmp/squashfs-root
|
||||
sudo rm -rf /opt/cursor && sudo cp -r /tmp/squashfs-root /opt/cursor
|
||||
sudo chown -R root:root /opt/cursor && rm -rf /tmp/squashfs-root
|
||||
```
|
||||
3. **No change needed** to the wrapper or menu entry — paths stay the same.
|
||||
Verify glibc first (must need ≤ 2.31): `objdump -T /opt/cursor/usr/share/cursor/cursor | grep -oE 'GLIBC_[0-9.]+' | sort -V | tail`.
|
||||
|
||||
### Cleanup note
|
||||
Old AppImages in `~/software/` (`Cursor-3.2.11`, `Cursor-3.7.27`) and the user's
|
||||
manual extract `~/software/squashfs-root` (3.2.11) are redundant now that `/opt/cursor`
|
||||
is the install — safe to delete to reclaim space.
|
||||
@@ -1,149 +0,0 @@
|
||||
# Local AI Coding Setup — Windows 11 (Aider + Ollama)
|
||||
|
||||
## Hardware
|
||||
- GPU: RTX 5070 Ti, 12GB VRAM
|
||||
- RAM: 32GB
|
||||
- OS: Windows 11
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Install Ollama
|
||||
Download from https://ollama.com and install.
|
||||
|
||||
### 2. Pull a model
|
||||
```powershell
|
||||
ollama pull qwen3-coder:30b # 18GB, strong coding model (MoE, VRAM+RAM split)
|
||||
ollama pull gemma4:12b # 12GB, responds in English more reliably
|
||||
```
|
||||
|
||||
Confirm models are available:
|
||||
```powershell
|
||||
ollama list
|
||||
```
|
||||
|
||||
Confirm Ollama is serving:
|
||||
```powershell
|
||||
curl http://localhost:11434/api/tags
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Install Aider
|
||||
|
||||
```powershell
|
||||
pip install aider-chat
|
||||
aider --version # should print version number
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project Config
|
||||
|
||||
Create `.aider.conf.yml` in your project root:
|
||||
|
||||
```yaml
|
||||
model: ollama/gemma4:12b
|
||||
openai-api-base: http://localhost:11434/v1
|
||||
openai-api-key: ollama
|
||||
edit-format: whole
|
||||
map-tokens: 2048
|
||||
no-auto-commits: true
|
||||
system-prompt: "Always respond in English only. Never use Chinese or any other language."
|
||||
```
|
||||
|
||||
**Config notes:**
|
||||
- `edit-format: whole` — more reliable file writes for local models (avoids code showing in chat only)
|
||||
- `map-tokens: 2048` — limits repo map size, important for smaller context windows
|
||||
- `no-auto-commits: true` — review changes before they hit git
|
||||
- `system-prompt` — forces English responses (important for Qwen3 which defaults to Chinese)
|
||||
|
||||
Switch models without editing the file:
|
||||
```
|
||||
/model ollama/qwen3-coder:30b
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional: Performance Tuning (if inference is slow)
|
||||
|
||||
Create a `Modelfile` in the project root:
|
||||
|
||||
```
|
||||
FROM qwen3-coder:30b
|
||||
PARAMETER num_ctx 8192
|
||||
PARAMETER num_gpu 99
|
||||
```
|
||||
|
||||
Register it:
|
||||
```powershell
|
||||
ollama create qwen3-coder-tuned -f Modelfile
|
||||
```
|
||||
|
||||
Update `.aider.conf.yml`:
|
||||
```yaml
|
||||
model: ollama/qwen3-coder-tuned
|
||||
```
|
||||
|
||||
**Why:** `num_gpu 99` maximizes VRAM usage. `num_ctx 8192` prevents the KV cache from
|
||||
spilling too much into RAM. Recommended context range: 4096–16384.
|
||||
|
||||
---
|
||||
|
||||
## Daily Aider Workflow
|
||||
|
||||
Start Aider (reads config automatically):
|
||||
```powershell
|
||||
aider
|
||||
```
|
||||
|
||||
| Command | What it does |
|
||||
|---------|-------------|
|
||||
| `/add src/foo.py` | Add file to context |
|
||||
| `/drop src/foo.py` | Remove file from context |
|
||||
| `/diff` | See pending changes |
|
||||
| `/undo` | Revert last change |
|
||||
| `/run pytest` | Run command, share output with model |
|
||||
| `/run git rm --cached file.py` | Run git commands from inside Aider |
|
||||
| `/clear` | Clear chat history (keep files) |
|
||||
| `/settings` | Show loaded config (verify auto_commits, system_prompt) |
|
||||
| `/model ollama/gemma4:12b` | Switch model on the fly |
|
||||
| `/exit` | Quit |
|
||||
|
||||
---
|
||||
|
||||
## Known Issues & Fixes
|
||||
|
||||
### Model responds in Chinese
|
||||
Cause: Qwen3 is a Chinese model and defaults to Chinese.
|
||||
Fix: `system-prompt` in `.aider.conf.yml` (already included above). Restart Aider after config change.
|
||||
|
||||
### Aider shows code in chat but doesn't write files
|
||||
Cause: Model didn't follow Aider's edit format.
|
||||
Fix 1: `/add target_file.py` before asking — hints the target file to the model.
|
||||
Fix 2: `edit-format: whole` in config (already included above).
|
||||
|
||||
### "File not found" / repo-map warnings for deleted files
|
||||
Cause: File deleted from disk but still tracked in git index.
|
||||
Fix:
|
||||
```powershell
|
||||
git rm --cached filename.py
|
||||
```
|
||||
Or from inside Aider:
|
||||
```
|
||||
/run git rm --cached filename.py
|
||||
```
|
||||
|
||||
### Config changes not taking effect
|
||||
Fix: Exit Aider (`/exit`) and restart. Config is only read at startup.
|
||||
|
||||
---
|
||||
|
||||
## Verify Everything Works
|
||||
|
||||
1. `ollama list` — model appears
|
||||
2. `aider --version` — prints version
|
||||
3. Start Aider, run `/settings` — confirm `auto_commits: False` and `system_prompt` present
|
||||
4. Ask Aider to create a simple file — it should write to disk, not just chat
|
||||
5. Response is in English
|
||||
-342
@@ -1,342 +0,0 @@
|
||||
# Local AI Coding Environment Implementation Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Build a local-first AI coding environment on a Windows 11 workstation with the following requirements:
|
||||
|
||||
### Hardware
|
||||
|
||||
* NVIDIA RTX 5070 Ti 12GB VRAM
|
||||
* 64GB+ system RAM (if available)
|
||||
* Windows 11
|
||||
* WSL2 Ubuntu
|
||||
* VS Code
|
||||
|
||||
### Existing Software
|
||||
|
||||
* Ollama installed and operational
|
||||
* Qwen3-Coder 30B installed in Ollama
|
||||
* Claude Code CLI installed
|
||||
* Git installed
|
||||
|
||||
### Constraints
|
||||
|
||||
* Source code is confidential.
|
||||
* All code must remain local by default.
|
||||
* Cloud models may be used only as an optional fallback.
|
||||
* Minimize API costs.
|
||||
* Support large codebases.
|
||||
|
||||
---
|
||||
|
||||
# Target Architecture
|
||||
|
||||
```text
|
||||
VS Code
|
||||
|
|
||||
+---- Continue Extension
|
||||
|
|
||||
+---- Claude Code CLI
|
||||
|
|
||||
+---- Aider
|
||||
|
|
||||
v
|
||||
Ollama
|
||||
|
|
||||
v
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Project Goals
|
||||
|
||||
Implement and validate the following workflows.
|
||||
|
||||
## Workflow 1: Local Chat
|
||||
|
||||
Developer can:
|
||||
|
||||
* Ask coding questions
|
||||
* Explain code
|
||||
* Generate code
|
||||
* Review code
|
||||
|
||||
using:
|
||||
|
||||
```text
|
||||
VS Code
|
||||
+
|
||||
Continue
|
||||
+
|
||||
Ollama
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow 2: Local Agent
|
||||
|
||||
Developer can:
|
||||
|
||||
* Refactor code
|
||||
* Create files
|
||||
* Modify files
|
||||
* Run git-aware edits
|
||||
|
||||
using:
|
||||
|
||||
```text
|
||||
Aider
|
||||
+
|
||||
Ollama
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow 3: Claude Code Optional
|
||||
|
||||
Developer can:
|
||||
|
||||
* Use Claude Code against local Ollama models
|
||||
* Compare behavior against Aider
|
||||
* Determine whether Claude Code provides additional value
|
||||
|
||||
using:
|
||||
|
||||
```text
|
||||
Claude Code
|
||||
+
|
||||
Ollama
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Deliverables
|
||||
|
||||
Produce the following:
|
||||
|
||||
## Deliverable 1
|
||||
|
||||
Environment verification script.
|
||||
|
||||
Verify:
|
||||
|
||||
* Ollama installed
|
||||
* NVIDIA GPU visible
|
||||
* CUDA available
|
||||
* WSL functioning
|
||||
* Qwen3-Coder model available
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
PASS: Ollama
|
||||
PASS: GPU
|
||||
PASS: CUDA
|
||||
PASS: Qwen3-Coder
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 2
|
||||
|
||||
Aider installation guide.
|
||||
|
||||
Include:
|
||||
|
||||
### Linux / WSL installation
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
pipx install aider-chat
|
||||
```
|
||||
|
||||
or preferred installation method.
|
||||
|
||||
### Ollama integration
|
||||
|
||||
Configuration examples.
|
||||
|
||||
### Verification steps
|
||||
|
||||
Simple repository test.
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 3
|
||||
|
||||
Continue configuration.
|
||||
|
||||
Create:
|
||||
|
||||
### VS Code setup instructions
|
||||
|
||||
Install Continue extension.
|
||||
|
||||
### Model configuration
|
||||
|
||||
Configure Continue to use:
|
||||
|
||||
```text
|
||||
Ollama
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
### Example config files
|
||||
|
||||
Include complete examples.
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 4
|
||||
|
||||
Claude Code local model integration.
|
||||
|
||||
Research and implement the best available approach for:
|
||||
|
||||
```text
|
||||
Claude Code
|
||||
->
|
||||
OpenAI-compatible endpoint
|
||||
->
|
||||
Ollama
|
||||
```
|
||||
|
||||
Requirements:
|
||||
|
||||
* Use officially supported methods where possible.
|
||||
* Avoid unsupported hacks.
|
||||
* Document limitations.
|
||||
* Provide rollback procedure.
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 5
|
||||
|
||||
Performance optimization.
|
||||
|
||||
Analyze:
|
||||
|
||||
### VRAM usage
|
||||
|
||||
Qwen3-Coder 30B on RTX 5070 Ti 12GB.
|
||||
|
||||
### Recommended quantization
|
||||
|
||||
Evaluate:
|
||||
|
||||
* Q4
|
||||
* Q5
|
||||
* IQ3
|
||||
|
||||
Recommend the best balance between:
|
||||
|
||||
* Quality
|
||||
* Speed
|
||||
* Memory usage
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 6
|
||||
|
||||
Benchmark suite.
|
||||
|
||||
Create repeatable tests:
|
||||
|
||||
### Test 1
|
||||
|
||||
Generate a REST API.
|
||||
|
||||
### Test 2
|
||||
|
||||
Refactor a medium-sized module.
|
||||
|
||||
### Test 3
|
||||
|
||||
Write unit tests.
|
||||
|
||||
### Test 4
|
||||
|
||||
Debug a failing application.
|
||||
|
||||
Measure:
|
||||
|
||||
* Completion time
|
||||
* Accuracy
|
||||
* Token throughput
|
||||
* User effort
|
||||
|
||||
Compare:
|
||||
|
||||
```text
|
||||
Aider + Qwen3-Coder 30B
|
||||
Claude Code + Qwen3-Coder 30B
|
||||
Continue + Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Preferred Outcome
|
||||
|
||||
Primary development workflow:
|
||||
|
||||
```text
|
||||
VS Code
|
||||
+
|
||||
Continue
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
Agent workflow:
|
||||
|
||||
```text
|
||||
Aider
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
Optional advanced workflow:
|
||||
|
||||
```text
|
||||
Claude Code
|
||||
+
|
||||
Qwen3-Coder 30B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Success Criteria
|
||||
|
||||
The project is successful if:
|
||||
|
||||
1. All code remains local.
|
||||
2. No cloud services are required.
|
||||
3. Developer can perform daily coding tasks locally.
|
||||
4. Aider successfully edits repositories.
|
||||
5. Continue provides a productive IDE experience.
|
||||
6. Claude Code local integration is evaluated and documented.
|
||||
7. Setup can be reproduced on a fresh machine.
|
||||
|
||||
---
|
||||
|
||||
# Final Report
|
||||
|
||||
Produce a final report containing:
|
||||
|
||||
* Architecture diagram
|
||||
* Installation steps
|
||||
* Configuration files
|
||||
* Benchmark results
|
||||
* Known limitations
|
||||
* Recommended workflow
|
||||
* Future upgrade path
|
||||
|
||||
The report should be suitable for long-term maintenance and onboarding of additional developers.
|
||||
@@ -0,0 +1,27 @@
|
||||
import os
|
||||
import pandas as pdc
|
||||
from openpyxl import load_workbook
|
||||
|
||||
input_folder = r"d:\input"
|
||||
|
||||
merged_dfs = []
|
||||
total_rows = 0
|
||||
|
||||
excel_files = [f for f in os.listdir(input_folder) if f.endswith(".xlsx")]
|
||||
|
||||
for excel_file in sorted(excel_files):
|
||||
try:
|
||||
wb = load_workbook(excel_file, data_only=False)
|
||||
sheet_name = wb.sheetnames[0] # Get first sheet name
|
||||
df = pdc.read_excel(excel_file, sheet_name=sheet_name)
|
||||
merged_dfs.append(df)
|
||||
total_rows += len(df)
|
||||
except Exception as e:
|
||||
print(f"Error processing {excel_file}: " + str(e))
|
||||
continue
|
||||
|
||||
if merged_dfs:
|
||||
merged_output = pd.concat(merged_dfs, ignore_index=True)
|
||||
output_file = r"d:\output\merged_file.xlsx"
|
||||
merged_output.to_excel(output_file, index=False)
|
||||
print(f"Successfully saved {total_rows} rows to output file")
|
||||
@@ -0,0 +1,403 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>MLP Playground — Neuron → Layer → MLP (section 5.1)</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/d3@7"></script>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0f1420;
|
||||
--panel: #161d2e;
|
||||
--panel-2: #1d2940;
|
||||
--ink: #e8edf6;
|
||||
--muted: #94a3b8;
|
||||
--line: #2a3550;
|
||||
--pos: #4ade80; /* positive weight / activation */
|
||||
--neg: #f87171; /* negative weight / activation */
|
||||
--accent: #60a5fa;
|
||||
--accent-2: #fbbf24;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { margin: 0; padding: 0; }
|
||||
body {
|
||||
background: radial-gradient(1200px 600px at 70% -10%, #1b2740 0%, var(--bg) 55%);
|
||||
color: var(--ink);
|
||||
font-family: "Segoe UI", system-ui, -apple-system, Roboto, sans-serif;
|
||||
line-height: 1.5;
|
||||
padding: 22px clamp(14px, 3vw, 40px) 60px;
|
||||
}
|
||||
h1 { font-size: clamp(20px, 2.6vw, 28px); margin: 0 0 4px; }
|
||||
.sub { color: var(--muted); margin: 0 0 18px; max-width: 70ch; }
|
||||
.sub b { color: var(--ink); }
|
||||
.layout { display: grid; grid-template-columns: 300px 1fr; gap: 18px; align-items: start; }
|
||||
@media (max-width: 900px) { .layout { grid-template-columns: 1fr; } }
|
||||
|
||||
.panel {
|
||||
background: linear-gradient(180deg, var(--panel) 0%, var(--panel-2) 100%);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 14px;
|
||||
padding: 16px 16px 18px;
|
||||
box-shadow: 0 10px 30px rgba(0,0,0,.35);
|
||||
}
|
||||
.panel h2 { font-size: 14px; text-transform: uppercase; letter-spacing: .08em; color: var(--muted); margin: 0 0 12px; }
|
||||
|
||||
.field { margin-bottom: 16px; }
|
||||
.field label { display: flex; justify-content: space-between; font-size: 13px; margin-bottom: 6px; }
|
||||
.field label .val { font-variant-numeric: tabular-nums; color: var(--accent); font-weight: 600; }
|
||||
input[type=range] { width: 100%; accent-color: var(--accent); }
|
||||
|
||||
.btn {
|
||||
appearance: none; border: 1px solid var(--line); cursor: pointer;
|
||||
background: var(--panel-2); color: var(--ink);
|
||||
padding: 9px 12px; border-radius: 9px; font-size: 13px; font-weight: 600;
|
||||
transition: .15s; width: 100%; margin-bottom: 9px;
|
||||
}
|
||||
.btn:hover { border-color: var(--accent); color: #fff; }
|
||||
.btn.primary { background: linear-gradient(180deg, #3b82f6, #2563eb); border-color: #2563eb; }
|
||||
.btn.primary:hover { background: linear-gradient(180deg, #4f93ff, #2f6fe0); }
|
||||
|
||||
.legend { display: flex; flex-direction: column; gap: 7px; font-size: 12.5px; color: var(--muted); }
|
||||
.legend .row { display: flex; align-items: center; gap: 8px; }
|
||||
.swatch { width: 22px; height: 4px; border-radius: 3px; }
|
||||
|
||||
.stage { background: rgba(0,0,0,.18); border-radius: 14px; }
|
||||
svg { display: block; width: 100%; height: auto; }
|
||||
|
||||
.out-card {
|
||||
margin-top: 14px; padding: 14px 16px; border-radius: 12px;
|
||||
border: 1px solid var(--line); background: rgba(0,0,0,.2);
|
||||
display: flex; align-items: center; justify-content: space-between; gap: 14px;
|
||||
}
|
||||
.out-card .label { color: var(--muted); font-size: 13px; }
|
||||
.out-card .verdict { font-size: 26px; font-weight: 800; font-variant-numeric: tabular-nums; }
|
||||
|
||||
/* tooltip */
|
||||
.tip {
|
||||
position: fixed; pointer-events: none; z-index: 50; opacity: 0;
|
||||
background: #0b1322; border: 1px solid var(--accent); color: var(--ink);
|
||||
padding: 8px 10px; border-radius: 9px; font-size: 12.5px; max-width: 240px;
|
||||
box-shadow: 0 8px 24px rgba(0,0,0,.5); transition: opacity .12s;
|
||||
}
|
||||
.tip b { color: var(--accent); }
|
||||
.tip .mono { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; }
|
||||
|
||||
/* node + edge base styles */
|
||||
.edge { transition: stroke-opacity .2s; }
|
||||
.node circle { stroke: #0b1322; stroke-width: 1.5px; cursor: pointer; }
|
||||
.node-label { font-size: 11px; fill: var(--muted); }
|
||||
.layer-title { font-size: 12px; fill: var(--ink); font-weight: 700; text-anchor: middle; }
|
||||
.layer-cap { font-size: 11px; fill: var(--muted); text-anchor: middle; }
|
||||
.hint { color: var(--muted); font-size: 12px; margin-top: 10px; }
|
||||
.pill { display:inline-block; padding:2px 7px; border:1px solid var(--line); border-radius:999px; font-size:11px; color:var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>MLP Playground — <span style="color:var(--accent)">Neuron → Layer → MLP</span></h1>
|
||||
<p class="sub">
|
||||
The reviewer-panel story from <b>section 5.1</b>, made live. Two inputs
|
||||
(<b>income</b>, <b>credit</b>) flow through two panels of 8 reviewers into one final judge:
|
||||
the <span class="pill">2 → 8 → 8 → 1</span> network. Every line is a weight, every dot is a
|
||||
<span class="mono">tanh(w·x + b)</span> opinion. Drag the inputs, hover anything, or press play.
|
||||
</p>
|
||||
|
||||
<div class="layout">
|
||||
<div class="panel">
|
||||
<h2>Inputs (the application)</h2>
|
||||
<div class="field">
|
||||
<label>Income (x₁) <span class="val" id="x1v">0.50</span></label>
|
||||
<input type="range" id="x1" min="-1" max="1" step="0.01" value="0.5" />
|
||||
</div>
|
||||
<div class="field">
|
||||
<label>Credit (x₂) <span class="val" id="x2v">-0.30</span></label>
|
||||
<input type="range" id="x2" min="-1" max="1" step="0.01" value="-0.3" />
|
||||
</div>
|
||||
|
||||
<h2 style="margin-top:6px">Controls</h2>
|
||||
<button class="btn primary" id="play">▶ Animate forward pass</button>
|
||||
<button class="btn" id="reroll">🎲 New random weights</button>
|
||||
<button class="btn" id="reset">↺ Reset inputs</button>
|
||||
|
||||
<h2 style="margin-top:18px">Read the colors</h2>
|
||||
<div class="legend">
|
||||
<div class="row"><span class="swatch" style="background:var(--pos)"></span> positive weight / activation</div>
|
||||
<div class="row"><span class="swatch" style="background:var(--neg)"></span> negative weight / activation</div>
|
||||
<div class="row"><span class="swatch" style="background:var(--muted);height:2px"></span> thin line = weak, thick = strong</div>
|
||||
</div>
|
||||
|
||||
<div class="out-card">
|
||||
<span class="label">Final judge<br>score (tanh)</span>
|
||||
<span class="verdict" id="verdict">—</span>
|
||||
</div>
|
||||
<p class="hint" id="verdict-text">Closer to <b style="color:var(--pos)">+1</b> = likely good tenant, closer to <b style="color:var(--neg)">−1</b> = likely risky.</p>
|
||||
</div>
|
||||
|
||||
<div class="panel stage">
|
||||
<svg id="net" viewBox="0 0 980 560" preserveAspectRatio="xMidYMid meet" aria-label="MLP diagram"></svg>
|
||||
<p class="hint" style="padding:0 6px 4px">
|
||||
Hover a <b>line</b> to see its weight, or a <b>dot</b> to see that reviewer's
|
||||
<span class="mono">w·x + b</span> and squashed opinion. Layers left→right: form → junior panel → senior panel → final judge.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="tip" id="tip"></div>
|
||||
|
||||
<script>
|
||||
// ----------------------------------------------------------------------------
|
||||
// Tiny MLP, faithful to section 5.1: Neuron = tanh(w·x + b), Layer = neurons in
|
||||
// parallel, MLP = layers in sequence. Shape: 2 -> 8 -> 8 -> 1.
|
||||
// ----------------------------------------------------------------------------
|
||||
const SIZES = [2, 8, 8, 1];
|
||||
const LAYER_TITLES = ["Inputs", "Hidden panel 1", "Hidden panel 2", "Final judge"];
|
||||
const LAYER_CAPS = ["income, credit", "8 junior reviewers", "8 senior reviewers", "1 score"];
|
||||
|
||||
const rand = () => Math.random() * 2 - 1; // uniform(-1, 1), like micrograd
|
||||
const tanh = (z) => Math.tanh(z);
|
||||
|
||||
// Build network: weights[l] is matrix [nout x nin], biases[l] is [nout].
|
||||
let weights, biases;
|
||||
function buildNetwork() {
|
||||
weights = []; biases = [];
|
||||
for (let l = 0; l < SIZES.length - 1; l++) {
|
||||
const nin = SIZES[l], nout = SIZES[l + 1];
|
||||
weights.push(Array.from({length: nout}, () => Array.from({length: nin}, rand)));
|
||||
biases.push(Array.from({length: nout}, rand));
|
||||
}
|
||||
}
|
||||
buildNetwork();
|
||||
|
||||
// Forward pass. Returns activations per layer (layer 0 = raw inputs) and the
|
||||
// pre-activation (w·x+b) for each neuron so the tooltip can show both.
|
||||
function forward(x) {
|
||||
const acts = [x.slice()];
|
||||
const preacts = [x.slice()]; // inputs have no pre-activation; mirror for indexing
|
||||
let cur = x.slice();
|
||||
for (let l = 0; l < weights.length; l++) {
|
||||
const z = [], a = [];
|
||||
for (let j = 0; j < weights[l].length; j++) {
|
||||
let s = biases[l][j];
|
||||
for (let i = 0; i < cur.length; i++) s += weights[l][j][i] * cur[i];
|
||||
z.push(s); a.push(tanh(s));
|
||||
}
|
||||
preacts.push(z); acts.push(a); cur = a;
|
||||
}
|
||||
return { acts, preacts };
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Layout geometry
|
||||
// ----------------------------------------------------------------------------
|
||||
const W = 980, H = 560, PAD_X = 90, PAD_TOP = 70, PAD_BOT = 50;
|
||||
const svg = d3.select("#net");
|
||||
const tip = d3.select("#tip");
|
||||
|
||||
function layerX(l) { return PAD_X + (W - 2 * PAD_X) * (l / (SIZES.length - 1)); }
|
||||
function nodeY(l, j) {
|
||||
const n = SIZES[l];
|
||||
const top = PAD_TOP, bot = H - PAD_BOT, span = bot - top;
|
||||
if (n === 1) return (top + bot) / 2;
|
||||
return top + span * (j / (n - 1));
|
||||
}
|
||||
|
||||
// color scales: weights & activations mapped to red(-)/green(+)
|
||||
function weightColor(w) { return w >= 0 ? "var(--pos)" : "var(--neg)"; }
|
||||
function actColor(a) { return a >= 0 ? "var(--pos)" : "var(--neg)"; }
|
||||
function weightWidth(w) { return 0.6 + Math.min(5.5, Math.abs(w) * 3.2); }
|
||||
|
||||
const gEdges = svg.append("g").attr("class", "edges");
|
||||
const gNodes = svg.append("g").attr("class", "nodes");
|
||||
const gTitles = svg.append("g").attr("class", "titles");
|
||||
|
||||
// layer titles + captions
|
||||
function drawTitles() {
|
||||
gTitles.selectAll("*").remove();
|
||||
SIZES.forEach((n, l) => {
|
||||
gTitles.append("text").attr("class", "layer-title")
|
||||
.attr("x", layerX(l)).attr("y", 28).text(LAYER_TITLES[l]);
|
||||
gTitles.append("text").attr("class", "layer-cap")
|
||||
.attr("x", layerX(l)).attr("y", 46).text(LAYER_CAPS[l]);
|
||||
});
|
||||
}
|
||||
|
||||
// build edges + nodes once (geometry is static; only color/width/anim update)
|
||||
let edgeSel, nodeSel;
|
||||
function drawStructure() {
|
||||
// edges
|
||||
const edges = [];
|
||||
for (let l = 0; l < weights.length; l++) {
|
||||
for (let j = 0; j < SIZES[l + 1]; j++) {
|
||||
for (let i = 0; i < SIZES[l]; i++) {
|
||||
edges.push({ l, i, j,
|
||||
x1: layerX(l), y1: nodeY(l, i),
|
||||
x2: layerX(l + 1), y2: nodeY(l + 1, j) });
|
||||
}
|
||||
}
|
||||
}
|
||||
edgeSel = gEdges.selectAll("line").data(edges).join("line")
|
||||
.attr("class", "edge")
|
||||
.attr("x1", d => d.x1).attr("y1", d => d.y1)
|
||||
.attr("x2", d => d.x2).attr("y2", d => d.y2)
|
||||
.on("mousemove", (e, d) => {
|
||||
const w = weights[d.l][d.j][d.i];
|
||||
showTip(e, `<b>Weight</b> · panel line<br>
|
||||
<span class="mono">layer ${d.l} → ${d.l + 1}</span><br>
|
||||
from neuron ${d.i} → neuron ${d.j}<br>
|
||||
<span class="mono">w = ${w.toFixed(3)}</span><br>
|
||||
<span style="color:${w>=0?'var(--pos)':'var(--neg)'}">${w>=0?'boosts':'damps'} this signal</span>`);
|
||||
})
|
||||
.on("mouseleave", hideTip);
|
||||
|
||||
// nodes
|
||||
const nodes = [];
|
||||
SIZES.forEach((n, l) => {
|
||||
for (let j = 0; j < n; j++) nodes.push({ l, j, x: layerX(l), y: nodeY(l, j) });
|
||||
});
|
||||
const g = gNodes.selectAll("g.node").data(nodes).join("g").attr("class", "node")
|
||||
.attr("transform", d => `translate(${d.x},${d.y})`);
|
||||
g.selectAll("circle").data(d => [d]).join("circle").attr("r", d => d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11);
|
||||
nodeSel = g;
|
||||
|
||||
// input row labels
|
||||
gNodes.selectAll("text.in-lab").remove();
|
||||
["x₁ income", "x₂ credit"].forEach((t, i) => {
|
||||
gNodes.append("text").attr("class", "node-label in-lab")
|
||||
.attr("x", layerX(0) - 22).attr("y", nodeY(0, i) + 4)
|
||||
.attr("text-anchor", "end").text(t);
|
||||
});
|
||||
|
||||
g.on("mousemove", (e, d) => {
|
||||
if (d.l === 0) {
|
||||
showTip(e, `<b>Input</b><br><span class="mono">x${d.j+1} = ${state.acts[0][d.j].toFixed(3)}</span><br>${d.j===0?'monthly income':'credit score'} (raw form field)`);
|
||||
} else {
|
||||
const z = state.preacts[d.l][d.j], a = state.acts[d.l][d.j];
|
||||
const role = d.l === SIZES.length - 1 ? "final judge" : `reviewer ${d.j} in panel ${d.l}`;
|
||||
showTip(e, `<b>Neuron</b> · ${role}<br>
|
||||
<span class="mono">w·x + b = ${z.toFixed(3)}</span><br>
|
||||
<span class="mono">tanh(...) = <b style="color:${a>=0?'var(--pos)':'var(--neg)'}">${a.toFixed(3)}</b></span><br>
|
||||
${a>=0?'leaning yes':'leaning no'}`);
|
||||
}
|
||||
}).on("mouseleave", hideTip);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Paint current state (colors + widths) onto edges and nodes
|
||||
// ----------------------------------------------------------------------------
|
||||
let state = { acts: [], preacts: [] };
|
||||
|
||||
function recompute() {
|
||||
const x1 = +d3.select("#x1").property("value");
|
||||
const x2 = +d3.select("#x2").property("value");
|
||||
state = forward([x1, x2]);
|
||||
paint();
|
||||
updateVerdict();
|
||||
}
|
||||
|
||||
function paint() {
|
||||
edgeSel
|
||||
.attr("stroke", d => weightColor(weights[d.l][d.j][d.i]))
|
||||
.attr("stroke-width", d => weightWidth(weights[d.l][d.j][d.i]))
|
||||
.attr("stroke-opacity", 0.45);
|
||||
|
||||
nodeSel.select("circle")
|
||||
.attr("fill", d => {
|
||||
const a = state.acts[d.l] ? state.acts[d.l][d.j] : 0;
|
||||
const mag = Math.min(1, Math.abs(a));
|
||||
const base = a >= 0 ? [74,222,128] : [248,113,113];
|
||||
// blend toward dark panel for low magnitude
|
||||
const mix = (c) => Math.round(40 + (c - 40) * (0.25 + 0.75 * mag));
|
||||
return `rgb(${mix(base[0])},${mix(base[1])},${mix(base[2])})`;
|
||||
});
|
||||
}
|
||||
|
||||
function updateVerdict() {
|
||||
const out = state.acts[state.acts.length - 1][0];
|
||||
const el = d3.select("#verdict");
|
||||
el.text((out >= 0 ? "+" : "") + out.toFixed(3))
|
||||
.style("color", out >= 0 ? "var(--pos)" : "var(--neg)");
|
||||
d3.select("#verdict-text").html(
|
||||
out >= 0
|
||||
? `Score <b style="color:var(--pos)">${out.toFixed(2)}</b> → leaning <b>good tenant</b>.`
|
||||
: `Score <b style="color:var(--neg)">${out.toFixed(2)}</b> → leaning <b>risky</b>.`
|
||||
);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Animated forward pass: light up layer by layer, pulse edges between them
|
||||
// ----------------------------------------------------------------------------
|
||||
let playing = false;
|
||||
function animate() {
|
||||
if (playing) return;
|
||||
playing = true;
|
||||
// dim everything first
|
||||
edgeSel.attr("stroke-opacity", 0.08);
|
||||
nodeSel.select("circle").attr("opacity", 0.25);
|
||||
d3.select("#play").attr("disabled", true).text("▶ playing…");
|
||||
|
||||
const STEP = 620;
|
||||
// light input layer
|
||||
lightLayer(0, 0);
|
||||
for (let l = 0; l < weights.length; l++) {
|
||||
pulseEdges(l, STEP * (l + 0.15));
|
||||
lightLayer(l + 1, STEP * (l + 1));
|
||||
}
|
||||
setTimeout(() => {
|
||||
edgeSel.transition().duration(300).attr("stroke-opacity", 0.45);
|
||||
nodeSel.select("circle").transition().duration(300).attr("opacity", 1);
|
||||
playing = false;
|
||||
d3.select("#play").attr("disabled", null).text("▶ Animate forward pass");
|
||||
}, STEP * (weights.length + 1) + 250);
|
||||
}
|
||||
|
||||
function lightLayer(l, delay) {
|
||||
nodeSel.filter(d => d.l === l).select("circle")
|
||||
.transition().delay(delay).duration(280)
|
||||
.attr("opacity", 1)
|
||||
.attr("r", d => (d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11) + 4)
|
||||
.transition().duration(220)
|
||||
.attr("r", d => d.l === 0 || d.l === SIZES.length - 1 ? 13 : 11);
|
||||
}
|
||||
|
||||
function pulseEdges(l, delay) {
|
||||
edgeSel.filter(d => d.l === l)
|
||||
.transition().delay(delay).duration(300).attr("stroke-opacity", 0.9)
|
||||
.transition().duration(320).attr("stroke-opacity", 0.18);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Tooltip helpers
|
||||
// ----------------------------------------------------------------------------
|
||||
function showTip(e, html) {
|
||||
tip.html(html).style("opacity", 1)
|
||||
.style("left", (e.clientX + 14) + "px")
|
||||
.style("top", (e.clientY + 14) + "px");
|
||||
}
|
||||
function hideTip() { tip.style("opacity", 0); }
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Wire up controls
|
||||
// ----------------------------------------------------------------------------
|
||||
function syncLabels() {
|
||||
d3.select("#x1v").text((+d3.select("#x1").property("value")).toFixed(2));
|
||||
d3.select("#x2v").text((+d3.select("#x2").property("value")).toFixed(2));
|
||||
}
|
||||
d3.select("#x1").on("input", () => { syncLabels(); recompute(); });
|
||||
d3.select("#x2").on("input", () => { syncLabels(); recompute(); });
|
||||
d3.select("#play").on("click", animate);
|
||||
d3.select("#reroll").on("click", () => { buildNetwork(); drawStructure(); recompute(); });
|
||||
d3.select("#reset").on("click", () => {
|
||||
d3.select("#x1").property("value", 0.5);
|
||||
d3.select("#x2").property("value", -0.3);
|
||||
syncLabels(); recompute();
|
||||
});
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Init
|
||||
// ----------------------------------------------------------------------------
|
||||
drawTitles();
|
||||
drawStructure();
|
||||
syncLabels();
|
||||
recompute();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,250 @@
|
||||
# Local AI Coding Environment — Reference & Lessons Learned
|
||||
|
||||
> Consolidates `local_ai_setup.md` and `local_llm.md`. Those files are superseded by this one.
|
||||
|
||||
---
|
||||
|
||||
## Hardware
|
||||
|
||||
| Component | Spec |
|
||||
|-----------|------|
|
||||
| GPU | NVIDIA RTX 5070 Ti, 12 GB VRAM |
|
||||
| RAM | 32 GB system RAM |
|
||||
| OS | Windows 11 |
|
||||
| WSL | WSL2 Ubuntu |
|
||||
|
||||
---
|
||||
|
||||
## Software Stack
|
||||
|
||||
| Tool | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| Ollama | Local model server | Serves OpenAI-compatible API at `localhost:11434/v1` |
|
||||
| OpenCode | Primary AI coding agent | Installed via npm (`opencode-ai`) |
|
||||
| Aider | Alternative AI coding agent | pip: `aider-chat` |
|
||||
| Claude Code | Cloud AI coding agent | Optional; can point at Ollama |
|
||||
| VS Code | Editor | With Continue extension for inline chat |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
VS Code
|
||||
├── Continue Extension ──────────────┐
|
||||
├── Claude Code CLI ────────────────┤
|
||||
└── OpenCode / Aider ───────────────┤
|
||||
▼
|
||||
Ollama
|
||||
│
|
||||
▼
|
||||
Qwen3-Coder 30B (primary)
|
||||
```
|
||||
|
||||
All code stays local. Cloud models are optional fallback only.
|
||||
|
||||
---
|
||||
|
||||
## Ollama — Available Models
|
||||
|
||||
| Model | Size | Use |
|
||||
|-------|------|-----|
|
||||
| `qwen3-coder:30b-fixed` | 18 GB | **Primary** — OpenCode agent (fixed config, see below) |
|
||||
| `qwen3-coder:30b` | 18 GB | Original pull (unmodified) |
|
||||
| `gemma4:latest` | 9.6 GB | English-reliable alternative |
|
||||
| `gemma4:12b` | 7.6 GB | Lighter English-reliable option |
|
||||
| `qwen2.5-coder:14b` | 9.0 GB | Mid-weight coding model |
|
||||
| `qwen2.5-coder:7b` | 4.7 GB | Fast, lightweight coding |
|
||||
| `qwen2.5-coder:1.5b` | 986 MB | Minimal footprint |
|
||||
| `qwen3:latest` | 5.2 GB | General chat |
|
||||
| `qwen3:1.7b` | 1.4 GB | Fast general chat |
|
||||
| `bge-m3:latest` | 1.2 GB | Embeddings |
|
||||
|
||||
---
|
||||
|
||||
## OpenCode Configuration
|
||||
|
||||
**Config file:** `~/.config/opencode/opencode.jsonc`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"ollama": {
|
||||
"options": {
|
||||
"baseURL": "http://localhost:11434/v1"
|
||||
},
|
||||
"models": {
|
||||
"qwen3-coder:30b-fixed": { "name": "Qwen3 Coder 30B" },
|
||||
"qwen2.5-coder:14b": { "name": "Qwen 2.5 Coder 14B" },
|
||||
"qwen2.5-coder:7b": { "name": "Qwen 2.5 Coder 7B" },
|
||||
"gemma4:latest": { "name": "Gemma 4 8B (latest)" },
|
||||
"qwen3:latest": { "name": "Qwen3 8B" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"model": "ollama/qwen3-coder:30b-fixed"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Active Model — qwen3-coder:30b-fixed Modelfile
|
||||
|
||||
```
|
||||
FROM <local blob: sha256-9124a16eb1a7...>
|
||||
TEMPLATE {{ .Prompt }}
|
||||
PARAMETER repeat_penalty 1.05
|
||||
PARAMETER stop <|im_start|>
|
||||
PARAMETER stop <|im_end|>
|
||||
PARAMETER stop <|endoftext|>
|
||||
PARAMETER temperature 0.7
|
||||
PARAMETER top_k 20
|
||||
PARAMETER top_p 0.8
|
||||
PARAMETER num_ctx 32768 ← fixed (was missing)
|
||||
PARAMETER num_predict 8192 ← fixed (was missing)
|
||||
```
|
||||
|
||||
Model specs: 30.5B params, MoE architecture (qwen3moe), Q4_K_M quantization, supports tools + completion.
|
||||
|
||||
---
|
||||
|
||||
## Issue & Fix — Tool Call JSON Truncation
|
||||
|
||||
### Error
|
||||
|
||||
```
|
||||
llama-server returned invalid tool call arguments for "edit":
|
||||
unexpected end of ... [retrying in 51s attempt N]
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
|
||||
Two Ollama parameters were missing from the model's Modelfile:
|
||||
|
||||
| Parameter | Default when missing | Required for tool calls |
|
||||
|-----------|---------------------|------------------------|
|
||||
| `num_ctx` | **2048 tokens** | Minimum ~8192; 32768 recommended |
|
||||
| `num_predict` | **128 tokens** | Minimum ~1024; 8192 recommended |
|
||||
|
||||
With only 128 output tokens allowed, the model began generating valid JSON for the `edit` tool call but ran out of budget before closing all braces, producing malformed JSON. The 2048-token context also filled up quickly during long sessions, compounding the problem.
|
||||
|
||||
### Fix Applied
|
||||
|
||||
Rebuilt `qwen3-coder:30b-fixed` with the two parameters added:
|
||||
|
||||
```bash
|
||||
# Export current Modelfile, append fixes, rebuild
|
||||
ollama show qwen3-coder:30b-fixed --modelfile > Modelfile_fixed
|
||||
# (add num_ctx 32768 and num_predict 8192)
|
||||
ollama create qwen3-coder:30b-fixed -f Modelfile_fixed
|
||||
```
|
||||
|
||||
No changes needed to the OpenCode config — same model name, same config file.
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
ollama show qwen3-coder:30b-fixed
|
||||
# Should show: num_ctx 32768 and num_predict 8192 in Parameters section
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Aider Configuration
|
||||
|
||||
**Config file:** `.aider.conf.yml` in project root
|
||||
|
||||
```yaml
|
||||
model: ollama/qwen3-coder:30b
|
||||
openai-api-base: http://localhost:11434/v1
|
||||
openai-api-key: ollama
|
||||
edit-format: whole
|
||||
map-tokens: 2048
|
||||
no-auto-commits: true
|
||||
system-prompt: "Always respond in English only. Never use Chinese or any other language."
|
||||
```
|
||||
|
||||
| Option | Why |
|
||||
|--------|-----|
|
||||
| `edit-format: whole` | More reliable file writes for local models |
|
||||
| `map-tokens: 2048` | Limits repo map size for smaller context windows |
|
||||
| `no-auto-commits: true` | Review changes before they hit git |
|
||||
| `system-prompt` | Qwen3 defaults to Chinese; forces English |
|
||||
|
||||
### Aider Quick Reference
|
||||
|
||||
| Command | What it does |
|
||||
|---------|-------------|
|
||||
| `/add src/foo.py` | Add file to context |
|
||||
| `/drop src/foo.py` | Remove file from context |
|
||||
| `/diff` | See pending changes |
|
||||
| `/undo` | Revert last change |
|
||||
| `/run pytest` | Run command, share output with model |
|
||||
| `/clear` | Clear chat history (keep files) |
|
||||
| `/settings` | Show loaded config |
|
||||
| `/model ollama/gemma4:12b` | Switch model on the fly |
|
||||
|
||||
---
|
||||
|
||||
## Known Issues & Fixes
|
||||
|
||||
### Tool call JSON truncated (OpenCode / llama-server)
|
||||
**Error:** `llama-server returned invalid tool call arguments for "edit": unexpected end of...`
|
||||
**Cause:** `num_predict` too low (default 128) cuts off JSON mid-generation.
|
||||
**Fix:** See "Issue & Fix" section above — add `num_ctx` and `num_predict` to Modelfile.
|
||||
|
||||
### Model responds in Chinese
|
||||
**Cause:** Qwen3 is a Chinese model and defaults to Chinese.
|
||||
**Fix (Aider):** `system-prompt` in `.aider.conf.yml`.
|
||||
**Fix (OpenCode):** Add a system prompt in OpenCode's project config if supported, or use `gemma4` models which default to English.
|
||||
|
||||
### Aider writes code to chat but not to disk
|
||||
**Cause:** Model didn't follow Aider's edit format.
|
||||
**Fix 1:** `/add target_file.py` before the request — hints the target.
|
||||
**Fix 2:** `edit-format: whole` in config (already included above).
|
||||
|
||||
### Config changes not taking effect (Aider)
|
||||
**Fix:** Exit (`/exit`) and restart. Config is read only at startup.
|
||||
|
||||
### "File not found" / repo-map warnings for deleted files
|
||||
**Cause:** File deleted from disk but still tracked in git index.
|
||||
**Fix:**
|
||||
```bash
|
||||
git rm --cached filename.py
|
||||
# or from inside Aider:
|
||||
/run git rm --cached filename.py
|
||||
```
|
||||
|
||||
### Model loops with filler text, never executes tool calls
|
||||
**Symptom:** Model repeatedly says "I'll continue..." but never reads/edits files. Context counter in OpenCode bottom-left shows a number near your `num_ctx` limit.
|
||||
**Cause:** Context window is full. No tokens left for tool call JSON, so the model outputs short filler and stalls.
|
||||
**Fix (immediate):** Start a new OpenCode session. Break large tasks (e.g. "translate all files") into one file per session.
|
||||
**Fix (permanent):** Increase `num_ctx` in the Modelfile (e.g. 65536), rebuild, restart Ollama. Watch RAM usage — each doubling of `num_ctx` roughly doubles KV cache memory.
|
||||
|
||||
### Context fills up, model performance degrades
|
||||
**Cause:** Long session accumulates tokens beyond `num_ctx`.
|
||||
**Fix (Aider):** `/clear` to reset chat history without losing file context.
|
||||
**Fix (Ollama model):** Increase `num_ctx` in Modelfile (current: 32768).
|
||||
|
||||
---
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- qwen3-coder:30b is a MoE model — fits in VRAM+RAM split on RTX 5070 Ti (12 GB VRAM).
|
||||
- `num_gpu 99` in Modelfile maximizes layers on VRAM; remainder spills to RAM.
|
||||
- Safe `num_ctx` range for this hardware: **8192–32768**. Above 32768 risks RAM pressure.
|
||||
- Q4_K_M quantization gives best quality/speed/memory balance for this size class.
|
||||
|
||||
---
|
||||
|
||||
## Environment Verification Checklist
|
||||
|
||||
```bash
|
||||
ollama list # models appear
|
||||
ollama show qwen3-coder:30b-fixed # num_ctx and num_predict present
|
||||
curl http://localhost:11434/api/tags # Ollama serving
|
||||
aider --version # Aider installed
|
||||
opencode --version # OpenCode installed
|
||||
nvidia-smi # GPU visible
|
||||
```
|
||||
@@ -0,0 +1,4 @@
|
||||
torch>=2.0.0
|
||||
numpy
|
||||
ipython
|
||||
ipykernel
|
||||
Binary file not shown.
@@ -0,0 +1,86 @@
|
||||
# Server Information
|
||||
|
||||
**SSH Access:** `ssh root@39.102.124.161` (Alibaba Cloud)
|
||||
**OS:** CentOS with Baota panel pre-configured
|
||||
**Website:** https://www.thereisnospoonadu.com/
|
||||
**Web Server:** Tengine (Alibaba's fork of Nginx) — use `tengine` commands, NOT `nginx`
|
||||
|
||||
## Gitea (self-hosted Git server)
|
||||
|
||||
**Web UI:** https://git.thereisnospoonadu.com
|
||||
**Admin username:** beastgitea2026
|
||||
**API token:** paste fresh when needed (not stored)
|
||||
**SSH remote:** `git@git.thereisnospoonadu.com:beastgitea2026/<repo>.git`
|
||||
**SSH port:** 222 — configured in `~/.ssh/config` on dev machine
|
||||
**Gitea data:** `/var/lib/gitea/` | **Docker container name:** `gitea`
|
||||
**Tengine config:** `/etc/tengine/conf.d/gitea.conf`
|
||||
|
||||
## Critical File Paths
|
||||
|
||||
**Web Root:** `/www/wwwroot/soft_download/`
|
||||
**Tengine Config (ACTUAL):** `/etc/tengine/conf.d/thereisnospoonadu.conf`
|
||||
- Site uses HTTPS (port 443), HTTP redirects to HTTPS
|
||||
- Always test changes with: `tengine -t && tengine -s reload`
|
||||
|
||||
**Auth File:** `/etc/tengine/.htpasswd`
|
||||
**Logs:** Check tengine log paths inside `/etc/tengine/conf.d/thereisnospoonadu.conf`
|
||||
|
||||
## User Management
|
||||
|
||||
Add a new end user (can login and download files):
|
||||
```bash
|
||||
# Add user (file already exists — do NOT use -c or it will overwrite all users)
|
||||
htpasswd /etc/tengine/.htpasswd username
|
||||
|
||||
# Verify a user's password
|
||||
htpasswd -v /etc/tengine/.htpasswd username
|
||||
|
||||
# Delete a user
|
||||
htpasswd -D /etc/tengine/.htpasswd username
|
||||
|
||||
# List current users
|
||||
cat /etc/tengine/.htpasswd
|
||||
```
|
||||
Note: `htpasswd` requires `httpd-tools` package (`yum install -y httpd-tools`).
|
||||
|
||||
## Architecture
|
||||
|
||||
**Download Portal System:**
|
||||
- Login page: `/login.html` (public)
|
||||
- File listing: `/list/` (requires HTTP Basic Auth via Authorization header)
|
||||
- File downloads: Direct links to `.exe`, `.md`, `.pdf`, etc. (NO auth required)
|
||||
- Security model: Must login to see file list, but downloads are public once you know the filename
|
||||
|
||||
**Key Files:**
|
||||
- `/www/wwwroot/soft_download/index.html` - Main file browser
|
||||
- `/www/wwwroot/soft_download/login.html` - Login page
|
||||
- `/www/wwwroot/soft_download/api/validate` - Auth validation endpoint
|
||||
|
||||
## Important Notes
|
||||
|
||||
- When making Tengine changes, ALWAYS edit `/etc/tengine/conf.d/thereisnospoonadu.conf`
|
||||
- Test against the actual domain: `curl -I https://www.thereisnospoonadu.com/filename`
|
||||
- PHP is NOT configured/working on this server - use Tengine-only solutions
|
||||
- Large files (170MB+ exe) need native browser downloads, not blob/XHR approach
|
||||
|
||||
## Verification Checklist (Before Making Changes)
|
||||
|
||||
```bash
|
||||
# 1. Verify which Tengine config is active
|
||||
tengine -T | grep -A 30 'server_name.*thereisnospoonadu'
|
||||
|
||||
# 2. Test current behavior against actual domain
|
||||
curl -I https://www.thereisnospoonadu.com/版本升级说明（用记事本打开）.md
|
||||
|
||||
# 3. After changes: test and reload
|
||||
tengine -t && tengine -s reload
|
||||
```
|
||||
|
||||
## Troubleshooting (Only When Issues Occur)
|
||||
|
||||
```bash
|
||||
# Find log paths from config
|
||||
grep 'access_log\|error_log' /etc/tengine/conf.d/thereisnospoonadu.conf
|
||||
```
|
||||
|
||||
**Note:** Logs are automatically rotated daily, keeping 7 days of history (compressed).
|
||||
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env python3
|
||||
import re
|
||||
import os
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
from datetime import datetime, timedelta
|
||||
from collections import defaultdict
|
||||
|
||||
LOG_FILE = "/usr/local/nginx/logs/download_access.log"
|
||||
|
||||
LOG_RE = re.compile(
|
||||
r'(?P<ip>\S+) - (?P<user>\S+) \[(?P<time>[^\]]+)\] "(?P<request>[^"]*)" '
|
||||
r'(?P<status>\d+) (?P<bytes>\d+) "(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
|
||||
)
|
||||
|
||||
DOWNLOAD_EXTS = re.compile(
|
||||
r'\.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)(\?|$)',
|
||||
re.IGNORECASE
|
||||
)
|
||||
|
||||
def fmt_bytes(b):
|
||||
b = int(b)
|
||||
for unit in ['B', 'KB', 'MB', 'GB']:
|
||||
if b < 1024:
|
||||
return "{:.1f} {}".format(b, unit)
|
||||
b /= 1024
|
||||
return "{:.1f} TB".format(b)
|
||||
|
||||
def parse_time(s):
|
||||
try:
|
||||
return datetime.strptime(s, "%d/%b/%Y:%H:%M:%S %z")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def parse_logs():
|
||||
entries = []
|
||||
try:
|
||||
with open(LOG_FILE, 'r', errors='replace') as f:
|
||||
for line in f:
|
||||
m = LOG_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
user = m.group('user')
|
||||
entries.append({
|
||||
'ip': m.group('ip'),
|
||||
'user': user,
|
||||
'time': parse_time(m.group('time')),
|
||||
'request': m.group('request'),
|
||||
'status': int(m.group('status')),
|
||||
'bytes': int(m.group('bytes')),
|
||||
})
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return entries
|
||||
|
||||
def build_stats(entries):
|
||||
users = defaultdict(lambda: {
|
||||
'logins': [], 'downloads': [], 'total_bytes': 0, 'ips': set()
|
||||
})
|
||||
all_downloads = []
|
||||
|
||||
for e in entries:
|
||||
u = e['user']
|
||||
t = e['time']
|
||||
req = e['request']
|
||||
parts = req.split(' ')
|
||||
path = parts[1] if len(parts) >= 2 else req
|
||||
|
||||
# For anonymous download entries, extract username from ?u= query param
|
||||
if u == '-' and DOWNLOAD_EXTS.search(path) and e['status'] in (200, 206):
|
||||
m = re.search(r'[?&]u=([^&\s]+)', path)
|
||||
if m:
|
||||
try:
|
||||
from urllib.parse import unquote
|
||||
u = unquote(m.group(1))
|
||||
except Exception:
|
||||
u = m.group(1)
|
||||
else:
|
||||
continue
|
||||
|
||||
if u == '-':
|
||||
continue
|
||||
|
||||
users[u]['ips'].add(e['ip'])
|
||||
|
||||
# Count login: either /api/validate or /list/ success
|
||||
if e['status'] == 200 and t:
|
||||
if '/api/validate' in path or path.rstrip('/') == '/list':
|
||||
users[u]['logins'].append(t)
|
||||
|
||||
if DOWNLOAD_EXTS.search(path) and e['status'] in (200, 206):
|
||||
filename = path.split('/')[-1].split('?')[0]
|
||||
try:
|
||||
from urllib.parse import unquote
|
||||
filename = unquote(filename)
|
||||
# Tengine logs non-ASCII bytes as \xNN sequences; decode them as UTF-8
|
||||
def _decode_nginx_escapes(s):
|
||||
parts = re.split(r'((?:\\x[0-9A-Fa-f]{2})+)', s)
|
||||
out = []
|
||||
for part in parts:
|
||||
if part.startswith('\\x'):
|
||||
raw = bytes(int(h, 16) for h in re.findall(r'\\x([0-9A-Fa-f]{2})', part))
|
||||
out.append(raw.decode('utf-8', errors='replace'))
|
||||
else:
|
||||
out.append(part)
|
||||
return ''.join(out)
|
||||
filename = _decode_nginx_escapes(filename)
|
||||
except Exception:
|
||||
pass
|
||||
users[u]['downloads'].append({'time': t, 'file': filename, 'bytes': e['bytes']})
|
||||
users[u]['total_bytes'] += e['bytes']
|
||||
all_downloads.append({'user': u, 'time': t, 'file': filename, 'bytes': e['bytes']})
|
||||
|
||||
def sort_key(x):
|
||||
t = x['time']
|
||||
if t is None:
|
||||
return datetime.min.replace(tzinfo=None)
|
||||
try:
|
||||
return t.replace(tzinfo=None)
|
||||
except Exception:
|
||||
return datetime.min.replace(tzinfo=None)
|
||||
|
||||
all_downloads.sort(key=sort_key, reverse=True)
|
||||
return users, all_downloads
|
||||
|
||||
def render_html(users, all_downloads):
|
||||
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
user_rows = ""
|
||||
for uname in sorted(users.keys()):
|
||||
data = users[uname]
|
||||
logins = sorted(data['logins'], reverse=True)
|
||||
last_login = logins[0].strftime("%Y-%m-%d %H:%M") if logins else "—"
|
||||
login_count = len(logins)
|
||||
dl_count = len(data['downloads'])
|
||||
total_data = fmt_bytes(data['total_bytes'])
|
||||
ips = ", ".join(sorted(data['ips']))
|
||||
|
||||
if logins:
|
||||
delta = datetime.now(logins[0].tzinfo) - logins[0]
|
||||
is_active = delta < timedelta(hours=24)
|
||||
else:
|
||||
is_active = False
|
||||
|
||||
status_class = "active" if is_active else "inactive"
|
||||
status_label = "Active 24h" if is_active else "Inactive"
|
||||
|
||||
user_rows += (
|
||||
"<tr>"
|
||||
"<td><span class='username'>{}</span></td>"
|
||||
"<td><span class='badge {}'>{}</span></td>"
|
||||
"<td>{}</td>"
|
||||
"<td class='num'>{}</td>"
|
||||
"<td class='num'>{}</td>"
|
||||
"<td class='num'>{}</td>"
|
||||
"<td class='ip'>{}</td>"
|
||||
"</tr>"
|
||||
).format(uname, status_class, status_label, last_login,
|
||||
login_count, dl_count, total_data, ips)
|
||||
|
||||
dl_rows = ""
|
||||
for d in all_downloads[:100]:
|
||||
t = d['time'].strftime("%Y-%m-%d %H:%M") if d['time'] else "—"
|
||||
dl_rows += (
|
||||
"<tr>"
|
||||
"<td>{}</td>"
|
||||
"<td><span class='username'>{}</span></td>"
|
||||
"<td class='filename'>{}</td>"
|
||||
"<td class='num'>{}</td>"
|
||||
"</tr>"
|
||||
).format(t, d['user'], d['file'], fmt_bytes(d['bytes']))
|
||||
|
||||
total_users = len(users)
|
||||
total_dls = sum(len(v['downloads']) for v in users.values())
|
||||
total_data_all = fmt_bytes(sum(v['total_bytes'] for v in users.values()))
|
||||
|
||||
no_users = "<tr><td colspan='7' class='empty'>No authenticated activity yet</td></tr>"
|
||||
no_dls = "<tr><td colspan='4' class='empty'>No downloads recorded yet</td></tr>"
|
||||
|
||||
html = """<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta http-equiv="refresh" content="60">
|
||||
<title>Admin Dashboard</title>
|
||||
<style>
|
||||
* {{ box-sizing: border-box; margin: 0; padding: 0; }}
|
||||
body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #f1f5f9; color: #1e293b; min-height: 100vh; }}
|
||||
header {{ background: #ffffff; border-bottom: 1px solid #e2e8f0; padding: 20px 32px; display: flex; align-items: center; justify-content: space-between; }}
|
||||
header h1 {{ font-size: 1.4rem; font-weight: 600; color: #7c3aed; }}
|
||||
.refresh-note {{ font-size: 0.75rem; color: #94a3b8; }}
|
||||
.stats-bar {{ display: flex; gap: 16px; padding: 24px 32px; }}
|
||||
.stat-card {{ background: #ffffff; border: 1px solid #e2e8f0; border-radius: 10px; padding: 18px 24px; flex: 1; }}
|
||||
.stat-card .label {{ font-size: 0.75rem; color: #94a3b8; text-transform: uppercase; letter-spacing: 0.05em; }}
|
||||
.stat-card .value {{ font-size: 2rem; font-weight: 700; color: #7c3aed; margin-top: 4px; }}
|
||||
.section {{ padding: 0 32px 32px; }}
|
||||
.section h2 {{ font-size: 1rem; font-weight: 600; color: #64748b; margin-bottom: 12px; text-transform: uppercase; letter-spacing: 0.05em; }}
|
||||
table {{ width: 100%; border-collapse: collapse; background: #ffffff; border: 1px solid #e2e8f0; border-radius: 10px; overflow: hidden; }}
|
||||
th {{ background: #f8fafc; padding: 12px 16px; text-align: left; font-size: 0.75rem; color: #94a3b8; text-transform: uppercase; letter-spacing: 0.05em; font-weight: 600; }}
|
||||
td {{ padding: 12px 16px; border-top: 1px solid #f1f5f9; font-size: 0.875rem; }}
|
||||
tr:hover td {{ background: #f8fafc; }}
|
||||
.username {{ color: #2563eb; font-weight: 500; }}
|
||||
.filename {{ color: #059669; font-family: monospace; font-size: 0.8rem; }}
|
||||
.ip {{ color: #64748b; font-size: 0.8rem; font-family: monospace; }}
|
||||
.num {{ text-align: right; color: #1e293b; }}
|
||||
.badge {{ padding: 2px 10px; border-radius: 999px; font-size: 0.7rem; font-weight: 600; }}
|
||||
.badge.active {{ background: #d1fae5; color: #065f46; }}
|
||||
.badge.inactive {{ background: #f1f5f9; color: #94a3b8; }}
|
||||
.empty {{ text-align: center; color: #94a3b8; padding: 32px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Admin Dashboard — thereisnospoonadu.com</h1>
|
||||
<span class="refresh-note">Last updated: {now} | Auto-refresh every 60s</span>
|
||||
</header>
|
||||
|
||||
<div class="stats-bar">
|
||||
<div class="stat-card"><div class="label">Total Users</div><div class="value">{total_users}</div></div>
|
||||
<div class="stat-card"><div class="label">Total Downloads</div><div class="value">{total_dls}</div></div>
|
||||
<div class="stat-card"><div class="label">Total Data Served</div><div class="value">{total_data_all}</div></div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>User Summary</h2>
|
||||
<table>
|
||||
<thead><tr>
|
||||
<th>Username</th><th>Status</th><th>Last Login</th>
|
||||
<th style="text-align:right">Logins</th>
|
||||
<th style="text-align:right">Downloads</th>
|
||||
<th style="text-align:right">Data Used</th>
|
||||
<th>IP Address(es)</th>
|
||||
</tr></thead>
|
||||
<tbody>{user_rows}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Download History (last 100)</h2>
|
||||
<table>
|
||||
<thead><tr><th>Time</th><th>User</th><th>File</th><th style="text-align:right">Size</th></tr></thead>
|
||||
<tbody>{dl_rows}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</body>
|
||||
</html>""".format(
|
||||
now=now,
|
||||
total_users=total_users,
|
||||
total_dls=total_dls,
|
||||
total_data_all=total_data_all,
|
||||
user_rows=user_rows if user_rows else no_users,
|
||||
dl_rows=dl_rows if dl_rows else no_dls,
|
||||
)
|
||||
return html
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
entries = parse_logs()
|
||||
users, all_downloads = build_stats(entries)
|
||||
html = render_html(users, all_downloads)
|
||||
body = html.encode('utf-8')
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/html; charset=utf-8')
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
server = HTTPServer(('127.0.0.1', 8765), Handler)
|
||||
print("Admin dashboard running on 127.0.0.1:8765")
|
||||
server.serve_forever()
|
||||
@@ -0,0 +1,14 @@
|
||||
<?php
|
||||
/**
|
||||
* Used by Nginx auth_request. Returns 200 if session valid, 401 otherwise.
|
||||
* Also returns X-Auth-User header so Tengine can log the authenticated username.
|
||||
* Deploy to /www/wwwroot/soft_download/auth_check.php
|
||||
*/
|
||||
session_start();
|
||||
if (!empty($_SESSION['soft_download_user'])) {
|
||||
header('X-Auth-User: ' . $_SESSION['soft_download_user']);
|
||||
http_response_code(200);
|
||||
exit;
|
||||
}
|
||||
http_response_code(401);
|
||||
exit;
|
||||
@@ -0,0 +1,197 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>软件下载页面</title>
|
||||
<style>
|
||||
body{font-size:18px!important;line-height:2;font-family:sans-serif;margin:.5em 1.5em 2em}
|
||||
h1{font-size:24px!important;margin-top:0;margin-bottom:.5em;border-bottom:2px solid #ccc;padding-bottom:.3em}
|
||||
.breadcrumb{font-size:18px;margin-bottom:.75em}
|
||||
.breadcrumb a{color:#0066cc;cursor:pointer}
|
||||
.tools-nav{margin-bottom:.75em}
|
||||
.tools-link{text-decoration:underline;font-style:italic}
|
||||
.loading{font-size:20px;color:#666}
|
||||
.error{color:#c00;font-size:18px}
|
||||
.user-bar{margin-bottom:.4em;padding:.25em 0;display:flex;align-items:center;justify-content:flex-end;gap:1em}
|
||||
.user-bar .logged-in{color:green}
|
||||
.user-bar .logout-btn{padding:.3em .8em;font-size:16px;cursor:pointer;background:#0066cc;color:#fff;border:none;border-radius:4px}
|
||||
.user-bar .logout-btn:hover{background:#0052a3}
|
||||
.file-list{margin-top:.25em;overflow-x:auto}
|
||||
.file-table{border-collapse:collapse;width:75%;table-layout:fixed}
|
||||
.file-table th,.file-table td{padding:.45em .65em;border-bottom:1px solid #ddd;text-align:left;vertical-align:middle}
|
||||
.file-table thead th{font-size:17px;font-weight:600;color:#333;background:#f5f5f5;border-bottom:2px solid #ccc}
|
||||
.file-table .col-name{width:70%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
|
||||
.file-table .col-size{width:10%;white-space:nowrap;font-size:16px;color:#555}
|
||||
.file-table .col-mtime{width:20%;white-space:nowrap;font-size:16px;color:#555}
|
||||
.file-table a{font-size:19px!important;color:#0066cc;text-decoration:none}
|
||||
.file-table a:hover{text-decoration:underline}
|
||||
.file-table .dir{font-weight:bold;cursor:pointer}
|
||||
.file-table .empty-msg{text-align:center;color:#666;padding:1em}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="user-bar">
|
||||
<span class="logged-in">当前已登录: <span id="username-display"></span></span>
|
||||
<button class="logout-btn" id="logout-btn">注销</button>
|
||||
</div>
|
||||
<h1>文件列表 <span id="path-display"></span></h1>
|
||||
<div class="breadcrumb" id="breadcrumb"></div>
|
||||
<div id="content">
|
||||
<div class="loading" id="loading">正在加载...</div>
|
||||
<div class="file-list" id="file-list" style="display:none"></div>
|
||||
<div class="error" id="error" style="display:none"></div>
|
||||
</div>
|
||||
<script>
|
||||
(function(){
|
||||
var authHeader=sessionStorage.getItem("auth");
|
||||
var username=sessionStorage.getItem("username");
|
||||
if(!authHeader){
|
||||
window.location.href="/login.html?return="+encodeURIComponent(window.location.pathname);
|
||||
return;
|
||||
}
|
||||
document.getElementById("username-display").textContent=username||"";
|
||||
loadFileList(getPath());
|
||||
|
||||
function getPath(){
|
||||
var p=window.location.pathname;
|
||||
if(p==="/"||p==="")return"";
|
||||
if(p.endsWith("/"))p=p.slice(0,-1);
|
||||
return p.replace(/^\//,"");
|
||||
}
|
||||
|
||||
function shouldHideFile(filename){
|
||||
var hiddenPatterns=[/\.php$/i,/\.html$/i,/\.backup$/i,/^api$/i,/^auth_check/i,/^login/i,/^logout/i,/^index\./i];
|
||||
return hiddenPatterns.some(function(pattern){return pattern.test(filename)});
|
||||
}
|
||||
|
||||
function escapeHtml(s){
|
||||
if(s==null||s==="")return"";
|
||||
return String(s).replace(/&/g,"&").replace(/</g,"<").replace(/>/g,">").replace(/"/g,""");
|
||||
}
|
||||
|
||||
function formatSize(bytes){
|
||||
if(!bytes)return"-";
|
||||
if(bytes<1024)return bytes+" B";
|
||||
if(bytes<1048576)return(bytes/1024).toFixed(1)+" K";
|
||||
if(bytes<1073741824)return(bytes/1048576).toFixed(1)+" M";
|
||||
return(bytes/1073741824).toFixed(1)+" G";
|
||||
}
|
||||
|
||||
function formatMtime(mtime){
|
||||
if(!mtime)return"";
|
||||
try{
|
||||
var d=new Date(mtime);
|
||||
return isNaN(d)?mtime:d.toLocaleString();
|
||||
}catch(e){
|
||||
return mtime;
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeName(name){
|
||||
return (name||"").replace(/\/$/,"");
|
||||
}
|
||||
|
||||
function isDirectory(item){
|
||||
return item.type==="directory"||/\/$/.test(item.name||"");
|
||||
}
|
||||
|
||||
function renderBreadcrumb(path){
|
||||
var parts=path?path.split("/").filter(Boolean):[];
|
||||
if(parts.length===0){
|
||||
document.getElementById("breadcrumb").innerHTML="";
|
||||
return;
|
||||
}
|
||||
var inTools=parts[0]==="tools";
|
||||
var html='<a'+(inTools?' class="tools-link"':'')+' onclick="navigateTo(\'/\')">根目录</a>';
|
||||
var acc="";
|
||||
for(var i=0;i<parts.length;i++){
|
||||
acc+=(acc?"/":"")+parts[i];
|
||||
var linkClass=parts[i]==="tools"?' class="tools-link"':'';
|
||||
html+=' / <a'+linkClass+' onclick="navigateTo(\'/'+acc+'/\')">'+escapeHtml(parts[i])+"</a>";
|
||||
}
|
||||
document.getElementById("breadcrumb").innerHTML=html;
|
||||
}
|
||||
|
||||
function buildToolsNavHtml(path,items){
|
||||
if(path)return "";
|
||||
var hasTools=items.some(function(item){return isDirectory(item)&&normalizeName(item.name)==="tools"});
|
||||
if(!hasTools)return "";
|
||||
return '<div class="breadcrumb tools-nav"><a class="tools-link" onclick="navigateTo(\'/tools/\')">tools</a></div>';
|
||||
}
|
||||
|
||||
window.navigateTo=function(path){
|
||||
window.location.href=path;
|
||||
};
|
||||
|
||||
function loadFileList(path){
|
||||
var loadingEl=document.getElementById("loading");
|
||||
var errorEl=document.getElementById("error");
|
||||
var listEl=document.getElementById("file-list");
|
||||
loadingEl.style.display="block";
|
||||
errorEl.style.display="none";
|
||||
listEl.style.display="none";
|
||||
|
||||
fetch("/list/"+(path?path+"/":""),{
|
||||
headers:{Authorization:authHeader}
|
||||
})
|
||||
.then(function(r){
|
||||
if(r.status===401){
|
||||
sessionStorage.clear();
|
||||
window.location.href="/login.html";
|
||||
return;
|
||||
}
|
||||
if(!r.ok)throw new Error("加载失败");
|
||||
return r.json();
|
||||
})
|
||||
.then(function(data){
|
||||
if(!data)return;
|
||||
loadingEl.style.display="none";
|
||||
document.getElementById("path-display").textContent=path?"/"+path:"";
|
||||
renderBreadcrumb(path);
|
||||
|
||||
var rows="";
|
||||
var items=Array.isArray(data)?data:[];
|
||||
var toolsHtml=buildToolsNavHtml(path,items);
|
||||
items.forEach(function(item){
|
||||
if(shouldHideFile(item.name))return;
|
||||
var name=normalizeName(item.name);
|
||||
if(!path&&isDirectory(item)&&name==="tools")return;
|
||||
var fullPath=(path?path+"/":"")+name;
|
||||
var isDir=isDirectory(item);
|
||||
var size=formatSize(item.size);
|
||||
var mtime=formatMtime(item.mtime);
|
||||
var nameEsc=escapeHtml(name)+(isDir?"/":"");
|
||||
|
||||
if(isDir){
|
||||
var navPath="/"+fullPath+"/";
|
||||
rows+="<tr><td class=\"col-name\"><a onclick='navigateTo("+JSON.stringify(navPath)+")' class=\"dir\" href=\"javascript:void(0)\">"+nameEsc+"</a></td><td class=\"col-size\">"+escapeHtml(size)+"</td><td class=\"col-mtime\">"+escapeHtml(mtime)+"</td></tr>";
|
||||
}else{
|
||||
rows+="<tr><td class=\"col-name\"><a href=\"/"+fullPath+"?u="+encodeURIComponent(username)+"\" download>"+nameEsc+"</a></td><td class=\"col-size\">"+escapeHtml(size)+"</td><td class=\"col-mtime\">"+escapeHtml(mtime)+"</td></tr>";
|
||||
}
|
||||
});
|
||||
|
||||
var tableStart='<table class="file-table"><thead><tr><th class="col-name">文件名称</th><th class="col-size">文件大小</th><th class="col-mtime">上传时间</th></tr></thead><tbody>';
|
||||
var tableEnd="</tbody></table>";
|
||||
if(!rows){
|
||||
listEl.innerHTML=toolsHtml+tableStart+'<tr><td class="empty-msg" colspan="3">目录为空</td></tr>'+tableEnd;
|
||||
}else{
|
||||
listEl.innerHTML=toolsHtml+tableStart+rows+tableEnd;
|
||||
}
|
||||
listEl.style.display="block";
|
||||
})
|
||||
.catch(function(err){
|
||||
loadingEl.style.display="none";
|
||||
errorEl.textContent="加载失败: "+(err.message||"未知错误");
|
||||
errorEl.style.display="block";
|
||||
});
|
||||
}
|
||||
|
||||
document.getElementById("logout-btn").addEventListener("click",function(){
|
||||
sessionStorage.clear();
|
||||
window.location.href="/login.html";
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,103 @@
|
||||
<!DOCTYPE html>
|
||||
<!-- Deploy: copy to /www/wwwroot/soft_download/login.html on the server. Must be excluded from auth_basic in Nginx. -->
|
||||
<html lang="zh">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>登录 - 软件下载页面</title>
|
||||
<style>
|
||||
* { box-sizing: border-box; }
|
||||
body { font-size: 18px; line-height: 1.6; font-family: sans-serif; margin: 0; padding: 2em; min-height: 100vh; display: flex; align-items: center; justify-content: center; background: #f5f5f5; }
|
||||
.login-box { background: #fff; padding: 2em 2.5em; border-radius: 8px; box-shadow: 0 2px 10px rgba(0,0,0,0.1); max-width: 380px; width: 100%; }
|
||||
h1 { font-size: 22px; margin: 0 0 1.2em 0; color: #333; border-bottom: 2px solid #ccc; padding-bottom: 0.4em; }
|
||||
.form-group { margin-bottom: 1.2em; }
|
||||
.form-group label { display: block; margin-bottom: 0.3em; color: #555; font-size: 16px; }
|
||||
.form-group input { width: 100%; padding: 0.5em 0.8em; font-size: 16px; border: 1px solid #ccc; border-radius: 4px; }
|
||||
.form-group input:focus { outline: none; border-color: #0066cc; }
|
||||
.login-btn { width: 100%; padding: 0.6em 1em; font-size: 18px; background: #0066cc; color: #fff; border: none; border-radius: 4px; cursor: pointer; }
|
||||
.login-btn:hover { background: #0052a3; }
|
||||
.error-msg { color: #c00; font-size: 14px; margin-top: 0.5em; display: none; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="login-box">
|
||||
<h1>登录</h1>
|
||||
<form id="login-form">
|
||||
<div class="form-group">
|
||||
<label for="username">用户名</label>
|
||||
<input type="text" id="username" name="username" required autocomplete="username">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="password">密码</label>
|
||||
<input type="password" id="password" name="password" required autocomplete="current-password">
|
||||
</div>
|
||||
<div class="error-msg" id="error-msg"></div>
|
||||
<button type="submit" class="login-btn">登录</button>
|
||||
</form>
|
||||
</div>
|
||||
<script>
|
||||
(function() {
|
||||
var form = document.getElementById('login-form');
|
||||
var errorEl = document.getElementById('error-msg');
|
||||
|
||||
function getReturnPath() {
|
||||
var params = new URLSearchParams(window.location.search);
|
||||
var ret = params.get('return');
|
||||
return ret && ret.length > 0 ? ret : '/';
|
||||
}
|
||||
|
||||
form.addEventListener('submit', function(e) {
|
||||
e.preventDefault();
|
||||
var user = document.getElementById('username').value.trim();
|
||||
var pass = document.getElementById('password').value;
|
||||
if (!user || !pass) {
|
||||
errorEl.textContent = '请输入用户名和密码';
|
||||
errorEl.style.display = 'block';
|
||||
return;
|
||||
}
|
||||
var returnPath = getReturnPath();
|
||||
if (!returnPath.startsWith('/')) returnPath = '/' + returnPath;
|
||||
|
||||
// Submit to login.php via POST
|
||||
var formData = new FormData();
|
||||
formData.append('username', user);
|
||||
formData.append('password', pass);
|
||||
formData.append('return', returnPath);
|
||||
|
||||
fetch('/login.php', {
|
||||
method: 'POST',
|
||||
body: formData,
|
||||
credentials: 'same-origin'
|
||||
})
|
||||
.then(function(response) {
|
||||
if (response.redirected) {
|
||||
window.location.href = response.url;
|
||||
} else if (response.ok) {
|
||||
window.location.href = returnPath;
|
||||
} else {
|
||||
return response.text();
|
||||
}
|
||||
})
|
||||
.then(function(html) {
|
||||
if (html) {
|
||||
// Login failed, extract error message from response
|
||||
var parser = new DOMParser();
|
||||
var doc = parser.parseFromString(html, 'text/html');
|
||||
var errorMsg = doc.querySelector('.error-msg');
|
||||
if (errorMsg) {
|
||||
errorEl.textContent = errorMsg.textContent;
|
||||
} else {
|
||||
errorEl.textContent = '登录失败，请重试';
|
||||
}
|
||||
errorEl.style.display = 'block';
|
||||
}
|
||||
})
|
||||
.catch(function(err) {
|
||||
errorEl.textContent = '网络错误，请重试';
|
||||
errorEl.style.display = 'block';
|
||||
});
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,91 @@
|
||||
<?php
|
||||
/**
|
||||
* Session-based login. POST: validate against .htpasswd and set session. GET: show form.
|
||||
* Deploy to /www/wwwroot/soft_download/login.php
|
||||
* Requires: PHP with session support, htpasswd in PATH for validation.
|
||||
*/
|
||||
session_start();
|
||||
|
||||
$HTPASSWD_FILE = '/usr/local/nginx/conf/.htpasswd';
|
||||
$HTPASSWD_CMD = '/usr/bin/htpasswd'; // or 'htpasswd' if in PATH
|
||||
|
||||
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
|
||||
$user = isset($_POST['username']) ? trim((string) $_POST['username']) : '';
|
||||
$pass = isset($_POST['password']) ? (string) $_POST['password'] : '';
|
||||
$return_path = isset($_POST['return']) ? trim((string) $_POST['return']) : '/';
|
||||
if (!strlen($return_path) || $return_path[0] !== '/') {
|
||||
$return_path = '/';
|
||||
}
|
||||
|
||||
$error = '';
|
||||
if ($user !== '' && $pass !== '' && is_readable($HTPASSWD_FILE)) {
|
||||
$user_esc = escapeshellarg($user);
|
||||
$pass_esc = escapeshellarg($pass);
|
||||
$file_esc = escapeshellarg($HTPASSWD_FILE);
|
||||
$cmd = sprintf('%s -vb %s %s %s 2>/dev/null', $HTPASSWD_CMD, $file_esc, $user_esc, $pass_esc);
|
||||
exec($cmd, $out, $code);
|
||||
if ($code === 0) {
|
||||
$_SESSION['soft_download_user'] = $user;
|
||||
header('Location: ' . $return_path);
|
||||
exit;
|
||||
}
|
||||
$error = '用户名或密码错误';
|
||||
} else {
|
||||
if ($user === '' && $pass === '') {
|
||||
$error = '请输入用户名和密码';
|
||||
} else {
|
||||
$error = '用户名或密码错误';
|
||||
}
|
||||
}
|
||||
} else {
|
||||
$error = '';
|
||||
$return_path = isset($_GET['return']) ? trim((string) $_GET['return']) : '/';
|
||||
if (!strlen($return_path) || $return_path[0] !== '/') {
|
||||
$return_path = '/';
|
||||
}
|
||||
}
|
||||
|
||||
$return_esc = htmlspecialchars($return_path, ENT_QUOTES, 'UTF-8');
|
||||
$error_esc = htmlspecialchars($error ?? '', ENT_QUOTES, 'UTF-8');
|
||||
?>
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>登录 - 软件下载页面</title>
|
||||
<style>
|
||||
* { box-sizing: border-box; }
|
||||
body { font-size: 18px; line-height: 1.6; font-family: sans-serif; margin: 0; padding: 2em; min-height: 100vh; display: flex; align-items: center; justify-content: center; background: #f5f5f5; }
|
||||
.login-box { background: #fff; padding: 2em 2.5em; border-radius: 8px; box-shadow: 0 2px 10px rgba(0,0,0,0.1); max-width: 380px; width: 100%; }
|
||||
h1 { font-size: 22px; margin: 0 0 1.2em 0; color: #333; border-bottom: 2px solid #ccc; padding-bottom: 0.4em; }
|
||||
.form-group { margin-bottom: 1.2em; }
|
||||
.form-group label { display: block; margin-bottom: 0.3em; color: #555; font-size: 16px; }
|
||||
.form-group input { width: 100%; padding: 0.5em 0.8em; font-size: 16px; border: 1px solid #ccc; border-radius: 4px; }
|
||||
.form-group input:focus { outline: none; border-color: #0066cc; }
|
||||
.login-btn { width: 100%; padding: 0.6em 1em; font-size: 18px; background: #0066cc; color: #fff; border: none; border-radius: 4px; cursor: pointer; }
|
||||
.login-btn:hover { background: #0052a3; }
|
||||
.error-msg { color: #c00; font-size: 14px; margin-top: 0.5em; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="login-box">
|
||||
<h1>登录</h1>
|
||||
<form method="post" action="login.php">
|
||||
<input type="hidden" name="return" value="<?php echo $return_esc; ?>">
|
||||
<div class="form-group">
|
||||
<label for="username">用户名</label>
|
||||
<input type="text" id="username" name="username" required autocomplete="username" value="<?php echo htmlspecialchars($_POST['username'] ?? '', ENT_QUOTES, 'UTF-8'); ?>">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="password">密码</label>
|
||||
<input type="password" id="password" name="password" required autocomplete="current-password">
|
||||
</div>
|
||||
<?php if ($error_esc !== ''): ?>
|
||||
<div class="error-msg"><?php echo $error_esc; ?></div>
|
||||
<?php endif; ?>
|
||||
<button type="submit" class="login-btn">登录</button>
|
||||
</form>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,14 @@
|
||||
<?php
|
||||
/**
|
||||
* Destroy session and redirect to login.
|
||||
* Deploy to /www/wwwroot/soft_download/logout.php
|
||||
*/
|
||||
session_start();
|
||||
$_SESSION = [];
|
||||
if (ini_get('session.use_cookies')) {
|
||||
$p = session_get_cookie_params();
|
||||
setcookie(session_name(), '', time() - 3600, $p['path'], $p['domain'], $p['secure'], $p['httponly']);
|
||||
}
|
||||
session_destroy();
|
||||
header('Location: login.php');
|
||||
exit;
|
||||
@@ -0,0 +1,168 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Email sender for user credentials
|
||||
Sends username and password to all users in user_data.csv
|
||||
"""
|
||||
|
||||
import csv
|
||||
import smtplib
|
||||
import time
|
||||
import os
|
||||
from email.mime.text import MIMEText
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.header import Header
|
||||
from datetime import datetime
|
||||
|
||||
# ============ CONFIGURATION ============
|
||||
# TODO: Fill in your NetEase email credentials before running
|
||||
SENDER_EMAIL = "sofomoupc@163.com" # Your NetEase email address
|
||||
SMTP_PASSWORD = "FERFPaJFJ89ckKw5" # SMTP authorization code (NOT login password)
|
||||
SMTP_SERVER = "smtp.163.com" # Use smtp.126.com for @126.com, smtp.yeah.net for @yeah.net
|
||||
SMTP_PORT = 465 # SSL port
|
||||
|
||||
# Email settings
|
||||
EMAIL_SUBJECT = "“海察”软件下载网址用户名、密码"
|
||||
# Use absolute path to CSV file
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
CSV_FILE = os.path.join(SCRIPT_DIR, "user_data.csv")
|
||||
|
||||
# Delay between emails (seconds) to avoid being flagged as spam
|
||||
DELAY_BETWEEN_EMAILS = 2
|
||||
# =======================================
|
||||
|
||||
|
||||
def create_email_body(username, password):
|
||||
"""Create email body with user credentials"""
|
||||
return f"""用户名：{username}
|
||||
密码：{password}"""
|
||||
|
||||
|
||||
def send_email(sender, password, recipient, subject, body):
|
||||
"""Send email via NetEase SMTP server"""
|
||||
try:
|
||||
# Create message
|
||||
msg = MIMEMultipart()
|
||||
msg['From'] = Header(sender)
|
||||
msg['To'] = Header(recipient)
|
||||
msg['Subject'] = Header(subject, 'utf-8')
|
||||
|
||||
# Attach body
|
||||
msg.attach(MIMEText(body, 'plain', 'utf-8'))
|
||||
|
||||
# Connect to SMTP server with SSL
|
||||
server = smtplib.SMTP_SSL(SMTP_SERVER, SMTP_PORT)
|
||||
server.login(sender, password)
|
||||
|
||||
# Send email
|
||||
server.sendmail(sender, recipient, msg.as_string())
|
||||
server.quit()
|
||||
|
||||
return True, None
|
||||
except Exception as e:
|
||||
return False, str(e)
|
||||
|
||||
|
||||
def read_users_from_csv(filename):
|
||||
"""Read user data from CSV file"""
|
||||
users = []
|
||||
try:
|
||||
with open(filename, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
users.append({
|
||||
'username': row['username'],
|
||||
'passwd': row['passwd'],
|
||||
'email': row['email']
|
||||
})
|
||||
return users
|
||||
except Exception as e:
|
||||
print(f"Error reading CSV file: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function to send emails to all users"""
|
||||
print("=" * 60)
|
||||
print("User Credentials Email Sender")
|
||||
print("=" * 60)
|
||||
|
||||
# Validate configuration
|
||||
if SENDER_EMAIL == "your_email@163.com" or SMTP_PASSWORD == "your_smtp_auth_code":
|
||||
print("\n❌ ERROR: Please configure your email credentials first!")
|
||||
print("Edit the CONFIGURATION section at the top of this script:")
|
||||
print(" - SENDER_EMAIL: Your NetEase email address")
|
||||
print(" - SMTP_PASSWORD: Your SMTP authorization code")
|
||||
print(" - SMTP_SERVER: smtp.163.com (or smtp.126.com, smtp.yeah.net)")
|
||||
return
|
||||
|
||||
# Read users from CSV
|
||||
print(f"\n📂 Reading users from {CSV_FILE}...")
|
||||
users = read_users_from_csv(CSV_FILE)
|
||||
|
||||
if not users:
|
||||
print("❌ Failed to read users from CSV file")
|
||||
return
|
||||
|
||||
print(f"✓ Found {len(users)} users")
|
||||
|
||||
# Confirm before sending
|
||||
print(f"\n📧 Ready to send {len(users)} emails")
|
||||
print(f" From: {SENDER_EMAIL}")
|
||||
print(f" Subject: {EMAIL_SUBJECT}")
|
||||
response = input("\nProceed? (yes/no): ")
|
||||
|
||||
if response.lower() not in ['yes', 'y']:
|
||||
print("Cancelled.")
|
||||
return
|
||||
|
||||
# Send emails
|
||||
print(f"\n🚀 Sending emails...\n")
|
||||
success_count = 0
|
||||
failed_users = []
|
||||
|
||||
for i, user in enumerate(users, 1):
|
||||
username = user['username']
|
||||
passwd = user['passwd']
|
||||
email = user['email']
|
||||
|
||||
print(f"[{i}/{len(users)}] Sending to {username} ({email})...", end=" ")
|
||||
|
||||
# Create email body
|
||||
body = create_email_body(username, passwd)
|
||||
|
||||
# Send email
|
||||
success, error = send_email(SENDER_EMAIL, SMTP_PASSWORD, email, EMAIL_SUBJECT, body)
|
||||
|
||||
if success:
|
||||
print("✓")
|
||||
success_count += 1
|
||||
else:
|
||||
print(f"✗ Failed: {error}")
|
||||
failed_users.append({'user': username, 'email': email, 'error': error})
|
||||
|
||||
# Delay between emails
|
||||
if i < len(users):
|
||||
time.sleep(DELAY_BETWEEN_EMAILS)
|
||||
|
||||
# Summary
|
||||
print("\n" + "=" * 60)
|
||||
print(f"✓ Successfully sent: {success_count}/{len(users)}")
|
||||
|
||||
if failed_users:
|
||||
print(f"✗ Failed: {len(failed_users)}")
|
||||
print("\nFailed users:")
|
||||
for failed in failed_users:
|
||||
print(f" - {failed['user']} ({failed['email']}): {failed['error']}")
|
||||
|
||||
# Save failed users to log
|
||||
log_file = f"failed_emails_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
with open(log_file, 'w', encoding='utf-8') as f:
|
||||
for failed in failed_users:
|
||||
f.write(f"{failed['user']},{failed['email']},{failed['error']}\n")
|
||||
print(f"\n📝 Failed users saved to: {log_file}")
|
||||
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,158 @@
|
||||
import paramiko
|
||||
import sys
|
||||
|
||||
HOST = "39.102.124.161"
|
||||
USER = "root"
|
||||
PASS = "Beast2026"
|
||||
|
||||
VHOST_CONFIG = r"""# Redirect HTTP to HTTPS
|
||||
server {
|
||||
listen 80;
|
||||
server_name www.thereisnospoonadu.com thereisnospoonadu.com;
|
||||
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /var/www/certbot;
|
||||
allow all;
|
||||
}
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
# HTTPS Server
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name www.thereisnospoonadu.com thereisnospoonadu.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/www.thereisnospoonadu.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/www.thereisnospoonadu.com/privkey.pem;
|
||||
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
|
||||
ssl_prefer_server_ciphers on;
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 10m;
|
||||
|
||||
root /www/wwwroot/soft_download;
|
||||
index index.html index.htm;
|
||||
charset utf-8;
|
||||
|
||||
client_max_body_size 500M;
|
||||
client_body_timeout 300s;
|
||||
send_timeout 300s;
|
||||
keepalive_timeout 300s;
|
||||
proxy_read_timeout 300s;
|
||||
|
||||
location = / {
|
||||
try_files /index.html =404;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
location = /index.html {
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
location = /login.html {
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
location = /api/validate {
|
||||
auth_basic "App Download Portal";
|
||||
auth_basic_user_file /etc/tengine/.htpasswd;
|
||||
add_header Content-Type application/json;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
try_files /api/validate =404;
|
||||
}
|
||||
|
||||
location ^~ /list/ {
|
||||
index nothing_will_match;
|
||||
auth_basic "App Download Portal";
|
||||
auth_basic_user_file /etc/tengine/.htpasswd;
|
||||
alias /www/wwwroot/soft_download/;
|
||||
autoindex on;
|
||||
autoindex_exact_size off;
|
||||
autoindex_localtime on;
|
||||
autoindex_format json;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
location ~ \.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)$ {
|
||||
add_header Content-Disposition "attachment";
|
||||
add_header X-Content-Type-Options "nosniff";
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_basic "App Download Portal";
|
||||
auth_basic_user_file /etc/tengine/.htpasswd;
|
||||
try_files $uri $uri/ =404;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
access_log /var/log/tengine/download_access.log;
|
||||
error_log /var/log/tengine/download_error.log warn;
|
||||
server_tokens off;
|
||||
}
|
||||
"""
|
||||
|
||||
def run_cmd(ssh, cmd, desc=""):
|
||||
if desc:
|
||||
print(f"\n{'='*60}")
|
||||
print(f">>> {desc}")
|
||||
print(f"CMD: {cmd}")
|
||||
print('='*60)
|
||||
stdin, stdout, stderr = ssh.exec_command(cmd, timeout=30)
|
||||
out = stdout.read().decode('utf-8', errors='replace')
|
||||
err = stderr.read().decode('utf-8', errors='replace')
|
||||
rc = stdout.channel.recv_exit_status()
|
||||
if out:
|
||||
print("STDOUT:", out)
|
||||
if err:
|
||||
print("STDERR:", err)
|
||||
print(f"EXIT CODE: {rc}")
|
||||
return rc, out, err
|
||||
|
||||
def main():
|
||||
print(f"Connecting to {HOST}...")
|
||||
ssh = paramiko.SSHClient()
|
||||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
ssh.connect(HOST, username=USER, password=PASS, timeout=15)
|
||||
print("Connected.\n")
|
||||
|
||||
# Step 1: Write the vhost config
|
||||
print("="*60)
|
||||
print(">>> STEP 1: Write vhost config")
|
||||
print("="*60)
|
||||
run_cmd(ssh, "mkdir -p /etc/tengine/conf.d", "Ensure conf.d dir exists")
|
||||
|
||||
sftp = ssh.open_sftp()
|
||||
with sftp.open("/etc/tengine/conf.d/thereisnospoonadu.conf", "w") as f:
|
||||
f.write(VHOST_CONFIG)
|
||||
sftp.close()
|
||||
print("File written: /etc/tengine/conf.d/thereisnospoonadu.conf")
|
||||
|
||||
# Verify write
|
||||
run_cmd(ssh, "cat /etc/tengine/conf.d/thereisnospoonadu.conf", "Verify written file")
|
||||
|
||||
# Step 2: Test the config
|
||||
run_cmd(ssh, "tengine -t", "STEP 2: tengine -t (config test)")
|
||||
|
||||
# Step 3: Enable and start tengine
|
||||
run_cmd(ssh, "systemctl enable tengine && systemctl start tengine", "STEP 3: Enable and start tengine")
|
||||
|
||||
# Step 4: Verify running and listening
|
||||
run_cmd(ssh, "systemctl status tengine --no-pager", "STEP 4a: systemctl status tengine")
|
||||
run_cmd(ssh, "ss -tlnp | grep -E '80|443'", "STEP 4b: ss -tlnp ports 80/443")
|
||||
|
||||
# Step 5: Quick local test
|
||||
run_cmd(ssh,
|
||||
'curl -sk https://localhost/ -o /dev/null -w "%{http_code}" --resolve www.thereisnospoonadu.com:443:127.0.0.1 -H "Host: www.thereisnospoonadu.com" 2>/dev/null || echo "curl test done"',
|
||||
"STEP 5: Quick local curl test"
|
||||
)
|
||||
|
||||
ssh.close()
|
||||
print("\nAll steps complete.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,91 @@
|
||||
# Redirect all HTTP traffic to HTTPS (forces secure connections)
|
||||
server {
|
||||
listen 80;
|
||||
server_name www.thereisnospoonadu.com thereisnospoonadu.com;
|
||||
|
||||
# Certbot ACME challenge - must be served over HTTP for Let's Encrypt validation
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /var/www/certbot;
|
||||
allow all;
|
||||
}
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
# Custom log format: $remote_user is populated by auth_basic, captures real username
|
||||
log_format download '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
|
||||
|
||||
# HTTPS Server Configuration (core SSL setup + Basic Auth download portal)
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name www.thereisnospoonadu.com thereisnospoonadu.com;
|
||||
|
||||
# Path to your Certbot SSL certificate (unchanged—valid path)
|
||||
ssl_certificate /etc/letsencrypt/live/www.thereisnospoonadu.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/www.thereisnospoonadu.com/privkey.pem;
|
||||
|
||||
# Modern SSL security settings (A+ grade)
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
|
||||
ssl_prefer_server_ciphers on;
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 10m;
|
||||
|
||||
root /www/wwwroot/soft_download;
|
||||
index index.html index.htm;
|
||||
charset utf-8;
|
||||
|
||||
# Entry points: no auth required
|
||||
location = / {
|
||||
try_files /index.html =404;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
location = /index.html {
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
location = /login.html {
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
# API validation endpoint: HTTP Basic Auth
|
||||
location = /api/validate {
|
||||
auth_basic "App Download Portal";
|
||||
auth_basic_user_file /etc/tengine/.htpasswd;
|
||||
default_type text/plain;
|
||||
alias /www/wwwroot/soft_download/api/validate.txt;
|
||||
}
|
||||
|
||||
# JSON file listing: HTTP Basic Auth
|
||||
location /list/ {
|
||||
auth_basic "App Download Portal";
|
||||
auth_basic_user_file /etc/tengine/.htpasswd;
|
||||
alias /www/wwwroot/soft_download/;
|
||||
index nothing_will_match;
|
||||
autoindex on;
|
||||
autoindex_exact_size off;
|
||||
autoindex_localtime on;
|
||||
autoindex_format json;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
# File downloads: no auth required (username tracked via ?u= query param)
|
||||
location ~ \.(exe|md|pdf|zip|rar|7z|tar|gz|txt|doc|docx|xls|xlsx|ppt|pptx|deb)$ {
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
# All other paths: serve static files (no auth)
|
||||
location / {
|
||||
try_files $uri $uri/ =404;
|
||||
error_page 403 = /index.html;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
|
||||
access_log /usr/local/nginx/logs/download_access.log download;
|
||||
error_log /usr/local/nginx/logs/download_error.log warn;
|
||||
|
||||
# Security: Hide Nginx version
|
||||
server_tokens off;
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
username,passwd,email
|
||||
huangjing,1090,179864000@qq.com
|
||||
douzhn,1111,29867370@qq.com
|
||||
zhenglu,0872,18622306979@163.com
|
||||
zhangshh,1081,nkuzsh@sina.com
|
||||
qiangdq,1165,eacone@sina.com
|
||||
zhangwd,1381,zhangwendi0820@163.com
|
||||
chengqi,1481,531740421@qq.com
|
||||
lizj,1231,tjlizejun@163.com
|
||||
sunyl,1270,39204996@qq.com
|
||||
caichl,1296,cwtjwd@126.com
|
||||
lijy,0818,lijiayan222@qq.com
|
||||
chzhl,1295,46113395@qq.com
|
||||
qinglu,1230,luqingsmiles@foxmail.com
|
||||
renzw,1653,2734310562@qq.com
|
||||
songht,1215,luck_0076@163.com
|
||||
wangwx,1280,wenxingone@qq.com
|
||||
yangbo,1837,yangbo_6@126.com
|
||||
wangyi,0870,691511321@qq.com
|
||||
wangzhr,1407,396400640@qq.com
|
||||
weihq,1816,hailife@163.com
|
||||
yushj,1126,122758312@qq.com
|
||||
zhbl,1468,sdzbxiaolang@163.com
|
||||
shxj,1120,13811022092@139.com
|
||||
zhangl,1665,452838289@qq.com
|
||||
heyb,1019,121867346@qq.com
|
||||
dingpc,1164,pchding@163.com
|
||||
chengyh,1026,359957622@qq.com
|
||||
|
@@ -0,0 +1,26 @@
|
||||
username,passwd,email
|
||||
douzhn,1111,29867370@qq.com
|
||||
zhenglu,0872,18622306979@163.com
|
||||
zhangshh,1081,nkuzsh@sina.com
|
||||
qiangdq,1165,eacone@sina.com
|
||||
zhangwd,1381,zhangwendi0820@163.com
|
||||
chengqi,1481,531740421@qq.com
|
||||
lizj,1231,tjlizejun@163.com
|
||||
sunyl,1270,39204996@qq.com
|
||||
caichl,1296,cwtjwd@126.com
|
||||
lijy,0818,lijiayan222@qq.com
|
||||
chzhl,1295,46113395@qq.com
|
||||
qinglu,1230,luqingsmiles@foxmail.com
|
||||
renzw,1653,2734310562@qq.com
|
||||
songht,1215,luck_0076@163.com
|
||||
wangwx,1280,wenxingone@qq.com
|
||||
yangbo,1837,yangbo_6@126.com
|
||||
wangyi,0870,691511321@qq.com
|
||||
weihq,1816,hailife@163.com
|
||||
yushj,1126,122758312@qq.com
|
||||
zhbl,1468,sdzbxiaolang@163.com
|
||||
shxj,1120,13811022092@139.com
|
||||
zhangl,1665,452838289@qq.com
|
||||
heyb,1019,121867346@qq.com
|
||||
dingpc,1164,pchding@163.com
|
||||
chengyh,1026,359957622@qq.com
|
||||
|
@@ -0,0 +1,105 @@
|
||||
From: <Saved by Blink>
|
||||
Snapshot-Content-Location: https://www.thereisnospoonadu.com/
|
||||
Subject: =?utf-8?Q?=E8=BD=AF=E4=BB=B6=E4=B8=8B=E8=BD=BD=E9=A1=B5=E9=9D=A2?=
|
||||
Date: Fri, 3 Apr 2026 14:22:37 +0800
|
||||
MIME-Version: 1.0
|
||||
Content-Type: multipart/related;
|
||||
type="text/html";
|
||||
boundary="----MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----"
|
||||
|
||||
|
||||
------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----
|
||||
Content-Type: text/html
|
||||
Content-ID: <frame-86EA5B71750C5F07310888D40FAFED31@mhtml.blink>
|
||||
Content-Transfer-Encoding: quoted-printable
|
||||
Content-Location: https://www.thereisnospoonadu.com/
|
||||
|
||||
<!DOCTYPE html><html lang=3D"zh"><head><meta http-equiv=3D"Content-Type" co=
|
||||
ntent=3D"text/html; charset=3DUTF-8"><link rel=3D"stylesheet" type=3D"text/=
|
||||
css" href=3D"cid:css-37d7331d-d968-49c2-b3bd-7d9250c044e7@mhtml.blink" />
|
||||
|
||||
<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=3D1.=
|
||||
0">
|
||||
<title>=E8=BD=AF=E4=BB=B6=E4=B8=8B=E8=BD=BD=E9=A1=B5=E9=9D=A2</title>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<div class=3D"user-bar">
|
||||
<span class=3D"logged-in">=E5=BD=93=E5=89=8D=E5=B7=B2=E7=99=BB=E5=BD=95: <s=
|
||||
pan id=3D"username-display">haoxp</span></span>
|
||||
<button class=3D"logout-btn" id=3D"logout-btn">=E6=B3=A8=E9=94=80</button>
|
||||
</div>
|
||||
<h1>=E6=96=87=E4=BB=B6=E5=88=97=E8=A1=A8 <span id=3D"path-display"></span><=
|
||||
/h1>
|
||||
<div class=3D"breadcrumb" id=3D"breadcrumb"></div>
|
||||
<div id=3D"content">
|
||||
<div class=3D"loading" id=3D"loading" style=3D"display: none;">=E6=AD=A3=E5=
|
||||
=9C=A8=E5=8A=A0=E8=BD=BD...</div>
|
||||
<div class=3D"file-list" id=3D"file-list" style=3D"display: block;"><a href=
|
||||
=3D"https://www.thereisnospoonadu.com/%E7%89%88%E6%9C%AC%E5%8D%87%E7%BA%A7%=
|
||||
E8%AF%B4%E6%98%8E%EF%BC%88%E7%94%A8%E8%AE%B0%E4%BA%8B%E6%9C%AC%E6%89%93%E5%=
|
||||
BC%80%EF%BC%89.md?u=3Dhaoxp" download=3D"">=E7=89=88=E6=9C=AC=E5=8D=87=E7=
|
||||
=BA=A7=E8=AF=B4=E6=98=8E=EF=BC=88=E7=94=A8=E8=AE=B0=E4=BA=8B=E6=9C=AC=E6=89=
|
||||
=93=E5=BC=80=EF=BC=89.md<span class=3D"size">13.0 K 2026/4/3 14:10:24</spa=
|
||||
n></a><a href=3D"https://www.thereisnospoonadu.com/%E7%BA%AA%E6%A3%80%E7%9B=
|
||||
%91%E5%AF%9F%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97%E5%88%86%E6%9E%90%E8%BD%AF=
|
||||
%E4%BB%B6_v2026.03.19.deb?u=3Dhaoxp" download=3D"">=E7=BA=AA=E6=A3=80=E7=9B=
|
||||
=91=E5=AF=9F=E6=95=B0=E6=8D=AE=E6=B8=85=E6=B4=97=E5=88=86=E6=9E=90=E8=BD=AF=
|
||||
=E4=BB=B6_v2026.03.19.deb<span class=3D"size">108.1 M 2026/4/3 14:10:31</s=
|
||||
pan></a><a href=3D"https://www.thereisnospoonadu.com/%E7%BA%AA%E6%A3%80%E7%=
|
||||
9B%91%E5%AF%9F%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97%E5%88%86%E6%9E%90%E8%BD%=
|
||||
AF%E4%BB%B6_v2026.04.03%EF%BC%88%E5%8F%B3%E9%94%AE-%E4%BB%A5%E7%AE%A1%E7%90=
|
||||
%86%E5%91%98%E8%BA%AB%E4%BB%BD%E8%BF%90%E8%A1%8C%EF%BC%89.exe?u=3Dhaoxp" do=
|
||||
wnload=3D"">=E7=BA=AA=E6=A3=80=E7=9B=91=E5=AF=9F=E6=95=B0=E6=8D=AE=E6=B8=85=
|
||||
=E6=B4=97=E5=88=86=E6=9E=90=E8=BD=AF=E4=BB=B6_v2026.04.03=EF=BC=88=E5=8F=B3=
|
||||
=E9=94=AE-=E4=BB=A5=E7=AE=A1=E7=90=86=E5=91=98=E8=BA=AB=E4=BB=BD=E8=BF=90=
|
||||
=E8=A1=8C=EF=BC=89.exe<span class=3D"size">120.1 M 2026/4/3 14:10:37</span=
|
||||
></a></div>
|
||||
<div class=3D"error" id=3D"error" style=3D"display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</body></html>
|
||||
------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU----
|
||||
Content-Type: text/css
|
||||
Content-Transfer-Encoding: quoted-printable
|
||||
Content-Location: cid:css-37d7331d-d968-49c2-b3bd-7d9250c044e7@mhtml.blink
|
||||
|
||||
@charset "utf-8";
|
||||
|
||||
body { line-height: 2; font-family: sans-serif; margin: 2em; font-size: 18p=
|
||||
x !important; }
|
||||
|
||||
h1 { margin-bottom: 0.5em; border-bottom: 2px solid rgb(204, 204, 204); pad=
|
||||
ding-bottom: 0.3em; font-size: 24px !important; }
|
||||
|
||||
.file-list a { color: rgb(0, 102, 204); text-decoration: none; display: blo=
|
||||
ck; margin: 0.3em 0px; font-size: 20px !important; }
|
||||
|
||||
.file-list a:hover { text-decoration: underline; }
|
||||
|
||||
.file-list .dir { font-weight: bold; cursor: pointer; }
|
||||
|
||||
.file-list .size { font-size: 16px; color: rgb(102, 102, 102); margin-left:=
|
||||
1em; }
|
||||
|
||||
.breadcrumb { font-size: 18px; margin-bottom: 1em; }
|
||||
|
||||
.breadcrumb a { color: rgb(0, 102, 204); cursor: pointer; }
|
||||
|
||||
.loading { font-size: 20px; color: rgb(102, 102, 102); }
|
||||
|
||||
.error { color: rgb(204, 0, 0); font-size: 18px; }
|
||||
|
||||
.user-bar { margin-bottom: 1em; padding: 0.5em 0px; display: flex; align-it=
|
||||
ems: center; justify-content: flex-end; gap: 1em; }
|
||||
|
||||
.user-bar .logged-in { color: green; }
|
||||
|
||||
.user-bar .logout-btn { padding: 0.3em 0.8em; font-size: 16px; cursor: poin=
|
||||
ter; background: rgb(0, 102, 204); color: rgb(255, 255, 255); border: none;=
|
||||
border-radius: 4px; }
|
||||
|
||||
.user-bar .logout-btn:hover { background: rgb(0, 82, 163); }
|
||||
------MultipartBoundary--2C2rn7PyrLSMK9sISEgqNX2BSrI2xA1ugtSUwK34wU------
|
||||
Reference in New Issue
Block a user