375 lines
14 KiB
Plaintext
375 lines
14 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "67a422d8",
|
|
"metadata": {},
|
|
"source": [
|
|
"# The GPT tokenizer — Byte Pair Encoding from scratch\n",
|
|
"\n",
|
|
"Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.\n",
|
|
"\n",
|
|
"This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model\n",
|
|
"either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**\n",
|
|
"tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then\n",
|
|
"see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's\n",
|
|
"*\"Let's build the GPT Tokenizer\"*."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2037ef52",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prologue — in plain English\n",
|
|
"\n",
|
|
"A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece\n",
|
|
"mapped to an id. Two extremes:\n",
|
|
"\n",
|
|
"- **One token per character** (what we did): tiny vocabulary, but sequences are very long and\n",
|
|
" the model must relearn common spellings everywhere.\n",
|
|
"- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.\n",
|
|
"\n",
|
|
"**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common\n",
|
|
"adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single\n",
|
|
"tokens; rare text still survives as smaller pieces.\n",
|
|
"\n",
|
|
"Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep\n",
|
|
"writing \"t-h-e\", so you invent one squiggle for \"the.\" Then you notice \" a-n-d\" and give it a\n",
|
|
"squiggle too. BPE does this automatically, keeping the most useful shorthands."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "75d40ed6",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)\n",
|
|
"\n",
|
|
"Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts\n",
|
|
"with exactly 256 tokens — every possible byte. Plain English letters are one byte each;\n",
|
|
"accented or non-Latin characters take several bytes (which is why they cost more tokens later)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1037d36e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"sample = \"Hello world! tokenization is fun.\"\n",
|
|
"the_bytes = sample.encode(\"utf-8\")\n",
|
|
"print(\"text :\", sample)\n",
|
|
"print(\"bytes :\", list(the_bytes))\n",
|
|
"print(\"length :\", len(the_bytes), \"bytes ->\", len(the_bytes), \"starting tokens\")\n",
|
|
"\n",
|
|
"multi = \"café naïve\" # accented letters take >1 byte each\n",
|
|
"print(\"\\nnon-ASCII example:\", multi)\n",
|
|
"print(\"bytes:\", list(multi.encode(\"utf-8\")), \"(note: more bytes than characters)\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5db66687",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 2.1 Two tiny helpers: count pairs, and merge a pair\n",
|
|
"\n",
|
|
"- `get_stats` counts how often each adjacent pair appears.\n",
|
|
"- `merge` replaces every occurrence of a chosen pair with a single new token id.\n",
|
|
"\n",
|
|
"That is the entire mechanism of BPE."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6bc93ad8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_stats(ids):\n",
|
|
" counts = {}\n",
|
|
" for pair in zip(ids, ids[1:]):\n",
|
|
" counts[pair] = counts.get(pair, 0) + 1\n",
|
|
" return counts\n",
|
|
"\n",
|
|
"def merge(ids, pair, idx):\n",
|
|
" new_ids = []\n",
|
|
" i = 0\n",
|
|
" while i < len(ids):\n",
|
|
" if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:\n",
|
|
" new_ids.append(idx)\n",
|
|
" i += 2\n",
|
|
" else:\n",
|
|
" new_ids.append(ids[i])\n",
|
|
" i += 1\n",
|
|
" return new_ids\n",
|
|
"\n",
|
|
"# demo on a toy list: merge the most common pair\n",
|
|
"demo = [1, 2, 3, 1, 2, 3, 1, 2]\n",
|
|
"st = get_stats(demo)\n",
|
|
"top = max(st, key=st.get)\n",
|
|
"print(\"counts:\", st)\n",
|
|
"print(\"most common pair:\", top, \"->\", st[top], \"times\")\n",
|
|
"print(\"after merging\", top, \"into 99:\", merge(demo, top, 99))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb961e0",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 2.2 Train BPE — learn the merges\n",
|
|
"\n",
|
|
"We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find\n",
|
|
"the most common pair and merge it, recording each merge. After `num_merges` rounds we have a\n",
|
|
"vocabulary of `256 + num_merges` tokens."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f62fbf5d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"train_text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()[:20000]\n",
|
|
"ids = list(train_text.encode(\"utf-8\"))\n",
|
|
"print(\"training bytes:\", len(ids))\n",
|
|
"\n",
|
|
"num_merges = 60\n",
|
|
"merges = {} # (a, b) -> new_id\n",
|
|
"work = list(ids)\n",
|
|
"for i in range(num_merges):\n",
|
|
" stats = get_stats(work)\n",
|
|
" pair = max(stats, key=stats.get)\n",
|
|
" idx = 256 + i\n",
|
|
" work = merge(work, pair, idx)\n",
|
|
" merges[pair] = idx\n",
|
|
"\n",
|
|
"print(\"merges learned:\", len(merges))\n",
|
|
"print(\"tokens after training:\", len(work), \"(was\", len(ids), \"bytes)\")\n",
|
|
"print(f\"compression: {len(ids) / len(work):.2f}x\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "34b2bc27",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 3.1 encode and decode\n",
|
|
"\n",
|
|
"- **decode**: turn token ids back into text. We build each token's byte string by stitching the\n",
|
|
" merges back together.\n",
|
|
"- **encode**: turn new text into token ids by applying the learned merges, always doing the\n",
|
|
" *earliest-learned* applicable merge first."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c4f1dcab",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# build the id -> bytes table\n",
|
|
"vocab = {idx: bytes([idx]) for idx in range(256)}\n",
|
|
"for (p0, p1), idx in merges.items():\n",
|
|
" vocab[idx] = vocab[p0] + vocab[p1]\n",
|
|
"\n",
|
|
"def decode(ids):\n",
|
|
" data = b\"\".join(vocab[idx] for idx in ids)\n",
|
|
" return data.decode(\"utf-8\", errors=\"replace\")\n",
|
|
"\n",
|
|
"def encode(text):\n",
|
|
" tokens = list(text.encode(\"utf-8\"))\n",
|
|
" while len(tokens) >= 2:\n",
|
|
" stats = get_stats(tokens)\n",
|
|
" # pick the pair whose merge was learned earliest\n",
|
|
" pair = min(stats, key=lambda p: merges.get(p, float(\"inf\")))\n",
|
|
" if pair not in merges:\n",
|
|
" break\n",
|
|
" tokens = merge(tokens, pair, merges[pair])\n",
|
|
" return tokens\n",
|
|
"\n",
|
|
"trial = \"the king and the queen\"\n",
|
|
"enc = encode(trial)\n",
|
|
"print(\"text :\", trial)\n",
|
|
"print(\"tokens:\", enc)\n",
|
|
"print(\"count :\", len(enc), \"tokens for\", len(trial), \"characters\")\n",
|
|
"print(\"decode round-trip ok:\", decode(enc) == trial)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a41ee40c",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 3.2 See what the learned tokens are\n",
|
|
"\n",
|
|
"The merges discovered the common chunks of English on their own. Let's print a few learned\n",
|
|
"tokens (ids >= 256) as the text they stand for."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "dadcc751",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(\"a few learned tokens:\")\n",
|
|
"for idx in range(256, 256 + min(20, num_merges)):\n",
|
|
" piece = vocab[idx].decode(\"utf-8\", errors=\"replace\")\n",
|
|
" print(f\" id {idx}: {piece!r}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9193a4cd",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 4.1 Regex pre-splitting (what real GPTs add)\n",
|
|
"\n",
|
|
"A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`\n",
|
|
"into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,\n",
|
|
"runs of spaces, punctuation), then runs BPE **inside** each chunk only.\n",
|
|
"\n",
|
|
"Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the\n",
|
|
"standard `re` module to show the idea."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "96cf29fc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import re\n",
|
|
"\n",
|
|
"# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})\n",
|
|
"pat = re.compile(r\"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+\")\n",
|
|
"\n",
|
|
"demo_text = \"Hello, world! It's 2026 already.\"\n",
|
|
"chunks = re.findall(pat, demo_text)\n",
|
|
"print(\"split into chunks (BPE then runs inside each one):\")\n",
|
|
"print(chunks)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e825ed0b",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 4.2 Special tokens\n",
|
|
"\n",
|
|
"Real tokenizers also reserve **special tokens** that are not learned from text but inserted to\n",
|
|
"mark structure, for example `<|endoftext|>` between documents, or chat markers like\n",
|
|
"`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.\n",
|
|
"\n",
|
|
"Real-life picture: punctuation the model never \"spells\" — single reserved symbols that mean\n",
|
|
"\"new document starts here\" or \"the user is speaking now.\"\n",
|
|
"\n",
|
|
"```\n",
|
|
"<|endoftext|> -> id 50256 in GPT-2 (separates documents)\n",
|
|
"<|im_start|> -> chat role marker in chat models\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "638d25d8",
|
|
"metadata": {},
|
|
"source": [
|
|
"### 5.1 Why tokenization causes LLM quirks\n",
|
|
"\n",
|
|
"Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.\n",
|
|
"\n",
|
|
"- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does\n",
|
|
" not \"see\" its letters.\n",
|
|
"- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.\n",
|
|
"- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more\n",
|
|
" tokens (and more money / context)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4f051e82",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def n_tokens(s):\n",
|
|
" return len(encode(s))\n",
|
|
"\n",
|
|
"print(\"spelling: 'extraordinary' ->\", n_tokens(\"extraordinary\"), \"token(s)\")\n",
|
|
"print(\" the model sees a chunk, not the 13 letters\\n\")\n",
|
|
"\n",
|
|
"print(\"numbers : '1234567' ->\", n_tokens(\"1234567\"), \"tokens (split oddly, not per-digit)\")\n",
|
|
"print(\" '127 + 677' ->\", n_tokens(\"127 + 677\"), \"tokens\\n\")\n",
|
|
"\n",
|
|
"eng = \"hello how are you\"\n",
|
|
"non = \"你好你好你好\" # non-English of similar visible length\n",
|
|
"print(\"non-English costs more tokens for similar content:\")\n",
|
|
"print(\" english :\", repr(eng), \"->\", n_tokens(eng), \"tokens\")\n",
|
|
"print(\" non-latin :\", repr(non), \"->\", n_tokens(non), \"tokens\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cda06321",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the\n",
|
|
"compression ratio and the token counts in section 5.1. More merges = better compression, but a\n",
|
|
"bigger vocabulary. This trade-off is exactly what real tokenizer designers tune."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35388d46",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Series wrap-up\n",
|
|
"\n",
|
|
"You built the whole stack, from the ground up:\n",
|
|
"\n",
|
|
"- `00_micrograd.ipynb` — backprop from scratch (the engine)\n",
|
|
"- `01_build_gpt.ipynb` — the bigram baseline\n",
|
|
"- `02_makemore_mlp.ipynb` — an MLP with embeddings\n",
|
|
"- `03_batchnorm_activations.ipynb` — keeping deep nets healthy\n",
|
|
"- `04_backprop_ninja.ipynb` — gradients by hand\n",
|
|
"- `05_wavenet.ipynb` — a deeper, hierarchical model\n",
|
|
"- `06_build_gpt_attention.ipynb` — the transformer (GPT)\n",
|
|
"- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)\n",
|
|
"\n",
|
|
"Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,\n",
|
|
"with runnable code at every step.\n",
|
|
"\n",
|
|
"**Checklist**\n",
|
|
"- [ ] Text -> UTF-8 bytes (256 base tokens)\n",
|
|
"- [ ] `get_stats` + `merge` = the BPE mechanism\n",
|
|
"- [ ] Trained merges; measured compression\n",
|
|
"- [ ] encode / decode round-trips\n",
|
|
"- [ ] Regex pre-splitting and special tokens\n",
|
|
"- [ ] Explained tokenization-caused LLM quirks"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": ".venv (3.12.10)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python",
|
|
"version": "3.12.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |