{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "67a422d8",
      "metadata": {},
      "source": [
        "# The GPT tokenizer — Byte Pair Encoding from scratch\n",
        "\n",
        "Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step.\n",
        "\n",
        "This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model\n",
        "either single characters (Shakespeare) or single letters (names). Real GPTs use **subword**\n",
        "tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then\n",
        "see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's\n",
        "*\"Let's build the GPT Tokenizer\"*."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2037ef52",
      "metadata": {},
      "source": [
        "## Prologue — in plain English\n",
        "\n",
        "A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece\n",
        "mapped to an id. Two extremes:\n",
        "\n",
        "- **One token per character** (what we did): tiny vocabulary, but sequences are very long and\n",
        "  the model must relearn common spellings everywhere.\n",
        "- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it.\n",
        "\n",
        "**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common\n",
        "adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single\n",
        "tokens; rare text still survives as smaller pieces.\n",
        "\n",
        "Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep\n",
        "writing \"t-h-e\", so you invent one squiggle for \"the.\" Then you notice \" a-n-d\" and give it a\n",
        "squiggle too. BPE does this automatically, keeping the most useful shorthands."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "75d40ed6",
      "metadata": {},
      "source": [
        "### 1.1 Text -> UTF-8 bytes (the 256 starting tokens)\n",
        "\n",
        "Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts\n",
        "with exactly 256 tokens — every possible byte. Plain English letters are one byte each;\n",
        "accented or non-Latin characters take several bytes (which is why they cost more tokens later)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1037d36e",
      "metadata": {},
      "outputs": [],
      "source": [
        "sample = \"Hello world! tokenization is fun.\"\n",
        "the_bytes = sample.encode(\"utf-8\")\n",
        "print(\"text   :\", sample)\n",
        "print(\"bytes  :\", list(the_bytes))\n",
        "print(\"length :\", len(the_bytes), \"bytes ->\", len(the_bytes), \"starting tokens\")\n",
        "\n",
        "multi = \"café  naïve\"   # accented letters take >1 byte each\n",
        "print(\"\\nnon-ASCII example:\", multi)\n",
        "print(\"bytes:\", list(multi.encode(\"utf-8\")), \"(note: more bytes than characters)\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5db66687",
      "metadata": {},
      "source": [
        "### 2.1 Two tiny helpers: count pairs, and merge a pair\n",
        "\n",
        "- `get_stats` counts how often each adjacent pair appears.\n",
        "- `merge` replaces every occurrence of a chosen pair with a single new token id.\n",
        "\n",
        "That is the entire mechanism of BPE."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6bc93ad8",
      "metadata": {},
      "outputs": [],
      "source": [
        "def get_stats(ids):\n",
        "    counts = {}\n",
        "    for pair in zip(ids, ids[1:]):\n",
        "        counts[pair] = counts.get(pair, 0) + 1\n",
        "    return counts\n",
        "\n",
        "def merge(ids, pair, idx):\n",
        "    new_ids = []\n",
        "    i = 0\n",
        "    while i < len(ids):\n",
        "        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:\n",
        "            new_ids.append(idx)\n",
        "            i += 2\n",
        "        else:\n",
        "            new_ids.append(ids[i])\n",
        "            i += 1\n",
        "    return new_ids\n",
        "\n",
        "# demo on a toy list: merge the most common pair\n",
        "demo = [1, 2, 3, 1, 2, 3, 1, 2]\n",
        "st = get_stats(demo)\n",
        "top = max(st, key=st.get)\n",
        "print(\"counts:\", st)\n",
        "print(\"most common pair:\", top, \"->\", st[top], \"times\")\n",
        "print(\"after merging\", top, \"into 99:\", merge(demo, top, 99))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cfb961e0",
      "metadata": {},
      "source": [
        "### 2.2 Train BPE — learn the merges\n",
        "\n",
        "We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find\n",
        "the most common pair and merge it, recording each merge. After `num_merges` rounds we have a\n",
        "vocabulary of `256 + num_merges` tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f62fbf5d",
      "metadata": {},
      "outputs": [],
      "source": [
        "train_text = open(\"input.txt\", \"r\", encoding=\"utf-8\").read()[:20000]\n",
        "ids = list(train_text.encode(\"utf-8\"))\n",
        "print(\"training bytes:\", len(ids))\n",
        "\n",
        "num_merges = 60\n",
        "merges = {}                      # (a, b) -> new_id\n",
        "work = list(ids)\n",
        "for i in range(num_merges):\n",
        "    stats = get_stats(work)\n",
        "    pair = max(stats, key=stats.get)\n",
        "    idx = 256 + i\n",
        "    work = merge(work, pair, idx)\n",
        "    merges[pair] = idx\n",
        "\n",
        "print(\"merges learned:\", len(merges))\n",
        "print(\"tokens after training:\", len(work), \"(was\", len(ids), \"bytes)\")\n",
        "print(f\"compression: {len(ids) / len(work):.2f}x\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "34b2bc27",
      "metadata": {},
      "source": [
        "### 3.1 encode and decode\n",
        "\n",
        "- **decode**: turn token ids back into text. We build each token's byte string by stitching the\n",
        "  merges back together.\n",
        "- **encode**: turn new text into token ids by applying the learned merges, always doing the\n",
        "  *earliest-learned* applicable merge first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c4f1dcab",
      "metadata": {},
      "outputs": [],
      "source": [
        "# build the id -> bytes table\n",
        "vocab = {idx: bytes([idx]) for idx in range(256)}\n",
        "for (p0, p1), idx in merges.items():\n",
        "    vocab[idx] = vocab[p0] + vocab[p1]\n",
        "\n",
        "def decode(ids):\n",
        "    data = b\"\".join(vocab[idx] for idx in ids)\n",
        "    return data.decode(\"utf-8\", errors=\"replace\")\n",
        "\n",
        "def encode(text):\n",
        "    tokens = list(text.encode(\"utf-8\"))\n",
        "    while len(tokens) >= 2:\n",
        "        stats = get_stats(tokens)\n",
        "        # pick the pair whose merge was learned earliest\n",
        "        pair = min(stats, key=lambda p: merges.get(p, float(\"inf\")))\n",
        "        if pair not in merges:\n",
        "            break\n",
        "        tokens = merge(tokens, pair, merges[pair])\n",
        "    return tokens\n",
        "\n",
        "trial = \"the king and the queen\"\n",
        "enc = encode(trial)\n",
        "print(\"text  :\", trial)\n",
        "print(\"tokens:\", enc)\n",
        "print(\"count :\", len(enc), \"tokens for\", len(trial), \"characters\")\n",
        "print(\"decode round-trip ok:\", decode(enc) == trial)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a41ee40c",
      "metadata": {},
      "source": [
        "### 3.2 See what the learned tokens are\n",
        "\n",
        "The merges discovered the common chunks of English on their own. Let's print a few learned\n",
        "tokens (ids >= 256) as the text they stand for."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dadcc751",
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"a few learned tokens:\")\n",
        "for idx in range(256, 256 + min(20, num_merges)):\n",
        "    piece = vocab[idx].decode(\"utf-8\", errors=\"replace\")\n",
        "    print(f\"  id {idx}: {piece!r}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9193a4cd",
      "metadata": {},
      "source": [
        "### 4.1 Regex pre-splitting (what real GPTs add)\n",
        "\n",
        "A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the`\n",
        "into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words,\n",
        "runs of spaces, punctuation), then runs BPE **inside** each chunk only.\n",
        "\n",
        "Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the\n",
        "standard `re` module to show the idea."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "96cf29fc",
      "metadata": {},
      "outputs": [],
      "source": [
        "import re\n",
        "\n",
        "# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N})\n",
        "pat = re.compile(r\"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+\")\n",
        "\n",
        "demo_text = \"Hello, world! It's 2026 already.\"\n",
        "chunks = re.findall(pat, demo_text)\n",
        "print(\"split into chunks (BPE then runs inside each one):\")\n",
        "print(chunks)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e825ed0b",
      "metadata": {},
      "source": [
        "### 4.2 Special tokens\n",
        "\n",
        "Real tokenizers also reserve **special tokens** that are not learned from text but inserted to\n",
        "mark structure, for example `<|endoftext|>` between documents, or chat markers like\n",
        "`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary.\n",
        "\n",
        "Real-life picture: punctuation the model never \"spells\" — single reserved symbols that mean\n",
        "\"new document starts here\" or \"the user is speaking now.\"\n",
        "\n",
        "```\n",
        "<|endoftext|>   -> id 50256 in GPT-2 (separates documents)\n",
        "<|im_start|>    -> chat role marker in chat models\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "638d25d8",
      "metadata": {},
      "source": [
        "### 5.1 Why tokenization causes LLM quirks\n",
        "\n",
        "Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few.\n",
        "\n",
        "- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does\n",
        "  not \"see\" its letters.\n",
        "- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits.\n",
        "- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more\n",
        "  tokens (and more money / context)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4f051e82",
      "metadata": {},
      "outputs": [],
      "source": [
        "def n_tokens(s):\n",
        "    return len(encode(s))\n",
        "\n",
        "print(\"spelling: 'extraordinary' ->\", n_tokens(\"extraordinary\"), \"token(s)\")\n",
        "print(\"          the model sees a chunk, not the 13 letters\\n\")\n",
        "\n",
        "print(\"numbers : '1234567'      ->\", n_tokens(\"1234567\"), \"tokens (split oddly, not per-digit)\")\n",
        "print(\"          '127 + 677'    ->\", n_tokens(\"127 + 677\"), \"tokens\\n\")\n",
        "\n",
        "eng = \"hello how are you\"\n",
        "non = \"你好你好你好\"   # non-English of similar visible length\n",
        "print(\"non-English costs more tokens for similar content:\")\n",
        "print(\"  english   :\", repr(eng), \"->\", n_tokens(eng), \"tokens\")\n",
        "print(\"  non-latin :\", repr(non), \"->\", n_tokens(non), \"tokens\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cda06321",
      "metadata": {},
      "source": [
        "**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the\n",
        "compression ratio and the token counts in section 5.1. More merges = better compression, but a\n",
        "bigger vocabulary. This trade-off is exactly what real tokenizer designers tune."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "35388d46",
      "metadata": {},
      "source": [
        "## Series wrap-up\n",
        "\n",
        "You built the whole stack, from the ground up:\n",
        "\n",
        "- `00_micrograd.ipynb` — backprop from scratch (the engine)\n",
        "- `01_build_gpt.ipynb` — the bigram baseline\n",
        "- `02_makemore_mlp.ipynb` — an MLP with embeddings\n",
        "- `03_batchnorm_activations.ipynb` — keeping deep nets healthy\n",
        "- `04_backprop_ninja.ipynb` — gradients by hand\n",
        "- `05_wavenet.ipynb` — a deeper, hierarchical model\n",
        "- `06_build_gpt_attention.ipynb` — the transformer (GPT)\n",
        "- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook)\n",
        "\n",
        "Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data,\n",
        "with runnable code at every step.\n",
        "\n",
        "**Checklist**\n",
        "- [ ] Text -> UTF-8 bytes (256 base tokens)\n",
        "- [ ] `get_stats` + `merge` = the BPE mechanism\n",
        "- [ ] Trained merges; measured compression\n",
        "- [ ] encode / decode round-trips\n",
        "- [ ] Regex pre-splitting and special tokens\n",
        "- [ ] Explained tokenization-caused LLM quirks"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".venv (3.12.10)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}