{ "cells": [ { "cell_type": "markdown", "id": "153d6f74", "metadata": {}, "source": [ "# Micrograd — backprop from scratch\n", "\n", "A tiny automatic-gradient engine in pure Python — **concept -> code -> Your turn** each step.\n", "\n", "This is notebook **00** of the series and the right place to start: it explains what a\n", "*gradient* and *backpropagation* really are, using single numbers you can follow by hand.\n", "Every later notebook (the bigram in `01`, the transformer in `06`) calls `loss.backward()`\n", "and trusts it. Here we build that machinery ourselves so it is never magic." ] }, { "cell_type": "markdown", "id": "aac17ea7", "metadata": {}, "source": [ "## Prologue — what this notebook does, in plain English\n", "\n", "We build a small object called a **`Value`**. A `Value` is just a number that also\n", "**remembers how it was made** (which other numbers, and which operation: plus, times, ...).\n", "\n", "Once numbers remember their own history, the computer can answer one very useful question\n", "automatically:\n", "\n", "> *If I nudge this input a tiny bit, how much does the final answer change?*\n", "\n", "That sensitivity is the **gradient**. Computing all those sensitivities in one efficient\n", "backward sweep is **backpropagation** — the single algorithm that trains essentially every\n", "neural network, including GPT.\n", "\n", "This notebook follows Andrej Karpathy's *\"The spelled-out intro to neural networks and\n", "backpropagation: building micrograd\"* lecture." ] }, { "cell_type": "markdown", "id": "44dd4b0d", "metadata": {}, "source": [ "### The whole idea, as a chain of gears\n", "\n", "Picture a row of **gears** connected together. You turn the first gear a little; the last\n", "gear also turns, by an amount that depends on all the gears in between.\n", "\n", "- The **forward pass** is turning the first gears and reading the last gear (the output).\n", "- The **gradient** answers: *if I turn this one gear slightly, how much does the final gear move?*\n", "- **Backpropagation** is figuring out that answer for **every** gear at once, by walking\n", " backward from the last gear to the first and multiplying the little ratios along the way\n", " (that multiplication is the **chain rule**).\n", "\n", "A neural network is just a very big gear train. Training = nudging each gear a hair in the\n", "direction that makes the output less wrong." ] }, { "cell_type": "markdown", "id": "334f0f03", "metadata": {}, "source": [ "### Mental picture — a factory line with tags\n", "\n", "Imagine a **cookie factory**. Each station does one small job and puts a **tag** on the box\n", "saying *who gave me ingredients* and *what I did* (mixed, baked, frosted).\n", "\n", "| Step | Real life | In code |\n", "|------|-----------|---------|\n", "| Forward pass | Ingredients move down the line; you get a final cookie score | `d = a * b + c` computes numbers left to right |\n", "| History | Every box keeps tags — no separate diary | `_prev` (parents) and `_op` (operation) on each `Value` |\n", "| Gradient | *If I change this one ingredient a tiny bit, how much does the final score change?* | `.grad` on each `Value` |\n", "| Backprop | Walk backward along the tags from the final score to every ingredient | `loss.backward()` |\n", "\n", "A normal float is like a score on a napkin: **4**. You cannot tell where it came from.\n", "A **`Value`** is the same score **on a tagged box** — you can always walk back.\n", "\n", "**Blame traveling backward (the chain rule)**\n", "\n", "The final taste score is off. At each station you ask: *if your output had been 1 point\n", "sweeter, how much would the final score have moved?*\n", "\n", "- **`+` (mix two bowls)** — nudge either bowl; the mix moves the same amount. Blame passes\n", " through to both inputs equally.\n", "- **`*` (recipe ratio, e.g. 2 cups × 3 batches)** — change flour a little → effect depends\n", " on batch count; change batches → effect depends on flour. Each side's blame depends on\n", " the **other** side's amount.\n", "\n", "**Backprop** starts at the output (\"downstream moved by 1\"), walks **backward** along the\n", "tags, and at each station multiplies by that station's local rule. Small effect here ×\n", "small effect there = total effect on the end.\n", "\n", "**Why gradients add (`+=`)**\n", "\n", "One bag of sugar might go into **both** dough and icing. Both paths affect the final taste,\n", "so sugar's total blame is the **sum** of both paths — same when one `Value` is used twice\n", "in an expression.\n", "\n", "**Training the MLP (section 6)**\n", "\n", "Same factory, but the knobs are **weights and biases** (how much each input counts):\n", "\n", "1. **Forward** — run points through the line; get predictions.\n", "2. **Loss** — how wrong were we? (distance from the target).\n", "3. **Backward** — `loss.backward()` sends blame to every knob.\n", "4. **Update** — turn each knob a tiny step **against** its gradient (downhill = less wrong).\n", "\n", "The **two moons** dataset is two swirls of red and blue beads. The network learns a curved\n", "boundary — not by memorizing dots, but by turning knobs until wrong guesses hurt (high loss)\n", "and right ones don't.\n", "\n", "> **One line to keep:** Forward = tagged boxes down the line. Backward = follow tags from\n", "> the final score back to each ingredient and assign fair blame. PyTorch does the same thing\n", "> at scale — millions of tagged boxes, fast hardware." ] }, { "cell_type": "markdown", "id": "84448625", "metadata": {}, "source": [ "### 0.1 Imports\n", "\n", "Pure Python plus a little NumPy/Matplotlib for the toy dataset and pictures. No deep-learning\n", "library is needed to build the engine itself — that is the whole point.\n", "\n", "**-> Training:** `math` gives us `exp`/`tanh` for activations; `random` initialises weights;\n", "NumPy and Matplotlib are only for the demo dataset and plots." ] }, { "cell_type": "code", "execution_count": 1, "id": "e31a52ef", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ready\n" ] } ], "source": [ "import math # exp, tanh used inside Value activations\n", "import random # random weight init for Neuron\n", "import numpy as np # dataset + optional plots only (not the autograd engine)\n", "\n", "random.seed(1337) # same random weights every run\n", "np.random.seed(1337) # same noisy moon dots every run\n", "\n", "# Plots are optional: the notebook runs fine without matplotlib (it just skips pictures).\n", "try:\n", " import matplotlib.pyplot as plt\n", " HAS_PLT = True # flag so later cells can skip plotting gracefully\n", "except Exception:\n", " HAS_PLT = False\n", " print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n", "\n", "print(\"ready\")" ] }, { "cell_type": "code", "execution_count": null, "id": "ace0626a", "metadata": {}, "outputs": [], "source": [ "from IPython.display import HTML, display\n", "import json\n", "\n", "_GRAPH_UID = 0 # unique id per render so multiple cells don't clash\n", "\n", "\n", "def show_graph(root, title=\"Computation graph — hover a node\"):\n", " \"\"\"Draw an interactive SVG graph for a Value expression.\n", "\n", " Hover (or click) any node to see its value, operation, and parents.\n", " After `.backward()`, gradients appear on each node automatically.\n", " \"\"\"\n", " global _GRAPH_UID\n", " _GRAPH_UID += 1\n", " gid = f\"mg{_GRAPH_UID}\"\n", "\n", " nodes, seen = [], set()\n", "\n", " def build_topo(v):\n", " if v not in seen:\n", " seen.add(v)\n", " for ch in v._prev:\n", " build_topo(ch)\n", " nodes.append(v)\n", "\n", " build_topo(root)\n", " id_map = {v: i for i, v in enumerate(nodes)}\n", "\n", " depth = {}\n", "\n", " def mark_depth(v, d=0):\n", " depth[v] = max(depth.get(v, 0), d)\n", " for ch in v._prev:\n", " mark_depth(ch, d + 1)\n", "\n", " mark_depth(root, 0)\n", " max_depth = max(depth.values()) if depth else 0\n", " layer_of = {v: max_depth - depth[v] for v in nodes}\n", "\n", " from collections import defaultdict\n", "\n", " layers = defaultdict(list)\n", " for v in nodes:\n", " layers[layer_of[v]].append(v)\n", "\n", " NODE_W, NODE_H, GAP_X, GAP_Y = 108, 58, 36, 72\n", " positions = {}\n", " max_w = 0\n", " for ly in sorted(layers.keys()):\n", " row = layers[ly]\n", " row_w = len(row) * NODE_W + max(0, len(row) - 1) * GAP_X\n", " max_w = max(max_w, row_w)\n", " x0 = (max_w - row_w) / 2\n", " for i, v in enumerate(row):\n", " positions[v] = (x0 + i * (NODE_W + GAP_X), ly * (NODE_H + GAP_Y))\n", "\n", " svg_w = int(max_w + 48)\n", " svg_h = int((max(layers.keys()) + 1) * (NODE_H + GAP_Y) + 36)\n", " show_grad = any(getattr(v, \"grad\", 0.0) for v in nodes)\n", "\n", " def node_name(v):\n", " lbl = getattr(v, \"label\", \"\") or \"\"\n", " return lbl if lbl else f\"n{id_map[v]}\"\n", "\n", " def esc(s):\n", " return (\n", " str(s)\n", " .replace(\"&\", \"&\")\n", " .replace(\"<\", \"<\")\n", " .replace(\">\", \">\")\n", " .replace('\"', \""\")\n", " )\n", "\n", " edge_lines = []\n", " for v in nodes:\n", " for ch in v._prev:\n", " x1, y1 = positions[ch]\n", " x2, y2 = positions[v]\n", " cx1, cy1 = x1 + NODE_W / 2, y1 + NODE_H\n", " cx2, cy2 = x2 + NODE_W / 2, y2\n", " eid = f\"{gid}-e{id_map[ch]}-{id_map[v]}\"\n", " edge_lines.append(\n", " f''\n", " )\n", "\n", " node_groups = []\n", " meta_nodes = {}\n", " for v in nodes:\n", " x, y = positions[v]\n", " nid = id_map[v]\n", " node_id = f\"{gid}-n{nid}\"\n", " nm = esc(node_name(v))\n", " op = esc(v._op or \"leaf\")\n", " val = f\"{v.data:.4f}\"\n", " grad = f\"{getattr(v, 'grad', 0.0):.4f}\"\n", " fill = \"#dcfce7\" if not v._prev else \"#dbeafe\"\n", " grad_svg = (\n", " f''\n", " f\"grad = {grad}\"\n", " if show_grad\n", " else \"\"\n", " )\n", " node_groups.append(\n", " f''\n", " f''\n", " f'{nm}'\n", " f''\n", " f\"op: {op} val: {val}\"\n", " f\"{grad_svg}\"\n", " f'{nm}: {val} ({op})'\n", " )\n", " meta_nodes[node_id] = {\n", " \"name\": node_name(v),\n", " \"op\": v._op or \"leaf\",\n", " \"val\": val,\n", " \"grad\": grad,\n", " \"parents\": \", \".join(node_name(p) for p in v._prev) or \"—\",\n", " }\n", "\n", " meta_json = json.dumps({\"gid\": gid, \"show_grad\": show_grad, \"nodes\": meta_nodes})\n", "\n", " html = f\"\"\"\n", "
\n", "
{esc(title)}
\n", " \n", " \n", " \n", " \n", " \n", " \n", " {''.join(edge_lines)}\n", " {''.join(node_groups)}\n", " \n", "
\n", " Hover a node to inspect it. Arrows flow from inputs (bottom) to output (top).\n", "
\n", "
\n", "\n", "\n", "\"\"\"\n", " display(HTML(html))\n", "\n", "\n", "print(\"interactive graph viewer ready\")" ] }, { "cell_type": "markdown", "id": "3315510f", "metadata": {}, "source": [ "### 0.2 A tiny dataset — two interleaving moons\n", "\n", "Our goal at the end is to train a small network to separate two groups of dots that curl\n", "around each other (\"two moons\"). Real-life picture: two handfuls of red and blue beads\n", "mixed in a swirl — can the network learn to draw the boundary between them?\n", "\n", "We try scikit-learn's `make_moons`; if it is not installed we generate the same shape with\n", "NumPy so the notebook stays dependency-light." ] }, { "cell_type": "code", "execution_count": 2, "id": "507b4df3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sklearn not available, using NumPy fallback: ModuleNotFoundError\n", "X shape: (100, 2) y shape: (100,)\n", "first 3 points: [[0.9296812690137589, -0.04902823627877147], [0.965763959785321, -0.11143765173168577], [1.0124564608446291, -0.07324929559695742]]\n", "first 3 labels: [0, 0, 0]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAF2CAYAAAA4MQK3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAaAZJREFUeJztnQd4FNXXxk9ooYgUqUE6CiJVmiBKlSJSFJGiNBGkKoLwBwuIqFQBQRQbzUJTioKCSBGVIgYQKfoBUqWXhCogzPe89zrJ7GZ2s5vsZtv7e56B7OzszJ075dxz7ilRhmEYQgghhJBUky71uyCEEEIIhSohhBDiQ6ipEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUKCiAMHDkhUVJTMnDkz0E0hYQDuI9xPv/76q4QSUVFR8uqrr0ooEpZCdf369eqCxMXFBbophNjy+eefy6RJk8K6d44ePaqew23btgW6KUHDu+++65cBk7/2G2l888038tBDD0nBggUlZ86c8uCDD8pvv/3m1T7CVqiOGDGCQpWEnFAtWrSoXLlyRTp27CjhIFTxHFKoJkKhGtw8/PDDkjFjRhk2bJgMHTpUduzYoQTrqVOnPN5HBr+2kBDitdkrc+bM7LUQALVI/vnnH8mSJUugm0LccOnSJcmWLZt4woYNG6RGjRoJnytXriyNGzeW7777Tp544onI1FRhbho0aJD6u3jx4uolhQVzVY8++qjcc889Dts3b95cff/VV18lrNu0aZNa9+233yas++uvv6RNmzaSO3duyZo1q9x7772ybNkyj9qEffXt21cWLFggZcuWVQ9hzZo15ffff1ffv//++1KqVCn1Mq1bt65qqzP4bZUqVdRv8+TJI08++aT8/fffSbZbvXq13H///eomgvmiZcuWsnv37iR9hDbt3btXunTporbLkSOHdO3aVS5fvuyw7cqVK6V27dpqm1tuuUVKly4tL774ogSKmzdvyttvvy3ly5dX/ZU3b15p0qSJw5zRjBkzpH79+pIvXz6Jjo5Wff7ee+8l2VexYsXUyPSnn36S6tWrq/2VKFFCZs+e7VFbxo8fL7Vq1ZLbbrtNXRdcny+++CLZ3+Ea4945ePBgwv2JtriaU8U1Qt/jerdq1Ur9jfN+4YUX5MaNGwkveOwD19sZvPhxfZ955hmP7tPFixdLuXLlVN/dfffdsnz58iTboi1PPfWU5M+fP2G76dOnJ3y/du1aqVatmvob95V5nu5MlBcuXJD+/fur88A+cf2gJWzZssVhOzyfuOY4JzyLderUkZ9//tn2Hv/jjz/k8ccfl1tvvVVdp+eee071hxVv75cVK1ZI1apV1TXHs+vpPvD7nTt3yg8//JDQH7gXTDBdhfMvXLiw2gfeCWPGjFH3vDuS2y+4evWqDBgwQN03eDc88sgjttoX3nnm+yN79uzSrFkztW9PQPuff/75hOt3++23S6dOneT06dMJ25w8eVK6deum7hs8bxUrVpRZs2Z5tP+tW7dK06ZN1bXEM9CgQQPZuHGj7Rwy+qJ3797qeqAdnt5fVoEKzAHutWvXxGOMMOO3334z2rdvj3J2xsSJE41PPvlELRcvXjQmTJhgpEuXzoiPj1fb3rx508iVK5da98ILLyTsY9y4cQ7bHT9+3MifP7+RPXt246WXXlL7qVixotpm4cKFybYJbalQoYJRuHBhY/To0WrJkSOHUaRIEeOdd94xypYta7z11lvGyy+/bGTKlMmoV6+ew+9nzJih9lGtWjV1TkOGDDGyZMliFCtWzDh37lzCditXrjQyZMhg3HnnncbYsWONESNGGHny5FHnuH///oTthg8frvZXuXJl49FHHzXeffdd4+mnn1brBg8enLDdjh07VHuqVq1qvP3228a0adNUPz3wwANGoOjSpYtqZ9OmTY1JkyYZ48ePN1q2bGlMmTIlYRv0E7ZDX2F9o0aN1G/Q11aKFi1qlC5dWl3bF198UX1/zz33GFFRUerck+P22283evfurX6He6J69erqOEuXLnX7u++++86oVKmSujbm/blo0SL1Ha4T9oFrbtK5c2cjc+bMxt1332089dRTxnvvvWe0bt1abYdrZ4J7M2PGjMaZM2ccjjd//ny17bp169y2C9vgvi5YsKAxcuRI1b8lSpQwsmbNapw+fTphOzwPOHfcz6+99ppqT4sWLRKeOXMbfId1PXr0SDjPffv2uTx+hw4d1P02YMAA46OPPjLGjBljNG/e3Pj0008Ttlm1apXapmbNmuqZwfHwbGHdpk2bktzj5cuXV/vANXryySfVuo4dOzoc15v7pVSpUup5wjOI52HNmjUe7wPXGP1WpkyZhP7AvQAuXbqkzuO2225T9yL23alTJ3UvPvfcc26vm7v9mu8OPOv169dXbRs4cKCRPn164/HHH3fYz+zZs9XxmjRporZD/+MdkzNnTof3hx0XLlwwypUrp/bbvXt3dU/gHkK/bN26VW1z+fJl46677lL36PPPP29MnjzZuP/++1X7cK9ZwTpcQxM8j9myZUu4N/EOLV68uBEdHW1s3LgxYTvzfPFOrVOnjjoPbOvp/WXlypUrxn333aeuifMz5Y6wE6qmUETHOt8ImzdvVuu/+eYb9Xn79u3qc5s2bYwaNWokbIcXBG5Ck/79+6vtfvzxR4ebCBcVN92NGzfctge/xcW3tuf9999X6wsUKGCcP38+Yf3QoUMd2n7t2jUjX7586obFRTbBixvbDRs2LGEdXtTY1noDYJAB4Y8H1PmFgxe0lUceeUTdQCZ4QWC7U6dOGcHA6tWrVXueffbZJN9hgGSCh9eZxo0bKwHh/JJ0FjYnT55U1wovnuRwPg6uFa4TXl7J0axZM3V8Z1wJVayDkLKCe7RKlSoJn//880+1HV5oVnA/4z619pEd+C1eOnv37nW4f7DeOmjp1q2berlZBS1o166dGiya/WI+b9ZzcQd+26dPH5ffo/133HGHupbO1xvP4oMPPpjkHse5W8EgCOtxXtbfe3O/LF++PMn2nu4DAyO87J2BoIDQ+L//+z+H9RDeEFSHDh1K8htP9msKmYYNGzr0GYQa9hsXF5fwPoPwhEC0gsERrovzemfwHsJxFtooGeZxITixjVWI4ZnBAOmWW25xeA86C9VWrVqpe9M6KDt69KhSdKyDfPN8a9eubfz7779e3V9Wrl+/bjz00EPqXbB27VrDG8LO/OsO2MdhNli3bp36/OOPPyaYKGACgOkT1xPmQJhArB5hMA/CDGqC/fTo0UOZ63bt2pXssWGqME18VjND69atlZnFeT3MzQBmTZhMYMqwzrXBLFOmTJkEE/SxY8eUQwhMhTBRm1SoUEGZOHAOzvTs2dPhM875zJkzcv78efUZJl+wZMmSZE1QacGXX36pTDvDhw9P8h3Wm1jnuOLj45X5CSZC9Ck+W4GZznqtYR6Didvsf3dYj3Pu3Dm1b+zL2VzpK+yul7Wdd955p7p/Pvvss4R1Z8+eVSY9zAdZ+8gVDRs2lJIlSzrcPzC3mcfB84HrgGkT/I2+NRfMPaEPUnr+uN9g2oWDkx24v/fs2SMdOnRQ96l5XMyZ4fnCc+18n/bp08fhc79+/dT/1ufBm/sFU0o4T2e82YcdmN7B9cyVK5dDn+J6wMRvvrNSCt5V1uuPY2G/mIIwp3lgvm3fvr3D8dOnT6/uqTVr1rjdP+4JmHIfeeSRJN+Zx0WfFyhQQB3DBE5Bzz77rFy8eFGZbO1AOzGniakPTM+YwEMX9wLe1+Y7y6R79+6q7d7cX1bgYIfnBs8SrqM3RJSjEjoZc5kQpgD/4+aCsMSFg30etn68iKwvWtx4zrZ2cNdddyV8jzkodxQpUsThM+aDAOZP7NbjJW3uG+BF7wyEKm6o5LZDOzEP5Dxh79wmPNDmsfEibdu2rXz00Ufy9NNPy5AhQ9SLC/PSjz32mKRL53o8hv7zag7CAgYEmTJlsv1u3759EhMT4zBosAPzaxC8cDpwniPGC87sY7s+MPvB7H93LF26VF5//XX1sseclYknwstbzPnj5NqJASLmRXE/wJMYL+vr16977E2cXH9gHg4v3w8++EAtdmAQmBLGjh0rnTt3Vs8E5qcR2oDzMV+kEKgA27gC19e8j8Edd9zh8D0GDLh3rX4L3twvEKp2eLMPO3Bu27dvT3KNU9unnjzr5vEB5oXtwPvAHXg2oSC4A/ckrofzu8P6HrUD9xz61NW7DQOpw4cPq3l9d9cpufvLyieffKKUkeTOSSJdqAII0DfeeEM5K0CovvTSS2oEA6GIzxCqwCpUfYHzqCm59doC4l+SOzZG3xghY5QKjRgOK/PmzVMPHkaOrn4Pwetq1JkcOJazk4U34OGG8MeAY8KECeoBgpDGKHnixIlJNJmU9j/ulRYtWsgDDzygwiQwasaoGw4rCJfxNa7a6Uy7du2UswhG2HAo+/TTT5VTjd0LyZvjmP1h9h8c5VwJN2i3KQEORXjuFi1apO6vcePGKUedhQsXKgcV89hYX6lSJdt9wILkDucBj7f3i52nr7f7sAPb4CU+ePBg2+9hhUgNnl5XCBNok85kyBBaoiKLzXVK7v6yAksInumUEFo95SHuNAV0KrSoOXPmKA9GU3ji5WgKVdzApnAFGPH/+eefSfYFz0Lze39h7hvHdx5FYp35vXU7u3bCY9hTt3IrGFXihYEFL4w333xTDUQg/GCasuOtt97ySNOzAyYkV0DLgMYNTdiVtvr1118rrRHe3NbReXLmK2+BuQvaI9oDT0ITCFVP8Ic2C9AvmBqAUIXJFxqUL5NMQJPCdAUsO66uf2rOES8yTHVggXYGb30MgvHSM83S0JqSO7YJNDCr1gKPdwgQcyrGF/eLN/tw1Sc4N5hAPT0vT/frKWbfwiM2JW3A7xHT6Q68o6CNo/+t2mpy71Hcc/DydvVuw76cLX4pub+czeWYLkwJYTmnagoPu4xKMONCo8AIBS8g02QA4QrzLzQsZy0VZoJffvlFmXZMYEqF+QsPJ+bl/AW0DNzo06ZNczAxwt6PUBm8QM2bBaN3uKdbzxs3OkZlOAdvgfByxtQQrG1xBqYVPJgpWaymO2dgisHIGvMdrkbc5ojcqmnC/OapsPMUHAcvMjOkBcCkiHAUT+9RT+baUgJMvZjnR2gZ2gnt1Vdgf7gOGFTYvUStYRrunkNn0I/O/YH7HuZ+817DfYWXN0KZIIDcHdtk6tSpDp+nTJmi/jdfor64X7zZB/rErj+gReH9gkGaM9j+33//ddsGV/v1FMwTY7CCQTOmC5xJLvkB7glkHlq0aFGS78x+wTvo+PHjytplgvPCNYGFwdXcJfq3UaNGyrfDarY/ceKEsgrB+picedqT+8t5ThaKREoIS00VDx+ARoUXCoQoHCtw42HEg+8hQM0YVVNThaDE4ixUMZ8IzRYPIibVIYwhvPbv369eLu7mF1OLOQBArB9uOkzy42ZCrCYEOkx9JjBnoI2YN0YsGDLz4IbFfE5K8mi+9tpryvwLwY1RJEZ2MHXCucvqtJVW1KtXTwmMyZMnKw0EsYoY9cLCgO8wl4iHD6Y3XFvEZeLl++GHH6oHCM5cvgJ9As0dbYCzBPoGL3DEFmI0nhy4B/FyQewg4jnxUkGbfdU2xGRiPhX3A87dl4wePVppYRig4uWDQSUGYHBQ+v777xMGYxCAmFrBgBDaLZ4//MZuvgsxhLivMF8PawX6A/vavHmzsnwAPGeY48c5YTCMZ6JQoULK4oT24MUKrdEKnlGY6XGdILRgDsf1Mi0ivrhfvNkHrjviVzEXj3sF28AChQEQNF3EwcLZENvhXYRYdsQ+Q5jA2uQKV/v1FPQdfo/nC9ob3pvQEA8dOqSmfu677z555513XP4e7Uc727Rpo+KX0R7cBzgnXH/0N7Q/xPXi/GJjY9X7C78xrSlWh01ncF5mzDy0TJijsS8IRMyVJocn95fzXC2mN1KU+tEIU+CiXqhQIRVO4hxeM2jQILUOcUpWEIOG9XaxdFj32GOPKbdzxAwiJjG5eEQT7NPZldsMnUD4jxXEvWH9ggULHNbPmzdPhVDAxTt37tzGE088YRw5ciTJsb7//nsVW4U41ltvvVXFYe3atcthGzPcwDlUxnRHN/sKMYGIAY2JiVHu7PgfMcDObv9pCdzk0WeIyUOb8ubNq2JWY2NjE7b56quvVMwfrhNCSXCdp0+fnuQ+QIgEQlucQWiCXXiCMx9//LEK8cA1QXvQf2bfJgfiphE3h/sJ25vhNa5CahBu4Yy7Y5mhI59//rnhKXb3KUDb0AYrJ06cUNsiVhVxhwgNa9CggfHBBx84bLdkyRIVM4j4aXfhNVevXlXPJeJkESaB88Xf1jhcE8Q9Ir4a4V/oe7QPMZe4X537Bvc+nlvsE/Glffv2dQhN88X94s0+EKKCfaA9+M56nyGsBSF1eA/h3kYcc61atVQsNkJP3OFqv+YzjfAmu/eMGWdrXY9QIISf4FxKliyp4m9//fVXIzkQyte3b1/13kX7ETuL+8YaeoX7pmvXrurcsA3iiO3uCeeQGrBlyxbVNoTfIHYa8fzr16932MbV+Xpzf5nHd77nPSXqvx0QQsIIWDA+/vhjZW6DdSbSgGUG0wQwW7rT8AjxNWE5p0pIJAPPdpg5Mc8ViQKVkEASlnOqhEQimNfFPBHmqRASgDy3hJC0hUKVkDABHr8Io4GTCpy5XMVyEkL8B+dUCSGEEB/BOVVCCCHER1CoEkIIIT4iIudUkTAAlQoQbOyvdHGEEEKCG0SUIjEEMiv5KolPRApVCFRPc0USQggJbw4fPqwyLvmCiBSqZjosdGRyOSMJIYSEJ+fPn1cKlrsUid4SkULVNPlCoFKoEkJIZBPlw2lAOioRQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4iIiMUyXhy7//iixbJhIbK1KggEi7diK5cwe6VYSQSIFClYQNJ0+K1K8vsnOnSIYMIjduiAwaJLJokUijRoFuHSEkEqD5l4QNvXqJ/PFHosZqGCJXroi0bi1y4UKgW0cIiQQoVElYEBcnsnix1k6tQLBevKi/I4QQf0OhSsKC+HiU9LP/DhWdTp1K6xYRQiIRClUSFhQqJJI/v/13ELY1a6Z1iwghkQiFKgkL4Jg0YkTS9enTizRoIHLvvYFoFSEk0qBQJWHDM8+IfPyxSNGi+nOWLNp5ackSlHYKdOsIIZFAlGHAlSPyCtPmyJFD4uPjWU81DMEdjTnWbNlEMmYMdGsIIZEkCxinSsIOaKU5cwa6FYSQSITmX0IIIcRHUKgSQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSoJW/btE+nfXyd+QFL9FSsC3SJCSLjDkBoSlmzeLFK3rsi1a7piDTIrLVwoMnKkyMsv++eYSIe4f7+OjS1SxD/HIIQEN9RUSVjSp4/IP/9ogQrM6jXDhokcPOj74y1dKlKqlF6Q0alKFV0oPVDs2SOyZo3I8eOBawMhkQiFKgk7TpzQmqqrqjVff+3b423YINKypciBA4nrfvtNa8qHD0uacuyYSL16InfeqQu23367yNNPi1y9mrbtICRSoVAlYUdyiTddCduUMmaMzuJkPS40YxRInzbN9e8gcP/6K/n2egr289BDIj/+6NiOGTNEBgzwzTEIIe6hUCVhR4ECIpUq6Tqqdjz8sG+Pt2VL0uLoAOu2bk26HmZhmIcx71qypMgdd4h8803q2/HTTyLbtiVtCwYRH32k8yETQkJYqK5bt06aN28uMTExEhUVJYsXL072N2vXrpV77rlHoqOjpVSpUjJz5swk20ydOlWKFSsmmTNnlho1asgvv/zipzMgoco772iHIZSEA3BUAkOGiJQo4dtjwcRqJ8BxbHznrJ3CLAzzsAm01RYtRDZtSl07/vjD9Xdw2PLHXDIhJA2F6qVLl6RixYpKCHrC/v37pVmzZlKvXj3Ztm2b9O/fX55++mlZYYmFmDdvngwYMECGDx8uW7ZsUftv3LixnDx50o9nQkKN++7TGmTnziLlyumaql9+KfLGG74/Vu/e9iZlOEl17+647r33tFnYqk3CbAvzMczIqcHdYMFOwBNC/ICRRuBQixYtcrvN4MGDjbvvvtthXdu2bY3GjRsnfK5evbrRp0+fhM83btwwYmJijFGjRnnclvj4eNUe/E9Iarl5E/euYURF4T7XS8aMhjFtWtJtmzRJ3MZ5KVw4de24ccMwSpc2jAwZHPebPr1hdOmSun0TEo7E+0EWBNWc6oYNG6Rhw4YO66CFYj24du2axMbGOmyTLl069dnchpC0xtQyYcZ9/33tGPT337poujOFCiWapK3AfOxOk4Rmi9mTp54S6dpVZNGipHOn2Mfy5SJlyzquh2kZ5nBCSIQlfzh+/Ljkz5/fYR0+o5DslStX5Ny5c3Ljxg3bbf5wM6F09epVtZhgf4T4mmLFRHr0cL8NzMEff5x0PczHiK21A2ZkZIT66qtEgQxXg2bNtHC1FmJHG+Cs9Ouvev62fHntCEUISRuCSlP1F6NGjVLV3c2lcOHCgW4S8ZDr1zE3Hz7dVaOGyLvvOgpCaLqDBol06GD/m9mztUA1BayZ0GLZMq0VO4P9Vasm8uijFKiERLRQLVCggJxA5L4FfL711lslS5YskidPHkmfPr3tNvitK4YOHSrx8fEJy+G0jsiPYC5fFoEPmbexoadOaSejbNlEbrlFpGJFkW+/lbCgVy+RI0e0xgrHJeQoHjtWC0M7PvvM/jusw3eEkOAhqIRqzZo1ZdWqVQ7rVq5cqdaDTJkySZUqVRy2uXnzpvpsbmMHwnMgmK0L8S9nzoh07CiSIwfM89oz9ZNPPPstLPV16miBAU0V7NihzZ0rV0pYkC+fnh/t2VOkeHH320JTt0sQgXUXL/qtiYSQYBOqFy9eVKExWMyQGfx96NChBA2yU6dOCdv37NlT/vrrLxk8eLCaI3333Xdl/vz58vzzzydsg3CaDz/8UGbNmiW7d++WXr16qdCdrvDeIEEBHGjgSzZnTqKpEjGSuNSff57877/4QmT3bkdHHGi60MyGD5eIo3HjxDhbK1jXtGkgWkQIcYnhR9asWaPclZ2Xzp07q+/xf506dZL8plKlSkamTJmMEiVKGDNmzEiy3ylTphhFihRR2yDEZuPGjV61iyE1/uWrr1yHjZQsqUNQ3IGIKYSk2P0+Xbrkfx9unDplGLffrkNjrGEyMTGGceJEoFtHSOgS74eQmij8IxEGvH/hsIT5VZqCfQ8qwYwalailOhMXp83C7n7/5pv2qf9y5RI5e1YijqNHRV5/XWvxAN7AKGGHEB1CSPDIgqCaUyXhQd68rh2TMmcWyZrV/e8xF2s31EMcpjVDEZygliwRmT9fO0OFMzEx2msY54kFDk4UqIQEHxSqxOe0a6dDRpw9VjEH2KWLYziJHYirRBIFCFEsZmzm/fdrLRYg5SAcvlu1EmnbVguYESN8V/GFEEJSAs2/9AT2C6hZ+vjj2pMXwhSmYOTjRTUWT7sc/mzz5unqKvAGRv5eCFk4MSGpgZ15GI5Q7dv7/HQIIWHIeT+YfylUKVT9GlazYIH+/957ddFsV7GY3gBncKTdc56zhcCtWjX11V4IIZHBeT8I1aBKU0jCi9tu03GYvgbhOXZaKuZx9+/3/fEIIcRTOKdKQg4kjLerXwozM8zChBASKChUSciBpPXR0UkFK7TXwYMD1SpCCKFQJUEI4li3bnUdJlOkiE5XWLKko6l51iydfSjYgQPW+PEir74q8sMP9FgmJJygoxIdlZIFYSrr14v8/LNI7tw68QCSMPiaa9dEBg4U+eAD/Tc00cce05/tkkWgXcgJ/M8/OuF+pkwS9EyblljiDecHZyukGly4UMfw+hvUokC40i+/JOYfrl3b/8clJBih928Qd2S4ggQLLVuKfP+9nrOEMxBe/nPn6uLXvgTCBkLHmjgCx4TX8HffSciza5dIuXJJNVMI11de0ZqrP0HJYYQ1IUQJpnLE/0Koo8A6zeYkEjnPjErEF6Cw9T336CQMKC07erTrlIJIhbd6tf4bL2IIBGiGbdporcdXIOwGGqlzJiYcE6be7dsl5IF52i4xPs75o4/8f/x+/RIFKjCv+ZAhIgcO+P/4hEQCdFSKMFDwGsWrf/tNv1RR1/PFF0W6dbN/2X/4YVJBB8GK33pSccZT9u51LdjB77+LX4CZGXmGixXTSSkwJwsztz/AwCEl3yUHciEjdhcl9mAmR0YraKVWzp/X1ga7UCTEDmOgRQhJPRSqEcSff2ptBVgFJYQkhC0yFVlBNiRX9TqhcR0/7ru2JZfHFhq1r8F5Y84WplfEvl64IIJSvcje5FTW1yfUqGE/cEBfIjlGSs3zmBOdMkU7dkF4IoVj9eoi//d/idu5G7BAqOJaE0JSD4VqBAAB2revSJky+qXrCniiWsHc6Z132mdBQvFwZC/yFbffLvLww0nNo/hcurR/nGmgkSKdonWAYZq4Bw3y/fE6dNDF2q3niL7F8VJaJxaF36GVWjVQCNArV7QGbgIHM5j87eJ78duHHkrZ8QkhjlCoRgDw9pw6Nfntsmd3/IwXPpxnnB1rIBQgoJHM3pfMmKE1LCsQQkuX2guD1AJt1EzWbwVCFiE97gYgKSFbNpEff9TasSlYkawC+ZDr1k35OdgNeiBYly93XDdxoj6uec7m72D6r1AhZccnhDjCNIURAASqqRHZge+yZBFp3jzpd0hOD630pZf0/CuE2yOP6Ny7yVWb8ZY8ebT2iHCPnTv1PCeEjT8EKkAJOlcl6iB8/BGigxJu8JyGuRVLap3PIajRP3bnge+sPPCAzosMb1/0M+Zgn3nGfj6dEJIyGKcaASE1OXNqr09XAhWaC6rBQFi6Ai/tY8e0NhsuXQaPVySQcBZIEKjoCxQDCHYQamSX8AKCFmXyUmpWJiQSOM+QGpISKlWyD+UAKKe2Z497gWq+pOFMFC4CFUATnjRJ/42BBfoIgwycJ0ylzkDTx/wltGhXGm5a8+CDIr17J56DadqFU9QLLwS0aYREJNRUw0lKuNFmmjRxNP9CgODU4RGcN69ENAgvQgzpqVMiNWuKdOyYdH4ZTlzdu+sBiJkqEWZ1OFcFGlxXtG/+fB1DDM0VYVO+Ns8TEm6cZz3V4O3IYAfmXaQA/PvvRE0GCQeQ4Ye4B6EpcOTB3LKpoUKjhfa+YYNItWrsQUJCkfM0/5KU0ratjsWE+RL/b9xIgeopcMpC2IlzbC8E64QJvCcJIYnQ+zeCMGM+iXds22afPAHrtmxhbxJCEmGcKiEeODTZxbNikILvCCHEhEKVkGTo2dNeU4VJGJmqCCHEhEKVkGSoVUuXpLMmg4DmOmqUfcKMUAfzxZh3R5EDVwlDCCH2UKgS4gHIPITkF599posPILsUSqaFG8hmhbhmmLXvuEOkVCmdRpEQ4hmMU42QkBpCPMkwdffdOtbVOXRo/fqkeZkJCXXOM6SGEOIv3n1X5yO2Cx0aN479Togn0PxLCFGgMo9dEXM4af36KzuJkKAQqlOnTpVixYpJ5syZpUaNGvILJm1cULduXYmKikqyNGvWLGGbLl26JPm+CXLwEUJSXdPWLnQI5l+kZURMLuquwmELsya9eomcPs1OJyTNhOq8efNkwIABMnz4cNmyZYtUrFhRGjduLCdPnrTdfuHChXLs2LGEZceOHZI+fXpp06aNw3YQotbt5syZ48/TICQi6NHDPnQI5mCMa++7T+eRRrrGCxdEPvxQ5P77RS5fDkRrCYlAoTphwgTp3r27dO3aVcqWLSvTpk2TrFmzyvTp0223z507txQoUCBhWblypdreWahGR0c7bJcrVy5/ngYhEQGKCaBIgDURP+ZTX3xR12GFMLWah/E30l7CI5oQ4meheu3aNYmNjZWGDRsmrEuXLp36vAFZyD3g448/lnbt2kk2p2rLa9eulXz58knp0qWlV69ecubMGZ+3n5BIBGXkUHRhxgyRDz7QHsFvvCGyerX9fCuySqFCDiHEz7l/T58+LTdu3JD8+fM7rMfnPzC8TQbMvcL8C8HqbPp99NFHpXjx4rJv3z558cUXpWnTpkpQw1Rsx9WrV9VidaMmhNiDUoBdujiuQym8uLik20KTZVQaISHg/QthWr58eanuFBwHzbVFixbqu1atWsnSpUtl8+bNSnt1xahRo1SpN3MpXLhwGpwBIeFD167aYckZzME+8YTWYjHHinnXsmVF+vUT2b8/EC0lJEyFap48eZTmeOLECYf1+Ix5UHdcunRJ5s6dK926dUv2OCVKlFDH2oucai4YOnSoqp1qLocPH/biTAgh//ufTtcI4CFsegm/9JJe36GDdnTCzM7u3TqtY+XK+m9CIgm/CdVMmTJJlSpVZNWqVQnrbt68qT7XhEeEGxYsWKDMtU8++WSyxzly5IiaUy1YsKDLbeDYhMxJ1oUQ4jlZs4qsWSOyaJFO2fj88zp29fXX9Zzq/Pl6OzNXMDTYixe1k1NywHsYheDtzMuEhBp+Nf8inObDDz+UWbNmye7du5VTEbRQeAODTp06KS3SzvQL0+5tt93msP7ixYsyaNAg2bhxoxw4cEAJ6JYtW0qpUqVUqA4hxH9AO23VShdtHztWpEoVvX7pUvv4VpiE8Z2rpPz4/pVXRPLl03V+MZfbuTN8HngVSeji1yLlbdu2lVOnTsmwYcPk+PHjUqlSJVm+fHmC89KhQ4eUR7CVP//8U3766Sf5DgFxTsCcvH37diWk4+LiJCYmRho1aiQjR45U2mikgMTu778vsnmzCCzpTz2l57IICQR2c61WRyZXvPyyyJgxjtotwnNwf9s8/oSEBEyoH2Km4J07dcA9RvMY6UNDwMtowgRtkiMkrcE8qjnfagXO+Agxt8vNguQR0FCRvN8OZG/CnCwh/oQJ9YmKIzQFKjAz4Lzwgi5HRkhac++9ep4VmFFt0F4xe4Oas3bAr9CVQAXbtvmhoYREckgNSQpyXKxbZx+EDxPa4sXsNZL2wMT73nsiX34p8vDDIg88oOdKt2/XdVntwLSFO9NwTIzfmktI6M6pEt+CNHGuwAvq2jX2OAkMuP8efVQvngBnfeQT/vZbx0EiNN1ChUQsidgICSmoqYYQ8O8qV87eMQRJz5s2TXvNedAgXcEEL8I+fXSKO0I8ASnAK1XSf5taK+5xeAy7SI5GSNBDR6UQc1RCDlYzegjzqRCwEKgwuaEkF7TVFi10ID5Sy/kLzOtWqyayb1+ipgGnKYRFxMZqTYSQ5MC0BeJcd+wQQaIzlJazJvQnJNQclShUQ0yoAggthCLA6xIj+0uXEIqU+JLCqB+p4n7+WSRHDv+04a23RAYP1gLdCjSMAQN0HCMhhAQz9P4lCgTdI4MNsi3CGxj1CSBMzXg//I/0cBMn+q/DMBfmLFCtAf+EEBKJcE41xEHaOFdzrGbqOH+QJYvroH98RwghkQiFaogD4ekqDZydJukr2ra13z9Mz0iuTgghkQiFaojTvLn9esxttm7tv+O2a6fzwJrHMrVWpEuESZoQQiIROiqFoKOSFWSlqVsXRd0TNVYIOATdY51TTQKfgvnTJUtEFi7UnsgI/H/8ce2FTAghwQ69f4O4IwMJSme9+67IvHlauLVsKfLssyK5cwe6ZYQQErxQqAZxR/qT334T+fRTkfh4HY+KJOURVJSHBJirV3UKTMSSFi2qrREh8NgQkiwUqhEoVMeP11mLzHqV0ESRVQkB89REib85cECkXj39P5Iy4P7DI7N8uU6kT0goc94PsoCOSkHMrl1aoAK8zMyKNIhBfemlgDaNRAgdO+p4aDP3NObtUbYNKTFxb2LQh/qnhBANhWoQ8/nniRqqs4PQJ58EokUkkoB2+tNPSasiIZQqLk4nF/nf/7RJGBVqCCEUqkENNAJX5bGuXHEdn0qILzh92v33ELYQsLCgIDb51Cn2OyHUVIMYzGXZlXtDyEzt2u7rURKSWu66SyRbtuS3w+AO9+mCBexzQihUgxjEfdas6VgGy0y08OabgWwZiQQgUIcO9Wxb3Jdnz/q7RYQEPxSqQQzmU7/7TmTgQF1SLXNmkfr1Rdat05mLCPE3L74oMmmSSIEC+rMr6whMwLwnCWFGpaAPqfE1qLcKp5JVq7Qm0r49QyNI8mDuFDV0ly0TefJJLVzNOX1oqRCoa9dySoI4geD6kyd1sVxoBUEGQ2pIqh2fMBcLp5JZs3QWJpiXPTXxkcgFUw45c4o88YSeOy1TRq/HwKxPHy1sOcdPEsAIrHNnkTx5RO68UyRfPpFXX/VvlY8ggbl/I0hTHTJExxU6h0gAFDSvVSsQrSKhCjzQkdnLVQlAu8xMc+aIfPONTiSBgg8oyuDp70kI0bChNl1YXzYYdSHAfuRICRaYUSmIOzIUiImxD9TH3O0zz4i8804gWkUiAeSnbtBAZONGLUTxfsX7FoIVOautzngkxImNFala1f67rFlFTpwQueUWCQZo/iWp4tIl+/WYG7t4kZ1L/Mfbb+uqSQAWQFOBwfw+Q3HCjK1b3Y+u/vpLwhkaXiLMImOnEeAFB69iQvyZHcxuOg1aKzRVEmYmMVfARJE/v4QzFKoRxPDhutapc9xrhQq68ggh/qz768pKgrlWEkY0aiRSqFDSETw+t2hBoUpCF2gGqCby3HM61hUm3g0bdFIJeG3CMQ91V1HxJgi93UkYgXvO1bwpkvOTMCJDBu0OjuB6YLqFV64s8tFHEu7Q+zdMHZWQNu7RR0WWLnUsG9e7t3ZIYvgDSUuOHhW55x6dT9icT4WQRbQF5lqDxG+F+DooftkykSNHtDkMxaCD7MVD798g7shgY/Jkkf797ZPuf/211hwISUvwbh01SmTJEj3QQ+KRwYNFcuXidSCBISS9f6dOnSrFihWTzJkzS40aNeQX0wXQhpkzZ0pUVJTDgt9ZMQxDhg0bJgULFpQsWbJIw4YNZc+ePf4+jZBj5kx7gQrt4NNPA9GiMAVq17ffamkxe7ZrF2sit9+O94EWrigrhy6jQCXhhl+F6rx582TAgAEyfPhw2bJli1SsWFEaN24sJ5G2ygUYLRw7dixhOXjwoMP3Y8eOlcmTJ8u0adNk06ZNki1bNrXPf1x5QkRwdjBXMgC1MIkPQLwd5okeekjklVd0BhlIDkxcE6/AABDhNfBCL1lSO865GX8TErwYfqR69epGnz59Ej7fuHHDiImJMUaNGmW7/YwZM4wcOXK43N/NmzeNAgUKGOPGjUtYFxcXZ0RHRxtz5szxuF3x8fHQ4dT/4UqPHoaRIQNeVY5LunSGMXq0Edns22cYPXsaxh13GEbVqobxzjuGce2a9/tp0SJpJ6ODb7vNMP75xx8tD1tefTWx+/A/ujV9esNYujTQLSPhTLwfZIHfNNVr165JbGysMs+apEuXTn3e4GYkf/HiRSlatKgULlxYWrZsKTt37kz4bv/+/XL8+HGHfcIeDrOyu31evXpV2c6tS7gzaJBIlixJw2cQQta9u0Quu3cneiFi2gDZX/r1E3nsMe/ykqIiNyan4f1lBfs4c0Y7aBCP+PtvkddeS+w+gG7F3337RkS6WBJG+E2onj59Wm7cuCH5nQJ98RmC0Y7SpUvL9OnTZcmSJfLpp5/KzZs3pVatWnIEkzAiCb/zZp9g1KhRSviaCwR2uFOqlE4J17y5jk1FdrCOHfW63LklckHuUcx7msLQ1DG/+kqX7vEUuLHaTVpbTcPEI1De0E5wonsx90qXCRJKBFXyh5o1a0qnTp2kUqVKUqdOHVm4cKHkzZtX3n///VTtd+jQocq7y1wOHz4skUDZsiKLFungesiRGTN0THZEAw3SrqIA3FERf+QpxYvDTOL6+2rVJBJA1rnfftPaZkoxQ75S+j0hESFU8+TJI+nTp5cTTiN2fC5gVjxOhowZM0rlypVl79696rP5O2/3GR0drRygrAuJUNxlbvfm7Q2vdFTwtts/Msq4SigeJkCLfOMNXdGrUiXtn4XTTolwhZ8XqtbYpTDEwLBECZ80mZDQFqqZMmWSKlWqyCqLSQ3mXHyGRuoJMB///vvvKnwGFC9eXAlP6z4xPwovYE/3SSIczJ3aCU+Yg+vW9X7iesIEnZoKoA7a009rN9YwB6f98suOEURr1uhKNM7TzMlx2206rhqYlwZjE3Tnhx8GXb4AQtxj+JG5c+cqz9yZM2cau3btMnr06GHkzJnTOH78uPq+Y8eOxpAhQxK2HzFihLFixQpj3759RmxsrNGuXTsjc+bMxs6dOxO2GT16tNrHkiVLjO3btxstW7Y0ihcvbly5csXjdkWC9y9xweHDhlGoUKKbqXWJijKMxx4zjHPnvOu+69cN4+hRw7h8OSK6/d9/DSNPnqTdZy5LlqRsvxs2GEaXLoZRv75hDBignbQJ8Sf+kAV+na1o27atnDp1SiVrgCMR5kqXL1+e4Gh06NAh5RFscu7cOenevbvaNleuXErTXb9+vZSFDeg/Bg8eLJcuXZIePXpIXFyc1K5dW+3TOUkEIbbATolJQMzTv/eezkRgApmASeizZ71zWoJ69Z81JRKAjxYWV12xfbvOm+4t996rF0JCGeb+5fxqZHLunK6WgSTJdmzZokNvSBLg+IZMSFeu2HfOrFkinTqx40jwE5JpCgkJSuD85kqggt9/T8vWhBSY60Sss8XIpMBnCNvWrQPVMkICD4VqkLB5s07NhhRt9eqJfPGF+zBIkkqQBcOdB0yRIuxiN4wendTEi0pfKDWIsoK+BvGqI0aI9OypnZeYYpkEKzT/BoH595tv9AsK73h4TmLEj2B4pJM1M80QP9CyZdK4VbidYmSDzEvOqhhJwo4dOkcvQmsaN7YPjUktCxaIdOigB5m4JHhGEG/9448ixYrxopCUw9JvYVj6DcIT2Y8wEnfWTPECQT0B+NYQP4B0gq1aifz0U+I6XAxUncH/JCimvmFUwDyu9fnA2AfZSqEZExJMsoC5SgLMvn3Iaexa4K5YIdKtW+plB8q9QXDfdZeuY5k9e+r2GRYgQHLdOm17h8pVtKi2vVNDDRoWLxaxK0AF4wLSG+LexmUkJFigUA30BfBzijYoYU2b6nRyGN3DdAaz8tq1WsBGPLC5V6+uFxJ0oPaFOR3iDDTXCxcoVElwwUmjAIM5oXLl7JUjzE81a+ao1SLE8to1z/YN51YkEIJAxUsJn/EiwujenKMiJJipU8d1lRrUxaA/GQk2KFSDQFGCNyPCFMy0tKZ2OmmSzoAHy2SVKnqaD3lWMceEymXJsXq1Lpbi/FKC6WzbNpE//vDDCRHiQ3C/t2njOOg0/x4zhpZ6EnzQ/BsEIIsMysZOnSqydasegffoIVKrlk7ug9F6fHzi9tA0ESeYXExgXJz74yb3PSHBwGefiVSoIPLuuyInT2pBO2xYyrI2EeJvGFITBCE17pg4UWTgQHvPYLxcUGPbFahwB98bOzMvYglRgvaWW3zfZkIICQWYUSkCgenXrloZTLrQbt0BjRfB8tYcB+bfqDBCgUoIIb6F5t8gB4LRlUORJ/GrU6ZobRXzs9BM4Rg1ZIg2HxNCSEDZulVkxgzt/IEaxIgfzJ07pC8Kzb9Bbv5F8oc77tChMM7CFYLyuec83xf2kdoQHUII8QnvvSfSu7d+KZnelPDMRBwgXnppAM2/EQi0zIULHU21MOHiXuzXz7t9UaASQoKCo0cTX2AY7UOoYoEXZt++EspQbwkBHn5Ym26Rkg3B7vAGZs5TQkjIsmiRfQCymSoLoQk5c0ooQqEaImTNKvLoo4FuBSGE+IArV7TJzZXDiF1uyhCByR8IIYSkLQ8+6DpV1p13iuTPH7JXhEKVEEJI2pI3r+t6xsg96a7WcZBDoUoIISRtmTvX9XeoHOVpgvMghEKVEEJI2hIXZ5/VBkCghvCcKh2VCCGEpC21a+tQGmdg9kVNyhAu+ExNlRBCSNrSsKEWrFZtFQnN4Q385pucUyWEEEI8BgL02291ogczsw1KEX39tUjLlhLKME1hkKcpJISkHuQUQL6BxYu1MtS8uchjjzHLWFBgGNoUnDFjmh/aH2kKOadKCAlrrl8XeeQRkWXLEq2Nn38u8tFHel10dKBbGOFERQVEoPoLzqkSQsKaTz7RwtPUWLGA1au1YCXEl1CoEkLCGmilKfmOkJRAoUoICVuWLBFZv971VN7ly2ndIhLucE6VEBKWwLkUc6mucrZjfhUVoAgJKU116tSpUqxYMcmcObPUqFFDfvnlF5fbfvjhh3L//fdLrly51NKwYcMk23fp0kWioqIcliZNmvj7NAghIcbw4a7DHbG+YEGRZ59N61aRcMevQnXevHkyYMAAGT58uGzZskUqVqwojRs3lpMnT9puv3btWmnfvr2sWbNGNmzYIIULF5ZGjRrJ33//7bAdhOixY8cSljlz5vjzNAjxHKhFmzeLLFggsmMHey6AlyE21nUhlHLlRDBeR153QkImThWaabVq1eSdd95Rn2/evKkEZb9+/WTIkCHJ/v7GjRtKY8XvO3XqlKCpxsXFyWIEnAVRbBIhcuSIDlzfssUxc8z8+SK5crGD0pjbbhM5e9be7PvKK1qTJZHNeT/IAr9pqteuXZPY2Fhlwk04WLp06jO0UE+4fPmyXL9+XXLnzp1Eo82XL5+ULl1aevXqJWfOnHG7n6tXr6rOsy6E+BSMTSFQt293XL9mjchTT7GzA0CPHjpxj92l6tgxEC0ikYDfhOrp06eVppnfqdgsPh8/ftyjffzvf/+TmJgYB8EM0+/s2bNl1apVMmbMGPnhhx+kadOm6liuGDVqlBqNmAu0ZUJ8Cky+0FCdk4TjvoQL6uHD7PAUAu9dCMjWrUXGjxc5d86z3w0bJtKggf47QwYtYPH/zJkiJUrwcpAI8/4dPXq0zJ07V2mlcHIyadeuXcLf5cuXlwoVKkjJkiXVdg3MJ8iJoUOHqrldE2iqaSVY8Y5dtUrk2DGRypVFKlZMk8OStOavv1x/B9Xo0CERDua8ZtQokRdf1MIQ4xPM+kycKPLzzyLFirn/bZYsIitWiPz4o8gPP4jkyCHy+OMiBQp43w5CAi5U8+TJI+nTp5cTJ044rMfnAsnc1ePHj1dC9fvvv1dC0x0lSpRQx9q7d69LoRodHa2WtAaWQLjsW5UUOCrDh8XMIU3CBJSrcgVUpJIl07I1YcH//Z8WqMA0AGB8glfKCy+IfPFF8vuAl+8DD+iFkJA2/2bKlEmqVKmizLQmcFTC55o1a7r83dixY2XkyJGyfPlyqVq1arLHOXLkiJpTLQj/+CACdXYbNxY5etRx/cqVIs89F6hWEb8BE0TdukkLL0OgPvkk1aMUAKFpV8faTI5/9WpKLxYhIRpSA5MrYk9nzZolu3fvVk5Fly5dkq5du6rv4dEL06wJ5khfeeUVmT59uoptxdwrlosXL6rv8f+gQYNk48aNcuDAASWgW7ZsKaVKlVKhOsHE0qUimDp2nurFZ+Qipa9UGPLllyJNmzoK1CeeEHnvvUC2KmT55x/XcaYIlXHjRkGCka+/1pPcsNvDhAcnvnDE8DNTpkwxihQpYmTKlMmoXr26sXHjxoTv6tSpY3Tu3Dnhc9GiRRHek2QZPny4+v7y5ctGo0aNjLx58xoZM2ZU23fv3t04fvy4V22Kj49X+8X//mLSJMNIlw7GKvtl716/HZoEmgMHDGPdOsM4dizQLQlpfvrJ/tnBc1WrVqBbR7zirbf0xUufPvH/qCjD+PTTgHakP2QB66n6KU4VVm+L07ID2bOLIP+Fxf+KEJJkwC/Spo3IwoWJqQZhDoYBAErOffexy3wC1P61a0V27RIpWlQ7fviyFNu5c3r6A3NiziBcEl6cmTJJIAipONVIp149Pc3mPCcEc9bzz1OgEpIceFaQLO2tt0TKlhXJl0+kVSuRjRspUH0G5qjuuUebZZGzsUUL7VS3e7fvjrF2rb1ABcjOYU2WEgZQqPqrY9Npd37rVC80UySSQvwcISR5oDBhELpzp/b6hfMSZADxEchUh84FpjkA3pWY83SV49FbMiaj9YZRgXJA828apClE6mK8EO64Q5t+CSEk4CB2GuZeV6CKO0xuqeXyZW3+vXAhqeZRqJDI/v32bt5pAM2/IQruG4yuKVAJIUED5jLd4RwPmFKyZkUJssSUVgD/Y5k+PWAC1V/Q/BtAYG1BeE2lSlrgVqkiMnduIFtEvOKPP0QmTBCZPFnk4EF2HgktSpd27yCEF5OvaNtWz50+/bTIgw+K9Okj8vvvrr05QxiafwNYpebNN0Veekk7ZEDAYiCHaQzkNx04MGDNIsmBi4UMHlOmJGZsxzpcUA+qLxESNCB966RJjpXcoTki3hpxpWHOeT94/1KoBkiouvMyh7UETnk0FwcpMC/8V4rQb/NQhKQFyP+IOniwtmDuEyZZ3Ntvvx0RuVTPM6QmfED1O1de5ri3UUCZBCkffGBfUwwvpI8/DkSLCEkZuGdRteDUKT2dcfq0vod9IVA3bBBp317HFsL8iyoIEQDnVAMEtFF3ZMuWVi0hXgMHDrtwA4z6k3P+ICRYX0iYY0UpH18wZ44OJkYMFCqLIIPH/feLzJ6d+n3v26drFMfEiJQqJfLaayKXLkmwQKEaIGrX1uZfZ4UH86sQqLgHd+wIVOuIW1AQwvRitIK5qOrV2XkksvnnH+2IhHlas7wQ/sfnfv1ErlxJ+b737NEenZiCwQAWAnbECO3w5Mr0l8ZQqAYIvJM/+0w735mp1wDuO9yTqBlZvrzIO+8EqoXEJYMG6QtmHRHhImI0hJcJCXrwnCEzE3xxEEdOfMjGja4ryaOSyE8/pXzfplZqCmsAqxGOiYIWQQCFagCpX1/XjHz5ZR2DbVbkQPUN856Bk6m7+tckAGCOCDX8rLV+ob2iGvbtt/OSBDlIIFSmjL5kyMpXpIhIz56O72niJXiRIc3h/feLvPGG+21dlR7yhGXL7C8UtJTlyyWsi5QTzyhcWAtVRGNYvdqt9x9iV81izSRIQNXrrVt1ZQRoqbfdFugWEQ+A5RFpbuGPY1V04HuWN6/IyJHsRq/BYLJRIy3s/v3XfmrEBHO2mPtKKdHRKfsuDaGmmkKQcQvz79aHM6XgPrx+3f47CFXn7F4kiECWdwrUkAG+CkgZ6lyLFQNahB1TW/USdFz37no+0zp/amJmS4Kgxcvs3XdTV02kQwf7DEw45uOPSzBAoeoluHYvvKDfpbAC5s+vvcbj41N+EXCPVatmH6WB4zHskRDfAL8WV4oUnmFXU4HEBXv3ivz5p+vk+/feK1K1qg6pQYgNhGJqgFkPXsoQ0FhMAQvBDhNEEEDzbwp8VBAXbZpqcS8tWKDDvL7/PuUXYvRobUHBPWKOoiFk69QJy0xehASEO+90rY3mzCmSK1datyjEcVb5nUFawi5dxGfgAiGIH96/eOEiFKhdO50BKjVztT6EGZW8yKgUF6c1U1ee20htWblyyi/GunUir76qneNQDQkvADgqYXAXoBq+xFfA63H+fJ0j+O67RR55JGjmgCKJq1d1uVBkLLPKA7yP8eyxLKOXQKtAh+K+dnYKSZ9e5PBhkYIFJVhhRqUgMB25C4XCHGtqfV/gEYz5VRwHcapdu4rUqKEFOglRNm3SLqY9eoiMGaPnC2DCOnAg0C2LODCOQSZJjGtMYA6G4yrycBMvgTkN86QQoKZdPf1/JlnEj5oCFQIXoTC+qtEaxHBO1Qtwf7izMKQ2mgLOpEjDCUxHOoBiDrg/SZCDFwbMFRCi5ugLI6RWrbS3GV4spkfakSMiHTsGtLmRCixA27bpBZFRSJCFnPJhVoEs7YDpFXGicBQqU0bPbS5ZokcpuOcRbI8wB6Q+zJNHZPhw156ZYQDNv14m1G/ZUodKWU1HeBgRZ4pkH3bORt7M1+LhtpvzwVTC2bMp3zfxM2vWaLOCWQIOHsGYfM+dW+Shh1z/DkHIxYvz8pDw5M3/SnFZgWbSubPIjBkSaGj+DQJQU9c5Ex0GYRC0qRGo4OJF198FUWpLYucBidE65o9MzpzRmihi+Nzhi5gsQoKRixftE0FAe501K2yz2tD86yVQQFBsYf16HTCOJB54p8LqkVrq1rXXUqEJM6wmiMGcEkwXzvNFGGXhRnEF0hqWLWv/3Q8/6El2eKjBZDZ4MEdWJPRSV12+7D5PZBhCoZoCYL1AijOERjVu7Lu5mEcf1SFd1v3hbyxIeUmClF277EdDELT792v3bTszxtCh9uWIVq3SHmsYvWHuCVrvW2+JNGmSfAgDIcHCbclkGcPUSBhCoRpEIIwG79P+/bVygqQQiF1FiA2LnwQxmBO1yygAQYoRETxhEHaQJYtej/JEmDx3lXvSXG/VfPE3boQVK/xxBoT4nlKldOiCs9aB5wKxiUGSrMHX0FHJS0clQpLw228i99xjHy4AswZMXWZWD4TTIHDdlXkDXsOu4lcx6sKIa+xYXgQSGuzdq+e1UAoI9y8sOtmz63kzmPsCDB2VCAlGkK8SdfwQMmAVphiRmwHxptkWxZuR5cMV0Hhd5UaF0OYgkISatrpnjy5OPnCg9j84dCgoBKq/oKbKlxTxFXDRRmYBaJsIr7GrhACh+cwz7gvlduumvSOd508hqPGCgimZEJJqqKkSEixAA120SMegwvTbr5/Ofde8uUjr1u4zxyTnbISsS8i4ZAphs8IHyqhQoBIS1DChPiEp4X//Exk3LnGuFGmvEMwO0y6ELIQrKi04C1DMKT38sPt9w0sNmZm++EI7J8FL8sknRe66i9eKkEj3/p06daoUK1ZMMmfOLDVq1JBfUGHADQsWLJAyZcqo7cuXLy/ffPONw/eGYciwYcOkYMGCkiVLFmnYsKHsgUmMkLRi924tUIEpNCEs//lHV0AAqHaNqQXTK9gsVdWsmU4UkRxwVnriCZH33tMB9BSohIQEfhWq8+bNkwEDBsjw4cNly5YtUrFiRWncuLGcPHnSdvv169dL+/btpVu3brJ161Zp1aqVWnYgs/x/jB07ViZPnizTpk2TTZs2SbZs2dQ+/8ELjZC04Ouv7eNOIWChWaL6ARw0kMy5Vy9tykUA8uTJ2mSc2tRbhJDgxfAj1atXN/r06ZPw+caNG0ZMTIwxatQo2+0ff/xxo1mzZg7ratSoYTzzzDPq75s3bxoFChQwxo0bl/B9XFycER0dbcyZM8fjdsXHx8MlU/1PiNeMGWMY6dJhVtV+OXeOnUpICOAPWeC3IfO1a9ckNjZWmWdN0qVLpz5vQAV4G7Deuj2AFmpuv3//fjl+/LjDNjly5FBmZVf7BFevXlVeXtaFkBTTooW9IxLmV2vX1tWuCSERid+E6unTp+XGjRuSH5kzLOAzBKMdWO9ue/N/b/YJRo0apYSvuRRGBnxCUgoSPSMXLzCTOOB/xJeiMg0hoQDsKsj2depUoFsSVkTE5M7QoUMlPj4+YTlsrSZCSEoYPVrPjyL5c5UqIr17J2ZWIiTY+f57kXLlRAoVEsmXT+T++7UHOwnekJo8efJI+vTp5cSJEw7r8bkAcp/agPXutjf/xzp4/1q3qVSpksu2REdHq4UQnwFPXhQfx0JIKPHrr9oD3RruhekzVEVCcQjLu5UEkaaaKVMmqVKliqxChvj/uHnzpvpc00WKKqy3bg9WrlyZsH3x4sWVYLVug/lReAG72ichhBCn5CLATKEJIGCRAWzaNHZVMCd/QDhN586dpWrVqlK9enWZNGmSXLp0SboihZuIdOrUSQoVKqTmPMFzzz0nderUkbfeekuaNWsmc+fOlV9//VU+QOFSpRxESf/+/eX111+XO+64QwnZV155RWJiYlToDSEk/DhyBPHuIghxhxL19NM6RztJIZs2uS5VuHkzuzWYhWrbtm3l1KlTKlkDHIlgol2+fHmCo9GhQ4eUR7BJrVq15PPPP5eXX35ZXnzxRSU4Fy9eLOVg+/+PwYMHK8Hco0cPiYuLk9q1a6t9IlkEIWHJuXO6woc1YX+EsH27nu5DWmW885FLA7ULMKWNpFYkBWBkgqoxzh7s6FyaflMNE+ozoT7xBDi3vfyyyJdf6pcRwmpef10nefAXa9bA3COybZuew0WRciSQ8OcxgwwIVEz3OWd7xFgc9d+LFAlUy0IYpNN86in779avD+sKMs4woT4hgQAhByi2DBUJKtOVKzovL9b5y5Mctk5UqIeqZs5/ffedyH33iZw5I5EAThMJquzqD6A7Fi8ORKvCgC5dRPr0SRydmGUKJ06MKIHqLyIipIaQVIEakEitaX274+/4eJEJE/zTuW++qSWH1USHY54+LfLRRxIJuCvmAzlgnRZcu1YbD4oX12ORZcvSpImhCToPpQeRw/qtt/TfqHHav3+gWxYWsEoNIZ7E9Nm94bFu5Ur/9J+dzRNA0CZTlCJcQPgkIuWgrDtP/+EzahMAGBA6dtTKFroMxgNclkmTEusbEBdJTLAQn0JNlZDkyJHDPgk+Rvz4zl8SBft3Bpmb8uaVSAEJqsySssC8DM8+q+sUXL2q/8ZYwxyDmP8PGaJrGxCSllCoEpIcUIPscv3iTd6pk3/6r0cP+/Wwef4XkhYJIB8BojzattX12WvVEvn0U62FApSdPXvW/rcoXAWzMCFpCc2/hCRHmzYi334rMmtWosoE4fbIIyLduvmn/5D2EPGEsG1COzUD9ceP1w5SEUSFClqQ2mFeDlcgEomQtIQhNT4OqcG7D04SeAnAj6VOHa105M7t08OQtAYX9ocfdL5f2BfhFYNqSf6ujRobq71+EYf96KMiRYv693ghBi4FugR54a0JgmA5z55d5NgxkaxZA9lCEmkhNRSqPhSqeKj79tXOolAu8MDjnRsTI7Jxo85dTQhJHf/3f/oZg/NqiRJak4VDEp4/GBCgvcJa//nn2mxM/AQ8hv/4Q6RYMZE77wzJbj7vB6FK86+PHTbxsFudJfBwY7SMvAGIuSaEpBx49T78sH6urAIUVvEDB0R27tRzr716ac9h4gcuXtTJIxCrbZoH6tcXmTNHO9hFONRUfaipDhyoE97YpdWECQp5AwghKQMDVShFyLBHU28A6dBBZP58x5AvjG6qVxf5+WcJJZhRKcixE6aefEcISR54+iK5vlWgAnw+f56evmkCzG5z5yaNocYLDikOt2yRSIchNT4EJQrthCfmVx96yJdHIhEFbJovvqgn7GFyu35dIpHkTjtCuyVt+euvpKMa5wnvCIdC1YcgPRrme6wx+7CKwPT7xhu+PBKJGDBZiCpN48aJoAQiwnsQrAnX8gijShWRXLnsv4uO1p72xGmU4U4ApgTkgbRLSmJSsmTEXwIKVR8CT9+FC3VgeuXK+v7DfD4sImXLRvy9RuzAS++rr0RQD7h2bV3PDDZOsGOHyKBB+m+YQExVbOtWkVdfjbj+hOCEzwLe6bD+APN/lGTOmTOgzQue++n99/XLJ1MmkQIFdOe4S6QMTpwQmTJFZMQInZbTLtkJQCjDY48ldrzznGrVqr47l1DFiEDi4+MxfFP/ExJQBg7Ea9Aw0qdP/D9XLsPYtcswhgwxjAwZ9HrnJWfOiL1wq1cbRrNmhlG0qGE0aGAYX38d6BYFEaNGJb1XoqIMo2dP17+ZO9cwMmY0jHTpEu+3++83jAsX7LfHe7N5c8dj3HefYRw9aoQa/pAF9P5lPVUSKH7/XQdZOgMtAHMJ0DY+/NB+shCaAScRiXOoS/78IpcvJ+0XqPeIOXIuQAurCO4zZ2cQ3IOYwzfzQdqxd68OFoZLdvnyIXktzvshTpXmX0ICBQqCOpvRAEx1y5eL3HuvveDEb1C9mxDnQZqdQAXQJxFI7wwyZNiZenEPosSgKzMwKFVKpHnzkBWo/oJClZBA4e6FhZcgUiFict4qeM2i0q+/niZNJCGEKy8ud9+jPq/dwA4gsJ6xgF5DoUpIoICruJ0DCV5yyFCDsnKrV+vk+vgb61G2BaVX4AFMiBXUwnMehJkDMZiF69VL2l9wLrKzhmDgBq9zODsRr6BQJSSQMSLdu//3JKZLnCtF8vy33tKf4dIKl1cUBoXWsGaNyH338ZoRe0GIqka33ab/Roke/I+Yvi+/tC/Z07KlFp5WQYzfwFLy2mvs5RRAoUpIIJk2TeSTT7QGirgrCNlt25i4lqSMu+4S2bdPxzT366ervB886HogBkGLgVr79olCF3OlCxbo0obEa+j9S+9fQgjRVd2vXNHWEXcJHsKI86xSQwghxC9g2gELSRU0/xJCCEkelAdC7CoKQ6My/ODBImfPsuecYD1VQghJ61AqzGOiyDdiPOGwFuzmVlSnqVZN5OTJRI/1CRNEliwR2bxZhNNoCVBTJYSQtAJVXBD60rChTgwOQVW3rsi5c65/A0/cq1dTlhz/1CmR558XKVhQFxDv0UMLc2+BALUKVIC/9+zRSSJIAhSqhBCSFkBDbdZMZP9+x/Uo7P3000m3hxCFF2+JEnquM29enfDe04QMqGSEeGYkyj9+XAvYGTO0IIcp1xuWLbOPqUYbv/3Wu32FORSqhBCSFsDki3y5zsIJnxct0oLPCmKVn3lG5+wFZ85ooWrGNicH8kaj/qn1eBDI2I8ZB+0prhyYEF+dJYt3+wpzKFQJISQtOHzY9XfQ+KzaI3L42iVfwHYzZ2rhnBwrVrjO6/vNN+IVHTrYz/ti/23berevMMdvQvXs2bPyxBNPqMz/OXPmlG7duslFVFFws32/fv2kdOnSkiVLFilSpIg8++yzqnqAlaioqCTL3Llz/XUahBDiG5C5yBVIB2gt8I3qLxcuuN7eLjm+M8ikZGbqsvvOG/r0SUwggX2aGZhQB7hdO+/2Feb4zfsXAvXYsWOycuVKuX79unTt2lV69Oghn6Mqgg1Hjx5Vy/jx46Vs2bJy8OBB6dmzp1r3xRdfOGw7Y8YMadKkScJnCG1CCAlq4OVbp47ITz85mmShAfbs6VhlPSXJ8Z2BsPvqq6Trcbwnn/Sm5drEu2qVyPz5IkuXaqHaurVOc+gqIX+kYviBXbt2qcKvmzdvTlj37bffGlFRUcbff//t8X7mz59vZMqUybh+/XrCOux30aJFqWofi5QTQgLC2bOG0bq1LhyO12+mTIbx3HOGce1a0m3vvTexeL25oJB4njyGcfVq8sf691/DaNNG/w7Fx8191a1rGFeu+OX0Qg1/yAK/mH83bNigtMeqVasmrGvYsKGkS5dONm3a5PF+zMKxGZBk3EKfPn0kT548Ur16dZk+fToGBm73c/XqVZWOyroQQkiaAw0TlrejR0ViY0VOnNCFwO2S3SMnNKrLADM5PjRGJMf3pHoMNEhMjUFbhWaK/L74/N13zJwUaubf48ePSz7ERFkPlCGD5M6dW33nCadPn5aRI0cqk7GV1157TerXry9Zs2aV7777Tnr37q3majH/6opRo0bJCHjNEUJIMFCggF7cgcT2iAOFyXXHDp3FCA5DqELjKZj/RCFxLCT4hOqQIUNkzJgxbrfZjQn2VAJNslmzZmpu9dVXX3X47pVXXkn4u3LlynLp0iUZN26cW6E6dOhQGTBggMP+CxcunOp2EkKIX4FDUZcu7ORwFaoDBw6ULslc4BIlSkiBAgXkJLJvWPj333+Vhy++c8eFCxeUE1L27Nll0aJFktHOLGKhRo0aSqOFiTc6Otp2G6x39R0hhBASEKGaN29etSRHzZo1JS4uTmJjY6UKPN5EZPXq1XLz5k0lBF0BDbJx48ZKAH711VeS2YOKCdu2bZNcuXJRaJLwx8xeAw96hKfVry/StatI9uyBbhkhxJ9zqnfddZfSNrt37y7Tpk1TITV9+/aVdu3aSUxMjNrm77//lgYNGsjs2bOVwxEEaqNGjeTy5cvy6aefOjgUQZCnT59evv76azlx4oTce++9SuAiXOfNN9+UF154wR+nQUhwCdTevXVRczigIOgeDihIQbd+vU5hR0IHhKfMni0SFydSu7ZOU+hJmAwJfgw/cebMGaN9+/bGLbfcYtx6661G165djQsXLiR8v3//fuXKvGbNGvUZ/+Oz3YJtzbCcSpUqqX1my5bNqFixojFt2jTjxo0bXrWNITUk5Fi71jG0wlwQJtG7d6BbR7xh6NDEMBczTKZQIcM4dIj9mMb4QxZE4R+JMPxR7Z0Qv4I6lu+/b59MHRoO61qGBtu3i1SsmHQ9wgYfe0xkzpzk9wHT/8KFuhxb5cq64o2rzEkkzWUB66kSEgpcu+b6u+vX07IlJDUsWKAFqPPgCJ8RfwqzvjsBiWxMDz+sK9BgGgCZme65R8eeehNqQ/wGhzeEhAJIy2mnpeLFinJiJPQHR7i+dgnwrUn2W7RIzAlspjr87TeRXr183FCSUihUCQkF8DJFMWurFgONJ1s2EadY7jQH1VXgMDVunH7Bk5QNjho00NfUFXBMQzFzZ8EL4Qot112hc5JmUKgSEgrgZYtwmjffhHu9yO23i3TurFPdlSkTuHZNnSpSpIhI//7IDiNSqZJOVmBX0JrogRGS0CPloFlKDdcWaQeTSayjUhq6Mg1D0HJePSigoxIdlQhJGRDolvzeiW+VKJG33xbp148962oOHIOR6dO1IKxXTw9I7r7bfX8hdMosv+ZM7tzaccmTnMDEr45K1FQJISkDQsGVuRKeysQeZImDZg9P4CNHdOL85AQqqFlTl46zK7X20ksUqEEChSohJGUgFamdmRdReh4WziBeAAsA5lVh9jfTtyLpB6rcPP88uzJIoFAlhKSMatUS5wWtQJO69172qj+AifLjj7VTErRclJB77jn760ACAoUqISRldOumYyOt5kjz5f7ii+xVfwKv70KF3HsLk4BAoUoISRkQqEhGgFAQE3gmf/ONSK1aodOriP8cP15r3kikgNrL9KQlKYTev/T+JST1IDH81asi+fKFlinyyhUd5vLrr4nxnwhbKV5cZNMmZikKc87T+5cQEpTkzCmSP39oCVQwa5bI5s2OCRXw94ED2gGIEC+h+ZcQErnAm9YOM0sRIV7CWW5CiGdm0sWLRQ4fFilfXqRRI/t4yVADpl5o13bFuiKvgBfxARSqhJDkMyc1bSpy6lRiZRQkK/j+e5ECBUK79x55RGTZMvvvELaCMmu33JLWrSIhDM2/hBD3KfWaN0/0hjWTPfzxh0jXrmnfc5jv/P13kW3bfJNfuGNHkRIlXOfaZWYo4iUUqoQQ15pajx46p6yzAMPn5ct18oG0AppxyZIiFSro4txI5L9kSer2iVy5yErkCs6rEi+h+ZcQkhSkGURWpEOHkk9VGBPj/x7cvVvkoYcchTuE/aOPimzYIFK9esr37W5uONS8mUnAoaZKCEnKa6/pNHjunHWQ1eeOO9Km91CvFW2xhr7gMxyNJk5M3b4hmO2EJ9bhO0K8gEKVkEjg/Hmd6WjlSp2kITnmzUt+znLQIC1Y0wJUdLEr7o11mF9NDT17anOyVbDib6zDd4R4AYUqIeEOanfCS7dZMx0KU7CgyKJF7n/jTqBiHhLF0l95RdIMZDiyy3ML060rRyNPwcBg3TqRt97S9Uqx4G+sS6tBAwkbmKaQaQpJOAPtFMLUCrQwmE2h4ZUrZ/+7Tp1E5syx1w4XLBB57DFJU9wV6MY5IuTHExAWNHmy/k2WLCLt24s8/bRIdLRPm0tCA6YpJIR4B+YbnR1xMBcJwfree65/9+qrIjlyJP4W22OBpovYzrQGCfo//FAkc+bEdagpOmGC5wIVzldVqoiMGiWyZYsW1P36aQcohA4R4gNo/iUknNmzx96UCw10717Xv4NJdetWkb59RUqX1tVbkAv3668Dl0kJGiUEIzTluXO19683xbkhTBECZPYHBhdYVq9m6AzxGQypISScKVtWe/E6C1bMT0JYuqNw4eBLKg/tOaWmZ8Sc2g0wMEhAvGu7dqluHiHUVAkJZwYMSCpITC/XXr0konAXc8p4VOIjKFQJCWcaNhT5+GOt4Zmg5imqs6CgeCTRpo296RqDjlatAtEiEobQ+5fevyRSqsz88ot27kH2IbvwFE+BOXntWh1u0rixSNas4jcQV/vuuyIHD+p5XcyhIpl/SoDnb40ael9IImFWp4GjEwYZqekTEpKc90ORcgpVClVCPAMCaPBg7XFrZjbKnl3kk09EWrb0fS8iVvSFFxIr40DoQRAi53D9+inbJwoDwOvZDKnBPGrnznqwQSKO834Qqn4z/549e1aeeOIJ1dCcOXNKt27d5CLKKLmhbt26EhUV5bD0dMpocujQIWnWrJlkzZpV8uXLJ4MGDZJ/7WLpCCG+BSEt48c7pgrEMw3HoX37fHssVIgZMkT/bc4J4znHgndCSmud5s4t8tJLIj//rBP0w6OYApX4EL8JVQjUnTt3ysqVK2Xp0qWybt066YGKF8nQvXt3OXbsWMIyduzYhO9u3LihBOq1a9dk/fr1MmvWLJk5c6YMGzbMX6dBCDFB0gRnhx4zLAXztr4E2qjdYBnHQpgQFkIiRaju3r1bli9fLh999JHUqFFDateuLVOmTJG5c+fK0WRKRUEDLVCgQMJiVcm/++472bVrl3z66adSqVIladq0qYwcOVKmTp2qBC0hxI+gYo0rDTG5ajYkKfHxIm+8oRNSYEHqR+RoJiGNX4Tqhg0blMm3atWqCesaNmwo6dKlk02bNrn97WeffSZ58uSRcuXKydChQ+Xy5csO+y1fvrzkz58/YV3jxo2VXRxasSuuXr2qtrEuhBAvQUpDpDd0BoI2pc5DrmjSxN5xCMdHZZy0qo7jL/AOQpYoWNmQ3QkLcikjFeOFC4FuHQk2oXr8+HE132klQ4YMkjt3bvWdKzp06KC00DVr1iiB+sknn8iTTz7psF+rQAXmZ3f7HTVqlJqMNpfCCGonhHgH5jit86kATkRwVurWzbe9iecaGZDMYwAIWfwNR6NQjyuFR/Mffzj2J/7etUtk2rRAtoykpVAdMmRIEkci5+UP3CgpBHOu0DyhjWJOdvbs2bJo0SLZl0onCAhoeHeZy+HDh1O1P0KCljNnROCH8PjjOq8tUg36ihYt9NzpbbclrkOs65o1OvbV18Dz97vvRJo312XY4KULja5BAwl5Fi9OOkABWIfvSMjiVWDWwIEDpUuXLm63KVGihJoLPXnypMN6eOjCIxjfeQrmY8HevXulZMmS6re/INbOwgl4CYq43W90dLRaCAlr4LwD8yEEq2kqfecdrRX5KnvSU0+JwHqE6RbEp955p3+1xgcf1Eu4YWdG9+Q7El5CNW/evGpJjpo1a0pcXJzExsZKFUzAC3JWr5abN28mCEpP2PZf8eGCqP/4337feOMNJbBN8zK8i+HMVBY5TgmJZHr31nGYpgZk/g+NFXGkMTG+OQ7qqUJzJCkHYUgbNyZ1/MIApXVr9mwI45ch0V133SVNmjRR4THQLH/++Wfp27evtGvXTmL+e7D//vtvKVOmTILmCRMvPHkhiA8cOCBfffWVdOrUSR544AGpUKGC2qZRo0ZKeHbs2FF+++03WbFihbz88svSp08faqIksjl3Tsdd2iWMh3BNrig5SVsQawtHTrOknrlUq4Z5MF6NEMZvebngxQtB2qBBA+X127p1a5mMOLf/uH79uvz5558J3r2ZMmWS77//XiZNmiSXLl1SzkT4DYSmSfr06VXMa69evZTWmi1bNuncubO89tpr/joNQkKDq1ddf4eXNdIUkuABpnOkepw+PXHAgzq1cPhCpicSsjBNIdMUknDADGuBo6BdLOlvv4n8Z/EJa3Du0NhR5g3JI1CAHKbvQNWAJRGXppAZpAkJB6CNIidvs2ba0QVmYNOBqFOnyBCoMHPDkWrWrMQYV3grw1t42TJ4LAa6hSQCoJsZIeECEib88AOcD3SOW4S7vP2271MIBisIRYFAteYJBqtXay/omTN1n0DglighMmWKfVgLIamA5l+afwkJD+A1axf/CY0dEQRIkWqWezMZOFAXCSARyflQqlJDCCFpyqVL9ponhOixY4l/W5k0KfE7QnwAhSohJDxo2NA+cQLWuSoEgLnn9ev93jQSOVCoEkLCA9RGLVrU0dMX86e33OL+d5wCIj6EQpWQSGXvXpHnntOpDdu10zl8Q5mcOVHKSuSZZ3R+YgjL9u1FYmO1g5JzWA00WGRmq1s3UC0mYQgdlThKJZEITJ4wl16/rr1kIXBgCkVYzvPPS9iBlKf164vExWntFeeaObPIN9+I1KmTuB3SPM6di7JXusYpQpTsStCRsOC8HxyVKFQpVEmkgfnFihV1Uny7Um5HjqBChYRlKsfZs0V279YhNah6Yy0liYo4yGqE7FPoBww2kFADITn+qMJDAg6FahB3JCEhw8GDIsWKuf7+o498Xx81FIqGIy850qZanZqgpbZqJbJgQSBbR/wEQ2oIIanHLum+lUhMiID8uwjJcfYShra6cKFIfHygWkZCDDoqERJpFC/uug4qnHeaNpWIAzVoXdUxxSADc7GEeACFKiGRBoQpCpdj3tB0wjE9Y4cNE7n9dok47r3XtYaO+eVI7BOSIihUCYlEkGT+1191yAnCTeAZCxPo8OESkdSsqfvETlsdMYJVbojH0PuXjkqEEIA51SFDdAECeAAjkQQGGV27sn/ClPMMqQnejiSEhAmI3YWAzZHDft6ZhA3nWU+VEEL8TMaMOjsTISmAc6qEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4CApVQgghxEdQqBJCCCE+IiKr7xr/VaJA4C8hhJDI5Px/MsCUCb4gIoXqhQsX1P+FCxcOdFMIIYQEgUxAlj1fEJG5f2/evClHjx6V7NmzS1QapyHDyAjC/PDhw2GVIjEczysczylczysczylcz+t8EJ0TxB8EakxMjKRzVfrPSyJSU0Xn3R7gUk64mQJ9Q/mDcDyvcDyncD2vcDyncD2vW4PknHyloZrQUYkQQgjxERSqhBBCiI+gUE1joqOjZfjw4er/cCIczysczylczysczylczys6DM9JIt1RiRBCCPEH1FQJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQjUNeOONN6RWrVqSNWtWyZkzp0e/gf/YsGHDpGDBgpIlSxZp2LCh7NmzR4KFs2fPyhNPPKGCt3FO3bp1k4sXL7r9Td26dVUGK+vSs2dPCSRTp06VYsWKSebMmaVGjRryyy+/uN1+wYIFUqZMGbV9+fLl5ZtvvpFgxJvzmjlzZpLrgt8FE+vWrZPmzZurzDdo3+LFi5P9zdq1a+Wee+5RXqalSpVS5xnK54Tzcb5OWI4fPy7BwqhRo6RatWoqW12+fPmkVatW8ueffyb7u1B5rjyBQjUNuHbtmrRp00Z69erl8W/Gjh0rkydPlmnTpsmmTZskW7Zs0rhxY/nnn38kGIBA3blzp6xcuVKWLl2qXhA9evRI9nfdu3eXY8eOJSw4z0Axb948GTBggHLv37Jli1SsWFH18cmTJ223X79+vbRv314NILZu3apeGFh27NghwYS35wUwOLJel4MHD0owcenSJXUeGCx4wv79+6VZs2ZSr1492bZtm/Tv31+efvppWbFihYTqOZlASFmvFYRXsPDDDz9Inz59ZOPGjerdcP36dWnUqJE6V1eEynPlMQipIWnDjBkzjBw5ciS73c2bN40CBQoY48aNS1gXFxdnREdHG3PmzDECza5duxCGZWzevDlh3bfffmtERUUZf//9t8vf1alTx3juueeMYKF69epGnz59Ej7fuHHDiImJMUaNGmW7/eOPP240a9bMYV2NGjWMZ555xggmvD0vT+/LYAH33qJFi9xuM3jwYOPuu+92WNe2bVujcePGRqie05o1a9R2586dM0KFkydPqjb/8MMPLrcJlefKU6ipBiEYZcOkA5OvNT8lzHgbNmyQQIM2wORbtWrVhHVoK3IqQ6t2x2effSZ58uSRcuXKydChQ+Xy5csSKOtBbGysQx+j/fjsqo+x3ro9gAYYDNckNecFYLovWrSoSnTesmVLZYUIZULhWqWUSpUqqWmhBx98UH7++WcJZuLj49X/uXPnjphrFZEJ9YMdc44kf/78DuvxORjmT9AGZ5NThgwZ1IPjrn0dOnRQL27MIW3fvl3+97//KVPWwoULJa05ffq03Lhxw7aP//jjD9vf4NyC9Zqk5rxKly4t06dPlwoVKqiX4Pjx45UPAARroAtPpBRX1woVUq5cuaL8FEINCFJMB2Ewe/XqVfnoo4+UnwIGspg7DsZqYP3795f77rtPDaJdEQrPlTdQqKaQIUOGyJgxY9xus3v3bjX5Hm7nlFKsc65wRsBLokGDBrJv3z4pWbJkivdLUkfNmjXVYgKBetddd8n7778vI0eOZPcGCRj8YLFeJzw7EydOlE8++USCjT59+qh50Z9++kkiCQrVFDJw4EDp0qWL221KlCiRon0XKFBA/X/ixAkleEzwGaafQJ8T2ufs9PLvv/8qj2Cz7Z4AczbYu3dvmgtVmKDTp0+v+tQKPrs6B6z3ZvtAkJLzciZjxoxSuXJldV1CFVfXCg5ZoailuqJ69epBKbT69u2b4MCYnLUjFJ4rb+CcagrJmzev0kLdLZkyZUrRvosXL65uqFWrViWsg9kKZh6rRhGoc0Ib4uLi1NydyerVq5W5xxSUngCvTGAdOKQVOI8qVao49DHaj8+u+hjrrdsDeDj685qkxXk5A/Px77//HpDr4itC4Vr5AjxDwXSdDMNQAnXRokXqnYB3WcRdq0B7SkUCBw8eNLZu3WqMGDHCuOWWW9TfWC5cuJCwTenSpY2FCxcmfB49erSRM2dOY8mSJcb27duNli1bGsWLFzeuXLliBANNmjQxKleubGzatMn46aefjDvuuMNo3759wvdHjhxR54Tvwd69e43XXnvN+PXXX439+/er8ypRooTxwAMPBOwc5s6dqzyqZ86cqTyae/Toofr8+PHj6vuOHTsaQ4YMSdj+559/NjJkyGCMHz/e2L17tzF8+HAjY8aMxu+//24EE96eF+7LFStWGPv27TNiY2ONdu3aGZkzZzZ27txpBAt4VsznBq+tCRMmqL/xbAGcD87L5K+//jKyZs1qDBo0SF2rqVOnGunTpzeWL19uhOo5TZw40Vi8eLGxZ88edc/Bkz5dunTG999/bwQLvXr1Up7ka9euNY4dO5awXL58OWGbUH2uPIVCNQ3o3LmzemicF7jIJ1wIERXaYA2reeWVV4z8+fOrF2SDBg2MP//80wgWzpw5o4QoBgm33nqr0bVrV4dBAgSn9RwPHTqkBGju3LnV+ZQqVUq98OLj4wN4FoYxZcoUo0iRIkamTJlUKMrGjRsdQoBw7azMnz/fuPPOO9X2CNlYtmyZEYx4c179+/dP2Bb320MPPWRs2bLFCCbMcBLnxTwP/I/zcv5NpUqV1HlhAGd9vkLxnMaMGWOULFlSDXjwHNWtW9dYvXq1EUyIzfk4v9tC+bnyBJZ+I4QQQnwE51QJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUIIIcRHUKgSQggh4hv+H/SXy5gKmyalAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def make_moons_fallback(n_samples=100, noise=0.1):\n", " n = n_samples // 2 # half the points per moon\n", " t = np.linspace(0, np.pi, n) # angle along each crescent\n", " # outer moon — upper arc (cos, sin)\n", " x1 = np.stack([np.cos(t), np.sin(t)], axis=1)\n", " # inner moon — lower arc, shifted down so the two curves interleave\n", " x2 = np.stack([1 - np.cos(t), 1 - np.sin(t) - 0.5], axis=1)\n", " X = np.concatenate([x1, x2], axis=0) # stack into one (100, 2) array\n", " X += noise * np.random.randn(*X.shape) # jitter dots so it's not perfectly clean\n", " y = np.array([0] * n + [1] * n) # label 0 = outer moon, 1 = inner moon\n", " return X, y\n", "\n", "try:\n", " from sklearn.datasets import make_moons\n", " X, y = make_moons(n_samples=100, noise=0.1, random_state=1337)\n", " print(\"using sklearn make_moons\")\n", "except Exception as e:\n", " X, y = make_moons_fallback(100, noise=0.1) # same shape, no sklearn needed\n", " print(\"sklearn not available, using NumPy fallback:\", type(e).__name__)\n", "\n", "print(\"X shape:\", X.shape, \" y shape:\", y.shape) # expect (100, 2) and (100,)\n", "print(\"first 3 points:\", X[:3].tolist())\n", "print(\"first 3 labels:\", y[:3].tolist())\n", "\n", "if HAS_PLT:\n", " plt.figure(figsize=(5, 4))\n", " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # red vs blue by class\n", " plt.title(\"two moons — can a tiny net separate the colors?\")\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "0d2fe98b", "metadata": {}, "source": [ "### 1.1 The `Value` object — a number that remembers\n", "\n", "A normal Python float forgets where it came from: once you compute `3.0`, nobody knows it\n", "was `1.0 + 2.0`. Our `Value` keeps that memory: the numbers that made it (`_prev`) and the\n", "operation (`_op`).\n", "\n", "We start with a **minimal** version that can only do the **forward pass** (no gradients yet),\n", "just so you can see the \"remembering\" working.\n", "\n", "**-> Training:** `_prev` is the list of parent numbers; later, gradients flow backward along\n", "exactly these links." ] }, { "cell_type": "code", "execution_count": 3, "id": "eea9bc18", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d = Value(data=4.0)\n", "d was made by op: '+'\n", "d's parents: {Value(data=-6.0), Value(data=10.0)}\n" ] } ], "source": [ "class Value:\n", " def __init__(self, data, _children=(), _op=''):\n", " self.data = data # the number itself (forward pass result)\n", " # HISTORY: who made me? _prev = parent Value objects; _op = the operation (+, *, ...)\n", " # Forward pass builds a linked tree/DAG automatically — no separate \"tape\" step.\n", " self._prev = set(_children) # set of input Values that fed into this one\n", " self._op = _op # string tag: '+', '*', or '' for a leaf (input)\n", "\n", " def __repr__(self):\n", " return f\"Value(data={self.data})\" # pretty print when you print(a)\n", "\n", " def __add__(self, other):\n", " # New number = sum; we pass (self, other) so this node remembers both parents.\n", " return Value(self.data + other.data, (self, other), '+')\n", "\n", " def __mul__(self, other):\n", " # Same idea: result remembers it came from self * other.\n", " return Value(self.data * other.data, (self, other), '*')\n", "\n", "\n", "# --- tiny demo: build d = a*b + c and inspect the graph tags ---\n", "a = Value(2.0) # leaf — no parents, _op is ''\n", "b = Value(-3.0)\n", "c = Value(10.0)\n", "e = a * b # e.data = -6; e._prev = {a, b}; e._op = '*'\n", "d = e + c # d.data = 4; d._prev = {e, c}; d._op = '+'\n", "print(\"d =\", d)\n", "print(\"d was made by op:\", repr(d._op))\n", "print(\"d's parents:\", d._prev)" ] }, { "cell_type": "markdown", "id": "e65b3200", "metadata": {}, "source": [ "### 1.2 Reading the expression graph\n", "\n", "`d = a*b + c` is really a little tree: `d` points back to `e` and `c`; `e` points back to\n", "`a` and `b`. Let's print it as an indented tree so the structure is visible. (No graphviz\n", "dependency — just recursion.) Section **1.3** draws the same wiring as an interactive picture." ] }, { "cell_type": "code", "execution_count": 4, "id": "3a436597", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4.0000 (op=+)\n", " -6.0000 (op=*)\n", " 2.0000 (op=leaf)\n", " -3.0000 (op=leaf)\n", " 10.0000 (op=leaf)\n" ] } ], "source": [ "def show(v, indent=0):\n", " # print this node's value and which op created it ('leaf' = raw input)\n", " print(\" \" * indent + f\"{v.data:.4f} (op={v._op or 'leaf'})\")\n", " for child in v._prev: # follow parent links (deeper indent = further back in time)\n", " show(child, indent + 1) # recurse until we hit leaves with no _prev\n", "\n", "show(d) # prints the tree: d (+) -> e (*) -> a, b and c" ] }, { "cell_type": "markdown", "id": "f4e9da6b", "metadata": {}, "source": [ "### 1.3 Interactive graph — see the wiring\n", "\n", "The text tree above is correct but flat. The same expression as a **picture**: green boxes are\n", "raw inputs (leaves), blue boxes are operations. **Hover** a node to highlight its connections;\n", "**click** to pin the detail panel. Arrows run from ingredients (bottom) up to the final score (top)." ] }, { "cell_type": "code", "execution_count": null, "id": "f149d4da", "metadata": {}, "outputs": [], "source": [ "# Re-use d = a*b + c from section 1.1 (run that cell first if you restarted the kernel)\n", "show_graph(d, title=\"Forward pass: d = a × b + c\")" ] }, { "cell_type": "markdown", "id": "5ee07490", "metadata": {}, "source": [ "### 2.1 What a gradient means — nudge and measure\n", "\n", "Before any clever math, here is the *definition* of a gradient, done the dumb way: change an\n", "input by a tiny amount `h`, recompute the output, and see how much it moved.\n", "\n", "slope = (output after nudge - output before) / h\n", "\n", "Real-life picture: to feel how steep a hill is, take one small step and notice how much your\n", "height changed. That ratio is the slope." ] }, { "cell_type": "code", "execution_count": null, "id": "fbfbee18", "metadata": {}, "outputs": [], "source": [ "def f_out(a_, b_, c_):\n", " return (a_ * b_) + c_ # plain floats — no graph, just the math\n", "\n", "h = 1e-6 # tiny nudge size (like a small step on a hill)\n", "base = f_out(2.0, -3.0, 10.0) # output before any nudge = 4.0\n", "\n", "# how sensitive is the output to a? slope ≈ (f(a+h) - f(a)) / h\n", "da = (f_out(2.0 + h, -3.0, 10.0) - base) / h\n", "# to b?\n", "db = (f_out(2.0, -3.0 + h, 10.0) - base) / h\n", "# to c?\n", "dc = (f_out(2.0, -3.0, 10.0 + h) - base) / h\n", "\n", "print(f\"output = {base}\")\n", "print(f\"d(out)/da = {da:.4f} (equals b = -3)\") # derivative of a*b w.r.t. a is b\n", "print(f\"d(out)/db = {db:.4f} (equals a = 2)\") # derivative w.r.t. b is a\n", "print(f\"d(out)/dc = {dc:.4f} (equals 1)\") # derivative of +c w.r.t. c is 1" ] }, { "cell_type": "markdown", "id": "d3d3c6e2", "metadata": {}, "source": [ "### 3.1 The chain rule — multiply the little ratios\n", "\n", "The nudge-and-measure trick is correct but slow: one re-run per input. Real networks have\n", "millions of inputs. The **chain rule** gets every sensitivity in *one* backward pass.\n", "\n", "#### The one idea (relay race, not one long jump)\n", "\n", "When one thing changes another thing, which then changes a third, you **multiply the two\n", "\"sensitivity\" steps**.\n", "\n", "- **a** — starting point (e.g. how many projects you take)\n", "- **e** — middle step (hours you work)\n", "- **d** — final result (your pay)\n", "\n", "**d** does not feel **a** directly. It only feels **e**. So ask two smaller questions:\n", "\n", "1. If **a** moves a little, how much does **e** move? → `d(e)/d(a)`\n", "2. If **e** moves a little, how much does **d** move? → `d(d)/d(e)`\n", "\n", "The full effect is those two multiplied:\n", "\n", "`d(d)/d(a) = d(d)/d(e) × d(e)/d(a)`\n", "\n", "> **One line to keep:** If **d** depends on **e**, and **e** depends on **a**, then \"how fast\n", "> **d** responds to **a**\" = \"how fast **d** responds to **e**\" × \"how fast **e** responds to **a**.\"\n", "\n", "#### Units make it click (the middle cancels)\n", "\n", "Like converting units:\n", "\n", "- Pay per hour → dollars/hour\n", "- Hours per project → hours/project\n", "\n", "Multiply: `(dollars/hour) × (hours/project) = dollars/project`\n", "\n", "The **hour** in the middle cancels. You get \"how pay changes when projects change.\" The formula\n", "looks like fraction cancellation — that is the story, not magic algebra.\n", "\n", "#### Tiny wiggle picture\n", "\n", "Bump **a** by a tiny amount Δa.\n", "\n", "- **e** moves about `(d(e)/d(a)) × Δa`\n", "- **d** moves about `(d(d)/d(e)) ×` that amount\n", "\n", "So total change in **d** is `(d(d)/d(e)) × (d(e)/d(a)) × Δa`. The chain rule is just: the\n", "multiplier on Δa is the **product of the two hops**.\n", "\n", "#### Worked example\n", "\n", "Let `a = 2`, `e = a²`, `d = 3e`.\n", "\n", "- `d(e)/d(a) = 2a = 4` (e changes 4 units per unit of a)\n", "- `d(d)/d(e) = 3` (d changes 3 units per unit of e)\n", "\n", "So `d(d)/d(a) = 3 × 4 = 12` — when **a** goes up by 1, **d** goes up by about 12.\n", "\n", "Check directly: `d = 3a²`, so `d(d)/d(a) = 6a = 12` at `a = 2`. Same answer.\n", "\n", "#### Why college notation felt confusing\n", "\n", "Textbooks often write `f(g(x))` and say \"derivative of outside × derivative of inside.\" That\n", "is correct, but it hides the story: **indirect influence through a middle variable**. The relay\n", "race picture is the same thing with names you can follow.\n", "\n", "#### Local rules (what each station does)\n", "\n", "Each operation knows its own **local** derivative:\n", "\n", "- for `+`: output changes 1-for-1 with each input, so the gradient just **passes through**.\n", "- for `*`: each input's local derivative is **the other input**.\n", "\n", "**Backprop** = start with gradient 1.0 at the output, then walk backward multiplying by each\n", "local derivative." ] }, { "cell_type": "markdown", "id": "0da2822d", "metadata": {}, "source": [ "### 4.1 Adding gradients and local backward rules\n", "\n", "#### What is autograd?\n", "\n", "**Autograd** = **automatic differentiation** — the computer figures out gradients for you\n", "after you write the forward math.\n", "\n", "Real-life picture: like a GPS that logged every turn on your drive. When you ask \"how do I get\n", "back?\", it does not guess — it rewinds those turns and tells you how each step affected the\n", "final destination.\n", "\n", "**Is it a library?** Two meanings:\n", "\n", "| Meaning | What it is | In this notebook |\n", "|---|---|---|\n", "| **The idea** | Any system that records ops, then runs `backward()` to fill `.grad` | Our homemade **micrograd** `Value` class |\n", "| **A PyPI package** | A library literally named [`autograd`](https://github.com/HIPS/autograd) | We do **not** use it — we build from scratch |\n", "\n", "Later, **PyTorch** ships its own autograd (`loss.backward()` in notebooks `01` and `06`). Same\n", "idea, industrial scale.\n", "\n", "When code says \"analytic gradients via autograd,\" it means: run forward through `Value` objects,\n", "call `.backward()` once, read `.grad` — instead of nudging each input by hand with\n", "`numeric_grad`.\n", "\n", "---\n", "\n", "Now the full `Value`. Each operation also defines a `_backward()` that says how to push the\n", "output's gradient onto its inputs. Note we **accumulate** (`+=`) gradients, because one value\n", "can feed into several places.\n", "\n", "We add `+`, `*`, `**` (power), `tanh`, and `exp` — enough to build a neural net. We also add\n", "convenience operators (`-`, `/`, right-hand versions) so expressions read naturally.\n", "\n", "#### `_op` and `_prev` — the tags on every box\n", "\n", "Every `Value` stores two pieces of history in `__init__`:\n", "\n", "- **`_op`** — *what happened at this station?* A stamp on the box: `'+'` (mixed), `'*'` (scaled),\n", " `'tanh'` (squashed), or `''` for a raw leaf input (flour, sugar — nothing done yet).\n", "- **`_prev`** — *who gave me ingredients?* The set of parent `Value`s that fed into this one.\n", " Like \"received from\" arrows on the tag.\n", "\n", "**Cookie-factory picture for `d = a * b + c`:**\n", "\n", "1. `e = a * b` → `e._op = '*'`, `e._prev = {a, b}` — \"made by multiplying; parents are a and b.\"\n", "2. `d = e + c` → `d._op = '+'`, `d._prev = {e, c}` — \"made by adding; parents are e and c.\"\n", "\n", "A normal float is a score on a napkin: **4**. You cannot tell how you got it. A `Value` is the\n", "same score **on a tagged box** — you can always walk back.\n", "\n", "**Why this matters:** forward pass moves ingredients down the line. Backprop starts at the final\n", "score and walks **backward** along `_prev`. At each stop, `_op` tells you which local rule to\n", "apply (add passes blame through; multiply splits blame using the other input). Without these\n", "tags there is no map from the final number back to the knobs you need to turn.\n", "\n", "> **One line:** `_op` = what was done here; `_prev` = who fed into it. Together they are the\n", "> breadcrumb trail from output back to inputs.\n", "\n", "**-> Training:** `tanh` is the neuron's \"squashing\" activation; its local derivative is\n", "`1 - tanh(x)^2`." ] }, { "cell_type": "code", "execution_count": null, "id": "9433f6b8", "metadata": {}, "outputs": [], "source": [ "class Value:\n", " def __init__(self, data, _children=(), _op='', label=''):\n", " self.data = data # the actual number (forward value)\n", " self.grad = 0.0 # how much the final loss changes if THIS number nudges +1\n", " self._backward = lambda: None # filled in by each op: \"push grad to my parents\"\n", " self._prev = set(_children) # graph edges pointing backward to parents\n", " self._op = _op # which op created this node ('+', '*', 'tanh', ...)\n", " self.label = label # optional name for debugging / graph drawing\n", "\n", " def __repr__(self):\n", " return f\"Value(data={self.data:.4f}, grad={self.grad:.4f})\"\n", "\n", " def __add__(self, other):\n", " other = other if isinstance(other, Value) else Value(other) # allow Value + 3\n", " out = Value(self.data + other.data, (self, other), '+') # forward: add numbers\n", " def _backward():\n", " # local rule for +: d(a+b)/da = 1, d(a+b)/db = 1 → grad flows 1:1 to both\n", " self.grad += 1.0 * out.grad\n", " other.grad += 1.0 * out.grad\n", " out._backward = _backward # closure captures self/other/out — runs later in backward()\n", " return out\n", "\n", " def __mul__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", " out = Value(self.data * other.data, (self, other), '*') # forward: multiply\n", " def _backward():\n", " # local rule for *: d(a*b)/da = b, d(a*b)/db = a (chain rule: × out.grad)\n", " self.grad += other.data * out.grad\n", " other.grad += self.data * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def __pow__(self, other):\n", " assert isinstance(other, (int, float)), \"only int/float powers\" # exponent is constant\n", " out = Value(self.data ** other, (self,), f'**{other}') # forward: x^n\n", " def _backward():\n", " # d(x^n)/dx = n * x^(n-1), then chain rule × upstream grad\n", " self.grad += other * (self.data ** (other - 1)) * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def tanh(self):\n", " x = self.data\n", " t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1) # forward tanh (squash to -1..1)\n", " out = Value(t, (self,), 'tanh')\n", " def _backward():\n", " # d tanh(x)/dx = 1 - tanh(x)^2 (use saved t, not recompute from self.data)\n", " self.grad += (1 - t ** 2) * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def exp(self):\n", " out = Value(math.exp(self.data), (self,), 'exp') # forward: e^x\n", " def _backward():\n", " # d e^x / dx = e^x — out.data IS e^x at forward time\n", " self.grad += out.data * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " # --- convenience ops so expressions like 3 + a or a / b work naturally ---\n", " def __neg__(self):\n", " return self * -1 # unary minus via multiply by -1\n", "\n", " def __sub__(self, other):\n", " return self + (-other if isinstance(other, Value) else Value(-other)) # a - b = a + (-b)\n", "\n", " def __radd__(self, other):\n", " return self + other # handles 3 + a (Python calls __radd__ on a)\n", "\n", " def __rmul__(self, other):\n", " return self * other # handles 3 * a\n", "\n", " def __truediv__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", " return self * other ** -1 # a / b = a * b^(-1)\n", "\n", " def backward(self):\n", " # Step 1: list every node in the graph, parents before children (topological sort).\n", " # We walk _prev links; reversed(topo) then visits output → inputs.\n", " topo = []\n", " visited = set()\n", " def build_topo(v):\n", " if v not in visited:\n", " visited.add(v)\n", " for child in v._prev: # visit all ancestors first\n", " build_topo(child)\n", " topo.append(v) # post-order: children before parent\n", " build_topo(self) # start from the output node (e.g. loss)\n", "\n", " # Step 2: seed — \"if output goes up 1, output goes up 1\" (dL/dL = 1).\n", " self.grad = 1.0\n", "\n", " # Step 3: reverse sweep; each node runs its local _backward (chain rule × parents).\n", " for node in reversed(topo): # output first, leaves last\n", " node._backward() # each op pushes blame to its _prev parents\n", "\n", "\n", "print(\"full Value class ready\")" ] }, { "cell_type": "markdown", "id": "7f14c265", "metadata": {}, "source": [ "### 4.2 One backward pass on the tiny expression\n", "\n", "Rebuild `d = a*b + c` with the full class, call `d.backward()` once, and read the gradients.\n", "They should match the nudge-and-measure numbers from section 2.1 (`a.grad = b = -3`,\n", "`b.grad = a = 2`, `c.grad = 1`)." ] }, { "cell_type": "code", "execution_count": null, "id": "9d1e739c", "metadata": {}, "outputs": [], "source": [ "a = Value(2.0, label='a') # leaf inputs — knobs with no parents yet\n", "b = Value(-3.0, label='b')\n", "c = Value(10.0, label='c')\n", "e = a * b # builds multiply node; graph remembers a, b\n", "d = e + c # builds add node; graph remembers e, c\n", "\n", "d.backward() # one backward pass fills .grad on every node in the graph\n", "\n", "print(\"d =\", d.data) # forward answer still 4.0\n", "print(\"a.grad =\", a.grad, \" (expect -3)\") # ∂d/∂a = b\n", "print(\"b.grad =\", b.grad, \" (expect 2)\") # ∂d/∂b = a\n", "print(\"c.grad =\", c.grad, \" (expect 1)\") # ∂d/∂c = 1" ] }, { "cell_type": "markdown", "id": "0083fec0", "metadata": {}, "source": [ "Same graph **after backprop** — orange gradient numbers show how much each node would nudge the\n", "final output. Hover `a` and notice its grad equals `b` (−3): the multiply rule in action." ] }, { "cell_type": "code", "execution_count": null, "id": "c14c82ca", "metadata": {}, "outputs": [], "source": [ "show_graph(d, title=\"After backward() — gradients filled in\")" ] }, { "cell_type": "markdown", "id": "1ac38e12", "metadata": {}, "source": [ "### 4.3 Trust but verify — numeric gradient check\n", "\n", "A good habit: confirm the analytic gradient (from `backward()`) matches the slow\n", "nudge-and-measure gradient. If they agree, the engine is correct.\n", "\n", "**Real-life picture:** change one recipe ingredient by a tiny pinch, cook again, and see\n", "how much the final score moves. That ratio — *change in score ÷ size of pinch* — is the\n", "gradient for that ingredient. Repeat for each ingredient, one at a time. Slow, but it is\n", "the definition of \"sensitivity.\"\n", "\n", "**Two ways to measure the same hill**\n", "\n", "| | Nudge-and-measure (numeric) | `backward()` (analytic) |\n", "|---|---|---|\n", "| Idea | Tiny change in one input → re-run forward | Chain rule walks the graph once |\n", "| Cost | One forward pass **per input** | One backward pass for **all** inputs |\n", "| Result | slope ≈ `(f(x+h) − f(x)) / h` | `.grad` on each leaf `Value` |\n", "\n", "**Outline — nudge-and-measure (what `numeric_grad` does)**\n", "\n", "1. Run the same math as the graph, but on plain floats (no `Value` objects).\n", "2. Pick a tiny step `h` (e.g. `1e-6`).\n", "3. Save the baseline output at the current inputs.\n", "4. For **each** input `i`, one at a time:\n", " - copy the inputs;\n", " - add `h` only to input `i`;\n", " - re-run forward;\n", " - slope = (new output − baseline) / `h`.\n", "5. That slope is the numeric gradient for input `i`.\n", "\n", "**Outline — analytic check (what we trust after verifying)**\n", "\n", "1. Build the expression with `Value` objects (the graph remembers parents via `_prev`).\n", "2. Call `.backward()` once on the final output.\n", "3. Read `.grad` on each leaf — chain rule applied in one reverse sweep.\n", "\n", "**Compare:** print analytic vs numeric side by side; they **match** when\n", "`|analytic − numeric| < 1e-4`. Same hill, two rulers — if they agree, the engine's local\n", "`_backward()` rules are wired correctly." ] }, { "cell_type": "code", "execution_count": null, "id": "fe4b711e", "metadata": {}, "outputs": [], "source": [ "def numeric_grad(f, inputs, i, h=1e-6):\n", " base = f(inputs) # output at current inputs\n", " bumped = list(inputs) # copy so we don't mutate the original list\n", " bumped[i] += h # nudge only input i by a tiny amount\n", " return (f(bumped) - base) / h # slope = rise / run\n", "\n", "f = lambda v: (v[0] * v[1]) + v[2] # same math as a*b+c but on plain floats\n", "inputs = [2.0, -3.0, 10.0]\n", "\n", "a = Value(2.0); b = Value(-3.0); c = Value(10.0)\n", "(a * b + c).backward() # analytic gradients via autograd\n", "\n", "for i, (name, val) in enumerate(zip(\"abc\", (a, b, c))):\n", " ng = numeric_grad(f, inputs, i) # slow finite-difference gradient for input i\n", " print(f\"{name}: analytic={val.grad:+.4f} numeric={ng:+.4f} match={abs(val.grad-ng)<1e-4}\")" ] }, { "cell_type": "markdown", "id": "6518159c", "metadata": {}, "source": [ "**Your turn 4** — Build `g = (a*b + c).tanh()` with `a=0.5, b=2.0, c=-1.0`, call\n", "`g.backward()`, and check `a.grad` against a numeric nudge. (Hint: reuse `numeric_grad`\n", "with `f = lambda v: math.tanh(v[0]*v[1] + v[2])`.)" ] }, { "cell_type": "code", "execution_count": null, "id": "d6bea75d", "metadata": {}, "outputs": [], "source": [ "# Your turn 4 — fill in and run\n", "a = Value(0.5); b = Value(2.0); c = Value(-1.0) # inputs\n", "g = (a * b + c).tanh() # forward: multiply, add, squash\n", "g.backward() # backprop through tanh, +, *\n", "\n", "f = lambda v: math.tanh(v[0]*v[1] + v[2]) # same formula on plain floats\n", "ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0) # numeric ∂g/∂a (nudge index 0)\n", "print(f\"a.grad analytic = {a.grad:+.4f} numeric = {ng_a:+.4f}\")" ] }, { "cell_type": "markdown", "id": "fef7ef8b", "metadata": {}, "source": [ "### 5.1 Neuron -> Layer -> MLP, all from `Value`\n", "\n", "Everything below is still built from plain `Value` objects — the same tagged boxes and\n", "`backward()` trail from section 4. Here we stack them into a small **neural network**.\n", "\n", "#### One story to hold in your head: renting an apartment\n", "\n", "You apply with two numbers on the form: **monthly income** and **credit score**.\n", "The landlord wants one answer: *likely good tenant* vs *likely risky*.\n", "\n", "**One neuron = one reviewer on the panel**\n", "\n", "A single neuron looks at your whole application and gives **one opinion**, squashed to a\n", "number between -1 and +1:\n", "\n", "`tanh(w · x + b)` means:\n", "\n", "- **`w`** — how much this reviewer cares about each field (income vs credit). Big weight =\n", " \"this matters a lot to me.\"\n", "- **`b`** — this reviewer's baseline strictness before seeing you (always picky, or always\n", " lenient).\n", "- **`w · x + b`** — weighted score: multiply each input by its importance, add bias.\n", "- **`tanh(...)`** — squash the score so it cannot run away to infinity; strong yes or strong no,\n", " but bounded (like capping enthusiasm at \"very pro\" or \"very against\").\n", "\n", "So one neuron is not magic — it is **one weighted vote**, softened at the edges.\n", "\n", "**One layer = a whole panel, all reading the same application**\n", "\n", "A **layer** is several neurons in **parallel**. Each sees the **same** inputs `x`, but each\n", "has its **own** weights and bias — different reviewers, different priorities:\n", "\n", "- Reviewer A might care mostly about income.\n", "- Reviewer B might care mostly about credit.\n", "- Reviewer C might look for a balance of both.\n", "\n", "The layer's output is a **list of opinions** (one per neuron). Eight neurons → eight numbers\n", "describing the application from eight angles.\n", "\n", "**An MLP = panels stacked in stages**\n", "\n", "An **MLP** (multi-layer perceptron) feeds one layer's opinions into the next:\n", "\n", "1. **First hidden layer** — junior reviewers turn raw form fields into rough themes (\"cash-flow\n", " looks fine\", \"credit is shaky\", …).\n", "2. **Second hidden layer** — senior reviewers debate those themes, not the raw form.\n", "3. **Output layer** — one final judge reads the debate and emits a single score (here: one\n", " neuron → one number for the two-moons task).\n", "\n", "That is why the code uses shapes like `MLP(2, [8, 8, 1)`:\n", "\n", "| Piece | In the story | In code |\n", "|---|---|---|\n", "| 2 inputs | income + credit (here: x, y on the plot) | `nin = 2` |\n", "| 8 + 8 hidden | two panels of 8 reviewers | two `Layer`s with `nout=8` |\n", "| 1 output | final hire / no-hire score | last layer with `nout=1` |\n", "\n", "Data flows **forward** only: form → junior panel → senior panel → final judge.\n", "Every multiply, add, and `tanh` is still a `Value` node, so `loss.backward()` can later\n", "blame the right weight when the answer was wrong.\n", "\n", "#### The three building blocks (code ↔ picture)\n", "\n", "| Class | What it is | Real-life |\n", "|---|---|---|\n", "| **`Neuron`** | one weighted vote + squash | one reviewer |\n", "| **`Layer`** | list of neurons, same inputs | one panel sitting together |\n", "| **`MLP`** | layers wired in sequence | whole hiring process, stage by stage |\n", "\n", "**-> Training:** `parameters()` collects every weight and bias in the network — all the knobs\n", "on all reviewers' score sheets. Section 6 nudges those knobs downhill until the final judge\n", "gets the moons right." ] }, { "cell_type": "code", "execution_count": 6, "id": "93e51095", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Interactive companion to section 5.1 — drag the inputs, hover lines/dots, press play.\n", "# Shows the same 2 -> 8 -> 8 -> 1 reviewer-panel network as the code below.\n", "import base64\n", "from pathlib import Path\n", "from IPython.display import IFrame, display\n", "\n", "html_path = Path(\"mlp_playground.html\")\n", "if not html_path.exists():\n", " html_path = Path.cwd() / \"mlp_playground.html\"\n", "\n", "if not html_path.exists():\n", " raise FileNotFoundError(\n", " \"mlp_playground.html not found. Keep it next to this notebook, then re-run.\"\n", " )\n", "\n", "# Notebook UIs often cannot load local files via IFrame src=\"file.html\".\n", "# Pack the page into a data URI so the browser runs it inline with scripts intact.\n", "b64 = base64.b64encode(html_path.read_bytes()).decode(\"ascii\")\n", "display(IFrame(src=f\"data:text/html;base64,{b64}\", width=\"100%\", height=640))" ] }, { "cell_type": "code", "execution_count": null, "id": "1d260369", "metadata": {}, "outputs": [], "source": [ "class Neuron:\n", " def __init__(self, nin):\n", " # one weight per input + one bias — all are Value objects (trainable knobs)\n", " self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]\n", " self.b = Value(random.uniform(-1, 1))\n", "\n", " def __call__(self, x):\n", " # weighted sum: w1*x1 + w2*x2 + ... + b (each * and + builds the graph)\n", " act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)\n", " return act.tanh() # squash activation to (-1, 1)\n", "\n", " def parameters(self):\n", " return self.w + [self.b] # every knob this neuron owns\n", "\n", "\n", "class Layer:\n", " def __init__(self, nin, nout):\n", " self.neurons = [Neuron(nin) for _ in range(nout)] # nout parallel neurons\n", "\n", " def __call__(self, x):\n", " outs = [n(x) for n in self.neurons] # each neuron sees the same input vector x\n", " return outs[0] if len(outs) == 1 else outs # scalar if one neuron, else list\n", "\n", " def parameters(self):\n", " return [p for n in self.neurons for p in n.parameters()] # flatten all weights\n", "\n", "\n", "class MLP:\n", " def __init__(self, nin, nouts):\n", " sizes = [nin] + nouts # e.g. [2, 8, 8, 1] for 2→8→8→1\n", " # layer i connects sizes[i] inputs to sizes[i+1] outputs\n", " self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(nouts))]\n", "\n", " def __call__(self, x):\n", " for layer in self.layers: # data flows forward through each layer\n", " x = layer(x)\n", " return x\n", "\n", " def parameters(self):\n", " return [p for layer in self.layers for p in layer.parameters()] # all trainable Values\n", "\n", "\n", "net = MLP(2, [8, 8, 1]) # 2 inputs -> 8 hidden -> 8 hidden -> 1 output\n", "print(\"parameter count:\", len(net.parameters()))\n", "print(\"one prediction:\", net([Value(0.5), Value(-0.2)]))" ] }, { "cell_type": "markdown", "id": "fc0b5b7b", "metadata": {}, "source": [ "### 6.1 Train the MLP on the two moons\n", "\n", "Loop the four familiar steps:\n", "\n", "1. **forward** — predict on every point\n", "2. **loss** — how wrong (here: mean-squared error against labels mapped to -1 / +1)\n", "3. **backward** — `loss.backward()` fills every parameter's `.grad`\n", "4. **update** — nudge each parameter a little **against** its gradient (downhill)\n", "\n", "Watch the loss fall. (Pure-Python micrograd is slow, so we keep the net and step count small.)" ] }, { "cell_type": "code", "execution_count": null, "id": "de95ef11", "metadata": {}, "outputs": [], "source": [ "net = MLP(2, [8, 8, 1]) # fresh network for training\n", "\n", "# map labels {0,1} -> {-1,+1} to match tanh output range (-1 to +1)\n", "ys = [1.0 if yi == 1 else -1.0 for yi in y]\n", "# wrap each (x, y) coordinate pair as Value objects so forward builds a graph\n", "Xv = [[Value(float(xi[0])), Value(float(xi[1]))] for xi in X]\n", "\n", "lr = 0.1 # learning rate — how big each knob turn per step\n", "for step in range(60): # 60 gradient-descent steps (small count: micrograd is slow)\n", "\n", " # --- 1. FORWARD: predict every point, build one big graph ending in loss ---\n", " preds = [net(xrow) for xrow in Xv]\n", " loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0)) # sum of squared errors\n", " loss = loss * (1.0 / len(ys)) # average MSE so scale doesn't grow with dataset size\n", "\n", " # --- 2. BACKWARD: zero old grads, then propagate blame from loss to every weight ---\n", " for p in net.parameters():\n", " p.grad = 0.0 # must reset — backward() accumulates with +=\n", " loss.backward() # fills p.grad for every parameter\n", "\n", " # --- 3. UPDATE: walk downhill — turn each knob opposite its gradient ---\n", " for p in net.parameters():\n", " p.data -= lr * p.grad # new value = old value - lr * (how loss rises if knob rises)\n", "\n", " if step % 10 == 0 or step == 59:\n", " # sign match: prediction > 0 same as target > 0 counts as correct\n", " acc = sum((1 if (p.data > 0) == (yt > 0) else 0) for p, yt in zip(preds, ys)) / len(ys)\n", " print(f\"step {step:3d} loss {loss.data:.4f} acc {acc:.2%}\")" ] }, { "cell_type": "markdown", "id": "9cb30391", "metadata": {}, "source": [ "### 6.2 See the boundary it learned\n", "\n", "Evaluate the trained net across a grid of points and color the regions. The swirl between the\n", "two moons should now be split by a curved boundary — the network \"drew the line.\"\n", "\n", "(Coarse grid for speed; each grid point is a full forward pass through micrograd.)" ] }, { "cell_type": "code", "execution_count": null, "id": "ee39626a", "metadata": {}, "outputs": [], "source": [ "if HAS_PLT:\n", " # pad plot area a little beyond the data points\n", " xmin, xmax = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " ymin, ymax = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xs = np.linspace(xmin, xmax, 30) # 30 grid columns\n", " ys_grid = np.linspace(ymin, ymax, 30) # 30 grid rows\n", "\n", " Z = np.zeros((len(ys_grid), len(xs))) # store net output at each grid cell\n", " for i, gy in enumerate(ys_grid):\n", " for j, gx in enumerate(xs):\n", " out = net([Value(float(gx)), Value(float(gy))]) # forward only (no training)\n", " Z[i, j] = out.data # >0 one side of boundary, <0 other side\n", "\n", " plt.figure(figsize=(5, 4))\n", " plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3) # colored regions\n", " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # training dots on top\n", " plt.title(\"decision boundary learned by the micrograd MLP\")\n", " plt.show()\n", "else:\n", " print(\"matplotlib missing - skipping the boundary plot (training above still ran)\")" ] }, { "cell_type": "markdown", "id": "b00ac0fd", "metadata": {}, "source": [ "**Your turn 6** — Try `MLP(2, [16, 16, 1])` or a different learning rate / step count and\n", "re-run 6.1 and 6.2. Does a bigger net reach higher accuracy? Does too-large a learning rate\n", "make the loss bounce instead of fall?" ] }, { "cell_type": "markdown", "id": "48623e88", "metadata": {}, "source": [ "### 7.1 Bridge to PyTorch — same gradients, industrial engine\n", "\n", "PyTorch is micrograd scaled up to fast tensors. The *idea* is identical: build an expression,\n", "call `.backward()`, read `.grad`. Here we redo the tiny `a*b + c` (then a tanh) in PyTorch and\n", "confirm the gradients match what our engine produced.\n", "\n", "**-> Training:** in notebooks `01` and `06` we let PyTorch's autograd do exactly this — now you\n", "know what it is doing under the hood." ] }, { "cell_type": "code", "execution_count": null, "id": "e6c3140e", "metadata": {}, "outputs": [], "source": [ "# On some Windows setups torch and numpy ship duplicate OpenMP DLLs; this avoids a clash.\n", "import os\n", "os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n", "\n", "try:\n", " import torch\n", "\n", " # requires_grad=True tells PyTorch to track history (like our Value graph)\n", " at = torch.tensor([0.5], requires_grad=True)\n", " bt = torch.tensor([2.0], requires_grad=True)\n", " ct = torch.tensor([-1.0], requires_grad=True)\n", "\n", " gt = torch.tanh(at * bt + ct) # same expression as micrograd Your turn 4\n", " gt.backward() # PyTorch autograd — same idea as Value.backward()\n", " print(\"PyTorch: a.grad =\", at.grad.item())\n", "except OSError as e:\n", " print(\"PyTorch could not load in this environment:\", e)\n", " print(\"(That is fine here - the micrograd result below is the point.)\")\n", "\n", "# our engine, same expression — should match PyTorch's a.grad\n", "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n", "(a * b + c).tanh().backward()\n", "print(\"micrograd: a.grad =\", a.grad)" ] }, { "cell_type": "markdown", "id": "49c1ebf8", "metadata": {}, "source": [ "## What's next\n", "\n", "You built the engine that powers all of deep learning:\n", "\n", "- a **number that remembers** its history (`Value`)\n", "- **backpropagation** via the chain rule (`backward()`)\n", "- a small **MLP** trained by nudging weights downhill\n", "\n", "Where this leads in the series:\n", "\n", "- `01_build_gpt.ipynb` — language modeling basics (the bigram), now that backprop is no longer\n", " mysterious.\n", "- `02`-`05` — scale these same ideas up with PyTorch on real character data (`names.txt`).\n", "- `06_build_gpt_attention.ipynb` — the transformer, where `loss.backward()` is doing exactly\n", " what you built here, just across millions of `Value`-like nodes.\n", "\n", "**Checklist**\n", "- [ ] A `Value` records data, parents, and operation\n", "- [ ] Each op defines a local `_backward`\n", "- [ ] `backward()` = topological order + chain rule\n", "- [ ] Neuron / Layer / MLP built from `Value`\n", "- [ ] Trained on two moons; boundary learned\n", "- [ ] Confirmed gradients match PyTorch" ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.12.10)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 }