diff --git a/00_micrograd.ipynb b/00_micrograd.ipynb index 5db4d43..4684e7c 100644 --- a/00_micrograd.ipynb +++ b/00_micrograd.ipynb @@ -58,6 +58,65 @@ "direction that makes the output less wrong." ] }, + { + "cell_type": "markdown", + "id": "334f0f03", + "metadata": {}, + "source": [ + "### Mental picture — a factory line with tags\n", + "\n", + "Imagine a **cookie factory**. Each station does one small job and puts a **tag** on the box\n", + "saying *who gave me ingredients* and *what I did* (mixed, baked, frosted).\n", + "\n", + "| Step | Real life | In code |\n", + "|------|-----------|---------|\n", + "| Forward pass | Ingredients move down the line; you get a final cookie score | `d = a * b + c` computes numbers left to right |\n", + "| History | Every box keeps tags — no separate diary | `_prev` (parents) and `_op` (operation) on each `Value` |\n", + "| Gradient | *If I change this one ingredient a tiny bit, how much does the final score change?* | `.grad` on each `Value` |\n", + "| Backprop | Walk backward along the tags from the final score to every ingredient | `loss.backward()` |\n", + "\n", + "A normal float is like a score on a napkin: **4**. You cannot tell where it came from.\n", + "A **`Value`** is the same score **on a tagged box** — you can always walk back.\n", + "\n", + "**Blame traveling backward (the chain rule)**\n", + "\n", + "The final taste score is off. At each station you ask: *if your output had been 1 point\n", + "sweeter, how much would the final score have moved?*\n", + "\n", + "- **`+` (mix two bowls)** — nudge either bowl; the mix moves the same amount. Blame passes\n", + " through to both inputs equally.\n", + "- **`*` (recipe ratio, e.g. 2 cups × 3 batches)** — change flour a little → effect depends\n", + " on batch count; change batches → effect depends on flour. Each side's blame depends on\n", + " the **other** side's amount.\n", + "\n", + "**Backprop** starts at the output (\"downstream moved by 1\"), walks **backward** along the\n", + "tags, and at each station multiplies by that station's local rule. Small effect here ×\n", + "small effect there = total effect on the end.\n", + "\n", + "**Why gradients add (`+=`)**\n", + "\n", + "One bag of sugar might go into **both** dough and icing. Both paths affect the final taste,\n", + "so sugar's total blame is the **sum** of both paths — same when one `Value` is used twice\n", + "in an expression.\n", + "\n", + "**Training the MLP (section 6)**\n", + "\n", + "Same factory, but the knobs are **weights and biases** (how much each input counts):\n", + "\n", + "1. **Forward** — run points through the line; get predictions.\n", + "2. **Loss** — how wrong were we? (distance from the target).\n", + "3. **Backward** — `loss.backward()` sends blame to every knob.\n", + "4. **Update** — turn each knob a tiny step **against** its gradient (downhill = less wrong).\n", + "\n", + "The **two moons** dataset is two swirls of red and blue beads. The network learns a curved\n", + "boundary — not by memorizing dots, but by turning knobs until wrong guesses hurt (high loss)\n", + "and right ones don't.\n", + "\n", + "> **One line to keep:** Forward = tagged boxes down the line. Backward = follow tags from\n", + "> the final score back to each ingredient and assign fair blame. PyTorch does the same thing\n", + "> at scale — millions of tagged boxes, fast hardware." + ] + }, { "cell_type": "markdown", "id": "84448625", @@ -74,22 +133,30 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "e31a52ef", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ready\n" + ] + } + ], "source": [ - "import math\n", - "import random\n", - "import numpy as np\n", + "import math # exp, tanh used inside Value activations\n", + "import random # random weight init for Neuron\n", + "import numpy as np # dataset + optional plots only (not the autograd engine)\n", "\n", - "random.seed(1337)\n", - "np.random.seed(1337)\n", + "random.seed(1337) # same random weights every run\n", + "np.random.seed(1337) # same noisy moon dots every run\n", "\n", "# Plots are optional: the notebook runs fine without matplotlib (it just skips pictures).\n", "try:\n", " import matplotlib.pyplot as plt\n", - " HAS_PLT = True\n", + " HAS_PLT = True # flag so later cells can skip plotting gracefully\n", "except Exception:\n", " HAS_PLT = False\n", " print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n", @@ -97,6 +164,205 @@ "print(\"ready\")" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "ace0626a", + "metadata": {}, + "outputs": [], + "source": [ + "from IPython.display import HTML, display\n", + "import json\n", + "\n", + "_GRAPH_UID = 0 # unique id per render so multiple cells don't clash\n", + "\n", + "\n", + "def show_graph(root, title=\"Computation graph — hover a node\"):\n", + " \"\"\"Draw an interactive SVG graph for a Value expression.\n", + "\n", + " Hover (or click) any node to see its value, operation, and parents.\n", + " After `.backward()`, gradients appear on each node automatically.\n", + " \"\"\"\n", + " global _GRAPH_UID\n", + " _GRAPH_UID += 1\n", + " gid = f\"mg{_GRAPH_UID}\"\n", + "\n", + " nodes, seen = [], set()\n", + "\n", + " def build_topo(v):\n", + " if v not in seen:\n", + " seen.add(v)\n", + " for ch in v._prev:\n", + " build_topo(ch)\n", + " nodes.append(v)\n", + "\n", + " build_topo(root)\n", + " id_map = {v: i for i, v in enumerate(nodes)}\n", + "\n", + " depth = {}\n", + "\n", + " def mark_depth(v, d=0):\n", + " depth[v] = max(depth.get(v, 0), d)\n", + " for ch in v._prev:\n", + " mark_depth(ch, d + 1)\n", + "\n", + " mark_depth(root, 0)\n", + " max_depth = max(depth.values()) if depth else 0\n", + " layer_of = {v: max_depth - depth[v] for v in nodes}\n", + "\n", + " from collections import defaultdict\n", + "\n", + " layers = defaultdict(list)\n", + " for v in nodes:\n", + " layers[layer_of[v]].append(v)\n", + "\n", + " NODE_W, NODE_H, GAP_X, GAP_Y = 108, 58, 36, 72\n", + " positions = {}\n", + " max_w = 0\n", + " for ly in sorted(layers.keys()):\n", + " row = layers[ly]\n", + " row_w = len(row) * NODE_W + max(0, len(row) - 1) * GAP_X\n", + " max_w = max(max_w, row_w)\n", + " x0 = (max_w - row_w) / 2\n", + " for i, v in enumerate(row):\n", + " positions[v] = (x0 + i * (NODE_W + GAP_X), ly * (NODE_H + GAP_Y))\n", + "\n", + " svg_w = int(max_w + 48)\n", + " svg_h = int((max(layers.keys()) + 1) * (NODE_H + GAP_Y) + 36)\n", + " show_grad = any(getattr(v, \"grad\", 0.0) for v in nodes)\n", + "\n", + " def node_name(v):\n", + " lbl = getattr(v, \"label\", \"\") or \"\"\n", + " return lbl if lbl else f\"n{id_map[v]}\"\n", + "\n", + " def esc(s):\n", + " return (\n", + " str(s)\n", + " .replace(\"&\", \"&\")\n", + " .replace(\"<\", \"<\")\n", + " .replace(\">\", \">\")\n", + " .replace('\"', \""\")\n", + " )\n", + "\n", + " edge_lines = []\n", + " for v in nodes:\n", + " for ch in v._prev:\n", + " x1, y1 = positions[ch]\n", + " x2, y2 = positions[v]\n", + " cx1, cy1 = x1 + NODE_W / 2, y1 + NODE_H\n", + " cx2, cy2 = x2 + NODE_W / 2, y2\n", + " eid = f\"{gid}-e{id_map[ch]}-{id_map[v]}\"\n", + " edge_lines.append(\n", + " f''\n", + " )\n", + "\n", + " node_groups = []\n", + " meta_nodes = {}\n", + " for v in nodes:\n", + " x, y = positions[v]\n", + " nid = id_map[v]\n", + " node_id = f\"{gid}-n{nid}\"\n", + " nm = esc(node_name(v))\n", + " op = esc(v._op or \"leaf\")\n", + " val = f\"{v.data:.4f}\"\n", + " grad = f\"{getattr(v, 'grad', 0.0):.4f}\"\n", + " fill = \"#dcfce7\" if not v._prev else \"#dbeafe\"\n", + " grad_svg = (\n", + " f''\n", + " f\"grad = {grad}\"\n", + " if show_grad\n", + " else \"\"\n", + " )\n", + " node_groups.append(\n", + " f''\n", + " f''\n", + " f'{nm}'\n", + " f''\n", + " f\"op: {op} val: {val}\"\n", + " f\"{grad_svg}\"\n", + " f'{nm}: {val} ({op})'\n", + " )\n", + " meta_nodes[node_id] = {\n", + " \"name\": node_name(v),\n", + " \"op\": v._op or \"leaf\",\n", + " \"val\": val,\n", + " \"grad\": grad,\n", + " \"parents\": \", \".join(node_name(p) for p in v._prev) or \"—\",\n", + " }\n", + "\n", + " meta_json = json.dumps({\"gid\": gid, \"show_grad\": show_grad, \"nodes\": meta_nodes})\n", + "\n", + " html = f\"\"\"\n", + "
\n", + "
{esc(title)}
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " {''.join(edge_lines)}\n", + " {''.join(node_groups)}\n", + " \n", + "
\n", + " Hover a node to inspect it. Arrows flow from inputs (bottom) to output (top).\n", + "
\n", + "
\n", + "\n", + "\n", + "\"\"\"\n", + " display(HTML(html))\n", + "\n", + "\n", + "print(\"interactive graph viewer ready\")" + ] + }, { "cell_type": "markdown", "id": "3315510f", @@ -114,21 +380,42 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "id": "507b4df3", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sklearn not available, using NumPy fallback: ModuleNotFoundError\n", + "X shape: (100, 2) y shape: (100,)\n", + "first 3 points: [[0.9296812690137589, -0.04902823627877147], [0.965763959785321, -0.11143765173168577], [1.0124564608446291, -0.07324929559695742]]\n", + "first 3 labels: [0, 0, 0]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAF2CAYAAAA4MQK3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAaAZJREFUeJztnQd4FNXXxk9ooYgUqUE6CiJVmiBKlSJSFJGiNBGkKoLwBwuIqFQBQRQbzUJTioKCSBGVIgYQKfoBUqWXhCogzPe89zrJ7GZ2s5vsZtv7e56B7OzszJ075dxz7ilRhmEYQgghhJBUky71uyCEEEIIhSohhBDiQ6ipEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUKCiAMHDkhUVJTMnDkz0E0hYQDuI9xPv/76q4QSUVFR8uqrr0ooEpZCdf369eqCxMXFBbophNjy+eefy6RJk8K6d44ePaqew23btgW6KUHDu+++65cBk7/2G2l888038tBDD0nBggUlZ86c8uCDD8pvv/3m1T7CVqiOGDGCQpWEnFAtWrSoXLlyRTp27CjhIFTxHFKoJkKhGtw8/PDDkjFjRhk2bJgMHTpUduzYoQTrqVOnPN5HBr+2kBDitdkrc+bM7LUQALVI/vnnH8mSJUugm0LccOnSJcmWLZt4woYNG6RGjRoJnytXriyNGzeW7777Tp544onI1FRhbho0aJD6u3jx4uolhQVzVY8++qjcc889Dts3b95cff/VV18lrNu0aZNa9+233yas++uvv6RNmzaSO3duyZo1q9x7772ybNkyj9qEffXt21cWLFggZcuWVQ9hzZo15ffff1ffv//++1KqVCn1Mq1bt65qqzP4bZUqVdRv8+TJI08++aT8/fffSbZbvXq13H///eomgvmiZcuWsnv37iR9hDbt3btXunTporbLkSOHdO3aVS5fvuyw7cqVK6V27dpqm1tuuUVKly4tL774ogSKmzdvyttvvy3ly5dX/ZU3b15p0qSJw5zRjBkzpH79+pIvXz6Jjo5Wff7ee+8l2VexYsXUyPSnn36S6tWrq/2VKFFCZs+e7VFbxo8fL7Vq1ZLbbrtNXRdcny+++CLZ3+Ea4945ePBgwv2JtriaU8U1Qt/jerdq1Ur9jfN+4YUX5MaNGwkveOwD19sZvPhxfZ955hmP7tPFixdLuXLlVN/dfffdsnz58iTboi1PPfWU5M+fP2G76dOnJ3y/du1aqVatmvob95V5nu5MlBcuXJD+/fur88A+cf2gJWzZssVhOzyfuOY4JzyLderUkZ9//tn2Hv/jjz/k8ccfl1tvvVVdp+eee071hxVv75cVK1ZI1apV1TXHs+vpPvD7nTt3yg8//JDQH7gXTDBdhfMvXLiw2gfeCWPGjFH3vDuS2y+4evWqDBgwQN03eDc88sgjttoX3nnm+yN79uzSrFkztW9PQPuff/75hOt3++23S6dOneT06dMJ25w8eVK6deum7hs8bxUrVpRZs2Z5tP+tW7dK06ZN1bXEM9CgQQPZuHGj7Rwy+qJ3797qeqAdnt5fVoEKzAHutWvXxGOMMOO3334z2rdvj3J2xsSJE41PPvlELRcvXjQmTJhgpEuXzoiPj1fb3rx508iVK5da98ILLyTsY9y4cQ7bHT9+3MifP7+RPXt246WXXlL7qVixotpm4cKFybYJbalQoYJRuHBhY/To0WrJkSOHUaRIEeOdd94xypYta7z11lvGyy+/bGTKlMmoV6+ew+9nzJih9lGtWjV1TkOGDDGyZMliFCtWzDh37lzCditXrjQyZMhg3HnnncbYsWONESNGGHny5FHnuH///oTthg8frvZXuXJl49FHHzXeffdd4+mnn1brBg8enLDdjh07VHuqVq1qvP3228a0adNUPz3wwANGoOjSpYtqZ9OmTY1JkyYZ48ePN1q2bGlMmTIlYRv0E7ZDX2F9o0aN1G/Q11aKFi1qlC5dWl3bF198UX1/zz33GFFRUerck+P22283evfurX6He6J69erqOEuXLnX7u++++86oVKmSujbm/blo0SL1Ha4T9oFrbtK5c2cjc+bMxt1332089dRTxnvvvWe0bt1abYdrZ4J7M2PGjMaZM2ccjjd//ny17bp169y2C9vgvi5YsKAxcuRI1b8lSpQwsmbNapw+fTphOzwPOHfcz6+99ppqT4sWLRKeOXMbfId1PXr0SDjPffv2uTx+hw4d1P02YMAA46OPPjLGjBljNG/e3Pj0008Ttlm1apXapmbNmuqZwfHwbGHdpk2bktzj5cuXV/vANXryySfVuo4dOzoc15v7pVSpUup5wjOI52HNmjUe7wPXGP1WpkyZhP7AvQAuXbqkzuO2225T9yL23alTJ3UvPvfcc26vm7v9mu8OPOv169dXbRs4cKCRPn164/HHH3fYz+zZs9XxmjRporZD/+MdkzNnTof3hx0XLlwwypUrp/bbvXt3dU/gHkK/bN26VW1z+fJl46677lL36PPPP29MnjzZuP/++1X7cK9ZwTpcQxM8j9myZUu4N/EOLV68uBEdHW1s3LgxYTvzfPFOrVOnjjoPbOvp/WXlypUrxn333aeuifMz5Y6wE6qmUETHOt8ImzdvVuu/+eYb9Xn79u3qc5s2bYwaNWokbIcXBG5Ck/79+6vtfvzxR4ebCBcVN92NGzfctge/xcW3tuf9999X6wsUKGCcP38+Yf3QoUMd2n7t2jUjX7586obFRTbBixvbDRs2LGEdXtTY1noDYJAB4Y8H1PmFgxe0lUceeUTdQCZ4QWC7U6dOGcHA6tWrVXueffbZJN9hgGSCh9eZxo0bKwHh/JJ0FjYnT55U1wovnuRwPg6uFa4TXl7J0axZM3V8Z1wJVayDkLKCe7RKlSoJn//880+1HV5oVnA/4z619pEd+C1eOnv37nW4f7DeOmjp1q2berlZBS1o166dGiya/WI+b9ZzcQd+26dPH5ffo/133HGHupbO1xvP4oMPPpjkHse5W8EgCOtxXtbfe3O/LF++PMn2nu4DAyO87J2BoIDQ+L//+z+H9RDeEFSHDh1K8htP9msKmYYNGzr0GYQa9hsXF5fwPoPwhEC0gsERrovzemfwHsJxFtooGeZxITixjVWI4ZnBAOmWW25xeA86C9VWrVqpe9M6KDt69KhSdKyDfPN8a9eubfz7779e3V9Wrl+/bjz00EPqXbB27VrDG8LO/OsO2MdhNli3bp36/OOPPyaYKGACgOkT1xPmQJhArB5hMA/CDGqC/fTo0UOZ63bt2pXssWGqME18VjND69atlZnFeT3MzQBmTZhMYMqwzrXBLFOmTJkEE/SxY8eUQwhMhTBRm1SoUEGZOHAOzvTs2dPhM875zJkzcv78efUZJl+wZMmSZE1QacGXX36pTDvDhw9P8h3Wm1jnuOLj45X5CSZC9Ck+W4GZznqtYR6Didvsf3dYj3Pu3Dm1b+zL2VzpK+yul7Wdd955p7p/Pvvss4R1Z8+eVSY9zAdZ+8gVDRs2lJIlSzrcPzC3mcfB84HrgGkT/I2+NRfMPaEPUnr+uN9g2oWDkx24v/fs2SMdOnRQ96l5XMyZ4fnCc+18n/bp08fhc79+/dT/1ufBm/sFU0o4T2e82YcdmN7B9cyVK5dDn+J6wMRvvrNSCt5V1uuPY2G/mIIwp3lgvm3fvr3D8dOnT6/uqTVr1rjdP+4JmHIfeeSRJN+Zx0WfFyhQQB3DBE5Bzz77rFy8eFGZbO1AOzGniakPTM+YwEMX9wLe1+Y7y6R79+6q7d7cX1bgYIfnBs8SrqM3RJSjEjoZc5kQpgD/4+aCsMSFg30etn68iKwvWtx4zrZ2cNdddyV8jzkodxQpUsThM+aDAOZP7NbjJW3uG+BF7wyEKm6o5LZDOzEP5Dxh79wmPNDmsfEibdu2rXz00Ufy9NNPy5AhQ9SLC/PSjz32mKRL53o8hv7zag7CAgYEmTJlsv1u3759EhMT4zBosAPzaxC8cDpwniPGC87sY7s+MPvB7H93LF26VF5//XX1sseclYknwstbzPnj5NqJASLmRXE/wJMYL+vr16977E2cXH9gHg4v3w8++EAtdmAQmBLGjh0rnTt3Vs8E5qcR2oDzMV+kEKgA27gC19e8j8Edd9zh8D0GDLh3rX4L3twvEKp2eLMPO3Bu27dvT3KNU9unnjzr5vEB5oXtwPvAHXg2oSC4A/ckrofzu8P6HrUD9xz61NW7DQOpw4cPq3l9d9cpufvLyieffKKUkeTOSSJdqAII0DfeeEM5K0CovvTSS2oEA6GIzxCqwCpUfYHzqCm59doC4l+SOzZG3xghY5QKjRgOK/PmzVMPHkaOrn4Pwetq1JkcOJazk4U34OGG8MeAY8KECeoBgpDGKHnixIlJNJmU9j/ulRYtWsgDDzygwiQwasaoGw4rCJfxNa7a6Uy7du2UswhG2HAo+/TTT5VTjd0LyZvjmP1h9h8c5VwJN2i3KQEORXjuFi1apO6vcePGKUedhQsXKgcV89hYX6lSJdt9wILkDucBj7f3i52nr7f7sAPb4CU+ePBg2+9hhUgNnl5XCBNok85kyBBaoiKLzXVK7v6yAksInumUEFo95SHuNAV0KrSoOXPmKA9GU3ji5WgKVdzApnAFGPH/+eefSfYFz0Lze39h7hvHdx5FYp35vXU7u3bCY9hTt3IrGFXihYEFL4w333xTDUQg/GCasuOtt97ySNOzAyYkV0DLgMYNTdiVtvr1118rrRHe3NbReXLmK2+BuQvaI9oDT0ITCFVP8Ic2C9AvmBqAUIXJFxqUL5NMQJPCdAUsO66uf2rOES8yTHVggXYGb30MgvHSM83S0JqSO7YJNDCr1gKPdwgQcyrGF/eLN/tw1Sc4N5hAPT0vT/frKWbfwiM2JW3A7xHT6Q68o6CNo/+t2mpy71Hcc/DydvVuw76cLX4pub+czeWYLkwJYTmnagoPu4xKMONCo8AIBS8g02QA4QrzLzQsZy0VZoJffvlFmXZMYEqF+QsPJ+bl/AW0DNzo06ZNczAxwt6PUBm8QM2bBaN3uKdbzxs3OkZlOAdvgfByxtQQrG1xBqYVPJgpWaymO2dgisHIGvMdrkbc5ojcqmnC/OapsPMUHAcvMjOkBcCkiHAUT+9RT+baUgJMvZjnR2gZ2gnt1Vdgf7gOGFTYvUStYRrunkNn0I/O/YH7HuZ+817DfYWXN0KZIIDcHdtk6tSpDp+nTJmi/jdfor64X7zZB/rErj+gReH9gkGaM9j+33//ddsGV/v1FMwTY7CCQTOmC5xJLvkB7glkHlq0aFGS78x+wTvo+PHjytplgvPCNYGFwdXcJfq3UaNGyrfDarY/ceKEsgrB+picedqT+8t5ThaKREoIS00VDx+ARoUXCoQoHCtw42HEg+8hQM0YVVNThaDE4ixUMZ8IzRYPIibVIYwhvPbv369eLu7mF1OLOQBArB9uOkzy42ZCrCYEOkx9JjBnoI2YN0YsGDLz4IbFfE5K8mi+9tpryvwLwY1RJEZ2MHXCucvqtJVW1KtXTwmMyZMnKw0EsYoY9cLCgO8wl4iHD6Y3XFvEZeLl++GHH6oHCM5cvgJ9As0dbYCzBPoGL3DEFmI0nhy4B/FyQewg4jnxUkGbfdU2xGRiPhX3A87dl4wePVppYRig4uWDQSUGYHBQ+v777xMGYxCAmFrBgBDaLZ4//MZuvgsxhLivMF8PawX6A/vavHmzsnwAPGeY48c5YTCMZ6JQoULK4oT24MUKrdEKnlGY6XGdILRgDsf1Mi0ivrhfvNkHrjviVzEXj3sF28AChQEQNF3EwcLZENvhXYRYdsQ+Q5jA2uQKV/v1FPQdfo/nC9ob3pvQEA8dOqSmfu677z555513XP4e7Uc727Rpo+KX0R7cBzgnXH/0N7Q/xPXi/GJjY9X7C78xrSlWh01ncF5mzDy0TJijsS8IRMyVJocn95fzXC2mN1KU+tEIU+CiXqhQIRVO4hxeM2jQILUOcUpWEIOG9XaxdFj32GOPKbdzxAwiJjG5eEQT7NPZldsMnUD4jxXEvWH9ggULHNbPmzdPhVDAxTt37tzGE088YRw5ciTJsb7//nsVW4U41ltvvVXFYe3atcthGzPcwDlUxnRHN/sKMYGIAY2JiVHu7PgfMcDObv9pCdzk0WeIyUOb8ubNq2JWY2NjE7b56quvVMwfrhNCSXCdp0+fnuQ+QIgEQlucQWiCXXiCMx9//LEK8cA1QXvQf2bfJgfiphE3h/sJ25vhNa5CahBu4Yy7Y5mhI59//rnhKXb3KUDb0AYrJ06cUNsiVhVxhwgNa9CggfHBBx84bLdkyRIVM4j4aXfhNVevXlXPJeJkESaB88Xf1jhcE8Q9Ir4a4V/oe7QPMZe4X537Bvc+nlvsE/Glffv2dQhN88X94s0+EKKCfaA9+M56nyGsBSF1eA/h3kYcc61atVQsNkJP3OFqv+YzjfAmu/eMGWdrXY9QIISf4FxKliyp4m9//fVXIzkQyte3b1/13kX7ETuL+8YaeoX7pmvXrurcsA3iiO3uCeeQGrBlyxbVNoTfIHYa8fzr16932MbV+Xpzf5nHd77nPSXqvx0QQsIIWDA+/vhjZW6DdSbSgGUG0wQwW7rT8AjxNWE5p0pIJAPPdpg5Mc8ViQKVkEASlnOqhEQimNfFPBHmqRASgDy3hJC0hUKVkDABHr8Io4GTCpy5XMVyEkL8B+dUCSGEEB/BOVVCCCHER1CoEkIIIT4iIudUkTAAlQoQbOyvdHGEEEKCG0SUIjEEMiv5KolPRApVCFRPc0USQggJbw4fPqwyLvmCiBSqZjosdGRyOSMJIYSEJ+fPn1cKlrsUid4SkULVNPlCoFKoEkJIZBPlw2lAOioRQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4iIiMUyXhy7//iixbJhIbK1KggEi7diK5cwe6VYSQSIFClYQNJ0+K1K8vsnOnSIYMIjduiAwaJLJokUijRoFuHSEkEqD5l4QNvXqJ/PFHosZqGCJXroi0bi1y4UKgW0cIiQQoVElYEBcnsnix1k6tQLBevKi/I4QQf0OhSsKC+HiU9LP/DhWdTp1K6xYRQiIRClUSFhQqJJI/v/13ELY1a6Z1iwghkQiFKgkL4Jg0YkTS9enTizRoIHLvvYFoFSEk0qBQJWHDM8+IfPyxSNGi+nOWLNp5ackSlHYKdOsIIZFAlGHAlSPyCtPmyJFD4uPjWU81DMEdjTnWbNlEMmYMdGsIIZEkCxinSsIOaKU5cwa6FYSQSITmX0IIIcRHUKgSQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSoJW/btE+nfXyd+QFL9FSsC3SJCSLjDkBoSlmzeLFK3rsi1a7piDTIrLVwoMnKkyMsv++eYSIe4f7+OjS1SxD/HIIQEN9RUSVjSp4/IP/9ogQrM6jXDhokcPOj74y1dKlKqlF6Q0alKFV0oPVDs2SOyZo3I8eOBawMhkQiFKgk7TpzQmqqrqjVff+3b423YINKypciBA4nrfvtNa8qHD0uacuyYSL16InfeqQu23367yNNPi1y9mrbtICRSoVAlYUdyiTddCduUMmaMzuJkPS40YxRInzbN9e8gcP/6K/n2egr289BDIj/+6NiOGTNEBgzwzTEIIe6hUCVhR4ECIpUq6Tqqdjz8sG+Pt2VL0uLoAOu2bk26HmZhmIcx71qypMgdd4h8803q2/HTTyLbtiVtCwYRH32k8yETQkJYqK5bt06aN28uMTExEhUVJYsXL072N2vXrpV77rlHoqOjpVSpUjJz5swk20ydOlWKFSsmmTNnlho1asgvv/zipzMgoco772iHIZSEA3BUAkOGiJQo4dtjwcRqJ8BxbHznrJ3CLAzzsAm01RYtRDZtSl07/vjD9Xdw2PLHXDIhJA2F6qVLl6RixYpKCHrC/v37pVmzZlKvXj3Ztm2b9O/fX55++mlZYYmFmDdvngwYMECGDx8uW7ZsUftv3LixnDx50o9nQkKN++7TGmTnziLlyumaql9+KfLGG74/Vu/e9iZlOEl17+647r33tFnYqk3CbAvzMczIqcHdYMFOwBNC/ICRRuBQixYtcrvN4MGDjbvvvtthXdu2bY3GjRsnfK5evbrRp0+fhM83btwwYmJijFGjRnnclvj4eNUe/E9Iarl5E/euYURF4T7XS8aMhjFtWtJtmzRJ3MZ5KVw4de24ccMwSpc2jAwZHPebPr1hdOmSun0TEo7E+0EWBNWc6oYNG6Rhw4YO66CFYj24du2axMbGOmyTLl069dnchpC0xtQyYcZ9/33tGPT337poujOFCiWapK3AfOxOk4Rmi9mTp54S6dpVZNGipHOn2Mfy5SJlyzquh2kZ5nBCSIQlfzh+/Ljkz5/fYR0+o5DslStX5Ny5c3Ljxg3bbf5wM6F09epVtZhgf4T4mmLFRHr0cL8NzMEff5x0PczHiK21A2ZkZIT66qtEgQxXg2bNtHC1FmJHG+Cs9Ouvev62fHntCEUISRuCSlP1F6NGjVLV3c2lcOHCgW4S8ZDr1zE3Hz7dVaOGyLvvOgpCaLqDBol06GD/m9mztUA1BayZ0GLZMq0VO4P9Vasm8uijFKiERLRQLVCggJxA5L4FfL711lslS5YskidPHkmfPr3tNvitK4YOHSrx8fEJy+G0jsiPYC5fFoEPmbexoadOaSejbNlEbrlFpGJFkW+/lbCgVy+RI0e0xgrHJeQoHjtWC0M7PvvM/jusw3eEkOAhqIRqzZo1ZdWqVQ7rVq5cqdaDTJkySZUqVRy2uXnzpvpsbmMHwnMgmK0L8S9nzoh07CiSIwfM89oz9ZNPPPstLPV16miBAU0V7NihzZ0rV0pYkC+fnh/t2VOkeHH320JTt0sQgXUXL/qtiYSQYBOqFy9eVKExWMyQGfx96NChBA2yU6dOCdv37NlT/vrrLxk8eLCaI3333Xdl/vz58vzzzydsg3CaDz/8UGbNmiW7d++WXr16qdCdrvDeIEEBHGjgSzZnTqKpEjGSuNSff57877/4QmT3bkdHHGi60MyGD5eIo3HjxDhbK1jXtGkgWkQIcYnhR9asWaPclZ2Xzp07q+/xf506dZL8plKlSkamTJmMEiVKGDNmzEiy3ylTphhFihRR2yDEZuPGjV61iyE1/uWrr1yHjZQsqUNQ3IGIKYSk2P0+Xbrkfx9unDplGLffrkNjrGEyMTGGceJEoFtHSOgS74eQmij8IxEGvH/hsIT5VZqCfQ8qwYwalailOhMXp83C7n7/5pv2qf9y5RI5e1YijqNHRV5/XWvxAN7AKGGHEB1CSPDIgqCaUyXhQd68rh2TMmcWyZrV/e8xF2s31EMcpjVDEZygliwRmT9fO0OFMzEx2msY54kFDk4UqIQEHxSqxOe0a6dDRpw9VjEH2KWLYziJHYirRBIFCFEsZmzm/fdrLRYg5SAcvlu1EmnbVguYESN8V/GFEEJSAs2/9AT2C6hZ+vjj2pMXwhSmYOTjRTUWT7sc/mzz5unqKvAGRv5eCFk4MSGpgZ15GI5Q7dv7/HQIIWHIeT+YfylUKVT9GlazYIH+/957ddFsV7GY3gBncKTdc56zhcCtWjX11V4IIZHBeT8I1aBKU0jCi9tu03GYvgbhOXZaKuZx9+/3/fEIIcRTOKdKQg4kjLerXwozM8zChBASKChUSciBpPXR0UkFK7TXwYMD1SpCCKFQJUEI4li3bnUdJlOkiE5XWLKko6l51iydfSjYgQPW+PEir74q8sMP9FgmJJygoxIdlZIFYSrr14v8/LNI7tw68QCSMPiaa9dEBg4U+eAD/Tc00cce05/tkkWgXcgJ/M8/OuF+pkwS9EyblljiDecHZyukGly4UMfw+hvUokC40i+/JOYfrl3b/8clJBih928Qd2S4ggQLLVuKfP+9nrOEMxBe/nPn6uLXvgTCBkLHmjgCx4TX8HffSciza5dIuXJJNVMI11de0ZqrP0HJYYQ1IUQJpnLE/0Koo8A6zeYkEjnPjErEF6Cw9T336CQMKC07erTrlIJIhbd6tf4bL2IIBGiGbdporcdXIOwGGqlzJiYcE6be7dsl5IF52i4xPs75o4/8f/x+/RIFKjCv+ZAhIgcO+P/4hEQCdFSKMFDwGsWrf/tNv1RR1/PFF0W6dbN/2X/4YVJBB8GK33pSccZT9u51LdjB77+LX4CZGXmGixXTSSkwJwsztz/AwCEl3yUHciEjdhcl9mAmR0YraKVWzp/X1ga7UCTEDmOgRQhJPRSqEcSff2ptBVgFJYQkhC0yFVlBNiRX9TqhcR0/7ru2JZfHFhq1r8F5Y84WplfEvl64IIJSvcje5FTW1yfUqGE/cEBfIjlGSs3zmBOdMkU7dkF4IoVj9eoi//d/idu5G7BAqOJaE0JSD4VqBAAB2revSJky+qXrCniiWsHc6Z132mdBQvFwZC/yFbffLvLww0nNo/hcurR/nGmgkSKdonWAYZq4Bw3y/fE6dNDF2q3niL7F8VJaJxaF36GVWjVQCNArV7QGbgIHM5j87eJ78duHHkrZ8QkhjlCoRgDw9pw6Nfntsmd3/IwXPpxnnB1rIBQgoJHM3pfMmKE1LCsQQkuX2guD1AJt1EzWbwVCFiE97gYgKSFbNpEff9TasSlYkawC+ZDr1k35OdgNeiBYly93XDdxoj6uec7m72D6r1AhZccnhDjCNIURAASqqRHZge+yZBFp3jzpd0hOD630pZf0/CuE2yOP6Ny7yVWb8ZY8ebT2iHCPnTv1PCeEjT8EKkAJOlcl6iB8/BGigxJu8JyGuRVLap3PIajRP3bnge+sPPCAzosMb1/0M+Zgn3nGfj6dEJIyGKcaASE1OXNqr09XAhWaC6rBQFi6Ai/tY8e0NhsuXQaPVySQcBZIEKjoCxQDCHYQamSX8AKCFmXyUmpWJiQSOM+QGpISKlWyD+UAKKe2Z497gWq+pOFMFC4CFUATnjRJ/42BBfoIgwycJ0ylzkDTx/wltGhXGm5a8+CDIr17J56DadqFU9QLLwS0aYREJNRUw0lKuNFmmjRxNP9CgODU4RGcN69ENAgvQgzpqVMiNWuKdOyYdH4ZTlzdu+sBiJkqEWZ1OFcFGlxXtG/+fB1DDM0VYVO+Ns8TEm6cZz3V4O3IYAfmXaQA/PvvRE0GCQeQ4Ye4B6EpcOTB3LKpoUKjhfa+YYNItWrsQUJCkfM0/5KU0ratjsWE+RL/b9xIgeopcMpC2IlzbC8E64QJvCcJIYnQ+zeCMGM+iXds22afPAHrtmxhbxJCEmGcKiEeODTZxbNikILvCCHEhEKVkGTo2dNeU4VJGJmqCCHEhEKVkGSoVUuXpLMmg4DmOmqUfcKMUAfzxZh3R5EDVwlDCCH2UKgS4gHIPITkF599posPILsUSqaFG8hmhbhmmLXvuEOkVCmdRpEQ4hmMU42QkBpCPMkwdffdOtbVOXRo/fqkeZkJCXXOM6SGEOIv3n1X5yO2Cx0aN479Togn0PxLCFGgMo9dEXM4af36KzuJkKAQqlOnTpVixYpJ5syZpUaNGvILJm1cULduXYmKikqyNGvWLGGbLl26JPm+CXLwEUJSXdPWLnQI5l+kZURMLuquwmELsya9eomcPs1OJyTNhOq8efNkwIABMnz4cNmyZYtUrFhRGjduLCdPnrTdfuHChXLs2LGEZceOHZI+fXpp06aNw3YQotbt5syZ48/TICQi6NHDPnQI5mCMa++7T+eRRrrGCxdEPvxQ5P77RS5fDkRrCYlAoTphwgTp3r27dO3aVcqWLSvTpk2TrFmzyvTp0223z507txQoUCBhWblypdreWahGR0c7bJcrVy5/ngYhEQGKCaBIgDURP+ZTX3xR12GFMLWah/E30l7CI5oQ4meheu3aNYmNjZWGDRsmrEuXLp36vAFZyD3g448/lnbt2kk2p2rLa9eulXz58knp0qWlV69ecubMGZ+3n5BIBGXkUHRhxgyRDz7QHsFvvCGyerX9fCuySqFCDiHEz7l/T58+LTdu3JD8+fM7rMfnPzC8TQbMvcL8C8HqbPp99NFHpXjx4rJv3z558cUXpWnTpkpQw1Rsx9WrV9VidaMmhNiDUoBdujiuQym8uLik20KTZVQaISHg/QthWr58eanuFBwHzbVFixbqu1atWsnSpUtl8+bNSnt1xahRo1SpN3MpXLhwGpwBIeFD167aYckZzME+8YTWYjHHinnXsmVF+vUT2b8/EC0lJEyFap48eZTmeOLECYf1+Ix5UHdcunRJ5s6dK926dUv2OCVKlFDH2oucai4YOnSoqp1qLocPH/biTAgh//ufTtcI4CFsegm/9JJe36GDdnTCzM7u3TqtY+XK+m9CIgm/CdVMmTJJlSpVZNWqVQnrbt68qT7XhEeEGxYsWKDMtU8++WSyxzly5IiaUy1YsKDLbeDYhMxJ1oUQ4jlZs4qsWSOyaJFO2fj88zp29fXX9Zzq/Pl6OzNXMDTYixe1k1NywHsYheDtzMuEhBp+Nf8inObDDz+UWbNmye7du5VTEbRQeAODTp06KS3SzvQL0+5tt93msP7ixYsyaNAg2bhxoxw4cEAJ6JYtW0qpUqVUqA4hxH9AO23VShdtHztWpEoVvX7pUvv4VpiE8Z2rpPz4/pVXRPLl03V+MZfbuTN8HngVSeji1yLlbdu2lVOnTsmwYcPk+PHjUqlSJVm+fHmC89KhQ4eUR7CVP//8U3766Sf5DgFxTsCcvH37diWk4+LiJCYmRho1aiQjR45U2mikgMTu778vsnmzCCzpTz2l57IICQR2c61WRyZXvPyyyJgxjtotwnNwf9s8/oSEBEyoH2Km4J07dcA9RvMY6UNDwMtowgRtkiMkrcE8qjnfagXO+Agxt8vNguQR0FCRvN8OZG/CnCwh/oQJ9YmKIzQFKjAz4Lzwgi5HRkhac++9ep4VmFFt0F4xe4Oas3bAr9CVQAXbtvmhoYREckgNSQpyXKxbZx+EDxPa4sXsNZL2wMT73nsiX34p8vDDIg88oOdKt2/XdVntwLSFO9NwTIzfmktI6M6pEt+CNHGuwAvq2jX2OAkMuP8efVQvngBnfeQT/vZbx0EiNN1ChUQsidgICSmoqYYQ8O8qV87eMQRJz5s2TXvNedAgXcEEL8I+fXSKO0I8ASnAK1XSf5taK+5xeAy7SI5GSNBDR6UQc1RCDlYzegjzqRCwEKgwuaEkF7TVFi10ID5Sy/kLzOtWqyayb1+ipgGnKYRFxMZqTYSQ5MC0BeJcd+wQQaIzlJazJvQnJNQclShUQ0yoAggthCLA6xIj+0uXEIqU+JLCqB+p4n7+WSRHDv+04a23RAYP1gLdCjSMAQN0HCMhhAQz9P4lCgTdI4MNsi3CGxj1CSBMzXg//I/0cBMn+q/DMBfmLFCtAf+EEBKJcE41xEHaOFdzrGbqOH+QJYvroH98RwghkQiFaogD4ekqDZydJukr2ra13z9Mz0iuTgghkQiFaojTvLn9esxttm7tv+O2a6fzwJrHMrVWpEuESZoQQiIROiqFoKOSFWSlqVsXRd0TNVYIOATdY51TTQKfgvnTJUtEFi7UnsgI/H/8ce2FTAghwQ69f4O4IwMJSme9+67IvHlauLVsKfLssyK5cwe6ZYQQErxQqAZxR/qT334T+fRTkfh4HY+KJOURVJSHBJirV3UKTMSSFi2qrREh8NgQkiwUqhEoVMeP11mLzHqV0ESRVQkB89REib85cECkXj39P5Iy4P7DI7N8uU6kT0goc94PsoCOSkHMrl1aoAK8zMyKNIhBfemlgDaNRAgdO+p4aDP3NObtUbYNKTFxb2LQh/qnhBANhWoQ8/nniRqqs4PQJ58EokUkkoB2+tNPSasiIZQqLk4nF/nf/7RJGBVqCCEUqkENNAJX5bGuXHEdn0qILzh92v33ELYQsLCgIDb51Cn2OyHUVIMYzGXZlXtDyEzt2u7rURKSWu66SyRbtuS3w+AO9+mCBexzQihUgxjEfdas6VgGy0y08OabgWwZiQQgUIcO9Wxb3Jdnz/q7RYQEPxSqQQzmU7/7TmTgQF1SLXNmkfr1Rdat05mLCPE3L74oMmmSSIEC+rMr6whMwLwnCWFGpaAPqfE1qLcKp5JVq7Qm0r49QyNI8mDuFDV0ly0TefJJLVzNOX1oqRCoa9dySoI4geD6kyd1sVxoBUEGQ2pIqh2fMBcLp5JZs3QWJpiXPTXxkcgFUw45c4o88YSeOy1TRq/HwKxPHy1sOcdPEsAIrHNnkTx5RO68UyRfPpFXX/VvlY8ggbl/I0hTHTJExxU6h0gAFDSvVSsQrSKhCjzQkdnLVQlAu8xMc+aIfPONTiSBgg8oyuDp70kI0bChNl1YXzYYdSHAfuRICRaYUSmIOzIUiImxD9TH3O0zz4i8804gWkUiAeSnbtBAZONGLUTxfsX7FoIVOautzngkxImNFala1f67rFlFTpwQueUWCQZo/iWp4tIl+/WYG7t4kZ1L/Mfbb+uqSQAWQFOBwfw+Q3HCjK1b3Y+u/vpLwhkaXiLMImOnEeAFB69iQvyZHcxuOg1aKzRVEmYmMVfARJE/v4QzFKoRxPDhutapc9xrhQq68ggh/qz768pKgrlWEkY0aiRSqFDSETw+t2hBoUpCF2gGqCby3HM61hUm3g0bdFIJeG3CMQ91V1HxJgi93UkYgXvO1bwpkvOTMCJDBu0OjuB6YLqFV64s8tFHEu7Q+zdMHZWQNu7RR0WWLnUsG9e7t3ZIYvgDSUuOHhW55x6dT9icT4WQRbQF5lqDxG+F+DooftkykSNHtDkMxaCD7MVD798g7shgY/Jkkf797ZPuf/211hwISUvwbh01SmTJEj3QQ+KRwYNFcuXidSCBISS9f6dOnSrFihWTzJkzS40aNeQX0wXQhpkzZ0pUVJTDgt9ZMQxDhg0bJgULFpQsWbJIw4YNZc+ePf4+jZBj5kx7gQrt4NNPA9GiMAVq17ffamkxe7ZrF2sit9+O94EWrigrhy6jQCXhhl+F6rx582TAgAEyfPhw2bJli1SsWFEaN24sJ5G2ygUYLRw7dixhOXjwoMP3Y8eOlcmTJ8u0adNk06ZNki1bNrXPf1x5QkRwdjBXMgC1MIkPQLwd5okeekjklVd0BhlIDkxcE6/AABDhNfBCL1lSO865GX8TErwYfqR69epGnz59Ej7fuHHDiImJMUaNGmW7/YwZM4wcOXK43N/NmzeNAgUKGOPGjUtYFxcXZ0RHRxtz5szxuF3x8fHQ4dT/4UqPHoaRIQNeVY5LunSGMXq0Edns22cYPXsaxh13GEbVqobxzjuGce2a9/tp0SJpJ6ODb7vNMP75xx8tD1tefTWx+/A/ujV9esNYujTQLSPhTLwfZIHfNNVr165JbGysMs+apEuXTn3e4GYkf/HiRSlatKgULlxYWrZsKTt37kz4bv/+/XL8+HGHfcIeDrOyu31evXpV2c6tS7gzaJBIlixJw2cQQta9u0Quu3cneiFi2gDZX/r1E3nsMe/ykqIiNyan4f1lBfs4c0Y7aBCP+PtvkddeS+w+gG7F3337RkS6WBJG+E2onj59Wm7cuCH5nQJ98RmC0Y7SpUvL9OnTZcmSJfLpp5/KzZs3pVatWnIEkzAiCb/zZp9g1KhRSviaCwR2uFOqlE4J17y5jk1FdrCOHfW63LklckHuUcx7msLQ1DG/+kqX7vEUuLHaTVpbTcPEI1De0E5wonsx90qXCRJKBFXyh5o1a0qnTp2kUqVKUqdOHVm4cKHkzZtX3n///VTtd+jQocq7y1wOHz4skUDZsiKLFungesiRGTN0THZEAw3SrqIA3FERf+QpxYvDTOL6+2rVJBJA1rnfftPaZkoxQ75S+j0hESFU8+TJI+nTp5cTTiN2fC5gVjxOhowZM0rlypVl79696rP5O2/3GR0drRygrAuJUNxlbvfm7Q2vdFTwtts/Msq4SigeJkCLfOMNXdGrUiXtn4XTTolwhZ8XqtbYpTDEwLBECZ80mZDQFqqZMmWSKlWqyCqLSQ3mXHyGRuoJMB///vvvKnwGFC9eXAlP6z4xPwovYE/3SSIczJ3aCU+Yg+vW9X7iesIEnZoKoA7a009rN9YwB6f98suOEURr1uhKNM7TzMlx2206rhqYlwZjE3Tnhx8GXb4AQtxj+JG5c+cqz9yZM2cau3btMnr06GHkzJnTOH78uPq+Y8eOxpAhQxK2HzFihLFixQpj3759RmxsrNGuXTsjc+bMxs6dOxO2GT16tNrHkiVLjO3btxstW7Y0ihcvbly5csXjdkWC9y9xweHDhlGoUKKbqXWJijKMxx4zjHPnvOu+69cN4+hRw7h8OSK6/d9/DSNPnqTdZy5LlqRsvxs2GEaXLoZRv75hDBignbQJ8Sf+kAV+na1o27atnDp1SiVrgCMR5kqXL1+e4Gh06NAh5RFscu7cOenevbvaNleuXErTXb9+vZSFDeg/Bg8eLJcuXZIePXpIXFyc1K5dW+3TOUkEIbbATolJQMzTv/eezkRgApmASeizZ71zWoJ69Z81JRKAjxYWV12xfbvOm+4t996rF0JCGeb+5fxqZHLunK6WgSTJdmzZokNvSBLg+IZMSFeu2HfOrFkinTqx40jwE5JpCgkJSuD85kqggt9/T8vWhBSY60Sss8XIpMBnCNvWrQPVMkICD4VqkLB5s07NhhRt9eqJfPGF+zBIkkqQBcOdB0yRIuxiN4wendTEi0pfKDWIsoK+BvGqI0aI9OypnZeYYpkEKzT/BoH595tv9AsK73h4TmLEj2B4pJM1M80QP9CyZdK4VbidYmSDzEvOqhhJwo4dOkcvQmsaN7YPjUktCxaIdOigB5m4JHhGEG/9448ixYrxopCUw9JvYVj6DcIT2Y8wEnfWTPECQT0B+NYQP4B0gq1aifz0U+I6XAxUncH/JCimvmFUwDyu9fnA2AfZSqEZExJMsoC5SgLMvn3Iaexa4K5YIdKtW+plB8q9QXDfdZeuY5k9e+r2GRYgQHLdOm17h8pVtKi2vVNDDRoWLxaxK0AF4wLSG+LexmUkJFigUA30BfBzijYoYU2b6nRyGN3DdAaz8tq1WsBGPLC5V6+uFxJ0oPaFOR3iDDTXCxcoVElwwUmjAIM5oXLl7JUjzE81a+ao1SLE8to1z/YN51YkEIJAxUsJn/EiwujenKMiJJipU8d1lRrUxaA/GQk2KFSDQFGCNyPCFMy0tKZ2OmmSzoAHy2SVKnqaD3lWMceEymXJsXq1Lpbi/FKC6WzbNpE//vDDCRHiQ3C/t2njOOg0/x4zhpZ6EnzQ/BsEIIsMysZOnSqydasegffoIVKrlk7ug9F6fHzi9tA0ESeYXExgXJz74yb3PSHBwGefiVSoIPLuuyInT2pBO2xYyrI2EeJvGFITBCE17pg4UWTgQHvPYLxcUGPbFahwB98bOzMvYglRgvaWW3zfZkIICQWYUSkCgenXrloZTLrQbt0BjRfB8tYcB+bfqDBCgUoIIb6F5t8gB4LRlUORJ/GrU6ZobRXzs9BM4Rg1ZIg2HxNCSEDZulVkxgzt/IEaxIgfzJ07pC8Kzb9Bbv5F8oc77tChMM7CFYLyuec83xf2kdoQHUII8QnvvSfSu7d+KZnelPDMRBwgXnppAM2/EQi0zIULHU21MOHiXuzXz7t9UaASQoKCo0cTX2AY7UOoYoEXZt++EspQbwkBHn5Ym26Rkg3B7vAGZs5TQkjIsmiRfQCymSoLoQk5c0ooQqEaImTNKvLoo4FuBSGE+IArV7TJzZXDiF1uyhCByR8IIYSkLQ8+6DpV1p13iuTPH7JXhEKVEEJI2pI3r+t6xsg96a7WcZBDoUoIISRtmTvX9XeoHOVpgvMghEKVEEJI2hIXZ5/VBkCghvCcKh2VCCGEpC21a+tQGmdg9kVNyhAu+ExNlRBCSNrSsKEWrFZtFQnN4Q385pucUyWEEEI8BgL02291ogczsw1KEX39tUjLlhLKME1hkKcpJISkHuQUQL6BxYu1MtS8uchjjzHLWFBgGNoUnDFjmh/aH2kKOadKCAlrrl8XeeQRkWXLEq2Nn38u8tFHel10dKBbGOFERQVEoPoLzqkSQsKaTz7RwtPUWLGA1au1YCXEl1CoEkLCGmilKfmOkJRAoUoICVuWLBFZv971VN7ly2ndIhLucE6VEBKWwLkUc6mucrZjfhUVoAgJKU116tSpUqxYMcmcObPUqFFDfvnlF5fbfvjhh3L//fdLrly51NKwYcMk23fp0kWioqIcliZNmvj7NAghIcbw4a7DHbG+YEGRZ59N61aRcMevQnXevHkyYMAAGT58uGzZskUqVqwojRs3lpMnT9puv3btWmnfvr2sWbNGNmzYIIULF5ZGjRrJ33//7bAdhOixY8cSljlz5vjzNAjxHKhFmzeLLFggsmMHey6AlyE21nUhlHLlRDBeR153QkImThWaabVq1eSdd95Rn2/evKkEZb9+/WTIkCHJ/v7GjRtKY8XvO3XqlKCpxsXFyWIEnAVRbBIhcuSIDlzfssUxc8z8+SK5crGD0pjbbhM5e9be7PvKK1qTJZHNeT/IAr9pqteuXZPY2Fhlwk04WLp06jO0UE+4fPmyXL9+XXLnzp1Eo82XL5+ULl1aevXqJWfOnHG7n6tXr6rOsy6E+BSMTSFQt293XL9mjchTT7GzA0CPHjpxj92l6tgxEC0ikYDfhOrp06eVppnfqdgsPh8/ftyjffzvf/+TmJgYB8EM0+/s2bNl1apVMmbMGPnhhx+kadOm6liuGDVqlBqNmAu0ZUJ8Cky+0FCdk4TjvoQL6uHD7PAUAu9dCMjWrUXGjxc5d86z3w0bJtKggf47QwYtYPH/zJkiJUrwcpAI8/4dPXq0zJ07V2mlcHIyadeuXcLf5cuXlwoVKkjJkiXVdg3MJ8iJoUOHqrldE2iqaSVY8Y5dtUrk2DGRypVFKlZMk8OStOavv1x/B9Xo0CERDua8ZtQokRdf1MIQ4xPM+kycKPLzzyLFirn/bZYsIitWiPz4o8gPP4jkyCHy+OMiBQp43w5CAi5U8+TJI+nTp5cTJ044rMfnAsnc1ePHj1dC9fvvv1dC0x0lSpRQx9q7d69LoRodHa2WtAaWQLjsW5UUOCrDh8XMIU3CBJSrcgVUpJIl07I1YcH//Z8WqMA0AGB8glfKCy+IfPFF8vuAl+8DD+iFkJA2/2bKlEmqVKmizLQmcFTC55o1a7r83dixY2XkyJGyfPlyqVq1arLHOXLkiJpTLQj/+CACdXYbNxY5etRx/cqVIs89F6hWEb8BE0TdukkLL0OgPvkk1aMUAKFpV8faTI5/9WpKLxYhIRpSA5MrYk9nzZolu3fvVk5Fly5dkq5du6rv4dEL06wJ5khfeeUVmT59uoptxdwrlosXL6rv8f+gQYNk48aNcuDAASWgW7ZsKaVKlVKhOsHE0qUimDp2nurFZ+Qipa9UGPLllyJNmzoK1CeeEHnvvUC2KmT55x/XcaYIlXHjRkGCka+/1pPcsNvDhAcnvnDE8DNTpkwxihQpYmTKlMmoXr26sXHjxoTv6tSpY3Tu3Dnhc9GiRRHek2QZPny4+v7y5ctGo0aNjLx58xoZM2ZU23fv3t04fvy4V22Kj49X+8X//mLSJMNIlw7GKvtl716/HZoEmgMHDGPdOsM4dizQLQlpfvrJ/tnBc1WrVqBbR7zirbf0xUufPvH/qCjD+PTTgHakP2QB66n6KU4VVm+L07ID2bOLIP+Fxf+KEJJkwC/Spo3IwoWJqQZhDoYBAErOffexy3wC1P61a0V27RIpWlQ7fviyFNu5c3r6A3NiziBcEl6cmTJJIAipONVIp149Pc3mPCcEc9bzz1OgEpIceFaQLO2tt0TKlhXJl0+kVSuRjRspUH0G5qjuuUebZZGzsUUL7VS3e7fvjrF2rb1ABcjOYU2WEgZQqPqrY9Npd37rVC80UySSQvwcISR5oDBhELpzp/b6hfMSZADxEchUh84FpjkA3pWY83SV49FbMiaj9YZRgXJA828apClE6mK8EO64Q5t+CSEk4CB2GuZeV6CKO0xuqeXyZW3+vXAhqeZRqJDI/v32bt5pAM2/IQruG4yuKVAJIUED5jLd4RwPmFKyZkUJssSUVgD/Y5k+PWAC1V/Q/BtAYG1BeE2lSlrgVqkiMnduIFtEvOKPP0QmTBCZPFnk4EF2HgktSpd27yCEF5OvaNtWz50+/bTIgw+K9Okj8vvvrr05QxiafwNYpebNN0Veekk7ZEDAYiCHaQzkNx04MGDNIsmBi4UMHlOmJGZsxzpcUA+qLxESNCB966RJjpXcoTki3hpxpWHOeT94/1KoBkiouvMyh7UETnk0FwcpMC/8V4rQb/NQhKQFyP+IOniwtmDuEyZZ3Ntvvx0RuVTPM6QmfED1O1de5ri3UUCZBCkffGBfUwwvpI8/DkSLCEkZuGdRteDUKT2dcfq0vod9IVA3bBBp317HFsL8iyoIEQDnVAMEtFF3ZMuWVi0hXgMHDrtwA4z6k3P+ICRYX0iYY0UpH18wZ44OJkYMFCqLIIPH/feLzJ6d+n3v26drFMfEiJQqJfLaayKXLkmwQKEaIGrX1uZfZ4UH86sQqLgHd+wIVOuIW1AQwvRitIK5qOrV2XkksvnnH+2IhHlas7wQ/sfnfv1ErlxJ+b737NEenZiCwQAWAnbECO3w5Mr0l8ZQqAYIvJM/+0w735mp1wDuO9yTqBlZvrzIO+8EqoXEJYMG6QtmHRHhImI0hJcJCXrwnCEzE3xxEEdOfMjGja4ryaOSyE8/pXzfplZqCmsAqxGOiYIWQQCFagCpX1/XjHz5ZR2DbVbkQPUN856Bk6m7+tckAGCOCDX8rLV+ob2iGvbtt/OSBDlIIFSmjL5kyMpXpIhIz56O72niJXiRIc3h/feLvPGG+21dlR7yhGXL7C8UtJTlyyWsi5QTzyhcWAtVRGNYvdqt9x9iV81izSRIQNXrrVt1ZQRoqbfdFugWEQ+A5RFpbuGPY1V04HuWN6/IyJHsRq/BYLJRIy3s/v3XfmrEBHO2mPtKKdHRKfsuDaGmmkKQcQvz79aHM6XgPrx+3f47CFXn7F4kiECWdwrUkAG+CkgZ6lyLFQNahB1TW/USdFz37no+0zp/amJmS4Kgxcvs3XdTV02kQwf7DEw45uOPSzBAoeoluHYvvKDfpbAC5s+vvcbj41N+EXCPVatmH6WB4zHskRDfAL8WV4oUnmFXU4HEBXv3ivz5p+vk+/feK1K1qg6pQYgNhGJqgFkPXsoQ0FhMAQvBDhNEEEDzbwp8VBAXbZpqcS8tWKDDvL7/PuUXYvRobUHBPWKOoiFk69QJy0xehASEO+90rY3mzCmSK1datyjEcVb5nUFawi5dxGfgAiGIH96/eOEiFKhdO50BKjVztT6EGZW8yKgUF6c1U1ee20htWblyyi/GunUir76qneNQDQkvADgqYXAXoBq+xFfA63H+fJ0j+O67RR55JGjmgCKJq1d1uVBkLLPKA7yP8eyxLKOXQKtAh+K+dnYKSZ9e5PBhkYIFJVhhRqUgMB25C4XCHGtqfV/gEYz5VRwHcapdu4rUqKEFOglRNm3SLqY9eoiMGaPnC2DCOnAg0C2LODCOQSZJjGtMYA6G4yrycBMvgTkN86QQoKZdPf1/JlnEj5oCFQIXoTC+qtEaxHBO1Qtwf7izMKQ2mgLOpEjDCUxHOoBiDrg/SZCDFwbMFRCi5ugLI6RWrbS3GV4spkfakSMiHTsGtLmRCixA27bpBZFRSJCFnPJhVoEs7YDpFXGicBQqU0bPbS5ZokcpuOcRbI8wB6Q+zJNHZPhw156ZYQDNv14m1G/ZUodKWU1HeBgRZ4pkH3bORt7M1+LhtpvzwVTC2bMp3zfxM2vWaLOCWQIOHsGYfM+dW+Shh1z/DkHIxYvz8pDw5M3/SnFZgWbSubPIjBkSaGj+DQJQU9c5Ex0GYRC0qRGo4OJF198FUWpLYucBidE65o9MzpzRmihi+Nzhi5gsQoKRixftE0FAe501K2yz2tD86yVQQFBsYf16HTCOJB54p8LqkVrq1rXXUqEJM6wmiMGcEkwXzvNFGGXhRnEF0hqWLWv/3Q8/6El2eKjBZDZ4MEdWJPRSV12+7D5PZBhCoZoCYL1AijOERjVu7Lu5mEcf1SFd1v3hbyxIeUmClF277EdDELT792v3bTszxtCh9uWIVq3SHmsYvWHuCVrvW2+JNGmSfAgDIcHCbclkGcPUSBhCoRpEIIwG79P+/bVygqQQiF1FiA2LnwQxmBO1yygAQYoRETxhEHaQJYtej/JEmDx3lXvSXG/VfPE3boQVK/xxBoT4nlKldOiCs9aB5wKxiUGSrMHX0FHJS0clQpLw228i99xjHy4AswZMXWZWD4TTIHDdlXkDXsOu4lcx6sKIa+xYXgQSGuzdq+e1UAoI9y8sOtmz63kzmPsCDB2VCAlGkK8SdfwQMmAVphiRmwHxptkWxZuR5cMV0Hhd5UaF0OYgkISatrpnjy5OPnCg9j84dCgoBKq/oKbKlxTxFXDRRmYBaJsIr7GrhACh+cwz7gvlduumvSOd508hqPGCgimZEJJqqKkSEixAA120SMegwvTbr5/Ofde8uUjr1u4zxyTnbISsS8i4ZAphs8IHyqhQoBIS1DChPiEp4X//Exk3LnGuFGmvEMwO0y6ELIQrKi04C1DMKT38sPt9w0sNmZm++EI7J8FL8sknRe66i9eKkEj3/p06daoUK1ZMMmfOLDVq1JBfUGHADQsWLJAyZcqo7cuXLy/ffPONw/eGYciwYcOkYMGCkiVLFmnYsKHsgUmMkLRi924tUIEpNCEs//lHV0AAqHaNqQXTK9gsVdWsmU4UkRxwVnriCZH33tMB9BSohIQEfhWq8+bNkwEDBsjw4cNly5YtUrFiRWncuLGcPHnSdvv169dL+/btpVu3brJ161Zp1aqVWnYgs/x/jB07ViZPnizTpk2TTZs2SbZs2dQ+/8ELjZC04Ouv7eNOIWChWaL6ARw0kMy5Vy9tykUA8uTJ2mSc2tRbhJDgxfAj1atXN/r06ZPw+caNG0ZMTIwxatQo2+0ff/xxo1mzZg7ratSoYTzzzDPq75s3bxoFChQwxo0bl/B9XFycER0dbcyZM8fjdsXHx8MlU/1PiNeMGWMY6dJhVtV+OXeOnUpICOAPWeC3IfO1a9ckNjZWmWdN0qVLpz5vQAV4G7Deuj2AFmpuv3//fjl+/LjDNjly5FBmZVf7BFevXlVeXtaFkBTTooW9IxLmV2vX1tWuCSERid+E6unTp+XGjRuSH5kzLOAzBKMdWO9ue/N/b/YJRo0apYSvuRRGBnxCUgoSPSMXLzCTOOB/xJeiMg0hoQDsKsj2depUoFsSVkTE5M7QoUMlPj4+YTlsrSZCSEoYPVrPjyL5c5UqIr17J2ZWIiTY+f57kXLlRAoVEsmXT+T++7UHOwnekJo8efJI+vTp5cSJEw7r8bkAcp/agPXutjf/xzp4/1q3qVSpksu2REdHq4UQnwFPXhQfx0JIKPHrr9oD3RruhekzVEVCcQjLu5UEkaaaKVMmqVKliqxChvj/uHnzpvpc00WKKqy3bg9WrlyZsH3x4sWVYLVug/lReAG72ichhBCn5CLATKEJIGCRAWzaNHZVMCd/QDhN586dpWrVqlK9enWZNGmSXLp0SboihZuIdOrUSQoVKqTmPMFzzz0nderUkbfeekuaNWsmc+fOlV9//VU+QOFSpRxESf/+/eX111+XO+64QwnZV155RWJiYlToDSEk/DhyBPHuIghxhxL19NM6RztJIZs2uS5VuHkzuzWYhWrbtm3l1KlTKlkDHIlgol2+fHmCo9GhQ4eUR7BJrVq15PPPP5eXX35ZXnzxRSU4Fy9eLOVg+/+PwYMHK8Hco0cPiYuLk9q1a6t9IlkEIWHJuXO6woc1YX+EsH27nu5DWmW885FLA7ULMKWNpFYkBWBkgqoxzh7s6FyaflMNE+ozoT7xBDi3vfyyyJdf6pcRwmpef10nefAXa9bA3COybZuew0WRciSQ8OcxgwwIVEz3OWd7xFgc9d+LFAlUy0IYpNN86in779avD+sKMs4woT4hgQAhByi2DBUJKtOVKzovL9b5y5Mctk5UqIeqZs5/ffedyH33iZw5I5EAThMJquzqD6A7Fi8ORKvCgC5dRPr0SRydmGUKJ06MKIHqLyIipIaQVIEakEitaX274+/4eJEJE/zTuW++qSWH1USHY54+LfLRRxIJuCvmAzlgnRZcu1YbD4oX12ORZcvSpImhCToPpQeRw/qtt/TfqHHav3+gWxYWsEoNIZ7E9Nm94bFu5Ur/9J+dzRNA0CZTlCJcQPgkIuWgrDtP/+EzahMAGBA6dtTKFroMxgNclkmTEusbEBdJTLAQn0JNlZDkyJHDPgk+Rvz4zl8SBft3Bpmb8uaVSAEJqsySssC8DM8+q+sUXL2q/8ZYwxyDmP8PGaJrGxCSllCoEpIcUIPscv3iTd6pk3/6r0cP+/Wwef4XkhYJIB8BojzattX12WvVEvn0U62FApSdPXvW/rcoXAWzMCFpCc2/hCRHmzYi334rMmtWosoE4fbIIyLduvmn/5D2EPGEsG1COzUD9ceP1w5SEUSFClqQ2mFeDlcgEomQtIQhNT4OqcG7D04SeAnAj6VOHa105M7t08OQtAYX9ocfdL5f2BfhFYNqSf6ujRobq71+EYf96KMiRYv693ghBi4FugR54a0JgmA5z55d5NgxkaxZA9lCEmkhNRSqPhSqeKj79tXOolAu8MDjnRsTI7Jxo85dTQhJHf/3f/oZg/NqiRJak4VDEp4/GBCgvcJa//nn2mxM/AQ8hv/4Q6RYMZE77wzJbj7vB6FK86+PHTbxsFudJfBwY7SMvAGIuSaEpBx49T78sH6urAIUVvEDB0R27tRzr716ac9h4gcuXtTJIxCrbZoH6tcXmTNHO9hFONRUfaipDhyoE97YpdWECQp5AwghKQMDVShFyLBHU28A6dBBZP58x5AvjG6qVxf5+WcJJZhRKcixE6aefEcISR54+iK5vlWgAnw+f56evmkCzG5z5yaNocYLDikOt2yRSIchNT4EJQrthCfmVx96yJdHIhEFbJovvqgn7GFyu35dIpHkTjtCuyVt+euvpKMa5wnvCIdC1YcgPRrme6wx+7CKwPT7xhu+PBKJGDBZiCpN48aJoAQiwnsQrAnX8gijShWRXLnsv4uO1p72xGmU4U4ApgTkgbRLSmJSsmTEXwIKVR8CT9+FC3VgeuXK+v7DfD4sImXLRvy9RuzAS++rr0RQD7h2bV3PDDZOsGOHyKBB+m+YQExVbOtWkVdfjbj+hOCEzwLe6bD+APN/lGTOmTOgzQue++n99/XLJ1MmkQIFdOe4S6QMTpwQmTJFZMQInZbTLtkJQCjDY48ldrzznGrVqr47l1DFiEDi4+MxfFP/ExJQBg7Ea9Aw0qdP/D9XLsPYtcswhgwxjAwZ9HrnJWfOiL1wq1cbRrNmhlG0qGE0aGAYX38d6BYFEaNGJb1XoqIMo2dP17+ZO9cwMmY0jHTpEu+3++83jAsX7LfHe7N5c8dj3HefYRw9aoQa/pAF9P5lPVUSKH7/XQdZOgMtAHMJ0DY+/NB+shCaAScRiXOoS/78IpcvJ+0XqPeIOXIuQAurCO4zZ2cQ3IOYwzfzQdqxd68OFoZLdvnyIXktzvshTpXmX0ICBQqCOpvRAEx1y5eL3HuvveDEb1C9mxDnQZqdQAXQJxFI7wwyZNiZenEPosSgKzMwKFVKpHnzkBWo/oJClZBA4e6FhZcgUiFict4qeM2i0q+/niZNJCGEKy8ud9+jPq/dwA4gsJ6xgF5DoUpIoICruJ0DCV5yyFCDsnKrV+vk+vgb61G2BaVX4AFMiBXUwnMehJkDMZiF69VL2l9wLrKzhmDgBq9zODsRr6BQJSSQMSLdu//3JKZLnCtF8vy33tKf4dIKl1cUBoXWsGaNyH338ZoRe0GIqka33ab/Roke/I+Yvi+/tC/Z07KlFp5WQYzfwFLy2mvs5RRAoUpIIJk2TeSTT7QGirgrCNlt25i4lqSMu+4S2bdPxzT366ervB886HogBkGLgVr79olCF3OlCxbo0obEa+j9S+9fQgjRVd2vXNHWEXcJHsKI86xSQwghxC9g2gELSRU0/xJCCEkelAdC7CoKQ6My/ODBImfPsuecYD1VQghJ61AqzGOiyDdiPOGwFuzmVlSnqVZN5OTJRI/1CRNEliwR2bxZhNNoCVBTJYSQtAJVXBD60rChTgwOQVW3rsi5c65/A0/cq1dTlhz/1CmR558XKVhQFxDv0UMLc2+BALUKVIC/9+zRSSJIAhSqhBCSFkBDbdZMZP9+x/Uo7P3000m3hxCFF2+JEnquM29enfDe04QMqGSEeGYkyj9+XAvYGTO0IIcp1xuWLbOPqUYbv/3Wu32FORSqhBCSFsDki3y5zsIJnxct0oLPCmKVn3lG5+wFZ85ooWrGNicH8kaj/qn1eBDI2I8ZB+0prhyYEF+dJYt3+wpzKFQJISQtOHzY9XfQ+KzaI3L42iVfwHYzZ2rhnBwrVrjO6/vNN+IVHTrYz/ti/23berevMMdvQvXs2bPyxBNPqMz/OXPmlG7duslFVFFws32/fv2kdOnSkiVLFilSpIg8++yzqnqAlaioqCTL3Llz/XUahBDiG5C5yBVIB2gt8I3qLxcuuN7eLjm+M8ikZGbqsvvOG/r0SUwggX2aGZhQB7hdO+/2Feb4zfsXAvXYsWOycuVKuX79unTt2lV69Oghn6Mqgg1Hjx5Vy/jx46Vs2bJy8OBB6dmzp1r3xRdfOGw7Y8YMadKkScJnCG1CCAlq4OVbp47ITz85mmShAfbs6VhlPSXJ8Z2BsPvqq6Trcbwnn/Sm5drEu2qVyPz5IkuXaqHaurVOc+gqIX+kYviBXbt2qcKvmzdvTlj37bffGlFRUcbff//t8X7mz59vZMqUybh+/XrCOux30aJFqWofi5QTQgLC2bOG0bq1LhyO12+mTIbx3HOGce1a0m3vvTexeL25oJB4njyGcfVq8sf691/DaNNG/w7Fx8191a1rGFeu+OX0Qg1/yAK/mH83bNigtMeqVasmrGvYsKGkS5dONm3a5PF+zMKxGZBk3EKfPn0kT548Ur16dZk+fToGBm73c/XqVZWOyroQQkiaAw0TlrejR0ViY0VOnNCFwO2S3SMnNKrLADM5PjRGJMf3pHoMNEhMjUFbhWaK/L74/N13zJwUaubf48ePSz7ERFkPlCGD5M6dW33nCadPn5aRI0cqk7GV1157TerXry9Zs2aV7777Tnr37q3majH/6opRo0bJCHjNEUJIMFCggF7cgcT2iAOFyXXHDp3FCA5DqELjKZj/RCFxLCT4hOqQIUNkzJgxbrfZjQn2VAJNslmzZmpu9dVXX3X47pVXXkn4u3LlynLp0iUZN26cW6E6dOhQGTBggMP+CxcunOp2EkKIX4FDUZcu7ORwFaoDBw6ULslc4BIlSkiBAgXkJLJvWPj333+Vhy++c8eFCxeUE1L27Nll0aJFktHOLGKhRo0aSqOFiTc6Otp2G6x39R0hhBASEKGaN29etSRHzZo1JS4uTmJjY6UKPN5EZPXq1XLz5k0lBF0BDbJx48ZKAH711VeS2YOKCdu2bZNcuXJRaJLwx8xeAw96hKfVry/StatI9uyBbhkhxJ9zqnfddZfSNrt37y7Tpk1TITV9+/aVdu3aSUxMjNrm77//lgYNGsjs2bOVwxEEaqNGjeTy5cvy6aefOjgUQZCnT59evv76azlx4oTce++9SuAiXOfNN9+UF154wR+nQUhwCdTevXVRczigIOgeDihIQbd+vU5hR0IHhKfMni0SFydSu7ZOU+hJmAwJfgw/cebMGaN9+/bGLbfcYtx6661G165djQsXLiR8v3//fuXKvGbNGvUZ/+Oz3YJtzbCcSpUqqX1my5bNqFixojFt2jTjxo0bXrWNITUk5Fi71jG0wlwQJtG7d6BbR7xh6NDEMBczTKZQIcM4dIj9mMb4QxZE4R+JMPxR7Z0Qv4I6lu+/b59MHRoO61qGBtu3i1SsmHQ9wgYfe0xkzpzk9wHT/8KFuhxb5cq64o2rzEkkzWUB66kSEgpcu+b6u+vX07IlJDUsWKAFqPPgCJ8RfwqzvjsBiWxMDz+sK9BgGgCZme65R8eeehNqQ/wGhzeEhAJIy2mnpeLFinJiJPQHR7i+dgnwrUn2W7RIzAlspjr87TeRXr183FCSUihUCQkF8DJFMWurFgONJ1s2EadY7jQH1VXgMDVunH7Bk5QNjho00NfUFXBMQzFzZ8EL4Qot112hc5JmUKgSEgrgZYtwmjffhHu9yO23i3TurFPdlSkTuHZNnSpSpIhI//7IDiNSqZJOVmBX0JrogRGS0CPloFlKDdcWaQeTSayjUhq6Mg1D0HJePSigoxIdlQhJGRDolvzeiW+VKJG33xbp148962oOHIOR6dO1IKxXTw9I7r7bfX8hdMosv+ZM7tzaccmTnMDEr45K1FQJISkDQsGVuRKeysQeZImDZg9P4CNHdOL85AQqqFlTl46zK7X20ksUqEEChSohJGUgFamdmRdReh4WziBeAAsA5lVh9jfTtyLpB6rcPP88uzJIoFAlhKSMatUS5wWtQJO69172qj+AifLjj7VTErRclJB77jn760ACAoUqISRldOumYyOt5kjz5f7ii+xVfwKv70KF3HsLk4BAoUoISRkQqEhGgFAQE3gmf/ONSK1aodOriP8cP15r3kikgNrL9KQlKYTev/T+JST1IDH81asi+fKFlinyyhUd5vLrr4nxnwhbKV5cZNMmZikKc87T+5cQEpTkzCmSP39oCVQwa5bI5s2OCRXw94ED2gGIEC+h+ZcQErnAm9YOM0sRIV7CWW5CiGdm0sWLRQ4fFilfXqRRI/t4yVADpl5o13bFuiKvgBfxARSqhJDkMyc1bSpy6lRiZRQkK/j+e5ECBUK79x55RGTZMvvvELaCMmu33JLWrSIhDM2/hBD3KfWaN0/0hjWTPfzxh0jXrmnfc5jv/P13kW3bfJNfuGNHkRIlXOfaZWYo4iUUqoQQ15pajx46p6yzAMPn5ct18oG0AppxyZIiFSro4txI5L9kSer2iVy5yErkCs6rEi+h+ZcQkhSkGURWpEOHkk9VGBPj/x7cvVvkoYcchTuE/aOPimzYIFK9esr37W5uONS8mUnAoaZKCEnKa6/pNHjunHWQ1eeOO9Km91CvFW2xhr7gMxyNJk5M3b4hmO2EJ9bhO0K8gEKVkEjg/Hmd6WjlSp2kITnmzUt+znLQIC1Y0wJUdLEr7o11mF9NDT17anOyVbDib6zDd4R4AYUqIeEOanfCS7dZMx0KU7CgyKJF7n/jTqBiHhLF0l95RdIMZDiyy3ML060rRyNPwcBg3TqRt97S9Uqx4G+sS6tBAwkbmKaQaQpJOAPtFMLUCrQwmE2h4ZUrZ/+7Tp1E5syx1w4XLBB57DFJU9wV6MY5IuTHExAWNHmy/k2WLCLt24s8/bRIdLRPm0tCA6YpJIR4B+YbnR1xMBcJwfree65/9+qrIjlyJP4W22OBpovYzrQGCfo//FAkc+bEdagpOmGC5wIVzldVqoiMGiWyZYsW1P36aQcohA4R4gNo/iUknNmzx96UCw10717Xv4NJdetWkb59RUqX1tVbkAv3668Dl0kJGiUEIzTluXO19683xbkhTBECZPYHBhdYVq9m6AzxGQypISScKVtWe/E6C1bMT0JYuqNw4eBLKg/tOaWmZ8Sc2g0wMEhAvGu7dqluHiHUVAkJZwYMSCpITC/XXr0konAXc8p4VOIjKFQJCWcaNhT5+GOt4Zmg5imqs6CgeCTRpo296RqDjlatAtEiEobQ+5fevyRSqsz88ot27kH2IbvwFE+BOXntWh1u0rixSNas4jcQV/vuuyIHD+p5XcyhIpl/SoDnb40ael9IImFWp4GjEwYZqekTEpKc90ORcgpVClVCPAMCaPBg7XFrZjbKnl3kk09EWrb0fS8iVvSFFxIr40DoQRAi53D9+inbJwoDwOvZDKnBPGrnznqwQSKO834Qqn4z/549e1aeeOIJ1dCcOXNKt27d5CLKKLmhbt26EhUV5bD0dMpocujQIWnWrJlkzZpV8uXLJ4MGDZJ/7WLpCCG+BSEt48c7pgrEMw3HoX37fHssVIgZMkT/bc4J4znHgndCSmud5s4t8tJLIj//rBP0w6OYApX4EL8JVQjUnTt3ysqVK2Xp0qWybt066YGKF8nQvXt3OXbsWMIyduzYhO9u3LihBOq1a9dk/fr1MmvWLJk5c6YMGzbMX6dBCDFB0gRnhx4zLAXztr4E2qjdYBnHQpgQFkIiRaju3r1bli9fLh999JHUqFFDateuLVOmTJG5c+fK0WRKRUEDLVCgQMJiVcm/++472bVrl3z66adSqVIladq0qYwcOVKmTp2qBC0hxI+gYo0rDTG5ajYkKfHxIm+8oRNSYEHqR+RoJiGNX4Tqhg0blMm3atWqCesaNmwo6dKlk02bNrn97WeffSZ58uSRcuXKydChQ+Xy5csO+y1fvrzkz58/YV3jxo2VXRxasSuuXr2qtrEuhBAvQUpDpDd0BoI2pc5DrmjSxN5xCMdHZZy0qo7jL/AOQpYoWNmQ3QkLcikjFeOFC4FuHQk2oXr8+HE132klQ4YMkjt3bvWdKzp06KC00DVr1iiB+sknn8iTTz7psF+rQAXmZ3f7HTVqlJqMNpfCCGonhHgH5jit86kATkRwVurWzbe9iecaGZDMYwAIWfwNR6NQjyuFR/Mffzj2J/7etUtk2rRAtoykpVAdMmRIEkci5+UP3CgpBHOu0DyhjWJOdvbs2bJo0SLZl0onCAhoeHeZy+HDh1O1P0KCljNnROCH8PjjOq8tUg36ihYt9NzpbbclrkOs65o1OvbV18Dz97vvRJo312XY4KULja5BAwl5Fi9OOkABWIfvSMjiVWDWwIEDpUuXLm63KVGihJoLPXnypMN6eOjCIxjfeQrmY8HevXulZMmS6re/INbOwgl4CYq43W90dLRaCAlr4LwD8yEEq2kqfecdrRX5KnvSU0+JwHqE6RbEp955p3+1xgcf1Eu4YWdG9+Q7El5CNW/evGpJjpo1a0pcXJzExsZKFUzAC3JWr5abN28mCEpP2PZf8eGCqP/4337feOMNJbBN8zK8i+HMVBY5TgmJZHr31nGYpgZk/g+NFXGkMTG+OQ7qqUJzJCkHYUgbNyZ1/MIApXVr9mwI45ch0V133SVNmjRR4THQLH/++Wfp27evtGvXTmL+e7D//vtvKVOmTILmCRMvPHkhiA8cOCBfffWVdOrUSR544AGpUKGC2qZRo0ZKeHbs2FF+++03WbFihbz88svSp08faqIksjl3Tsdd2iWMh3BNrig5SVsQawtHTrOknrlUq4Z5MF6NEMZvebngxQtB2qBBA+X127p1a5mMOLf/uH79uvz5558J3r2ZMmWS77//XiZNmiSXLl1SzkT4DYSmSfr06VXMa69evZTWmi1bNuncubO89tpr/joNQkKDq1ddf4eXNdIUkuABpnOkepw+PXHAgzq1cPhCpicSsjBNIdMUknDADGuBo6BdLOlvv4n8Z/EJa3Du0NhR5g3JI1CAHKbvQNWAJRGXppAZpAkJB6CNIidvs2ba0QVmYNOBqFOnyBCoMHPDkWrWrMQYV3grw1t42TJ4LAa6hSQCoJsZIeECEib88AOcD3SOW4S7vP2271MIBisIRYFAteYJBqtXay/omTN1n0DglighMmWKfVgLIamA5l+afwkJD+A1axf/CY0dEQRIkWqWezMZOFAXCSARyflQqlJDCCFpyqVL9ponhOixY4l/W5k0KfE7QnwAhSohJDxo2NA+cQLWuSoEgLnn9ev93jQSOVCoEkLCA9RGLVrU0dMX86e33OL+d5wCIj6EQpWQSGXvXpHnntOpDdu10zl8Q5mcOVHKSuSZZ3R+YgjL9u1FYmO1g5JzWA00WGRmq1s3UC0mYQgdlThKJZEITJ4wl16/rr1kIXBgCkVYzvPPS9iBlKf164vExWntFeeaObPIN9+I1KmTuB3SPM6di7JXusYpQpTsStCRsOC8HxyVKFQpVEmkgfnFihV1Uny7Um5HjqBChYRlKsfZs0V279YhNah6Yy0liYo4yGqE7FPoBww2kFADITn+qMJDAg6FahB3JCEhw8GDIsWKuf7+o498Xx81FIqGIy850qZanZqgpbZqJbJgQSBbR/wEQ2oIIanHLum+lUhMiID8uwjJcfYShra6cKFIfHygWkZCDDoqERJpFC/uug4qnHeaNpWIAzVoXdUxxSADc7GEeACFKiGRBoQpCpdj3tB0wjE9Y4cNE7n9dok47r3XtYaO+eVI7BOSIihUCYlEkGT+1191yAnCTeAZCxPo8OESkdSsqfvETlsdMYJVbojH0PuXjkqEEIA51SFDdAECeAAjkQQGGV27sn/ClPMMqQnejiSEhAmI3YWAzZHDft6ZhA3nWU+VEEL8TMaMOjsTISmAc6qEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4CApVQgghxEdQqBJCCCE+IiKr7xr/VaJA4C8hhJDI5Px/MsCUCb4gIoXqhQsX1P+FCxcOdFMIIYQEgUxAlj1fEJG5f2/evClHjx6V7NmzS1QapyHDyAjC/PDhw2GVIjEczysczylczysczylcz+t8EJ0TxB8EakxMjKRzVfrPSyJSU0Xn3R7gUk64mQJ9Q/mDcDyvcDyncD2vcDyncD2vW4PknHyloZrQUYkQQgjxERSqhBBCiI+gUE1joqOjZfjw4er/cCIczysczylczysczylczys6DM9JIt1RiRBCCPEH1FQJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQjUNeOONN6RWrVqSNWtWyZkzp0e/gf/YsGHDpGDBgpIlSxZp2LCh7NmzR4KFs2fPyhNPPKGCt3FO3bp1k4sXL7r9Td26dVUGK+vSs2dPCSRTp06VYsWKSebMmaVGjRryyy+/uN1+wYIFUqZMGbV9+fLl5ZtvvpFgxJvzmjlzZpLrgt8FE+vWrZPmzZurzDdo3+LFi5P9zdq1a+Wee+5RXqalSpVS5xnK54Tzcb5OWI4fPy7BwqhRo6RatWoqW12+fPmkVatW8ueffyb7u1B5rjyBQjUNuHbtmrRp00Z69erl8W/Gjh0rkydPlmnTpsmmTZskW7Zs0rhxY/nnn38kGIBA3blzp6xcuVKWLl2qXhA9evRI9nfdu3eXY8eOJSw4z0Axb948GTBggHLv37Jli1SsWFH18cmTJ223X79+vbRv314NILZu3apeGFh27NghwYS35wUwOLJel4MHD0owcenSJXUeGCx4wv79+6VZs2ZSr1492bZtm/Tv31+efvppWbFihYTqOZlASFmvFYRXsPDDDz9Inz59ZOPGjerdcP36dWnUqJE6V1eEynPlMQipIWnDjBkzjBw5ciS73c2bN40CBQoY48aNS1gXFxdnREdHG3PmzDECza5duxCGZWzevDlh3bfffmtERUUZf//9t8vf1alTx3juueeMYKF69epGnz59Ej7fuHHDiImJMUaNGmW7/eOPP240a9bMYV2NGjWMZ555xggmvD0vT+/LYAH33qJFi9xuM3jwYOPuu+92WNe2bVujcePGRqie05o1a9R2586dM0KFkydPqjb/8MMPLrcJlefKU6ipBiEYZcOkA5OvNT8lzHgbNmyQQIM2wORbtWrVhHVoK3IqQ6t2x2effSZ58uSRcuXKydChQ+Xy5csSKOtBbGysQx+j/fjsqo+x3ro9gAYYDNckNecFYLovWrSoSnTesmVLZYUIZULhWqWUSpUqqWmhBx98UH7++WcJZuLj49X/uXPnjphrFZEJ9YMdc44kf/78DuvxORjmT9AGZ5NThgwZ1IPjrn0dOnRQL27MIW3fvl3+97//KVPWwoULJa05ffq03Lhxw7aP//jjD9vf4NyC9Zqk5rxKly4t06dPlwoVKqiX4Pjx45UPAARroAtPpBRX1woVUq5cuaL8FEINCFJMB2Ewe/XqVfnoo4+UnwIGspg7DsZqYP3795f77rtPDaJdEQrPlTdQqKaQIUOGyJgxY9xus3v3bjX5Hm7nlFKsc65wRsBLokGDBrJv3z4pWbJkivdLUkfNmjXVYgKBetddd8n7778vI0eOZPcGCRj8YLFeJzw7EydOlE8++USCjT59+qh50Z9++kkiCQrVFDJw4EDp0qWL221KlCiRon0XKFBA/X/ixAkleEzwGaafQJ8T2ufs9PLvv/8qj2Cz7Z4AczbYu3dvmgtVmKDTp0+v+tQKPrs6B6z3ZvtAkJLzciZjxoxSuXJldV1CFVfXCg5ZoailuqJ69epBKbT69u2b4MCYnLUjFJ4rb+CcagrJmzev0kLdLZkyZUrRvosXL65uqFWrViWsg9kKZh6rRhGoc0Ib4uLi1NydyerVq5W5xxSUngCvTGAdOKQVOI8qVao49DHaj8+u+hjrrdsDeDj685qkxXk5A/Px77//HpDr4itC4Vr5AjxDwXSdDMNQAnXRokXqnYB3WcRdq0B7SkUCBw8eNLZu3WqMGDHCuOWWW9TfWC5cuJCwTenSpY2FCxcmfB49erSRM2dOY8mSJcb27duNli1bGsWLFzeuXLliBANNmjQxKleubGzatMn46aefjDvuuMNo3759wvdHjhxR54Tvwd69e43XXnvN+PXXX439+/er8ypRooTxwAMPBOwc5s6dqzyqZ86cqTyae/Toofr8+PHj6vuOHTsaQ4YMSdj+559/NjJkyGCMHz/e2L17tzF8+HAjY8aMxu+//24EE96eF+7LFStWGPv27TNiY2ONdu3aGZkzZzZ27txpBAt4VsznBq+tCRMmqL/xbAGcD87L5K+//jKyZs1qDBo0SF2rqVOnGunTpzeWL19uhOo5TZw40Vi8eLGxZ88edc/Bkz5dunTG999/bwQLvXr1Up7ka9euNY4dO5awXL58OWGbUH2uPIVCNQ3o3LmzemicF7jIJ1wIERXaYA2reeWVV4z8+fOrF2SDBg2MP//80wgWzpw5o4QoBgm33nqr0bVrV4dBAgSn9RwPHTqkBGju3LnV+ZQqVUq98OLj4wN4FoYxZcoUo0iRIkamTJlUKMrGjRsdQoBw7azMnz/fuPPOO9X2CNlYtmyZEYx4c179+/dP2Bb320MPPWRs2bLFCCbMcBLnxTwP/I/zcv5NpUqV1HlhAGd9vkLxnMaMGWOULFlSDXjwHNWtW9dYvXq1EUyIzfk4v9tC+bnyBJZ+I4QQQnwE51QJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUIIIcRHUKgSQggh4hv+H/SXy5gKmyalAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "def make_moons_fallback(n_samples=100, noise=0.1):\n", - " n = n_samples // 2\n", - " t = np.linspace(0, np.pi, n)\n", - " # outer moon\n", + " n = n_samples // 2 # half the points per moon\n", + " t = np.linspace(0, np.pi, n) # angle along each crescent\n", + " # outer moon — upper arc (cos, sin)\n", " x1 = np.stack([np.cos(t), np.sin(t)], axis=1)\n", - " # inner moon, shifted\n", + " # inner moon — lower arc, shifted down so the two curves interleave\n", " x2 = np.stack([1 - np.cos(t), 1 - np.sin(t) - 0.5], axis=1)\n", - " X = np.concatenate([x1, x2], axis=0)\n", - " X += noise * np.random.randn(*X.shape)\n", - " y = np.array([0] * n + [1] * n)\n", + " X = np.concatenate([x1, x2], axis=0) # stack into one (100, 2) array\n", + " X += noise * np.random.randn(*X.shape) # jitter dots so it's not perfectly clean\n", + " y = np.array([0] * n + [1] * n) # label 0 = outer moon, 1 = inner moon\n", " return X, y\n", "\n", "try:\n", @@ -136,16 +423,16 @@ " X, y = make_moons(n_samples=100, noise=0.1, random_state=1337)\n", " print(\"using sklearn make_moons\")\n", "except Exception as e:\n", - " X, y = make_moons_fallback(100, noise=0.1)\n", + " X, y = make_moons_fallback(100, noise=0.1) # same shape, no sklearn needed\n", " print(\"sklearn not available, using NumPy fallback:\", type(e).__name__)\n", "\n", - "print(\"X shape:\", X.shape, \" y shape:\", y.shape)\n", + "print(\"X shape:\", X.shape, \" y shape:\", y.shape) # expect (100, 2) and (100,)\n", "print(\"first 3 points:\", X[:3].tolist())\n", "print(\"first 3 labels:\", y[:3].tolist())\n", "\n", "if HAS_PLT:\n", " plt.figure(figsize=(5, 4))\n", - " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n", + " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # red vs blue by class\n", " plt.title(\"two moons — can a tiny net separate the colors?\")\n", " plt.show()" ] @@ -170,32 +457,47 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "eea9bc18", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "d = Value(data=4.0)\n", + "d was made by op: '+'\n", + "d's parents: {Value(data=-6.0), Value(data=10.0)}\n" + ] + } + ], "source": [ "class Value:\n", " def __init__(self, data, _children=(), _op=''):\n", - " self.data = data\n", - " self._prev = set(_children)\n", - " self._op = _op\n", + " self.data = data # the number itself (forward pass result)\n", + " # HISTORY: who made me? _prev = parent Value objects; _op = the operation (+, *, ...)\n", + " # Forward pass builds a linked tree/DAG automatically — no separate \"tape\" step.\n", + " self._prev = set(_children) # set of input Values that fed into this one\n", + " self._op = _op # string tag: '+', '*', or '' for a leaf (input)\n", "\n", " def __repr__(self):\n", - " return f\"Value(data={self.data})\"\n", + " return f\"Value(data={self.data})\" # pretty print when you print(a)\n", "\n", " def __add__(self, other):\n", + " # New number = sum; we pass (self, other) so this node remembers both parents.\n", " return Value(self.data + other.data, (self, other), '+')\n", "\n", " def __mul__(self, other):\n", + " # Same idea: result remembers it came from self * other.\n", " return Value(self.data * other.data, (self, other), '*')\n", "\n", "\n", - "a = Value(2.0)\n", + "# --- tiny demo: build d = a*b + c and inspect the graph tags ---\n", + "a = Value(2.0) # leaf — no parents, _op is ''\n", "b = Value(-3.0)\n", "c = Value(10.0)\n", - "e = a * b\n", - "d = e + c\n", + "e = a * b # e.data = -6; e._prev = {a, b}; e._op = '*'\n", + "d = e + c # d.data = 4; d._prev = {e, c}; d._op = '+'\n", "print(\"d =\", d)\n", "print(\"d was made by op:\", repr(d._op))\n", "print(\"d's parents:\", d._prev)" @@ -210,22 +512,58 @@ "\n", "`d = a*b + c` is really a little tree: `d` points back to `e` and `c`; `e` points back to\n", "`a` and `b`. Let's print it as an indented tree so the structure is visible. (No graphviz\n", - "dependency — just recursion.)" + "dependency — just recursion.) Section **1.3** draws the same wiring as an interactive picture." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3a436597", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4.0000 (op=+)\n", + " -6.0000 (op=*)\n", + " 2.0000 (op=leaf)\n", + " -3.0000 (op=leaf)\n", + " 10.0000 (op=leaf)\n" + ] + } + ], + "source": [ + "def show(v, indent=0):\n", + " # print this node's value and which op created it ('leaf' = raw input)\n", + " print(\" \" * indent + f\"{v.data:.4f} (op={v._op or 'leaf'})\")\n", + " for child in v._prev: # follow parent links (deeper indent = further back in time)\n", + " show(child, indent + 1) # recurse until we hit leaves with no _prev\n", + "\n", + "show(d) # prints the tree: d (+) -> e (*) -> a, b and c" + ] + }, + { + "cell_type": "markdown", + "id": "f4e9da6b", + "metadata": {}, + "source": [ + "### 1.3 Interactive graph — see the wiring\n", + "\n", + "The text tree above is correct but flat. The same expression as a **picture**: green boxes are\n", + "raw inputs (leaves), blue boxes are operations. **Hover** a node to highlight its connections;\n", + "**click** to pin the detail panel. Arrows run from ingredients (bottom) up to the final score (top)." ] }, { "cell_type": "code", "execution_count": null, - "id": "3a436597", + "id": "f149d4da", "metadata": {}, "outputs": [], "source": [ - "def show(v, indent=0):\n", - " print(\" \" * indent + f\"{v.data:.4f} (op={v._op or 'leaf'})\")\n", - " for child in v._prev:\n", - " show(child, indent + 1)\n", - "\n", - "show(d)" + "# Re-use d = a*b + c from section 1.1 (run that cell first if you restarted the kernel)\n", + "show_graph(d, title=\"Forward pass: d = a × b + c\")" ] }, { @@ -252,12 +590,12 @@ "outputs": [], "source": [ "def f_out(a_, b_, c_):\n", - " return (a_ * b_) + c_ # plain floats\n", + " return (a_ * b_) + c_ # plain floats — no graph, just the math\n", "\n", - "h = 1e-6\n", - "base = f_out(2.0, -3.0, 10.0)\n", + "h = 1e-6 # tiny nudge size (like a small step on a hill)\n", + "base = f_out(2.0, -3.0, 10.0) # output before any nudge = 4.0\n", "\n", - "# how sensitive is the output to a?\n", + "# how sensitive is the output to a? slope ≈ (f(a+h) - f(a)) / h\n", "da = (f_out(2.0 + h, -3.0, 10.0) - base) / h\n", "# to b?\n", "db = (f_out(2.0, -3.0 + h, 10.0) - base) / h\n", @@ -265,9 +603,9 @@ "dc = (f_out(2.0, -3.0, 10.0 + h) - base) / h\n", "\n", "print(f\"output = {base}\")\n", - "print(f\"d(out)/da = {da:.4f} (equals b = -3)\")\n", - "print(f\"d(out)/db = {db:.4f} (equals a = 2)\")\n", - "print(f\"d(out)/dc = {dc:.4f} (equals 1)\")" + "print(f\"d(out)/da = {da:.4f} (equals b = -3)\") # derivative of a*b w.r.t. a is b\n", + "print(f\"d(out)/db = {db:.4f} (equals a = 2)\") # derivative w.r.t. b is a\n", + "print(f\"d(out)/dc = {dc:.4f} (equals 1)\") # derivative of +c w.r.t. c is 1" ] }, { @@ -280,17 +618,74 @@ "The nudge-and-measure trick is correct but slow: one re-run per input. Real networks have\n", "millions of inputs. The **chain rule** gets every sensitivity in *one* backward pass.\n", "\n", - "The rule, in words: *if a affects e, and e affects d, then a's effect on d is the product of\n", - "the two local effects.*\n", + "#### The one idea (relay race, not one long jump)\n", "\n", - "d(d)/d(a) = d(d)/d(e) * d(e)/d(a)\n", + "When one thing changes another thing, which then changes a third, you **multiply the two\n", + "\"sensitivity\" steps**.\n", + "\n", + "- **a** — starting point (e.g. how many projects you take)\n", + "- **e** — middle step (hours you work)\n", + "- **d** — final result (your pay)\n", + "\n", + "**d** does not feel **a** directly. It only feels **e**. So ask two smaller questions:\n", + "\n", + "1. If **a** moves a little, how much does **e** move? → `d(e)/d(a)`\n", + "2. If **e** moves a little, how much does **d** move? → `d(d)/d(e)`\n", + "\n", + "The full effect is those two multiplied:\n", + "\n", + "`d(d)/d(a) = d(d)/d(e) × d(e)/d(a)`\n", + "\n", + "> **One line to keep:** If **d** depends on **e**, and **e** depends on **a**, then \"how fast\n", + "> **d** responds to **a**\" = \"how fast **d** responds to **e**\" × \"how fast **e** responds to **a**.\"\n", + "\n", + "#### Units make it click (the middle cancels)\n", + "\n", + "Like converting units:\n", + "\n", + "- Pay per hour → dollars/hour\n", + "- Hours per project → hours/project\n", + "\n", + "Multiply: `(dollars/hour) × (hours/project) = dollars/project`\n", + "\n", + "The **hour** in the middle cancels. You get \"how pay changes when projects change.\" The formula\n", + "looks like fraction cancellation — that is the story, not magic algebra.\n", + "\n", + "#### Tiny wiggle picture\n", + "\n", + "Bump **a** by a tiny amount Δa.\n", + "\n", + "- **e** moves about `(d(e)/d(a)) × Δa`\n", + "- **d** moves about `(d(d)/d(e)) ×` that amount\n", + "\n", + "So total change in **d** is `(d(d)/d(e)) × (d(e)/d(a)) × Δa`. The chain rule is just: the\n", + "multiplier on Δa is the **product of the two hops**.\n", + "\n", + "#### Worked example\n", + "\n", + "Let `a = 2`, `e = a²`, `d = 3e`.\n", + "\n", + "- `d(e)/d(a) = 2a = 4` (e changes 4 units per unit of a)\n", + "- `d(d)/d(e) = 3` (d changes 3 units per unit of e)\n", + "\n", + "So `d(d)/d(a) = 3 × 4 = 12` — when **a** goes up by 1, **d** goes up by about 12.\n", + "\n", + "Check directly: `d = 3a²`, so `d(d)/d(a) = 6a = 12` at `a = 2`. Same answer.\n", + "\n", + "#### Why college notation felt confusing\n", + "\n", + "Textbooks often write `f(g(x))` and say \"derivative of outside × derivative of inside.\" That\n", + "is correct, but it hides the story: **indirect influence through a middle variable**. The relay\n", + "race picture is the same thing with names you can follow.\n", + "\n", + "#### Local rules (what each station does)\n", "\n", "Each operation knows its own **local** derivative:\n", "\n", "- for `+`: output changes 1-for-1 with each input, so the gradient just **passes through**.\n", "- for `*`: each input's local derivative is **the other input**.\n", "\n", - "Backprop = start with gradient 1.0 at the output, then walk backward multiplying by each\n", + "**Backprop** = start with gradient 1.0 at the output, then walk backward multiplying by each\n", "local derivative." ] }, @@ -301,6 +696,31 @@ "source": [ "### 4.1 Adding gradients and local backward rules\n", "\n", + "#### What is autograd?\n", + "\n", + "**Autograd** = **automatic differentiation** — the computer figures out gradients for you\n", + "after you write the forward math.\n", + "\n", + "Real-life picture: like a GPS that logged every turn on your drive. When you ask \"how do I get\n", + "back?\", it does not guess — it rewinds those turns and tells you how each step affected the\n", + "final destination.\n", + "\n", + "**Is it a library?** Two meanings:\n", + "\n", + "| Meaning | What it is | In this notebook |\n", + "|---|---|---|\n", + "| **The idea** | Any system that records ops, then runs `backward()` to fill `.grad` | Our homemade **micrograd** `Value` class |\n", + "| **A PyPI package** | A library literally named [`autograd`](https://github.com/HIPS/autograd) | We do **not** use it — we build from scratch |\n", + "\n", + "Later, **PyTorch** ships its own autograd (`loss.backward()` in notebooks `01` and `06`). Same\n", + "idea, industrial scale.\n", + "\n", + "When code says \"analytic gradients via autograd,\" it means: run forward through `Value` objects,\n", + "call `.backward()` once, read `.grad` — instead of nudging each input by hand with\n", + "`numeric_grad`.\n", + "\n", + "---\n", + "\n", "Now the full `Value`. Each operation also defines a `_backward()` that says how to push the\n", "output's gradient onto its inputs. Note we **accumulate** (`+=`) gradients, because one value\n", "can feed into several places.\n", @@ -308,6 +728,31 @@ "We add `+`, `*`, `**` (power), `tanh`, and `exp` — enough to build a neural net. We also add\n", "convenience operators (`-`, `/`, right-hand versions) so expressions read naturally.\n", "\n", + "#### `_op` and `_prev` — the tags on every box\n", + "\n", + "Every `Value` stores two pieces of history in `__init__`:\n", + "\n", + "- **`_op`** — *what happened at this station?* A stamp on the box: `'+'` (mixed), `'*'` (scaled),\n", + " `'tanh'` (squashed), or `''` for a raw leaf input (flour, sugar — nothing done yet).\n", + "- **`_prev`** — *who gave me ingredients?* The set of parent `Value`s that fed into this one.\n", + " Like \"received from\" arrows on the tag.\n", + "\n", + "**Cookie-factory picture for `d = a * b + c`:**\n", + "\n", + "1. `e = a * b` → `e._op = '*'`, `e._prev = {a, b}` — \"made by multiplying; parents are a and b.\"\n", + "2. `d = e + c` → `d._op = '+'`, `d._prev = {e, c}` — \"made by adding; parents are e and c.\"\n", + "\n", + "A normal float is a score on a napkin: **4**. You cannot tell how you got it. A `Value` is the\n", + "same score **on a tagged box** — you can always walk back.\n", + "\n", + "**Why this matters:** forward pass moves ingredients down the line. Backprop starts at the final\n", + "score and walks **backward** along `_prev`. At each stop, `_op` tells you which local rule to\n", + "apply (add passes blame through; multiply splits blame using the other input). Without these\n", + "tags there is no map from the final number back to the knobs you need to turn.\n", + "\n", + "> **One line:** `_op` = what was done here; `_prev` = who fed into it. Together they are the\n", + "> breadcrumb trail from output back to inputs.\n", + "\n", "**-> Training:** `tanh` is the neuron's \"squashing\" activation; its local derivative is\n", "`1 - tanh(x)^2`." ] @@ -321,90 +766,99 @@ "source": [ "class Value:\n", " def __init__(self, data, _children=(), _op='', label=''):\n", - " self.data = data\n", - " self.grad = 0.0\n", - " self._backward = lambda: None\n", - " self._prev = set(_children)\n", - " self._op = _op\n", - " self.label = label\n", + " self.data = data # the actual number (forward value)\n", + " self.grad = 0.0 # how much the final loss changes if THIS number nudges +1\n", + " self._backward = lambda: None # filled in by each op: \"push grad to my parents\"\n", + " self._prev = set(_children) # graph edges pointing backward to parents\n", + " self._op = _op # which op created this node ('+', '*', 'tanh', ...)\n", + " self.label = label # optional name for debugging / graph drawing\n", "\n", " def __repr__(self):\n", " return f\"Value(data={self.data:.4f}, grad={self.grad:.4f})\"\n", "\n", " def __add__(self, other):\n", - " other = other if isinstance(other, Value) else Value(other)\n", - " out = Value(self.data + other.data, (self, other), '+')\n", + " other = other if isinstance(other, Value) else Value(other) # allow Value + 3\n", + " out = Value(self.data + other.data, (self, other), '+') # forward: add numbers\n", " def _backward():\n", + " # local rule for +: d(a+b)/da = 1, d(a+b)/db = 1 → grad flows 1:1 to both\n", " self.grad += 1.0 * out.grad\n", " other.grad += 1.0 * out.grad\n", - " out._backward = _backward\n", + " out._backward = _backward # closure captures self/other/out — runs later in backward()\n", " return out\n", "\n", " def __mul__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", - " out = Value(self.data * other.data, (self, other), '*')\n", + " out = Value(self.data * other.data, (self, other), '*') # forward: multiply\n", " def _backward():\n", + " # local rule for *: d(a*b)/da = b, d(a*b)/db = a (chain rule: × out.grad)\n", " self.grad += other.data * out.grad\n", " other.grad += self.data * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def __pow__(self, other):\n", - " assert isinstance(other, (int, float)), \"only int/float powers\"\n", - " out = Value(self.data ** other, (self,), f'**{other}')\n", + " assert isinstance(other, (int, float)), \"only int/float powers\" # exponent is constant\n", + " out = Value(self.data ** other, (self,), f'**{other}') # forward: x^n\n", " def _backward():\n", + " # d(x^n)/dx = n * x^(n-1), then chain rule × upstream grad\n", " self.grad += other * (self.data ** (other - 1)) * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def tanh(self):\n", " x = self.data\n", - " t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1)\n", + " t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1) # forward tanh (squash to -1..1)\n", " out = Value(t, (self,), 'tanh')\n", " def _backward():\n", + " # d tanh(x)/dx = 1 - tanh(x)^2 (use saved t, not recompute from self.data)\n", " self.grad += (1 - t ** 2) * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", " def exp(self):\n", - " out = Value(math.exp(self.data), (self,), 'exp')\n", + " out = Value(math.exp(self.data), (self,), 'exp') # forward: e^x\n", " def _backward():\n", + " # d e^x / dx = e^x — out.data IS e^x at forward time\n", " self.grad += out.data * out.grad\n", " out._backward = _backward\n", " return out\n", "\n", - " # convenience\n", + " # --- convenience ops so expressions like 3 + a or a / b work naturally ---\n", " def __neg__(self):\n", - " return self * -1\n", + " return self * -1 # unary minus via multiply by -1\n", "\n", " def __sub__(self, other):\n", - " return self + (-other if isinstance(other, Value) else Value(-other))\n", + " return self + (-other if isinstance(other, Value) else Value(-other)) # a - b = a + (-b)\n", "\n", " def __radd__(self, other):\n", - " return self + other\n", + " return self + other # handles 3 + a (Python calls __radd__ on a)\n", "\n", " def __rmul__(self, other):\n", - " return self * other\n", + " return self * other # handles 3 * a\n", "\n", " def __truediv__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", - " return self * other ** -1\n", + " return self * other ** -1 # a / b = a * b^(-1)\n", "\n", " def backward(self):\n", - " # build topological order so children come before parents\n", + " # Step 1: list every node in the graph, parents before children (topological sort).\n", + " # We walk _prev links; reversed(topo) then visits output → inputs.\n", " topo = []\n", " visited = set()\n", " def build_topo(v):\n", " if v not in visited:\n", " visited.add(v)\n", - " for child in v._prev:\n", + " for child in v._prev: # visit all ancestors first\n", " build_topo(child)\n", - " topo.append(v)\n", - " build_topo(self)\n", - " # seed the output gradient, then sweep backward\n", + " topo.append(v) # post-order: children before parent\n", + " build_topo(self) # start from the output node (e.g. loss)\n", + "\n", + " # Step 2: seed — \"if output goes up 1, output goes up 1\" (dL/dL = 1).\n", " self.grad = 1.0\n", - " for node in reversed(topo):\n", - " node._backward()\n", + "\n", + " # Step 3: reverse sweep; each node runs its local _backward (chain rule × parents).\n", + " for node in reversed(topo): # output first, leaves last\n", + " node._backward() # each op pushes blame to its _prev parents\n", "\n", "\n", "print(\"full Value class ready\")" @@ -429,18 +883,37 @@ "metadata": {}, "outputs": [], "source": [ - "a = Value(2.0, label='a')\n", + "a = Value(2.0, label='a') # leaf inputs — knobs with no parents yet\n", "b = Value(-3.0, label='b')\n", "c = Value(10.0, label='c')\n", - "e = a * b\n", - "d = e + c\n", + "e = a * b # builds multiply node; graph remembers a, b\n", + "d = e + c # builds add node; graph remembers e, c\n", "\n", - "d.backward()\n", + "d.backward() # one backward pass fills .grad on every node in the graph\n", "\n", - "print(\"d =\", d.data)\n", - "print(\"a.grad =\", a.grad, \" (expect -3)\")\n", - "print(\"b.grad =\", b.grad, \" (expect 2)\")\n", - "print(\"c.grad =\", c.grad, \" (expect 1)\")" + "print(\"d =\", d.data) # forward answer still 4.0\n", + "print(\"a.grad =\", a.grad, \" (expect -3)\") # ∂d/∂a = b\n", + "print(\"b.grad =\", b.grad, \" (expect 2)\") # ∂d/∂b = a\n", + "print(\"c.grad =\", c.grad, \" (expect 1)\") # ∂d/∂c = 1" + ] + }, + { + "cell_type": "markdown", + "id": "0083fec0", + "metadata": {}, + "source": [ + "Same graph **after backprop** — orange gradient numbers show how much each node would nudge the\n", + "final output. Hover `a` and notice its grad equals `b` (−3): the multiply rule in action." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c14c82ca", + "metadata": {}, + "outputs": [], + "source": [ + "show_graph(d, title=\"After backward() — gradients filled in\")" ] }, { @@ -451,7 +924,42 @@ "### 4.3 Trust but verify — numeric gradient check\n", "\n", "A good habit: confirm the analytic gradient (from `backward()`) matches the slow\n", - "nudge-and-measure gradient. If they agree, the engine is correct." + "nudge-and-measure gradient. If they agree, the engine is correct.\n", + "\n", + "**Real-life picture:** change one recipe ingredient by a tiny pinch, cook again, and see\n", + "how much the final score moves. That ratio — *change in score ÷ size of pinch* — is the\n", + "gradient for that ingredient. Repeat for each ingredient, one at a time. Slow, but it is\n", + "the definition of \"sensitivity.\"\n", + "\n", + "**Two ways to measure the same hill**\n", + "\n", + "| | Nudge-and-measure (numeric) | `backward()` (analytic) |\n", + "|---|---|---|\n", + "| Idea | Tiny change in one input → re-run forward | Chain rule walks the graph once |\n", + "| Cost | One forward pass **per input** | One backward pass for **all** inputs |\n", + "| Result | slope ≈ `(f(x+h) − f(x)) / h` | `.grad` on each leaf `Value` |\n", + "\n", + "**Outline — nudge-and-measure (what `numeric_grad` does)**\n", + "\n", + "1. Run the same math as the graph, but on plain floats (no `Value` objects).\n", + "2. Pick a tiny step `h` (e.g. `1e-6`).\n", + "3. Save the baseline output at the current inputs.\n", + "4. For **each** input `i`, one at a time:\n", + " - copy the inputs;\n", + " - add `h` only to input `i`;\n", + " - re-run forward;\n", + " - slope = (new output − baseline) / `h`.\n", + "5. That slope is the numeric gradient for input `i`.\n", + "\n", + "**Outline — analytic check (what we trust after verifying)**\n", + "\n", + "1. Build the expression with `Value` objects (the graph remembers parents via `_prev`).\n", + "2. Call `.backward()` once on the final output.\n", + "3. Read `.grad` on each leaf — chain rule applied in one reverse sweep.\n", + "\n", + "**Compare:** print analytic vs numeric side by side; they **match** when\n", + "`|analytic − numeric| < 1e-4`. Same hill, two rulers — if they agree, the engine's local\n", + "`_backward()` rules are wired correctly." ] }, { @@ -462,19 +970,19 @@ "outputs": [], "source": [ "def numeric_grad(f, inputs, i, h=1e-6):\n", - " base = f(inputs)\n", - " bumped = list(inputs)\n", - " bumped[i] += h\n", - " return (f(bumped) - base) / h\n", + " base = f(inputs) # output at current inputs\n", + " bumped = list(inputs) # copy so we don't mutate the original list\n", + " bumped[i] += h # nudge only input i by a tiny amount\n", + " return (f(bumped) - base) / h # slope = rise / run\n", "\n", - "f = lambda v: (v[0] * v[1]) + v[2]\n", + "f = lambda v: (v[0] * v[1]) + v[2] # same math as a*b+c but on plain floats\n", "inputs = [2.0, -3.0, 10.0]\n", "\n", "a = Value(2.0); b = Value(-3.0); c = Value(10.0)\n", - "(a * b + c).backward()\n", + "(a * b + c).backward() # analytic gradients via autograd\n", "\n", "for i, (name, val) in enumerate(zip(\"abc\", (a, b, c))):\n", - " ng = numeric_grad(f, inputs, i)\n", + " ng = numeric_grad(f, inputs, i) # slow finite-difference gradient for input i\n", " print(f\"{name}: analytic={val.grad:+.4f} numeric={ng:+.4f} match={abs(val.grad-ng)<1e-4}\")" ] }, @@ -496,12 +1004,12 @@ "outputs": [], "source": [ "# Your turn 4 — fill in and run\n", - "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n", - "g = (a * b + c).tanh()\n", - "g.backward()\n", + "a = Value(0.5); b = Value(2.0); c = Value(-1.0) # inputs\n", + "g = (a * b + c).tanh() # forward: multiply, add, squash\n", + "g.backward() # backprop through tanh, +, *\n", "\n", - "f = lambda v: math.tanh(v[0]*v[1] + v[2])\n", - "ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0)\n", + "f = lambda v: math.tanh(v[0]*v[1] + v[2]) # same formula on plain floats\n", + "ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0) # numeric ∂g/∂a (nudge index 0)\n", "print(f\"a.grad analytic = {a.grad:+.4f} numeric = {ng_a:+.4f}\")" ] }, @@ -512,13 +1020,126 @@ "source": [ "### 5.1 Neuron -> Layer -> MLP, all from `Value`\n", "\n", - "A **neuron** computes `tanh(w . x + b)` — a weighted vote of its inputs, squashed to (-1, 1).\n", - "A **layer** is a list of neurons. An **MLP** (multi-layer perceptron) is a stack of layers.\n", + "Everything below is still built from plain `Value` objects — the same tagged boxes and\n", + "`backward()` trail from section 4. Here we stack them into a small **neural network**.\n", "\n", - "Real-life picture: each neuron is a tiny committee member weighing the evidence; layers stack\n", - "committees so later ones judge the opinions of earlier ones.\n", + "#### One story to hold in your head: renting an apartment\n", "\n", - "**-> Training:** `parameters()` returns every weight and bias — exactly the knobs we nudge." + "You apply with two numbers on the form: **monthly income** and **credit score**.\n", + "The landlord wants one answer: *likely good tenant* vs *likely risky*.\n", + "\n", + "**One neuron = one reviewer on the panel**\n", + "\n", + "A single neuron looks at your whole application and gives **one opinion**, squashed to a\n", + "number between -1 and +1:\n", + "\n", + "`tanh(w · x + b)` means:\n", + "\n", + "- **`w`** — how much this reviewer cares about each field (income vs credit). Big weight =\n", + " \"this matters a lot to me.\"\n", + "- **`b`** — this reviewer's baseline strictness before seeing you (always picky, or always\n", + " lenient).\n", + "- **`w · x + b`** — weighted score: multiply each input by its importance, add bias.\n", + "- **`tanh(...)`** — squash the score so it cannot run away to infinity; strong yes or strong no,\n", + " but bounded (like capping enthusiasm at \"very pro\" or \"very against\").\n", + "\n", + "So one neuron is not magic — it is **one weighted vote**, softened at the edges.\n", + "\n", + "**One layer = a whole panel, all reading the same application**\n", + "\n", + "A **layer** is several neurons in **parallel**. Each sees the **same** inputs `x`, but each\n", + "has its **own** weights and bias — different reviewers, different priorities:\n", + "\n", + "- Reviewer A might care mostly about income.\n", + "- Reviewer B might care mostly about credit.\n", + "- Reviewer C might look for a balance of both.\n", + "\n", + "The layer's output is a **list of opinions** (one per neuron). Eight neurons → eight numbers\n", + "describing the application from eight angles.\n", + "\n", + "**An MLP = panels stacked in stages**\n", + "\n", + "An **MLP** (multi-layer perceptron) feeds one layer's opinions into the next:\n", + "\n", + "1. **First hidden layer** — junior reviewers turn raw form fields into rough themes (\"cash-flow\n", + " looks fine\", \"credit is shaky\", …).\n", + "2. **Second hidden layer** — senior reviewers debate those themes, not the raw form.\n", + "3. **Output layer** — one final judge reads the debate and emits a single score (here: one\n", + " neuron → one number for the two-moons task).\n", + "\n", + "That is why the code uses shapes like `MLP(2, [8, 8, 1)`:\n", + "\n", + "| Piece | In the story | In code |\n", + "|---|---|---|\n", + "| 2 inputs | income + credit (here: x, y on the plot) | `nin = 2` |\n", + "| 8 + 8 hidden | two panels of 8 reviewers | two `Layer`s with `nout=8` |\n", + "| 1 output | final hire / no-hire score | last layer with `nout=1` |\n", + "\n", + "Data flows **forward** only: form → junior panel → senior panel → final judge.\n", + "Every multiply, add, and `tanh` is still a `Value` node, so `loss.backward()` can later\n", + "blame the right weight when the answer was wrong.\n", + "\n", + "#### The three building blocks (code ↔ picture)\n", + "\n", + "| Class | What it is | Real-life |\n", + "|---|---|---|\n", + "| **`Neuron`** | one weighted vote + squash | one reviewer |\n", + "| **`Layer`** | list of neurons, same inputs | one panel sitting together |\n", + "| **`MLP`** | layers wired in sequence | whole hiring process, stage by stage |\n", + "\n", + "**-> Training:** `parameters()` collects every weight and bias in the network — all the knobs\n", + "on all reviewers' score sheets. Section 6 nudges those knobs downhill until the final judge\n", + "gets the moons right." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "93e51095", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Interactive companion to section 5.1 — drag the inputs, hover lines/dots, press play.\n", + "# Shows the same 2 -> 8 -> 8 -> 1 reviewer-panel network as the code below.\n", + "import base64\n", + "from pathlib import Path\n", + "from IPython.display import IFrame, display\n", + "\n", + "html_path = Path(\"mlp_playground.html\")\n", + "if not html_path.exists():\n", + " html_path = Path.cwd() / \"mlp_playground.html\"\n", + "\n", + "if not html_path.exists():\n", + " raise FileNotFoundError(\n", + " \"mlp_playground.html not found. Keep it next to this notebook, then re-run.\"\n", + " )\n", + "\n", + "# Notebook UIs often cannot load local files via IFrame src=\"file.html\".\n", + "# Pack the page into a data URI so the browser runs it inline with scripts intact.\n", + "b64 = base64.b64encode(html_path.read_bytes()).decode(\"ascii\")\n", + "display(IFrame(src=f\"data:text/html;base64,{b64}\", width=\"100%\", height=640))" ] }, { @@ -530,44 +1151,47 @@ "source": [ "class Neuron:\n", " def __init__(self, nin):\n", + " # one weight per input + one bias — all are Value objects (trainable knobs)\n", " self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]\n", " self.b = Value(random.uniform(-1, 1))\n", "\n", " def __call__(self, x):\n", + " # weighted sum: w1*x1 + w2*x2 + ... + b (each * and + builds the graph)\n", " act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)\n", - " return act.tanh()\n", + " return act.tanh() # squash activation to (-1, 1)\n", "\n", " def parameters(self):\n", - " return self.w + [self.b]\n", + " return self.w + [self.b] # every knob this neuron owns\n", "\n", "\n", "class Layer:\n", " def __init__(self, nin, nout):\n", - " self.neurons = [Neuron(nin) for _ in range(nout)]\n", + " self.neurons = [Neuron(nin) for _ in range(nout)] # nout parallel neurons\n", "\n", " def __call__(self, x):\n", - " outs = [n(x) for n in self.neurons]\n", - " return outs[0] if len(outs) == 1 else outs\n", + " outs = [n(x) for n in self.neurons] # each neuron sees the same input vector x\n", + " return outs[0] if len(outs) == 1 else outs # scalar if one neuron, else list\n", "\n", " def parameters(self):\n", - " return [p for n in self.neurons for p in n.parameters()]\n", + " return [p for n in self.neurons for p in n.parameters()] # flatten all weights\n", "\n", "\n", "class MLP:\n", " def __init__(self, nin, nouts):\n", - " sizes = [nin] + nouts\n", + " sizes = [nin] + nouts # e.g. [2, 8, 8, 1] for 2→8→8→1\n", + " # layer i connects sizes[i] inputs to sizes[i+1] outputs\n", " self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(nouts))]\n", "\n", " def __call__(self, x):\n", - " for layer in self.layers:\n", + " for layer in self.layers: # data flows forward through each layer\n", " x = layer(x)\n", " return x\n", "\n", " def parameters(self):\n", - " return [p for layer in self.layers for p in layer.parameters()]\n", + " return [p for layer in self.layers for p in layer.parameters()] # all trainable Values\n", "\n", "\n", - "net = MLP(2, [8, 8, 1]) # 2 inputs -> 8 -> 8 -> 1 output\n", + "net = MLP(2, [8, 8, 1]) # 2 inputs -> 8 hidden -> 8 hidden -> 1 output\n", "print(\"parameter count:\", len(net.parameters()))\n", "print(\"one prediction:\", net([Value(0.5), Value(-0.2)]))" ] @@ -596,30 +1220,32 @@ "metadata": {}, "outputs": [], "source": [ - "net = MLP(2, [8, 8, 1])\n", + "net = MLP(2, [8, 8, 1]) # fresh network for training\n", "\n", - "# map labels {0,1} -> {-1,+1} to match tanh output range\n", + "# map labels {0,1} -> {-1,+1} to match tanh output range (-1 to +1)\n", "ys = [1.0 if yi == 1 else -1.0 for yi in y]\n", + "# wrap each (x, y) coordinate pair as Value objects so forward builds a graph\n", "Xv = [[Value(float(xi[0])), Value(float(xi[1]))] for xi in X]\n", "\n", - "lr = 0.1\n", - "for step in range(60):\n", - " # forward + MSE loss\n", + "lr = 0.1 # learning rate — how big each knob turn per step\n", + "for step in range(60): # 60 gradient-descent steps (small count: micrograd is slow)\n", + "\n", + " # --- 1. FORWARD: predict every point, build one big graph ending in loss ---\n", " preds = [net(xrow) for xrow in Xv]\n", - " loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0))\n", - " loss = loss * (1.0 / len(ys))\n", + " loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0)) # sum of squared errors\n", + " loss = loss * (1.0 / len(ys)) # average MSE so scale doesn't grow with dataset size\n", "\n", - " # backward (reset grads first)\n", + " # --- 2. BACKWARD: zero old grads, then propagate blame from loss to every weight ---\n", " for p in net.parameters():\n", - " p.grad = 0.0\n", - " loss.backward()\n", + " p.grad = 0.0 # must reset — backward() accumulates with +=\n", + " loss.backward() # fills p.grad for every parameter\n", "\n", - " # update\n", + " # --- 3. UPDATE: walk downhill — turn each knob opposite its gradient ---\n", " for p in net.parameters():\n", - " p.data -= lr * p.grad\n", + " p.data -= lr * p.grad # new value = old value - lr * (how loss rises if knob rises)\n", "\n", " if step % 10 == 0 or step == 59:\n", - " # accuracy\n", + " # sign match: prediction > 0 same as target > 0 counts as correct\n", " acc = sum((1 if (p.data > 0) == (yt > 0) else 0) for p, yt in zip(preds, ys)) / len(ys)\n", " print(f\"step {step:3d} loss {loss.data:.4f} acc {acc:.2%}\")" ] @@ -645,20 +1271,21 @@ "outputs": [], "source": [ "if HAS_PLT:\n", + " # pad plot area a little beyond the data points\n", " xmin, xmax = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " ymin, ymax = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", - " xs = np.linspace(xmin, xmax, 30)\n", - " ys_grid = np.linspace(ymin, ymax, 30)\n", + " xs = np.linspace(xmin, xmax, 30) # 30 grid columns\n", + " ys_grid = np.linspace(ymin, ymax, 30) # 30 grid rows\n", "\n", - " Z = np.zeros((len(ys_grid), len(xs)))\n", + " Z = np.zeros((len(ys_grid), len(xs))) # store net output at each grid cell\n", " for i, gy in enumerate(ys_grid):\n", " for j, gx in enumerate(xs):\n", - " out = net([Value(float(gx)), Value(float(gy))])\n", - " Z[i, j] = out.data\n", + " out = net([Value(float(gx)), Value(float(gy))]) # forward only (no training)\n", + " Z[i, j] = out.data # >0 one side of boundary, <0 other side\n", "\n", " plt.figure(figsize=(5, 4))\n", - " plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3)\n", - " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\")\n", + " plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3) # colored regions\n", + " plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # training dots on top\n", " plt.title(\"decision boundary learned by the micrograd MLP\")\n", " plt.show()\n", "else:\n", @@ -704,18 +1331,19 @@ "try:\n", " import torch\n", "\n", + " # requires_grad=True tells PyTorch to track history (like our Value graph)\n", " at = torch.tensor([0.5], requires_grad=True)\n", " bt = torch.tensor([2.0], requires_grad=True)\n", " ct = torch.tensor([-1.0], requires_grad=True)\n", "\n", - " gt = torch.tanh(at * bt + ct)\n", - " gt.backward()\n", + " gt = torch.tanh(at * bt + ct) # same expression as micrograd Your turn 4\n", + " gt.backward() # PyTorch autograd — same idea as Value.backward()\n", " print(\"PyTorch: a.grad =\", at.grad.item())\n", "except OSError as e:\n", " print(\"PyTorch could not load in this environment:\", e)\n", " print(\"(That is fine here - the micrograd result below is the point.)\")\n", "\n", - "# our engine, same expression\n", + "# our engine, same expression — should match PyTorch's a.grad\n", "a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n", "(a * b + c).tanh().backward()\n", "print(\"micrograd: a.grad =\", a.grad)" @@ -759,10 +1387,18 @@ "name": "python3" }, "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +} diff --git a/_build_07.py b/_build_07.py deleted file mode 100644 index f017d16..0000000 --- a/_build_07.py +++ /dev/null @@ -1,276 +0,0 @@ -import json -import uuid -from pathlib import Path - -cells = [] - - -def _src(text): - lines = text.split("\n") - out = [line + "\n" if i < len(lines) - 1 else line for i, line in enumerate(lines)] - if out and out[-1] == "": - out.pop() - return out - - -def md(text): - cells.append({"cell_type": "markdown", "id": uuid.uuid4().hex[:8], - "metadata": {}, "source": _src(text)}) - - -def code(text): - cells.append({"cell_type": "code", "execution_count": None, "id": uuid.uuid4().hex[:8], - "metadata": {}, "outputs": [], "source": _src(text)}) - - -# ---------------------------------------------------------------------------- -md("""# The GPT tokenizer — Byte Pair Encoding from scratch - -Turn raw text into tokens the model can read — **concept -> code -> Your turn** each step. - -This is notebook **07**, the final piece of the pipeline. Every notebook so far fed the model -either single characters (Shakespeare) or single letters (names). Real GPTs use **subword** -tokens built by **Byte Pair Encoding (BPE)**. Here we build a BPE tokenizer from scratch, then -see why tokenization quietly causes many famous LLM quirks. Follows Karpathy's -*"Let's build the GPT Tokenizer"*.""") - -md("""## Prologue — in plain English - -A neural net only eats numbers, so text must be chopped into pieces (**tokens**) and each piece -mapped to an id. Two extremes: - -- **One token per character** (what we did): tiny vocabulary, but sequences are very long and - the model must relearn common spellings everywhere. -- **One token per word**: short sequences, but the vocabulary explodes and unseen words break it. - -**BPE** is the practical middle: start from raw bytes, then repeatedly **merge the most common -adjacent pair** into a new token. Frequent chunks like `the`, `ing`, or ` and` become single -tokens; rare text still survives as smaller pieces. - -Real-life picture: inventing shorthand. You start writing letter by letter, notice you keep -writing "t-h-e", so you invent one squiggle for "the." Then you notice " a-n-d" and give it a -squiggle too. BPE does this automatically, keeping the most useful shorthands.""") - -# ---------------------------------------------------------------------------- -md("""### 1.1 Text -> UTF-8 bytes (the 256 starting tokens) - -Computers already store text as **bytes** (numbers 0-255) via UTF-8. So our alphabet starts -with exactly 256 tokens — every possible byte. Plain English letters are one byte each; -accented or non-Latin characters take several bytes (which is why they cost more tokens later).""") - -code("""sample = "Hello world! tokenization is fun." -the_bytes = sample.encode("utf-8") -print("text :", sample) -print("bytes :", list(the_bytes)) -print("length :", len(the_bytes), "bytes ->", len(the_bytes), "starting tokens") - -multi = "café na\u00efve" # accented letters take >1 byte each -print("\\nnon-ASCII example:", multi) -print("bytes:", list(multi.encode("utf-8")), "(note: more bytes than characters)")""") - -# ---------------------------------------------------------------------------- -md("""### 2.1 Two tiny helpers: count pairs, and merge a pair - -- `get_stats` counts how often each adjacent pair appears. -- `merge` replaces every occurrence of a chosen pair with a single new token id. - -That is the entire mechanism of BPE.""") - -code("""def get_stats(ids): - counts = {} - for pair in zip(ids, ids[1:]): - counts[pair] = counts.get(pair, 0) + 1 - return counts - -def merge(ids, pair, idx): - new_ids = [] - i = 0 - while i < len(ids): - if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]: - new_ids.append(idx) - i += 2 - else: - new_ids.append(ids[i]) - i += 1 - return new_ids - -# demo on a toy list: merge the most common pair -demo = [1, 2, 3, 1, 2, 3, 1, 2] -st = get_stats(demo) -top = max(st, key=st.get) -print("counts:", st) -print("most common pair:", top, "->", st[top], "times") -print("after merging", top, "into 99:", merge(demo, top, 99))""") - -md("""### 2.2 Train BPE — learn the merges - -We train on a few KB of real text (Shakespeare). Starting from raw bytes, we repeatedly find -the most common pair and merge it, recording each merge. After `num_merges` rounds we have a -vocabulary of `256 + num_merges` tokens.""") - -code("""train_text = open("input.txt", "r", encoding="utf-8").read()[:20000] -ids = list(train_text.encode("utf-8")) -print("training bytes:", len(ids)) - -num_merges = 60 -merges = {} # (a, b) -> new_id -work = list(ids) -for i in range(num_merges): - stats = get_stats(work) - pair = max(stats, key=stats.get) - idx = 256 + i - work = merge(work, pair, idx) - merges[pair] = idx - -print("merges learned:", len(merges)) -print("tokens after training:", len(work), "(was", len(ids), "bytes)") -print(f"compression: {len(ids) / len(work):.2f}x")""") - -# ---------------------------------------------------------------------------- -md("""### 3.1 encode and decode - -- **decode**: turn token ids back into text. We build each token's byte string by stitching the - merges back together. -- **encode**: turn new text into token ids by applying the learned merges, always doing the - *earliest-learned* applicable merge first.""") - -code("""# build the id -> bytes table -vocab = {idx: bytes([idx]) for idx in range(256)} -for (p0, p1), idx in merges.items(): - vocab[idx] = vocab[p0] + vocab[p1] - -def decode(ids): - data = b"".join(vocab[idx] for idx in ids) - return data.decode("utf-8", errors="replace") - -def encode(text): - tokens = list(text.encode("utf-8")) - while len(tokens) >= 2: - stats = get_stats(tokens) - # pick the pair whose merge was learned earliest - pair = min(stats, key=lambda p: merges.get(p, float("inf"))) - if pair not in merges: - break - tokens = merge(tokens, pair, merges[pair]) - return tokens - -trial = "the king and the queen" -enc = encode(trial) -print("text :", trial) -print("tokens:", enc) -print("count :", len(enc), "tokens for", len(trial), "characters") -print("decode round-trip ok:", decode(enc) == trial)""") - -md("""### 3.2 See what the learned tokens are - -The merges discovered the common chunks of English on their own. Let's print a few learned -tokens (ids >= 256) as the text they stand for.""") - -code("""print("a few learned tokens:") -for idx in range(256, 256 + min(20, num_merges)): - piece = vocab[idx].decode("utf-8", errors="replace") - print(f" id {idx}: {piece!r}")""") - -# ---------------------------------------------------------------------------- -md("""### 4.1 Regex pre-splitting (what real GPTs add) - -A subtlety: plain BPE might merge across spaces and punctuation (e.g. glue `dog.` or `the the` -into weird tokens). GPT-2 first **splits** the text into sensible chunks with a regex (words, -runs of spaces, punctuation), then runs BPE **inside** each chunk only. - -Real GPT uses the `regex` library with Unicode classes; here is a simplified version with the -standard `re` module to show the idea.""") - -code("""import re - -# simplified GPT-2-style splitter (real one uses the `regex` lib with \\p{L}, \\p{N}) -pat = re.compile(r"'s|'t|'re|'ve|'m|'ll|'d| ?\\w+| ?[^\\s\\w]+|\\s+") - -demo_text = "Hello, world! It's 2026 already." -chunks = re.findall(pat, demo_text) -print("split into chunks (BPE then runs inside each one):") -print(chunks)""") - -md("""### 4.2 Special tokens - -Real tokenizers also reserve **special tokens** that are not learned from text but inserted to -mark structure, for example `<|endoftext|>` between documents, or chat markers like -`<|im_start|>` / `<|im_end|>`. They get their own ids above the learned vocabulary. - -Real-life picture: punctuation the model never "spells" — single reserved symbols that mean -"new document starts here" or "the user is speaking now." - -``` -<|endoftext|> -> id 50256 in GPT-2 (separates documents) -<|im_start|> -> chat role marker in chat models -```""") - -# ---------------------------------------------------------------------------- -md("""### 5.1 Why tokenization causes LLM quirks - -Many puzzling LLM behaviours trace back to how text is tokenized. We measure a few. - -- **Spelling / reversing words is hard**: a word is often a *single* token, so the model does - not "see" its letters. -- **Arithmetic is fragile**: numbers split into arbitrary chunks, not clean digits. -- **Non-English costs more**: it falls back to many byte-tokens, so the same meaning uses more - tokens (and more money / context).""") - -code("""def n_tokens(s): - return len(encode(s)) - -print("spelling: 'extraordinary' ->", n_tokens("extraordinary"), "token(s)") -print(" the model sees a chunk, not the 13 letters\\n") - -print("numbers : '1234567' ->", n_tokens("1234567"), "tokens (split oddly, not per-digit)") -print(" '127 + 677' ->", n_tokens("127 + 677"), "tokens\\n") - -eng = "hello how are you" -non = "\u4f60\u597d\u4f60\u597d\u4f60\u597d" # non-English of similar visible length -print("non-English costs more tokens for similar content:") -print(" english :", repr(eng), "->", n_tokens(eng), "tokens") -print(" non-latin :", repr(non), "->", n_tokens(non), "tokens")""") - -md("""**Your turn 7** — Raise `num_merges` (e.g. to 300) in section 2.2, retrain, and re-check the -compression ratio and the token counts in section 5.1. More merges = better compression, but a -bigger vocabulary. This trade-off is exactly what real tokenizer designers tune.""") - -# ---------------------------------------------------------------------------- -md("""## Series wrap-up - -You built the whole stack, from the ground up: - -- `00_micrograd.ipynb` — backprop from scratch (the engine) -- `01_build_gpt.ipynb` — the bigram baseline -- `02_makemore_mlp.ipynb` — an MLP with embeddings -- `03_batchnorm_activations.ipynb` — keeping deep nets healthy -- `04_backprop_ninja.ipynb` — gradients by hand -- `05_wavenet.ipynb` — a deeper, hierarchical model -- `06_build_gpt_attention.ipynb` — the transformer (GPT) -- `07_gpt_tokenizer.ipynb` — turning text into tokens (this notebook) - -Together these cover Andrej Karpathy's *Neural Networks: Zero to Hero* course, on real data, -with runnable code at every step. - -**Checklist** -- [ ] Text -> UTF-8 bytes (256 base tokens) -- [ ] `get_stats` + `merge` = the BPE mechanism -- [ ] Trained merges; measured compression -- [ ] encode / decode round-trips -- [ ] Regex pre-splitting and special tokens -- [ ] Explained tokenization-caused LLM quirks""") - -# ---------------------------------------------------------------------------- -nb = { - "cells": cells, - "metadata": { - "kernelspec": {"display_name": ".venv (3.12.10)", "language": "python", "name": "python3"}, - "language_info": {"name": "python", "version": "3.12.10"}, - }, - "nbformat": 4, - "nbformat_minor": 5, -} - -out_path = Path(r"d:\BeastProjects\gpt_from_scratch\07_gpt_tokenizer.ipynb") -out_path.write_text(json.dumps(nb, ensure_ascii=False, indent=2), encoding="utf-8") -print("wrote", out_path, "with", len(cells), "cells") diff --git a/kylin_remote_setup.md b/kylin_remote_setup.md new file mode 100644 index 0000000..ecb369b --- /dev/null +++ b/kylin_remote_setup.md @@ -0,0 +1,475 @@ +# Kylin Remote Control & Monitoring Setup + +Complete reference for controlling and monitoring a Kylin Linux machine from a +Windows 11 PC over a local phone-hotspot network. No VPN is used for the SSH +connection itself — only the local Wi-Fi/hotspot. + +--- + +## 0. Bootstrap — START HERE in a new session + +**The connection is already fully set up** (SSH key login + passwordless sudo). +A new session does **NOT** need to repeat the setup in §2–§3 — those sections are +records of what was already done. Just connect and go. + +**Verify everything with one command** (run from the Windows machine, e.g. Claude Code's Bash tool): + +```bash +ssh -o BatchMode=yes -o ConnectTimeout=10 kylin 'echo SSH_OK; whoami; hostname; sudo -n true && echo SUDO_OK' +``` + +Expected: `SSH_OK` / `caichl` / `caichl-HUAWEIQINGYUNL420` / `SUDO_OK`. + +- ✅ **`SSH_OK` + `SUDO_OK`** → full control. Run anything (incl. `sudo`) via + `ssh -o BatchMode=yes kylin ""`. No password needed. +- ❌ **asks for password / times out** → most likely the **hotspot IP changed** + (`192.168.43.248` is DHCP-assigned and can change between sessions). Ask the user + to re-check the Kylin IP (`hostname -I` on that machine), update + `C:\Users\adusu\.ssh\config`, then re-verify. If it asks for a password, the key + may have been removed — see §2.3 / §2.4 to re-establish. +- ℹ️ SSH prints a harmless **post-quantum warning** to stderr — ignore it (the + examples in this doc filter it out). + +**Connect manually:** `ssh kylin` (alias in `C:\Users\adusu\.ssh\config` → `caichl@192.168.43.248`, +with the pinned cipher/MAC already configured — see §2). + +> A fresh Claude Code session also auto-loads memory pointers (`MEMORY.md` → +> `kylin-remote-machine`, `kylin-command-logging`, `kylin-mihomo-vpn`) that link +> back here. This file is the authoritative, self-contained reference. + +--- + +## 1. The two machines + +| Role | Details | +|------|---------| +| **Controller** | Windows 11 (`C:\Users\adusu`), where Claude Code + VPN run | +| **Target** | Kylin V10 SP1, Huawei QINGYUN L420, **ARM64 / aarch64** | +| **Network** | Shared phone hotspot, subnet `192.168.43.x` | +| **Target IP** | `192.168.43.248` | +| **Target user** | `caichl` | + +> **Key idea:** Claude Code needs the VPN to reach Anthropic's servers, but the +> SSH hop from Windows → Kylin is **local LAN traffic** and needs no VPN. The two +> connections are independent. + +--- + +## 2. SSH connection + +### 2.1 Required cipher/MAC (important) + +The default cipher fails over this hotspot link with +`Corrupted MAC on input`. The connection must pin a specific cipher and MAC: + +``` +-c aes256-ctr -m hmac-sha2-256 +``` + +### 2.2 Windows SSH config + +File: `C:\Users\adusu\.ssh\config` + +```sshconfig +Host kylin kyln + HostName 192.168.43.248 + User caichl + Ciphers aes256-ctr + MACs hmac-sha2-256 +``` + +With this in place, simply run: + +```powershell +ssh kylin # or: ssh kyln (both aliases work) +``` + +The equivalent explicit command (no config) is: + +```powershell +ssh -m hmac-sha2-256 -c aes256-ctr caichl@192.168.43.248 +``` + +### 2.3 Passwordless login (SSH key) + +The Windows key `~/.ssh/id_ed25519.pub` was copied to the Kylin machine: + +```powershell +Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub | ` + ssh kylin "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys" +``` + +Result: `ssh kylin` logs in with **no password**. + +### 2.4 Passwordless sudo + +So admin commands can run non-interactively from Windows: + +```bash +# Run once on the Kylin machine (asks for password the one time) +sudo bash -c 'echo "caichl ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/caichl-nopasswd; \ + chmod 440 /etc/sudoers.d/caichl-nopasswd; \ + visudo -cf /etc/sudoers.d/caichl-nopasswd' +``` + +> **Security note:** This makes `caichl` a passwordless-sudo account — acceptable +> for a disposable machine only. To revert: `sudo rm /etc/sudoers.d/caichl-nopasswd`. + +After this, from Windows you can run, e.g.: + +```powershell +ssh -o BatchMode=yes kylin "sudo systemctl status ssh" +``` + +### 2.5 Account credentials (recovery only) + +> ⚠️ **Plaintext password below — keep this file private; do NOT commit or share it.** + +| Item | Value | +|------|-------| +| User | `caichl` | +| Password | `Caicai@2026` | + +**Normally not needed** — SSH login uses the key (§2.3) and sudo is passwordless +(§2.4). Use this only to **recover** if the key auth or sudoers drop-in is ever lost +(e.g. to log in interactively and re-copy the key, or to re-create +`/etc/sudoers.d/caichl-nopasswd`). To change it: run `passwd` on the Kylin machine. + +--- + +## 3. Initial SSH server setup (done on the Kylin machine) + +```bash +sudo apt update +sudo apt install -y openssh-server +sudo systemctl enable --now ssh +hostname -I # find the IP (192.168.43.248) +whoami # confirm the username (caichl) +``` + +--- + +## 4. Monitoring + +### 4.1 SSH logins & sessions (works out of the box) + +```bash +who # who is logged in + source +w # sessions + current command +last # login history with IPs +lastb # FAILED login attempts (sudo) +sudo grep "Accepted" /var/log/auth.log # successful SSH logins +sudo grep "Failed" /var/log/auth.log # failed/suspicious attempts +sudo tail -f /var/log/auth.log # live SSH activity +``` + +### 4.2 Command logging — no reboot (WORKING) + +Logs every command typed in an interactive shell, with user, tty, source IP, +and the command text. + +**`/etc/profile.d/cmdlog.sh`** +```bash +# Log interactive shell commands to syslog (facility local6) -> /var/log/cmdlog.log +if [ -n "$BASH" ] && [ -n "$PS1" ]; then + export PROMPT_COMMAND='logger -p local6.notice -t cmdlog "user=$USER tty=$(tty 2>/dev/null) from=$(who am i 2>/dev/null | sed -n "s/.*(\(.*\)).*/\1/p") cmd=$(history 1 | sed "s/^ *[0-9]* *//")"' +fi +``` + +**`/etc/rsyslog.d/30-cmdlog.conf`** +``` +local6.* /var/log/cmdlog.log +``` + +**Setup commands** +```bash +sudo touch /var/log/cmdlog.log +sudo chown syslog:adm /var/log/cmdlog.log # MUST be syslog-owned (rsyslog drops privs) +sudo chmod 640 /var/log/cmdlog.log +sudo systemctl restart rsyslog +``` + +**View the logs** +```bash +sudo cat /var/log/cmdlog.log # all logged commands +sudo tail -f /var/log/cmdlog.log # live +``` + +Example line: +``` +Jun 11 10:54:22 caichl-HUAWEIQINGYUNL420 cmdlog: user=caichl tty=/dev/pts/5 from=192.168.43.180 cmd=echo hello +``` + +> **Gotcha:** rsyslog runs as the `syslog` user. If `/var/log/cmdlog.log` is owned +> by `root:root`, nothing is written. It must be `chown syslog:adm`. + +### 4.3 System-wide auditing — auditd (QUEUED, UNVERIFIED) + +`auditd` is installed and enabled, with a rule logging all `execve` for real +users: + +**`/etc/audit/rules.d/cmdlog.rules`** +``` +-a exit,always -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog +``` + +It cannot run yet because the kernel booted with `audit=0`. This was changed to +`audit=1`: + +```bash +sudo sed -i 's/audit=0/audit=1/' /etc/default/grub +# /boot is read-only on this device — remount to rebuild grub +sudo mount -o remount,rw /boot +sudo update-grub +sudo mount -o remount,ro /boot +``` + +> **Caveat:** This device appears to boot via **Huawei firmware, not standard +> GRUB** (running kernel `-27` is not the one in `grub.cfg` which lists `-11`, and +> the kernel cmdline has Huawei-specific params). So the `audit=1` change **may +> not take effect** on reboot. The no-reboot logging in §4.2 already covers +> interactive command logging, so auditd is a bonus. + +**Verify after any reboot** +```bash +cat /proc/sys/kernel/audit_enabled # want: 1 +sudo systemctl is-active auditd # want: active +sudo ausearch -k cmdlog -i | tail -40 # human-readable command audit +sudo aureport -x --summary # summary of executables run +``` + +--- + +## 5. Device-specific quirks (Huawei Kylin ARM64) + +- **Architecture is ARM64 (aarch64).** Software must have an ARM64 build; x86-only + proprietary apps will not install/run. +- **`/boot` is a separate ext4 partition mounted read-only.** Remount rw to change + GRUB, then back to ro (see §4.3). +- **Boot likely controlled by Huawei firmware**, limiting effectiveness of GRUB + cmdline edits. +- **Hotspot link needs `aes256-ctr` / `hmac-sha2-256`** or SSH fails with + "Corrupted MAC on input". + +--- + +## 6. Quick reference + +```powershell +# Connect +ssh kylin + +# Run a one-off command from Windows +ssh kylin "uptime" + +# Run an admin command from Windows +ssh kylin "sudo systemctl status ssh" + +# See logged commands +ssh kylin "sudo tail -50 /var/log/cmdlog.log" + +# Watch live +ssh kylin "sudo tail -f /var/log/cmdlog.log" + +# SSH login history +ssh kylin "last" +``` + +--- + +## 7. Troubleshooting + +| Symptom | Cause / Fix | +|---------|-------------| +| `Corrupted MAC on input` | Use `-c aes256-ctr -m hmac-sha2-256` (already in config) | +| `Connection timed out` | Both devices not on same hotspot, or AP/client isolation on the phone — disable isolation | +| `ssh kyln` "could not resolve" | Alias spelling — config now accepts both `kylin` and `kyln` | +| Still asks for password | SSH key not copied (§2.3) | +| `sudo` asks for password | sudoers drop-in empty/missing (§2.4) | +| `audit support not in kernel` | Kernel booted `audit=0`; needs reboot with `audit=1` (§4.3) | +| `cmdlog.log` empty | File not owned by `syslog:adm`; `sudo chown syslog:adm /var/log/cmdlog.log` then restart rsyslog | +| `grub.cfg: Read-only file system` | `/boot` is read-only; `sudo mount -o remount,rw /boot` first | +| VPN GUI app won't run (`GLIBC_2.34 not found`) | Kylin has glibc 2.31; modern Flutter/Electron clients need 2.34+. Use the Go-based mihomo core instead (see §8) | + +--- + +## 8. VPN / proxy: mihomo + web dashboard + +**Why not FlClash / GUI clients:** Kylin V10 SP1 has **glibc 2.31**, but FlClash 0.8.93 +and similar modern GUI clients require **glibc 2.34+** and fail to run. Upgrading +glibc would risk breaking the OS. mihomo is a **statically-linked Go binary** with +no glibc dependency, so it runs fine — controlled via a browser dashboard. + +### Current state +> **VPN is currently OFF.** The `mihomo` service is **stopped** and **disabled** +> (auto-start removed), so it stays off across reboots until re-enabled. Ports +> `7890`/`9090` are closed. The full configuration below is intact and ready to +> resume. + +### Turn the VPN on / off (run on Kylin, or via `ssh kylin`) +```bash +# TURN ON (and restore auto-start on boot) +sudo systemctl enable --now mihomo + +# TURN OFF for this session only (comes back on next reboot) +sudo systemctl stop mihomo + +# TURN OFF and keep it off across reboots <-- current state +sudo systemctl stop mihomo && sudo systemctl disable mihomo + +# Check state +systemctl is-active mihomo # active / inactive +systemctl is-enabled mihomo # enabled / disabled +``` +> Reminder: if you had set the Kylin **system proxy** to `127.0.0.1:7890`, switch +> it back to "None" while the VPN is off, or apps will fail to reach the network. + +### Desktop access +A launcher **"Mihomo VPN Dashboard"** was added to the app menu (Network category) +at `/usr/share/applications/mihomo-dashboard.desktop`; it opens +`http://127.0.0.1:9090/ui/`. (Only works while the service is running.) + +### Installed components +| Item | Location / value | +|------|------------------| +| mihomo core | `/usr/local/bin/mihomo` (Clash.Meta v1.19.27, arm64) | +| Config | `/etc/mihomo/config.yaml` | +| Web dashboard (zashboard) | `/etc/mihomo/ui/` | +| systemd service | `/etc/systemd/system/mihomo.service` (auto-starts on boot) | +| Proxy port (HTTP+SOCKS) | `7890` | +| Dashboard API | `0.0.0.0:9090` | +| Dashboard secret | `428968aebf7c301565c6b6d3a11826c4` | + +### Install steps (done; for reference / rebuild) +```bash +# binary (downloaded on Windows through VPN, scp'd over to dodge chicken-and-egg) +gunzip mihomo-linux-arm64-vX.gz +sudo install -m 755 mihomo-linux-arm64 /usr/local/bin/mihomo +# dashboard +sudo mkdir -p /etc/mihomo/ui && sudo unzip zashboard-dist.zip -d /etc/mihomo/ui +# config.yaml: mixed-port 7890, external-controller 0.0.0.0:9090, secret, external-ui: ui, +# proxy-providers (subscription URL), PROXY (select) + AUTO (url-test) groups +sudo systemctl enable --now mihomo +``` +> The subscription URL lives in `/etc/mihomo/config.yaml` under `proxy-providers:`. +> `external-ui-name` must NOT be set (it makes mihomo look in a `ui/` subdir +> and auto-download); serve `/etc/mihomo/ui` directly. + +### Open the dashboard +- **From Windows browser:** `http://192.168.43.248:9090/ui/` + - backend/API: `http://192.168.43.248:9090` • secret: `428968aebf7c301565c6b6d3a11826c4` +- **From Kylin desktop browser:** `http://127.0.0.1:9090/ui/` + +### Use the proxy +Point apps at `127.0.0.1:7890` (HTTP/SOCKS), or set the Kylin system proxy +(Settings → Network → Proxy → Manual → `127.0.0.1:7890`). For transparent +whole-system routing, enable mihomo **TUN mode** (`/dev/net/tun` is present). + +### Manage via API (examples) +```bash +S=428968aebf7c301565c6b6d3a11826c4 +# switch PROXY group to a node / AUTO +curl -X PUT -H "Authorization: Bearer $S" http://127.0.0.1:9090/proxies/PROXY -d '{"name":"AUTO"}' +# verify exit IP goes through the proxy +curl -x http://127.0.0.1:7890 -s https://api.ip.sb/geoip +# update subscription / reload +sudo systemctl restart mihomo +``` + +**Verified working:** exit IP resolved to Singapore and `google.com/generate_204` +returned HTTP 204 through the proxy. + +--- + +## 9. Browsers + +| Browser | App-menu name | Type / location | Notes | +|---------|---------------|-----------------|-------| +| **Chromium** | "Chromium" | snap `chromium` v149 (`/snap/bin/chromium`) | Installed as the Chrome/Edge equivalent — same engine, Chrome Web Store extensions work. **Set as the default browser.** | + +- **Chrome & Edge cannot be installed** — neither has an official ARM64 Linux build + (same architecture wall as the VPN GUI clients). Chromium is the substitute. +- Installing the chromium snap first required **updating snapd** (Kylin shipped 2.54, + too old → error `assumes unsupported features: snapd2.55`). Fix: + `sudo snap install snapd` (brought it to 2.75.2), then `sudo snap install chromium`. +- Default browser is recorded in `~/.config/mimeapps.list` + (`text/html`, `x-scheme-handler/http(s)` → `chromium_chromium.desktop`). If a + double-click opens the wrong app, right-click the file → **Open with → Chromium → + Set as default**. + +### Desktop instruction files (for the user at the machine) +- `~/桌面/VPN_Guide.html` + `~/桌面/VPN_Guide.md` (copies also in `~/Desktop/`): + a user-facing "how to turn the VPN on/off" guide. Double-click the **`.html`** to + read it rendered in a browser; the **`.md`** opens in **VS Code** (`code`, installed) + with `Ctrl+Shift+V` preview. +- App-menu launcher **"Mihomo VPN Dashboard"** → opens `http://127.0.0.1:9090/ui/` + (only works while the mihomo service is running). + +--- + +## 10. Inventory — everything set up on the Kylin machine + +| Area | What | State | +|------|------|-------| +| Access | SSH alias `kylin`/`kyln`, key login, passwordless sudo | ✅ active | +| Monitoring | SSH login logs (`auth.log`); command logging → `/var/log/cmdlog.log` | ✅ active | +| Monitoring | auditd execve rule | ⏳ queued (needs reboot + `audit=1`; may not take on Huawei firmware) | +| VPN | mihomo core + zashboard dashboard, subscription configured | ⛔ installed, **OFF** (stopped + disabled) | +| Browser | Chromium v149 (default), 360/CNOOC browser | ✅ installed | +| Editors | VS Code (`code`), pluma, WPS Office | ✅ pre-installed | +| IDE | Cursor 3.7.27 (`/opt/cursor`) via `env -i` wrapper | ✅ installed (see §11) | +| Desktop docs | `VPN_Guide.html` / `.md`, dashboard launcher | ✅ on desktop | + +--- + +## 11. Cursor IDE (Electron) — install + crash fixes + +Cursor (AI code editor, Electron-based) installed as an **extracted AppImage** at +`/opt/cursor` (currently **v3.7.27**, ARM64). It will NOT run with a normal launcher +on this Huawei/Kylin box — it needs a special wrapper. + +### Why the special launcher (three Kylin/Huawei ARM64 problems) +1. **EFAULT machine-ID probe crash** — Cursor runs a command to generate a unique + machine ID; the Kylin kernel blocks that memory `write` as suspicious → + `Error: EFAULT: bad address in system call argument, write` → instant crash. + **Fix:** `env -i` strips the environment (especially D-Bus) so Cursor skips the probe. +2. **Mali GPU crash (段错误 / segfault)** — hardware rendering conflicts with Huawei's + Mali GPU driver (`dmesg`: `mali gpu: kctx ... create/destroyed` at crash time). + **Fix:** `--disable-gpu` (software rendering). +3. **Blocked sandbox** — `--no-sandbox`. + Also installed `xdg-desktop-portal` + `xdg-desktop-portal-gtk` (1.6.0) for the + Wayland file picker (necessary but not sufficient alone). + +### The launcher (already installed) +- **Wrapper:** `/usr/local/bin/cursor-launch` +- **Start-menu entry:** `/usr/share/applications/cursor.desktop` → `Exec=/usr/local/bin/cursor-launch %F` + +```sh +#!/bin/sh +exec env -i \ + HOME="$HOME" DISPLAY="${DISPLAY:-:0}" PATH="$PATH" \ + /opt/cursor/usr/share/cursor/cursor --no-sandbox --disable-gpu "$@" +``` + +### Updating Cursor to a newer version +1. Get the latest ARM64 AppImage URL (downloads on Windows through the VPN): + ```bash + curl -sL -A "Mozilla/5.0" -H "Accept: application/json" \ + "https://www.cursor.com/api/download?platform=linux-arm64&releaseTrack=stable" + # -> JSON with "downloadUrl" (…/Cursor-X.Y.Z-aarch64.AppImage) and "version" + curl -L -C - -o Cursor-new.AppImage "" + ``` +2. `scp` it to Kylin, then: + ```bash + chmod +x Cursor-new.AppImage + cd /tmp && ./Cursor-new.AppImage --appimage-extract # -> /tmp/squashfs-root + sudo rm -rf /opt/cursor && sudo cp -r /tmp/squashfs-root /opt/cursor + sudo chown -R root:root /opt/cursor && rm -rf /tmp/squashfs-root + ``` +3. **No change needed** to the wrapper or menu entry — paths stay the same. + Verify glibc first (must need ≤ 2.31): `objdump -T /opt/cursor/usr/share/cursor/cursor | grep -oE 'GLIBC_[0-9.]+' | sort -V | tail`. + +### Cleanup note +Old AppImages in `~/software/` (`Cursor-3.2.11`, `Cursor-3.7.27`) and the user's +manual extract `~/software/squashfs-root` (3.2.11) are redundant now that `/opt/cursor` +is the install — safe to delete to reclaim space. diff --git a/merge_sheet.py b/merge_sheet.py new file mode 100644 index 0000000..89c0305 --- /dev/null +++ b/merge_sheet.py @@ -0,0 +1,27 @@ +import os +import pandas as pdc +from openpyxl import load_workbook + +input_folder = r"d:\input" + +merged_dfs = [] +total_rows = 0 + +excel_files = [f for f in os.listdir(input_folder) if f.endswith(".xlsx")] + +for excel_file in sorted(excel_files): + try: + wb = load_workbook(excel_file, data_only=False) + sheet_name = wb.sheetnames[0] # Get first sheet name + df = pdc.read_excel(excel_file, sheet_name=sheet_name) + merged_dfs.append(df) + total_rows += len(df) + except Exception as e: + print(f"Error processing {excel_file}: " + str(e)) + continue + +if merged_dfs: + merged_output = pd.concat(merged_dfs, ignore_index=True) + output_file = r"d:\output\merged_file.xlsx" + merged_output.to_excel(output_file, index=False) + print(f"Successfully saved {total_rows} rows to output file") \ No newline at end of file diff --git a/mlp_playground.html b/mlp_playground.html new file mode 100644 index 0000000..a3732a8 --- /dev/null +++ b/mlp_playground.html @@ -0,0 +1,403 @@ + + + + + +MLP Playground — Neuron → Layer → MLP (section 5.1) + + + + +

MLP Playground — Neuron → Layer → MLP

+

+ The reviewer-panel story from section 5.1, made live. Two inputs + (income, credit) flow through two panels of 8 reviewers into one final judge: + the 2 → 8 → 8 → 1 network. Every line is a weight, every dot is a + tanh(w·x + b) opinion. Drag the inputs, hover anything, or press play. +

+ +
+
+

Inputs (the application)

+
+ + +
+
+ + +
+ +

Controls

+ + + + +

Read the colors

+
+
positive weight / activation
+
negative weight / activation
+
thin line = weak, thick = strong
+
+ +
+ Final judge
score (tanh) + — +
+

Closer to +1 = likely good tenant, closer to −1 = likely risky.

+
+ +
+ +

+ Hover a line to see its weight, or a dot to see that reviewer's + w·x + b and squashed opinion. Layers left→right: form → junior panel → senior panel → final judge. +

+
+
+ +
+ + + + diff --git a/temp.txt b/temp.txt new file mode 100644 index 0000000..e69de29 diff --git a/temp_script.txt b/temp_script.txt new file mode 100644 index 0000000..0d744a6 Binary files /dev/null and b/temp_script.txt differ diff --git a/website/CLAUDE.md b/website/CLAUDE.md new file mode 100644 index 0000000..34918b6 --- /dev/null +++ b/website/CLAUDE.md @@ -0,0 +1,86 @@ +# Server Information + +**SSH Access:** `ssh root@39.102.124.161` (Alibaba Cloud) +**OS:** CentOS with Baota panel pre-configured +**Website:** https://www.thereisnospoonadu.com/ +**Web Server:** Tengine (Alibaba's fork of Nginx) — use `tengine` commands, NOT `nginx` + +## Gitea (self-hosted Git server) + +**Web UI:** https://git.thereisnospoonadu.com +**Admin username:** beastgitea2026 +**API token:** paste fresh when needed (not stored) +**SSH remote:** `git@git.thereisnospoonadu.com:beastgitea2026/.git` +**SSH port:** 222 — configured in `~/.ssh/config` on dev machine +**Gitea data:** `/var/lib/gitea/` | **Docker container name:** `gitea` +**Tengine config:** `/etc/tengine/conf.d/gitea.conf` + +## Critical File Paths + +**Web Root:** `/www/wwwroot/soft_download/` +**Tengine Config (ACTUAL):** `/etc/tengine/conf.d/thereisnospoonadu.conf` + - Site uses HTTPS (port 443), HTTP redirects to HTTPS + - Always test changes with: `tengine -t && tengine -s reload` + +**Auth File:** `/etc/tengine/.htpasswd` +**Logs:** Check tengine log paths inside `/etc/tengine/conf.d/thereisnospoonadu.conf` + +## User Management + +Add a new end user (can login and download files): +```bash +# Add user (file already exists — do NOT use -c or it will overwrite all users) +htpasswd /etc/tengine/.htpasswd username + +# Verify a user's password +htpasswd -v /etc/tengine/.htpasswd username + +# Delete a user +htpasswd -D /etc/tengine/.htpasswd username + +# List current users +cat /etc/tengine/.htpasswd +``` +Note: `htpasswd` requires `httpd-tools` package (`yum install -y httpd-tools`). + +## Architecture + +**Download Portal System:** +- Login page: `/login.html` (public) +- File listing: `/list/` (requires HTTP Basic Auth via Authorization header) +- File downloads: Direct links to `.exe`, `.md`, `.pdf`, etc. (NO auth required) +- Security model: Must login to see file list, but downloads are public once you know the filename + +**Key Files:** +- `/www/wwwroot/soft_download/index.html` - Main file browser +- `/www/wwwroot/soft_download/login.html` - Login page +- `/www/wwwroot/soft_download/api/validate` - Auth validation endpoint + +## Important Notes + +- When making Tengine changes, ALWAYS edit `/etc/tengine/conf.d/thereisnospoonadu.conf` +- Test against the actual domain: `curl -I https://www.thereisnospoonadu.com/filename` +- PHP is NOT configured/working on this server - use Tengine-only solutions +- Large files (170MB+ exe) need native browser downloads, not blob/XHR approach + +## Verification Checklist (Before Making Changes) + +```bash +# 1. Verify which Tengine config is active +tengine -T | grep -A 30 'server_name.*thereisnospoonadu' + +# 2. Test current behavior against actual domain +curl -I https://www.thereisnospoonadu.com/版本升级说明（用记事本打开）.md + +# 3. After changes: test and reload +tengine -t && tengine -s reload +``` + +## Troubleshooting (Only When Issues Occur) + +```bash +# Find log paths from config +grep 'access_log\|error_log' /etc/tengine/conf.d/thereisnospoonadu.conf +``` + +**Note:** Logs are automatically rotated daily, keeping 7 days of history (compressed). diff --git a/website/admin_server.py b/website/admin_server.py new file mode 100644 index 0000000..9dfb847 --- /dev/null +++ b/website/admin_server.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +import re +import os +from http.server import HTTPServer, BaseHTTPRequestHandler +from datetime import datetime, timedelta +from collections import defaultdict + +LOG_FILE = "/usr/local/nginx/logs/download_access.log" + +LOG_RE = re.compile( + r'(?P\S+) - (?P\S+) \[(?P