beastgitea2026/gpt_from_scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
gpt_from_scratch/00_micrograd.ipynb
T
beastgitea2026 8d2e18c5a4 c
2026-06-12 17:19:26 +08:00

1405 lines
121 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"id": "153d6f74",
"metadata": {},
"source": [
"# Micrograd — backprop from scratch\n",
"\n",
"A tiny automatic-gradient engine in pure Python — **concept -> code -> Your turn** each step.\n",
"\n",
"This is notebook **00** of the series and the right place to start: it explains what a\n",
"*gradient* and *backpropagation* really are, using single numbers you can follow by hand.\n",
"Every later notebook (the bigram in `01`, the transformer in `06`) calls `loss.backward()`\n",
"and trusts it. Here we build that machinery ourselves so it is never magic."
]
},
{
"cell_type": "markdown",
"id": "aac17ea7",
"metadata": {},
"source": [
"## Prologue — what this notebook does, in plain English\n",
"\n",
"We build a small object called a **`Value`**. A `Value` is just a number that also\n",
"**remembers how it was made** (which other numbers, and which operation: plus, times, ...).\n",
"\n",
"Once numbers remember their own history, the computer can answer one very useful question\n",
"automatically:\n",
"\n",
"> *If I nudge this input a tiny bit, how much does the final answer change?*\n",
"\n",
"That sensitivity is the **gradient**. Computing all those sensitivities in one efficient\n",
"backward sweep is **backpropagation** — the single algorithm that trains essentially every\n",
"neural network, including GPT.\n",
"\n",
"This notebook follows Andrej Karpathy's *\"The spelled-out intro to neural networks and\n",
"backpropagation: building micrograd\"* lecture."
]
},
{
"cell_type": "markdown",
"id": "44dd4b0d",
"metadata": {},
"source": [
"### The whole idea, as a chain of gears\n",
"\n",
"Picture a row of **gears** connected together. You turn the first gear a little; the last\n",
"gear also turns, by an amount that depends on all the gears in between.\n",
"\n",
"- The **forward pass** is turning the first gears and reading the last gear (the output).\n",
"- The **gradient** answers: *if I turn this one gear slightly, how much does the final gear move?*\n",
"- **Backpropagation** is figuring out that answer for **every** gear at once, by walking\n",
" backward from the last gear to the first and multiplying the little ratios along the way\n",
" (that multiplication is the **chain rule**).\n",
"\n",
"A neural network is just a very big gear train. Training = nudging each gear a hair in the\n",
"direction that makes the output less wrong."
]
},
{
"cell_type": "markdown",
"id": "334f0f03",
"metadata": {},
"source": [
"### Mental picture — a factory line with tags\n",
"\n",
"Imagine a **cookie factory**. Each station does one small job and puts a **tag** on the box\n",
"saying *who gave me ingredients* and *what I did* (mixed, baked, frosted).\n",
"\n",
"| Step | Real life | In code |\n",
"|------|-----------|---------|\n",
"| Forward pass | Ingredients move down the line; you get a final cookie score | `d = a * b + c` computes numbers left to right |\n",
"| History | Every box keeps tags — no separate diary | `_prev` (parents) and `_op` (operation) on each `Value` |\n",
"| Gradient | *If I change this one ingredient a tiny bit, how much does the final score change?* | `.grad` on each `Value` |\n",
"| Backprop | Walk backward along the tags from the final score to every ingredient | `loss.backward()` |\n",
"\n",
"A normal float is like a score on a napkin: **4**. You cannot tell where it came from.\n",
"A **`Value`** is the same score **on a tagged box** — you can always walk back.\n",
"\n",
"**Blame traveling backward (the chain rule)**\n",
"\n",
"The final taste score is off. At each station you ask: *if your output had been 1 point\n",
"sweeter, how much would the final score have moved?*\n",
"\n",
"- **`+` (mix two bowls)** — nudge either bowl; the mix moves the same amount. Blame passes\n",
" through to both inputs equally.\n",
"- **`*` (recipe ratio, e.g. 2 cups × 3 batches)** — change flour a little → effect depends\n",
" on batch count; change batches → effect depends on flour. Each side's blame depends on\n",
" the **other** side's amount.\n",
"\n",
"**Backprop** starts at the output (\"downstream moved by 1\"), walks **backward** along the\n",
"tags, and at each station multiplies by that station's local rule. Small effect here ×\n",
"small effect there = total effect on the end.\n",
"\n",
"**Why gradients add (`+=`)**\n",
"\n",
"One bag of sugar might go into **both** dough and icing. Both paths affect the final taste,\n",
"so sugar's total blame is the **sum** of both paths — same when one `Value` is used twice\n",
"in an expression.\n",
"\n",
"**Training the MLP (section 6)**\n",
"\n",
"Same factory, but the knobs are **weights and biases** (how much each input counts):\n",
"\n",
"1. **Forward** — run points through the line; get predictions.\n",
"2. **Loss** — how wrong were we? (distance from the target).\n",
"3. **Backward** — `loss.backward()` sends blame to every knob.\n",
"4. **Update** — turn each knob a tiny step **against** its gradient (downhill = less wrong).\n",
"\n",
"The **two moons** dataset is two swirls of red and blue beads. The network learns a curved\n",
"boundary — not by memorizing dots, but by turning knobs until wrong guesses hurt (high loss)\n",
"and right ones don't.\n",
"\n",
"> **One line to keep:** Forward = tagged boxes down the line. Backward = follow tags from\n",
"> the final score back to each ingredient and assign fair blame. PyTorch does the same thing\n",
"> at scale — millions of tagged boxes, fast hardware."
]
},
{
"cell_type": "markdown",
"id": "84448625",
"metadata": {},
"source": [
"### 0.1 Imports\n",
"\n",
"Pure Python plus a little NumPy/Matplotlib for the toy dataset and pictures. No deep-learning\n",
"library is needed to build the engine itself — that is the whole point.\n",
"\n",
"**-> Training:** `math` gives us `exp`/`tanh` for activations; `random` initialises weights;\n",
"NumPy and Matplotlib are only for the demo dataset and plots."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e31a52ef",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ready\n"
]
}
],
"source": [
"import math # exp, tanh used inside Value activations\n",
"import random # random weight init for Neuron\n",
"import numpy as np # dataset + optional plots only (not the autograd engine)\n",
"\n",
"random.seed(1337) # same random weights every run\n",
"np.random.seed(1337) # same noisy moon dots every run\n",
"\n",
"# Plots are optional: the notebook runs fine without matplotlib (it just skips pictures).\n",
"try:\n",
" import matplotlib.pyplot as plt\n",
" HAS_PLT = True # flag so later cells can skip plotting gracefully\n",
"except Exception:\n",
" HAS_PLT = False\n",
" print(\"matplotlib not installed -> plots will be skipped (everything else still works)\")\n",
"\n",
"print(\"ready\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ace0626a",
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import HTML, display\n",
"import json\n",
"\n",
"_GRAPH_UID = 0 # unique id per render so multiple cells don't clash\n",
"\n",
"\n",
"def show_graph(root, title=\"Computation graph — hover a node\"):\n",
" \"\"\"Draw an interactive SVG graph for a Value expression.\n",
"\n",
" Hover (or click) any node to see its value, operation, and parents.\n",
" After `.backward()`, gradients appear on each node automatically.\n",
" \"\"\"\n",
" global _GRAPH_UID\n",
" _GRAPH_UID += 1\n",
" gid = f\"mg{_GRAPH_UID}\"\n",
"\n",
" nodes, seen = [], set()\n",
"\n",
" def build_topo(v):\n",
" if v not in seen:\n",
" seen.add(v)\n",
" for ch in v._prev:\n",
" build_topo(ch)\n",
" nodes.append(v)\n",
"\n",
" build_topo(root)\n",
" id_map = {v: i for i, v in enumerate(nodes)}\n",
"\n",
" depth = {}\n",
"\n",
" def mark_depth(v, d=0):\n",
" depth[v] = max(depth.get(v, 0), d)\n",
" for ch in v._prev:\n",
" mark_depth(ch, d + 1)\n",
"\n",
" mark_depth(root, 0)\n",
" max_depth = max(depth.values()) if depth else 0\n",
" layer_of = {v: max_depth - depth[v] for v in nodes}\n",
"\n",
" from collections import defaultdict\n",
"\n",
" layers = defaultdict(list)\n",
" for v in nodes:\n",
" layers[layer_of[v]].append(v)\n",
"\n",
" NODE_W, NODE_H, GAP_X, GAP_Y = 108, 58, 36, 72\n",
" positions = {}\n",
" max_w = 0\n",
" for ly in sorted(layers.keys()):\n",
" row = layers[ly]\n",
" row_w = len(row) * NODE_W + max(0, len(row) - 1) * GAP_X\n",
" max_w = max(max_w, row_w)\n",
" x0 = (max_w - row_w) / 2\n",
" for i, v in enumerate(row):\n",
" positions[v] = (x0 + i * (NODE_W + GAP_X), ly * (NODE_H + GAP_Y))\n",
"\n",
" svg_w = int(max_w + 48)\n",
" svg_h = int((max(layers.keys()) + 1) * (NODE_H + GAP_Y) + 36)\n",
" show_grad = any(getattr(v, \"grad\", 0.0) for v in nodes)\n",
"\n",
" def node_name(v):\n",
" lbl = getattr(v, \"label\", \"\") or \"\"\n",
" return lbl if lbl else f\"n{id_map[v]}\"\n",
"\n",
" def esc(s):\n",
" return (\n",
" str(s)\n",
" .replace(\"&\", \"&\")\n",
" .replace(\"<\", \"&lt;\")\n",
" .replace(\">\", \"&gt;\")\n",
" .replace('\"', \"&quot;\")\n",
" )\n",
"\n",
" edge_lines = []\n",
" for v in nodes:\n",
" for ch in v._prev:\n",
" x1, y1 = positions[ch]\n",
" x2, y2 = positions[v]\n",
" cx1, cy1 = x1 + NODE_W / 2, y1 + NODE_H\n",
" cx2, cy2 = x2 + NODE_W / 2, y2\n",
" eid = f\"{gid}-e{id_map[ch]}-{id_map[v]}\"\n",
" edge_lines.append(\n",
" f'<line class=\"edge\" data-edge=\"{eid}\" data-from=\"{id_map[ch]}\" data-to=\"{id_map[v]}\" '\n",
" f'x1=\"{cx1:.1f}\" y1=\"{cy1:.1f}\" '\n",
" f'x2=\"{cx2:.1f}\" y2=\"{cy2:.1f}\" stroke=\"#94a3b8\" stroke-width=\"2\" '\n",
" f'marker-end=\"url(#{gid}-arrow)\"/>'\n",
" )\n",
"\n",
" node_groups = []\n",
" meta_nodes = {}\n",
" for v in nodes:\n",
" x, y = positions[v]\n",
" nid = id_map[v]\n",
" node_id = f\"{gid}-n{nid}\"\n",
" nm = esc(node_name(v))\n",
" op = esc(v._op or \"leaf\")\n",
" val = f\"{v.data:.4f}\"\n",
" grad = f\"{getattr(v, 'grad', 0.0):.4f}\"\n",
" fill = \"#dcfce7\" if not v._prev else \"#dbeafe\"\n",
" grad_svg = (\n",
" f'<text x=\"{x + 8:.1f}\" y=\"{y + 50:.1f}\" font-size=\"11\" fill=\"#c2410c\">'\n",
" f\"grad = {grad}</text>\"\n",
" if show_grad\n",
" else \"\"\n",
" )\n",
" node_groups.append(\n",
" f'<g class=\"node\" data-node=\"{node_id}\" tabindex=\"0\" role=\"button\" '\n",
" f'aria-label=\"{nm} value {val}\">'\n",
" f'<rect x=\"{x:.1f}\" y=\"{y:.1f}\" width=\"{NODE_W}\" height=\"{NODE_H}\" rx=\"8\" '\n",
" f'fill=\"{fill}\" stroke=\"#64748b\" stroke-width=\"1.5\"/>'\n",
" f'<text x=\"{x + 8:.1f}\" y=\"{y + 18:.1f}\" font-size=\"13\" font-weight=\"600\" '\n",
" f'fill=\"#0f172a\">{nm}</text>'\n",
" f'<text x=\"{x + 8:.1f}\" y=\"{y + 34:.1f}\" font-size=\"11\" fill=\"#475569\">'\n",
" f\"op: {op} val: {val}</text>\"\n",
" f\"{grad_svg}\"\n",
" f'<title>{nm}: {val} ({op})</title></g>'\n",
" )\n",
" meta_nodes[node_id] = {\n",
" \"name\": node_name(v),\n",
" \"op\": v._op or \"leaf\",\n",
" \"val\": val,\n",
" \"grad\": grad,\n",
" \"parents\": \", \".join(node_name(p) for p in v._prev) or \"—\",\n",
" }\n",
"\n",
" meta_json = json.dumps({\"gid\": gid, \"show_grad\": show_grad, \"nodes\": meta_nodes})\n",
"\n",
" html = f\"\"\"\n",
"<div class=\"mg-wrap\" style=\"font-family:system-ui,sans-serif;max-width:{svg_w + 20}px\">\n",
" <div style=\"font-weight:600;margin-bottom:6px;color:#334155\">{esc(title)}</div>\n",
" <svg width=\"{svg_w}\" height=\"{svg_h}\" style=\"background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px\">\n",
" <defs>\n",
" <marker id=\"{gid}-arrow\" markerWidth=\"8\" markerHeight=\"8\" refX=\"6\" refY=\"3\" orient=\"auto\">\n",
" <path d=\"M0,0 L6,3 L0,6 Z\" fill=\"#94a3b8\"/>\n",
" </marker>\n",
" </defs>\n",
" {''.join(edge_lines)}\n",
" {''.join(node_groups)}\n",
" </svg>\n",
" <div id=\"{gid}-info\" style=\"margin-top:8px;padding:8px 10px;background:#fff;border:1px solid #e2e8f0;\n",
" border-radius:6px;font-size:13px;color:#334155;min-height:1.2em\">\n",
" Hover a node to inspect it. Arrows flow from inputs (bottom) to output (top).\n",
" </div>\n",
"</div>\n",
"<style>\n",
" .mg-wrap .node {{ cursor: pointer; }}\n",
" .mg-wrap .node rect {{ transition: stroke 0.12s, filter 0.12s; }}\n",
" .mg-wrap .node:hover rect, .mg-wrap .node:focus rect {{ stroke: #f59e0b; stroke-width: 2.5; filter: drop-shadow(0 1px 3px rgba(0,0,0,.15)); }}\n",
" .mg-wrap .edge.dim {{ opacity: 0.15; }}\n",
" .mg-wrap .edge.hl {{ stroke: #f59e0b; stroke-width: 3; opacity: 1; }}\n",
"</style>\n",
"<script>\n",
"(function() {{\n",
" const meta = {meta_json};\n",
" const info = document.getElementById(meta.gid + \"-info\");\n",
" const edges = Array.from(document.querySelectorAll('[data-edge^=\"' + meta.gid + '-\"]'));\n",
" function clearHL() {{\n",
" document.querySelectorAll('.node[data-node^=\"' + meta.gid + '-\"]').forEach(n => n.classList.remove('pinned'));\n",
" edges.forEach(e => {{ e.classList.remove('hl', 'dim'); }});\n",
" }}\n",
" function highlight(nodeId) {{\n",
" clearHL();\n",
" edges.forEach(e => e.classList.add('dim'));\n",
" const nid = nodeId.split('-n')[1];\n",
" edges.forEach(e => {{\n",
" if (e.getAttribute('data-from') === nid || e.getAttribute('data-to') === nid)\n",
" e.classList.remove('dim'), e.classList.add('hl');\n",
" }});\n",
" const m = meta.nodes[nodeId];\n",
" if (!m) return;\n",
" let txt = '<b>' + m.name + '</b> &nbsp; op=' + m.op + ' &nbsp; value=' + m.val;\n",
" if (meta.show_grad) txt += ' &nbsp; grad=' + m.grad;\n",
" txt += '<br>parents: ' + m.parents;\n",
" info.innerHTML = txt;\n",
" }}\n",
" document.querySelectorAll('.node[data-node^=\"' + meta.gid + '-\"]').forEach(el => {{\n",
" el.addEventListener('mouseenter', () => highlight(el.getAttribute('data-node')));\n",
" el.addEventListener('focus', () => highlight(el.getAttribute('data-node')));\n",
" el.addEventListener('click', () => {{\n",
" el.classList.toggle('pinned');\n",
" highlight(el.getAttribute('data-node'));\n",
" }});\n",
" }});\n",
"}})();\n",
"</script>\n",
"\"\"\"\n",
" display(HTML(html))\n",
"\n",
"\n",
"print(\"interactive graph viewer ready\")"
]
},
{
"cell_type": "markdown",
"id": "3315510f",
"metadata": {},
"source": [
"### 0.2 A tiny dataset — two interleaving moons\n",
"\n",
"Our goal at the end is to train a small network to separate two groups of dots that curl\n",
"around each other (\"two moons\"). Real-life picture: two handfuls of red and blue beads\n",
"mixed in a swirl — can the network learn to draw the boundary between them?\n",
"\n",
"We try scikit-learn's `make_moons`; if it is not installed we generate the same shape with\n",
"NumPy so the notebook stays dependency-light."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "507b4df3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sklearn not available, using NumPy fallback: ModuleNotFoundError\n",
"X shape: (100, 2) y shape: (100,)\n",
"first 3 points: [[0.9296812690137589, -0.04902823627877147], [0.965763959785321, -0.11143765173168577], [1.0124564608446291, -0.07324929559695742]]\n",
"first 3 labels: [0, 0, 0]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAF2CAYAAAA4MQK3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAaAZJREFUeJztnQd4FNXXxk9ooYgUqUE6CiJVmiBKlSJSFJGiNBGkKoLwBwuIqFQBQRQbzUJTioKCSBGVIgYQKfoBUqWXhCogzPe89zrJ7GZ2s5vsZtv7e56B7OzszJ075dxz7ilRhmEYQgghhJBUky71uyCEEEIIhSohhBDiQ6ipEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUKCiAMHDkhUVJTMnDkz0E0hYQDuI9xPv/76q4QSUVFR8uqrr0ooEpZCdf369eqCxMXFBbophNjy+eefy6RJk8K6d44ePaqew23btgW6KUHDu+++65cBk7/2G2l888038tBDD0nBggUlZ86c8uCDD8pvv/3m1T7CVqiOGDGCQpWEnFAtWrSoXLlyRTp27CjhIFTxHFKoJkKhGtw8/PDDkjFjRhk2bJgMHTpUduzYoQTrqVOnPN5HBr+2kBDitdkrc+bM7LUQALVI/vnnH8mSJUugm0LccOnSJcmWLZt4woYNG6RGjRoJnytXriyNGzeW7777Tp544onI1FRhbho0aJD6u3jx4uolhQVzVY8++qjcc889Dts3b95cff/VV18lrNu0aZNa9+233yas++uvv6RNmzaSO3duyZo1q9x7772ybNkyj9qEffXt21cWLFggZcuWVQ9hzZo15ffff1ffv//++1KqVCn1Mq1bt65qqzP4bZUqVdRv8+TJI08++aT8/fffSbZbvXq13H///eomgvmiZcuWsnv37iR9hDbt3btXunTporbLkSOHdO3aVS5fvuyw7cqVK6V27dpqm1tuuUVKly4tL774ogSKmzdvyttvvy3ly5dX/ZU3b15p0qSJw5zRjBkzpH79+pIvXz6Jjo5Wff7ee+8l2VexYsXUyPSnn36S6tWrq/2VKFFCZs+e7VFbxo8fL7Vq1ZLbbrtNXRdcny+++CLZ3+Ea4945ePBgwv2JtriaU8U1Qt/jerdq1Ur9jfN+4YUX5MaNGwkveOwD19sZvPhxfZ955hmP7tPFixdLuXLlVN/dfffdsnz58iTboi1PPfWU5M+fP2G76dOnJ3y/du1aqVatmvob95V5nu5MlBcuXJD+/fur88A+cf2gJWzZssVhOzyfuOY4JzyLderUkZ9//tn2Hv/jjz/k8ccfl1tvvVVdp+eee071hxVv75cVK1ZI1apV1TXHs+vpPvD7nTt3yg8//JDQH7gXTDBdhfMvXLiw2gfeCWPGjFH3vDuS2y+4evWqDBgwQN03eDc88sgjttoX3nnm+yN79uzSrFkztW9PQPuff/75hOt3++23S6dOneT06dMJ25w8eVK6deum7hs8bxUrVpRZs2Z5tP+tW7dK06ZN1bXEM9CgQQPZuHGj7Rwy+qJ3797qeqAdnt5fVoEKzAHutWvXxGOMMOO3334z2rdvj3J2xsSJE41PPvlELRcvXjQmTJhgpEuXzoiPj1fb3rx508iVK5da98ILLyTsY9y4cQ7bHT9+3MifP7+RPXt246WXXlL7qVixotpm4cKFybYJbalQoYJRuHBhY/To0WrJkSOHUaRIEeOdd94xypYta7z11lvGyy+/bGTKlMmoV6+ew+9nzJih9lGtWjV1TkOGDDGyZMliFCtWzDh37lzCditXrjQyZMhg3HnnncbYsWONESNGGHny5FHnuH///oTthg8frvZXuXJl49FHHzXeffdd4+mnn1brBg8enLDdjh07VHuqVq1qvP3228a0adNUPz3wwANGoOjSpYtqZ9OmTY1JkyYZ48ePN1q2bGlMmTIlYRv0E7ZDX2F9o0aN1G/Q11aKFi1qlC5dWl3bF198UX1/zz33GFFRUerck+P22283evfurX6He6J69erqOEuXLnX7u++++86oVKmSujbm/blo0SL1Ha4T9oFrbtK5c2cjc+bMxt1332089dRTxnvvvWe0bt1abYdrZ4J7M2PGjMaZM2ccjjd//ny17bp169y2C9vgvi5YsKAxcuRI1b8lSpQwsmbNapw+fTphOzwPOHfcz6+99ppqT4sWLRKeOXMbfId1PXr0SDjPffv2uTx+hw4d1P02YMAA46OPPjLGjBljNG/e3Pj0008Ttlm1apXapmbNmuqZwfHwbGHdpk2bktzj5cuXV/vANXryySfVuo4dOzoc15v7pVSpUup5wjOI52HNmjUe7wPXGP1WpkyZhP7AvQAuXbqkzuO2225T9yL23alTJ3UvPvfcc26vm7v9mu8OPOv169dXbRs4cKCRPn164/HHH3fYz+zZs9XxmjRporZD/+MdkzNnTof3hx0XLlwwypUrp/bbvXt3dU/gHkK/bN26VW1z+fJl46677lL36PPPP29MnjzZuP/++1X7cK9ZwTpcQxM8j9myZUu4N/EOLV68uBEdHW1s3LgxYTvzfPFOrVOnjjoPbOvp/WXlypUrxn333aeuifMz5Y6wE6qmUETHOt8ImzdvVuu/+eYb9Xn79u3qc5s2bYwaNWokbIcXBG5Ck/79+6vtfvzxR4ebCBcVN92NGzfctge/xcW3tuf9999X6wsUKGCcP38+Yf3QoUMd2n7t2jUjX7586obFRTbBixvbDRs2LGEdXtTY1noDYJAB4Y8H1PmFgxe0lUceeUTdQCZ4QWC7U6dOGcHA6tWrVXueffbZJN9hgGSCh9eZxo0bKwHh/JJ0FjYnT55U1wovnuRwPg6uFa4TXl7J0axZM3V8Z1wJVayDkLKCe7RKlSoJn//880+1HV5oVnA/4z619pEd+C1eOnv37nW4f7DeOmjp1q2berlZBS1o166dGiya/WI+b9ZzcQd+26dPH5ffo/133HGHupbO1xvP4oMPPpjkHse5W8EgCOtxXtbfe3O/LF++PMn2nu4DAyO87J2BoIDQ+L//+z+H9RDeEFSHDh1K8htP9msKmYYNGzr0GYQa9hsXF5fwPoPwhEC0gsERrovzemfwHsJxFtooGeZxITixjVWI4ZnBAOmWW25xeA86C9VWrVqpe9M6KDt69KhSdKyDfPN8a9eubfz7779e3V9Wrl+/bjz00EPqXbB27VrDG8LO/OsO2MdhNli3bp36/OOPPyaYKGACgOkT1xPmQJhArB5hMA/CDGqC/fTo0UOZ63bt2pXssWGqME18VjND69atlZnFeT3MzQBmTZhMYMqwzrXBLFOmTJkEE/SxY8eUQwhMhTBRm1SoUEGZOHAOzvTs2dPhM875zJkzcv78efUZJl+wZMmSZE1QacGXX36pTDvDhw9P8h3Wm1jnuOLj45X5CSZC9Ck+W4GZznqtYR6Didvsf3dYj3Pu3Dm1b+zL2VzpK+yul7Wdd955p7p/Pvvss4R1Z8+eVSY9zAdZ+8gVDRs2lJIlSzrcPzC3mcfB84HrgGkT/I2+NRfMPaEPUnr+uN9g2oWDkx24v/fs2SMdOnRQ96l5XMyZ4fnCc+18n/bp08fhc79+/dT/1ufBm/sFU0o4T2e82YcdmN7B9cyVK5dDn+J6wMRvvrNSCt5V1uuPY2G/mIIwp3lgvm3fvr3D8dOnT6/uqTVr1rjdP+4JmHIfeeSRJN+Zx0WfFyhQQB3DBE5Bzz77rFy8eFGZbO1AOzGniakPTM+YwEMX9wLe1+Y7y6R79+6q7d7cX1bgYIfnBs8SrqM3RJSjEjoZc5kQpgD/4+aCsMSFg30etn68iKwvWtx4zrZ2cNdddyV8jzkodxQpUsThM+aDAOZP7NbjJW3uG+BF7wyEKm6o5LZDOzEP5Dxh79wmPNDmsfEibdu2rXz00Ufy9NNPy5AhQ9SLC/PSjz32mKRL53o8hv7zag7CAgYEmTJlsv1u3759EhMT4zBosAPzaxC8cDpwniPGC87sY7s+MPvB7H93LF26VF5//XX1sseclYknwstbzPnj5NqJASLmRXE/wJMYL+vr16977E2cXH9gHg4v3w8++EAtdmAQmBLGjh0rnTt3Vs8E5qcR2oDzMV+kEKgA27gC19e8j8Edd9zh8D0GDLh3rX4L3twvEKp2eLMPO3Bu27dvT3KNU9unnjzr5vEB5oXtwPvAHXg2oSC4A/ckrofzu8P6HrUD9xz61NW7DQOpw4cPq3l9d9cpufvLyieffKKUkeTOSSJdqAII0DfeeEM5K0CovvTSS2oEA6GIzxCqwCpUfYHzqCm59doC4l+SOzZG3xghY5QKjRgOK/PmzVMPHkaOrn4Pwetq1JkcOJazk4U34OGG8MeAY8KECeoBgpDGKHnixIlJNJmU9j/ulRYtWsgDDzygwiQwasaoGw4rCJfxNa7a6Uy7du2UswhG2HAo+/TTT5VTjd0LyZvjmP1h9h8c5VwJN2i3KQEORXjuFi1apO6vcePGKUedhQsXKgcV89hYX6lSJdt9wILkDucBj7f3i52nr7f7sAPb4CU+ePBg2+9hhUgNnl5XCBNok85kyBBaoiKLzXVK7v6yAksInumUEFo95SHuNAV0KrSoOXPmKA9GU3ji5WgKVdzApnAFGPH/+eefSfYFz0Lze39h7hvHdx5FYp35vXU7u3bCY9hTt3IrGFXihYEFL4w333xTDUQg/GCasuOtt97ySNOzAyYkV0DLgMYNTdiVtvr1118rrRHe3NbReXLmK2+BuQvaI9oDT0ITCFVP8Ic2C9AvmBqAUIXJFxqUL5NMQJPCdAUsO66uf2rOES8yTHVggXYGb30MgvHSM83S0JqSO7YJNDCr1gKPdwgQcyrGF/eLN/tw1Sc4N5hAPT0vT/frKWbfwiM2JW3A7xHT6Q68o6CNo/+t2mpy71Hcc/DydvVuw76cLX4pub+czeWYLkwJYTmnagoPu4xKMONCo8AIBS8g02QA4QrzLzQsZy0VZoJffvlFmXZMYEqF+QsPJ+bl/AW0DNzo06ZNczAxwt6PUBm8QM2bBaN3uKdbzxs3OkZlOAdvgfByxtQQrG1xBqYVPJgpWaymO2dgisHIGvMdrkbc5ojcqmnC/OapsPMUHAcvMjOkBcCkiHAUT+9RT+baUgJMvZjnR2gZ2gnt1Vdgf7gOGFTYvUStYRrunkNn0I/O/YH7HuZ+817DfYWXN0KZIIDcHdtk6tSpDp+nTJmi/jdfor64X7zZB/rErj+gReH9gkGaM9j+33//ddsGV/v1FMwTY7CCQTOmC5xJLvkB7glkHlq0aFGS78x+wTvo+PHjytplgvPCNYGFwdXcJfq3UaNGyrfDarY/ceKEsgrB+picedqT+8t5ThaKREoIS00VDx+ARoUXCoQoHCtw42HEg+8hQM0YVVNThaDE4ixUMZ8IzRYPIibVIYwhvPbv369eLu7mF1OLOQBArB9uOkzy42ZCrCYEOkx9JjBnoI2YN0YsGDLz4IbFfE5K8mi+9tpryvwLwY1RJEZ2MHXCucvqtJVW1KtXTwmMyZMnKw0EsYoY9cLCgO8wl4iHD6Y3XFvEZeLl++GHH6oHCM5cvgJ9As0dbYCzBPoGL3DEFmI0nhy4B/FyQewg4jnxUkGbfdU2xGRiPhX3A87dl4wePVppYRig4uWDQSUGYHBQ+v777xMGYxCAmFrBgBDaLZ4//MZuvgsxhLivMF8PawX6A/vavHmzsnwAPGeY48c5YTCMZ6JQoULK4oT24MUKrdEKnlGY6XGdILRgDsf1Mi0ivrhfvNkHrjviVzEXj3sF28AChQEQNF3EwcLZENvhXYRYdsQ+Q5jA2uQKV/v1FPQdfo/nC9ob3pvQEA8dOqSmfu677z555513XP4e7Uc727Rpo+KX0R7cBzgnXH/0N7Q/xPXi/GJjY9X7C78xrSlWh01ncF5mzDy0TJijsS8IRMyVJocn95fzXC2mN1KU+tEIU+CiXqhQIRVO4hxeM2jQILUOcUpWEIOG9XaxdFj32GOPKbdzxAwiJjG5eEQT7NPZldsMnUD4jxXEvWH9ggULHNbPmzdPhVDAxTt37tzGE088YRw5ciTJsb7//nsVW4U41ltvvVXFYe3atcthGzPcwDlUxnRHN/sKMYGIAY2JiVHu7PgfMcDObv9pCdzk0WeIyUOb8ubNq2JWY2NjE7b56quvVMwfrhNCSXCdp0+fnuQ+QIgEQlucQWiCXXiCMx9//LEK8cA1QXvQf2bfJgfiphE3h/sJ25vhNa5CahBu4Yy7Y5mhI59//rnhKXb3KUDb0AYrJ06cUNsiVhVxhwgNa9CggfHBBx84bLdkyRIVM4j4aXfhNVevXlXPJeJkESaB88Xf1jhcE8Q9Ir4a4V/oe7QPMZe4X537Bvc+nlvsE/Glffv2dQhN88X94s0+EKKCfaA9+M56nyGsBSF1eA/h3kYcc61atVQsNkJP3OFqv+YzjfAmu/eMGWdrXY9QIISf4FxKliyp4m9//fVXIzkQyte3b1/13kX7ETuL+8YaeoX7pmvXrurcsA3iiO3uCeeQGrBlyxbVNoTfIHYa8fzr16932MbV+Xpzf5nHd77nPSXqvx0QQsIIWDA+/vhjZW6DdSbSgGUG0wQwW7rT8AjxNWE5p0pIJAPPdpg5Mc8ViQKVkEASlnOqhEQimNfFPBHmqRASgDy3hJC0hUKVkDABHr8Io4GTCpy5XMVyEkL8B+dUCSGEEB/BOVVCCCHER1CoEkIIIT4iIudUkTAAlQoQbOyvdHGEEEKCG0SUIjEEMiv5KolPRApVCFRPc0USQggJbw4fPqwyLvmCiBSqZjosdGRyOSMJIYSEJ+fPn1cKlrsUid4SkULVNPlCoFKoEkJIZBPlw2lAOioRQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4iIiMUyXhy7//iixbJhIbK1KggEi7diK5cwe6VYSQSIFClYQNJ0+K1K8vsnOnSIYMIjduiAwaJLJokUijRoFuHSEkEqD5l4QNvXqJ/PFHosZqGCJXroi0bi1y4UKgW0cIiQQoVElYEBcnsnix1k6tQLBevKi/I4QQf0OhSsKC+HiU9LP/DhWdTp1K6xYRQiIRClUSFhQqJJI/v/13ELY1a6Z1iwghkQiFKgkL4Jg0YkTS9enTizRoIHLvvYFoFSEk0qBQJWHDM8+IfPyxSNGi+nOWLNp5ackSlHYKdOsIIZFAlGHAlSPyCtPmyJFD4uPjWU81DMEdjTnWbNlEMmYMdGsIIZEkCxinSsIOaKU5cwa6FYSQSITmX0IIIcRHUKgSQgghPoJClRBCCPERFKqEEEKIj6BQJYQQQnwEhSoJW/btE+nfXyd+QFL9FSsC3SJCSLjDkBoSlmzeLFK3rsi1a7piDTIrLVwoMnKkyMsv++eYSIe4f7+OjS1SxD/HIIQEN9RUSVjSp4/IP/9ogQrM6jXDhokcPOj74y1dKlKqlF6Q0alKFV0oPVDs2SOyZo3I8eOBawMhkQiFKgk7TpzQmqqrqjVff+3b423YINKypciBA4nrfvtNa8qHD0uacuyYSL16InfeqQu23367yNNPi1y9mrbtICRSoVAlYUdyiTddCduUMmaMzuJkPS40YxRInzbN9e8gcP/6K/n2egr289BDIj/+6NiOGTNEBgzwzTEIIe6hUCVhR4ECIpUq6Tqqdjz8sG+Pt2VL0uLoAOu2bk26HmZhmIcx71qypMgdd4h8803q2/HTTyLbtiVtCwYRH32k8yETQkJYqK5bt06aN28uMTExEhUVJYsXL072N2vXrpV77rlHoqOjpVSpUjJz5swk20ydOlWKFSsmmTNnlho1asgvv/zipzMgoco772iHIZSEA3BUAkOGiJQo4dtjwcRqJ8BxbHznrJ3CLAzzsAm01RYtRDZtSl07/vjD9Xdw2PLHXDIhJA2F6qVLl6RixYpKCHrC/v37pVmzZlKvXj3Ztm2b9O/fX55++mlZYYmFmDdvngwYMECGDx8uW7ZsUftv3LixnDx50o9nQkKN++7TGmTnziLlyumaql9+KfLGG74/Vu/e9iZlOEl17+647r33tFnYqk3CbAvzMczIqcHdYMFOwBNC/ICRRuBQixYtcrvN4MGDjbvvvtthXdu2bY3GjRsnfK5evbrRp0+fhM83btwwYmJijFGjRnnclvj4eNUe/E9Iarl5E/euYURF4T7XS8aMhjFtWtJtmzRJ3MZ5KVw4de24ccMwSpc2jAwZHPebPr1hdOmSun0TEo7E+0EWBNWc6oYNG6Rhw4YO66CFYj24du2axMbGOmyTLl069dnchpC0xtQyYcZ9/33tGPT337poujOFCiWapK3AfOxOk4Rmi9mTp54S6dpVZNGipHOn2Mfy5SJlyzquh2kZ5nBCSIQlfzh+/Ljkz5/fYR0+o5DslStX5Ny5c3Ljxg3bbf5wM6F09epVtZhgf4T4mmLFRHr0cL8NzMEff5x0PczHiK21A2ZkZIT66qtEgQxXg2bNtHC1FmJHG+Cs9Ouvev62fHntCEUISRuCSlP1F6NGjVLV3c2lcOHCgW4S8ZDr1zE3Hz7dVaOGyLvvOgpCaLqDBol06GD/m9mztUA1BayZ0GLZMq0VO4P9Vasm8uijFKiERLRQLVCggJxA5L4FfL711lslS5YskidPHkmfPr3tNvitK4YOHSrx8fEJy+G0jsiPYC5fFoEPmbexoadOaSejbNlEbrlFpGJFkW+/lbCgVy+RI0e0xgrHJeQoHjtWC0M7PvvM/jusw3eEkOAhqIRqzZo1ZdWqVQ7rVq5cqdaDTJkySZUqVRy2uXnzpvpsbmMHwnMgmK0L8S9nzoh07CiSIwfM89oz9ZNPPPstLPV16miBAU0V7NihzZ0rV0pYkC+fnh/t2VOkeHH320JTt0sQgXUXL/qtiYSQYBOqFy9eVKExWMyQGfx96NChBA2yU6dOCdv37NlT/vrrLxk8eLCaI3333Xdl/vz58vzzzydsg3CaDz/8UGbNmiW7d++WXr16qdCdrvDeIEEBHGjgSzZnTqKpEjGSuNSff57877/4QmT3bkdHHGi60MyGD5eIo3HjxDhbK1jXtGkgWkQIcYnhR9asWaPclZ2Xzp07q+/xf506dZL8plKlSkamTJmMEiVKGDNmzEiy3ylTphhFihRR2yDEZuPGjV61iyE1/uWrr1yHjZQsqUNQ3IGIKYSk2P0+Xbrkfx9unDplGLffrkNjrGEyMTGGceJEoFtHSOgS74eQmij8IxEGvH/hsIT5VZqCfQ8qwYwalailOhMXp83C7n7/5pv2qf9y5RI5e1YijqNHRV5/XWvxAN7AKGGHEB1CSPDIgqCaUyXhQd68rh2TMmcWyZrV/e8xF2s31EMcpjVDEZygliwRmT9fO0OFMzEx2msY54kFDk4UqIQEHxSqxOe0a6dDRpw9VjEH2KWLYziJHYirRBIFCFEsZmzm/fdrLRYg5SAcvlu1EmnbVguYESN8V/GFEEJSAs2/9AT2C6hZ+vjj2pMXwhSmYOTjRTUWT7sc/mzz5unqKvAGRv5eCFk4MSGpgZ15GI5Q7dv7/HQIIWHIeT+YfylUKVT9GlazYIH+/957ddFsV7GY3gBncKTdc56zhcCtWjX11V4IIZHBeT8I1aBKU0jCi9tu03GYvgbhOXZaKuZx9+/3/fEIIcRTOKdKQg4kjLerXwozM8zChBASKChUSciBpPXR0UkFK7TXwYMD1SpCCKFQJUEI4li3bnUdJlOkiE5XWLKko6l51iydfSjYgQPW+PEir74q8sMP9FgmJJygoxIdlZIFYSrr14v8/LNI7tw68QCSMPiaa9dEBg4U+eAD/Tc00cce05/tkkWgXcgJ/M8/OuF+pkwS9EyblljiDecHZyukGly4UMfw+hvUokC40i+/JOYfrl3b/8clJBih928Qd2S4ggQLLVuKfP+9nrOEMxBe/nPn6uLXvgTCBkLHmjgCx4TX8HffSciza5dIuXJJNVMI11de0ZqrP0HJYYQ1IUQJpnLE/0Koo8A6zeYkEjnPjErEF6Cw9T336CQMKC07erTrlIJIhbd6tf4bL2IIBGiGbdporcdXIOwGGqlzJiYcE6be7dsl5IF52i4xPs75o4/8f/x+/RIFKjCv+ZAhIgcO+P/4hEQCdFSKMFDwGsWrf/tNv1RR1/PFF0W6dbN/2X/4YVJBB8GK33pSccZT9u51LdjB77+LX4CZGXmGixXTSSkwJwsztz/AwCEl3yUHciEjdhcl9mAmR0YraKVWzp/X1ga7UCTEDmOgRQhJPRSqEcSff2ptBVgFJYQkhC0yFVlBNiRX9TqhcR0/7ru2JZfHFhq1r8F5Y84WplfEvl64IIJSvcje5FTW1yfUqGE/cEBfIjlGSs3zmBOdMkU7dkF4IoVj9eoi//d/idu5G7BAqOJaE0JSD4VqBAAB2revSJky+qXrCniiWsHc6Z132mdBQvFwZC/yFbffLvLww0nNo/hcurR/nGmgkSKdonWAYZq4Bw3y/fE6dNDF2q3niL7F8VJaJxaF36GVWjVQCNArV7QGbgIHM5j87eJ78duHHkrZ8QkhjlCoRgDw9pw6Nfntsmd3/IwXPpxnnB1rIBQgoJHM3pfMmKE1LCsQQkuX2guD1AJt1EzWbwVCFiE97gYgKSFbNpEff9TasSlYkawC+ZDr1k35OdgNeiBYly93XDdxoj6uec7m72D6r1AhZccnhDjCNIURAASqqRHZge+yZBFp3jzpd0hOD630pZf0/CuE2yOP6Ny7yVWb8ZY8ebT2iHCPnTv1PCeEjT8EKkAJOlcl6iB8/BGigxJu8JyGuRVLap3PIajRP3bnge+sPPCAzosMb1/0M+Zgn3nGfj6dEJIyGKcaASE1OXNqr09XAhWaC6rBQFi6Ai/tY8e0NhsuXQaPVySQcBZIEKjoCxQDCHYQamSX8AKCFmXyUmpWJiQSOM+QGpISKlWyD+UAKKe2Z497gWq+pOFMFC4CFUATnjRJ/42BBfoIgwycJ0ylzkDTx/wltGhXGm5a8+CDIr17J56DadqFU9QLLwS0aYREJNRUw0lKuNFmmjRxNP9CgODU4RGcN69ENAgvQgzpqVMiNWuKdOyYdH4ZTlzdu+sBiJkqEWZ1OFcFGlxXtG/+fB1DDM0VYVO+Ns8TEm6cZz3V4O3IYAfmXaQA/PvvRE0GCQeQ4Ye4B6EpcOTB3LKpoUKjhfa+YYNItWrsQUJCkfM0/5KU0ratjsWE+RL/b9xIgeopcMpC2IlzbC8E64QJvCcJIYnQ+zeCMGM+iXds22afPAHrtmxhbxJCEmGcKiEeODTZxbNikILvCCHEhEKVkGTo2dNeU4VJGJmqCCHEhEKVkGSoVUuXpLMmg4DmOmqUfcKMUAfzxZh3R5EDVwlDCCH2UKgS4gHIPITkF599posPILsUSqaFG8hmhbhmmLXvuEOkVCmdRpEQ4hmMU42QkBpCPMkwdffdOtbVOXRo/fqkeZkJCXXOM6SGEOIv3n1X5yO2Cx0aN479Togn0PxLCFGgMo9dEXM4af36KzuJkKAQqlOnTpVixYpJ5syZpUaNGvILJm1cULduXYmKikqyNGvWLGGbLl26JPm+CXLwEUJSXdPWLnQI5l+kZURMLuquwmELsya9eomcPs1OJyTNhOq8efNkwIABMnz4cNmyZYtUrFhRGjduLCdPnrTdfuHChXLs2LGEZceOHZI+fXpp06aNw3YQotbt5syZ48/TICQi6NHDPnQI5mCMa++7T+eRRrrGCxdEPvxQ5P77RS5fDkRrCYlAoTphwgTp3r27dO3aVcqWLSvTpk2TrFmzyvTp0223z507txQoUCBhWblypdreWahGR0c7bJcrVy5/ngYhEQGKCaBIgDURP+ZTX3xR12GFMLWah/E30l7CI5oQ4meheu3aNYmNjZWGDRsmrEuXLp36vAFZyD3g448/lnbt2kk2p2rLa9eulXz58knp0qWlV69ecubMGZ+3n5BIBGXkUHRhxgyRDz7QHsFvvCGyerX9fCuySqFCDiHEz7l/T58+LTdu3JD8+fM7rMfnPzC8TQbMvcL8C8HqbPp99NFHpXjx4rJv3z558cUXpWnTpkpQw1Rsx9WrV9VidaMmhNiDUoBdujiuQym8uLik20KTZVQaISHg/QthWr58eanuFBwHzbVFixbqu1atWsnSpUtl8+bNSnt1xahRo1SpN3MpXLhwGpwBIeFD167aYckZzME+8YTWYjHHinnXsmVF+vUT2b8/EC0lJEyFap48eZTmeOLECYf1+Ix5UHdcunRJ5s6dK926dUv2OCVKlFDH2oucai4YOnSoqp1qLocPH/biTAgh//ufTtcI4CFsegm/9JJe36GDdnTCzM7u3TqtY+XK+m9CIgm/CdVMmTJJlSpVZNWqVQnrbt68qT7XhEeEGxYsWKDMtU8++WSyxzly5IiaUy1YsKDLbeDYhMxJ1oUQ4jlZs4qsWSOyaJFO2fj88zp29fXX9Zzq/Pl6OzNXMDTYixe1k1NywHsYheDtzMuEhBp+Nf8inObDDz+UWbNmye7du5VTEbRQeAODTp06KS3SzvQL0+5tt93msP7ixYsyaNAg2bhxoxw4cEAJ6JYtW0qpUqVUqA4hxH9AO23VShdtHztWpEoVvX7pUvv4VpiE8Z2rpPz4/pVXRPLl03V+MZfbuTN8HngVSeji1yLlbdu2lVOnTsmwYcPk+PHjUqlSJVm+fHmC89KhQ4eUR7CVP//8U3766Sf5DgFxTsCcvH37diWk4+LiJCYmRho1aiQjR45U2mikgMTu778vsnmzCCzpTz2l57IICQR2c61WRyZXvPyyyJgxjtotwnNwf9s8/oSEBEyoH2Km4J07dcA9RvMY6UNDwMtowgRtkiMkrcE8qjnfagXO+Agxt8vNguQR0FCRvN8OZG/CnCwh/oQJ9YmKIzQFKjAz4Lzwgi5HRkhac++9ep4VmFFt0F4xe4Oas3bAr9CVQAXbtvmhoYREckgNSQpyXKxbZx+EDxPa4sXsNZL2wMT73nsiX34p8vDDIg88oOdKt2/XdVntwLSFO9NwTIzfmktI6M6pEt+CNHGuwAvq2jX2OAkMuP8efVQvngBnfeQT/vZbx0EiNN1ChUQsidgICSmoqYYQ8O8qV87eMQRJz5s2TXvNedAgXcEEL8I+fXSKO0I8ASnAK1XSf5taK+5xeAy7SI5GSNBDR6UQc1RCDlYzegjzqRCwEKgwuaEkF7TVFi10ID5Sy/kLzOtWqyayb1+ipgGnKYRFxMZqTYSQ5MC0BeJcd+wQQaIzlJazJvQnJNQclShUQ0yoAggthCLA6xIj+0uXEIqU+JLCqB+p4n7+WSRHDv+04a23RAYP1gLdCjSMAQN0HCMhhAQz9P4lCgTdI4MNsi3CGxj1CSBMzXg//I/0cBMn+q/DMBfmLFCtAf+EEBKJcE41xEHaOFdzrGbqOH+QJYvroH98RwghkQiFaogD4ekqDZydJukr2ra13z9Mz0iuTgghkQiFaojTvLn9esxttm7tv+O2a6fzwJrHMrVWpEuESZoQQiIROiqFoKOSFWSlqVsXRd0TNVYIOATdY51TTQKfgvnTJUtEFi7UnsgI/H/8ce2FTAghwQ69f4O4IwMJSme9+67IvHlauLVsKfLssyK5cwe6ZYQQErxQqAZxR/qT334T+fRTkfh4HY+KJOURVJSHBJirV3UKTMSSFi2qrREh8NgQkiwUqhEoVMeP11mLzHqV0ESRVQkB89REib85cECkXj39P5Iy4P7DI7N8uU6kT0goc94PsoCOSkHMrl1aoAK8zMyKNIhBfemlgDaNRAgdO+p4aDP3NObtUbYNKTFxb2LQh/qnhBANhWoQ8/nniRqqs4PQJ58EokUkkoB2+tNPSasiIZQqLk4nF/nf/7RJGBVqCCEUqkENNAJX5bGuXHEdn0qILzh92v33ELYQsLCgIDb51Cn2OyHUVIMYzGXZlXtDyEzt2u7rURKSWu66SyRbtuS3w+AO9+mCBexzQihUgxjEfdas6VgGy0y08OabgWwZiQQgUIcO9Wxb3Jdnz/q7RYQEPxSqQQzmU7/7TmTgQF1SLXNmkfr1Rdat05mLCPE3L74oMmmSSIEC+rMr6whMwLwnCWFGpaAPqfE1qLcKp5JVq7Qm0r49QyNI8mDuFDV0ly0TefJJLVzNOX1oqRCoa9dySoI4geD6kyd1sVxoBUEGQ2pIqh2fMBcLp5JZs3QWJpiXPTXxkcgFUw45c4o88YSeOy1TRq/HwKxPHy1sOcdPEsAIrHNnkTx5RO68UyRfPpFXX/VvlY8ggbl/I0hTHTJExxU6h0gAFDSvVSsQrSKhCjzQkdnLVQlAu8xMc+aIfPONTiSBgg8oyuDp70kI0bChNl1YXzYYdSHAfuRICRaYUSmIOzIUiImxD9TH3O0zz4i8804gWkUiAeSnbtBAZONGLUTxfsX7FoIVOautzngkxImNFala1f67rFlFTpwQueUWCQZo/iWp4tIl+/WYG7t4kZ1L/Mfbb+uqSQAWQFOBwfw+Q3HCjK1b3Y+u/vpLwhkaXiLMImOnEeAFB69iQvyZHcxuOg1aKzRVEmYmMVfARJE/v4QzFKoRxPDhutapc9xrhQq68ggh/qz768pKgrlWEkY0aiRSqFDSETw+t2hBoUpCF2gGqCby3HM61hUm3g0bdFIJeG3CMQ91V1HxJgi93UkYgXvO1bwpkvOTMCJDBu0OjuB6YLqFV64s8tFHEu7Q+zdMHZWQNu7RR0WWLnUsG9e7t3ZIYvgDSUuOHhW55x6dT9icT4WQRbQF5lqDxG+F+DooftkykSNHtDkMxaCD7MVD798g7shgY/Jkkf797ZPuf/211hwISUvwbh01SmTJEj3QQ+KRwYNFcuXidSCBISS9f6dOnSrFihWTzJkzS40aNeQX0wXQhpkzZ0pUVJTDgt9ZMQxDhg0bJgULFpQsWbJIw4YNZc+ePf4+jZBj5kx7gQrt4NNPA9GiMAVq17ffamkxe7ZrF2sit9+O94EWrigrhy6jQCXhhl+F6rx582TAgAEyfPhw2bJli1SsWFEaN24sJ5G2ygUYLRw7dixhOXjwoMP3Y8eOlcmTJ8u0adNk06ZNki1bNrXPf1x5QkRwdjBXMgC1MIkPQLwd5okeekjklVd0BhlIDkxcE6/AABDhNfBCL1lSO865GX8TErwYfqR69epGnz59Ej7fuHHDiImJMUaNGmW7/YwZM4wcOXK43N/NmzeNAgUKGOPGjUtYFxcXZ0RHRxtz5szxuF3x8fHQ4dT/4UqPHoaRIQNeVY5LunSGMXq0Edns22cYPXsaxh13GEbVqobxzjuGce2a9/tp0SJpJ6ODb7vNMP75xx8tD1tefTWx+/A/ujV9esNYujTQLSPhTLwfZIHfNNVr165JbGysMs+apEuXTn3e4GYkf/HiRSlatKgULlxYWrZsKTt37kz4bv/+/XL8+HGHfcIeDrOyu31evXpV2c6tS7gzaJBIlixJw2cQQta9u0Quu3cneiFi2gDZX/r1E3nsMe/ykqIiNyan4f1lBfs4c0Y7aBCP+PtvkddeS+w+gG7F3337RkS6WBJG+E2onj59Wm7cuCH5nQJ98RmC0Y7SpUvL9OnTZcmSJfLpp5/KzZs3pVatWnIEkzAiCb/zZp9g1KhRSviaCwR2uFOqlE4J17y5jk1FdrCOHfW63LklckHuUcx7msLQ1DG/+kqX7vEUuLHaTVpbTcPEI1De0E5wonsx90qXCRJKBFXyh5o1a0qnTp2kUqVKUqdOHVm4cKHkzZtX3n///VTtd+jQocq7y1wOHz4skUDZsiKLFungesiRGTN0THZEAw3SrqIA3FERf+QpxYvDTOL6+2rVJBJA1rnfftPaZkoxQ75S+j0hESFU8+TJI+nTp5cTTiN2fC5gVjxOhowZM0rlypVl79696rP5O2/3GR0drRygrAuJUNxlbvfm7Q2vdFTwtts/Msq4SigeJkCLfOMNXdGrUiXtn4XTTolwhZ8XqtbYpTDEwLBECZ80mZDQFqqZMmWSKlWqyCqLSQ3mXHyGRuoJMB///vvvKnwGFC9eXAlP6z4xPwovYE/3SSIczJ3aCU+Yg+vW9X7iesIEnZoKoA7a009rN9YwB6f98suOEURr1uhKNM7TzMlx2206rhqYlwZjE3Tnhx8GXb4AQtxj+JG5c+cqz9yZM2cau3btMnr06GHkzJnTOH78uPq+Y8eOxpAhQxK2HzFihLFixQpj3759RmxsrNGuXTsjc+bMxs6dOxO2GT16tNrHkiVLjO3btxstW7Y0ihcvbly5csXjdkWC9y9xweHDhlGoUKKbqXWJijKMxx4zjHPnvOu+69cN4+hRw7h8OSK6/d9/DSNPnqTdZy5LlqRsvxs2GEaXLoZRv75hDBignbQJ8Sf+kAV+na1o27atnDp1SiVrgCMR5kqXL1+e4Gh06NAh5RFscu7cOenevbvaNleuXErTXb9+vZSFDeg/Bg8eLJcuXZIePXpIXFyc1K5dW+3TOUkEIbbATolJQMzTv/eezkRgApmASeizZ71zWoJ69Z81JRKAjxYWV12xfbvOm+4t996rF0JCGeb+5fxqZHLunK6WgSTJdmzZokNvSBLg+IZMSFeu2HfOrFkinTqx40jwE5JpCgkJSuD85kqggt9/T8vWhBSY60Sss8XIpMBnCNvWrQPVMkICD4VqkLB5s07NhhRt9eqJfPGF+zBIkkqQBcOdB0yRIuxiN4wendTEi0pfKDWIsoK+BvGqI0aI9OypnZeYYpkEKzT/BoH595tv9AsK73h4TmLEj2B4pJM1M80QP9CyZdK4VbidYmSDzEvOqhhJwo4dOkcvQmsaN7YPjUktCxaIdOigB5m4JHhGEG/9448ixYrxopCUw9JvYVj6DcIT2Y8wEnfWTPECQT0B+NYQP4B0gq1aifz0U+I6XAxUncH/JCimvmFUwDyu9fnA2AfZSqEZExJMsoC5SgLMvn3Iaexa4K5YIdKtW+plB8q9QXDfdZeuY5k9e+r2GRYgQHLdOm17h8pVtKi2vVNDDRoWLxaxK0AF4wLSG+LexmUkJFigUA30BfBzijYoYU2b6nRyGN3DdAaz8tq1WsBGPLC5V6+uFxJ0oPaFOR3iDDTXCxcoVElwwUmjAIM5oXLl7JUjzE81a+ao1SLE8to1z/YN51YkEIJAxUsJn/EiwujenKMiJJipU8d1lRrUxaA/GQk2KFSDQFGCNyPCFMy0tKZ2OmmSzoAHy2SVKnqaD3lWMceEymXJsXq1Lpbi/FKC6WzbNpE//vDDCRHiQ3C/t2njOOg0/x4zhpZ6EnzQ/BsEIIsMysZOnSqydasegffoIVKrlk7ug9F6fHzi9tA0ESeYXExgXJz74yb3PSHBwGefiVSoIPLuuyInT2pBO2xYyrI2EeJvGFITBCE17pg4UWTgQHvPYLxcUGPbFahwB98bOzMvYglRgvaWW3zfZkIICQWYUSkCgenXrloZTLrQbt0BjRfB8tYcB+bfqDBCgUoIIb6F5t8gB4LRlUORJ/GrU6ZobRXzs9BM4Rg1ZIg2HxNCSEDZulVkxgzt/IEaxIgfzJ07pC8Kzb9Bbv5F8oc77tChMM7CFYLyuec83xf2kdoQHUII8QnvvSfSu7d+KZnelPDMRBwgXnppAM2/EQi0zIULHU21MOHiXuzXz7t9UaASQoKCo0cTX2AY7UOoYoEXZt++EspQbwkBHn5Ym26Rkg3B7vAGZs5TQkjIsmiRfQCymSoLoQk5c0ooQqEaImTNKvLoo4FuBSGE+IArV7TJzZXDiF1uyhCByR8IIYSkLQ8+6DpV1p13iuTPH7JXhEKVEEJI2pI3r+t6xsg96a7WcZBDoUoIISRtmTvX9XeoHOVpgvMghEKVEEJI2hIXZ5/VBkCghvCcKh2VCCGEpC21a+tQGmdg9kVNyhAu+ExNlRBCSNrSsKEWrFZtFQnN4Q385pucUyWEEEI8BgL02291ogczsw1KEX39tUjLlhLKME1hkKcpJISkHuQUQL6BxYu1MtS8uchjjzHLWFBgGNoUnDFjmh/aH2kKOadKCAlrrl8XeeQRkWXLEq2Nn38u8tFHel10dKBbGOFERQVEoPoLzqkSQsKaTz7RwtPUWLGA1au1YCXEl1CoEkLCGmilKfmOkJRAoUoICVuWLBFZv971VN7ly2ndIhLucE6VEBKWwLkUc6mucrZjfhUVoAgJKU116tSpUqxYMcmcObPUqFFDfvnlF5fbfvjhh3L//fdLrly51NKwYcMk23fp0kWioqIcliZNmvj7NAghIcbw4a7DHbG+YEGRZ59N61aRcMevQnXevHkyYMAAGT58uGzZskUqVqwojRs3lpMnT9puv3btWmnfvr2sWbNGNmzYIIULF5ZGjRrJ33//7bAdhOixY8cSljlz5vjzNAjxHKhFmzeLLFggsmMHey6AlyE21nUhlHLlRDBeR153QkImThWaabVq1eSdd95Rn2/evKkEZb9+/WTIkCHJ/v7GjRtKY8XvO3XqlKCpxsXFyWIEnAVRbBIhcuSIDlzfssUxc8z8+SK5crGD0pjbbhM5e9be7PvKK1qTJZHNeT/IAr9pqteuXZPY2Fhlwk04WLp06jO0UE+4fPmyXL9+XXLnzp1Eo82XL5+ULl1aevXqJWfOnHG7n6tXr6rOsy6E+BSMTSFQt293XL9mjchTT7GzA0CPHjpxj92l6tgxEC0ikYDfhOrp06eVppnfqdgsPh8/ftyjffzvf/+TmJgYB8EM0+/s2bNl1apVMmbMGPnhhx+kadOm6liuGDVqlBqNmAu0ZUJ8Cky+0FCdk4TjvoQL6uHD7PAUAu9dCMjWrUXGjxc5d86z3w0bJtKggf47QwYtYPH/zJkiJUrwcpAI8/4dPXq0zJ07V2mlcHIyadeuXcLf5cuXlwoVKkjJkiXVdg3MJ8iJoUOHqrldE2iqaSVY8Y5dtUrk2DGRypVFKlZMk8OStOavv1x/B9Xo0CERDua8ZtQokRdf1MIQ4xPM+kycKPLzzyLFirn/bZYsIitWiPz4o8gPP4jkyCHy+OMiBQp43w5CAi5U8+TJI+nTp5cTJ044rMfnAsnc1ePHj1dC9fvvv1dC0x0lSpRQx9q7d69LoRodHa2WtAaWQLjsW5UUOCrDh8XMIU3CBJSrcgVUpJIl07I1YcH//Z8WqMA0AGB8glfKCy+IfPFF8vuAl+8DD+iFkJA2/2bKlEmqVKmizLQmcFTC55o1a7r83dixY2XkyJGyfPlyqVq1arLHOXLkiJpTLQj/+CACdXYbNxY5etRx/cqVIs89F6hWEb8BE0TdukkLL0OgPvkk1aMUAKFpV8faTI5/9WpKLxYhIRpSA5MrYk9nzZolu3fvVk5Fly5dkq5du6rv4dEL06wJ5khfeeUVmT59uoptxdwrlosXL6rv8f+gQYNk48aNcuDAASWgW7ZsKaVKlVKhOsHE0qUimDp2nurFZ+Qipa9UGPLllyJNmzoK1CeeEHnvvUC2KmT55x/XcaYIlXHjRkGCka+/1pPcsNvDhAcnvnDE8DNTpkwxihQpYmTKlMmoXr26sXHjxoTv6tSpY3Tu3Dnhc9GiRRHek2QZPny4+v7y5ctGo0aNjLx58xoZM2ZU23fv3t04fvy4V22Kj49X+8X//mLSJMNIlw7GKvtl716/HZoEmgMHDGPdOsM4dizQLQlpfvrJ/tnBc1WrVqBbR7zirbf0xUufPvH/qCjD+PTTgHakP2QB66n6KU4VVm+L07ID2bOLIP+Fxf+KEJJkwC/Spo3IwoWJqQZhDoYBAErOffexy3wC1P61a0V27RIpWlQ7fviyFNu5c3r6A3NiziBcEl6cmTJJIAipONVIp149Pc3mPCcEc9bzz1OgEpIceFaQLO2tt0TKlhXJl0+kVSuRjRspUH0G5qjuuUebZZGzsUUL7VS3e7fvjrF2rb1ABcjOYU2WEgZQqPqrY9Npd37rVC80UySSQvwcISR5oDBhELpzp/b6hfMSZADxEchUh84FpjkA3pWY83SV49FbMiaj9YZRgXJA828apClE6mK8EO64Q5t+CSEk4CB2GuZeV6CKO0xuqeXyZW3+vXAhqeZRqJDI/v32bt5pAM2/IQruG4yuKVAJIUED5jLd4RwPmFKyZkUJssSUVgD/Y5k+PWAC1V/Q/BtAYG1BeE2lSlrgVqkiMnduIFtEvOKPP0QmTBCZPFnk4EF2HgktSpd27yCEF5OvaNtWz50+/bTIgw+K9Okj8vvvrr05QxiafwNYpebNN0Veekk7ZEDAYiCHaQzkNx04MGDNIsmBi4UMHlOmJGZsxzpcUA+qLxESNCB966RJjpXcoTki3hpxpWHOeT94/1KoBkiouvMyh7UETnk0FwcpMC/8V4rQb/NQhKQFyP+IOniwtmDuEyZZ3Ntvvx0RuVTPM6QmfED1O1de5ri3UUCZBCkffGBfUwwvpI8/DkSLCEkZuGdRteDUKT2dcfq0vod9IVA3bBBp317HFsL8iyoIEQDnVAMEtFF3ZMuWVi0hXgMHDrtwA4z6k3P+ICRYX0iYY0UpH18wZ44OJkYMFCqLIIPH/feLzJ6d+n3v26drFMfEiJQqJfLaayKXLkmwQKEaIGrX1uZfZ4UH86sQqLgHd+wIVOuIW1AQwvRitIK5qOrV2XkksvnnH+2IhHlas7wQ/sfnfv1ErlxJ+b737NEenZiCwQAWAnbECO3w5Mr0l8ZQqAYIvJM/+0w735mp1wDuO9yTqBlZvrzIO+8EqoXEJYMG6QtmHRHhImI0hJcJCXrwnCEzE3xxEEdOfMjGja4ryaOSyE8/pXzfplZqCmsAqxGOiYIWQQCFagCpX1/XjHz5ZR2DbVbkQPUN856Bk6m7+tckAGCOCDX8rLV+ob2iGvbtt/OSBDlIIFSmjL5kyMpXpIhIz56O72niJXiRIc3h/feLvPGG+21dlR7yhGXL7C8UtJTlyyWsi5QTzyhcWAtVRGNYvdqt9x9iV81izSRIQNXrrVt1ZQRoqbfdFugWEQ+A5RFpbuGPY1V04HuWN6/IyJHsRq/BYLJRIy3s/v3XfmrEBHO2mPtKKdHRKfsuDaGmmkKQcQvz79aHM6XgPrx+3f47CFXn7F4kiECWdwrUkAG+CkgZ6lyLFQNahB1TW/USdFz37no+0zp/amJmS4Kgxcvs3XdTV02kQwf7DEw45uOPSzBAoeoluHYvvKDfpbAC5s+vvcbj41N+EXCPVatmH6WB4zHskRDfAL8WV4oUnmFXU4HEBXv3ivz5p+vk+/feK1K1qg6pQYgNhGJqgFkPXsoQ0FhMAQvBDhNEEEDzbwp8VBAXbZpqcS8tWKDDvL7/PuUXYvRobUHBPWKOoiFk69QJy0xehASEO+90rY3mzCmSK1datyjEcVb5nUFawi5dxGfgAiGIH96/eOEiFKhdO50BKjVztT6EGZW8yKgUF6c1U1ee20htWblyyi/GunUir76qneNQDQkvADgqYXAXoBq+xFfA63H+fJ0j+O67RR55JGjmgCKJq1d1uVBkLLPKA7yP8eyxLKOXQKtAh+K+dnYKSZ9e5PBhkYIFJVhhRqUgMB25C4XCHGtqfV/gEYz5VRwHcapdu4rUqKEFOglRNm3SLqY9eoiMGaPnC2DCOnAg0C2LODCOQSZJjGtMYA6G4yrycBMvgTkN86QQoKZdPf1/JlnEj5oCFQIXoTC+qtEaxHBO1Qtwf7izMKQ2mgLOpEjDCUxHOoBiDrg/SZCDFwbMFRCi5ugLI6RWrbS3GV4spkfakSMiHTsGtLmRCixA27bpBZFRSJCFnPJhVoEs7YDpFXGicBQqU0bPbS5ZokcpuOcRbI8wB6Q+zJNHZPhw156ZYQDNv14m1G/ZUodKWU1HeBgRZ4pkH3bORt7M1+LhtpvzwVTC2bMp3zfxM2vWaLOCWQIOHsGYfM+dW+Shh1z/DkHIxYvz8pDw5M3/SnFZgWbSubPIjBkSaGj+DQJQU9c5Ex0GYRC0qRGo4OJF198FUWpLYucBidE65o9MzpzRmihi+Nzhi5gsQoKRixftE0FAe501K2yz2tD86yVQQFBsYf16HTCOJB54p8LqkVrq1rXXUqEJM6wmiMGcEkwXzvNFGGXhRnEF0hqWLWv/3Q8/6El2eKjBZDZ4MEdWJPRSV12+7D5PZBhCoZoCYL1AijOERjVu7Lu5mEcf1SFd1v3hbyxIeUmClF277EdDELT792v3bTszxtCh9uWIVq3SHmsYvWHuCVrvW2+JNGmSfAgDIcHCbclkGcPUSBhCoRpEIIwG79P+/bVygqQQiF1FiA2LnwQxmBO1yygAQYoRETxhEHaQJYtej/JEmDx3lXvSXG/VfPE3boQVK/xxBoT4nlKldOiCs9aB5wKxiUGSrMHX0FHJS0clQpLw228i99xjHy4AswZMXWZWD4TTIHDdlXkDXsOu4lcx6sKIa+xYXgQSGuzdq+e1UAoI9y8sOtmz63kzmPsCDB2VCAlGkK8SdfwQMmAVphiRmwHxptkWxZuR5cMV0Hhd5UaF0OYgkISatrpnjy5OPnCg9j84dCgoBKq/oKbKlxTxFXDRRmYBaJsIr7GrhACh+cwz7gvlduumvSOd508hqPGCgimZEJJqqKkSEixAA120SMegwvTbr5/Ofde8uUjr1u4zxyTnbISsS8i4ZAphs8IHyqhQoBIS1DChPiEp4X//Exk3LnGuFGmvEMwO0y6ELIQrKi04C1DMKT38sPt9w0sNmZm++EI7J8FL8sknRe66i9eKkEj3/p06daoUK1ZMMmfOLDVq1JBfUGHADQsWLJAyZcqo7cuXLy/ffPONw/eGYciwYcOkYMGCkiVLFmnYsKHsgUmMkLRi924tUIEpNCEs//lHV0AAqHaNqQXTK9gsVdWsmU4UkRxwVnriCZH33tMB9BSohIQEfhWq8+bNkwEDBsjw4cNly5YtUrFiRWncuLGcPHnSdvv169dL+/btpVu3brJ161Zp1aqVWnYgs/x/jB07ViZPnizTpk2TTZs2SbZs2dQ+/8ELjZC04Ouv7eNOIWChWaL6ARw0kMy5Vy9tykUA8uTJ2mSc2tRbhJDgxfAj1atXN/r06ZPw+caNG0ZMTIwxatQo2+0ff/xxo1mzZg7ratSoYTzzzDPq75s3bxoFChQwxo0bl/B9XFycER0dbcyZM8fjdsXHx8MlU/1PiNeMGWMY6dJhVtV+OXeOnUpICOAPWeC3IfO1a9ckNjZWmWdN0qVLpz5vQAV4G7Deuj2AFmpuv3//fjl+/LjDNjly5FBmZVf7BFevXlVeXtaFkBTTooW9IxLmV2vX1tWuCSERid+E6unTp+XGjRuSH5kzLOAzBKMdWO9ue/N/b/YJRo0apYSvuRRGBnxCUgoSPSMXLzCTOOB/xJeiMg0hoQDsKsj2depUoFsSVkTE5M7QoUMlPj4+YTlsrSZCSEoYPVrPjyL5c5UqIr17J2ZWIiTY+f57kXLlRAoVEsmXT+T++7UHOwnekJo8efJI+vTp5cSJEw7r8bkAcp/agPXutjf/xzp4/1q3qVSpksu2REdHq4UQnwFPXhQfx0JIKPHrr9oD3RruhekzVEVCcQjLu5UEkaaaKVMmqVKliqxChvj/uHnzpvpc00WKKqy3bg9WrlyZsH3x4sWVYLVug/lReAG72ichhBCn5CLATKEJIGCRAWzaNHZVMCd/QDhN586dpWrVqlK9enWZNGmSXLp0SboihZuIdOrUSQoVKqTmPMFzzz0nderUkbfeekuaNWsmc+fOlV9//VU+QOFSpRxESf/+/eX111+XO+64QwnZV155RWJiYlToDSEk/DhyBPHuIghxhxL19NM6RztJIZs2uS5VuHkzuzWYhWrbtm3l1KlTKlkDHIlgol2+fHmCo9GhQ4eUR7BJrVq15PPPP5eXX35ZXnzxRSU4Fy9eLOVg+/+PwYMHK8Hco0cPiYuLk9q1a6t9IlkEIWHJuXO6woc1YX+EsH27nu5DWmW885FLA7ULMKWNpFYkBWBkgqoxzh7s6FyaflMNE+ozoT7xBDi3vfyyyJdf6pcRwmpef10nefAXa9bA3COybZuew0WRciSQ8OcxgwwIVEz3OWd7xFgc9d+LFAlUy0IYpNN86in779avD+sKMs4woT4hgQAhByi2DBUJKtOVKzovL9b5y5Mctk5UqIeqZs5/ffedyH33iZw5I5EAThMJquzqD6A7Fi8ORKvCgC5dRPr0SRydmGUKJ06MKIHqLyIipIaQVIEakEitaX274+/4eJEJE/zTuW++qSWH1USHY54+LfLRRxIJuCvmAzlgnRZcu1YbD4oX12ORZcvSpImhCToPpQeRw/qtt/TfqHHav3+gWxYWsEoNIZ7E9Nm94bFu5Ur/9J+dzRNA0CZTlCJcQPgkIuWgrDtP/+EzahMAGBA6dtTKFroMxgNclkmTEusbEBdJTLAQn0JNlZDkyJHDPgk+Rvz4zl8SBft3Bpmb8uaVSAEJqsySssC8DM8+q+sUXL2q/8ZYwxyDmP8PGaJrGxCSllCoEpIcUIPscv3iTd6pk3/6r0cP+/Wwef4XkhYJIB8BojzattX12WvVEvn0U62FApSdPXvW/rcoXAWzMCFpCc2/hCRHmzYi334rMmtWosoE4fbIIyLduvmn/5D2EPGEsG1COzUD9ceP1w5SEUSFClqQ2mFeDlcgEomQtIQhNT4OqcG7D04SeAnAj6VOHa105M7t08OQtAYX9ocfdL5f2BfhFYNqSf6ujRobq71+EYf96KMiRYv693ghBi4FugR54a0JgmA5z55d5NgxkaxZA9lCEmkhNRSqPhSqeKj79tXOolAu8MDjnRsTI7Jxo85dTQhJHf/3f/oZg/NqiRJak4VDEp4/GBCgvcJa//nn2mxM/AQ8hv/4Q6RYMZE77wzJbj7vB6FK86+PHTbxsFudJfBwY7SMvAGIuSaEpBx49T78sH6urAIUVvEDB0R27tRzr716ac9h4gcuXtTJIxCrbZoH6tcXmTNHO9hFONRUfaipDhyoE97YpdWECQp5AwghKQMDVShFyLBHU28A6dBBZP58x5AvjG6qVxf5+WcJJZhRKcixE6aefEcISR54+iK5vlWgAnw+f56evmkCzG5z5yaNocYLDikOt2yRSIchNT4EJQrthCfmVx96yJdHIhEFbJovvqgn7GFyu35dIpHkTjtCuyVt+euvpKMa5wnvCIdC1YcgPRrme6wx+7CKwPT7xhu+PBKJGDBZiCpN48aJoAQiwnsQrAnX8gijShWRXLnsv4uO1p72xGmU4U4ApgTkgbRLSmJSsmTEXwIKVR8CT9+FC3VgeuXK+v7DfD4sImXLRvy9RuzAS++rr0RQD7h2bV3PDDZOsGOHyKBB+m+YQExVbOtWkVdfjbj+hOCEzwLe6bD+APN/lGTOmTOgzQue++n99/XLJ1MmkQIFdOe4S6QMTpwQmTJFZMQInZbTLtkJQCjDY48ldrzznGrVqr47l1DFiEDi4+MxfFP/ExJQBg7Ea9Aw0qdP/D9XLsPYtcswhgwxjAwZ9HrnJWfOiL1wq1cbRrNmhlG0qGE0aGAYX38d6BYFEaNGJb1XoqIMo2dP17+ZO9cwMmY0jHTpEu+3++83jAsX7LfHe7N5c8dj3HefYRw9aoQa/pAF9P5lPVUSKH7/XQdZOgMtAHMJ0DY+/NB+shCaAScRiXOoS/78IpcvJ+0XqPeIOXIuQAurCO4zZ2cQ3IOYwzfzQdqxd68OFoZLdvnyIXktzvshTpXmX0ICBQqCOpvRAEx1y5eL3HuvveDEb1C9mxDnQZqdQAXQJxFI7wwyZNiZenEPosSgKzMwKFVKpHnzkBWo/oJClZBA4e6FhZcgUiFict4qeM2i0q+/niZNJCGEKy8ud9+jPq/dwA4gsJ6xgF5DoUpIoICruJ0DCV5yyFCDsnKrV+vk+vgb61G2BaVX4AFMiBXUwnMehJkDMZiF69VL2l9wLrKzhmDgBq9zODsRr6BQJSSQMSLdu//3JKZLnCtF8vy33tKf4dIKl1cUBoXWsGaNyH338ZoRe0GIqka33ab/Roke/I+Yvi+/tC/Z07KlFp5WQYzfwFLy2mvs5RRAoUpIIJk2TeSTT7QGirgrCNlt25i4lqSMu+4S2bdPxzT366ervB886HogBkGLgVr79olCF3OlCxbo0obEa+j9S+9fQgjRVd2vXNHWEXcJHsKI86xSQwghxC9g2gELSRU0/xJCCEkelAdC7CoKQ6My/ODBImfPsuecYD1VQghJ61AqzGOiyDdiPOGwFuzmVlSnqVZN5OTJRI/1CRNEliwR2bxZhNNoCVBTJYSQtAJVXBD60rChTgwOQVW3rsi5c65/A0/cq1dTlhz/1CmR558XKVhQFxDv0UMLc2+BALUKVIC/9+zRSSJIAhSqhBCSFkBDbdZMZP9+x/Uo7P3000m3hxCFF2+JEnquM29enfDe04QMqGSEeGYkyj9+XAvYGTO0IIcp1xuWLbOPqUYbv/3Wu32FORSqhBCSFsDki3y5zsIJnxct0oLPCmKVn3lG5+wFZ85ooWrGNicH8kaj/qn1eBDI2I8ZB+0prhyYEF+dJYt3+wpzKFQJISQtOHzY9XfQ+KzaI3L42iVfwHYzZ2rhnBwrVrjO6/vNN+IVHTrYz/ti/23berevMMdvQvXs2bPyxBNPqMz/OXPmlG7duslFVFFws32/fv2kdOnSkiVLFilSpIg8++yzqnqAlaioqCTL3Llz/XUahBDiG5C5yBVIB2gt8I3qLxcuuN7eLjm+M8ikZGbqsvvOG/r0SUwggX2aGZhQB7hdO+/2Feb4zfsXAvXYsWOycuVKuX79unTt2lV69Oghn6Mqgg1Hjx5Vy/jx46Vs2bJy8OBB6dmzp1r3xRdfOGw7Y8YMadKkScJnCG1CCAlq4OVbp47ITz85mmShAfbs6VhlPSXJ8Z2BsPvqq6Trcbwnn/Sm5drEu2qVyPz5IkuXaqHaurVOc+gqIX+kYviBXbt2qcKvmzdvTlj37bffGlFRUcbff//t8X7mz59vZMqUybh+/XrCOux30aJFqWofi5QTQgLC2bOG0bq1LhyO12+mTIbx3HOGce1a0m3vvTexeL25oJB4njyGcfVq8sf691/DaNNG/w7Fx8191a1rGFeu+OX0Qg1/yAK/mH83bNigtMeqVasmrGvYsKGkS5dONm3a5PF+zMKxGZBk3EKfPn0kT548Ur16dZk+fToGBm73c/XqVZWOyroQQkiaAw0TlrejR0ViY0VOnNCFwO2S3SMnNKrLADM5PjRGJMf3pHoMNEhMjUFbhWaK/L74/N13zJwUaubf48ePSz7ERFkPlCGD5M6dW33nCadPn5aRI0cqk7GV1157TerXry9Zs2aV7777Tnr37q3majH/6opRo0bJCHjNEUJIMFCggF7cgcT2iAOFyXXHDp3FCA5DqELjKZj/RCFxLCT4hOqQIUNkzJgxbrfZjQn2VAJNslmzZmpu9dVXX3X47pVXXkn4u3LlynLp0iUZN26cW6E6dOhQGTBggMP+CxcunOp2EkKIX4FDUZcu7ORwFaoDBw6ULslc4BIlSkiBAgXkJLJvWPj333+Vhy++c8eFCxeUE1L27Nll0aJFktHOLGKhRo0aSqOFiTc6Otp2G6x39R0hhBASEKGaN29etSRHzZo1JS4uTmJjY6UKPN5EZPXq1XLz5k0lBF0BDbJx48ZKAH711VeS2YOKCdu2bZNcuXJRaJLwx8xeAw96hKfVry/StatI9uyBbhkhxJ9zqnfddZfSNrt37y7Tpk1TITV9+/aVdu3aSUxMjNrm77//lgYNGsjs2bOVwxEEaqNGjeTy5cvy6aefOjgUQZCnT59evv76azlx4oTce++9SuAiXOfNN9+UF154wR+nQUhwCdTevXVRczigIOgeDihIQbd+vU5hR0IHhKfMni0SFydSu7ZOU+hJmAwJfgw/cebMGaN9+/bGLbfcYtx6661G165djQsXLiR8v3//fuXKvGbNGvUZ/+Oz3YJtzbCcSpUqqX1my5bNqFixojFt2jTjxo0bXrWNITUk5Fi71jG0wlwQJtG7d6BbR7xh6NDEMBczTKZQIcM4dIj9mMb4QxZE4R+JMPxR7Z0Qv4I6lu+/b59MHRoO61qGBtu3i1SsmHQ9wgYfe0xkzpzk9wHT/8KFuhxb5cq64o2rzEkkzWUB66kSEgpcu+b6u+vX07IlJDUsWKAFqPPgCJ8RfwqzvjsBiWxMDz+sK9BgGgCZme65R8eeehNqQ/wGhzeEhAJIy2mnpeLFinJiJPQHR7i+dgnwrUn2W7RIzAlspjr87TeRXr183FCSUihUCQkF8DJFMWurFgONJ1s2EadY7jQH1VXgMDVunH7Bk5QNjho00NfUFXBMQzFzZ8EL4Qot112hc5JmUKgSEgrgZYtwmjffhHu9yO23i3TurFPdlSkTuHZNnSpSpIhI//7IDiNSqZJOVmBX0JrogRGS0CPloFlKDdcWaQeTSayjUhq6Mg1D0HJePSigoxIdlQhJGRDolvzeiW+VKJG33xbp148962oOHIOR6dO1IKxXTw9I7r7bfX8hdMosv+ZM7tzaccmTnMDEr45K1FQJISkDQsGVuRKeysQeZImDZg9P4CNHdOL85AQqqFlTl46zK7X20ksUqEEChSohJGUgFamdmRdReh4WziBeAAsA5lVh9jfTtyLpB6rcPP88uzJIoFAlhKSMatUS5wWtQJO69172qj+AifLjj7VTErRclJB77jn760ACAoUqISRldOumYyOt5kjz5f7ii+xVfwKv70KF3HsLk4BAoUoISRkQqEhGgFAQE3gmf/ONSK1aodOriP8cP15r3kikgNrL9KQlKYTev/T+JST1IDH81asi+fKFlinyyhUd5vLrr4nxnwhbKV5cZNMmZikKc87T+5cQEpTkzCmSP39oCVQwa5bI5s2OCRXw94ED2gGIEC+h+ZcQErnAm9YOM0sRIV7CWW5CiGdm0sWLRQ4fFilfXqRRI/t4yVADpl5o13bFuiKvgBfxARSqhJDkMyc1bSpy6lRiZRQkK/j+e5ECBUK79x55RGTZMvvvELaCMmu33JLWrSIhDM2/hBD3KfWaN0/0hjWTPfzxh0jXrmnfc5jv/P13kW3bfJNfuGNHkRIlXOfaZWYo4iUUqoQQ15pajx46p6yzAMPn5ct18oG0AppxyZIiFSro4txI5L9kSer2iVy5yErkCs6rEi+h+ZcQkhSkGURWpEOHkk9VGBPj/x7cvVvkoYcchTuE/aOPimzYIFK9esr37W5uONS8mUnAoaZKCEnKa6/pNHjunHWQ1eeOO9Km91CvFW2xhr7gMxyNJk5M3b4hmO2EJ9bhO0K8gEKVkEjg/Hmd6WjlSp2kITnmzUt+znLQIC1Y0wJUdLEr7o11mF9NDT17anOyVbDib6zDd4R4AYUqIeEOanfCS7dZMx0KU7CgyKJF7n/jTqBiHhLF0l95RdIMZDiyy3ML060rRyNPwcBg3TqRt97S9Uqx4G+sS6tBAwkbmKaQaQpJOAPtFMLUCrQwmE2h4ZUrZ/+7Tp1E5syx1w4XLBB57DFJU9wV6MY5IuTHExAWNHmy/k2WLCLt24s8/bRIdLRPm0tCA6YpJIR4B+YbnR1xMBcJwfree65/9+qrIjlyJP4W22OBpovYzrQGCfo//FAkc+bEdagpOmGC5wIVzldVqoiMGiWyZYsW1P36aQcohA4R4gNo/iUknNmzx96UCw10717Xv4NJdetWkb59RUqX1tVbkAv3668Dl0kJGiUEIzTluXO19683xbkhTBECZPYHBhdYVq9m6AzxGQypISScKVtWe/E6C1bMT0JYuqNw4eBLKg/tOaWmZ8Sc2g0wMEhAvGu7dqluHiHUVAkJZwYMSCpITC/XXr0konAXc8p4VOIjKFQJCWcaNhT5+GOt4Zmg5imqs6CgeCTRpo296RqDjlatAtEiEobQ+5fevyRSqsz88ot27kH2IbvwFE+BOXntWh1u0rixSNas4jcQV/vuuyIHD+p5XcyhIpl/SoDnb40ael9IImFWp4GjEwYZqekTEpKc90ORcgpVClVCPAMCaPBg7XFrZjbKnl3kk09EWrb0fS8iVvSFFxIr40DoQRAi53D9+inbJwoDwOvZDKnBPGrnznqwQSKO834Qqn4z/549e1aeeOIJ1dCcOXNKt27d5CLKKLmhbt26EhUV5bD0dMpocujQIWnWrJlkzZpV8uXLJ4MGDZJ/7WLpCCG+BSEt48c7pgrEMw3HoX37fHssVIgZMkT/bc4J4znHgndCSmud5s4t8tJLIj//rBP0w6OYApX4EL8JVQjUnTt3ysqVK2Xp0qWybt066YGKF8nQvXt3OXbsWMIyduzYhO9u3LihBOq1a9dk/fr1MmvWLJk5c6YMGzbMX6dBCDFB0gRnhx4zLAXztr4E2qjdYBnHQpgQFkIiRaju3r1bli9fLh999JHUqFFDateuLVOmTJG5c+fK0WRKRUEDLVCgQMJiVcm/++472bVrl3z66adSqVIladq0qYwcOVKmTp2qBC0hxI+gYo0rDTG5ajYkKfHxIm+8oRNSYEHqR+RoJiGNX4Tqhg0blMm3atWqCesaNmwo6dKlk02bNrn97WeffSZ58uSRcuXKydChQ+Xy5csO+y1fvrzkz58/YV3jxo2VXRxasSuuXr2qtrEuhBAvQUpDpDd0BoI2pc5DrmjSxN5xCMdHZZy0qo7jL/AOQpYoWNmQ3QkLcikjFeOFC4FuHQk2oXr8+HE132klQ4YMkjt3bvWdKzp06KC00DVr1iiB+sknn8iTTz7psF+rQAXmZ3f7HTVqlJqMNpfCCGonhHgH5jit86kATkRwVurWzbe9iecaGZDMYwAIWfwNR6NQjyuFR/Mffzj2J/7etUtk2rRAtoykpVAdMmRIEkci5+UP3CgpBHOu0DyhjWJOdvbs2bJo0SLZl0onCAhoeHeZy+HDh1O1P0KCljNnROCH8PjjOq8tUg36ihYt9NzpbbclrkOs65o1OvbV18Dz97vvRJo312XY4KULja5BAwl5Fi9OOkABWIfvSMjiVWDWwIEDpUuXLm63KVGihJoLPXnypMN6eOjCIxjfeQrmY8HevXulZMmS6re/INbOwgl4CYq43W90dLRaCAlr4LwD8yEEq2kqfecdrRX5KnvSU0+JwHqE6RbEp955p3+1xgcf1Eu4YWdG9+Q7El5CNW/evGpJjpo1a0pcXJzExsZKFUzAC3JWr5abN28mCEpP2PZf8eGCqP/4337feOMNJbBN8zK8i+HMVBY5TgmJZHr31nGYpgZk/g+NFXGkMTG+OQ7qqUJzJCkHYUgbNyZ1/MIApXVr9mwI45ch0V133SVNmjRR4THQLH/++Wfp27evtGvXTmL+e7D//vtvKVOmTILmCRMvPHkhiA8cOCBfffWVdOrUSR544AGpUKGC2qZRo0ZKeHbs2FF+++03WbFihbz88svSp08faqIksjl3Tsdd2iWMh3BNrig5SVsQawtHTrOknrlUq4Z5MF6NEMZvebngxQtB2qBBA+X127p1a5mMOLf/uH79uvz5558J3r2ZMmWS77//XiZNmiSXLl1SzkT4DYSmSfr06VXMa69evZTWmi1bNuncubO89tpr/joNQkKDq1ddf4eXNdIUkuABpnOkepw+PXHAgzq1cPhCpicSsjBNIdMUknDADGuBo6BdLOlvv4n8Z/EJa3Du0NhR5g3JI1CAHKbvQNWAJRGXppAZpAkJB6CNIidvs2ba0QVmYNOBqFOnyBCoMHPDkWrWrMQYV3grw1t42TJ4LAa6hSQCoJsZIeECEib88AOcD3SOW4S7vP2271MIBisIRYFAteYJBqtXay/omTN1n0DglighMmWKfVgLIamA5l+afwkJD+A1axf/CY0dEQRIkWqWezMZOFAXCSARyflQqlJDCCFpyqVL9ponhOixY4l/W5k0KfE7QnwAhSohJDxo2NA+cQLWuSoEgLnn9ev93jQSOVCoEkLCA9RGLVrU0dMX86e33OL+d5wCIj6EQpWQSGXvXpHnntOpDdu10zl8Q5mcOVHKSuSZZ3R+YgjL9u1FYmO1g5JzWA00WGRmq1s3UC0mYQgdlThKJZEITJ4wl16/rr1kIXBgCkVYzvPPS9iBlKf164vExWntFeeaObPIN9+I1KmTuB3SPM6di7JXusYpQpTsStCRsOC8HxyVKFQpVEmkgfnFihV1Uny7Um5HjqBChYRlKsfZs0V279YhNah6Yy0liYo4yGqE7FPoBww2kFADITn+qMJDAg6FahB3JCEhw8GDIsWKuf7+o498Xx81FIqGIy850qZanZqgpbZqJbJgQSBbR/wEQ2oIIanHLum+lUhMiID8uwjJcfYShra6cKFIfHygWkZCDDoqERJpFC/uug4qnHeaNpWIAzVoXdUxxSADc7GEeACFKiGRBoQpCpdj3tB0wjE9Y4cNE7n9dok47r3XtYaO+eVI7BOSIihUCYlEkGT+1191yAnCTeAZCxPo8OESkdSsqfvETlsdMYJVbojH0PuXjkqEEIA51SFDdAECeAAjkQQGGV27sn/ClPMMqQnejiSEhAmI3YWAzZHDft6ZhA3nWU+VEEL8TMaMOjsTISmAc6qEEEKIj6BQJYQQQnwEhSohhBDiIyhUCSGEEB9BoUoIIYT4CApVQgghxEdQqBJCCCE+IiKr7xr/VaJA4C8hhJDI5Px/MsCUCb4gIoXqhQsX1P+FCxcOdFMIIYQEgUxAlj1fEJG5f2/evClHjx6V7NmzS1QapyHDyAjC/PDhw2GVIjEczysczylczysczylcz+t8EJ0TxB8EakxMjKRzVfrPSyJSU0Xn3R7gUk64mQJ9Q/mDcDyvcDyncD2vcDyncD2vW4PknHyloZrQUYkQQgjxERSqhBBCiI+gUE1joqOjZfjw4er/cCIczysczylczysczylczys6DM9JIt1RiRBCCPEH1FQJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQjUNeOONN6RWrVqSNWtWyZkzp0e/gf/YsGHDpGDBgpIlSxZp2LCh7NmzR4KFs2fPyhNPPKGCt3FO3bp1k4sXL7r9Td26dVUGK+vSs2dPCSRTp06VYsWKSebMmaVGjRryyy+/uN1+wYIFUqZMGbV9+fLl5ZtvvpFgxJvzmjlzZpLrgt8FE+vWrZPmzZurzDdo3+LFi5P9zdq1a+Wee+5RXqalSpVS5xnK54Tzcb5OWI4fPy7BwqhRo6RatWoqW12+fPmkVatW8ueffyb7u1B5rjyBQjUNuHbtmrRp00Z69erl8W/Gjh0rkydPlmnTpsmmTZskW7Zs0rhxY/nnn38kGIBA3blzp6xcuVKWLl2qXhA9evRI9nfdu3eXY8eOJSw4z0Axb948GTBggHLv37Jli1SsWFH18cmTJ223X79+vbRv314NILZu3apeGFh27NghwYS35wUwOLJel4MHD0owcenSJXUeGCx4wv79+6VZs2ZSr1492bZtm/Tv31+efvppWbFihYTqOZlASFmvFYRXsPDDDz9Inz59ZOPGjerdcP36dWnUqJE6V1eEynPlMQipIWnDjBkzjBw5ciS73c2bN40CBQoY48aNS1gXFxdnREdHG3PmzDECza5duxCGZWzevDlh3bfffmtERUUZf//9t8vf1alTx3juueeMYKF69epGnz59Ej7fuHHDiImJMUaNGmW7/eOPP240a9bMYV2NGjWMZ555xggmvD0vT+/LYAH33qJFi9xuM3jwYOPuu+92WNe2bVujcePGRqie05o1a9R2586dM0KFkydPqjb/8MMPLrcJlefKU6ipBiEYZcOkA5OvNT8lzHgbNmyQQIM2wORbtWrVhHVoK3IqQ6t2x2effSZ58uSRcuXKydChQ+Xy5csSKOtBbGysQx+j/fjsqo+x3ro9gAYYDNckNecFYLovWrSoSnTesmVLZYUIZULhWqWUSpUqqWmhBx98UH7++WcJZuLj49X/uXPnjphrFZEJ9YMdc44kf/78DuvxORjmT9AGZ5NThgwZ1IPjrn0dOnRQL27MIW3fvl3+97//KVPWwoULJa05ffq03Lhxw7aP//jjD9vf4NyC9Zqk5rxKly4t06dPlwoVKqiX4Pjx45UPAARroAtPpBRX1woVUq5cuaL8FEINCFJMB2Ewe/XqVfnoo4+UnwIGspg7DsZqYP3795f77rtPDaJdEQrPlTdQqKaQIUOGyJgxY9xus3v3bjX5Hm7nlFKsc65wRsBLokGDBrJv3z4pWbJkivdLUkfNmjXVYgKBetddd8n7778vI0eOZPcGCRj8YLFeJzw7EydOlE8++USCjT59+qh50Z9++kkiCQrVFDJw4EDp0qWL221KlCiRon0XKFBA/X/ixAkleEzwGaafQJ8T2ufs9PLvv/8qj2Cz7Z4AczbYu3dvmgtVmKDTp0+v+tQKPrs6B6z3ZvtAkJLzciZjxoxSuXJldV1CFVfXCg5ZoailuqJ69epBKbT69u2b4MCYnLUjFJ4rb+CcagrJmzev0kLdLZkyZUrRvosXL65uqFWrViWsg9kKZh6rRhGoc0Ib4uLi1NydyerVq5W5xxSUngCvTGAdOKQVOI8qVao49DHaj8+u+hjrrdsDeDj685qkxXk5A/Px77//HpDr4itC4Vr5AjxDwXSdDMNQAnXRokXqnYB3WcRdq0B7SkUCBw8eNLZu3WqMGDHCuOWWW9TfWC5cuJCwTenSpY2FCxcmfB49erSRM2dOY8mSJcb27duNli1bGsWLFzeuXLliBANNmjQxKleubGzatMn46aefjDvuuMNo3759wvdHjhxR54Tvwd69e43XXnvN+PXXX439+/er8ypRooTxwAMPBOwc5s6dqzyqZ86cqTyae/Toofr8+PHj6vuOHTsaQ4YMSdj+559/NjJkyGCMHz/e2L17tzF8+HAjY8aMxu+//24EE96eF+7LFStWGPv27TNiY2ONdu3aGZkzZzZ27txpBAt4VsznBq+tCRMmqL/xbAGcD87L5K+//jKyZs1qDBo0SF2rqVOnGunTpzeWL19uhOo5TZw40Vi8eLGxZ88edc/Bkz5dunTG999/bwQLvXr1Up7ka9euNY4dO5awXL58OWGbUH2uPIVCNQ3o3LmzemicF7jIJ1wIERXaYA2reeWVV4z8+fOrF2SDBg2MP//80wgWzpw5o4QoBgm33nqr0bVrV4dBAgSn9RwPHTqkBGju3LnV+ZQqVUq98OLj4wN4FoYxZcoUo0iRIkamTJlUKMrGjRsdQoBw7azMnz/fuPPOO9X2CNlYtmyZEYx4c179+/dP2Bb320MPPWRs2bLFCCbMcBLnxTwP/I/zcv5NpUqV1HlhAGd9vkLxnMaMGWOULFlSDXjwHNWtW9dYvXq1EUyIzfk4v9tC+bnyBJZ+I4QQQnwE51QJIYQQH0GhSgghhPgIClVCCCHER1CoEkIIIT6CQpUQQgjxERSqhBBCiI+gUCWEEEJ8BIUqIYQQ4iMoVAkhhBAfQaFKCCGE+AgKVUIIIcRHUKgSQggh4hv+H/SXy5gKmyalAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 500x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def make_moons_fallback(n_samples=100, noise=0.1):\n",
" n = n_samples // 2 # half the points per moon\n",
" t = np.linspace(0, np.pi, n) # angle along each crescent\n",
" # outer moon — upper arc (cos, sin)\n",
" x1 = np.stack([np.cos(t), np.sin(t)], axis=1)\n",
" # inner moon — lower arc, shifted down so the two curves interleave\n",
" x2 = np.stack([1 - np.cos(t), 1 - np.sin(t) - 0.5], axis=1)\n",
" X = np.concatenate([x1, x2], axis=0) # stack into one (100, 2) array\n",
" X += noise * np.random.randn(*X.shape) # jitter dots so it's not perfectly clean\n",
" y = np.array([0] * n + [1] * n) # label 0 = outer moon, 1 = inner moon\n",
" return X, y\n",
"\n",
"try:\n",
" from sklearn.datasets import make_moons\n",
" X, y = make_moons(n_samples=100, noise=0.1, random_state=1337)\n",
" print(\"using sklearn make_moons\")\n",
"except Exception as e:\n",
" X, y = make_moons_fallback(100, noise=0.1) # same shape, no sklearn needed\n",
" print(\"sklearn not available, using NumPy fallback:\", type(e).__name__)\n",
"\n",
"print(\"X shape:\", X.shape, \" y shape:\", y.shape) # expect (100, 2) and (100,)\n",
"print(\"first 3 points:\", X[:3].tolist())\n",
"print(\"first 3 labels:\", y[:3].tolist())\n",
"\n",
"if HAS_PLT:\n",
" plt.figure(figsize=(5, 4))\n",
" plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # red vs blue by class\n",
" plt.title(\"two moons — can a tiny net separate the colors?\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "0d2fe98b",
"metadata": {},
"source": [
"### 1.1 The `Value` object — a number that remembers\n",
"\n",
"A normal Python float forgets where it came from: once you compute `3.0`, nobody knows it\n",
"was `1.0 + 2.0`. Our `Value` keeps that memory: the numbers that made it (`_prev`) and the\n",
"operation (`_op`).\n",
"\n",
"We start with a **minimal** version that can only do the **forward pass** (no gradients yet),\n",
"just so you can see the \"remembering\" working.\n",
"\n",
"**-> Training:** `_prev` is the list of parent numbers; later, gradients flow backward along\n",
"exactly these links."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "eea9bc18",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"d = Value(data=4.0)\n",
"d was made by op: '+'\n",
"d's parents: {Value(data=-6.0), Value(data=10.0)}\n"
]
}
],
"source": [
"class Value:\n",
" def __init__(self, data, _children=(), _op=''):\n",
" self.data = data # the number itself (forward pass result)\n",
" # HISTORY: who made me? _prev = parent Value objects; _op = the operation (+, *, ...)\n",
" # Forward pass builds a linked tree/DAG automatically — no separate \"tape\" step.\n",
" self._prev = set(_children) # set of input Values that fed into this one\n",
" self._op = _op # string tag: '+', '*', or '' for a leaf (input)\n",
"\n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\" # pretty print when you print(a)\n",
"\n",
" def __add__(self, other):\n",
" # New number = sum; we pass (self, other) so this node remembers both parents.\n",
" return Value(self.data + other.data, (self, other), '+')\n",
"\n",
" def __mul__(self, other):\n",
" # Same idea: result remembers it came from self * other.\n",
" return Value(self.data * other.data, (self, other), '*')\n",
"\n",
"\n",
"# --- tiny demo: build d = a*b + c and inspect the graph tags ---\n",
"a = Value(2.0) # leaf — no parents, _op is ''\n",
"b = Value(-3.0)\n",
"c = Value(10.0)\n",
"e = a * b # e.data = -6; e._prev = {a, b}; e._op = '*'\n",
"d = e + c # d.data = 4; d._prev = {e, c}; d._op = '+'\n",
"print(\"d =\", d)\n",
"print(\"d was made by op:\", repr(d._op))\n",
"print(\"d's parents:\", d._prev)"
]
},
{
"cell_type": "markdown",
"id": "e65b3200",
"metadata": {},
"source": [
"### 1.2 Reading the expression graph\n",
"\n",
"`d = a*b + c` is really a little tree: `d` points back to `e` and `c`; `e` points back to\n",
"`a` and `b`. Let's print it as an indented tree so the structure is visible. (No graphviz\n",
"dependency — just recursion.) Section **1.3** draws the same wiring as an interactive picture."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "3a436597",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"4.0000 (op=+)\n",
" -6.0000 (op=*)\n",
" 2.0000 (op=leaf)\n",
" -3.0000 (op=leaf)\n",
" 10.0000 (op=leaf)\n"
]
}
],
"source": [
"def show(v, indent=0):\n",
" # print this node's value and which op created it ('leaf' = raw input)\n",
" print(\" \" * indent + f\"{v.data:.4f} (op={v._op or 'leaf'})\")\n",
" for child in v._prev: # follow parent links (deeper indent = further back in time)\n",
" show(child, indent + 1) # recurse until we hit leaves with no _prev\n",
"\n",
"show(d) # prints the tree: d (+) -> e (*) -> a, b and c"
]
},
{
"cell_type": "markdown",
"id": "f4e9da6b",
"metadata": {},
"source": [
"### 1.3 Interactive graph — see the wiring\n",
"\n",
"The text tree above is correct but flat. The same expression as a **picture**: green boxes are\n",
"raw inputs (leaves), blue boxes are operations. **Hover** a node to highlight its connections;\n",
"**click** to pin the detail panel. Arrows run from ingredients (bottom) up to the final score (top)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f149d4da",
"metadata": {},
"outputs": [],
"source": [
"# Re-use d = a*b + c from section 1.1 (run that cell first if you restarted the kernel)\n",
"show_graph(d, title=\"Forward pass: d = a × b + c\")"
]
},
{
"cell_type": "markdown",
"id": "5ee07490",
"metadata": {},
"source": [
"### 2.1 What a gradient means — nudge and measure\n",
"\n",
"Before any clever math, here is the *definition* of a gradient, done the dumb way: change an\n",
"input by a tiny amount `h`, recompute the output, and see how much it moved.\n",
"\n",
"slope = (output after nudge - output before) / h\n",
"\n",
"Real-life picture: to feel how steep a hill is, take one small step and notice how much your\n",
"height changed. That ratio is the slope."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fbfbee18",
"metadata": {},
"outputs": [],
"source": [
"def f_out(a_, b_, c_):\n",
" return (a_ * b_) + c_ # plain floats — no graph, just the math\n",
"\n",
"h = 1e-6 # tiny nudge size (like a small step on a hill)\n",
"base = f_out(2.0, -3.0, 10.0) # output before any nudge = 4.0\n",
"\n",
"# how sensitive is the output to a? slope ≈ (f(a+h) - f(a)) / h\n",
"da = (f_out(2.0 + h, -3.0, 10.0) - base) / h\n",
"# to b?\n",
"db = (f_out(2.0, -3.0 + h, 10.0) - base) / h\n",
"# to c?\n",
"dc = (f_out(2.0, -3.0, 10.0 + h) - base) / h\n",
"\n",
"print(f\"output = {base}\")\n",
"print(f\"d(out)/da = {da:.4f} (equals b = -3)\") # derivative of a*b w.r.t. a is b\n",
"print(f\"d(out)/db = {db:.4f} (equals a = 2)\") # derivative w.r.t. b is a\n",
"print(f\"d(out)/dc = {dc:.4f} (equals 1)\") # derivative of +c w.r.t. c is 1"
]
},
{
"cell_type": "markdown",
"id": "d3d3c6e2",
"metadata": {},
"source": [
"### 3.1 The chain rule — multiply the little ratios\n",
"\n",
"The nudge-and-measure trick is correct but slow: one re-run per input. Real networks have\n",
"millions of inputs. The **chain rule** gets every sensitivity in *one* backward pass.\n",
"\n",
"#### The one idea (relay race, not one long jump)\n",
"\n",
"When one thing changes another thing, which then changes a third, you **multiply the two\n",
"\"sensitivity\" steps**.\n",
"\n",
"- **a** — starting point (e.g. how many projects you take)\n",
"- **e** — middle step (hours you work)\n",
"- **d** — final result (your pay)\n",
"\n",
"**d** does not feel **a** directly. It only feels **e**. So ask two smaller questions:\n",
"\n",
"1. If **a** moves a little, how much does **e** move? → `d(e)/d(a)`\n",
"2. If **e** moves a little, how much does **d** move? → `d(d)/d(e)`\n",
"\n",
"The full effect is those two multiplied:\n",
"\n",
"`d(d)/d(a) = d(d)/d(e) × d(e)/d(a)`\n",
"\n",
"> **One line to keep:** If **d** depends on **e**, and **e** depends on **a**, then \"how fast\n",
"> **d** responds to **a**\" = \"how fast **d** responds to **e**\" × \"how fast **e** responds to **a**.\"\n",
"\n",
"#### Units make it click (the middle cancels)\n",
"\n",
"Like converting units:\n",
"\n",
"- Pay per hour → dollars/hour\n",
"- Hours per project → hours/project\n",
"\n",
"Multiply: `(dollars/hour) × (hours/project) = dollars/project`\n",
"\n",
"The **hour** in the middle cancels. You get \"how pay changes when projects change.\" The formula\n",
"looks like fraction cancellation — that is the story, not magic algebra.\n",
"\n",
"#### Tiny wiggle picture\n",
"\n",
"Bump **a** by a tiny amount Δa.\n",
"\n",
"- **e** moves about `(d(e)/d(a)) × Δa`\n",
"- **d** moves about `(d(d)/d(e)) ×` that amount\n",
"\n",
"So total change in **d** is `(d(d)/d(e)) × (d(e)/d(a)) × Δa`. The chain rule is just: the\n",
"multiplier on Δa is the **product of the two hops**.\n",
"\n",
"#### Worked example\n",
"\n",
"Let `a = 2`, `e = a²`, `d = 3e`.\n",
"\n",
"- `d(e)/d(a) = 2a = 4` (e changes 4 units per unit of a)\n",
"- `d(d)/d(e) = 3` (d changes 3 units per unit of e)\n",
"\n",
"So `d(d)/d(a) = 3 × 4 = 12` — when **a** goes up by 1, **d** goes up by about 12.\n",
"\n",
"Check directly: `d = 3a²`, so `d(d)/d(a) = 6a = 12` at `a = 2`. Same answer.\n",
"\n",
"#### Why college notation felt confusing\n",
"\n",
"Textbooks often write `f(g(x))` and say \"derivative of outside × derivative of inside.\" That\n",
"is correct, but it hides the story: **indirect influence through a middle variable**. The relay\n",
"race picture is the same thing with names you can follow.\n",
"\n",
"#### Local rules (what each station does)\n",
"\n",
"Each operation knows its own **local** derivative:\n",
"\n",
"- for `+`: output changes 1-for-1 with each input, so the gradient just **passes through**.\n",
"- for `*`: each input's local derivative is **the other input**.\n",
"\n",
"**Backprop** = start with gradient 1.0 at the output, then walk backward multiplying by each\n",
"local derivative."
]
},
{
"cell_type": "markdown",
"id": "0da2822d",
"metadata": {},
"source": [
"### 4.1 Adding gradients and local backward rules\n",
"\n",
"#### What is autograd?\n",
"\n",
"**Autograd** = **automatic differentiation** — the computer figures out gradients for you\n",
"after you write the forward math.\n",
"\n",
"Real-life picture: like a GPS that logged every turn on your drive. When you ask \"how do I get\n",
"back?\", it does not guess — it rewinds those turns and tells you how each step affected the\n",
"final destination.\n",
"\n",
"**Is it a library?** Two meanings:\n",
"\n",
"| Meaning | What it is | In this notebook |\n",
"|---|---|---|\n",
"| **The idea** | Any system that records ops, then runs `backward()` to fill `.grad` | Our homemade **micrograd** `Value` class |\n",
"| **A PyPI package** | A library literally named [`autograd`](https://github.com/HIPS/autograd) | We do **not** use it — we build from scratch |\n",
"\n",
"Later, **PyTorch** ships its own autograd (`loss.backward()` in notebooks `01` and `06`). Same\n",
"idea, industrial scale.\n",
"\n",
"When code says \"analytic gradients via autograd,\" it means: run forward through `Value` objects,\n",
"call `.backward()` once, read `.grad` — instead of nudging each input by hand with\n",
"`numeric_grad`.\n",
"\n",
"---\n",
"\n",
"Now the full `Value`. Each operation also defines a `_backward()` that says how to push the\n",
"output's gradient onto its inputs. Note we **accumulate** (`+=`) gradients, because one value\n",
"can feed into several places.\n",
"\n",
"We add `+`, `*`, `**` (power), `tanh`, and `exp` — enough to build a neural net. We also add\n",
"convenience operators (`-`, `/`, right-hand versions) so expressions read naturally.\n",
"\n",
"#### `_op` and `_prev` — the tags on every box\n",
"\n",
"Every `Value` stores two pieces of history in `__init__`:\n",
"\n",
"- **`_op`** — *what happened at this station?* A stamp on the box: `'+'` (mixed), `'*'` (scaled),\n",
" `'tanh'` (squashed), or `''` for a raw leaf input (flour, sugar — nothing done yet).\n",
"- **`_prev`** — *who gave me ingredients?* The set of parent `Value`s that fed into this one.\n",
" Like \"received from\" arrows on the tag.\n",
"\n",
"**Cookie-factory picture for `d = a * b + c`:**\n",
"\n",
"1. `e = a * b` → `e._op = '*'`, `e._prev = {a, b}` — \"made by multiplying; parents are a and b.\"\n",
"2. `d = e + c` → `d._op = '+'`, `d._prev = {e, c}` — \"made by adding; parents are e and c.\"\n",
"\n",
"A normal float is a score on a napkin: **4**. You cannot tell how you got it. A `Value` is the\n",
"same score **on a tagged box** — you can always walk back.\n",
"\n",
"**Why this matters:** forward pass moves ingredients down the line. Backprop starts at the final\n",
"score and walks **backward** along `_prev`. At each stop, `_op` tells you which local rule to\n",
"apply (add passes blame through; multiply splits blame using the other input). Without these\n",
"tags there is no map from the final number back to the knobs you need to turn.\n",
"\n",
"> **One line:** `_op` = what was done here; `_prev` = who fed into it. Together they are the\n",
"> breadcrumb trail from output back to inputs.\n",
"\n",
"**-> Training:** `tanh` is the neuron's \"squashing\" activation; its local derivative is\n",
"`1 - tanh(x)^2`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9433f6b8",
"metadata": {},
"outputs": [],
"source": [
"class Value:\n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data # the actual number (forward value)\n",
" self.grad = 0.0 # how much the final loss changes if THIS number nudges +1\n",
" self._backward = lambda: None # filled in by each op: \"push grad to my parents\"\n",
" self._prev = set(_children) # graph edges pointing backward to parents\n",
" self._op = _op # which op created this node ('+', '*', 'tanh', ...)\n",
" self.label = label # optional name for debugging / graph drawing\n",
"\n",
" def __repr__(self):\n",
" return f\"Value(data={self.data:.4f}, grad={self.grad:.4f})\"\n",
"\n",
" def __add__(self, other):\n",
" other = other if isinstance(other, Value) else Value(other) # allow Value + 3\n",
" out = Value(self.data + other.data, (self, other), '+') # forward: add numbers\n",
" def _backward():\n",
" # local rule for +: d(a+b)/da = 1, d(a+b)/db = 1 → grad flows 1:1 to both\n",
" self.grad += 1.0 * out.grad\n",
" other.grad += 1.0 * out.grad\n",
" out._backward = _backward # closure captures self/other/out — runs later in backward()\n",
" return out\n",
"\n",
" def __mul__(self, other):\n",
" other = other if isinstance(other, Value) else Value(other)\n",
" out = Value(self.data * other.data, (self, other), '*') # forward: multiply\n",
" def _backward():\n",
" # local rule for *: d(a*b)/da = b, d(a*b)/db = a (chain rule: × out.grad)\n",
" self.grad += other.data * out.grad\n",
" other.grad += self.data * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
" def __pow__(self, other):\n",
" assert isinstance(other, (int, float)), \"only int/float powers\" # exponent is constant\n",
" out = Value(self.data ** other, (self,), f'**{other}') # forward: x^n\n",
" def _backward():\n",
" # d(x^n)/dx = n * x^(n-1), then chain rule × upstream grad\n",
" self.grad += other * (self.data ** (other - 1)) * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
" def tanh(self):\n",
" x = self.data\n",
" t = (math.exp(2 * x) - 1) / (math.exp(2 * x) + 1) # forward tanh (squash to -1..1)\n",
" out = Value(t, (self,), 'tanh')\n",
" def _backward():\n",
" # d tanh(x)/dx = 1 - tanh(x)^2 (use saved t, not recompute from self.data)\n",
" self.grad += (1 - t ** 2) * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
" def exp(self):\n",
" out = Value(math.exp(self.data), (self,), 'exp') # forward: e^x\n",
" def _backward():\n",
" # d e^x / dx = e^x — out.data IS e^x at forward time\n",
" self.grad += out.data * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
" # --- convenience ops so expressions like 3 + a or a / b work naturally ---\n",
" def __neg__(self):\n",
" return self * -1 # unary minus via multiply by -1\n",
"\n",
" def __sub__(self, other):\n",
" return self + (-other if isinstance(other, Value) else Value(-other)) # a - b = a + (-b)\n",
"\n",
" def __radd__(self, other):\n",
" return self + other # handles 3 + a (Python calls __radd__ on a)\n",
"\n",
" def __rmul__(self, other):\n",
" return self * other # handles 3 * a\n",
"\n",
" def __truediv__(self, other):\n",
" other = other if isinstance(other, Value) else Value(other)\n",
" return self * other ** -1 # a / b = a * b^(-1)\n",
"\n",
" def backward(self):\n",
" # Step 1: list every node in the graph, parents before children (topological sort).\n",
" # We walk _prev links; reversed(topo) then visits output → inputs.\n",
" topo = []\n",
" visited = set()\n",
" def build_topo(v):\n",
" if v not in visited:\n",
" visited.add(v)\n",
" for child in v._prev: # visit all ancestors first\n",
" build_topo(child)\n",
" topo.append(v) # post-order: children before parent\n",
" build_topo(self) # start from the output node (e.g. loss)\n",
"\n",
" # Step 2: seed — \"if output goes up 1, output goes up 1\" (dL/dL = 1).\n",
" self.grad = 1.0\n",
"\n",
" # Step 3: reverse sweep; each node runs its local _backward (chain rule × parents).\n",
" for node in reversed(topo): # output first, leaves last\n",
" node._backward() # each op pushes blame to its _prev parents\n",
"\n",
"\n",
"print(\"full Value class ready\")"
]
},
{
"cell_type": "markdown",
"id": "7f14c265",
"metadata": {},
"source": [
"### 4.2 One backward pass on the tiny expression\n",
"\n",
"Rebuild `d = a*b + c` with the full class, call `d.backward()` once, and read the gradients.\n",
"They should match the nudge-and-measure numbers from section 2.1 (`a.grad = b = -3`,\n",
"`b.grad = a = 2`, `c.grad = 1`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d1e739c",
"metadata": {},
"outputs": [],
"source": [
"a = Value(2.0, label='a') # leaf inputs — knobs with no parents yet\n",
"b = Value(-3.0, label='b')\n",
"c = Value(10.0, label='c')\n",
"e = a * b # builds multiply node; graph remembers a, b\n",
"d = e + c # builds add node; graph remembers e, c\n",
"\n",
"d.backward() # one backward pass fills .grad on every node in the graph\n",
"\n",
"print(\"d =\", d.data) # forward answer still 4.0\n",
"print(\"a.grad =\", a.grad, \" (expect -3)\") # ∂d/∂a = b\n",
"print(\"b.grad =\", b.grad, \" (expect 2)\") # ∂d/∂b = a\n",
"print(\"c.grad =\", c.grad, \" (expect 1)\") # ∂d/∂c = 1"
]
},
{
"cell_type": "markdown",
"id": "0083fec0",
"metadata": {},
"source": [
"Same graph **after backprop** — orange gradient numbers show how much each node would nudge the\n",
"final output. Hover `a` and notice its grad equals `b` (−3): the multiply rule in action."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c14c82ca",
"metadata": {},
"outputs": [],
"source": [
"show_graph(d, title=\"After backward() — gradients filled in\")"
]
},
{
"cell_type": "markdown",
"id": "1ac38e12",
"metadata": {},
"source": [
"### 4.3 Trust but verify — numeric gradient check\n",
"\n",
"A good habit: confirm the analytic gradient (from `backward()`) matches the slow\n",
"nudge-and-measure gradient. If they agree, the engine is correct.\n",
"\n",
"**Real-life picture:** change one recipe ingredient by a tiny pinch, cook again, and see\n",
"how much the final score moves. That ratio — *change in score ÷ size of pinch* — is the\n",
"gradient for that ingredient. Repeat for each ingredient, one at a time. Slow, but it is\n",
"the definition of \"sensitivity.\"\n",
"\n",
"**Two ways to measure the same hill**\n",
"\n",
"| | Nudge-and-measure (numeric) | `backward()` (analytic) |\n",
"|---|---|---|\n",
"| Idea | Tiny change in one input → re-run forward | Chain rule walks the graph once |\n",
"| Cost | One forward pass **per input** | One backward pass for **all** inputs |\n",
"| Result | slope ≈ `(f(x+h) − f(x)) / h` | `.grad` on each leaf `Value` |\n",
"\n",
"**Outline — nudge-and-measure (what `numeric_grad` does)**\n",
"\n",
"1. Run the same math as the graph, but on plain floats (no `Value` objects).\n",
"2. Pick a tiny step `h` (e.g. `1e-6`).\n",
"3. Save the baseline output at the current inputs.\n",
"4. For **each** input `i`, one at a time:\n",
" - copy the inputs;\n",
" - add `h` only to input `i`;\n",
" - re-run forward;\n",
" - slope = (new output − baseline) / `h`.\n",
"5. That slope is the numeric gradient for input `i`.\n",
"\n",
"**Outline — analytic check (what we trust after verifying)**\n",
"\n",
"1. Build the expression with `Value` objects (the graph remembers parents via `_prev`).\n",
"2. Call `.backward()` once on the final output.\n",
"3. Read `.grad` on each leaf — chain rule applied in one reverse sweep.\n",
"\n",
"**Compare:** print analytic vs numeric side by side; they **match** when\n",
"`|analytic − numeric| < 1e-4`. Same hill, two rulers — if they agree, the engine's local\n",
"`_backward()` rules are wired correctly."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fe4b711e",
"metadata": {},
"outputs": [],
"source": [
"def numeric_grad(f, inputs, i, h=1e-6):\n",
" base = f(inputs) # output at current inputs\n",
" bumped = list(inputs) # copy so we don't mutate the original list\n",
" bumped[i] += h # nudge only input i by a tiny amount\n",
" return (f(bumped) - base) / h # slope = rise / run\n",
"\n",
"f = lambda v: (v[0] * v[1]) + v[2] # same math as a*b+c but on plain floats\n",
"inputs = [2.0, -3.0, 10.0]\n",
"\n",
"a = Value(2.0); b = Value(-3.0); c = Value(10.0)\n",
"(a * b + c).backward() # analytic gradients via autograd\n",
"\n",
"for i, (name, val) in enumerate(zip(\"abc\", (a, b, c))):\n",
" ng = numeric_grad(f, inputs, i) # slow finite-difference gradient for input i\n",
" print(f\"{name}: analytic={val.grad:+.4f} numeric={ng:+.4f} match={abs(val.grad-ng)<1e-4}\")"
]
},
{
"cell_type": "markdown",
"id": "6518159c",
"metadata": {},
"source": [
"**Your turn 4** — Build `g = (a*b + c).tanh()` with `a=0.5, b=2.0, c=-1.0`, call\n",
"`g.backward()`, and check `a.grad` against a numeric nudge. (Hint: reuse `numeric_grad`\n",
"with `f = lambda v: math.tanh(v[0]*v[1] + v[2])`.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d6bea75d",
"metadata": {},
"outputs": [],
"source": [
"# Your turn 4 — fill in and run\n",
"a = Value(0.5); b = Value(2.0); c = Value(-1.0) # inputs\n",
"g = (a * b + c).tanh() # forward: multiply, add, squash\n",
"g.backward() # backprop through tanh, +, *\n",
"\n",
"f = lambda v: math.tanh(v[0]*v[1] + v[2]) # same formula on plain floats\n",
"ng_a = numeric_grad(f, [0.5, 2.0, -1.0], 0) # numeric ∂g/∂a (nudge index 0)\n",
"print(f\"a.grad analytic = {a.grad:+.4f} numeric = {ng_a:+.4f}\")"
]
},
{
"cell_type": "markdown",
"id": "fef7ef8b",
"metadata": {},
"source": [
"### 5.1 Neuron -> Layer -> MLP, all from `Value`\n",
"\n",
"Everything below is still built from plain `Value` objects — the same tagged boxes and\n",
"`backward()` trail from section 4. Here we stack them into a small **neural network**.\n",
"\n",
"#### One story to hold in your head: renting an apartment\n",
"\n",
"You apply with two numbers on the form: **monthly income** and **credit score**.\n",
"The landlord wants one answer: *likely good tenant* vs *likely risky*.\n",
"\n",
"**One neuron = one reviewer on the panel**\n",
"\n",
"A single neuron looks at your whole application and gives **one opinion**, squashed to a\n",
"number between -1 and +1:\n",
"\n",
"`tanh(w · x + b)` means:\n",
"\n",
"- **`w`** — how much this reviewer cares about each field (income vs credit). Big weight =\n",
" \"this matters a lot to me.\"\n",
"- **`b`** — this reviewer's baseline strictness before seeing you (always picky, or always\n",
" lenient).\n",
"- **`w · x + b`** — weighted score: multiply each input by its importance, add bias.\n",
"- **`tanh(...)`** — squash the score so it cannot run away to infinity; strong yes or strong no,\n",
" but bounded (like capping enthusiasm at \"very pro\" or \"very against\").\n",
"\n",
"So one neuron is not magic — it is **one weighted vote**, softened at the edges.\n",
"\n",
"**One layer = a whole panel, all reading the same application**\n",
"\n",
"A **layer** is several neurons in **parallel**. Each sees the **same** inputs `x`, but each\n",
"has its **own** weights and bias — different reviewers, different priorities:\n",
"\n",
"- Reviewer A might care mostly about income.\n",
"- Reviewer B might care mostly about credit.\n",
"- Reviewer C might look for a balance of both.\n",
"\n",
"The layer's output is a **list of opinions** (one per neuron). Eight neurons → eight numbers\n",
"describing the application from eight angles.\n",
"\n",
"**An MLP = panels stacked in stages**\n",
"\n",
"An **MLP** (multi-layer perceptron) feeds one layer's opinions into the next:\n",
"\n",
"1. **First hidden layer** — junior reviewers turn raw form fields into rough themes (\"cash-flow\n",
" looks fine\", \"credit is shaky\", …).\n",
"2. **Second hidden layer** — senior reviewers debate those themes, not the raw form.\n",
"3. **Output layer** — one final judge reads the debate and emits a single score (here: one\n",
" neuron → one number for the two-moons task).\n",
"\n",
"That is why the code uses shapes like `MLP(2, [8, 8, 1)`:\n",
"\n",
"| Piece | In the story | In code |\n",
"|---|---|---|\n",
"| 2 inputs | income + credit (here: x, y on the plot) | `nin = 2` |\n",
"| 8 + 8 hidden | two panels of 8 reviewers | two `Layer`s with `nout=8` |\n",
"| 1 output | final hire / no-hire score | last layer with `nout=1` |\n",
"\n",
"Data flows **forward** only: form → junior panel → senior panel → final judge.\n",
"Every multiply, add, and `tanh` is still a `Value` node, so `loss.backward()` can later\n",
"blame the right weight when the answer was wrong.\n",
"\n",
"#### The three building blocks (code ↔ picture)\n",
"\n",
"| Class | What it is | Real-life |\n",
"|---|---|---|\n",
"| **`Neuron`** | one weighted vote + squash | one reviewer |\n",
"| **`Layer`** | list of neurons, same inputs | one panel sitting together |\n",
"| **`MLP`** | layers wired in sequence | whole hiring process, stage by stage |\n",
"\n",
"**-> Training:** `parameters()` collects every weight and bias in the network — all the knobs\n",
"on all reviewers' score sheets. Section 6 nudges those knobs downhill until the final judge\n",
"gets the moons right."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "93e51095",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <iframe\n",
" width=\"100%\"\n",
" height=\"640\"\n",
" src=\"data:text/html;base64,PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4KPGhlYWQ+CjxtZXRhIGNoYXJzZXQ9IlVURi04IiAvPgo8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEuMCIgLz4KPHRpdGxlPk1MUCBQbGF5Z3JvdW5kIOKAlCBOZXVyb24g4oaSIExheWVyIOKGkiBNTFAgKHNlY3Rpb24gNS4xKTwvdGl0bGU+CjxzY3JpcHQgc3JjPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2QzQDciPjwvc2NyaXB0Pgo8c3R5bGU+CiAgOnJvb3QgewogICAgLS1iZzogIzBmMTQyMDsKICAgIC0tcGFuZWw6ICMxNjFkMmU7CiAgICAtLXBhbmVsLTI6ICMxZDI5NDA7CiAgICAtLWluazogI2U4ZWRmNjsKICAgIC0tbXV0ZWQ6ICM5NGEzYjg7CiAgICAtLWxpbmU6ICMyYTM1NTA7CiAgICAtLXBvczogIzRhZGU4MDsgICAvKiBwb3NpdGl2ZSB3ZWlnaHQgLyBhY3RpdmF0aW9uICovCiAgICAtLW5lZzogI2Y4NzE3MTsgICAvKiBuZWdhdGl2ZSB3ZWlnaHQgLyBhY3RpdmF0aW9uICovCiAgICAtLWFjY2VudDogIzYwYTVmYTsKICAgIC0tYWNjZW50LTI6ICNmYmJmMjQ7CiAgfQogICogeyBib3gtc2l6aW5nOiBib3JkZXItYm94OyB9CiAgaHRtbCwgYm9keSB7IG1hcmdpbjogMDsgcGFkZGluZzogMDsgfQogIGJvZHkgewogICAgYmFja2dyb3VuZDogcmFkaWFsLWdyYWRpZW50KDEyMDBweCA2MDBweCBhdCA3MCUgLTEwJSwgIzFiMjc0MCAwJSwgdmFyKC0tYmcpIDU1JSk7CiAgICBjb2xvcjogdmFyKC0taW5rKTsKICAgIGZvbnQtZmFtaWx5OiAiU2Vnb2UgVUkiLCBzeXN0ZW0tdWksIC1hcHBsZS1zeXN0ZW0sIFJvYm90bywgc2Fucy1zZXJpZjsKICAgIGxpbmUtaGVpZ2h0OiAxLjU7CiAgICBwYWRkaW5nOiAyMnB4IGNsYW1wKDE0cHgsIDN2dywgNDBweCkgNjBweDsKICB9CiAgaDEgeyBmb250LXNpemU6IGNsYW1wKDIwcHgsIDIuNnZ3LCAyOHB4KTsgbWFyZ2luOiAwIDAgNHB4OyB9CiAgLnN1YiB7IGNvbG9yOiB2YXIoLS1tdXRlZCk7IG1hcmdpbjogMCAwIDE4cHg7IG1heC13aWR0aDogNzBjaDsgfQogIC5zdWIgYiB7IGNvbG9yOiB2YXIoLS1pbmspOyB9CiAgLmxheW91dCB7IGRpc3BsYXk6IGdyaWQ7IGdyaWQtdGVtcGxhdGUtY29sdW1uczogMzAwcHggMWZyOyBnYXA6IDE4cHg7IGFsaWduLWl0ZW1zOiBzdGFydDsgfQogIEBtZWRpYSAobWF4LXdpZHRoOiA5MDBweCkgeyAubGF5b3V0IHsgZ3JpZC10ZW1wbGF0ZS1jb2x1bW5zOiAxZnI7IH0gfQoKICAucGFuZWwgewogICAgYmFja2dyb3VuZDogbGluZWFyLWdyYWRpZW50KDE4MGRlZywgdmFyKC0tcGFuZWwpIDAlLCB2YXIoLS1wYW5lbC0yKSAxMDAlKTsKICAgIGJvcmRlcjogMXB4IHNvbGlkIHZhcigtLWxpbmUpOwogICAgYm9yZGVyLXJhZGl1czogMTRweDsKICAgIHBhZGRpbmc6IDE2cHggMTZweCAxOHB4OwogICAgYm94LXNoYWRvdzogMCAxMHB4IDMwcHggcmdiYSgwLDAsMCwuMzUpOwogIH0KICAucGFuZWwgaDIgeyBmb250LXNpemU6IDE0cHg7IHRleHQtdHJhbnNmb3JtOiB1cHBlcmNhc2U7IGxldHRlci1zcGFjaW5nOiAuMDhlbTsgY29sb3I6IHZhcigtLW11dGVkKTsgbWFyZ2luOiAwIDAgMTJweDsgfQoKICAuZmllbGQgeyBtYXJnaW4tYm90dG9tOiAxNnB4OyB9CiAgLmZpZWxkIGxhYmVsIHsgZGlzcGxheTogZmxleDsganVzdGlmeS1jb250ZW50OiBzcGFjZS1iZXR3ZWVuOyBmb250LXNpemU6IDEzcHg7IG1hcmdpbi1ib3R0b206IDZweDsgfQogIC5maWVsZCBsYWJlbCAudmFsIHsgZm9udC12YXJpYW50LW51bWVyaWM6IHRhYnVsYXItbnVtczsgY29sb3I6IHZhcigtLWFjY2VudCk7IGZvbnQtd2VpZ2h0OiA2MDA7IH0KICBpbnB1dFt0eXBlPXJhbmdlXSB7IHdpZHRoOiAxMDAlOyBhY2NlbnQtY29sb3I6IHZhcigtLWFjY2VudCk7IH0KCiAgLmJ0biB7CiAgICBhcHBlYXJhbmNlOiBub25lOyBib3JkZXI6IDFweCBzb2xpZCB2YXIoLS1saW5lKTsgY3Vyc29yOiBwb2ludGVyOwogICAgYmFja2dyb3VuZDogdmFyKC0tcGFuZWwtMik7IGNvbG9yOiB2YXIoLS1pbmspOwogICAgcGFkZGluZzogOXB4IDEycHg7IGJvcmRlci1yYWRpdXM6IDlweDsgZm9udC1zaXplOiAxM3B4OyBmb250LXdlaWdodDogNjAwOwogICAgdHJhbnNpdGlvbjogLjE1czsgd2lkdGg6IDEwMCU7IG1hcmdpbi1ib3R0b206IDlweDsKICB9CiAgLmJ0bjpob3ZlciB7IGJvcmRlci1jb2xvcjogdmFyKC0tYWNjZW50KTsgY29sb3I6ICNmZmY7IH0KICAuYnRuLnByaW1hcnkgeyBiYWNrZ3JvdW5kOiBsaW5lYXItZ3JhZGllbnQoMTgwZGVnLCAjM2I4MmY2LCAjMjU2M2ViKTsgYm9yZGVyLWNvbG9yOiAjMjU2M2ViOyB9CiAgLmJ0bi5wcmltYXJ5OmhvdmVyIHsgYmFja2dyb3VuZDogbGluZWFyLWdyYWRpZW50KDE4MGRlZywgIzRmOTNmZiwgIzJmNmZlMCk7IH0KCiAgLmxlZ2VuZCB7IGRpc3BsYXk6IGZsZXg7IGZsZXgtZGlyZWN0aW9uOiBjb2x1bW47IGdhcDogN3B4OyBmb250LXNpemU6IDEyLjVweDsgY29sb3I6IHZhcigtLW11dGVkKTsgfQogIC5sZWdlbmQgLnJvdyB7IGRpc3BsYXk6IGZsZXg7IGFsaWduLWl0ZW1zOiBjZW50ZXI7IGdhcDogOHB4OyB9CiAgLnN3YXRjaCB7IHdpZHRoOiAyMnB4OyBoZWlnaHQ6IDRweDsgYm9yZGVyLXJhZGl1czogM3B4OyB9CgogIC5zdGFnZSB7IGJhY2tncm91bmQ6IHJnYmEoMCwwLDAsLjE4KTsgYm9yZGVyLXJhZGl1czogMTRweDsgfQogIHN2ZyB7IGRpc3BsYXk6IGJsb2NrOyB3aWR0aDogMTAwJTsgaGVpZ2h0OiBhdXRvOyB9CgogIC5vdXQtY2FyZCB7CiAgICBtYXJnaW4tdG9wOiAxNHB4OyBwYWRkaW5nOiAxNHB4IDE2cHg7IGJvcmRlci1yYWRpdXM6IDEycHg7CiAgICBib3JkZXI6IDFweCBzb2xpZCB2YXIoLS1saW5lKTsgYmFja2dyb3VuZDogcmdiYSgwLDAsMCwuMik7CiAgICBkaXNwbGF5OiBmbGV4OyBhbGlnbi1pdGVtczogY2VudGVyOyBqdXN0aWZ5LWNvbnRlbnQ6IHNwYWNlLWJldHdlZW47IGdhcDogMTRweDsKICB9CiAgLm91dC1jYXJkIC5sYWJlbCB7IGNvbG9yOiB2YXIoLS1tdXRlZCk7IGZvbnQtc2l6ZTogMTNweDsgfQogIC5vdXQtY2FyZCAudmVyZGljdCB7IGZvbnQtc2l6ZTogMjZweDsgZm9udC13ZWlnaHQ6IDgwMDsgZm9udC12YXJpYW50LW51bWVyaWM6IHRhYnVsYXItbnVtczsgfQoKICAvKiB0b29sdGlwICovCiAgLnRpcCB7CiAgICBwb3NpdGlvbjogZml4ZWQ7IHBvaW50ZXItZXZlbnRzOiBub25lOyB6LWluZGV4OiA1MDsgb3BhY2l0eTogMDsKICAgIGJhY2tncm91bmQ6ICMwYjEzMjI7IGJvcmRlcjogMXB4IHNvbGlkIHZhcigtLWFjY2VudCk7IGNvbG9yOiB2YXIoLS1pbmspOwogICAgcGFkZGluZzogOHB4IDEwcHg7IGJvcmRlci1yYWRpdXM6IDlweDsgZm9udC1zaXplOiAxMi41cHg7IG1heC13aWR0aDogMjQwcHg7CiAgICBib3gtc2hhZG93OiAwIDhweCAyNHB4IHJnYmEoMCwwLDAsLjUpOyB0cmFuc2l0aW9uOiBvcGFjaXR5IC4xMnM7CiAgfQogIC50aXAgYiB7IGNvbG9yOiB2YXIoLS1hY2NlbnQpOyB9CiAgLnRpcCAubW9ubyB7IGZvbnQtZmFtaWx5OiB1aS1tb25vc3BhY2UsICJDYXNjYWRpYSBDb2RlIiwgQ29uc29sYXMsIG1vbm9zcGFjZTsgfQoKICAvKiBub2RlICsgZWRnZSBiYXNlIHN0eWxlcyAqLwogIC5lZGdlIHsgdHJhbnNpdGlvbjogc3Ryb2tlLW9wYWNpdHkgLjJzOyB9CiAgLm5vZGUgY2lyY2xlIHsgc3Ryb2tlOiAjMGIxMzIyOyBzdHJva2Utd2lkdGg6IDEuNXB4OyBjdXJzb3I6IHBvaW50ZXI7IH0KICAubm9kZS1sYWJlbCB7IGZvbnQtc2l6ZTogMTFweDsgZmlsbDogdmFyKC0tbXV0ZWQpOyB9CiAgLmxheWVyLXRpdGxlIHsgZm9udC1zaXplOiAxMnB4OyBmaWxsOiB2YXIoLS1pbmspOyBmb250LXdlaWdodDogNzAwOyB0ZXh0LWFuY2hvcjogbWlkZGxlOyB9CiAgLmxheWVyLWNhcCB7IGZvbnQtc2l6ZTogMTFweDsgZmlsbDogdmFyKC0tbXV0ZWQpOyB0ZXh0LWFuY2hvcjogbWlkZGxlOyB9CiAgLmhpbnQgeyBjb2xvcjogdmFyKC0tbXV0ZWQpOyBmb250LXNpemU6IDEycHg7IG1hcmdpbi10b3A6IDEwcHg7IH0KICAucGlsbCB7IGRpc3BsYXk6aW5saW5lLWJsb2NrOyBwYWRkaW5nOjJweCA3cHg7IGJvcmRlcjoxcHggc29saWQgdmFyKC0tbGluZSk7IGJvcmRlci1yYWRpdXM6OTk5cHg7IGZvbnQtc2l6ZToxMXB4OyBjb2xvcjp2YXIoLS1tdXRlZCk7IH0KPC9zdHlsZT4KPC9oZWFkPgo8Ym9keT4KICA8aDE+TUxQIFBsYXlncm91bmQg4oCUIDxzcGFuIHN0eWxlPSJjb2xvcjp2YXIoLS1hY2NlbnQpIj5OZXVyb24g4oaSIExheWVyIOKGkiBNTFA8L3NwYW4+PC9oMT4KICA8cCBjbGFzcz0ic3ViIj4KICAgIFRoZSByZXZpZXdlci1wYW5lbCBzdG9yeSBmcm9tIDxiPnNlY3Rpb24gNS4xPC9iPiwgbWFkZSBsaXZlLiBUd28gaW5wdXRzCiAgICAoPGI+aW5jb21lPC9iPiwgPGI+Y3JlZGl0PC9iPikgZmxvdyB0aHJvdWdoIHR3byBwYW5lbHMgb2YgOCByZXZpZXdlcnMgaW50byBvbmUgZmluYWwganVkZ2U6CiAgICB0aGUgPHNwYW4gY2xhc3M9InBpbGwiPjIg4oaSIDgg4oaSIDgg4oaSIDE8L3NwYW4+IG5ldHdvcmsuIEV2ZXJ5IGxpbmUgaXMgYSB3ZWlnaHQsIGV2ZXJ5IGRvdCBpcyBhCiAgICA8c3BhbiBjbGFzcz0ibW9ubyI+dGFuaCh3wrd4ICsgYik8L3NwYW4+IG9waW5pb24uIERyYWcgdGhlIGlucHV0cywgaG92ZXIgYW55dGhpbmcsIG9yIHByZXNzIHBsYXkuCiAgPC9wPgoKICA8ZGl2IGNsYXNzPSJsYXlvdXQiPgogICAgPGRpdiBjbGFzcz0icGFuZWwiPgogICAgICA8aDI+SW5wdXRzICh0aGUgYXBwbGljYXRpb24pPC9oMj4KICAgICAgPGRpdiBjbGFzcz0iZmllbGQiPgogICAgICAgIDxsYWJlbD5JbmNvbWUgKHjigoEpIDxzcGFuIGNsYXNzPSJ2YWwiIGlkPSJ4MXYiPjAuNTA8L3NwYW4+PC9sYWJlbD4KICAgICAgICA8aW5wdXQgdHlwZT0icmFuZ2UiIGlkPSJ4MSIgbWluPSItMSIgbWF4PSIxIiBzdGVwPSIwLjAxIiB2YWx1ZT0iMC41IiAvPgogICAgICA8L2Rpdj4KICAgICAgPGRpdiBjbGFzcz0iZmllbGQiPgogICAgICAgIDxsYWJlbD5DcmVkaXQgKHjigoIpIDxzcGFuIGNsYXNzPSJ2YWwiIGlkPSJ4MnYiPi0wLjMwPC9zcGFuPjwvbGFiZWw+CiAgICAgICAgPGlucHV0IHR5cGU9InJhbmdlIiBpZD0ieDIiIG1pbj0iLTEiIG1heD0iMSIgc3RlcD0iMC4wMSIgdmFsdWU9Ii0wLjMiIC8+CiAgICAgIDwvZGl2PgoKICAgICAgPGgyIHN0eWxlPSJtYXJnaW4tdG9wOjZweCI+Q29udHJvbHM8L2gyPgogICAgICA8YnV0dG9uIGNsYXNzPSJidG4gcHJpbWFyeSIgaWQ9InBsYXkiPuKWtiBBbmltYXRlIGZvcndhcmQgcGFzczwvYnV0dG9uPgogICAgICA8YnV0dG9uIGNsYXNzPSJidG4iIGlkPSJyZXJvbGwiPvCfjrIgTmV3IHJhbmRvbSB3ZWlnaHRzPC9idXR0b24+CiAgICAgIDxidXR0b24gY2xhc3M9ImJ0biIgaWQ9InJlc2V0Ij7ihrogUmVzZXQgaW5wdXRzPC9idXR0b24+CgogICAgICA8aDIgc3R5bGU9Im1hcmdpbi10b3A6MThweCI+UmVhZCB0aGUgY29sb3JzPC9oMj4KICAgICAgPGRpdiBjbGFzcz0ibGVnZW5kIj4KICAgICAgICA8ZGl2IGNsYXNzPSJyb3ciPjxzcGFuIGNsYXNzPSJzd2F0Y2giIHN0eWxlPSJiYWNrZ3JvdW5kOnZhcigtLXBvcykiPjwvc3Bhbj4gcG9zaXRpdmUgd2VpZ2h0IC8gYWN0aXZhdGlvbjwvZGl2PgogICAgICAgIDxkaXYgY2xhc3M9InJvdyI+PHNwYW4gY2xhc3M9InN3YXRjaCIgc3R5bGU9ImJhY2tncm91bmQ6dmFyKC0tbmVnKSI+PC9zcGFuPiBuZWdhdGl2ZSB3ZWlnaHQgLyBhY3RpdmF0aW9uPC9kaXY+CiAgICAgICAgPGRpdiBjbGFzcz0icm93Ij48c3BhbiBjbGFzcz0ic3dhdGNoIiBzdHlsZT0iYmFja2dyb3VuZDp2YXIoLS1tdXRlZCk7aGVpZ2h0OjJweCI+PC9zcGFuPiB0aGluIGxpbmUgPSB3ZWFrLCB0aGljayA9IHN0cm9uZzwvZGl2PgogICAgICA8L2Rpdj4KCiAgICAgIDxkaXYgY2xhc3M9Im91dC1jYXJkIj4KICAgICAgICA8c3BhbiBjbGFzcz0ibGFiZWwiPkZpbmFsIGp1ZGdlPGJyPnNjb3JlICh0YW5oKTwvc3Bhbj4KICAgICAgICA8c3BhbiBjbGFzcz0idmVyZGljdCIgaWQ9InZlcmRpY3QiPuKAlDwvc3Bhbj4KICAgICAgPC9kaXY+CiAgICAgIDxwIGNsYXNzPSJoaW50IiBpZD0idmVyZGljdC10ZXh0Ij5DbG9zZXIgdG8gPGIgc3R5bGU9ImNvbG9yOnZhcigtLXBvcykiPisxPC9iPiA9IGxpa2VseSBnb29kIHRlbmFudCwgY2xvc2VyIHRvIDxiIHN0eWxlPSJjb2xvcjp2YXIoLS1uZWcpIj7iiJIxPC9iPiA9IGxpa2VseSByaXNreS48L3A+CiAgICA8L2Rpdj4KCiAgICA8ZGl2IGNsYXNzPSJwYW5lbCBzdGFnZSI+CiAgICAgIDxzdmcgaWQ9Im5ldCIgdmlld0JveD0iMCAwIDk4MCA1NjAiIHByZXNlcnZlQXNwZWN0UmF0aW89InhNaWRZTWlkIG1lZXQiIGFyaWEtbGFiZWw9Ik1MUCBkaWFncmFtIj48L3N2Zz4KICAgICAgPHAgY2xhc3M9ImhpbnQiIHN0eWxlPSJwYWRkaW5nOjAgNnB4IDRweCI+CiAgICAgICAgSG92ZXIgYSA8Yj5saW5lPC9iPiB0byBzZWUgaXRzIHdlaWdodCwgb3IgYSA8Yj5kb3Q8L2I+IHRvIHNlZSB0aGF0IHJldmlld2VyJ3MKICAgICAgICA8c3BhbiBjbGFzcz0ibW9ubyI+d8K3eCArIGI8L3NwYW4+IGFuZCBzcXVhc2hlZCBvcGluaW9uLiBMYXllcnMgbGVmdOKGknJpZ2h0OiBmb3JtIOKGkiBqdW5pb3IgcGFuZWwg4oaSIHNlbmlvciBwYW5lbCDihpIgZmluYWwganVkZ2UuCiAgICAgIDwvcD4KICAgIDwvZGl2PgogIDwvZGl2PgoKICA8ZGl2IGNsYXNzPSJ0aXAiIGlkPSJ0aXAiPjwvZGl2PgoKPHNjcmlwdD4KLy8gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQovLyBUaW55IE1MUCwgZmFpdGhmdWwgdG8gc2VjdGlvbiA1LjE6IE5ldXJvbiA9IHRhbmgod8K3eCArIGIpLCBMYXllciA9IG5ldXJvbnMgaW4KLy8gcGFyYWxsZWwsIE1MUCA9IGxheWVycyBpbiBzZXF1ZW5jZS4gU2hhcGU6IDIgLT4gOCAtPiA4IC0+IDEuCi8vIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KY29uc3QgU0laRVMgPSBbMiwgOCwgOCwgMV07CmNvbnN0IExBWUVSX1RJVExFUyA9IFsiSW5wdXRzIiwgIkhpZGRlbiBwYW5lbCAxIiwgIkhpZGRlbiBwYW5lbCAyIiwgIkZpbmFsIGp1ZGdlIl07CmNvbnN0IExBWUVSX0NBUFMgICA9IFsiaW5jb21lLCBjcmVkaXQiLCAiOCBqdW5pb3IgcmV2aWV3ZXJzIiwgIjggc2VuaW9yIHJldmlld2VycyIsICIxIHNjb3JlIl07Cgpjb25zdCByYW5kID0gKCkgPT4gTWF0aC5yYW5kb20oKSAqIDIgLSAxOyAgICAgICAgICAvLyB1bmlmb3JtKC0xLCAxKSwgbGlrZSBtaWNyb2dyYWQKY29uc3QgdGFuaCA9ICh6KSA9PiBNYXRoLnRhbmgoeik7CgovLyBCdWlsZCBuZXR3b3JrOiB3ZWlnaHRzW2xdIGlzIG1hdHJpeCBbbm91dCB4IG5pbl0sIGJpYXNlc1tsXSBpcyBbbm91dF0uCmxldCB3ZWlnaHRzLCBiaWFzZXM7CmZ1bmN0aW9uIGJ1aWxkTmV0d29yaygpIHsKICB3ZWlnaHRzID0gW107IGJpYXNlcyA9IFtdOwogIGZvciAobGV0IGwgPSAwOyBsIDwgU0laRVMubGVuZ3RoIC0gMTsgbCsrKSB7CiAgICBjb25zdCBuaW4gPSBTSVpFU1tsXSwgbm91dCA9IFNJWkVTW2wgKyAxXTsKICAgIHdlaWdodHMucHVzaChBcnJheS5mcm9tKHtsZW5ndGg6IG5vdXR9LCAoKSA9PiBBcnJheS5mcm9tKHtsZW5ndGg6IG5pbn0sIHJhbmQpKSk7CiAgICBiaWFzZXMucHVzaChBcnJheS5mcm9tKHtsZW5ndGg6IG5vdXR9LCByYW5kKSk7CiAgfQp9CmJ1aWxkTmV0d29yaygpOwoKLy8gRm9yd2FyZCBwYXNzLiBSZXR1cm5zIGFjdGl2YXRpb25zIHBlciBsYXllciAobGF5ZXIgMCA9IHJhdyBpbnB1dHMpIGFuZCB0aGUKLy8gcHJlLWFjdGl2YXRpb24gKHfCt3grYikgZm9yIGVhY2ggbmV1cm9uIHNvIHRoZSB0b29sdGlwIGNhbiBzaG93IGJvdGguCmZ1bmN0aW9uIGZvcndhcmQoeCkgewogIGNvbnN0IGFjdHMgPSBbeC5zbGljZSgpXTsKICBjb25zdCBwcmVhY3RzID0gW3guc2xpY2UoKV07ICAgLy8gaW5wdXRzIGhhdmUgbm8gcHJlLWFjdGl2YXRpb247IG1pcnJvciBmb3IgaW5kZXhpbmcKICBsZXQgY3VyID0geC5zbGljZSgpOwogIGZvciAobGV0IGwgPSAwOyBsIDwgd2VpZ2h0cy5sZW5ndGg7IGwrKykgewogICAgY29uc3QgeiA9IFtdLCBhID0gW107CiAgICBmb3IgKGxldCBqID0gMDsgaiA8IHdlaWdodHNbbF0ubGVuZ3RoOyBqKyspIHsKICAgICAgbGV0IHMgPSBiaWFzZXNbbF1bal07CiAgICAgIGZvciAobGV0IGkgPSAwOyBpIDwgY3VyLmxlbmd0aDsgaSsrKSBzICs9IHdlaWdodHNbbF1bal1baV0gKiBjdXJbaV07CiAgICAgIHoucHVzaChzKTsgYS5wdXNoKHRhbmgocykpOwogICAgfQogICAgcHJlYWN0cy5wdXNoKHopOyBhY3RzLnB1c2goYSk7IGN1ciA9IGE7CiAgfQogIHJldHVybiB7IGFjdHMsIHByZWFjdHMgfTsKfQoKLy8gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQovLyBMYXlvdXQgZ2VvbWV0cnkKLy8gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpjb25zdCBXID0gOTgwLCBIID0gNTYwLCBQQURfWCA9IDkwLCBQQURfVE9QID0gNzAsIFBBRF9CT1QgPSA1MDsKY29uc3Qgc3ZnID0gZDMuc2VsZWN0KCIjbmV0Iik7CmNvbnN0IHRpcCA9IGQzLnNlbGVjdCgiI3RpcCIpOwoKZnVuY3Rpb24gbGF5ZXJYKGwpIHsgcmV0dXJuIFBBRF9YICsgKFcgLSAyICogUEFEX1gpICogKGwgLyAoU0laRVMubGVuZ3RoIC0gMSkpOyB9CmZ1bmN0aW9uIG5vZGVZKGwsIGopIHsKICBjb25zdCBuID0gU0laRVNbbF07CiAgY29uc3QgdG9wID0gUEFEX1RPUCwgYm90ID0gSCAtIFBBRF9CT1QsIHNwYW4gPSBib3QgLSB0b3A7CiAgaWYgKG4gPT09IDEpIHJldHVybiAodG9wICsgYm90KSAvIDI7CiAgcmV0dXJuIHRvcCArIHNwYW4gKiAoaiAvIChuIC0gMSkpOwp9CgovLyBjb2xvciBzY2FsZXM6IHdlaWdodHMgJiBhY3RpdmF0aW9ucyBtYXBwZWQgdG8gcmVkKC0pL2dyZWVuKCspCmZ1bmN0aW9uIHdlaWdodENvbG9yKHcpIHsgcmV0dXJuIHcgPj0gMCA/ICJ2YXIoLS1wb3MpIiA6ICJ2YXIoLS1uZWcpIjsgfQpmdW5jdGlvbiBhY3RDb2xvcihhKSAgICB7IHJldHVybiBhID49IDAgPyAidmFyKC0tcG9zKSIgOiAidmFyKC0tbmVnKSI7IH0KZnVuY3Rpb24gd2VpZ2h0V2lkdGgodykgeyByZXR1cm4gMC42ICsgTWF0aC5taW4oNS41LCBNYXRoLmFicyh3KSAqIDMuMik7IH0KCmNvbnN0IGdFZGdlcyAgPSBzdmcuYXBwZW5kKCJnIikuYXR0cigiY2xhc3MiLCAiZWRnZXMiKTsKY29uc3QgZ05vZGVzICA9IHN2Zy5hcHBlbmQoImciKS5hdHRyKCJjbGFzcyIsICJub2RlcyIpOwpjb25zdCBnVGl0bGVzID0gc3ZnLmFwcGVuZCgiZyIpLmF0dHIoImNsYXNzIiwgInRpdGxlcyIpOwoKLy8gbGF5ZXIgdGl0bGVzICsgY2FwdGlvbnMKZnVuY3Rpb24gZHJhd1RpdGxlcygpIHsKICBnVGl0bGVzLnNlbGVjdEFsbCgiKiIpLnJlbW92ZSgpOwogIFNJWkVTLmZvckVhY2goKG4sIGwpID0+IHsKICAgIGdUaXRsZXMuYXBwZW5kKCJ0ZXh0IikuYXR0cigiY2xhc3MiLCAibGF5ZXItdGl0bGUiKQogICAgICAuYXR0cigieCIsIGxheWVyWChsKSkuYXR0cigieSIsIDI4KS50ZXh0KExBWUVSX1RJVExFU1tsXSk7CiAgICBnVGl0bGVzLmFwcGVuZCgidGV4dCIpLmF0dHIoImNsYXNzIiwgImxheWVyLWNhcCIpCiAgICAgIC5hdHRyKCJ4IiwgbGF5ZXJYKGwpKS5hdHRyKCJ5IiwgNDYpLnRleHQoTEFZRVJfQ0FQU1tsXSk7CiAgfSk7Cn0KCi8vIGJ1aWxkIGVkZ2VzICsgbm9kZXMgb25jZSAoZ2VvbWV0cnkgaXMgc3RhdGljOyBvbmx5IGNvbG9yL3dpZHRoL2FuaW0gdXBkYXRlKQpsZXQgZWRnZVNlbCwgbm9kZVNlbDsKZnVuY3Rpb24gZHJhd1N0cnVjdHVyZSgpIHsKICAvLyBlZGdlcwogIGNvbnN0IGVkZ2VzID0gW107CiAgZm9yIChsZXQgbCA9IDA7IGwgPCB3ZWlnaHRzLmxlbmd0aDsgbCsrKSB7CiAgICBmb3IgKGxldCBqID0gMDsgaiA8IFNJWkVTW2wgKyAxXTsgaisrKSB7CiAgICAgIGZvciAobGV0IGkgPSAwOyBpIDwgU0laRVNbbF07IGkrKykgewogICAgICAgIGVkZ2VzLnB1c2goeyBsLCBpLCBqLAogICAgICAgICAgeDE6IGxheWVyWChsKSwgICAgIHkxOiBub2RlWShsLCBpKSwKICAgICAgICAgIHgyOiBsYXllclgobCArIDEpLCB5Mjogbm9kZVkobCArIDEsIGopIH0pOwogICAgICB9CiAgICB9CiAgfQogIGVkZ2VTZWwgPSBnRWRnZXMuc2VsZWN0QWxsKCJsaW5lIikuZGF0YShlZGdlcykuam9pbigibGluZSIpCiAgICAuYXR0cigiY2xhc3MiLCAiZWRnZSIpCiAgICAuYXR0cigieDEiLCBkID0+IGQueDEpLmF0dHIoInkxIiwgZCA9PiBkLnkxKQogICAgLmF0dHIoIngyIiwgZCA9PiBkLngyKS5hdHRyKCJ5MiIsIGQgPT4gZC55MikKICAgIC5vbigibW91c2Vtb3ZlIiwgKGUsIGQpID0+IHsKICAgICAgY29uc3QgdyA9IHdlaWdodHNbZC5sXVtkLmpdW2QuaV07CiAgICAgIHNob3dUaXAoZSwgYDxiPldlaWdodDwvYj4gwrcgcGFuZWwgbGluZTxicj4KICAgICAgICA8c3BhbiBjbGFzcz0ibW9ubyI+bGF5ZXIgJHtkLmx9IOKGkiAke2QubCArIDF9PC9zcGFuPjxicj4KICAgICAgICBmcm9tIG5ldXJvbiAke2QuaX0g4oaSIG5ldXJvbiAke2Quan08YnI+CiAgICAgICAgPHNwYW4gY2xhc3M9Im1vbm8iPncgPSAke3cudG9GaXhlZCgzKX08L3NwYW4+PGJyPgogICAgICAgIDxzcGFuIHN0eWxlPSJjb2xvcjoke3c+PTA/J3ZhcigtLXBvcyknOid2YXIoLS1uZWcpJ30iPiR7dz49MD8nYm9vc3RzJzonZGFtcHMnfSB0aGlzIHNpZ25hbDwvc3Bhbj5gKTsKICAgIH0pCiAgICAub24oIm1vdXNlbGVhdmUiLCBoaWRlVGlwKTsKCiAgLy8gbm9kZXMKICBjb25zdCBub2RlcyA9IFtdOwogIFNJWkVTLmZvckVhY2goKG4sIGwpID0+IHsKICAgIGZvciAobGV0IGogPSAwOyBqIDwgbjsgaisrKSBub2Rlcy5wdXNoKHsgbCwgaiwgeDogbGF5ZXJYKGwpLCB5OiBub2RlWShsLCBqKSB9KTsKICB9KTsKICBjb25zdCBnID0gZ05vZGVzLnNlbGVjdEFsbCgiZy5ub2RlIikuZGF0YShub2Rlcykuam9pbigiZyIpLmF0dHIoImNsYXNzIiwgIm5vZGUiKQogICAgLmF0dHIoInRyYW5zZm9ybSIsIGQgPT4gYHRyYW5zbGF0ZSgke2QueH0sJHtkLnl9KWApOwogIGcuc2VsZWN0QWxsKCJjaXJjbGUiKS5kYXRhKGQgPT4gW2RdKS5qb2luKCJjaXJjbGUiKS5hdHRyKCJyIiwgZCA9PiBkLmwgPT09IDAgfHwgZC5sID09PSBTSVpFUy5sZW5ndGggLSAxID8gMTMgOiAxMSk7CiAgbm9kZVNlbCA9IGc7CgogIC8vIGlucHV0IHJvdyBsYWJlbHMKICBnTm9kZXMuc2VsZWN0QWxsKCJ0ZXh0LmluLWxhYiIpLnJlbW92ZSgpOwogIFsieOKCgSBpbmNvbWUiLCAieOKCgiBjcmVkaXQiXS5mb3JFYWNoKCh0LCBpKSA9PiB7CiAgICBnTm9kZXMuYXBwZW5kKCJ0ZXh0IikuYXR0cigiY2xhc3MiLCAibm9kZS1sYWJlbCBpbi1sYWIiKQogICAgICAuYXR0cigieCIsIGxheWVyWCgwKSAtIDIyKS5hdHRyKCJ5Iiwgbm9kZVkoMCwgaSkgKyA0KQogICAgICAuYXR0cigidGV4dC1hbmNob3IiLCAiZW5kIikudGV4dCh0KTsKICB9KTsKCiAgZy5vbigibW91c2Vtb3ZlIiwgKGUsIGQpID0+IHsKICAgIGlmIChkLmwgPT09IDApIHsKICAgICAgc2hvd1RpcChlLCBgPGI+SW5wdXQ8L2I+PGJyPjxzcGFuIGNsYXNzPSJtb25vIj54JHtkLmorMX0gPSAke3N0YXRlLmFjdHNbMF1bZC5qXS50b0ZpeGVkKDMpfTwvc3Bhbj48YnI+JHtkLmo9PT0wPydtb250aGx5IGluY29tZSc6J2NyZWRpdCBzY29yZSd9IChyYXcgZm9ybSBmaWVsZClgKTsKICAgIH0gZWxzZSB7CiAgICAgIGNvbnN0IHogPSBzdGF0ZS5wcmVhY3RzW2QubF1bZC5qXSwgYSA9IHN0YXRlLmFjdHNbZC5sXVtkLmpdOwogICAgICBjb25zdCByb2xlID0gZC5sID09PSBTSVpFUy5sZW5ndGggLSAxID8gImZpbmFsIGp1ZGdlIiA6IGByZXZpZXdlciAke2Quan0gaW4gcGFuZWwgJHtkLmx9YDsKICAgICAgc2hvd1RpcChlLCBgPGI+TmV1cm9uPC9iPiDCtyAke3JvbGV9PGJyPgogICAgICAgIDxzcGFuIGNsYXNzPSJtb25vIj53wrd4ICsgYiA9ICR7ei50b0ZpeGVkKDMpfTwvc3Bhbj48YnI+CiAgICAgICAgPHNwYW4gY2xhc3M9Im1vbm8iPnRhbmgoLi4uKSA9IDxiIHN0eWxlPSJjb2xvcjoke2E+PTA/J3ZhcigtLXBvcyknOid2YXIoLS1uZWcpJ30iPiR7YS50b0ZpeGVkKDMpfTwvYj48L3NwYW4+PGJyPgogICAgICAgICR7YT49MD8nbGVhbmluZyB5ZXMnOidsZWFuaW5nIG5vJ31gKTsKICAgIH0KICB9KS5vbigibW91c2VsZWF2ZSIsIGhpZGVUaXApOwp9CgovLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCi8vIFBhaW50IGN1cnJlbnQgc3RhdGUgKGNvbG9ycyArIHdpZHRocykgb250byBlZGdlcyBhbmQgbm9kZXMKLy8gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpsZXQgc3RhdGUgPSB7IGFjdHM6IFtdLCBwcmVhY3RzOiBbXSB9OwoKZnVuY3Rpb24gcmVjb21wdXRlKCkgewogIGNvbnN0IHgxID0gK2QzLnNlbGVjdCgiI3gxIikucHJvcGVydHkoInZhbHVlIik7CiAgY29uc3QgeDIgPSArZDMuc2VsZWN0KCIjeDIiKS5wcm9wZXJ0eSgidmFsdWUiKTsKICBzdGF0ZSA9IGZvcndhcmQoW3gxLCB4Ml0pOwogIHBhaW50KCk7CiAgdXBkYXRlVmVyZGljdCgpOwp9CgpmdW5jdGlvbiBwYWludCgpIHsKICBlZGdlU2VsCiAgICAuYXR0cigic3Ryb2tlIiwgZCA9PiB3ZWlnaHRDb2xvcih3ZWlnaHRzW2QubF1bZC5qXVtkLmldKSkKICAgIC5hdHRyKCJzdHJva2Utd2lkdGgiLCBkID0+IHdlaWdodFdpZHRoKHdlaWdodHNbZC5sXVtkLmpdW2QuaV0pKQogICAgLmF0dHIoInN0cm9rZS1vcGFjaXR5IiwgMC40NSk7CgogIG5vZGVTZWwuc2VsZWN0KCJjaXJjbGUiKQogICAgLmF0dHIoImZpbGwiLCBkID0+IHsKICAgICAgY29uc3QgYSA9IHN0YXRlLmFjdHNbZC5sXSA/IHN0YXRlLmFjdHNbZC5sXVtkLmpdIDogMDsKICAgICAgY29uc3QgbWFnID0gTWF0aC5taW4oMSwgTWF0aC5hYnMoYSkpOwogICAgICBjb25zdCBiYXNlID0gYSA+PSAwID8gWzc0LDIyMiwxMjhdIDogWzI0OCwxMTMsMTEzXTsKICAgICAgLy8gYmxlbmQgdG93YXJkIGRhcmsgcGFuZWwgZm9yIGxvdyBtYWduaXR1ZGUKICAgICAgY29uc3QgbWl4ID0gKGMpID0+IE1hdGgucm91bmQoNDAgKyAoYyAtIDQwKSAqICgwLjI1ICsgMC43NSAqIG1hZykpOwogICAgICByZXR1cm4gYHJnYigke21peChiYXNlWzBdKX0sJHttaXgoYmFzZVsxXSl9LCR7bWl4KGJhc2VbMl0pfSlgOwogICAgfSk7Cn0KCmZ1bmN0aW9uIHVwZGF0ZVZlcmRpY3QoKSB7CiAgY29uc3Qgb3V0ID0gc3RhdGUuYWN0c1tzdGF0ZS5hY3RzLmxlbmd0aCAtIDFdWzBdOwogIGNvbnN0IGVsID0gZDMuc2VsZWN0KCIjdmVyZGljdCIpOwogIGVsLnRleHQoKG91dCA+PSAwID8gIisiIDogIiIpICsgb3V0LnRvRml4ZWQoMykpCiAgICAuc3R5bGUoImNvbG9yIiwgb3V0ID49IDAgPyAidmFyKC0tcG9zKSIgOiAidmFyKC0tbmVnKSIpOwogIGQzLnNlbGVjdCgiI3ZlcmRpY3QtdGV4dCIpLmh0bWwoCiAgICBvdXQgPj0gMAogICAgICA/IGBTY29yZSA8YiBzdHlsZT0iY29sb3I6dmFyKC0tcG9zKSI+JHtvdXQudG9GaXhlZCgyKX08L2I+IOKGkiBsZWFuaW5nIDxiPmdvb2QgdGVuYW50PC9iPi5gCiAgICAgIDogYFNjb3JlIDxiIHN0eWxlPSJjb2xvcjp2YXIoLS1uZWcpIj4ke291dC50b0ZpeGVkKDIpfTwvYj4g4oaSIGxlYW5pbmcgPGI+cmlza3k8L2I+LmAKICApOwp9CgovLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCi8vIEFuaW1hdGVkIGZvcndhcmQgcGFzczogbGlnaHQgdXAgbGF5ZXIgYnkgbGF5ZXIsIHB1bHNlIGVkZ2VzIGJldHdlZW4gdGhlbQovLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmxldCBwbGF5aW5nID0gZmFsc2U7CmZ1bmN0aW9uIGFuaW1hdGUoKSB7CiAgaWYgKHBsYXlpbmcpIHJldHVybjsKICBwbGF5aW5nID0gdHJ1ZTsKICAvLyBkaW0gZXZlcnl0aGluZyBmaXJzdAogIGVkZ2VTZWwuYXR0cigic3Ryb2tlLW9wYWNpdHkiLCAwLjA4KTsKICBub2RlU2VsLnNlbGVjdCgiY2lyY2xlIikuYXR0cigib3BhY2l0eSIsIDAuMjUpOwogIGQzLnNlbGVjdCgiI3BsYXkiKS5hdHRyKCJkaXNhYmxlZCIsIHRydWUpLnRleHQoIuKWtiBwbGF5aW5n4oCmIik7CgogIGNvbnN0IFNURVAgPSA2MjA7CiAgLy8gbGlnaHQgaW5wdXQgbGF5ZXIKICBsaWdodExheWVyKDAsIDApOwogIGZvciAobGV0IGwgPSAwOyBsIDwgd2VpZ2h0cy5sZW5ndGg7IGwrKykgewogICAgcHVsc2VFZGdlcyhsLCBTVEVQICogKGwgKyAwLjE1KSk7CiAgICBsaWdodExheWVyKGwgKyAxLCBTVEVQICogKGwgKyAxKSk7CiAgfQogIHNldFRpbWVvdXQoKCkgPT4gewogICAgZWRnZVNlbC50cmFuc2l0aW9uKCkuZHVyYXRpb24oMzAwKS5hdHRyKCJzdHJva2Utb3BhY2l0eSIsIDAuNDUpOwogICAgbm9kZVNlbC5zZWxlY3QoImNpcmNsZSIpLnRyYW5zaXRpb24oKS5kdXJhdGlvbigzMDApLmF0dHIoIm9wYWNpdHkiLCAxKTsKICAgIHBsYXlpbmcgPSBmYWxzZTsKICAgIGQzLnNlbGVjdCgiI3BsYXkiKS5hdHRyKCJkaXNhYmxlZCIsIG51bGwpLnRleHQoIuKWtiBBbmltYXRlIGZvcndhcmQgcGFzcyIpOwogIH0sIFNURVAgKiAod2VpZ2h0cy5sZW5ndGggKyAxKSArIDI1MCk7Cn0KCmZ1bmN0aW9uIGxpZ2h0TGF5ZXIobCwgZGVsYXkpIHsKICBub2RlU2VsLmZpbHRlcihkID0+IGQubCA9PT0gbCkuc2VsZWN0KCJjaXJjbGUiKQogICAgLnRyYW5zaXRpb24oKS5kZWxheShkZWxheSkuZHVyYXRpb24oMjgwKQogICAgLmF0dHIoIm9wYWNpdHkiLCAxKQogICAgLmF0dHIoInIiLCBkID0+IChkLmwgPT09IDAgfHwgZC5sID09PSBTSVpFUy5sZW5ndGggLSAxID8gMTMgOiAxMSkgKyA0KQogICAgLnRyYW5zaXRpb24oKS5kdXJhdGlvbigyMjApCiAgICAuYXR0cigiciIsIGQgPT4gZC5sID09PSAwIHx8IGQubCA9PT0gU0laRVMubGVuZ3RoIC0gMSA/IDEzIDogMTEpOwp9CgpmdW5jdGlvbiBwdWxzZUVkZ2VzKGwsIGRlbGF5KSB7CiAgZWRnZVNlbC5maWx0ZXIoZCA9PiBkLmwgPT09IGwpCiAgICAudHJhbnNpdGlvbigpLmRlbGF5KGRlbGF5KS5kdXJhdGlvbigzMDApLmF0dHIoInN0cm9rZS1vcGFjaXR5IiwgMC45KQogICAgLnRyYW5zaXRpb24oKS5kdXJhdGlvbigzMjApLmF0dHIoInN0cm9rZS1vcGFjaXR5IiwgMC4xOCk7Cn0KCi8vIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KLy8gVG9vbHRpcCBoZWxwZXJzCi8vIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KZnVuY3Rpb24gc2hvd1RpcChlLCBodG1sKSB7CiAgdGlwLmh0bWwoaHRtbCkuc3R5bGUoIm9wYWNpdHkiLCAxKQogICAgLnN0eWxlKCJsZWZ0IiwgKGUuY2xpZW50WCArIDE0KSArICJweCIpCiAgICAuc3R5bGUoInRvcCIsICAoZS5jbGllbnRZICsgMTQpICsgInB4Iik7Cn0KZnVuY3Rpb24gaGlkZVRpcCgpIHsgdGlwLnN0eWxlKCJvcGFjaXR5IiwgMCk7IH0KCi8vIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KLy8gV2lyZSB1cCBjb250cm9scwovLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmZ1bmN0aW9uIHN5bmNMYWJlbHMoKSB7CiAgZDMuc2VsZWN0KCIjeDF2IikudGV4dCgoK2QzLnNlbGVjdCgiI3gxIikucHJvcGVydHkoInZhbHVlIikpLnRvRml4ZWQoMikpOwogIGQzLnNlbGVjdCgiI3gydiIpLnRleHQoKCtkMy5zZWxlY3QoIiN4MiIpLnByb3BlcnR5KCJ2YWx1ZSIpKS50b0ZpeGVkKDIpKTsKfQpkMy5zZWxlY3QoIiN4MSIpLm9uKCJpbnB1dCIsICgpID0+IHsgc3luY0xhYmVscygpOyByZWNvbXB1dGUoKTsgfSk7CmQzLnNlbGVjdCgiI3gyIikub24oImlucHV0IiwgKCkgPT4geyBzeW5jTGFiZWxzKCk7IHJlY29tcHV0ZSgpOyB9KTsKZDMuc2VsZWN0KCIjcGxheSIpLm9uKCJjbGljayIsIGFuaW1hdGUpOwpkMy5zZWxlY3QoIiNyZXJvbGwiKS5vbigiY2xpY2siLCAoKSA9PiB7IGJ1aWxkTmV0d29yaygpOyBkcmF3U3RydWN0dXJlKCk7IHJlY29tcHV0ZSgpOyB9KTsKZDMuc2VsZWN0KCIjcmVzZXQiKS5vbigiY2xpY2siLCAoKSA9PiB7CiAgZDMuc2VsZWN0KCIjeDEiKS5wcm9wZXJ0eSgidmFsdWUiLCAwLjUpOwogIGQzLnNlbGVjdCgiI3gyIikucHJvcGVydHkoInZhbHVlIiwgLTAuMyk7CiAgc3luY0xhYmVscygpOyByZWNvbXB1dGUoKTsKfSk7CgovLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCi8vIEluaXQKLy8gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpkcmF3VGl0bGVzKCk7CmRyYXdTdHJ1Y3R1cmUoKTsKc3luY0xhYmVscygpOwpyZWNvbXB1dGUoKTsKPC9zY3JpcHQ+CjwvYm9keT4KPC9odG1sPgo=\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" \n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x236e4d71ac0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Interactive companion to section 5.1 — drag the inputs, hover lines/dots, press play.\n",
"# Shows the same 2 -> 8 -> 8 -> 1 reviewer-panel network as the code below.\n",
"import base64\n",
"from pathlib import Path\n",
"from IPython.display import IFrame, display\n",
"\n",
"html_path = Path(\"mlp_playground.html\")\n",
"if not html_path.exists():\n",
" html_path = Path.cwd() / \"mlp_playground.html\"\n",
"\n",
"if not html_path.exists():\n",
" raise FileNotFoundError(\n",
" \"mlp_playground.html not found. Keep it next to this notebook, then re-run.\"\n",
" )\n",
"\n",
"# Notebook UIs often cannot load local files via IFrame src=\"file.html\".\n",
"# Pack the page into a data URI so the browser runs it inline with scripts intact.\n",
"b64 = base64.b64encode(html_path.read_bytes()).decode(\"ascii\")\n",
"display(IFrame(src=f\"data:text/html;base64,{b64}\", width=\"100%\", height=640))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1d260369",
"metadata": {},
"outputs": [],
"source": [
"class Neuron:\n",
" def __init__(self, nin):\n",
" # one weight per input + one bias — all are Value objects (trainable knobs)\n",
" self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]\n",
" self.b = Value(random.uniform(-1, 1))\n",
"\n",
" def __call__(self, x):\n",
" # weighted sum: w1*x1 + w2*x2 + ... + b (each * and + builds the graph)\n",
" act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)\n",
" return act.tanh() # squash activation to (-1, 1)\n",
"\n",
" def parameters(self):\n",
" return self.w + [self.b] # every knob this neuron owns\n",
"\n",
"\n",
"class Layer:\n",
" def __init__(self, nin, nout):\n",
" self.neurons = [Neuron(nin) for _ in range(nout)] # nout parallel neurons\n",
"\n",
" def __call__(self, x):\n",
" outs = [n(x) for n in self.neurons] # each neuron sees the same input vector x\n",
" return outs[0] if len(outs) == 1 else outs # scalar if one neuron, else list\n",
"\n",
" def parameters(self):\n",
" return [p for n in self.neurons for p in n.parameters()] # flatten all weights\n",
"\n",
"\n",
"class MLP:\n",
" def __init__(self, nin, nouts):\n",
" sizes = [nin] + nouts # e.g. [2, 8, 8, 1] for 2→8→8→1\n",
" # layer i connects sizes[i] inputs to sizes[i+1] outputs\n",
" self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(nouts))]\n",
"\n",
" def __call__(self, x):\n",
" for layer in self.layers: # data flows forward through each layer\n",
" x = layer(x)\n",
" return x\n",
"\n",
" def parameters(self):\n",
" return [p for layer in self.layers for p in layer.parameters()] # all trainable Values\n",
"\n",
"\n",
"net = MLP(2, [8, 8, 1]) # 2 inputs -> 8 hidden -> 8 hidden -> 1 output\n",
"print(\"parameter count:\", len(net.parameters()))\n",
"print(\"one prediction:\", net([Value(0.5), Value(-0.2)]))"
]
},
{
"cell_type": "markdown",
"id": "fc0b5b7b",
"metadata": {},
"source": [
"### 6.1 Train the MLP on the two moons\n",
"\n",
"Loop the four familiar steps:\n",
"\n",
"1. **forward** — predict on every point\n",
"2. **loss** — how wrong (here: mean-squared error against labels mapped to -1 / +1)\n",
"3. **backward** — `loss.backward()` fills every parameter's `.grad`\n",
"4. **update** — nudge each parameter a little **against** its gradient (downhill)\n",
"\n",
"Watch the loss fall. (Pure-Python micrograd is slow, so we keep the net and step count small.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de95ef11",
"metadata": {},
"outputs": [],
"source": [
"net = MLP(2, [8, 8, 1]) # fresh network for training\n",
"\n",
"# map labels {0,1} -> {-1,+1} to match tanh output range (-1 to +1)\n",
"ys = [1.0 if yi == 1 else -1.0 for yi in y]\n",
"# wrap each (x, y) coordinate pair as Value objects so forward builds a graph\n",
"Xv = [[Value(float(xi[0])), Value(float(xi[1]))] for xi in X]\n",
"\n",
"lr = 0.1 # learning rate — how big each knob turn per step\n",
"for step in range(60): # 60 gradient-descent steps (small count: micrograd is slow)\n",
"\n",
" # --- 1. FORWARD: predict every point, build one big graph ending in loss ---\n",
" preds = [net(xrow) for xrow in Xv]\n",
" loss = sum(((p - yt) ** 2 for p, yt in zip(preds, ys)), Value(0.0)) # sum of squared errors\n",
" loss = loss * (1.0 / len(ys)) # average MSE so scale doesn't grow with dataset size\n",
"\n",
" # --- 2. BACKWARD: zero old grads, then propagate blame from loss to every weight ---\n",
" for p in net.parameters():\n",
" p.grad = 0.0 # must reset — backward() accumulates with +=\n",
" loss.backward() # fills p.grad for every parameter\n",
"\n",
" # --- 3. UPDATE: walk downhill — turn each knob opposite its gradient ---\n",
" for p in net.parameters():\n",
" p.data -= lr * p.grad # new value = old value - lr * (how loss rises if knob rises)\n",
"\n",
" if step % 10 == 0 or step == 59:\n",
" # sign match: prediction > 0 same as target > 0 counts as correct\n",
" acc = sum((1 if (p.data > 0) == (yt > 0) else 0) for p, yt in zip(preds, ys)) / len(ys)\n",
" print(f\"step {step:3d} loss {loss.data:.4f} acc {acc:.2%}\")"
]
},
{
"cell_type": "markdown",
"id": "9cb30391",
"metadata": {},
"source": [
"### 6.2 See the boundary it learned\n",
"\n",
"Evaluate the trained net across a grid of points and color the regions. The swirl between the\n",
"two moons should now be split by a curved boundary — the network \"drew the line.\"\n",
"\n",
"(Coarse grid for speed; each grid point is a full forward pass through micrograd.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ee39626a",
"metadata": {},
"outputs": [],
"source": [
"if HAS_PLT:\n",
" # pad plot area a little beyond the data points\n",
" xmin, xmax = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n",
" ymin, ymax = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n",
" xs = np.linspace(xmin, xmax, 30) # 30 grid columns\n",
" ys_grid = np.linspace(ymin, ymax, 30) # 30 grid rows\n",
"\n",
" Z = np.zeros((len(ys_grid), len(xs))) # store net output at each grid cell\n",
" for i, gy in enumerate(ys_grid):\n",
" for j, gx in enumerate(xs):\n",
" out = net([Value(float(gx)), Value(float(gy))]) # forward only (no training)\n",
" Z[i, j] = out.data # >0 one side of boundary, <0 other side\n",
"\n",
" plt.figure(figsize=(5, 4))\n",
" plt.contourf(xs, ys_grid, Z, levels=[-1e9, 0, 1e9], cmap=\"bwr\", alpha=0.3) # colored regions\n",
" plt.scatter(X[:, 0], X[:, 1], c=y, s=20, cmap=\"bwr\") # training dots on top\n",
" plt.title(\"decision boundary learned by the micrograd MLP\")\n",
" plt.show()\n",
"else:\n",
" print(\"matplotlib missing - skipping the boundary plot (training above still ran)\")"
]
},
{
"cell_type": "markdown",
"id": "b00ac0fd",
"metadata": {},
"source": [
"**Your turn 6** — Try `MLP(2, [16, 16, 1])` or a different learning rate / step count and\n",
"re-run 6.1 and 6.2. Does a bigger net reach higher accuracy? Does too-large a learning rate\n",
"make the loss bounce instead of fall?"
]
},
{
"cell_type": "markdown",
"id": "48623e88",
"metadata": {},
"source": [
"### 7.1 Bridge to PyTorch — same gradients, industrial engine\n",
"\n",
"PyTorch is micrograd scaled up to fast tensors. The *idea* is identical: build an expression,\n",
"call `.backward()`, read `.grad`. Here we redo the tiny `a*b + c` (then a tanh) in PyTorch and\n",
"confirm the gradients match what our engine produced.\n",
"\n",
"**-> Training:** in notebooks `01` and `06` we let PyTorch's autograd do exactly this — now you\n",
"know what it is doing under the hood."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e6c3140e",
"metadata": {},
"outputs": [],
"source": [
"# On some Windows setups torch and numpy ship duplicate OpenMP DLLs; this avoids a clash.\n",
"import os\n",
"os.environ.setdefault(\"KMP_DUPLICATE_LIB_OK\", \"TRUE\")\n",
"\n",
"try:\n",
" import torch\n",
"\n",
" # requires_grad=True tells PyTorch to track history (like our Value graph)\n",
" at = torch.tensor([0.5], requires_grad=True)\n",
" bt = torch.tensor([2.0], requires_grad=True)\n",
" ct = torch.tensor([-1.0], requires_grad=True)\n",
"\n",
" gt = torch.tanh(at * bt + ct) # same expression as micrograd Your turn 4\n",
" gt.backward() # PyTorch autograd — same idea as Value.backward()\n",
" print(\"PyTorch: a.grad =\", at.grad.item())\n",
"except OSError as e:\n",
" print(\"PyTorch could not load in this environment:\", e)\n",
" print(\"(That is fine here - the micrograd result below is the point.)\")\n",
"\n",
"# our engine, same expression — should match PyTorch's a.grad\n",
"a = Value(0.5); b = Value(2.0); c = Value(-1.0)\n",
"(a * b + c).tanh().backward()\n",
"print(\"micrograd: a.grad =\", a.grad)"
]
},
{
"cell_type": "markdown",
"id": "49c1ebf8",
"metadata": {},
"source": [
"## What's next\n",
"\n",
"You built the engine that powers all of deep learning:\n",
"\n",
"- a **number that remembers** its history (`Value`)\n",
"- **backpropagation** via the chain rule (`backward()`)\n",
"- a small **MLP** trained by nudging weights downhill\n",
"\n",
"Where this leads in the series:\n",
"\n",
"- `01_build_gpt.ipynb` — language modeling basics (the bigram), now that backprop is no longer\n",
" mysterious.\n",
"- `02`-`05` — scale these same ideas up with PyTorch on real character data (`names.txt`).\n",
"- `06_build_gpt_attention.ipynb` — the transformer, where `loss.backward()` is doing exactly\n",
" what you built here, just across millions of `Value`-like nodes.\n",
"\n",
"**Checklist**\n",
"- [ ] A `Value` records data, parents, and operation\n",
"- [ ] Each op defines a local `_backward`\n",
"- [ ] `backward()` = topological order + chain rule\n",
"- [ ] Neuron / Layer / MLP built from `Value`\n",
"- [ ] Trained on two moons; boundary learned\n",
"- [ ] Confirmed gradients match PyTorch"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.10)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink