This simulation explains how a neural network with backpropagation is trained, using a very, very simple example. We use only one data point, with a value for the outcome \(y = 0.8\) and a covariate \(x = 1.5\). The button 2 SAMPLES adds one more data point: \(x = -0.5\) and \(y = 0.6\), which lets us compare updating the weights after each observation (SGD) with averaging both gradients first (BATCH). The button 2 INPUTS adds a second covariate, \(x_2\), so the example becomes \((x_1, x_2, y) = (1.5, -1.0, 0.8)\) — and, with two samples, \((-0.5, 0.8, 0.6)\) as the second observation.
FIG. S1 — Layout of the simulator (shown in its 1-input, 1-sample configuration).
1 Network graph — the live diagram. Nodes display their pre-activation \(z\) and output \(a\); weight boxes sit on the edges and update after every gradient step. 2 Status readout — current sample, epoch counter, latest loss. 3 Phase chips — light up to show which stage is running (cyan = forward, gold = loss, red = backward). 4 Controls — see S3. 5 Trace panel — every formula with the current numbers substituted, one line per phase. 6 Loss chart — mean loss per epoch. 7 Configuration toggles — architecture and training scheme (see S3).
Pressing STEP advances one phase at a time. A full pass over one observation has six:
| PHASE | ON SCREEN | QUANTITY PRODUCED |
|---|---|---|
| [1] FWD in→n1 | cyan pulse(s) travel input → neuron 1 | \(z_1,\; a_1\) |
| [2] FWD a₁→n2 | cyan pulse neuron 1 → neuron 2 | \(z_2,\; \hat y\) |
| [3] LOSS | gold pulse reaches the loss box | \(L\) |
| [4] BWD | red pulse travels loss → neuron 2 | \(\delta_2,\; \partial L/\partial w_{\text{out}}\) |
| [5] BWD | red pulse neuron 2 → neuron 1, fanning to each input weight | \(\delta_1,\; \partial L/\partial w_{\text{in}}\) |
| [6] UPDATE | weight boxes flash red and change value | new \(w_j\) |
FIG. S2 — Mid-backward-pass: the error signal travels right-to-left; the gradient it prices appears under the weight, and the active phase chip is lit red.
All numbers appearing on screen also land, with their substitutions, in the trace panel — the worked example in Part II reproduces the first epoch line by line.
| CONTROL | WHAT IT DOES |
|---|---|
| STEP ▸ | advance one phase — the lecture pace; pause on any stage to discuss |
| RUN × N | execute exactly N epochs (1–200) at accelerated speed |
| AUTO ∞ | loop epochs continuously until paused |
| RESET | restore initial weights and clear the loss history |
| η slider | learning rate, 0.1–5, applied to every subsequent update |
| INPUTS 1 | 2 | one input weight, or two inputs feeding neuron 1 (\(z_1 = w_1x_1 + w_2x_2\)) |
| DATA 1 | 2 SAMPLES | train on one observation, or on two — a gold badge shows which sample is flowing |
| SGD | BATCH | with 2 samples: update after each sample, or average both gradients into one update per epoch |
Suggested classroom experiments: (a) RUN 30 at \(\eta=2\), RESET, RUN 30 at \(\eta=5\) — step size vs overshoot; (b) with 2 samples, RUN 30 under SGD then under BATCH — noisy vs smooth descent; (c) in 2-input mode, ask why the two input gradients carry opposite signs before revealing \(\partial L/\partial w_i = \delta_1 x_i\) (here \(x_2 < 0\)).
Switching any toggle resets training, so curves from different configurations are never mixed.
Both neurons use the logistic (sigmoid) activation:
The derivative identity follows from the quotient rule and is what makes the arithmetic tidy: once a neuron's output \(a=\sigma(z)\) is known from the forward pass, its local slope is just \(a(1-a)\) — no new computation needed. Each weight \(w_j\) multiplies the signal on one edge; biases are omitted to keep the chain rule bare.
Each neuron computes a weighted sum \(z\) (its pre-activation), then squashes it: \(a = \sigma(z)\).
The whole network is one nested function, e.g. \( \hat{y} = \sigma\!\big(w_2\,\sigma(w_1 x)\big) \) — a composition. That nesting is exactly why differentiation will call for the chain rule.
For one observation with target \(y\), we score the prediction with squared error:
The \(\tfrac12\) is cosmetic — it cancels the 2 from the power rule so the derivative is simply the residual \(\hat y - y\). Familiar territory: this is the same residual that drives least squares.
Output weight. \(L\) depends on \(w_2\) only through the chain \(w_2 \to z_2 \to \hat y \to L\), so multiply the three local derivatives:
Define the error signal (delta) at the output neuron as everything up to, but not including, the last factor:
Hidden weight. Going one layer deeper just extends the same chain: \(w_1 \to z_1 \to a_1 \to z_2 \to \hat y \to L\). The front of that chain is \(\delta_2\) again — we reuse it instead of recomputing:
Read \(\delta_1\) as: the error at the output (\(\delta_2\)), carried backward across the connection (\(\times\,w_2\)), and attenuated by the neuron's local slope (\(\times\,\sigma'(z_1)\)). This recursion — each layer's delta is built from the next layer's delta — is the entire backpropagation algorithm.
Two inputs. With \(z_1 = w_1 x_1 + w_2 x_2\), the single \(\delta_1\) fans out to every weight feeding the neuron:
The general pattern: gradient of a weight = (delta of the neuron it feeds) × (signal on its input side). One backward sweep prices every weight in the network — that is why training a million-parameter model costs roughly one extra forward pass, not a million.
Each weight steps downhill, against its gradient, scaled by the learning rate \(\eta\):
Too small an \(\eta\) → slow crawl; too large → the update overshoots the minimum and the loss can oscillate or diverge. (In the sim: run 30 epochs at \(\eta=2\), reset, repeat at \(\eta=5\).)
With a training set \(\{(x^{(i)},y^{(i)})\}_{i=1}^{n}\), the objective is the mean loss \( \bar L = \tfrac1n \sum_i L^{(i)} \), whose gradient is the mean of the per-sample gradients. The two schemes differ only in when the update happens:
A statistical framing: the batch gradient is the sample mean of the per-observation gradients — lower variance, higher cost per step. Mini-batch SGD (average over 32–256 samples) is the practical compromise; the sim's batch mode is a mini-batch of size 2 that happens to be the full data.
Data \(x=1.5,\; y=0.8\); initial weights \(w_1=0.8,\; w_2=-0.5\); learning rate \(\eta=2\).
| STEP | FORMULA | VALUE |
|---|---|---|
| \(z_1\) | \(w_1 x = 0.8 \times 1.5\) | 1.200 |
| \(a_1\) | \(\sigma(1.2)\) | 0.769 |
| \(z_2\) | \(w_2 a_1 = -0.5 \times 0.769\) | −0.384 |
| \(\hat y\) | \(\sigma(-0.384)\) | 0.405 |
| \(L\) | \(\tfrac12(0.405-0.8)^2\) | 0.078 |
| \(\delta_2\) | \((\hat y - y)\,\hat y(1-\hat y) = (-0.395)(0.405)(0.595)\) | −0.095 |
| \(\partial L/\partial w_2\) | \(\delta_2\, a_1 = (-0.095)(0.769)\) | −0.073 |
| \(\delta_1\) | \(\delta_2\, w_2\, a_1(1-a_1) = (-0.095)(-0.5)(0.178)\) | 0.008 |
| \(\partial L/\partial w_1\) | \(\delta_1\, x = (0.008)(1.5)\) | 0.013 |
| update | \(w_1 \leftarrow 0.8 - 2(0.013);\;\; w_2 \leftarrow -0.5 - 2(-0.073)\) | w₁=0.775, w₂=−0.354 |
Note the signs: \(\hat y\) is below the target, and increasing \(w_2\) raises \(\hat y\) (since \(a_1>0\)), so its gradient is negative and the update pushes \(w_2\) up. Every number here matches the sim's trace panel — step through epoch 1 and verify.
backprop.sim // Developed by Armando Teixeira-Pinto and Claude Fable 5