A network that learns XOR
A small neural network in your browser. Press start and watch it adjust its weights with backpropagation until it can do XOR.
Step 0, loss 0.2575.
The four examples
| x1 | x2 | Target | Output | Show |
|---|---|---|---|---|
| 0 | 0 | 0 | 0.44 correct | |
| 0 | 1 | 1 | 0.38 still wrong | |
| 1 | 0 | 1 | 0.45 still wrong | |
| 1 | 1 | 0 | 0.39 correct |
What the network thinks
The loss per step
What XOR is
XOR is a rule with two inputs. The result is 1 when exactly one of the two is 1.
| x1 | x2 | XOR |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Why one neuron cannot do it
One neuron computes a weighted sum, z = x1·w1 + x2·w2 + b, and passes it through a sigmoid. All inputs for which z is exactly 0 lie on a straight line. On one side of that line the output is above 0.5, on the other side below.
In XOR the two ones sit diagonally opposite each other, at (0, 1) and (1, 0). No straight line separates them from (0, 0) and (1, 1). That is why there is a hidden layer in between. Each hidden neuron draws its own line, and the output neuron combines those lines. You can see it in the map above: the area becomes a band instead of a half.
How it learns
Every step goes like this:
- Forward. For each of the four examples: weighted sum and sigmoid in the hidden layer, then the same in the output neuron.
- The loss. Per example
(output - target)², averaged over the four. That is the number in the chart. - Backward. With the chain rule, each weight gets a number for how the loss changes when that weight grows a tiny bit. Through a sigmoid that is always
output · (1 - output). - Adjust. Every weight moves a small step against that direction:
weight = weight - learning rate · gradient.
Use "One step" to watch a single round. With 2 hidden neurons the network sometimes gets stuck, depending on the start weights. Press "Reset" then.