Loss landscape

Each spot on this paper map is one setting of a tiny network's two weights, and its height is the loss: how wrong the network is with those weights. Release the marbles to watch four real optimisers walk downhill, drag to raise hills or dig valleys, and push the learning rate until they bounce out.

learning rate η 0.050mini-batch noise none (full batch)landscapelow losshigh loss

step 0loss –w ← w − η·∇L
step 0loss –v ← 0.9v + ∇L; w ← w − η·v
step 0loss –s ← 0.9s + 0.1(∇L)²; w ← w − η·∇L/√s
step 0loss –m, v ← running mean of ∇L, (∇L)²; w ← w − η·m̂/√v̂

Training a neural network means finding the lowest point on a landscape like this one, but with billions of directions instead of two. Every model you've used was found by marbles like these.

Started from a hand-cut layered paper terrain.