This is the honest sequel to the language model. That one was the smallest thing that could be called a model of language - an n-gram chain that I was careful not to oversell. This is the smallest thing that is unambiguously a neural network: a 784-32-10 multilayer perceptron that reads handwritten digits and, more to the point, runs entirely in integer arithmetic on the 604e. Digits are a classic first task, and the interesting part is not the task but whether a network can be made to live inside this machine's strengths - 16-bit integers and its 1 MiB L2.

It gets 96.1% of the MNIST test set right - 961 of the first 1000 - at about 600 inferences a second. Each one is 25,408 integer multiply-accumulates, which is what a 250 MHz scalar core can do in about 1.6 milliseconds. The same C code runs at about 10,000 inferences a second off the machine - and the machine it runs on is not a modern one either: kootenai, the Linux machine I build and test on, is a Wyse Zx0Q thin client built around a 2014 AMD Jaguar chip. One core of that decade-old thin client is still only about seventeen times faster than this PowerPC.

Training. This part happens off the machine, and it is worth saying plainly what it is, because most of us have never done it and never had a reason to. The whole network is 25,450 numbers - a 784-by-32 grid of weights, 32 biases, a 32-by-10 grid, and 10 more biases - and it starts as random noise. Training is nothing more than nudging those numbers until the noise recognises digits.

For each image you push 784 pixel values forward through the network, add a bias, keep the positives (that is the "rectifier"), and get ten scores, one per digit. You compare that with the right answer and collapse the difference into a single number that measures how wrong the network was. Then the chain rule of calculus gives you, for every one of the 25,450 numbers, the direction that would have made that error a little smaller. That is all backpropagation is - derivative bookkeeping, not insight. Adam takes those directions and steps downhill, choosing a slightly different step size for every parameter so that rarely-moved weights can move faster. Repeat over all 60,000 training images, 128 at a time, fifteen times through - about 7,000 steps - and the noise becomes a recogniser.

It is not magic, and it is not really ordinary either. It is on the order of 900,000 forward runs and as many backward ones, each touching every weight: about 70 billion multiply-accumulates, a couple of minutes of CPU on the training machine - which is itself an old thin client, not a modern one. Accuracy climbs from 85.8% after the first pass over the data to 97.0% after the fifteenth, each pass buying less than the last. On the 604e the same training would take an hour or two, because learning costs about three times as much as using, per image. That is the whole reason for the split below, and the reason the modern world trains on big machines and runs on small ones - doing it on 1997 hardware just makes the split visible.

Running it, in integers. The machine never sees a float. Each layer's weights are squeezed to 16-bit integers with a per-layer power-of-two scale, which turns every rescale into a shift; the biases are folded in as integers; the hidden layer accumulates in 64 bits, is rectified, and is requantised with a single arithmetic right shift; and the output is just the argmax of the integer accumulators, since scaling a logit does not change which one is largest. The 1/255 input scaling is folded into the first layer, so the 604e feeds raw pixel bytes straight in.

On the graphics processor. A note on the two machines, because "the host" is ambiguous here: obsidian is the subject of this site, and kootenai is the Linux machine I build, train and test on - a workbench and a bridge to obsidian, not the focus. kootenai has a Radeon (a Kabini HD 8400E, a GCN-era integrated GPU), and none of this used it. A GPU is the wrong tool for a network this small: the whole computation is under a tenth of a millisecond, comparable to the cost of just launching a kernel, and every weight fits in cache. On this machine it is not even reachable - the radeon driver is loaded, but there is no working OpenCL at all (zero platforms), so there is nothing to talk to. A GPU pays off for large, batched, floating-point work; a 25,450-parameter MLP and a per-node chess evaluation are neither. Making a GPU path run on hardware this old would be a project worth doing for its own sake, and in the spirit of this machine - but it is a different project.

A caveat, from a later probe: integer is not automatically the fast path on this machine. Timing the same network in plain float32, the 604e turns out to be faster in floating point than in the integer version above - about 1.7x, compiled with XL C - because the integers accumulate in 64 bits (which this 32-bit chip does in software) while the FPU is hardware and XL C drives it well. With GCC the float version is the slowest of the four. So the usual "float is expensive, integer is cheap" inverts here. The integer model is still the one that ports cleanly and the one this page is built around, but the choice was made on principle, not on measurement - and the measurement disagrees.

The scheme was simulated in numpy before any C was written, and the three results agree (96.25 / 96.25 / 96.10). The native output is identical under XL C and GCC and identical on kootenai too - across a big-endian 604e and a little-endian x86 - which is the evidence that the integer path is correct. The montage above is rendered on the machine too, with the same GIF encoder as the fractals.