Lecture 00: How Machines Learn

The idea in one sentence

A machine learns when we stop writing down the answer and start writing down a way to search for it, and every step of that search is arithmetic on lists of numbers.

Every program is a promise kept by the layer below

Here is a line that runs a few trillion times a day, inside every model you have ever used:

y = Wx + b .

That line says what comes out. How it comes out is the business of a tower of floors underneath it, each one keeping a promise for the floor above.

Fig. 1

The stack under one line of code. Python promises that a name holds a number. The instruction set promises that an ADD instruction adds. The transistor promises that a voltage above a threshold counts as a 1.

Notice what the whole tower has in common. Nothing on it is approximate. A transistor either conducts or it does not. An ADD either adds or the chip is broken. Every floor does exactly the same thing every time you ask. So if every layer is exact, where does learning come from?

A machine that only knows where it is

The answer needs a running start. Begin at the bottom, with the simplest thing that deserves to be called a machine. Give it a handful of states and a rule for moving between them, and give it no other memory at all. The only thing it knows about its whole past is which state it is in now. That is a finite state machine, and it is the honest floor plan of a traffic light and of the parser reading this page.

Here is one with four states. The machine is a cat named Gauss.

Try it. Four states, and no other memory

state = Asleep

Asleep

The orange ring is where Gauss is standing; the blue arrows are the four exits from there. Find two different histories that end in the same state, then ask what the machine could possibly know about the difference.

Click around and watch what the machine cannot do. It never counts. Make a noise nine times from Asleep and Gauss arrives in exactly the same state as one noise leaves him in. Four states means four distinguishable pasts, and that is not enough to tell nine from one.

To count, you need somewhere to write.

Turing’s tape

In 1936 Alan Turing added the missing piece. Take a finite state machine and hand it an endless strip of paper divided into cells. The machine reads the cell under its head, writes a symbol back, moves one cell left or right, and changes state. The rules fit in a small table.

Here is a complete machine. It adds one to a binary number, and the head starts on the rightmost digit.

Fig. 2

One step of the increment machine. Blue is wherever the head is standing, green is the cell whose contents just changed. The head sits on a 1, so rule 1 fires: write a 0, move one cell left. The dashed cells are the rest of the tape, which runs on as far as the machine needs.

Run it on 1011, which is eleven. Rule 1 fires, giving 1010. Rule 1 fires again, giving 1000. Now the head sits on a 0, so rule 2 fires: write a 1 and stop. The tape reads 1100, which is twelve. Three rules and three steps, and the machine has done arithmetic without knowing what a number is.

That tiny table is the whole of computing. Anything your laptop can compute, this machine can compute, given enough tape and enough patience. Turing proved that, and in the same paper he proved the other half: no program can read an arbitrary program and always say whether it finishes. Some questions have no recipe at all, and we know that for certain.

Look at what everything so far has in common. Each machine is exact, and each machine was told what to do. Nobody searched for anything.

Where the recipes run out

Solve this:

5x^3 - 7x^2 - 40x + 100 = 0 .

There is a closed formula for cubics. Almost nobody uses it, and for degree five and above no formula in radicals exists at all. So how does a computer answer? It guesses, then improves the guess.

Newton’s move is the one every good approximation makes: a curve is hard, a straight line is easy. Stand at your current guess $x_k$ . Replace the curve by its tangent there, and solve the tangent for zero instead. The tangent through $(x_k, f(x_k))$ with slope $f'(x_k)$ meets the horizontal axis at

x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} .

Start at $x_0 = -4$ , where $f(-4) = -172$ and $f'(-4) = 256$ . One step lands at $-4 + 172/256 = -3.328125$ .

Fig. 3

Two Newton steps. Orange is a guess, blue is the step the tangent takes, green is the root. The tangent at $x_0$ meets the axis at $x_1$ , the tangent at $x_1$ meets it at $x_2$ , and by then $x_2$ and the true root are a pixel apart.

Keep going and watch the error, the distance from the current guess to the true root $-3.1517194252$ :

0.848, \quad 0.176, \quad 0.0101, \quad 0.0000359, \quad 0.00000000046 .

Each error is roughly the square of the one before it, so the count of correct digits doubles every step. Four steps take a guess with no correct digits to an answer good to nine.

Try it. Where you start decides what you find

start x₀ = -4.0

after 0 steps
x = -4.000000
f(x) = -172.0000

Orange is where you stand, blue is the tangent, green is where that tangent lands. Start at −5 and it locks on in four steps. Start at 4 and watch it wander: two steps in, the guess reaches a stretch where the tangent is nearly flat, and a flat line meets the axis a long way off.

Move the starting point and the honest half shows up. From $x_0 = 4$ the guess bounces around for a dozen steps with no sign of settling. The reason is visible in the formula: when $f'(x_k)$ is near zero the tangent is nearly flat, and a nearly flat line crosses the axis a very long way from where you stand. Newton’s method is fast when it works, and it does not always work.

python

import numpy as np

f  = lambda x: 5 * x**3 - 7 * x**2 - 40 * x + 100
fp = lambda x: 15 * x**2 - 14 * x - 40

x = np.float64(-4.0)
for k in range(5):
    print(k, x, f(x))
    x = x - f(x) / fp(x)

# 0 -4.0                 -172.0
# 1 -3.328125            -28.728397369384766
# 2 -3.1618149444627393  -1.551404230694402
# 3 -3.151755360837241   -0.005502685162468879
# 4 -3.1517194256388494  -7.008890179349692e-08

python

import torch

f  = lambda x: 5 * x**3 - 7 * x**2 - 40 * x + 100
fp = lambda x: 15 * x**2 - 14 * x - 40

x = torch.tensor(-4.0, dtype=torch.float64)
for k in range(5):
    print(k, x.item(), f(x).item())
    x = x - f(x) / fp(x)

# 0 -4.0                 -172.0
# 1 -3.328125            -28.728397369384766
# 2 -3.1618149444627393  -1.551404230694402
# 3 -3.151755360837241   -0.005502685162468879
# 4 -3.1517194256388494  -7.008890179349692e-08

python

import jax
import jax.numpy as jnp
jax.config.update('jax_enable_x64', True)

f  = lambda x: 5 * x**3 - 7 * x**2 - 40 * x + 100
fp = lambda x: 15 * x**2 - 14 * x - 40

x = jnp.float64(-4.0)
for k in range(5):
    print(k, x, f(x))
    x = x - f(x) / fp(x)

# 0 -4.0                 -172.0
# 1 -3.328125            -28.728397369384766
# 2 -3.1618149444627393  -1.551404230694402
# 3 -3.151755360837241   -0.005502685162468879
# 4 -3.1517194256388494  -7.008890179349692e-08

python

import tensorflow as tf

f  = lambda x: 5 * x**3 - 7 * x**2 - 40 * x + 100
fp = lambda x: 15 * x**2 - 14 * x - 40

x = tf.constant(-4.0, dtype=tf.float64)
for k in range(5):
    print(k, x.numpy(), f(x).numpy())
    x = x - f(x) / fp(x)

# 0 -4.0                 -172.0
# 1 -3.328125            -28.728397369384766
# 2 -3.1618149444627393  -1.551404230694402
# 3 -3.151755360837241   -0.005502685162468879
# 4 -3.1517194256388494  -7.008890179349692e-08

Downhill instead of across

Newton hunts for a place where a function is zero. Training hunts for a place where a function is smallest, because that function is the model’s error and small error is the point. The target changes and the habit stays: stand somewhere, look at the slope, take a step.

Take $g(x) = x^2 - 4x + 7$ , whose lowest point sits at $x = 2$ with $g(2) = 3$ . Its slope is $g'(x) = 2x - 4$ , positive to the right of 2 and negative to the left. So walking against the slope always walks toward the bottom:

x_{k+1} = x_k - \eta \, g'(x_k) .

The number $\eta$ is the step size, and it decides everything. Subtract 2 from both sides of the update and the pattern falls out:

x_{k+1} - 2 = (1 - 2\eta)(x_k - 2) .

The distance to the bottom gets multiplied by $1 - 2\eta$ at every step. Now each case is a number you can check. With $\eta = 0.4$ the multiplier is $0.2$ and the distance shrinks fast. With $\eta = 0.5$ the multiplier is $0$ , so one step lands exactly on the answer. With $\eta = 1.1$ the multiplier is $-1.2$ , so each step overshoots further than the last and the walk leaves for good.

Fig. 4

The same valley walked with two step sizes, both starting at $x = 6$ . Orange is a guess, green is the bottom of the valley, blue is the step. On the left the walk reaches $2.8$ , then $2.16$ , then $2.032$ . On the right it reaches $-2.8$ , then $7.76$ , then $-4.912$ , climbing higher on the wall each time.

python

import numpy as np

gp = lambda x: 2 * x - 4          # slope of x^2 - 4x + 7

for eta in (0.4, 0.5, 1.1):
    x, walk = np.float64(6.0), []
    for _ in range(4):
        walk.append(round(float(x), 6))
        x = x - eta * gp(x)
    print(eta, walk)

# 0.4 [6.0, 2.8, 2.16, 2.032]
# 0.5 [6.0, 2.0, 2.0, 2.0]
# 1.1 [6.0, -2.8, 7.76, -4.912]

python

import torch

gp = lambda x: 2 * x - 4          # slope of x^2 - 4x + 7

for eta in (0.4, 0.5, 1.1):
    x, walk = torch.tensor(6.0, dtype=torch.float64), []
    for _ in range(4):
        walk.append(round(x.item(), 6))
        x = x - eta * gp(x)
    print(eta, walk)

# 0.4 [6.0, 2.8, 2.16, 2.032]
# 0.5 [6.0, 2.0, 2.0, 2.0]
# 1.1 [6.0, -2.8, 7.76, -4.912]

python

import jax
import jax.numpy as jnp
jax.config.update('jax_enable_x64', True)

gp = lambda x: 2 * x - 4          # slope of x^2 - 4x + 7

for eta in (0.4, 0.5, 1.1):
    x, walk = jnp.float64(6.0), []
    for _ in range(4):
        walk.append(round(float(x), 6))
        x = x - eta * gp(x)
    print(eta, walk)

# 0.4 [6.0, 2.8, 2.16, 2.032]
# 0.5 [6.0, 2.0, 2.0, 2.0]
# 1.1 [6.0, -2.8, 7.76, -4.912]

python

import tensorflow as tf

gp = lambda x: 2 * x - 4          # slope of x^2 - 4x + 7

for eta in (0.4, 0.5, 1.1):
    x, walk = tf.constant(6.0, dtype=tf.float64), []
    for _ in range(4):
        walk.append(round(float(x.numpy()), 6))
        x = x - eta * gp(x)
    print(eta, walk)

# 0.4 [6.0, 2.8, 2.16, 2.032]
# 0.5 [6.0, 2.0, 2.0, 2.0]
# 1.1 [6.0, -2.8, 7.76, -4.912]

Now scale it up. A trained model does not have one number to tune, it has billions. The slope $g'$ becomes a list of billions of slopes, one for each weight, and the step becomes a subtraction of two very long lists. The multiplier $1 - 2\eta$ becomes a matrix, and the numbers that decide whether training converges or explodes are that matrix’s eigenvalues. That is Lecture 15, and it is the same picture you just drew with one variable.

Giving a machine a memory

The line at the top of this lecture, $y = Wx + b$ , is one layer. Squash the output through a bent curve so the layer can do something other than stretch, stack a few of these, and you have the network of the 1980s. It has no memory. Feed it a word and it has already forgotten the previous word.

The first repair was a loop. Keep a running vector $h$ , and at every step mix the new input into it, so that $h_t$ comes from $h_{t-1}$ and the new input through the same weights every time. Now the machine carries its past. The trouble is that it carries the past in one fixed-size vector, overwritten at every step, so what happened forty words ago is a faint smear.

The second repair, in 1997, was gates. Learn a number between 0 and 1 for each slot of $h$ saying how much of the old value to keep, and another saying how much of the new value to let in. Multiply by those numbers instead of overwriting. Memory now survives long gaps, and this was the state of the art for two decades.

The third repair threw out the compression. Do not squeeze the past into one vector at all. Keep every past vector, and when you need the past, take a weighted average of all of them.

Fig. 5

Four answers to one question. Blue is the path the past takes to reach the present, and the thickness of the three blue arrows on the right is the weight each piece of the past is given. The yellow ring is the gate, the only part of the picture that learns what to keep.

Here is that last idea with numbers on it. Three earlier positions each hand in two vectors: a key $k_i$ , which advertises what that position is about, and a value $v_i$ , which is what it will contribute if chosen. The present position hands in a query $q$ , saying what it is looking for. A dot product scores each key against the query:

q = (1, 2), \quad k_1 = (1, 1), \quad k_2 = (1, 0), \quad k_3 = (2, -1) \;\Longrightarrow\; \text{scores } 3, \; 1, \; 0 .

Scores are not weights yet, since they can be negative and they do not add to one. Exponentiate and divide by the total, which is the softmax:

\frac{(e^3,\; e^1,\; e^0)}{e^3 + e^1 + e^0} = \frac{(20.0855,\; 2.7183,\; 1)}{23.8038} = (0.8438,\; 0.1142,\; 0.0420) .

Now spend those weights on the values $v_1 = (4, 1)$ , $v_2 = (0, 4)$ , $v_3 = (2, 0)$ :

0.8438\,v_1 + 0.1142\,v_2 + 0.0420\,v_3 = (3.4592,\; 1.3006) .

Fig. 6

Left: bar length is the weight, and one score is much larger than the others, so the softmax turns that gap into bars that are far apart. Right: the blue dot is the mixed result. It is a blend of the three value arrows, so it lands inside the dashed triangle, pulled toward $v_1$ .

That output is a linear combination of the value vectors, with weights chosen by dot products. Two operations, and both of them are the subject of Lecture 1. Every attention head in every model working today is doing this, a few billion times a second, on vectors with hundreds of entries instead of two.

Where this is going

Go back to the tower in Figure 1. A new floor has been quietly inserted near the top, above the algorithm and below the application, and that floor is made of numbers nobody wrote down. They arrived by search, and the search was a long walk down a slope.

Fig. 7

The same tower, one floor taller. Every other floor was written by a person. The blue one was searched for, and everything it does is a matrix meeting a vector.

The weights are matrices, the inputs are vectors, the forward pass is a chain of multiplications, and the training step is a subtraction. The questions that decide whether any of it works are questions about matrices: which directions does this one flatten, and can it be undone. That is what the rest of this course is about, and it starts with the simplest object in it, an arrow.

Check yourself

Do one Newton step by hand from $x_0 = -4$ . Compute $f(-4)$ and $f'(-4)$ , then the quotient, and check that you land on $-3.328125$ .
Run the increment machine on the tape 111. Which rules fire, in what order, and what does the tape read at the end?
For $g(x) = x^2 - 4x + 7$ , show that $x_{k+1} - 2 = (1 - 2\eta)(x_k - 2)$ . Which value of $\eta$ leaves the walk bouncing forever at a distance that never shrinks?
Starting from Asleep, find a two-click sequence for Gauss that ends in Smug. Is there an input that leaves the state unchanged no matter where the machine starts?
In the attention example, change the query to $q = (1, -1)$ and recompute the three scores. Which value vector now takes most of the weight?
Redraw Figure 3 from memory, labels included. The drawing is the test.