Lecture 7.5: The Null Space

The idea in one sentence

The vectors a matrix sends to zero form a flat space through the origin, and one pass of elimination hands you a basis for it.

Two receipts

A shop sells two items. The first costs one dollar, the second costs two. A basket is a pair of counts $x = (x_1, x_2)$ , and returns are allowed, so a count is any real number. The till computes

Ax = \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = x_1 + 2x_2 .

Two questions, same $A$ , different targets. Which baskets total six dollars? Which baskets total nothing? Both answers are infinite sets of points, and both are lines in the plane. They look like twins.

They are not. One of those lines is a subspace and the other is not, and everything this course does with solutions rests on that difference. Lecture 7 gave you the test: a subspace holds the zero vector, and it survives scaling and addition. Point the test at both lines.

Add two answers together

Two baskets that total six: $u = (6, 0)$ and $v = (0, 3)$ . Check them against the till. Six items of the first kind cost $6 + 0 = 6$ . Three of the second cost $0 + 6 = 6$ .

Now put both baskets into one cart. The counts add, so the combined basket is $u + v = (6, 3)$ , and the till says $6 + 6 = 12$ . Twelve dollars. The sum of two six-dollar baskets is not a six-dollar basket. Doubling fails for the same reason: $2u = (12, 0)$ costs twelve.

Two baskets that total nothing: $p = (2, -1)$ , which buys two of the first item and returns one of the second, and $q = (-6, 3)$ , which returns six and buys three. The till says $2 - 2 = 0$ and $-6 + 6 = 0$ . Add them: $p + q = (-4, 2)$ , and the till says $-4 + 4 = 0$ . Still nothing. Scale $p$ by any number you like and the total stays at zero, because scaling the basket scales the receipt.

Fig. 1

Left: the baskets costing nothing. The green line runs through the origin, and the sum of two of its points is another of its points. Right: the baskets costing six. The orange line misses the origin, and adding two of its points lands you on the twelve-dollar line above it. The blue arrow is the addition in both panels.

The picture says it faster than the algebra. Adding two arrows on the right pushes you outward, away from the line you started on. On the left the line already runs through the origin, so pushing outward along it lands you back on it. That is the whole difference.

The set that misses the origin

The zero-dollar line contains the origin: the empty basket costs nothing. The six-dollar line does not, since the empty basket does not cost six dollars. That alone disqualifies it, because every subspace contains the zero vector.

A flat set pushed off the origin like this is called an affine set. It can have any dimension, and it is perfectly straight. Neither of those saves it. An affine set is a subspace only when the push is zero. The six-dollar line is the zero-dollar line shifted by any single six-dollar basket. Shift by $u = (6,0)$ or by $v = (0,3)$ and you land on the same line.

Why Ax = 0 always gives a subspace

Nothing above depended on the numbers 1 and 2. Take any matrix $A$ , and let $x$ and $y$ be two vectors it sends to zero. Then

A(x + y) = Ax + Ay = 0 + 0 = 0, \qquad A(cx) = c\,(Ax) = c\,0 = 0 .

Both lines use only the property that $Ax$ distributes over sums and scalars. And $A0 = 0$ for every matrix, so the zero vector is always in. The closure test passes, for every $A$ ever written down.

This set has a name. For an $m$ by $n$ matrix $A$ , the null space $N(A)$ is the set of all $x$ with $Ax = 0$ . It is also called the kernel. Since $x$ has $n$ entries, $N(A)$ lives inside $\R^n$ , on the input side. The column space $C(A)$ from Lecture 7 lives inside $\R^m$ , on the output side. Two spaces in two different rooms, and mixing up which room is which is the most common early mistake in this subject.

The solution set of $Ax = b$ for a nonzero $b$ gets no such name, because it is not a subspace. It is an affine set, and Lecture 8 shows exactly which shift produces it.

Which columns carry new information

Time to compute a null space instead of guessing one. Start from a small matrix and write it twice, with the second copy doubled:

A = \begin{bmatrix} 1 & 2 & 2 & 4 \\ 3 & 8 & 6 & 16 \end{bmatrix} .

Call this the doubled-columns matrix. Column 3 is twice column 1, and column 4 is twice column 2. So only two of the four columns carry new information.

That redundancy is already a null vector in disguise. Column 3 minus twice column 1 is the zero vector. Write down the recipe for that cancellation, one coefficient per column, and what you have written down is a solution of $Ax = 0$ .

Fig. 2

The two tinted columns are the repeats, and the blue arcs say what they repeat. Every redundant column is a null vector waiting to be written down: read the cancellation as a list of coefficients and you have a solution of $Ax = 0$ before any elimination happens.

Check it by hand: $-2(1,3) + (2,6) = (0,0)$ . The other pair cancels the same way. What follows is the machine that finds both vectors without needing you to spot anything.

Run elimination. Subtract 3 times row 1 from row 2 to reach the echelon form $U$ . Then divide row 2 by its pivot and clear the entry above it to reach the reduced row echelon form $R$ .

Fig. 3

One elimination step, then one cleanup step. Yellow rings the pivots and cyan tints the free columns, a pairing that holds for the rest of the chapter. Columns 1 and 2 hold a pivot; columns 3 and 4 do not, and their variables are free to be anything.

A column of $R$ that holds a pivot is a pivot column, and its variable is determined. A column with no pivot is a free column, and its variable can be set to any number you want. The count of pivots is the rank $r$ , so the count of free columns is $n - r$ . Here $r = 2$ and $n = 4$ , so two of the four columns are free, and the null space has dimension two for exactly that reason.

Elimination did not disturb the answer. Every row step is multiplication on the left by an invertible matrix $E$ . If $Ax = 0$ then $EAx = 0$ , and if $EAx = 0$ then multiplying by $E^{-1}$ brings back $Ax = 0$ . So

N(A) = N(U) = N(R) .

Fig. 4

Three different matrices, one shared null space. Because every step between them can be undone, no vector enters or leaves the solution set along the way, which is why it is safe to hunt for $N(A)$ in the tidiest of the three.

The block machine

Suppose the pivot columns come first, so $R$ splits into an identity block beside a leftover block:

R = \begin{bmatrix} I & F \end{bmatrix},

where $I$ is $r$ by $r$ and $F$ is $r$ by $n - r$ . Split the unknown the same way. Write $y$ for the pivot entries of $x$ and $z$ for the free entries. Then $Rx = 0$ reads

I y + F z = 0 \qquad\Longrightarrow\qquad y = -F z .

That is the whole algorithm. Pick the free entries, and the pivot entries follow. The cleanest picks are the columns of the identity, one free variable switched on at a time. Stack those choices side by side and the answers stack with them:

N = \begin{bmatrix} -F \\ I \end{bmatrix}, \qquad R N = \begin{bmatrix} I & F \end{bmatrix} \begin{bmatrix} -F \\ I \end{bmatrix} = -F + F = 0 .

Fig. 5

The block machine, drawn on the doubled-columns example. $R$ is the identity beside $F$ , yellow half and cyan half, and $N$ is $-F$ stacked on the identity. Multiply and the two halves cancel exactly.

For the doubled-columns matrix the leftover block is $F = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$ , so

N = \begin{bmatrix} -2 & 0 \\ 0 & -2 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}, \qquad s_1 = (-2, 0, 1, 0), \qquad s_2 = (0, -2, 0, 1).

These are the two vectors you guessed from looking at the columns. The columns of $N$ are called the special solutions, one per free column, and every vector in $N(A)$ is a combination of them. That is what makes them a basis, and it is why $\dim N(A) = n - r$ .

Fig. 6

One free variable at a time. Set the free entries inside the dashed box, and $y = -Fz$ fills in the pivot entries above. Two free columns give two special solutions.

python

import numpy as np

A = np.array([[1., 2., 2., 4.], [3., 8., 6., 16.]])
U = A.copy(); U[1] -= 3 * U[0]                  # one elimination step
R = U.copy(); R[1] /= 2; R[0] -= 2 * R[1]       # clear above the second pivot
print(R)                     # [[1. 0. 2. 0.]
                             #  [0. 1. 0. 2.]]

F = R[:, 2:]                                    # the free columns of R
N = np.vstack([-F, np.eye(2)]) + 0.0            # the block machine
print(N.T)                   # [[-2.  0.  1.  0.]   <- s1
                             #  [ 0. -2.  0.  1.]]  <- s2
print(A @ N)                 # [[0. 0.]
                             #  [0. 0.]]

# The computer's own answer, from the SVD, spans the same sheet.
_, s, Vt = np.linalg.svd(A)
print(s)                     # [19.7419204   0.50653633]  two nonzero: rank 2
ns = Vt[2:].T                # the last n - r right singular vectors
print(np.linalg.norm(A @ ns))                 # 2.06e-15
P = ns @ ns.T                                 # project s1 onto that sheet
print(np.linalg.norm(P @ N[:, 0] - N[:, 0]))  # 3.69e-15: it does not move

python

import torch

A = torch.tensor([[1., 2., 2., 4.], [3., 8., 6., 16.]])
U = A.clone(); U[1] -= 3 * U[0]                 # one elimination step
R = U.clone(); R[1] /= 2; R[0] -= 2 * R[1]      # clear above the second pivot
print(R)                     # tensor([[1., 0., 2., 0.],
                             #         [0., 1., 0., 2.]])

F = R[:, 2:]                                    # the free columns of R
N = torch.cat([-F, torch.eye(2)]) + 0.0         # the block machine
print(N.T)                   # tensor([[-2.,  0.,  1.,  0.],   <- s1
                             #         [ 0., -2.,  0.,  1.]])  <- s2
print(A @ N)                 # tensor([[0., 0.],
                             #         [0., 0.]])

# The computer's own answer, from the SVD, spans the same sheet.
s = torch.linalg.svdvals(A)
print(s)                     # tensor([19.7419,  0.5065])  two nonzero: rank 2
Vt = torch.linalg.svd(A, full_matrices=True).Vh
ns = Vt[2:].T                # the last n - r right singular vectors
print(torch.linalg.norm(A @ ns))                 # ~1e-6 in float32
P = ns @ ns.T                                    # project s1 onto that sheet
print(torch.linalg.norm(P @ N[:, 0] - N[:, 0]))  # ~1e-6: it does not move

python

import jax.numpy as jnp

A = jnp.array([[1., 2., 2., 4.], [3., 8., 6., 16.]])
U = A.at[1].add(-3 * A[0])                      # one elimination step
R = U.at[1].divide(2)
R = R.at[0].add(-2 * R[1])                      # clear above the second pivot
print(R)                     # [[1. 0. 2. 0.]
                             #  [0. 1. 0. 2.]]

F = R[:, 2:]                                    # the free columns of R
N = jnp.vstack([-F, jnp.eye(2)]) + 0.0          # the block machine
print(N.T)                   # [[-2.  0.  1.  0.]   <- s1
                             #  [ 0. -2.  0.  1.]]  <- s2
print(A @ N)                 # [[0. 0.]
                             #  [0. 0.]]

# The computer's own answer, from the SVD, spans the same sheet.
_, s, Vt = jnp.linalg.svd(A)
print(s)                     # [19.7419     0.50653636]  two nonzero: rank 2
ns = Vt[2:].T                # the last n - r right singular vectors
print(jnp.linalg.norm(A @ ns))                 # ~1e-6 in float32
P = ns @ ns.T                                  # project s1 onto that sheet
print(jnp.linalg.norm(P @ N[:, 0] - N[:, 0]))  # ~1e-6: it does not move

python

import tensorflow as tf

A = tf.constant([[1., 2., 2., 4.], [3., 8., 6., 16.]])
r1 = A[1] - 3 * A[0]                            # one elimination step
r1 = r1 / 2
r0 = A[0] - 2 * r1                              # clear above the second pivot
R = tf.stack([r0, r1])
print(R)                     # [[1. 0. 2. 0.]
                             #  [0. 1. 0. 2.]]

F = R[:, 2:]                                    # the free columns of R
N = tf.concat([-F, tf.eye(2)], axis=0) + 0.0    # the block machine
print(tf.transpose(N))       # [[-2.  0.  1.  0.]   <- s1
                             #  [ 0. -2.  0.  1.]]  <- s2
print(A @ N)                 # [[0. 0.]
                             #  [0. 0.]]

# The computer's own answer, from the SVD, spans the same sheet.
s, _, v = tf.linalg.svd(A, full_matrices=True)
print(s)                     # [19.7419     0.50653636]  two nonzero: rank 2
ns = v[:, 2:]                # the last n - r right singular vectors
print(tf.norm(A @ ns))                       # ~1e-6 in float32
P = ns @ tf.transpose(ns)                    # project s1 onto that sheet
print(tf.norm(P @ N[:, :1] - N[:, :1]))      # ~1e-6: it does not move

The last three lines are the interesting part. The SVD returns a completely different-looking basis for the null space, full of numbers like $0.4$ and $-0.8$ . Project $s_1$ onto the sheet that basis spans and $s_1$ does not move, which means both bases describe the same flat sheet. A subspace does not care which basis you hand it.

When the pivots are scattered

The tidy split $R = \begin{bmatrix} I & F \end{bmatrix}$ assumed the pivot columns come first. They usually do not. Nothing important changes: the free variables still go in the free slots, and each pivot entry is still read off from the free column with its sign flipped.

Fig. 7

Pivots in columns 1, 2 and 4. To build the special solution for column 3, put a 1 in slot 3 and a 0 in slot 5, then copy column 3 of $R$ into the pivot slots with the sign flipped. Check it: $-2 + 2 = 0$ on the first row, $-3 + 3 = 0$ on the second, and $0 = 0$ on the third.

Compare the two special solutions with the free columns of $R$ . Column 3 is $(2, 3, 0)$ and $s_1$ carries $-2$ and $-3$ in the pivot slots. Column 5 is $(5, 6, 7)$ and $s_2$ carries $-5$ , $-6$ and $-7$ in the same slots. The recipe is the one line from before, $y = -Fz$ , written out by hand.

What a null space looks like

Two free columns for the doubled-columns matrix means two knobs, which means a flat two-dimensional sheet of solutions sitting inside $\R^4$ . Every point of that sheet is sent to the single point $0$ in $\R^2$ . Nothing on the sheet survives the trip.

Fig. 8

The null space of the doubled-columns matrix is a flat two-dimensional sheet inside $\R^4$ , and every point on it lands on the single point $0$ in $\R^2$ . The sheet runs through the origin, which is what makes it a subspace.

You cannot draw $\R^4$ , so draw the sheet instead. Both knobs turn independently, the point wanders over the whole sheet, and the readout on the far side never leaves zero. Turn them yourself:

Try it. Turn the two free variables

x₃ = 1.2x₄ = 0.7

x = (-2.4, -1.4, 1.2, 0.7)

Ax = (0.00, 0.00)

The two pivot entries follow the two knobs, and the readout never leaves zero. Every point of the shaded sheet is a solution of Ax = 0.

Where this is going

You can now answer $Ax = 0$ completely, for any matrix. Eliminate, find the pivots, put a 1 in each free slot in turn, and read the pivot entries off with a flipped sign.

The harder question is the one this lecture opened with and then set aside. When $b$ is not zero the solutions form an affine set, a shifted copy of the null space. Shifted by what? The answer is short: find any single solution, then add the whole null space to it. That single solution is called the particular solution, and the recipe is $x = x_p + x_n$ . Lecture 8 builds it, checks whether it exists at all, and sorts every matrix in the world into four cases by rank.

Check yourself

For $A = \begin{bmatrix} 1 & 2 \end{bmatrix}$ , write down three baskets that cost nine dollars. Add two of them. What does the total tell you about the set they came from?
Find $N(A)$ by hand for $A = \begin{bmatrix} 1 & 3 & 5 \\ 2 & 6 & 10 \end{bmatrix}$ . How many free columns, and what are the special solutions? Check each one by computing $Ax$ .
A matrix has 7 columns and rank 4. What is $\dim N(A)$ ? If instead it has 4 columns and rank 4, what is $N(A)$ ?
Take $R = \begin{bmatrix} 1 & 0 & 2 & 0 & 5 \\ 0 & 1 & 3 & 0 & 6 \\ 0 & 0 & 0 & 1 & 7 \end{bmatrix}$ from Figure 7 and build the special solution for column 5 without looking at the answer. Then verify $R s_2 = 0$ row by row.
Why can a matrix with more columns than rows never have a null space of only the zero vector? Answer with pivots, in one sentence.
Redraw Figure 1 from memory, both panels and all labels. The drawing is the test.