Lecture 08: The Complete Solution & Rank

The idea in one sentence

Find one solution of $Ax = b$ , add every vector in the null space to it, and you have found them all.

One anchor, and the whole null space

Lecture 7.5 ended with a shape and no formula. The solutions of $Ax = b$ form an affine set, a flat sheet pushed off the origin. That left one question open: what does the pushing?

The answer takes one line. Suppose $x_p$ is any single vector with $A x_p = b$ . Call it the particular solution. Suppose $x_n$ is any vector in the null space, so $A x_n = 0$ . Then

A(x_p + x_n) = A x_p + A x_n = b + 0 = b .

So $x_p + x_n$ solves the system too. Every choice of $x_n$ gives a solution, and there are no others: if $x$ and $x_p$ both solve the system then $A(x - x_p) = b - b = 0$ , which puts $x - x_p$ in the null space. Write $x_n = x - x_p$ and you are back where you started.

x = x_p + x_n , \qquad x_n \in N(A) .

One anchor, and a whole subspace of drift. Watch it in a case small enough to draw. Take

A = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}, \qquad b = (3, 6) .

The second row is twice the first, so both equations say the same thing, $x_1 + 2x_2 = 3$ . One solution is $x_p = (3, 0)$ . The null space is the line through $x_n = (-2, 1)$ , since $-2 + 2 = 0$ . The complete solution is $x = (3, 0) + t(-2, 1)$ for every real $t$ .

Fig. 1

The dashed line is the null space, which passes through the origin. The blue line is the whole solution set, the same line pushed over so it passes through $x_p$ . Orange is the anchor, green is the drift you add to it. Land on the blue line once and you can slide along it forever.

Notice what changes when $b$ changes and what does not. The direction of the drift belongs to $A$ alone, so every solvable $b$ produces a line parallel to that dashed one. Only the anchor moves.

Fig. 2

Three right-hand sides, three parallel lines. The dashed one runs through the origin and is $N(A)$ itself, the case $b = 0$ . Orange and blue are the other two, colored apart only so you can follow them. They are the same line slid sideways, and no choice of $b$ can ever tilt it.

Try it. Slide along the solution line

t = 1.2

x = xₚ + t xₙ = (0.6, 1.2)

Ax = (3.0, 6.0)
b = (3.0, 6.0)

‖Ax − b‖ = 0.00

Every t on the blue line solves the system.

Does b even fit?

The recipe assumed a particular solution exists. Often it does not, and elimination tells you which case you are in without any extra work. Put $b$ next to $A$ as one more column and reduce the whole thing:

A = \begin{bmatrix} 1 & 3 & 0 & 2 \\ 0 & 0 & 1 & 4 \\ 1 & 3 & 1 & 6 \end{bmatrix}, \qquad b = (1, 6, 7) .

Row 3 of $A$ is row 1 plus row 2. Whatever the third equation claims, it is the sum of the first two, so the only value of $b_3$ that can possibly work is $b_1 + b_2 = 7$ . It is 7. Subtract rows 1 and 2 from row 3 of the augmented matrix and the last row goes completely flat, right side included.

Fig. 3

The same matrix, two right-hand sides. Elimination flattens the third row either way, and what sits to the right of the divider decides everything. A zero there is the harmless statement $0 = 0$ . Anything else is a contradiction, which is what the pink row is.

That is the whole consistency test. Reduce $[A \mid b]$ , look at every row of the reduced matrix that is all zeros on the left, and check that its right-hand entry is zero too. If one of them is not, some combination of your equations says $0 = 1$ and no $x$ on earth will fix it.

When the test passes, the particular solution comes out almost for free. Set every free variable to zero. In the reduced matrix above the pivots sit in columns 1 and 3, so $x_2$ and $x_4$ are free. Setting both to zero leaves $x_1 = 1$ and $x_3 = 6$ , which is exactly the right-hand column:

x_p = (1, 0, 6, 0) .

Check it: the first equation gives $1 + 0 + 0 + 0 = 1$ , the second gives $0 + 0 + 6 + 0 = 6$ , and the third gives $1 + 0 + 6 + 0 = 7$ . Now add the null space. The free columns give two special solutions by the method of Lecture 7.5, $s_1 = (-3, 1, 0, 0)$ and $s_2 = (-2, 0, -4, 1)$ , so the complete solution is

x = (1, 0, 6, 0) + c\,(-3, 1, 0, 0) + d\,(-2, 0, -4, 1) .

An anchor in $\R^4$ , plus a two-dimensional sheet of drift, for any $c$ and $d$ you like.

python

import numpy as np

A = np.array([[1., 3., 0., 2.],
              [0., 0., 1., 4.],
              [1., 3., 1., 6.]])       # row 3 = row 1 + row 2
b = np.array([1., 6., 7.])             # and 7 = 1 + 6, so it fits

M = np.hstack([A, b.reshape(-1, 1)])   # the augmented matrix
M[2] -= M[0] + M[1]                    # the only step elimination needs
print(M)                    # [[1. 3. 0. 2. 1.]
                            #  [0. 0. 1. 4. 6.]
                            #  [0. 0. 0. 0. 0.]]   <- 0 = 0, consistent

xp = np.array([1., 0., 6., 0.])        # free variables set to zero
s1 = np.array([-3., 1., 0., 0.])       # switch on x2
s2 = np.array([-2., 0., -4., 1.])      # switch on x4
print(A @ xp, A @ s1, A @ s2)          # [1. 6. 7.] [0. 0. 0.] [0. 0. 0.]
for c, d in [(0., 0.), (2., -1.), (-0.5, 3.)]:
    print(A @ (xp + c * s1 + d * s2))  # [1. 6. 7.] every time

bad = np.array([1., 6., 8.])           # 8 is not 1 + 6
Mbad = np.hstack([A, bad.reshape(-1, 1)])
Mbad[2] -= Mbad[0] + Mbad[1]
print(Mbad[2])                         # [0. 0. 0. 0. 1.]  <- 0 = 1, no good
print(np.linalg.matrix_rank(A), np.linalg.matrix_rank(Mbad))   # 2 3

python

import torch

A = torch.tensor([[1., 3., 0., 2.],
                  [0., 0., 1., 4.],
                  [1., 3., 1., 6.]])   # row 3 = row 1 + row 2
b = torch.tensor([1., 6., 7.])         # and 7 = 1 + 6, so it fits

M = torch.cat([A, b.reshape(-1, 1)], dim=1)   # the augmented matrix
M[2] -= M[0] + M[1]                    # the only step elimination needs
print(M)                    # tensor([[1., 3., 0., 2., 1.],
                            #         [0., 0., 1., 4., 6.],
                            #         [0., 0., 0., 0., 0.]])   <- 0 = 0

xp = torch.tensor([1., 0., 6., 0.])    # free variables set to zero
s1 = torch.tensor([-3., 1., 0., 0.])   # switch on x2
s2 = torch.tensor([-2., 0., -4., 1.])  # switch on x4
print(A @ xp, A @ s1, A @ s2)          # [1., 6., 7.] and two zero vectors
for c, d in [(0., 0.), (2., -1.), (-0.5, 3.)]:
    print(A @ (xp + c * s1 + d * s2))  # tensor([1., 6., 7.]) every time

bad = torch.tensor([1., 6., 8.])       # 8 is not 1 + 6
Mbad = torch.cat([A, bad.reshape(-1, 1)], dim=1)
Mbad[2] -= Mbad[0] + Mbad[1]
print(Mbad[2])                         # tensor([0., 0., 0., 0., 1.])  <- 0 = 1
print(torch.linalg.matrix_rank(A), torch.linalg.matrix_rank(Mbad))   # 2 3

python

import jax.numpy as jnp

A = jnp.array([[1., 3., 0., 2.],
               [0., 0., 1., 4.],
               [1., 3., 1., 6.]])      # row 3 = row 1 + row 2
b = jnp.array([1., 6., 7.])            # and 7 = 1 + 6, so it fits

M = jnp.hstack([A, b.reshape(-1, 1)])  # the augmented matrix
M = M.at[2].add(-(M[0] + M[1]))        # the only step elimination needs
print(M)                    # [[1. 3. 0. 2. 1.]
                            #  [0. 0. 1. 4. 6.]
                            #  [0. 0. 0. 0. 0.]]   <- 0 = 0, consistent

xp = jnp.array([1., 0., 6., 0.])       # free variables set to zero
s1 = jnp.array([-3., 1., 0., 0.])      # switch on x2
s2 = jnp.array([-2., 0., -4., 1.])     # switch on x4
print(A @ xp, A @ s1, A @ s2)          # [1. 6. 7.] [0. 0. 0.] [0. 0. 0.]
for c, d in [(0., 0.), (2., -1.), (-0.5, 3.)]:
    print(A @ (xp + c * s1 + d * s2))  # [1. 6. 7.] every time

bad = jnp.array([1., 6., 8.])          # 8 is not 1 + 6
Mbad = jnp.hstack([A, bad.reshape(-1, 1)])
Mbad = Mbad.at[2].add(-(Mbad[0] + Mbad[1]))
print(Mbad[2])                         # [0. 0. 0. 0. 1.]  <- 0 = 1, no good
print(jnp.linalg.matrix_rank(A), jnp.linalg.matrix_rank(Mbad))   # 2 3

python

import tensorflow as tf

A = tf.constant([[1., 3., 0., 2.],
                 [0., 0., 1., 4.],
                 [1., 3., 1., 6.]])    # row 3 = row 1 + row 2
b = tf.constant([1., 6., 7.])          # and 7 = 1 + 6, so it fits

M = tf.concat([A, tf.reshape(b, [-1, 1])], axis=1)   # the augmented matrix
M = tf.tensor_scatter_nd_update(M, [[2]], [M[2] - M[0] - M[1]])
print(M)                    # [[1. 3. 0. 2. 1.]
                            #  [0. 0. 1. 4. 6.]
                            #  [0. 0. 0. 0. 0.]]   <- 0 = 0, consistent

xp = tf.constant([1., 0., 6., 0.])     # free variables set to zero
s1 = tf.constant([-3., 1., 0., 0.])    # switch on x2
s2 = tf.constant([-2., 0., -4., 1.])   # switch on x4
print(A @ tf.reshape(xp, [-1, 1]))     # [[1.] [6.] [7.]]
print(A @ tf.reshape(s1, [-1, 1]))     # [[0.] [0.] [0.]]
print(A @ tf.reshape(s2, [-1, 1]))     # [[0.] [0.] [0.]]
for c, d in [(0., 0.), (2., -1.), (-0.5, 3.)]:
    x = xp + c * s1 + d * s2
    print(tf.squeeze(A @ tf.reshape(x, [-1, 1])))    # [1. 6. 7.] every time

bad = tf.constant([1., 6., 8.])        # 8 is not 1 + 6
Mbad = tf.concat([A, tf.reshape(bad, [-1, 1])], axis=1)
print(Mbad[2] - Mbad[0] - Mbad[1])     # [0. 0. 0. 0. 1.]  <- 0 = 1, no good

Rank

Everything above turned on two counts: how many pivots the matrix has, and how many columns are left over. The first count has a name.

The rank $r$ of a matrix is the number of pivots elimination produces. Equivalently, it is the number of columns that carry information no earlier column already carried, which makes it the dimension of the column space $C(A)$ . The rows tell the same story, so $r$ is also the dimension of the row space $C(A\T)$ . That last claim is the one surprise in this chapter, and Lecture 9 proves it properly.

Two consequences you already used. The null space has dimension $n - r$ , one direction per free column. And a system has a solution exactly when adding $b$ as an extra column leaves the rank alone, since a column that raises the rank is a column the old ones could not reach. So $Ax = b$ is solvable precisely when

\rank\begin{bmatrix} A & b \end{bmatrix} = \rank A .

Rank is bounded by both dimensions, so $r \le m$ and $r \le n$ . Those two comparisons, each either tight or slack, sort every matrix into four cases.

Four shapes, four answers

Fig. 4

Two questions, four answers. Cyan marks a free column, which gives the answer set room to move. Pink marks a zero row, which is where $b$ can fail. Case 1 has neither and case 4 has both, and you can read every verdict off which colors are present.

Case 1, $r = m = n$ . Square with a pivot in every row and column. No free columns means $N(A) = \lbrace 0 \rbrace$ and no drift, and no zero rows means $b$ always fits. Exactly one solution, for every $b$ . Take

A = \begin{bmatrix} 1 & 2 \\ 3 & 8 \end{bmatrix}, \qquad b = (5, 17) .

Eliminate and back-substitute to get $x = (3, 1)$ , and check: $3 + 2 = 5$ and $9 + 8 = 17$ . There is nothing to add, because the null space holds only the zero vector. This is the invertible case from Lecture 5, seen from the rank side.

Case 2, $r = m < n$ . Short and wide with a pivot in every row, so elimination never produces a zero row and $b$ always fits. But $n - r$ free columns remain, so the answer is never unique. Infinitely many solutions, for every $b$ . Take

A = \begin{bmatrix} 1 & 2 & 3 \\ 0 & 1 & 4 \end{bmatrix}, \qquad b = (6, 5) .

Subtract twice row 2 from row 1 to reach $R = \begin{bmatrix} 1 & 0 & -5 \\ 0 & 1 & 4 \end{bmatrix}$ and $d = (-4, 5)$ . Column 3 is free, so $x_p = (-4, 5, 0)$ and $s = (5, -4, 1)$ . Check the special solution: $5 - 8 + 3 = 0$ and $0 - 4 + 4 = 0$ . The complete solution is $(-4, 5, 0) + t\,(5, -4, 1)$ .

Case 3, $r = n < m$ . Tall and thin with a pivot in every column. No free columns means at most one solution, and the leftover rows mean $b$ has to cooperate. No solution or exactly one. Take

A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} .

Its column space is the plane of all $(u, v, u+v)$ , which is the set of vectors satisfying $b_1 + b_2 - b_3 = 0$ . So $b = (2, 3, 5)$ works, with the unique answer $x = (2, 3)$ , and $b = (2, 3, 6)$ does not, because $2 + 3 - 6 = -1$ . One extra unit in the third slot and the target lifts clean off the plane.

Fig. 5

Two columns in $\R^3$ span a plane, never all of space. The green target sits on the plane and has exactly one recipe, since the columns are independent. The orange one sits above it and has none, no matter how you mix.

Case 4, $r < m$ and $r < n$ . Neither full. Free columns bring back the drift, leftover rows bring back the consistency test, and you get the worst of both. No solution, or infinitely many. This case is usually left symbolic. Here it is with real numbers:

A = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 2 & 4 \end{bmatrix} .

Row 2 is twice row 1, so $r = 2$ while $m = n = 3$ . With $b = (1, 2, 3)$ the second equation is twice the first, which is consistent, and subtracting the first equation from the third gives $x_3 = 2$ , leaving $x_1 + 2x_2 = -5$ . One answer is $x_p = (-5, 0, 2)$ , the null space is the line through $(-2, 1, 0)$ , and the complete solution is $(-5, 0, 2) + t\,(-2, 1, 0)$ . Check the anchor: $-5 + 0 + 6 = 1$ , then $-10 + 0 + 12 = 2$ , then $-5 + 0 + 8 = 3$ . With $b = (1, 3, 3)$ instead, the second equation demands 3 where the first forces 2. Nothing works.

The measurements table

Here is a table you will meet again. Five people, two measurements each, and one target column:

weight $w$	height $h$	target
5	2	3
3	4	8
2	8	7
1	5	9
4	7	2

The question is the one from Lecture 1, asked with real data. Is there a pair of numbers $x = (x_1, x_2)$ with $x_1 w + x_2 h$ equal to the target column? That is $Ax = b$ with $A$ the 5 by 2 matrix of measurements, so $m = 5$ and $n = 2$ . The two columns are independent, so $r = 2$ and this is case 3: tall and thin, no solution or exactly one.

It is the no-solution branch. The rank of $A$ is 2 and the rank of $A$ with the target attached is 3, so the target column adds something the measurements never had. In geometry: $C(A)$ is a two-dimensional plane sitting inside $\R^5$ , and the target sits off it.

Fig. 6

Two columns of five numbers each span the green plane inside $\R^5$ . The orange target is a vector in $\R^5$ that does not lie on that plane, so $Ax = b$ has no solution and the dashed gap can never be closed to zero.

That is the normal situation for every real dataset. Five equations and two unknowns almost never agree. Lecture 11 measures the gap, and Lecture 12 makes it as small as possible, which is what fitting a model means. Until then it is enough to know that the answer is honestly “none”, and that “none” has a size.

python

import numpy as np

square = (np.array([[1., 2.], [3., 8.]]), np.array([5., 17.]))
wide   = (np.array([[1., 2., 3.], [0., 1., 4.]]), np.array([6., 5.]))
tall   = (np.array([[1., 0.], [0., 1.], [1., 1.]]), np.array([2., 3., 5.]))
flat   = (np.array([[1., 2., 3.], [2., 4., 6.], [1., 2., 4.]]),
          np.array([1., 2., 3.]))
table  = (np.array([[5., 2.], [3., 4.], [2., 8.], [1., 5.], [4., 7.]]),
          np.array([3., 8., 7., 9., 2.]))

cases = {"square, invertible": square, "short and wide": wide,
         "tall and thin": tall, "not full rank": flat,
         "measurements table": table}

for name, (A, b) in cases.items():
    m, n = A.shape
    r = np.linalg.matrix_rank(A)
    fits = np.linalg.matrix_rank(np.hstack([A, b.reshape(-1, 1)])) == r
    how = "none" if not fits else ("one" if r == n else "infinitely many")
    print(f"{name:20s} m={m} n={n} r={r} fits:{str(fits):6s}{how}")

# square, invertible   m=2 n=2 r=2 fits:True  one
# short and wide       m=2 n=3 r=2 fits:True  infinitely many
# tall and thin        m=3 n=2 r=2 fits:True  one
# not full rank        m=3 n=3 r=2 fits:True  infinitely many
# measurements table   m=5 n=2 r=2 fits:False none

A, b = table
best = np.linalg.pinv(A) @ b     # the closest attempt, Lecture 12's answer
print(A @ best - b)
# [-0.84187449 -4.06166073  0.66072897 -4.22554124  4.82460948]
print(np.linalg.norm(A @ best - b))   # 7.666450205657343

python

import torch

square = (torch.tensor([[1., 2.], [3., 8.]]), torch.tensor([5., 17.]))
wide   = (torch.tensor([[1., 2., 3.], [0., 1., 4.]]), torch.tensor([6., 5.]))
tall   = (torch.tensor([[1., 0.], [0., 1.], [1., 1.]]),
          torch.tensor([2., 3., 5.]))
flat   = (torch.tensor([[1., 2., 3.], [2., 4., 6.], [1., 2., 4.]]),
          torch.tensor([1., 2., 3.]))
table  = (torch.tensor([[5., 2.], [3., 4.], [2., 8.], [1., 5.], [4., 7.]]),
          torch.tensor([3., 8., 7., 9., 2.]))

cases = {"square, invertible": square, "short and wide": wide,
         "tall and thin": tall, "not full rank": flat,
         "measurements table": table}

for name, (A, b) in cases.items():
    m, n = A.shape
    r = int(torch.linalg.matrix_rank(A))
    aug = torch.cat([A, b.reshape(-1, 1)], dim=1)
    fits = int(torch.linalg.matrix_rank(aug)) == r
    how = "none" if not fits else ("one" if r == n else "infinitely many")
    print(f"{name:20s} m={m} n={n} r={r} fits:{str(fits):6s}{how}")

# square, invertible   m=2 n=2 r=2 fits:True  one
# short and wide       m=2 n=3 r=2 fits:True  infinitely many
# tall and thin        m=3 n=2 r=2 fits:True  one
# not full rank        m=3 n=3 r=2 fits:True  infinitely many
# measurements table   m=5 n=2 r=2 fits:False none

A, b = table
best = torch.linalg.pinv(A) @ b  # the closest attempt, Lecture 12's answer
print(A @ best - b)
# tensor([-0.8419, -4.0617,  0.6607, -4.2255,  4.8246])
print(torch.linalg.norm(A @ best - b))   # tensor(7.6665)

python

import jax.numpy as jnp

square = (jnp.array([[1., 2.], [3., 8.]]), jnp.array([5., 17.]))
wide   = (jnp.array([[1., 2., 3.], [0., 1., 4.]]), jnp.array([6., 5.]))
tall   = (jnp.array([[1., 0.], [0., 1.], [1., 1.]]), jnp.array([2., 3., 5.]))
flat   = (jnp.array([[1., 2., 3.], [2., 4., 6.], [1., 2., 4.]]),
          jnp.array([1., 2., 3.]))
table  = (jnp.array([[5., 2.], [3., 4.], [2., 8.], [1., 5.], [4., 7.]]),
          jnp.array([3., 8., 7., 9., 2.]))

cases = {"square, invertible": square, "short and wide": wide,
         "tall and thin": tall, "not full rank": flat,
         "measurements table": table}

for name, (A, b) in cases.items():
    m, n = A.shape
    r = int(jnp.linalg.matrix_rank(A))
    aug = jnp.hstack([A, b.reshape(-1, 1)])
    fits = int(jnp.linalg.matrix_rank(aug)) == r
    how = "none" if not fits else ("one" if r == n else "infinitely many")
    print(f"{name:20s} m={m} n={n} r={r} fits:{str(fits):6s}{how}")

# square, invertible   m=2 n=2 r=2 fits:True  one
# short and wide       m=2 n=3 r=2 fits:True  infinitely many
# tall and thin        m=3 n=2 r=2 fits:True  one
# not full rank        m=3 n=3 r=2 fits:True  infinitely many
# measurements table   m=5 n=2 r=2 fits:False none

A, b = table
best = jnp.linalg.pinv(A) @ b    # the closest attempt, Lecture 12's answer
print(A @ best - b)
# [-0.8418745 -4.0616603  0.660729  -4.225541   4.8246098]
print(jnp.linalg.norm(A @ best - b))     # 7.66645

python

import tensorflow as tf

square = (tf.constant([[1., 2.], [3., 8.]]), tf.constant([[5.], [17.]]))
wide   = (tf.constant([[1., 2., 3.], [0., 1., 4.]]), tf.constant([[6.], [5.]]))
tall   = (tf.constant([[1., 0.], [0., 1.], [1., 1.]]),
          tf.constant([[2.], [3.], [5.]]))
flat   = (tf.constant([[1., 2., 3.], [2., 4., 6.], [1., 2., 4.]]),
          tf.constant([[1.], [2.], [3.]]))
table  = (tf.constant([[5., 2.], [3., 4.], [2., 8.], [1., 5.], [4., 7.]]),
          tf.constant([[3.], [8.], [7.], [9.], [2.]]))

cases = {"square, invertible": square, "short and wide": wide,
         "tall and thin": tall, "not full rank": flat,
         "measurements table": table}

def rank(M):        # tf has no matrix_rank: count the nonzero singular values
    s = tf.linalg.svd(M, compute_uv=False)
    return int(tf.reduce_sum(tf.cast(s > 1e-8 * s[0], tf.int32)))

for name, (A, b) in cases.items():
    m, n = A.shape
    r = rank(A)
    fits = rank(tf.concat([A, b], axis=1)) == r
    how = "none" if not fits else ("one" if r == n else "infinitely many")
    print(f"{name:20s} m={m} n={n} r={r} fits:{str(fits):6s}{how}")

# square, invertible   m=2 n=2 r=2 fits:True  one
# short and wide       m=2 n=3 r=2 fits:True  infinitely many
# tall and thin        m=3 n=2 r=2 fits:True  one
# not full rank        m=3 n=3 r=2 fits:True  infinitely many
# measurements table   m=5 n=2 r=2 fits:False none

A, b = table
best = tf.linalg.pinv(A) @ b     # the closest attempt, Lecture 12's answer
print(tf.squeeze(A @ best - b))
# [-0.8418745 -4.0616603  0.660729  -4.225541   4.8246098]
print(tf.norm(A @ best - b))     # 7.66645

Basis, independence, dimension

Three words have been doing quiet work throughout, and they deserve one paragraph each.

Vectors $v_1, \dots, v_k$ are independent when the only combination of them equal to zero is the one with all coefficients zero. Stack them as the columns of a matrix and that sentence becomes a null space statement: the columns are independent exactly when $N(A) = \lbrace 0 \rbrace$ . Free columns are the dependent ones, and each free column hands you a nonzero vector in the null space that proves the dependence.

A set spans a space when every vector in that space is some combination of the set. A basis is a set that does both jobs, independent and spanning. Independence keeps it from being wasteful and spanning keeps it from being short, and together they buy you something worth having: in terms of a basis, every vector in the space has exactly one recipe. Two different recipes for the same vector would subtract to a nonzero combination equal to zero.

Fig. 7

The two orange vectors are the basis, the blue steps are the walk, and green is where it lands: $w = (7, 4)$ from two steps of $v_1 = (3, 1)$ and one of $v_2 = (1, 2)$ . Independence is what makes the walk unique. Throw in a third vector like $v_1 + v_2$ and the same $w$ suddenly has many walks, which is exactly what dependence costs you.

Every basis of a given space has the same number of vectors, and that number is the dimension. For $C(A)$ the pivot columns of $A$ form a basis, so the dimension is $r$ . For $N(A)$ the special solutions form a basis, so the dimension is $n - r$ . The space $\lbrace 0 \rbrace$ has dimension 0 and its basis is the empty set, which is the only sensible answer: the zero vector can never belong to a basis, since $1$ times it is already a nonzero combination equal to zero.

Where this is going

Count the dimensions on both sides of $A$ . On the input side $\R^n$ holds the null space with $n - r$ dimensions and the row space $C(A\T)$ with $r$ . On the output side $\R^m$ holds the column space $C(A)$ with $r$ , and one more space: the vectors $y$ with $A\T y = 0$ . That one is the left null space $N(A\T)$ , and it has $m - r$ dimensions. Four spaces, two on each side, and the counts add up exactly.

Fig. 8

A thumbnail of the four fundamental subspaces. The four colors are fixed for the whole course: orange row space, cyan null space, green column space, purple left null space. Lecture 9 finds a basis for each one and explains why both $r$ counts are the same number. It also draws the version of this picture that everything later refers back to.

Two of those four spaces you have already met and can compute. The other two arrive next, along with the reason the two $r$ counts match. After that comes the question this chapter left open: when $b$ misses the column space, as it did for the measurements table, what is the closest thing to an answer?

Check yourself

Solve $x_1 + 2x_2 + x_3 = 4$ and $2x_1 + 4x_2 + 2x_3 = 8$ completely. What is $x_p$ , what is the null space, and how many free variables are there?
For the matrix in Figure 3, which right-hand sides $(b_1, b_2, b_3)$ are consistent? Write the condition as one equation.
Sort each of these into one of the four cases, using only $m$ , $n$ and $r$ : a 3 by 3 matrix of rank 3; a 2 by 5 matrix of rank 2; a 100 by 3 matrix of rank 3; a 4 by 4 matrix of rank 2. For each one, say how many solutions $Ax = b$ can have.
Build a 3 by 4 matrix of rank 2 and a right-hand side $b$ for which $Ax = b$ has no solution. Then change one entry of $b$ so that it does.
The measurements table has $r = 2$ and $m = 5$ . What is the dimension of $N(A\T)$ , and what does a vector in it tell you about the five rows of the table?
Redraw Figure 4 from memory, all four cells and both axis labels. The drawing is the test.