Lecture 07: Vector Spaces & Subspaces

The idea in one sentence

A vector space is any collection where you can add and scale without ever falling out, and once you know what to look for you find them everywhere, including inside the columns of your data.

The rules that make adding worth the name

Every result so far has come from two moves. Scale a thing. Add two things. Lecture 1 called the combination of those moves a linear combination and never asked what “thing” meant. It never had to. But at some point you have to say what you are assuming about addition, because the whole subject rests on it.

Start with what addition must never do. It must not care about the order you write things in, and it must not care how you group them. If $u + v$ and $v + u$ could differ, or if $(u + v) + w$ and $u + (v + w)$ could differ, then a sum of three vectors would not be a single object and the phrase “the sum” would be meaningless.

Next, addition needs a place to stand still. There has to be one element, called $0$ , that changes nothing when you add it. And every move has to be undoable: for each $u$ there is a $-u$ that brings you back to $0$ . Without those two you could walk but never return, and an algebra you cannot run backwards is not much use.

Now scaling. Scaling twice should be the same as scaling once by the product, so $a(bu) = (ab)u$ . Scaling has to distribute over adding, in both senses: scaling a sum is the same as scaling the parts, $a(u + v) = au + av$ , and adding two scale factors is the same as scaling twice and adding, $(a + b)u = au + bu$ . These two are what tie the two operations together. Drop them, and scaling and adding go their separate ways.

The last rule looks too obvious to state. Scaling by 1 must do nothing:

1\,u = u .

It is not obvious and it is not free. Take $\R^2$ with ordinary addition, but define $c\,(x, y)$ to be $(cx, 0)$ for every scalar. Check the other seven rules and they all hold. But $1(0,1) = (0,0)$ , so the last rule fails, and the object it produces is useless: every vector is scaled into the horizontal axis and information disappears for no reason. That eighth rule is what stops scaling from quietly destroying the space.

Fig. 1

Why the eighth rule is not free. Keep ordinary addition but define $c\,(x, y)$ to be $(cx, 0)$ . The other seven rules survive. This one dies, and the blue vectors land on the pink dots, every second coordinate gone.

Those are the eight rules, and together they are the definition. A vector space is a set with an addition and a scalar multiplication that obey them. Nothing in the list mentions arrows or coordinates.

Three spaces that do not look like arrows

Since the rules never mention arrows, other things qualify. Here are three, and each one is worth holding in your head, because they are the reason the abstraction earns its keep.

Two-by-two matrices. Call the set $M$ . Add two of them entry by entry, scale one by multiplying every entry. Both operations stay inside $M$ : a $2\times 2$ matrix plus a $2\times 2$ matrix is a $2\times 2$ matrix. Watch a real combination:

2\begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} + \begin{bmatrix} 0 & -1 \\ 4 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 4 & 7 \end{bmatrix}.

The zero vector of this space is the zero matrix, and the additive inverse of a matrix is the matrix with every sign flipped. The scaling identity reads

1 \begin{bmatrix} a & b \\ c & d \end{bmatrix} = \begin{bmatrix} a & b \\ c & d \end{bmatrix},

which is scaling, and has nothing to do with the matrix of all ones. Notice that naming an element of $M$ takes four numbers, exactly as naming a point of $\R^4$ does. That resemblance is real, and Lecture 8 gives it a name.

Functions. Call the set $F$ : every real function $f(x)$ . Adding two functions means adding their values at each point, $(f + g)(x) = f(x) + g(x)$ . Scaling means multiplying every value. Take $f(x) = x^2$ and $g(x) = 1 - x/2$ . Then $(f+g)(x) = x^2 - x/2 + 1$ , which at $x = -2$ is $4 + 1 + 1 = 6$ . The zero vector is the function that is zero everywhere. Nothing here has coordinates. To name one element of $F$ you have to give a number for every real input, so this space is not $\R^n$ for any $n$ . It is where the mathematics behind kernel methods and Fourier analysis lives.

The zero space. Call it $Z$ , and let it contain the single element $0$ . Adding gives $0 + 0 = 0$ , staying inside. Scaling gives $c\,0 = 0$ , staying inside. Every one of the eight rules holds, dully. $Z$ is a vector space. The empty set is not, because a vector space has to contain a zero, and the empty set contains nothing at all. The smallest possible vector space has one element, and it will keep turning up.

Fig. 2

Three vector spaces, none of them made of arrows. Orange is what goes in, green is what comes out, and the two moves are the same in all three: add the parts, scale every part. The zero vector is the zero matrix, the zero function, and the only element there is.

The function space is the one that resists a picture, because its elements have no coordinates. There is a way in. Stop asking for the whole function at once and just read it at a few points.

Fig. 3

A function turns into a vector the moment you sample it. The five yellow readings off the blue curve are the five entries of the column. Take a hundred inputs and it is a point of $\R^{100}$ . Take every real input at once and you have the function itself, which is why $F$ is not $\R^n$ for any $n$ .

python

import numpy as np

F = np.array([[1., 2], [0, 3]])
G = np.array([[0., -1], [4, 1]])
print(2 * F + G)             # [[2. 3.] [4. 7.]]   a mix of two matrices
print(1 * F)                 # [[1. 2.] [0. 3.]]   1 changes nothing
print(F + np.zeros((2, 2)))  # F again: the zero vector is the zero matrix

# a function becomes a vector the moment you sample it
x = np.linspace(-2, 2, 5)  # [-2. -1.  0.  1.  2.]
f, g = x ** 2, 1 - x / 2
print(f + g)                          # [6.  2.5 1.  1.5 4. ]
print(3 * (f + g) - (3 * f + 3 * g))  # zeros: scaling distributes

python

import torch

F = torch.tensor([[1., 2], [0, 3]])
G = torch.tensor([[0., -1], [4, 1]])
print(2 * F + G)              # tensor([[2., 3.], [4., 7.]])   a mix of two
print(1 * F)                  # tensor([[1., 2.], [0., 3.]])   1 does nothing
print(F + torch.zeros(2, 2))  # F again: the zero vector is the zero matrix

# a function becomes a vector the moment you sample it
x = torch.linspace(-2, 2, 5)  # tensor([-2., -1., 0., 1., 2.])
f, g = x ** 2, 1 - x / 2
print(f + g)                # tensor([6.0000, 2.5000, 1.0000, 1.5000, 4.0000])
print(3 * (f + g) - (3 * f + 3 * g))  # zeros: scaling distributes

python

import jax.numpy as jnp

F = jnp.array([[1., 2], [0, 3]])
G = jnp.array([[0., -1], [4, 1]])
print(2 * F + G)              # [[2. 3.] [4. 7.]]   a mix of two matrices
print(1 * F)                  # [[1. 2.] [0. 3.]]   1 changes nothing
print(F + jnp.zeros((2, 2)))  # F again: the zero vector is the zero matrix

# a function becomes a vector the moment you sample it
x = jnp.linspace(-2, 2, 5)  # [-2. -1.  0.  1.  2.]
f, g = x ** 2, 1 - x / 2
print(f + g)                          # [6.  2.5 1.  1.5 4. ]
print(3 * (f + g) - (3 * f + 3 * g))  # zeros: scaling distributes

python

import tensorflow as tf

F = tf.constant([[1., 2], [0, 3]])
G = tf.constant([[0., -1], [4, 1]])
print(2 * F + G)             # [[2. 3.] [4. 7.]]   a mix of two matrices
print(1 * F)                 # [[1. 2.] [0. 3.]]   1 changes nothing
print(F + tf.zeros((2, 2)))  # F again: the zero vector is the zero matrix

# a function becomes a vector the moment you sample it
x = tf.linspace(-2., 2., 5)  # [-2. -1.  0.  1.  2.]
f, g = x ** 2, 1 - x / 2
print(f + g)                          # [6.  2.5 1.  1.5 4. ]
print(3 * (f + g) - (3 * f + 3 * g))  # zeros: scaling distributes

Subspaces: two questions, and that is all

Whole vector spaces are useful. The pieces inside them matter more. A subspace of a vector space $V$ is a subset that is a vector space in its own right, using the same addition and the same scaling.

Checking eight rules on every candidate would be miserable, and it is unnecessary. Six of the eight say things like “addition does not care about order”, and if that is true for all of $V$ it is automatically true for anything sitting inside $V$ . Only one thing can go wrong: a sum or a scaling can walk out of the subset. So the whole test is two questions.

Take any $v$ and $w$ in the subset $S$ , and any scalar $c$ :

Is $v + w$ still in $S$ ?
Is $c\,v$ still in $S$ ?

If both answers are yes for every choice, $S$ is a subspace. That property has a name, closure, and that is the entire definition.

One consequence comes for free. Put $c = 0$ in the second question. Then $0\,v$ is in $S$ , and $0\,v$ is the zero vector. Every subspace contains the zero vector, so a plane in $\R^3$ that misses the origin cannot be a subspace, and you can rule such a set out at a glance.

Fig. 4

Left, $v = (2, 1)$ and $w = (-4, -2)$ both sit on the green line $y = x/2$ , and their sum $(-2, -1)$ lands back on it. Right, $v = (2, 3)$ and $w = (-2, 1)$ sit on $y = x/2 + 2$ , and their sum $(0, 4)$ misses it by 2. Shifting a line off the origin breaks it.

The gallery, and the one that fails

Inside $\R^3$ there are exactly four kinds of subspace, and they line up by dimension. The zero vector on its own. Any line through the origin. Any plane through the origin. All of $\R^3$ . Every subspace of $\R^3$ is one of those four.

Fig. 5

The complete gallery for $\R^3$ , in green because every one of them passes the two-question test. Each contains the origin, where the axes cross, and there is nothing else on the list.

Now the instructive failure. Take the first quadrant of the plane, all points with both coordinates at least zero. It looks well behaved. Test it.

Add $v = (2, 3)$ and $w = (3, 1)$ . The sum is $(5, 4)$ , still in the quadrant. Try other pairs and the same thing happens, because adding two non-negative numbers never gives a negative one. The first question passes, every time.

Now scale. Take $c = -1$ :

-1\,(2, 3) = (-2, -3),

which is in the opposite corner of the plane. The second question fails on the first try. The first quadrant is not a subspace, and it fails for a clean reason: a set can be perfectly closed under addition and still be broken, because going backwards is part of what scaling means.

Fig. 6

The pink region is the candidate set. Blue is what works: $v = (2, 3)$ and $w = (3, 1)$ add to $(5, 4)$ , still inside. Orange is what breaks it: scaling $v$ by $-1$ gives $(-2, -3)$ , and the shading stops at the axes. One failing example is enough.

Try it. Break the set if you can

v along the setw along the setc = -1.0

v + w inside
c v inside

Both questions pass so far. Keep looking.

Blue is what you move. A test dot turns green when it stays in the set and pink when it leaves.

python

import numpy as np

def in_quadrant(u):
    return bool(np.all(u >= 0))

v = np.array([2., 3])
w = np.array([3., 1])
print(v + w, in_quadrant(v + w))    # [5. 4.] True    adding is fine
print(-1 * v, in_quadrant(-1 * v))  # [-2. -3.] False  scaling breaks it

python

import torch

def in_quadrant(u):
    return bool(torch.all(u >= 0))

v = torch.tensor([2., 3])
w = torch.tensor([3., 1])
print(v + w, in_quadrant(v + w))    # tensor([5., 4.]) True   adding is fine
print(-1 * v, in_quadrant(-1 * v))  # tensor([-2., -3.]) False  it breaks

python

import jax.numpy as jnp

def in_quadrant(u):
    return bool(jnp.all(u >= 0))

v = jnp.array([2., 3])
w = jnp.array([3., 1])
print(v + w, in_quadrant(v + w))    # [5. 4.] True    adding is fine
print(-1 * v, in_quadrant(-1 * v))  # [-2. -3.] False  scaling breaks it

python

import tensorflow as tf

def in_quadrant(u):
    return bool(tf.reduce_all(u >= 0))  # tf spells all "reduce_all"

v = tf.constant([2., 3])
w = tf.constant([3., 1])
print(v + w, in_quadrant(v + w))    # [5. 4.] True    adding is fine
print(-1 * v, in_quadrant(-1 * v))  # [-2. -3.] False  scaling breaks it

Every span is a subspace

Lecture 1 defined the span of a set of vectors as everything you can reach by combining them. That was a picture. Now it is a theorem, and the proof takes two lines.

Let $S$ be the span of $v$ and $w$ , so a typical element is $a v + b w$ . Add two of them:

(a_1 v + b_1 w) + (a_2 v + b_2 w) = (a_1 + a_2) v + (b_1 + b_2) w,

which is again a combination of $v$ and $w$ , so it is in $S$ . Scale one of them:

c\,(a v + b w) = (ca) v + (cb) w,

also a combination, also in $S$ . Both questions answered. The span is a subspace, and the same two lines work for any number of vectors.

This is why the gallery looks the way it does. The span of one nonzero vector in $\R^3$ is a line through the origin. The span of two vectors pointing in different directions is a plane through the origin, and Lecture 1 met one: the combinations of $(1,1,1)$ and $(2,3,4)$ fill a plane, and $(1,0,0)$ is off it. Spans always run through the origin, because taking all the coefficients zero is allowed.

The column space of a matrix

Now put the pieces together, and the payoff arrives.

Here is a ratings table. Five people, three films, each entry a score out of ten:

A = \begin{bmatrix} 5 & 2 & 3 \\ 3 & 4 & 8 \\ 2 & 8 & 7 \\ 1 & 5 & 9 \\ 4 & 7 & 2 \end{bmatrix}.

Read it by columns, the way Lecture 2 taught. Column 1 is film 1, described by the five scores it received: $m_1 = (5, 3, 2, 1, 4)$ . Column 2 is film 2 and column 3 is film 3. Each film is a vector in $\R^5$ , one coordinate per viewer.

Mixing the films means combining the columns. Take two parts of film 1, one part of film 2 and none of film 3:

2 m_1 + 1 m_2 + 0 m_3 = (12, 10, 12, 7, 15),

and that is $Ax$ with $x = (2, 1, 0)$ . It is a prediction: if a new film were exactly this blend, those five numbers are what the five viewers would give it.

Now ask the question this course keeps asking. As $x$ ranges over all of $\R^3$ , what can $Ax$ be? The answer is the span of the three columns. It is a subspace of $\R^5$ , by the theorem in the last section, and it has a name and a symbol:

C(A) \;=\; \{\, Ax : x \in \R^3 \,\}.

This is the column space. It contains a lot more than the three columns themselves. It contains every blend of them, and the columns are just three of its members.

Fig. 7

Five viewers, three films. The green columns are the three films, each living in $\R^5$ ; yellow holds the weights. Every blend of the columns is a point of the column space $C(A)$ , and this blend, $2m_1 + m_2$ , predicts the scores $(12, 10, 12, 7, 15)$ .

Why the ambient space matters

$C(A)$ lives inside $\R^5$ , and it is a small part of it. Three columns can span at most three dimensions, so $C(A)$ is at most a three-dimensional slab inside a five-dimensional room. Almost nothing in $\R^5$ is in it.

That smallness is the shape of the problem. Take the reachable vector $b = (12, 10, 12, 7, 15)$ and change one entry by one, to $b' = (12, 10, 12, 7, 16)$ . Nothing about $b'$ looks unusual. But no weights on the three films produce it. The closest the column space can get misses by

(0.1712,\; -0.1117,\; 0.3497,\; -0.1715,\; -0.2622)

and that miss has squared length $0.2622$ . One viewer rated the blend one point higher than the films can explain, and the whole vector fell out of the space.

That gap is the story of everything that follows. It is why no real ratings matrix is reproduced exactly by a handful of factors, and why “solve $Ax = b$ ” usually turns into “get as close to $b$ as $C(A)$ allows”. Part IV of this course does that properly and calls it least squares.

python

import numpy as np

A = np.array([[5., 2, 3], [3, 4, 8], [2, 8, 7], [1, 5, 9], [4, 7, 2]])
print(A.shape, np.linalg.matrix_rank(A))  # (5, 3) 3   a slab inside R^5

x = np.array([2., 1, 0])
b = A @ x
print(b)  # [12. 10. 12.  7. 15.]  a blend

# ask for the best weights, see what is left over
print((np.linalg.pinv(A) @ b).round(6))  # [2. 1. 0.]  reached exactly

b2 = np.array([12., 10, 12, 7, 16])  # one score, one point higher
best = np.linalg.pinv(A) @ b2
print((A @ best - b2).round(4))    # [ 0.1712 -0.1117  0.3497 -0.1715 -0.2622]
print(np.sum((A @ best - b2) ** 2))  # 0.2622   b2 is outside C(A)

python

import torch

A = torch.tensor([[5., 2, 3], [3, 4, 8], [2, 8, 7], [1, 5, 9], [4, 7, 2]])
print(A.shape, torch.linalg.matrix_rank(A))  # torch.Size([5, 3]) tensor(3)

x = torch.tensor([2., 1, 0])
b = A @ x
print(b)  # tensor([12., 10., 12.,  7., 15.])

# ask for the best weights, see what is left over
print((torch.linalg.pinv(A) @ b).round(decimals=6))  # tensor([2., 1., 0.])

b2 = torch.tensor([12., 10, 12, 7, 16])  # one score, one point higher
best = torch.linalg.pinv(A) @ b2
print((A @ best - b2).round(decimals=4))
# [ 0.1712 -0.1117  0.3497 -0.1715 -0.2622]
print(torch.sum((A @ best - b2) ** 2))  # tensor(0.2622)   outside C(A)

python

import jax.numpy as jnp

A = jnp.array([[5., 2, 3], [3, 4, 8], [2, 8, 7], [1, 5, 9], [4, 7, 2]])
print(A.shape, jnp.linalg.matrix_rank(A))  # (5, 3) 3   a slab inside R^5

x = jnp.array([2., 1, 0])
b = A @ x
print(b)  # [12. 10. 12.  7. 15.]  a blend

# ask for the best weights, see what is left over
print((jnp.linalg.pinv(A) @ b).round(6))  # [2. 1. 0.]  reached exactly

b2 = jnp.array([12., 10, 12, 7, 16])  # one score, one point higher
best = jnp.linalg.pinv(A) @ b2
print((A @ best - b2).round(4))    # [ 0.1712 -0.1117  0.3497 -0.1715 -0.2622]
print(jnp.sum((A @ best - b2) ** 2))  # 0.2622   b2 is outside C(A)

python

import tensorflow as tf

A = tf.constant([[5., 2, 3], [3, 4, 8], [2, 8, 7], [1, 5, 9], [4, 7, 2]])
print(A.shape, tf.linalg.matrix_rank(A))  # (5, 3) 3   a slab inside R^5

x = tf.constant([[2.], [1.], [0.]])  # tf keeps vectors as columns
b = A @ x
print(tf.squeeze(b))  # [12. 10. 12.  7. 15.]  a blend

# ask for the best weights, see what is left over
print(tf.squeeze(tf.linalg.pinv(A) @ b))  # [2. 1. 0.]  reached exactly

b2 = tf.constant([[12.], [10.], [12.], [7.], [16.]])  # one point higher
best = tf.linalg.pinv(A) @ b2
print(tf.squeeze(A @ best - b2))   # [ 0.1712 -0.1117  0.3497 -0.1715 -0.2622]
print(tf.reduce_sum((A @ best - b2) ** 2))  # 0.2622   b2 is outside C(A)

Fig. 8

The green slab is the column space, at most three directions wide, inside a five-dimensional room. $b$ lies on it and is reachable. The pink $b'$ differs from $b$ in one coordinate by one point and floats just off the slab, so no weights produce it.

Where this is going

Adding and scaling turned out to define everything, and the sets that survive both moves are the subspaces. The columns of a matrix span one of them, $C(A)$ , and asking whether $Ax = b$ has a solution is asking whether $b$ lies in it.

That is half of the picture, and it is the half about outputs. The other half is about inputs. Some $x$ get sent to zero: the ring road machine from Lecture 2 flattened $(1,1,1)$ to nothing, and a matrix with a repeated column will flatten something too. Those blind directions form a subspace of their own, sitting in the room where $x$ lives. It is called the null space, and finding it is where the next chapter starts.

Check yourself

Which of these are subspaces of $\R^2$ : the line $y = 3x$ , the line $y = 3x + 1$ , the single point $(0,0)$ , the set of vectors with $x_1 x_2 = 0$ ? For each failure, give one vector or one scalar that breaks it.
Is the set of $2 \times 2$ matrices with $a + d = 0$ a subspace of $M$ ? Check both questions. What about the ones with $a + d = 1$ ?
Is the set of functions with $f(0) = 0$ a subspace of $F$ ? What about $f(0) = 3$ ?
Using the ratings matrix, compute $A x$ for $x = (1, 1, 1)$ by hand. Then find weights that produce $(5, 3, 2, 1, 4)$ exactly, without doing any arithmetic.
The span of two vectors in $\R^3$ is usually a plane. Give two vectors whose span is only a line, and say which of the four gallery cases you landed in.
Redraw Figure 6 from memory, arrows and labels included. The drawing is the test.