Matrices as Transformations

The idea in one sentence

A matrix is a machine: a vector goes in, a moved vector comes out, and the columns of the matrix tell you everything the machine will ever do.

One multiplication, two readings

Start with the numbers:

There are two ways to compute $Ax$, and they teach two different things.

Read $A$ by columns. Then $Ax$ is the move from Lecture 1, a linear combination, with the entries of $x$ as the scalars:

Read $A$ by rows. Then each entry of the answer is a dot product: the first entry is $1 \cdot 7 + 2 \cdot 8 = 23$, and so on down.

Both readings matter, for different reasons. The row reading is how you compute by hand. The column reading answers the deeper question: since $Ax$ is always a mix of the columns, the outputs a matrix can produce are exactly the span of its columns. Reachability, the question Lecture 1 ended on, is a question about columns.

A machine that moves every point at once

Now change the point of view. Fix $A$ and let $x$ vary. The matrix becomes a function: feed it any vector, it returns a moved vector. To see what the machine does, you do not need to test every input. Two are enough. Watch $e_1 = (1, 0)$ and $e_2 = (0, 1)$:

The columns of a matrix are the landing spots of the basis vectors. Every other input follows along, because $x = x_1 e_1 + x_2 e_2$ forces $Ax = x_1(\text{column 1}) + x_2(\text{column 2})$. Grid lines stay straight, parallel lines stay parallel, and the origin stays put. That is what the word linear means, drawn.

Every 2 by 2 matrix is one of these machines. Rotations, shears, reflections, scalings: each is four numbers, and the four numbers are just the two landing spots written side by side. Try it:

The machine view is what AI runs on. A language model stores words as vectors, and in word2vec those vectors did combination arithmetic: king minus man plus woman landed near queen. Subtract one vector, add another, and you have walked from one word to another.

The operations inside every neural network layer are the two moves of this lecture, a matrix multiplication and an addition, repeated at scale.

The machine that destroys information

Here is a machine with a story. Three cities sit on a ring road, with altitudes $x_1, x_2, x_3$. The machine reports the climbs between them, one number for each stretch of road:

Now raise all three cities by one meter. The climbs do not change: $C$ applied to $(1, 1, 1)$ gives $(0, 0, 0)$. Two different inputs, one output. The machine cannot see the direction $(1, 1, 1)$, and what a machine cannot see, no one can recover from its output. $C$ destroys information, and a machine that destroys information cannot be undone. Matrices like this are called singular.

Singularity also limits the outputs. Add the three rows of $Cx$ and everything cancels, so any output $b$ this machine can produce must satisfy $b_1 + b_2 + b_3 = 0$. That one equation sorts every right-hand side into one of two endings:

• $b = (1, 1, -2)$ sums to zero. Solvable, with altitudes $(1, 2, 0)$, and once you have one answer, adding $(1, 1, 1)$ gives another. Infinitely many solutions.
• $b = (1, 2, 3)$ sums to six. No altitudes on earth produce these climbs. No solution.

Walking the ring shows why.

A singular machine never gives the clean ending: exactly one answer. The machines that do, for every $b$, are called invertible, and telling the two kinds apart quickly is where this course goes next.

import numpy as np

A = np.array([[1., 2.], [3., 4.], [5., 6.]])
x = np.array([7., 8.])
print(A @ x)                        # [23. 53. 83.]
print(7 * A[:, 0] + 8 * A[:, 1])    # the same: Ax mixes the columns

C = np.array([[1., 0., -1.], [-1., 1., 0.], [0., -1., 1.]])
print(C @ np.array([1., 1., 1.]))   # [0. 0. 0.]  the invisible direction
print(np.linalg.det(C))             # 0.0  singular

b = np.array([1., 2., 3.])          # climbs that don't cancel
best = np.linalg.pinv(C) @ b        # the best possible attempt
print(C @ best - b)                 # [-2. -2. -2.]  misses: no solution

b = np.array([1., 1., -2.])         # climbs that cancel
print(C @ (np.linalg.pinv(C) @ b))  # [ 1.  1. -2.]  reached exactly

import torch

A = torch.tensor([[1., 2.], [3., 4.], [5., 6.]])
x = torch.tensor([7., 8.])
print(A @ x)                        # tensor([23., 53., 83.])
print(7 * A[:, 0] + 8 * A[:, 1])    # the same: Ax mixes the columns

C = torch.tensor([[1., 0., -1.], [-1., 1., 0.], [0., -1., 1.]])
print(C @ torch.ones(3))            # tensor([0., 0., 0.])  invisible direction
print(torch.linalg.det(C))          # tensor(0.)  singular

b = torch.tensor([1., 2., 3.])      # climbs that don't cancel
best = torch.linalg.pinv(C) @ b     # the best possible attempt
print(C @ best - b)                 # tensor([-2., -2., -2.])  no solution

b = torch.tensor([1., 1., -2.])     # climbs that cancel
print(C @ (torch.linalg.pinv(C) @ b))  # tensor([ 1.,  1., -2.])  reached

import jax.numpy as jnp

A = jnp.array([[1., 2.], [3., 4.], [5., 6.]])
x = jnp.array([7., 8.])
print(A @ x)                        # [23. 53. 83.]
print(7 * A[:, 0] + 8 * A[:, 1])    # the same: Ax mixes the columns

C = jnp.array([[1., 0., -1.], [-1., 1., 0.], [0., -1., 1.]])
print(C @ jnp.ones(3))              # [0. 0. 0.]  the invisible direction
print(jnp.linalg.det(C))            # 0.0  singular

b = jnp.array([1., 2., 3.])         # climbs that don't cancel
best = jnp.linalg.pinv(C) @ b       # the best possible attempt
print(C @ best - b)                 # [-2. -2. -2.]  misses: no solution

b = jnp.array([1., 1., -2.])        # climbs that cancel
print(C @ (jnp.linalg.pinv(C) @ b)) # [ 1.  1. -2.]  reached exactly

import tensorflow as tf

A = tf.constant([[1., 2.], [3., 4.], [5., 6.]])
x = tf.constant([[7.], [8.]])
print(A @ x)                        # [[23.] [53.] [83.]]
print(7 * A[:, 0] + 8 * A[:, 1])    # the same: Ax mixes the columns

C = tf.constant([[1., 0., -1.], [-1., 1., 0.], [0., -1., 1.]])
print(C @ tf.ones([3, 1]))          # zeros: the invisible direction
print(tf.linalg.det(C))             # 0.0  singular

b = tf.constant([[1.], [2.], [3.]])  # climbs that don't cancel
best = tf.linalg.pinv(C) @ b         # the best possible attempt
print(C @ best - b)                  # [[-2.] [-2.] [-2.]]  no solution

b = tf.constant([[1.], [1.], [-2.]])  # climbs that cancel
print(C @ (tf.linalg.pinv(C) @ b))    # [[ 1.] [ 1.] [-2.]]  reached

Where this is going

You now know what a matrix does: it mixes its columns, it moves the whole space, and sometimes it flattens the space and cannot be undone. What you do not have yet is a procedure. Given a big system $Ax = b$, how do you decide which ending you are in, and find $x$ fast? The answer is elimination, the algorithm inside every solver, and it is the next lecture.