The Four Subspaces & the Rank Theorem

The idea in one sentence

Every matrix carries four subspaces, two in the space it reads from and two in the space it writes to, and a single run of elimination hands you a basis for all four.

One matrix, carried the whole way

Here is the matrix that will show you all four:

It is the doubled-columns matrix from Lecture 7.5 with one row added, and it earns its place by repeating itself in both directions. Read it down the columns: column 3 is twice column 1, and column 4 is twice column 2. Read it across the rows: row 3 is row 1 plus row 2. Nothing below uses any other matrix.

$A$ has 3 rows and 4 columns. It takes a list of four numbers and returns three, so it maps $\R^4$ to $\R^3$. Four subspaces hang off it, two on each side.

On the input side, in $\R^4$, the row space $C(A\T)$ is every combination of the rows of $A$. The transpose sits in the name because the rows of $A$ are the columns of $A\T$, and $C(\cdot)$ always means the space some columns span. Next to it lives the null space $N(A)$, every $x$ with $Ax = 0$. These are the directions $A$ is blind to, and the ring road machine from Lecture 2 was blind to $(1, 1, 1)$.

On the output side, in $\R^3$, the column space $C(A)$ is every combination of the columns, which by the column reading of $Ax$ is exactly the set of outputs $A$ can produce. Next to it lives the left null space $N(A\T)$, every $y$ with $A\T y = 0$. Transpose that equation and it reads $y\T A = 0$: the vector $y$ sits on the left of $A$ and kills it. In words, $y$ holds the coefficients of a combination of the rows of $A$ that comes out zero, and this matrix has one in plain sight, because row 1 plus row 2 minus row 3 is the zero row.

Four definitions, four sets of vectors to hunt down. One elimination finds them all.

Elimination, run once, paid four times

Reduce $A$ to its row reduced form $R$, and carry the identity along so the record of what you did comes out on the right. Subtract 3 times row 1 from row 2 and 4 times row 1 from row 3, subtract the new row 2 from the new row 3, halve row 2, then subtract twice row 2 from row 1:

Two pivots, so the rank is $r = 2$. Columns 1 and 2 are the pivot columns, columns 3 and 4 are free. The third row of $A$ was never new information, and $R$ says so with a row of zeros.

Where the four bases come from

Each subspace takes its basis from a different place, and three of the four places are already on the page.

Row space: the pivot rows of $R$

Every elimination step replaces a row by a combination of rows, so each row of $R$ is a combination of rows of $A$. Every step is reversible, since $E$ has an inverse, so each row of $A$ is a combination of rows of $R$. Two matrices whose rows are combinations of each other span the same space. So $A$ and $R$ have the same row space, and the zero row of $R$ contributes nothing. The $r$ pivot rows are a basis:

Both directions of that argument are on the page already. $E$ says how to build $R$‘s rows out of $A$‘s, and $E^{-1}$, which for this matrix comes out in whole numbers, says how to build $A$‘s rows back out of $R$‘s.

Column space: the pivot columns of $A$ itself

Here the pattern breaks, and this is the one place readers get burned. Elimination does not preserve the column space. What it preserves is which columns depend on which, because $Ax = 0$ and $Rx = 0$ have the same solutions, and a dependence among columns is exactly a solution of $Ax = 0$. Column 3 of $R$ is twice column 1 of $R$, so column 3 of $A$ is twice column 1 of $A$. The recipe carries over. The vectors do not.

So take the columns in the pivot positions from the original matrix:

The columns of $R$ in those same positions are $(1, 0, 0)$ and $(0, 1, 0)$, and they span a different plane in $\R^3$. Figure 4 shows how different.

Null space: the special solutions

Lecture 7.5 built these. Set one free variable to 1, the rest to 0, and solve $Rx = 0$ upward. With $x_3 = 1$ the first row gives $x_1 + 2 = 0$, so $x_1 = -2$. With $x_4 = 1$ the second row gives $x_2 + 2 = 0$, so $x_2 = -2$:

One special solution per free column, so the count is forced to be $n - r$.

Left null space: the last $m - r$ rows of $E$

This is the one people never remember, and it is the easiest of the four once you know where to look. $EA = R$ says that row $i$ of $E$ times $A$ gives row $i$ of $R$. The bottom $m - r$ rows of $R$ are zero. So the bottom $m - r$ rows of $E$, applied to $A$, produce zero rows. Those rows are precisely the combinations of $A$‘s rows that vanish, which is what the left null space is.

Row 3 of $E$ is $(-1, -1, 1)$, and sure enough $-\text{row}_1 - \text{row}_2 + \text{row}_3 = 0$:

The big picture

Now put the four in one frame. On the input side, $\R^4$ holds the row space of dimension $r = 2$ and the null space of dimension $n - r = 2$, and $2 + 2 = 4$. On the output side, $\R^3$ holds the column space of dimension $r = 2$ and the left null space of dimension $m - r = 1$, and $2 + 1 = 3$. The counts close on both sides, which is the first sign that nothing has been left out.

Every vector $x$ in $\R^4$ splits into a row space piece $x_r$ and a null space piece $x_n$:

The null piece dies on the way through. Whatever $A$ produces, it produces from the row space alone. Take $x = (1, 1, 1, 1)$. Its two pieces are

and you can check by hand that $x_r$ is $\tfrac{3}{5}$ of the first row space basis vector plus $\tfrac{3}{5}$ of the second, that $x_n$ is $-\tfrac{1}{5}s_1 - \tfrac{1}{5}s_2$, and that $Ax = Ax_r = (9, 33, 42)$ while $Ax_n = 0$.

Every later chapter of this course points back at this frame. Projections land inside the column space. Least squares takes a $b$ that is out of reach and moves it onto that space. Eigenvectors are found by asking which null space they sit in. Learn the four rooms once and the rest of the course has an address system.

Every attention head in a transformer carries these four spaces, and the numbers are lopsided enough to change how you picture a layer.

Why the two dimensions match

Look at the two dimension labels in Figure 7 again. Both say 2, and that is no accident of this matrix. For every matrix of every shape, the row space and the column space have the same dimension, and that shared number is what the word rank means.

It sounds like it should be hard to prove. A $3 \times 4$ matrix has rows of length 4 and columns of length 3, so the two spaces do not even live in the same place. The proof takes two paragraphs and one factorization.

Start with the column space, since a basis for it is already sitting inside $A$. Call its dimension $k$. Put a basis for $C(A)$ into the columns of a matrix $B$. Every column of $A$ is some combination of those $k$ vectors, and if you write the coefficients for column $j$ down the $j$-th column of a new matrix $C$, the column reading of matrix multiplication says exactly $A = BC$. For our matrix, $k = 2$ and both factors are already computed:

$B$ is $3 \times 2$ and holds the pivot columns of $A$. $C$ is $2 \times 4$ and holds the pivot rows of $R$, which is the same thing as the recipe for rebuilding every column. Multiply them and $A$ comes back.

Now read the same product the other way. The row reading of $BC$ says row $i$ of $A$ is a combination of the rows of $C$, with row $i$ of $B$ supplying the coefficients. Row 3 of $B$ is $(4, 10)$, so

Every row of $A$ is a combination of the $k$ rows of $C$. So the row space sits inside the span of $k$ vectors, and such a space has dimension at most $k$:

That is one inequality. For the other, run the identical argument on $A\T$. A basis for its column space sits in its first two columns, so

and the same reading gives $\dim C(A) \le \dim C(A\T)$, because the row space of $A\T$ is the column space of $A$ and the column space of $A\T$ is the row space of $A$. Two inequalities pointing at each other leave one possibility:

Checking it in code

Everything above is small enough to check by hand and small enough to check in four lines. The first block confirms the four bases. Watch $E$ do two jobs at once: it certifies the reduction, and its last row is the left null space.

import numpy as np

A = np.array([[1., 2., 2., 4.],
              [3., 8., 6., 16.],
              [4., 10., 8., 20.]])
R = np.array([[1., 0., 2., 0.],
              [0., 1., 0., 2.],
              [0., 0., 0., 0.]])
E = np.array([[4., -1., 0.],
              [-1.5, 0.5, 0.],
              [-1., -1., 1.]])

print(np.linalg.matrix_rank(A))     # 2
print(np.allclose(E @ A, R))        # True   the record checks out

s1 = np.array([-2., 0., 1., 0.])    # special solutions -> null space
s2 = np.array([0., -2., 0., 1.])
print(A @ s1, A @ s2)               # [0. 0. 0.] [0. 0. 0.]

y = np.array([-1., -1., 1.])        # last row of E -> left null space
print(A.T @ y)                      # [0. 0. 0. 0.]

import torch

A = torch.tensor([[1., 2., 2., 4.],
                  [3., 8., 6., 16.],
                  [4., 10., 8., 20.]])
R = torch.tensor([[1., 0., 2., 0.],
                  [0., 1., 0., 2.],
                  [0., 0., 0., 0.]])
E = torch.tensor([[4., -1., 0.],
                  [-1.5, 0.5, 0.],
                  [-1., -1., 1.]])

print(torch.linalg.matrix_rank(A))  # tensor(2)
print(torch.allclose(E @ A, R))     # True   the record checks out

s1 = torch.tensor([-2., 0., 1., 0.])   # special solutions -> null space
s2 = torch.tensor([0., -2., 0., 1.])
print(A @ s1, A @ s2)               # tensor([0., 0., 0.]) tensor([0., 0., 0.])

y = torch.tensor([-1., -1., 1.])    # last row of E -> left null space
print(A.T @ y)                      # tensor([0., 0., 0., 0.])

import jax.numpy as jnp

A = jnp.array([[1., 2., 2., 4.],
               [3., 8., 6., 16.],
               [4., 10., 8., 20.]])
R = jnp.array([[1., 0., 2., 0.],
               [0., 1., 0., 2.],
               [0., 0., 0., 0.]])
E = jnp.array([[4., -1., 0.],
               [-1.5, 0.5, 0.],
               [-1., -1., 1.]])

print(jnp.linalg.matrix_rank(A))    # 2
print(jnp.allclose(E @ A, R))       # True   the record checks out

s1 = jnp.array([-2., 0., 1., 0.])   # special solutions -> null space
s2 = jnp.array([0., -2., 0., 1.])
print(A @ s1, A @ s2)               # [0. 0. 0.] [0. 0. 0.]

y = jnp.array([-1., -1., 1.])       # last row of E -> left null space
print(A.T @ y)                      # [0. 0. 0. 0.]

import tensorflow as tf

A = tf.constant([[1., 2., 2., 4.],
                 [3., 8., 6., 16.],
                 [4., 10., 8., 20.]])
R = tf.constant([[1., 0., 2., 0.],
                 [0., 1., 0., 2.],
                 [0., 0., 0., 0.]])
E = tf.constant([[4., -1., 0.],
                 [-1.5, 0.5, 0.],
                 [-1., -1., 1.]])

print(tf.linalg.matrix_rank(A))     # tf.Tensor(2, ...)
print(tf.reduce_all(tf.abs(E @ A - R) < 1e-6))   # True

s1 = tf.constant([[-2.], [0.], [1.], [0.]])   # special solutions
s2 = tf.constant([[0.], [-2.], [0.], [1.]])
print(A @ s1, A @ s2)               # [[0.] [0.] [0.]]  [[0.] [0.] [0.]]

y = tf.constant([[-1.], [-1.], [1.]])   # last row of E
print(tf.transpose(A) @ y)          # [[0.] [0.] [0.] [0.]]

The second block is the rank theorem itself. There is almost nothing to it once $A = BC$ is on the table, because the shapes of $B$ and $C$ are the whole argument.

import numpy as np

A = np.array([[1., 2., 2., 4.], [3., 8., 6., 16.], [4., 10., 8., 20.]])

B = A[:, :2]                                        # pivot columns of A
C = np.array([[1., 0., 2., 0.], [0., 1., 0., 2.]])  # pivot rows of R
print(np.allclose(B @ C, A))                        # True
print(B.shape, C.shape)                             # (3, 2) (2, 4)

print(2 * B[:, 0] + 0 * B[:, 1])   # [2. 6. 8.]        = column 3 of A
print(4 * C[0] + 10 * C[1])        # [ 4. 10.  8. 20.] = row 3 of A

print(np.linalg.matrix_rank(A), np.linalg.matrix_rank(A.T))   # 2 2
print(A.shape[1] - np.linalg.matrix_rank(A))        # 2 = dim N(A)
print(A.shape[0] - np.linalg.matrix_rank(A))        # 1 = dim N(A.T)

import torch

A = torch.tensor([[1., 2., 2., 4.], [3., 8., 6., 16.], [4., 10., 8., 20.]])

B = A[:, :2]                                            # pivot columns of A
C = torch.tensor([[1., 0., 2., 0.], [0., 1., 0., 2.]])  # pivot rows of R
print(torch.allclose(B @ C, A))                         # True
print(B.shape, C.shape)   # torch.Size([3, 2]) torch.Size([2, 4])

print(2 * B[:, 0] + 0 * B[:, 1])   # tensor([2., 6., 8.])
print(4 * C[0] + 10 * C[1])        # tensor([ 4., 10.,  8., 20.])

print(torch.linalg.matrix_rank(A), torch.linalg.matrix_rank(A.T))  # 2 2
print(A.shape[1] - torch.linalg.matrix_rank(A))         # 2 = dim N(A)
print(A.shape[0] - torch.linalg.matrix_rank(A))         # 1 = dim N(A.T)

import jax.numpy as jnp

A = jnp.array([[1., 2., 2., 4.], [3., 8., 6., 16.], [4., 10., 8., 20.]])

B = A[:, :2]                                         # pivot columns of A
C = jnp.array([[1., 0., 2., 0.], [0., 1., 0., 2.]])  # pivot rows of R
print(jnp.allclose(B @ C, A))                        # True
print(B.shape, C.shape)                              # (3, 2) (2, 4)

print(2 * B[:, 0] + 0 * B[:, 1])   # [2. 6. 8.]
print(4 * C[0] + 10 * C[1])        # [ 4. 10.  8. 20.]

print(jnp.linalg.matrix_rank(A), jnp.linalg.matrix_rank(A.T))   # 2 2
print(A.shape[1] - jnp.linalg.matrix_rank(A))        # 2 = dim N(A)
print(A.shape[0] - jnp.linalg.matrix_rank(A))        # 1 = dim N(A.T)

import tensorflow as tf

A = tf.constant([[1., 2., 2., 4.], [3., 8., 6., 16.], [4., 10., 8., 20.]])

B = A[:, :2]                                            # pivot columns of A
C = tf.constant([[1., 0., 2., 0.], [0., 1., 0., 2.]])   # pivot rows of R
print(tf.reduce_all(tf.abs(B @ C - A) < 1e-6))          # True
print(B.shape, C.shape)                                 # (3, 2) (2, 4)

print(2 * B[:, 0] + 0 * B[:, 1])   # [2. 6. 8.]
print(4 * C[0] + 10 * C[1])        # [ 4. 10.  8. 20.]

print(tf.linalg.matrix_rank(A), tf.linalg.matrix_rank(tf.transpose(A)))  # 2 2
print(A.shape[1] - tf.linalg.matrix_rank(A))            # 2 = dim N(A)
print(A.shape[0] - tf.linalg.matrix_rank(A))            # 1 = dim N(A.T)

Where this is going

You now have the four subspaces, their dimensions, and a basis for each, all out of a single elimination. What you do not have is the geometry. Figure 7 drew the row space above the null space as though they were two halves of $\R^4$, and it split $x$ into two pieces as though there were only one way to do it. Both of those are true, and both are true for the same reason: the row space and the null space meet at a right angle, and so do the column space and the left null space. That right angle is the next chapter, and it turns this filing cabinet into the fundamental theorem of linear algebra.