Lecture 01: Vectors & Dimensions

The idea in one sentence

A vector is a list of numbers you can draw as an arrow, and almost everything in this course comes from one move: scaling arrows and adding them together.

What is a dimension?

Start with a dot. A dot has no freedom. There is nowhere to go. That is zero dimensions.

Let the dot slide along a track. Now it has one kind of freedom: forward or back. One dimension. Let it leave the track and roam a tabletop, and it has two independent freedoms. Two dimensions. Lift it off the table: three.

Fig. 1

A dimension is a direction you can move in. Each shape is the one before it swept along a new direction, drawn here in blue. Count the independent directions and you have the dimension.

That is the whole definition. A dataset with 784 measurements per sample lives in a 784-dimensional space for the same reason a point on a table lives in two: you need 784 numbers to say where you are.

Vectors, and the one move that matters

A vector in the plane is two numbers, $v = (1, 1)$ , drawn as an arrow from the origin. You can do two things with it:

Scale it. $3v$ points the same way and goes three times as far.
Add another one. $v + w$ means: walk along $v$ , then walk along $w$ .

Do both at once and you get a linear combination, the most important operation in the course:

3v + 5w \;=\; 3\begin{bmatrix} 1 \\ 1 \end{bmatrix} + 5\begin{bmatrix} 2 \\ 3 \end{bmatrix} \;=\; \begin{bmatrix} 3 + 10 \\ 3 + 15 \end{bmatrix} \;=\; \begin{bmatrix} 13 \\ 18 \end{bmatrix}.

Fig. 2

A linear combination is a walk: two steps of $v$ in orange, then one step of $w$ in green. The blue path is the walk and the blue dot is where it ends. Every place you can end up is the span.

Now the question that unlocks everything. Let the scalars $c$ and $d$ range over all numbers. What can $c\,v + d\,w$ reach? For the two arrows above, the answer is every point of the plane, and the set of reachable points is called the span. Asking “is this point in the span?” is the same as asking “does this system of equations have a solution?” That link between a picture and a computation is what linear algebra is.

Try it. Move c and d

c = 1.0d = 1.0

v = (1, 1), w = (2, 3)
cv + dw = (3.0, 4.0)

Orange walks c steps of v, green walks d steps of w, blue is where you land. Every point the blue dot can reach is the span. Try to find a point it cannot reach.

Reaching can fail. The combinations of $(1, 1, 1)$ and $(2, 3, 4)$ fill a flat plane inside three-dimensional space, and the point $(1, 0, 0)$ is off that plane. So the system

c + 2d = 1, \qquad c + 3d = 0, \qquad c + 4d = 0

has no solution, and the picture tells you how badly it fails.

Fig. 3

The plane holds every point the two vectors can reach, drawn almost edge on. The target $b$ in orange floats above it. The green dot is the closest the plane can get, and the dashed drop is what is missing. Its squared length is $1/6$ , the number the code below prints.

python

import numpy as np

v = np.array([1.0, 1.0])
w = np.array([2.0, 3.0])
print(3 * v + 5 * w)                    # [13. 18.]

# Is (1, 0, 0) in the span of (1,1,1) and (2,3,4)?
# Ask for the best c, d and measure what is left over.
A = np.array([[1.0, 2.0], [1.0, 3.0], [1.0, 4.0]])
b = np.array([1.0, 0.0, 0.0])
(c, d), residual, *_ = np.linalg.lstsq(A, b)
print(np.round([c, d], 3))              # [ 1.833 -0.5  ]  the best attempt
print(residual)                         # [0.16666667]  above zero, so b is out of reach

python

import torch

v = torch.tensor([1.0, 1.0])
w = torch.tensor([2.0, 3.0])
print(3 * v + 5 * w)                    # tensor([13., 18.])

# Is (1, 0, 0) in the span of (1,1,1) and (2,3,4)?
A = torch.tensor([[1.0, 2.0], [1.0, 3.0], [1.0, 4.0]])
b = torch.tensor([[1.0], [0.0], [0.0]])
best = torch.linalg.lstsq(A, b).solution
leftover = ((A @ best - b) ** 2).sum()
print(best.squeeze())                   # 1.833, -0.5  the best attempt
print(leftover)                         # 0.16666667  above zero, so b is out of reach

python

import jax.numpy as jnp

v = jnp.array([1.0, 1.0])
w = jnp.array([2.0, 3.0])
print(3 * v + 5 * w)                    # [13. 18.]

# Is (1, 0, 0) in the span of (1,1,1) and (2,3,4)?
A = jnp.array([[1.0, 2.0], [1.0, 3.0], [1.0, 4.0]])
b = jnp.array([1.0, 0.0, 0.0])
best, residual, *_ = jnp.linalg.lstsq(A, b)
print(best)                             # 1.833, -0.5  the best attempt
print(residual)                         # 0.16666667  above zero, so b is out of reach

python

import tensorflow as tf

v = tf.constant([1.0, 1.0])
w = tf.constant([2.0, 3.0])
print(3 * v + 5 * w)                    # [13. 18.]

# Is (1, 0, 0) in the span of (1,1,1) and (2,3,4)?
A = tf.constant([[1.0, 2.0], [1.0, 3.0], [1.0, 4.0]])
b = tf.constant([[1.0], [0.0], [0.0]])
best = tf.linalg.lstsq(A, b)
leftover = tf.reduce_sum((A @ best - b) ** 2)
print(tf.squeeze(best))                 # 1.833, -0.5  the best attempt
print(leftover)                         # 0.16666667  above zero, so b is out of reach

The dot product

So far vectors have been things you walk along. They can also measure each other. The dot product multiplies matching components and adds the results:

v \cdot w \;=\; v_1 w_1 + v_2 w_2 .

The same number has a geometric life. It equals the lengths of the two vectors times the cosine of the angle between them:

v \cdot w \;=\; \norm{v}\,\norm{w}\,\cos\theta .

Picture shining a flashlight straight down onto the ground. The dot product measures the shadow: how much of one vector lies along the other.

Fig. 4

The dot product is a shadow. The orange vector $v$ drops straight down onto the green vector $w$ , and the blue segment is what lands. When $\theta$ is a right angle the shadow has length zero, and the vectors share nothing.

When the dot product is zero, the vectors are perpendicular. Later this one number gives us projections and least squares. It is also what a transformer computes when it compares a query with a key.

Search and attention both run on that one number. Score every candidate against the query, then sort.

Fig. 5

One query and three candidates, drawn with the same shadow construction as Figure 4. Each score is the dot product $q \cdot x$ , which is $\norm{q}$ times the shadow the candidate casts on the query, so $4 \times 3 = 12$ , $4 \times 2 = 8$ and $4 \times 1 = 4$ . Doc B is the longest vector here and still loses: what counts is how much of it lies along the query. Every ranked search result and every attention score is this one number, computed against millions of candidates at once.

Where this is going

Applied math has two halves. First you model: turn the problem into equations. Then you compute: solve them. Linear algebra writes the modeling half in one line:

A\,x = b .

A matrix $A$ holds your data. The vector $x$ is unknown. The vector $b$ is the target. The question is the one we just met: is $b$ in the span of the columns of $A$ ? The rest of the course builds the machinery that answers it quickly, in any dimension.