Vectors and Linear Combinations

A list of numbers and an arrow with a length and a direction are the same object. Which reading is useful depends on the question. A cost dotted with a plan is money, a cost dotted with a direction is a rate.

Two operations and one product. Everything later in this subject, a constraint, a cost, a basis, a pivot, is assembled from them, so the arithmetic here is worth being fluent in rather than merely able to do.

Each row below is one object read twice. Fill the missing cell before reading on: the point is that the translation is mechanical, and that fluency in it is what later algorithms assume.

A numerical check worth doing once. Take $u=(3,4)$ and $v=(4,-3)$.

Now verify geometrically: $u$ has length $5$ and so does $v$; the angle between them comes from $\cos \theta =\frac{u^{\mathsf{T}}v}{\Vert u\Vert \Vert v\Vert }=\frac{0}{25}=0$, so $\theta =90°$. Two unrelated-looking calculations, one answer.

Where the translation earns its keep. A constraint $2x_1+3x_2\le 12$ is a dot product $c^{\mathsf{T}}x$ with $c=(2,3)$. Testing the point $(3,1)$:

so the point satisfies it. Geometrically, $c$ points perpendicular to the boundary line and the test asks which side $(3,1)$ falls on. The simplex method performs exactly this test thousands of times, and reads the same number as "how much does moving in this direction improve the objective".

One number, three questions. For a direction $d$ and coefficient vector $c$, the single quantity $c^{\mathsf{T}}d$ answers: does this direction improve the objective (sign), by how much per unit travelled (magnitude), and is this direction along a contour (zero). No separate machinery is introduced later for any of the three.

The canonical statement above fixes the objects and the two operations. Three things follow from it that the statement does not say, and each is used later without comment.

The zero vector is a linear combination of every set. Take every coefficient zero in ${\lambda }_1{v}_1+\cdots +{\lambda }_k{v}_k$. Nothing in the definition excludes it, and the independence test depends on this case being available: independence asks whether the zero vector has any other representation.

The dot product is the only place two vectors become a scalar. Addition and scaling return vectors; $x^{\mathsf{T}}y$ returns a number. Every later quantity that is a single figure — a cost, an objective value, the left side of a constraint — is a dot product somewhere, which is why a constraint row applied to a candidate point produces something comparable against $b_i$.

The norm is defined from the dot product, not beside it. $\Vert x\Vert =\sqrt{x^{\mathsf{T}}x}$ means length is not a separate notion to be supplied: it is what the dot product of a vector with itself measures. Orthogonality is the same economy in reverse: $x^{\mathsf{T}}y=0$ is a statement about angle written without mentioning angles, which is what makes it computable in any dimension.

A note on shape. $x^{\mathsf{T}}y$ is written with a transpose because the two operands are columns and the product of a $1\times n$ with an $n\times 1$ is $1\times 1$. The transpose is doing real work here, and the same symbols without it, $x{y}^{\mathsf{T}}$, produce an $n\times n$ matrix instead.

Take $x=(2,-1,3)$ and $y=(4,5,1)$ in ${\mathbb{R}}^3$.

Addition and scaling.

Each is another vector in ${\mathbb{R}}^3$.

A linear combination. With coefficients $2$ and $-1$:

The dot product.

One number, not a vector. Its sign says the two vectors broadly agree in direction; had it come out zero they would be perpendicular.

The norm.

A length, so it is never negative, and it says nothing about which way $x$ points.

Problem. A workshop makes three products. The production plan is $x=(10,4,6)$ units, and the unit profits are $c=(3,7,2)$. A proposed change to the plan is the direction $d=(1,-1,0)$.

Find the total profit, decide whether moving along $d$ improves it, and say what $c^{\mathsf{T}}d=0$ would have meant.

Goal. Read each dot product as the quantity it reports.

Relevant principle. A dot product measures how much of one vector lies along another. With a cost vector on one side, it measures money.

Step 1: total profit.

Reason: each product's profit times its quantity, summed, which is exactly the dot product of the two lists.

Step 2: the effect of the change.

Reason: $d$ says make one more of the first product and one fewer of the second. The profit changes by the dot product of the cost with that direction.

Step 3: read the sign. Negative, so the change loses £4 per unit step. Moving along $d$ makes the plan worse.

Step 4: what zero would have meant. $c^{\mathsf{T}}d=0$ would say the direction is orthogonal to the cost: the plan changes but the profit does not. That is a flat direction, and it is precisely how a linear program comes to have several optimal plans with the same value.

Result. Profit $70$; the direction $d$ is worsening at $-4$ per step.

Check. Does the sign make sense without the arithmetic? $d$ trades a unit of the £7 product for a unit of the £3 product, so it should lose £4. It does.

Interpretation. Nothing here required a diagram. The geometry, improving, worsening, flat, arrived entirely through one scalar, which is why the dot product does so much work later.

Adding vectors of different dimensions. $(1,2)+(1,2,3)$ is undefined. The shorter vector is not padded with a zero; the operation simply does not apply.

A dot product that returns a vector. $x^{\mathsf{T}}y$ is a single number. Multiplying entry by entry to get $(x_1{y}_1,x_2{y}_2,\dots )$ is a different operation and is not what any later formula means by the notation.

Concluding that a zero dot product means a zero vector. $(1,0{)}^{\mathsf{T}}(0,1)=0$ with neither vector zero. Orthogonality is a relationship, not a statement that something vanished.

Reading a larger norm as a better direction. The norm measures length only. Scaling a worsening direction makes it longer and no less worsening.

Treating the zero vector as outside the space. It is a perfectly ordinary member, and it is a linear combination of every set, which is the fact the definition of independence is built on.

Why the confusion is common. Both are written as a product of two vectors, and a calculator will happily do either. The notation $x^{\mathsf{T}}y$ is what distinguishes them: the transpose is doing real work, turning a column into a row so the multiplication collapses to a single number.

The test. Ask what the answer is supposed to be. A cost applied to a plan is money, one number. A constraint applied to a point is a quantity to compare with a bound, one number. If the answer should be a single quantity, it is a dot product.

Where the entrywise product does appear. Rescaling: multiplying each entry of a plan by a per-unit factor. That is a legitimate operation and it is never written $x^{\mathsf{T}}y$.

1: fully structured. Let $u=(1,3)$ and $v=(4,-2)$.

(a) Compute $u+v$ and $2u$. (b) Compute $u^{\mathsf{T}}v$. (c) Are $u$ and $v$ orthogonal?

Check: (a) $(5,1)$ and $(2,6)$; (b) $(1)(4)+(3)(-2)=4-6=-2$; (c) no, orthogonality requires the product to be zero, and this is $-2$.

2: partly structured. A plan is $x=(5,5)$ with cost vector $c=(2,6)$, and a proposed direction is $d=(3,-1)$.

(a) What is the current cost? (b) Does $d$ increase or decrease it, and by how much per step? (c) Find a nonzero direction that leaves the cost unchanged.

Check: (a) $10+30=40$; (b) $c^{\mathsf{T}}d=6-6=0$, so the cost is unchanged, $d$ is already a flat direction; (c) any nonzero multiple of $(3,-1)$, since scaling a direction orthogonal to $c$ keeps it orthogonal.

3: unstructured. A logistics planner writes: "Our two route vectors are $r_1=(6,2)$ and $r_2=(-1,3)$. Their dot product is zero, so neither route carries any load and both can be dropped."

Assess the reasoning, and say what the zero actually tells the planner.

Check: the conclusion is wrong and the arithmetic is right. $r_1^{\mathsf{T}}{r}_2=-6+6=0$, but a zero dot product means the two vectors are perpendicular, not that either is the zero vector, and here both are plainly nonzero, with lengths $\sqrt{40}$ and $\sqrt{10}$. What the planner has discovered is that the two routes are independent in direction: neither is a multiple of the other, and changes along one do not affect the other's contribution. That is a useful fact about the pair, and it is the opposite of grounds for dropping them.

A vector is a list of $n$ numbers, read as data or as a direction with a length.

Two operations. $(x+y{)}_i=x_i+y_i$ and $(\lambda x{)}_i=\lambda x_i$, both entrywise, both requiring matching dimensions.

A linear combination. ${\lambda }_1{v}_1+\cdots +{\lambda }_k{v}_k$. Every coefficient zero gives the zero vector, which is therefore a combination of any set.

The dot product. $x^{\mathsf{T}}y=\sum _i{x}_i{y}_i$. One scalar, never a vector.

What it measures. How much of one vector lies along the other. A cost dotted with a plan is money; a cost dotted with a direction is the rate of change along it.

Orthogonality. $x^{\mathsf{T}}y=0$, which two nonzero vectors can satisfy. A flat direction for an objective is exactly one orthogonal to it.

The norm. $\Vert x\Vert =\sqrt{x^{\mathsf{T}}x}$. A length, never negative, silent about direction.

The recurring error. Treating the dot product as a vector.