Orthogonality and Projection

Finding coordinates in an arbitrary basis means solving a linear system: every coordinate depends on every other. In an orthonormal basis it means taking one inner product per coordinate, because the directions do not interfere.

That independence follows from orthogonality, and it answers a second question at the same time. Given a point off a subspace, the closest point in it is found by keeping the part that lies along the subspace and discarding the part perpendicular to it, and the discarded part being perpendicular is exactly what makes the remainder closest.

Least squares is that construction applied to a system with no solution. When $Ax=b$ cannot be solved, replace $b$ by the nearest point of the column space. The normal equations that result are not a trick; they say the residual is perpendicular to every column.

Positive definiteness is what makes a norm. Symmetry and linearity alone would permit , and $\sqrt{⟨v,v⟩}$ would not exist. Requiring for $v\ne 0$ is what lets an inner product define a length, and with it distance, angle and the whole geometry.

Orthogonality is relative to the inner product. The vectors $(1,1)$ and $(1,-1)$ are orthogonal under the dot product. Under $⟨u,v⟩=u_1{v}_1+2u_2{v}_2$, also a valid inner product, their pairing is $1-2=-1\ne 0$, so they are not. The word means nothing until the inner product is named, which is why a space with a chosen inner product is a single object, an inner product space, rather than a space and a separate decoration.

Why an orthogonal set is independent. Suppose $\sum _j{c}_j{u}_j=0$ with the $u_j$ pairwise orthogonal and nonzero. Take the inner product of both sides with $u_k$: every cross term vanishes, leaving $c_k⟨u_k,u_k⟩=0$, and positive definiteness forces $c_k=0$. So orthogonality is a strictly stronger condition than independence, and an orthogonal spanning set is automatically a basis.

Why the coefficients are inner products. For an orthonormal basis, apply $⟨\cdot ,e_k⟩$ to $v=\sum _j{c}_j{e}_j$. All terms but one vanish and $⟨e_k,e_k⟩=1$, leaving $c_k=⟨v,e_k⟩$. The same computation with a merely orthogonal basis leaves $⟨u_k,u_k⟩$ behind, which is where the denominator in the projection formula comes from. It is not a normalisation added by hand.

The denominator is the commonest omission. Writing $\sum _j⟨b,u_j⟩u_j$ for an orthogonal but not orthonormal basis scales each component by $\Vert u_j{\Vert }^2$, giving a vector in the right subspace with the wrong length. The error is invisible unless the residual is checked for orthogonality.

Projection is idempotent. ${\mathrm{proj}}_W$ is a linear map with ${\mathrm{proj}}_W\circ {\mathrm{proj}}_W={\mathrm{proj}}_W$: a point already in $W$ is its own closest point. Its kernel is $W^{⟂}$, the set of vectors orthogonal to all of $W$, and its image is $W$, so by rank–nullity $\dim W+\dim W^{⟂}=n$.

The same definition, applied wherever an inner product exists.

The standard basis of ${\mathbb{R}}^3$. $e_1,e_2,e_3$ pair to zero with each other and to one with themselves, so the set is orthonormal. This is why coordinates in the standard basis are so easy to read: each is already an inner product, $x_j=⟨x,e_j⟩$.

A rotated pair in ${\mathbb{R}}^2$. $(1,1)$ and $(1,-1)$ have $⟨u,v⟩=1-1=0$, so they are orthogonal, but $\Vert u{\Vert }^2=\Vert v{\Vert }^2=2$, so the set is orthogonal, not orthonormal. Dividing each by $\sqrt{2}$ gives

which pair to zero and have norm 1. This is the distinction that decides whether the denominators in the projection formula may be dropped.

Polynomials on $[0,1]$, with $⟨f,g⟩={\int }_0^{1}fgdx$. The basis $\{1,x\}$ is not orthogonal:

One Gram–Schmidt step replaces $x$ by $x-\frac{1}{2}$, and $⟨1,x-\frac{1}{2}⟩=\frac{1}{2}-\frac{1}{2}=0$. Since $\Vert x-\frac{1}{2}{\Vert }^2={\int }_0^1(x-\frac{1}{2}{)}^{2}dx=\frac{1}{12}$, normalising gives $\sqrt{12}(x-\frac{1}{2})$. These are the first two Legendre polynomials for this interval.

Two functions can look entirely unalike and still fail to be orthogonal; the integral decides, not the appearance.

Trigonometric functions on $[0,2\pi ]$, with $⟨f,g⟩={\int }_0^{2\pi }fgdx$.

while $⟨\sin ,\sin ⟩=⟨\cos ,\cos ⟩=\pi$. So $\{1,\sin x,\cos x,\sin 2x,\cos 2x,\dots \}$ is an orthogonal set. The basis Fourier series are written in. Projecting a function onto its span is exactly how Fourier coefficients are computed, and the $\pi$ in the denominators of the usual formulas is the $⟨u_j,u_j⟩$ of the projection formula, not a convention.

Symmetric matrices, with $⟨A,B⟩=\mathrm{tr}(A^{\mathsf{T}}B)$. This inner product is the sum of entrywise products. The basis from the basis-and-dimension unit,

is pairwise orthogonal, with squared norms $1$, $1$ and $2$. Again orthogonal but not orthonormal: the third must be divided by $\sqrt{2}$, because its single off-diagonal value appears twice.

Nothing about the objects, lists of numbers, polynomials, functions, matrices. What they share is an inner product satisfying the three axioms, and every construction in this unit is written in those terms alone. What differs is which sets count as orthogonal, and that depends entirely on the inner product chosen.

Gram–Schmidt. Given a basis $v_1,\dots ,v_k$ of a subspace:

1. $u_1=v_1$.
2. For $j=2,\dots ,k$: subtract from $v_j$ its projection onto each earlier $u_i$,

1. Optionally divide each $u_j$ by $\Vert u_j\Vert$ to normalise.

At each step, check $⟨u_j,u_i⟩=0$ for every before continuing. An error at step $j$ contaminates every later step, and the check costs one inner product per pair.

Projecting onto a subspace $W$. With an orthogonal basis $u_1,\dots ,u_k$ of $W$:

If the basis is orthonormal the denominators are 1 and may be dropped, but only then.

Verification, in two independent ways.

The first is the definitive one: the projection is characterised by having an orthogonal residual, so a residual that fails the test means the answer is not the projection.

Least squares for $Ax\approx b$.

1. Form $A^{\mathsf{T}}A$ and $A^{\mathsf{T}}b$.
2. Solve $A^{\mathsf{T}}A\hat{x}=A^{\mathsf{T}}b$.
3. The fitted vector is $A\hat{x}={\mathrm{proj}}_{\mathrm{col}A}(b)$, and the residual is $e=b-A\hat{x}$.
4. Check $A^{\mathsf{T}}e=0$. The residual is orthogonal to every column.

For fitting $y=a+bx$ to points $(x_i,y_i)$, the columns of $A$ are a column of ones and the column of $x_i$, so the normal equations read

The two rows say $\sum _i{e}_i=0$ and $\sum _i{x}_i{e}_i=0$, which are the orthogonality conditions written out.

Let $W=\mathrm{span}\{(1,1,0),(1,0,1)\}\subseteq {\mathbb{R}}^3$, and project $b=(1,2,3)$ onto it.

---

Step 1: orthogonalise. Take $u_1=v_1=(1,1,0)$, so $⟨u_1,u_1⟩=1+1+0=2$.

For $v_2=(1,0,1)$: $⟨v_2,u_1⟩=1+0+0=1$, so the projection of $v_2$ onto $u_1$ is $\frac{1}{2}(1,1,0)=(\frac{1}{2},\frac{1}{2},0)$, and

Check. $⟨u_1,u_2⟩=\frac{1}{2}-\frac{1}{2}+0=0$. Also $⟨u_2,u_2⟩=\frac{1}{4}+\frac{1}{4}+1=\frac{3}{2}$.

The pair $\{u_1,u_2\}$ is an orthogonal basis of the same plane $W$, Gram–Schmidt never leaves the span.

Step 2: the coefficients.

The denominators matter: dropping them would give $c_2=\frac{5}{2}$ instead of $\frac{5}{3}$, and a projection pointing the right way with the wrong length.

Step 3: the projection.

Step 4: the residual and the checks.

Orthogonality. $⟨e,u_1⟩=-\frac{4}{3}+\frac{4}{3}+0=0$ and $⟨e,u_2⟩=-\frac{2}{3}-\frac{2}{3}+\frac{4}{3}=0$. Since $e$ is orthogonal to a basis of $W$, it is orthogonal to all of $W$.

Pythagoras. $\Vert b{\Vert }^2=1+4+9=14$. And

and $\frac{26}{3}+\frac{16}{3}=\frac{42}{3}=14$.

$(\frac{7}{3},\frac{2}{3},\frac{5}{3})$ is the point of the plane $W$ closest to $(1,2,3)$, and the distance between them is $\Vert e\Vert =\sqrt{16/3}\approx 2.31$. Any other point of $W$ is further, because moving within $W$ adds a vector orthogonal to $e$, and by Pythagoras that only increases the total.

On the ordering. Starting Gram–Schmidt from $v_2$ instead would give a different orthogonal basis of the same plane, and the same projection, since the projection depends on $W$ and not on which basis describes it. That invariance is worth checking once.

Theorem. Let $W$ be a subspace and $b$ any vector. The point $p={\mathrm{proj}}_W(b)$ is the unique element of $W$ minimising $\Vert b-w\Vert$ over $w\in W$.

Proof. Write $e=b-p$, which is orthogonal to every element of $W$ by construction. For any $w\in W$,

and $p-w\in W$ because $W$ is a subspace, so $e⟂(p-w)$. By Pythagoras,

with equality exactly when $\Vert p-w\Vert =0$, that is $w=p$. $◼$

The proof uses only that $e$ is orthogonal to $W$ and that $W$ is closed under subtraction. Nothing depends on coordinates or on the space being ${\mathbb{R}}^n$, so the result holds in any inner product space, including spaces of functions, where it underlies Fourier approximation.

Corollary (Pythagoras for the decomposition). Taking $w=0$ gives $\Vert b{\Vert }^2=\Vert p{\Vert }^2+\Vert e{\Vert }^2$, which is the second verification check.

Corollary (the normal equations). Let $W=\mathrm{col}A$. Then $\hat{x}$ minimises $\Vert b-Ax\Vert$ exactly when $A\hat{x}={\mathrm{proj}}_W(b)$, which holds exactly when the residual is orthogonal to every column:

So the normal equations are the orthogonality condition rearranged, not a separate construction. They are solvable for any $A$ and $b$, and the solution is unique exactly when the columns of $A$ are independent, in which case $A^{\mathsf{T}}A$ is invertible.

Why uniqueness can fail. If the columns are dependent, many $\hat{x}$ produce the same $A\hat{x}$. The projection is still unique, it is a point of $W$, and the theorem gives uniqueness there, but its representation in terms of the columns is not. Distinguishing the two is what separates "the fit is unique" from "the coefficients are unique", and only the first is guaranteed.