The Singular Value Decomposition

Every matrix factors as $U Σ V^{T}$ with orthonormal $U$ and $V$ and nonnegative diagonal $Σ$ , including rectangular and singular ones, which diagonalization cannot handle. The singular values measure how much the map stretches along each of a set of orthogonal directions, and truncating them gives the best low-rank approximation.

Definition

For any real $m \times n$ matrix $A$ there exist an orthogonal $m \times m$ matrix $U$ , an orthogonal $n \times n$ matrix $V$ , and an $m \times n$ matrix $Σ$ that is zero off the diagonal with $σ_{1} \geq σ_{2} \geq \dots \geq 0$ on it, such that

A = U Σ V^{T} .

The $σ_{i}$ are the singular values, the columns of $V$ the right singular vectors, and the columns of $U$ the left singular vectors.

Where they come from. $A^{T} A$ is symmetric and positive semidefinite, so it has an orthonormal basis of eigenvectors with nonnegative eigenvalues. Those eigenvectors are the columns of $V$ , and

σ_{i} = \sqrt{λ_{i} (A^{T} A)} .

For each $σ_{i} > 0$ , the corresponding left singular vector is $u_{i} = A v_{i} / σ_{i}$ ; these are orthonormal, and any remaining columns of $U$ are filled by extending to an orthonormal basis.

The action, read off. Writing $A v_{i} = σ_{i} u_{i}$ says the map sends one orthonormal basis to another, scaled: $V$ supplies orthogonal input directions, $U$ orthogonal output directions, and $Σ$ the stretch factor along each.

Rank and the four subspaces. The number of nonzero singular values is $rank A$ . The first $r$ columns of $V$ span the row space and the rest span $\ker A$ ; the first $r$ columns of $U$ span $col A$ and the rest span $\ker A^{T}$ .

Sum of rank-one pieces.

A = \sum_{i = 1}^{r} σ_{i} u_{i} v_{i}^{T},

and truncating after $k$ terms gives $A_{k}$ , the best rank- $k$ approximation in both the Frobenius and spectral norms. The Eckart–Young theorem. The Frobenius error of the truncation is $\sqrt{σ_{k + 1}^{2} + \dots + σ_{r}^{2}}$ .

Two invariants. $\prod_{i} σ_{i} = | det A |$ for square $A$ , and $\sum_{i} σ_{i}^{2} = tr (A^{T} A) = ‖ A ‖_{F}^{2}$ .

Assumptions and scope

Singular values are unique and nonnegative; singular vectors are not unique. A pair $(u_{i}, v_{i})$ may be replaced by $(- u_{i}, - v_{i})$ , and a repeated singular value admits any orthonormal basis of its subspace, so an answer differing from a reference by signs may still be correct.
The decomposition exists for every real matrix, rectangular or singular. This is the sense in which it is more general than diagonalization, which requires square and non-defective.
Singular values are not eigenvalues. For a symmetric positive semidefinite matrix they coincide; in general $σ_{i} = \sqrt{λ_{i} (A^{T} A)}$ , and a matrix with complex or negative eigenvalues still has real nonnegative singular values.
Forming $A^{T} A$ squares the condition number, so computing an SVD this way is numerically poor. Production algorithms bidiagonalise $A$ directly; the route here is for understanding and for hand computation.
The Eckart–Young optimality is over all matrices of rank at most $k$ , in the Frobenius and spectral norms. It is not a claim about other norms or about structured approximations such as nonnegative or sparse ones.

Worked material

Example

Four matrices diagonalization could not handle

A rectangular matrix. $A = (\begin{matrix} 1 & 0 & 0 \\ 0 & 2 & 0 \end{matrix})$ is $2 \times 3$ , so it has no eigenvalues, $A v = λ v$ compares a vector in $R^{2}$ with one in $R^{3}$ . It has singular values regardless: $A^{T} A = diag (1, 4, 0)$ , so $σ_{1} = 2$ , $σ_{2} = 1$ , and the rank is 2. The third right singular vector $(0, 0, 1)$ spans the kernel.

A singular square matrix. $A = (\begin{matrix} 1 & 2 \\ 2 & 4 \end{matrix})$ has $det = 0$ . Its $A^{T} A = (\begin{matrix} 5 & 10 \\ 10 & 20 \end{matrix})$ has trace 25 and determinant 0, so eigenvalues 25 and 0, giving $σ_{1} = 5$ and $σ_{2} = 0$ . One nonzero singular value, so rank 1, and indeed $A_{1} = A$ exactly, with zero approximation error.

A defective matrix. The shear $(\begin{matrix} 3 & 1 \\ 0 & 3 \end{matrix})$ is not diagonalizable over any field, as the diagonalization unit showed. It still has a singular value decomposition: $A^{T} A = (\begin{matrix} 9 & 3 \\ 3 & 10 \end{matrix})$ , trace 19, determinant $90 - 9 = 81$ , so $λ = \frac{19 \pm \sqrt{361 - 324}}{2} = \frac{19 \pm \sqrt{37}}{2}$ , giving $λ \approx 12.5414$ and $6.4586$ , both positive. The singular values are their square roots, $σ_{1} \approx 3.5414$ and $σ_{2} \approx 2.5414$ , whose product is $\sqrt{81} = 9 = | det A |$ .

This is the sharpest contrast available: a matrix with one eigenvector and two singular directions.

A rotation. $(\begin{matrix} 0 & - 1 \\ 1 & 0 \end{matrix})$ has no real eigenvalues. Its $A^{T} A = I$ , so both singular values are 1. It stretches nothing, which is exactly what a rotation does. Every orthogonal matrix has all singular values equal to 1, and conversely.

Rectangular, singular, defective, and eigenvalue-free over $R$ : each defeats diagonalization for a different reason, and none defeats the SVD. The decomposition asks only that $A^{T} A$ be symmetric and positive semidefinite, which it is for every real matrix without exception.

Non-example

Five errors about singular values

Taking singular values to be eigenvalues. For $A = (\begin{matrix} 3 & 1 \\ 0 & 3 \end{matrix})$ the eigenvalues are 3 and 3; the singular values are $\sqrt{(19 \pm \sqrt{37}) / 2}$ , roughly 3.54 and 2.54. They are different quantities: $σ_{i} = \sqrt{λ_{i} (A^{T} A)}$ , not $λ_{i} (A)$ . They coincide only when $A$ is symmetric positive semidefinite, and then only because $A^{T} A = A^{2}$ .

Expecting negative or complex singular values. A matrix with eigenvalues $- 2$ and $- 5$ has singular values 2 and 5; a rotation with eigenvalues $\pm i$ has singular values 1 and 1. Singular values are square roots of nonnegative numbers and are always real and nonnegative, which is why they measure stretching rather than describing dynamics.

Concluding that a defective matrix has no decomposition. Diagonalization fails for the shear above; the SVD does not. The existence proof needs only that $A^{T} A$ is symmetric and positive semidefinite, which holds for every real matrix. "Not diagonalizable" and "no SVD" are unrelated claims, and the second is never true.

Flipping one singular vector without its partner. Replacing $v_{1}$ by $- v_{1}$ while leaving $u_{1}$ alone breaks $A v_{1} = σ_{1} u_{1}$ and the factorisation no longer reproduces $A$ . Sign freedom is real but paired: both members of $(u_{i}, v_{i})$ flip together, because the outer product $u_{i} v_{i}^{T}$ must be unchanged.

Treating the decomposition as unique. With a repeated singular value, any orthonormal basis of that subspace serves, so two correct answers may share no singular vectors at all. The identity matrix has $σ_{1} = σ_{2} = 1$ and every orthonormal pair is a valid choice. Comparing an answer against a reference by matching vectors is therefore the wrong check; verifying $A v_{i} = σ_{i} u_{i}$ is the right one.

What separates these. The first three mistake the SVD for diagonalization and import its limitations. The last two mistake a construction with genuine freedom for one with a single answer. Both errors are caught by the same habit: verify the factorisation rather than the vectors.

Common errors

Common misconception

The singular values of a matrix are its eigenvalues, so a matrix with negative or complex eigenvalues has negative or complex singular values, and a defective matrix has no usable decomposition.

The Singular Value Decomposition

Assumptions and scope

Worked material

Four matrices diagonalization could not handle

Five errors about singular values

Common errors

Related units

Requires

Learn this topic

Used in

Sources

Settings

Focus timer

Today

Last seven days

Where the time went

Recent sessions

Settings

Notifications

Your history