Practice: The Singular Value Decomposition

(a) Singular values of $A$.

Diagonal, so the eigenvalues are 4 and 9. Ordering decreasingly, ${\lambda }_1=9$ and ${\lambda }_2=4$, giving

Checks. ${\sigma }_1^2+{\sigma }_2^2=9+4=13=\mathrm{tr}(A^{\mathsf{T}}A)$ ✓, and ${\sigma }_1{\sigma }_2=6=|detA|=|-6|$ ✓. (b) The factorisation. The eigenvector for ${\lambda }_1=9$ is $e_2=(0,1)$ and for ${\lambda }_2=4$ is $e_1=(1,0)$, so in decreasing order

Left singular vectors:

So

Verification. $A{v}_1=A(0,1{)}^{\mathsf{T}}=(0,-3{)}^{\mathsf{T}}$ and ${\sigma }_1{u}_1=3(0,-1{)}^{\mathsf{T}}=(0,-3{)}^{\mathsf{T}}$ ✓. $A{v}_2=(2,0{)}^{\mathsf{T}}$ and ${\sigma }_2{u}_2=2(1,0{)}^{\mathsf{T}}=(2,0{)}^{\mathsf{T}}$ ✓. Both $U$ and $V$ have orthonormal columns. What is unusual. $A$ is diagonal with eigenvalues 2 and $-3$, and its singular values are 2 and 3. The absolute values of the eigenvalues, reordered. That is what happens for a diagonal (or more generally symmetric) matrix: ${\sigma }_i=|{\lambda }_i|$. The minus sign has not disappeared; it has moved into $u_1$, which is $(0,-1)$ rather than $(0,1)$. (c) $B$ is rank deficient.

Trace 25, determinant $100-100=0$, so the eigenvalues are 25 and 0. Hence

Check. ${\sigma }_1^2+{\sigma }_2^2=25=\mathrm{tr}(B^{\mathsf{T}}B)=5+20$ ✓, and ${\sigma }_1{\sigma }_2=0=|detB|$ ✓. One nonzero singular value, so $\mathrm{rank}B=1$. Kernel. The right singular vector for $\lambda =0$ spans it. Solving $B^{\mathsf{T}}Bv=0$, or directly $Bv=0$: $v_1+2v_2=0$ gives $v\propto (2,-1)$. Checking, $B(2,-1{)}^{\mathsf{T}}=(2-2,4-4{)}^{\mathsf{T}}=(0,0{)}^{\mathsf{T}}$ ✓. Normalised, $v_2=\frac{1}{\sqrt{5}}(2,-1)$. The other right singular vector is $v_1=\frac{1}{\sqrt{5}}(1,2)$, and $u_1=B{v}_1/{\sigma }_1=\frac{1}{5\sqrt{5}}(5,10{)}^{\mathsf{T}}=\frac{1}{\sqrt{5}}(1,2{)}^{\mathsf{T}}$. (d) Rank-one expansion of $B$. Since $r=1$ there is a single term:

The expansion reproduces $B$ exactly, so the best rank-1 approximation is $B$ and the error is

That is the general statement in a degenerate case: a matrix of rank 1 is already its own best rank-1 approximation, and nothing is discarded. (e) Negative and zero eigenvalues. For $A$, the eigenvalue $-3$ produced the singular value $3$. Squaring in $A^{\mathsf{T}}A$ destroys the sign: $(-3{)}^2=9$, and the nonnegative square root is 3. The sign is not lost from the decomposition. It reappears in the left singular vector $u_1=(0,-1)$, which records that the map reverses that direction. For $B$, the eigenvalue 0 produced the singular value 0. Here nothing is destroyed: a direction sent to zero is stretched by a factor of zero, and the two notions agree. This is why ${\sigma }_i=0$ and "$B$ is singular" are the same statement. Why always real and nonnegative. The singular values are $\sqrt{{\lambda }_i(A^{\mathsf{T}}A)}$, and every eigenvalue of $A^{\mathsf{T}}A$ satisfies

so each square root is a real nonnegative number. The quantity being rooted is a squared length, which is what makes the sign question disappear before it arises. This also explains the interpretation: ${\sigma }_i=\Vert A{v}_i\Vert$ is literally how far the unit vector $v_i$ is stretched, and a length is never negative.