Singular Value Decomposition (SVD) and Its Applications in Linear Algebra

Chapter 7: The Singular Value Decomposition (SVD)

Introduction to SVD

The Singular Value Decomposition (SVD) is a powerful matrix factorization technique in linear algebra, widely used in mathematics, engineering, and data science. SVD expresses any real or complex matrix as a product of three specific matrices, revealing important properties about the original matrix.

• Definition: For any $m \times n$ matrix $A$, the SVD is $A = U \Sigma V^T$, where $U$ and $V$ are orthogonal (or unitary) matrices, and $\Sigma$ is a diagonal matrix containing the singular values.

• Applications: SVD is used in solving linear systems, computing matrix rank, data compression, and principal component analysis.

• Key Properties: The singular values in $\Sigma$ are always non-negative and sorted in decreasing order.

Basis and Matrices in the SVD

SVD provides orthonormal bases for the column and row spaces of a matrix. These bases are given by the columns of $U$ and $V$, respectively.

• Orthonormal Basis: The columns of $U$ form an orthonormal basis for the column space of $A$.

• Orthonormal Basis: The columns of $V$ form an orthonormal basis for the row space of $A$.

• Diagonalization: $A^TA$ and $AA^T$ are diagonalized by $V$ and $U$, respectively.

Computation and Example of SVD

To compute the SVD of a matrix $A$, follow these steps:

1. Find the eigenvalues and eigenvectors of $A^TA$ and $AA^T$.

2. Construct $V$ from the eigenvectors of $A^TA$ and $U$ from the eigenvectors of $AA^T$.

3. Form $\Sigma$ from the square roots of the nonzero eigenvalues (these are the singular values).

Example: For $A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}$, the SVD is:

• $U = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

• $\Sigma = \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix}$

• $V^T = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}$

Formulas and Equations

• SVD Formula: $A = U \Sigma V^T$

• Eigenvalue Relations: $A^TA v_i = \sigma_i^2 v_i$ $AA^T u_i = \sigma_i^2 u_i$

• Singular Values: $\sigma_i = \sqrt{\lambda_i}$, where $\lambda_i$ are the eigenvalues of $A^TA$ or $AA^T$

Comparison Table: SVD Components

Matrix	Role in SVD	Properties
$U$	Left singular vectors	Orthonormal columns
$\Sigma$	Singular values	Diagonal, non-negative, sorted
$V$	Right singular vectors	Orthonormal columns

Singular Value Stability vs. Eigenvalue Instability

Singular values are more stable under small perturbations of the matrix than eigenvalues, making SVD a preferred method in numerical applications.

• Stability: Small changes in $A$ lead to small changes in singular values.

• Eigenvalue Instability: Eigenvalues can change dramatically with small changes in $A$.

Worked Examples

Several examples illustrate the computation of SVD for different matrices, including diagonal and non-diagonal cases. These examples reinforce the step-by-step process and highlight the interpretation of singular values and vectors.

• Example 1: Compute the SVD for $A = \begin{bmatrix} 3 & 0 \\ 0 & 2 \end{bmatrix}$.

• Example 2: Find the singular values and vectors for a non-square matrix.

Review of Key Ideas

• The SVD factors any $m \times n$ matrix $A$ as $A = U \Sigma V^T$.

• $U$ and $V$ are orthogonal matrices; $\Sigma$ is diagonal with singular values $\sigma_i$.

• Singular values are the square roots of the eigenvalues of $A^TA$ and $AA^T$.

• SVD provides orthonormal bases for the column and row spaces of $A$.

Additional info: SVD is a foundational concept in linear algebra and is essential for understanding advanced topics in mathematics, engineering, and data science. It is particularly relevant for Precalculus students as an introduction to matrix decompositions and their applications.