Singular Value Decomposition (SVD): Concepts, Construction, and Applications

Singular Value Decomposition (SVD)

8.1 Introduction

Singular Value Decomposition (SVD) is a fundamental matrix factorization technique in linear algebra. It expresses any real or complex matrix as a product of three matrices, revealing important geometric and algebraic properties. SVD is widely used in data analysis, signal processing, and numerical methods.

• Matrix Factorization: Writing a matrix as a product of other matrices to simplify analysis and computation.

• Applications: SVD is used for solving linear systems, computing pseudoinverses, and dimensionality reduction.

8.2 Definitions

• Singular Value Decomposition (SVD): For an m × n real matrix A, the SVD is a factorization of the form: $A = U \, ext{diag}(oldsymbol{ au}) \, V^T$

  • U: An m × m orthogonal matrix (columns are left singular vectors).

  • diag(τ): An m × n diagonal matrix with non-negative real numbers (singular values) on the diagonal.

  • V: An n × n orthogonal matrix (columns are right singular vectors).

• Orthogonal Matrix: A square matrix whose columns and rows are orthonormal vectors.

• Diagonal Matrix: A matrix with nonzero entries only on the main diagonal.

8.3 Constructing the SVD

To construct the SVD of a matrix, we use properties of eigenvalues and eigenvectors of ATA and AAT. The process involves the following steps:

8.3.1 Observations 1

• For any matrix A, the nonzero eigenvalues of ATA and AAT are the same.

• The singular values of A are the square roots of the positive eigenvalues of ATA (or AAT).

8.3.2 Observations 2

• Let v be an eigenvector of ATA with eigenvalue λ. Then Av is an eigenvector of AAT with the same eigenvalue λ (if Av ≠ 0).

• There is a bijection between the nonzero eigenvalues and eigenvectors of ATA and AAT.

8.3.3 Construction in Brief

1. Find the positive (nonzero) eigenvalues of ATA and AAT. The singular values are the square roots of these eigenvalues.

2. For each singular value, find the corresponding eigenvectors of ATA (right singular vectors) and AAT (left singular vectors).

3. Form the matrices U, diag(τ), and V from these vectors and values.

Example 8.1

Given $A = \begin{bmatrix} 4 & 2 \\ 3 & 3 \end{bmatrix}$, compute $A^T A$ and $A A^T$:

• $A^T A = \begin{bmatrix} 25 & 18 \\ 18 & 13 \end{bmatrix}$

• $A A^T = \begin{bmatrix} 20 & 18 \\ 18 & 18 \end{bmatrix}$

Find eigenvalues, eigenvectors, and construct the SVD as shown in the detailed steps in the notes.

8.4 Matlab

Matlab can be used to compute the SVD efficiently. The command [U, S, V] = svd(A) returns the matrices U, S, and V for a given matrix A.

A = [2 2; 1 1]; [U, S, V] = svd(A)

This will output the orthogonal matrices and the diagonal matrix of singular values.

8.5 Exercises

Practice problems involve finding the SVD of given matrices by hand or using computational tools. Example matrices are provided for students to work through the process of finding singular values and vectors.

Step	Description
1	Compute $A^T A$ and $A A^T$
2	Find eigenvalues and eigenvectors
3	Compute singular values (square roots of eigenvalues)
4	Form U, S, V matrices

Additional info: SVD is a key tool in numerical linear algebra and is foundational for advanced topics such as Principal Component Analysis (PCA) and matrix approximations.