Singular Value Decomposition (SVD)

Overview

Introduction

• Presenter: Louis Sorano
• Purpose: Explain singular value decomposition (SVD) and its applications, such as image compression.
• Key Topics:
  • Transformations (rotations, stretchings, compressions)
  • Linear transformations and matrices
  • Dimensionality reduction
  • Image compression

Announcements

• Code available on GitHub: Luis Disserano / singular_value_decomposition
• Book: Rocking Machine Learning
  • Link and discount code available in the video comments.

Transformations

Types of Transformations

• Stretching and compressing
  • Horizontally
  • Vertically
• Rotations

Example

• Transforming a circle into an ellipse using rotations and scalings.

Puzzle 1: Shape Transformation

• Steps involved:
  1. Stretch horizontally
  2. Compress vertically
  3. Rotate counterclockwise

Puzzle 2: Color-Coded Transformation

• Ensuring colors maintain their location through transformation steps:
  1. Rotate to correct orientation
  2. Stretch and compress
  3. Rotate back to the correct orientation

Moral

• Any linear transformation can be mimicked by a combination of rotations and scalings.

Linear Transformations

Definition

• A linear transformation is a matrix.
• Matrices define transformations of points in the plane.

Matrix Example

• Matrix A = [[3, 0], [4, 5]]
• Transformation Examples:
  • Point (1,0)  (3,4)
  • Point (0,1)  (0,5)
  • Point (-1,0)  (-3,-4)
  • Point (0,-1)  (0,-5)

Unit Circle Transformation

• Units circle mapped to an ellipse by a 2x2 matrix.
• Rotation matrix represented by cos(θ) and sin(θ).
• Diagonal matrix for scaling (sigma values for stretching).

Singular Value Decomposition (SVD)

Main Equation

• Any matrix A can be decomposed:
  • A = U * Σ * V*
  • U and V are rotation matrices.
  • Σ (sigma) is a diagonal or scaling matrix.

Calculation Methods

• Tools: Wolfram Alpha and numpy (Python)
  • Function: svd() in numpy

Example Decomposition

• Sequence of transformations explained.
  • Example matrices for rotations and scalings provided.
  • Understanding of transformations in sequence.

Dimensionality Reduction

Concept

• Simplifying our matrix by reducing its rank.
• Example: a skinny ellipse representing a near-degenerate transformation.

Rank Approximation

• Transform a nearly singular matrix (small sigma values).

High-Dimensional Examples

• Rank 1 matrix: Predictable, can be reduced into product of 2 vectors.
• Rank 2+ matrix: More complex, but can be approximated by sum of rank 1 matrices.

Singular Value Decomposition Application

• Approximating any matrix as a sum of rank 1 matrices.
• Practical for large matrices (e.g., 1000x1000) to save space.

Image Compression

Methodology

• Using SVD to compress images.
• Steps:
  1. Encode image as a matrix.
  2. Perform SVD to decompose matrix.
  3. Use top singular values to approximate and compress the image.

Example

• Compressing an image of a heart
• Process detailed with intermediate matrix representations.
  • Each step adds more details.

Summary

• SVD allows control over the degree of compression and quality.
• Choices impact storage requirements and image sharpness.

Conclusion

Additional Resources

• Videos on matrix factorization and principal component analysis (PCA).

Contact

• Louis Sorano's book: Rocking Machine Learning
• Twitter: @louislikesmath
• Website: serrano.academy