Lecture Notes: Math in Machine Learning

Introduction

• Importance of Math in Machine Learning:
  • Programming in machine learning involves optimization.
  • Understanding optimization requires mathematics.
  • Machine learning is more mathematical than other programming branches like web development.

Key Topics Covered

1. Linear Algebra
   • Helps understand the objects being optimized.
   • Core operation: Matrix multiplication.
2. Calculus
   • Aids in understanding how we optimize.
3. Probability and Statistics
   • Helps clarify what we are optimizing.

Linear Algebra

Importance of Linear Algebra

• Definition:
  • Mathematics of arrays (collections of numbers).
• Application in Machine Learning:
  • Data and models are represented as arrays.
  • Core manipulation is via matrix multiplication.

Examples of Matrix Multiplication in Action

• Inner product, correlation, covariance in statistics.
• Classical algorithms: Linear & logistic regression, PCA, hidden layers in neural networks.
• GPUs:
  • Originally for video games, now crucial for fast machine learning due to linear algebra requirements.

Matrix Multiplication Explained

• Process:
  • Multiply rows of the first matrix with columns of the second.
  • Sum the products for output matrix entries.
• Rule:
  • Takes matrices with compatible dimensions and outputs a new matrix.

Understanding Linear Algebra Differently

Linear Algebra vs. Algebra

• Linear algebra often confused with traditional algebra; it involves more than equations.
• Geometric View:
  • Studies linear transformations (rotations, reflections, scalings).
• Graphical Linear Algebra:
  • Uses diagrammatic reasoning instead of equations.
• Programming Perspective:
  • Matrices as functions, shapes as data types, multiplication as function composition.

Function Composition in Programming

• Example: Combined functions to check string length.
• Matrix-Vector Multiplication as Function Application:
  • Matrix behaves like a function applying to a vector.

Linear Functions and Their Properties

Definition of Linear Functions

• Respect two rules:
  1. Additivity:
     • f(a + b) = f(a) + f(b)
  2. Scalar Multiplication:
     • f(λa) = λf(a)
• Zero Value:
  • Always sends zero to zero.
• Kernel:
  • Collection of inputs that map to zero; informs about the rank of a function.

Singular Value Decomposition (SVD)

Concept of SVD

• Equivalent to refactoring a program in linear algebra.
• Breaks down functions:
  • Separation of Concerns:
    • Eigen decomposition separates components of a matrix.
  • Low Rank Approximation:
    • Simplifies matrices while retaining behavior.

Example of Refactoring in Programming

• Function breakdown (e.g., is odd function) as canonical decomposition.
• Steps Discussed:
  1. Identify representatives (even/odd).
  2. Perform reversible renaming.
  3. Match outputs to desired format.

Application of SVD in Machine Learning

Practical Uses

• Low Rank Approximations:
  • Useful in data compression and extracting significant patterns (e.g., images, videos).
• Example:
  • Security camera footage analyzed through low rank approximation.

Conclusion

Key Takeaways

1. Linear algebra is crucial in machine learning for optimization.
2. Matrix multiplication is fundamental despite its simplicity.
3. Linear algebra is conceptually distinct from traditional algebra; it's more akin to programming practices.
4. Singular Value Decomposition (SVD):
   • Essential for understanding and manipulating matrices in machine learning.

Further Resources

• Essence of Linear Algebra by 3Blue1Brown:
  • Visual learning on linear algebra concepts.
• Graphical Linear Algebra by Pawel Sobachinski:
  • Insight into abstract math.
• Numerical Linear Algebra by fast.ai:
  • Focus on applications in machine learning.
• Linear Algebra Done Right by Sheldon Axler:
  • Traditional approach with lecture series available online.