Matrix Calculus Lecture Notes

Introduction

• Professors: Alan Edelman and Stephen Johnson
• Departments: Mathematics, CSE, Julia Lab
• Platform: Zoom and GitHub
• Course Details: 16 lectures, problem sets, no prior experience with Julia required but helpful

Course Context

• Calculus at MIT
  • Single Variable Calculus (1801): Derivative/integral of one-variable functions
  • Multivariable Calculus (1802): Gradient, Jacobian
• What is Matrix Calculus?
  • Extends single and multivariable calculus to matrices and higher-dimensional arrays
  • Significant for disciplines like machine learning, statistics, engineering

Importance of Matrix Calculus

• Applications
  • Machine Learning: Gradient descent, parameter optimization, backpropagation
  • Engineering: Topology optimization in aerodynamics and fluid dynamics
  • Physics: PDE-constrained optimization
• Current Trends: Linear algebra’s importance has increased significantly due to machine learning and computational advances

Key Concepts and Motivations

• Why Matrix Calculus?: Needed for complex gradient calculations in machine learning and other fields
• Challenges: More complicated than scalar and vector calculus, many unfamiliar with its nuances
• Understanding Derivatives in Matrix Context: Not always intuitive—specific methods and rules are required

Understanding Linearization

• Concept of Linearization: Approximating non-linear functions as locally linear
• 1D Calculus: Change in y = f’(x) * change in x
• Multivariable Calculus: Generalizes to n-dimensional vectors and matrices

Tools and Notations

• Julia Notation: Point-wise operations (e.g., .* for element-wise multiplication)
• Trace of a Matrix: Sum of diagonal elements
• Mathematical Websites: Example, matrixcalculus.org for derivative calculations
• New Notations: δ for finite perturbations, d for infinitesimal changes

Product Rule in Matrix Calculus

• Extension from Scalar to Matrix/Vector Calculus
  • Example: d(AB) = A dB + dA B
  • Special Cases: e.g., d(X^TX) = 2X^T dX

Experiments and Examples

• Demonstrations in Julia: Experimentally verify Matrix Calculus rules
  • Squaring a matrix and using the product rule to verify results

Gradient and Jacobian

• Scalar to Vector Gradients: Scalar functions (loss functions) and their gradients
• Vector to Matrix Gradients: Higher-dimensional applications
• Jacobians and Hessians: Understanding first and second derivatives

Recapitulation and Conclusion

• Quadratic Forms: Abstraction for second derivatives in advanced linear algebra
• Next Steps: Break before continuing with more detailed exploration