Machine Learning Foundations: Week 5 - Lecture on Linear Algebra

Introduction

• Focus on linear algebra continuation.
• Goal: Understand the spectral theorem in the context of complex vector spaces.
• Importance of Hermitian matrices (generalize real symmetric matrices).
• Main result: Every Hermitian matrix is orthogonally diagonalizable.

Complex Matrices and Vectors

• Complex Vector Space:

  • Denoted by (\mathbb{C}^n), analogue of Euclidean space (\mathbb{R}^n).
  • Elements are complex numbers (x_i) (e.g., (x_1, x_2, \ldots, x_n)).

• Operations with Complex Numbers:

  • Addition: ((a + ib) + (c + id) = (a+c) + i(b+d)).
  • Multiplication: ((a+ib)(c+id) = (ac-bd) + i(bc+ad)).

• Complex Conjugate:

  • Of (a + ib) is (a - ib).
  • Geometric representation using polar coordinates.

Inner Product in Complex Spaces

• Inner Product Definition:

  • Different from Euclidean space due to non-real nature.
  • Defined using conjugates: (x \cdot y = x^* y) (conjugate transpose of (x) times (y)).
  • Key property: (x^* y \neq y^* x) for complex numbers.

• Length of a Vector:

  • Defined as (||x|| = \sqrt{x^* x}).
  • Example: Length of ((1, i)) is (\sqrt{2}), not zero.

Conjugate Transpose and Hermitian Matrices

• Conjugate Transpose (A^*):

  • Combination of transposition and conjugation.
  • For real matrices, (A^* = A^T).

• Hermitian Matrices:

  • Equivalent to symmetric matrices in complex spaces.
  • Defined as (A^* = A).
  • Significant because they are orthogonally diagonalizable.*

Recap

• Key Concepts:
  • Defined inner product for complex vectors (involves conjugation).
  • Introduced conjugate transpose for matrices.
  • Identified Hermitian matrices, a crucial class of matrices.
• Significance:
  • Hermitian matrices simplify diagonalization, essential for spectral theorem.
• Upcoming lectures will further explore Hermitian matrices and their properties.