Lecture: Generative Models and Generative AI

Introduction

• Focus on generative models or generative AI
• Buzz around generative AI; diving into foundational concepts
• Generative models: Systems identifying patterns in data to generate new instances
• Example: Generating realistic images of faces using models like StyleGAN

Types of Machine Learning Models

Supervised Learning

• Supervised models: Perform supervised learning
  • Use data with instances and labels
  • Learn mapping between data and labels for tasks like classification, regression, segmentation

Unsupervised Learning

• Unsupervised models: Learn patterns in data without labels
  • Learning underlying data structure
  • Techniques include density estimation and sample generation

Generative Models

• Generative models: Learn a probability distribution to generate new data
• Applications: Fairness, outlier detection, failure modes in autonomous systems
• Focus on latent variable models (key topics in this lecture)

Understanding Latent Variables

• Illustration via Plato's Myth of the Cave:
  • Observed variables (shadows) vs latent variables (objects causing shadows)
  • Goal: Learn underlying drivers of observed data

Autoencoders

Concept

• Autoencoder: Compress data to lower dimensions (latent space) for reconstruction
• Learn encoding (Z) that efficiently compresses data

Training an Autoencoder

• Loss function: Minimize difference between original data and reconstruction (e.g., Mean Squared Error)
• Represent original data in a compressed format and decode back to the original

Benefits

• Eliminate redundancy, improve efficiency
• Output (latent variable Z) allows data reconstruction

Variational Autoencoders (VAEs)

Introduction of Stochasticity

• Variational Autoencoder (VAE): Introduces randomness to latent variables (sampling)
• Latent variables as a vector of means (μ) and standard deviations (σ)

Training VAEs

• Encoder: Estimates probability distribution of latent variables given data (q)
• Decoder: Reconstructs data from latent variables
• Loss function: Reconstruction loss + regularization term (distance between learned distribution and prior, often normal distribution)

Regularization and Prior

• Regularization term: Encourages continuity and completeness in latent space
• Normal Prior: Ensures latent space distribution is centered around mean 0 and standard deviation 1

Interpreting Latent Variables

• Perturbation analysis: Hold other variables constant, vary one, and observe changes in output
• Disentanglement: Encouraged by weighting reconstruction terms more heavily
• Beta-VAE: A variant encouraging disentanglement of latent variables

Generative Adversarial Networks (GANs)

Basic Concept

• Two Networks: Generator (produces fake data) and Discriminator (distinguishes fake from real data)
• Training: Networks compete, with generator improving to produce data indistinguishably from real data

Process Visualization

• Generator training: Maps noise to real data distribution
• Discriminator training: Differentiates real from fake data

Application

• Noise Sampling: Generate new instances from random samples
• Interpolation: Traverse between samples to generate continuous transformations

Model Stability and Growth

• Paired Translation Gans: E.g. Image segmentation, converting day to night, maps
• Cycle GANs: Unpaired data translation, e.g. Horses to Zebras

Future Directions

• Mention of advanced generative models like Diffusion Models
• Diffusion Models: High fidelity generation, conditioned on various types of inputs
• Application in multiple fields beyond images

Hands-On Component

• Explore latent variable models, computer vision, facial detection in lab sessions

Conclusion

• Summary of key concepts: Latent variable models, autoencoders, VAEs, GANs
• Upcoming topics: Diffusion models and their application in generative AI

[Applause]