Lecture 2: Deep Sequence Modeling

Introduction

• Instructor: Ava
• Focus: Foundations of sequence modeling
• Upcoming: Lectures on large language models

Overview of Sequence Modeling

• Builds on basics of neural networks
• Application to sequential data problems
• Intuitive example: Predicting ball's movement in 2D space
• Importance: Audio processing, text, medical signals, financial data, etc.

Problem Formulations

• Sequence processing to produce outputs
• Tasks: Sentiment analysis, text generation, language translation

Building Neural Networks for Sequential Data

• Introduction of recurrent neural networks (RNNs)
• Essentials:
  • Internal state h(t) to maintain memory
  • Output depends on current input and past state
• RNNs have recurrent relations and are unrolled across time

Fundamentals of RNNs

• Key operations:
  • Hidden state updates
  • Output predictions
• Implementing RNNs:
  • Example pseudo code
  • Demonstrated in frameworks like TensorFlow and PyTorch

Challenges in Sequence Modeling

• Variability in sequence lengths
• Long-term dependencies
• Maintaining order

Operationalization in Practice

• Task: Predicting the next word in a sequence
• Importance of vectorizing inputs (embeddings)
• Handling sequence complexities

Training RNNs

• Backpropagation Through Time (BPTT)
• Challenges: Vanishing/exploding gradients
• Advanced architectures like LSTM to mitigate issues

Applications of RNNs

• Music generation example
• Experience in software labs

Limitations of RNNs

• Bottleneck in state size
• Sequential processing hinders parallelization

Introduction to Attention and Transformers

• Need for attention mechanism
• Paper: "Attention Is All You Need"

Core Concept of Attention

• Intuition: Human-like focus on important features
• Mechanism: Query, key, value matrices
• Process:
  • Compute similarity (dot product)
  • Attention weights
  • Output features
• Applications beyond language: Vision Transformers

Conclusion

• Overview of sequence modeling and RNNs
• Introduction to attention and Transformers
• Sequence modeling as a complex, rich field

Next Steps

• Hands-on labs on GitHub
• Further exploration of large language models
• Reception at One Kendall Square

• Note: This lecture emphasized both the theoretical and practical aspects of sequence modeling, preparing students for advanced topics in the field.