Lecture Notes: Practical Optimization & Stochastic Control

Introduction

• Theme: Practical, real-life details in optimization and control.
• Previous Topic Recap: Basic algorithms like trajectory optimization, MPC, convex/nonconvex methods.
• Focus Shift: Real-world application on hardware vs. simulation.

Iterative Learning Control (ILC)

• Issue: Model error/mismatch between real system and predictions from simulations.
• Goal: Use data from real hardware to adjust feedback policy.
• Pros:
  • Slots into existing tools
  • Highly data-efficient
• Cons: Narrow scope, best for repetitive tasks in industrial settings.
• Applications: Industrial robotics, drones, etc.
• Core Idea: Adapting open loop controls for repetitive tasks.

Adaptive Control & Online Learning

• Broad Focus: Techniques to improve control policy using hardware data.
• Challenges: Variations between hardware units, wear over time, etc.
• Origins: 1980s/70s industrial robotics.
• Modern Uses: Complex, large systems with limited data.

Stochastic Control Introduction

• Context: Moving from deterministic to stochastic settings.
• Problem: Incomplete state access, noisy measurements.
• Goal: Estimate state using noisy partial observations.
• Key Idea: States are fictional; only actuator commands and sensor data are real.
• Measurement Model: Function (G) mapping state (X) to measurement (Y) with noise.
• State Estimation: Process to recover X from Y.

Dynamic Programming & Stochastic Control

• Framework: Use probability distributions rather than definite states.
• Objective: Minimize expected cost, not just cost.
• Challenge: High dimension space makes writing this intractable.
• Solution: Approximate methods—Gaussian mixtures, particle methods, neural networks.
• LQG Problem: Linear dynamics, quadratic costs, Gaussian noise.
• Usefulness: Insights and local approximations for nonlinear problems.

Linear Quadratic Gaussian (LQG) Control

• Define: Generalization of LQR to stochastic settings.
• **Components: **
  • Linear Dynamics (Ax + Bu + W)
  • Measurement model (C*x + V)
• Noise: Process noise (W) and measurement noise (V) assumed to be Gaussian.
• Expectation: Impacts calculations; dominates next steps.

Gaussian Distributions & Expectations

• Multivariate Gaussian: Basics for probability distribution.
• Mean and Covariance Definitions: Centering and spread measurement.
• Uncorrelated Variables: Zero covariance off-diagonal terms.
• Expectation Formula: Integral of function weighted by probability distribution.

Dynamic Programming in LQG

• Cost Function: Minimize expected costs.
• Backup Process: Use dynamic programming principles.
• Breakdown: Split terms and calculate expected values.
• Significance: Noise terms don’t impact optimal control design.
• Consequence: Separation between control and estimation in design.

Certainty Equivalence Principle

• Definition: Optimal LQG controller equals deterministic LQR controller plus expected state.
• Implication: Noise increases cost but doesn't impact control policy.

Separation Principle

• Definition: Separate design of optimal controller and estimator.
• Implication: Facilitates manageable design processes.

Generalization Beyond LQG

• Limitations: Principles only hold under LQG assumptions (Gaussian noise, linear dynamics).
• Practice: Often used for approximations; adding sensors can help achieve assumptions.
• Challenges: Multimodal distributions and nonlinear couplings require complex reasoning.

State Estimation Overview

• Problem: Given measurements, deduce state.
• **Approaches: **
  • Maximum a Posteriori (MAP): Arg max of probability of state given measurements.
  • Minimum Mean Squared Error (MMSE): Minimize sum of squared errors, expectation of errors.
• Gaussian Case: Both methods coincide with straightforward results.