Reinforcement Learning Introductory Lecture

Instructor Introduction

• Harvan Husselt, Research Scientist at DeepMind
• Course conducted annually at UCL, this year pre-recorded due to COVID-19
• Co-instructors: Diana Bursa and Matteo Hessel

Course Overview

• Focus on Reinforcement Learning (RL)
• Recommended Text: "Reinforcement Learning: An Introduction" by Rich Sutton and Andy Barto
• Communication through Moodle for UCL students
• No final exams, graded based on assignments

Key Concepts

What is Reinforcement Learning?

• Relation to AI: Automating solutions, allowing machines to learn and find solutions autonomously
• Inspired by Alan Turing's ideas on machine learning from his 1950 paper
• Central question: How can machines learn to make decisions to achieve goals?
• Involves interaction and learning from environment

Concepts of Artificial Intelligence

• Automation of Repeated Tasks: Industrial and digital revolutions
• AI Goal: Enable machines to learn and make decisions autonomously
• Principles: Learning, autonomy, making decisions

Elements of Reinforcement Learning

Interaction with Environment

• Active Learning: Learner influences the data they receive
• Sequential Interaction: Decisions affect future interactions
• Goal-Directed Behavior: Actions are purposeful
• Learning Without Examples: Not always guided by explicit correct actions

The Agent-Environment Interaction Loop

• Agent: Learns by interacting with the environment
• Environment: Provides observations and rewards to the agent
• Reward Signal: Guides agent behavior
• Cumulative Reward (Return): Sum of rewards over time; the goal is to maximize this

Reward Hypothesis

• Any goal can be specified as maximizing cumulative reward
• Designing a reward function is crucial to defining goals

Reinforcement Learning Problems and Solutions

Examples

• Applications in gaming, robotics, finance, etc.
• RL can adapt online to unforeseen changes (e.g., wear and tear in robots)

Formalizing Reinforcement Learning

• Markov Decision Process (MDP): Mathematical model for RL problems
• Markov Property: Future states depend only on the current state, not past history
• Agent State: Summarizes the history affecting future decisions
• Policy: Mapping from states to actions, can be deterministic or stochastic
• Value Function: Predicts future rewards from a state
• Model: Predicts state transitions and rewards

Reinforcement Learning Algorithms

• Value-Based: Learns value functions (e.g., Q-learning)
• Policy-Based: Learns policies directly
• Actor-Critic: Combines policy and value functions

Learning and Planning

• Learning: Directly from interaction, updating policies
• Planning: Uses models for internal computation to enhance decision-making

Deep Reinforcement Learning

• Combines RL with deep learning techniques to handle complex problems

Conclusion

• RL is about learning decision-making through interaction
• The course will cover various RL algorithms and their applications
• Next lecture will focus on exploration strategies in RL.