TagsArticle Source
- Title: CS 294: Deep Reinforcement Learning, Fall 2017
- Authors: Sergey Levine
Deep Reinforcement Learning
Fall 2017 Materials
Lecture Videos
The course lectures are available below. The course is not being offered as an online course, and the videos are provided only for your personal informational and entertainment purposes. They are not part of any course requirement or degree-bearing university program. For all videos, click here. For live stream, click here.
Lectures, Readings, and Assignments
Below you can find an outline of the course. Slides and references will be posted as the course proceeds.
- Aug 23: Introduction and course overview (Levine)
- Aug 28: Supervised learning and imitation (Levine)
- Slides
- Homework 1 is out: Imitation Learning
- Aug 30: Reinforcement learning introduction (Levine)
- Sep 4: Holiday - no class.
- Sep 6: Policy gradients introduction (Levine)
- Slides
- Homework 1 milestone due
- Sep 8: Neural networks review (Achiam)
- Sep 11: Actor-critic introduction (Levine)
- Slides
- Homework 1 is due, Homework 2 is out: Policy Gradients
- Sep 13: Value functions introduction (Levine)
- Sep 18: Advanced Q-learning algorithms (Levine)
- Sep 20: Optimal control and planning (Levine)
- Sep 25: Learning dynamical systems from data (Levine)
- Sep 27: Learning policies by imitating optimal controllers (Levine)
- Oct 2: Advanced model learning and images (Guest lecture: Chelsea Finn)
- Oct 4: Connection between inference and control (Levine)
- Slides
- Homework 3 is due, Homework 4 is out: Model Based RL
- Oct 9: Inverse reinforcement learning (Levine)
- Slides
- Project proposal is due
- Oct 11: Advanced policy gradients (natural gradient, importance sampling) (Achiam)
- Oct 16: Exploration (Levine)
- Oct 18: Exploration (part 2) and transfer learning (Levine)
- Slides
- Homework 4 is due
- Oct 23: Multi-task learning and transfer (Levine)
- Oct 25: Meta-learning and parallelism (Levine)
- Oct 30: Advanced imitation learning and open problems (Levine)
- Nov 1: Guest lecture: Igor Mordatch, OpenAI
- Nov 6: Guest lecture: Mohammad Norouzi, Google
- Project milestone
- Nov 8: Guest lecture: Pieter Abbeel, UC Berkeley and OpenAI
- Nov 13: Guest lecture: Emma Brunskill, Stanford University
- Nov 15: Guest lecture: Aviv Tamar, UC Berkeley
- Nov 20: Guest lecture: John Schulman, OpenAI
- Nov 22: Holiday - no class
- Nov 27: Final project presentations 1
- Nov 29: Final project presentations 2
Previous Offerings
A full version of this course was offered in Spring 2017. Lecture videos from Spring 2017, are available here
An abbreviated version of this course was offered in Fall 2015.
Prerequisites
CS189 or equivalent is a prerequisite for the course. This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We’ll review this material in class, but it will be rather cursory.
- Reinforcement learning and MDPs
- Definition of MDPs
- Exact algorithms: policy and value iteration
- Search algorithms
- Numerical Optimization
- gradient descent, stochastic gradient descent
- backpropagation algorithm
- Machine Learning
- Classification and regression problems: what loss functions are used, how to fit linear and nonlinear models
- Training/test error, overfitting.
For introductory material on RL and MDPs, see
- CS188 EdX course, starting with Markov Decision Processes I
- Sutton & Barto, Ch 3 and 4.
- For a concise intro to MDPs, see Ch 1-2 of Andrew Ng’s thesis
- David Silver’s course, links below
For introductory material on machine learning and neural networks, see
Related Materials
John’s lecture series at MLSS
- Lecture 1: intro, derivative free optimization
- Lecture 2: score function gradient estimation and policy gradients
- Lecture 3: actor critic methods
- Lecture 4: trust region and natural gradient methods, open problems
Courses
- Dave Silver’s course on reinforcement learning / Lecture Videos
- Nando de Freitas’ course on machine learning
- Andrej Karpathy’s course on neural networks
Relevant Textbooks
- Deep Learning
- Sutton & Barto, Reinforcement Learning: An Introduction
- Szepesvari, Algorithms for Reinforcement Learning
- Bertsekas, Dynamic Programming and Optimal Control, Vols I and II
- Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming
- Powell, Approximate Dynamic Programming