Stop Thinking, Just Do!

Sung-Soo Kim's Blog

Deep Reinforcement Learning

tagsTags

10 April 2018


Article Source


Deep Reinforcement Learning

Fall 2017 Materials

Lecture Videos

The course lectures are available below. The course is not being offered as an online course, and the videos are provided only for your personal informational and entertainment purposes. They are not part of any course requirement or degree-bearing university program. For all videos, click here. For live stream, click here.

Lectures, Readings, and Assignments

Below you can find an outline of the course. Slides and references will be posted as the course proceeds.

  • Aug 23: Introduction and course overview (Levine)
  • Aug 28: Supervised learning and imitation (Levine)
  • Aug 30: Reinforcement learning introduction (Levine)
  • Sep 4: Holiday - no class.
  • Sep 6: Policy gradients introduction (Levine)
    • Slides
    • Homework 1 milestone due
  • Sep 8: Neural networks review (Achiam)
  • Sep 11: Actor-critic introduction (Levine)
  • Sep 13: Value functions introduction (Levine)
  • Sep 18: Advanced Q-learning algorithms (Levine)
  • Sep 20: Optimal control and planning (Levine)
    • Slides
    • Homework 2 is due, Homework 3 is out: DQN
  • Sep 25: Learning dynamical systems from data (Levine)
  • Sep 27: Learning policies by imitating optimal controllers (Levine)
  • Oct 2: Advanced model learning and images (Guest lecture: Chelsea Finn)
  • Oct 4: Connection between inference and control (Levine)
  • Oct 9: Inverse reinforcement learning (Levine)
    • Slides
    • Project proposal is due
  • Oct 11: Advanced policy gradients (natural gradient, importance sampling) (Achiam)
  • Oct 16: Exploration (Levine)
  • Oct 18: Exploration (part 2) and transfer learning (Levine)
  • Oct 23: Multi-task learning and transfer (Levine)
  • Oct 25: Meta-learning and parallelism (Levine)
  • Oct 30: Advanced imitation learning and open problems (Levine)
  • Nov 1: Guest lecture: Igor Mordatch, OpenAI
  • Nov 6: Guest lecture: Mohammad Norouzi, Google
    • Project milestone
  • Nov 8: Guest lecture: Pieter Abbeel, UC Berkeley and OpenAI
  • Nov 13: Guest lecture: Emma Brunskill, Stanford University
  • Nov 15: Guest lecture: Aviv Tamar, UC Berkeley
  • Nov 20: Guest lecture: John Schulman, OpenAI
  • Nov 22: Holiday - no class
  • Nov 27: Final project presentations 1
  • Nov 29: Final project presentations 2

Previous Offerings

A full version of this course was offered in Spring 2017. Lecture videos from Spring 2017, are available here

An abbreviated version of this course was offered in Fall 2015.

Prerequisites

CS189 or equivalent is a prerequisite for the course. This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We’ll review this material in class, but it will be rather cursory.

  • Reinforcement learning and MDPs
    • Definition of MDPs
    • Exact algorithms: policy and value iteration
    • Search algorithms
  • Numerical Optimization
    • gradient descent, stochastic gradient descent
    • backpropagation algorithm
  • Machine Learning
    • Classification and regression problems: what loss functions are used, how to fit linear and nonlinear models
    • Training/test error, overfitting.

For introductory material on RL and MDPs, see

For introductory material on machine learning and neural networks, see

John’s lecture series at MLSS

  • Lecture 1: intro, derivative free optimization
  • Lecture 2: score function gradient estimation and policy gradients
  • Lecture 3: actor critic methods
  • Lecture 4: trust region and natural gradient methods, open problems

Courses

Relevant Textbooks


comments powered by Disqus