Article Source

• Title: CS 294: Deep Reinforcement Learning, Spring 2017
• Authors: Sergey Levine, John Schulman, Chelsea Finn

CS 294: Deep Reinforcement Learning, Spring 2017

Instructors: Sergey Levine, John Schulman, Chelsea Finn Lectures: Mondays and Wednesdays, 9:00am-10:30am in 306 Soda Hall. Office Hours: MW 10:30-11:30, by sign-up only, room TBD Communication: Piazza will be used for announcements, general questions and discussions, clarifications about assignments, student questions to each other, and so on. To sign up, go to Piazza and sign up with “UC Berkeley” and “CS294-112”. The course is now full, and enrollment has closed. For people who are not enrolled, but interested in following and discussing the course, there is a subreddit forum here: reddit.com/r/berkeleydeeprlcourse/

Prerequisites

This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We’ll review this material in class, but it will be rather cursory.

• Reinforcement learning and MDPs
  • Definition of MDPs
  • Exact algorithms: policy and value iteration
  • Search algorithms
• Numerical Optimization
  • gradient descent, stochastic gradient descent
  • backpropagation algorithm
• Machine Learning
  • Classification and regression problems: what loss functions are used, how to fit linear and nonlinear models
  • Training/test error, overfitting.

For introductory material on RL and MDPs, see

• CS188 EdX course, starting with Markov Decision Processes I
• Sutton & Barto, Ch 3 and 4.
• For a concise intro to MDPs, see Ch 1-2 of Andrew Ng’s thesis
• David Silver’s course, links below

For introductory material on machine learning and neural networks, see

• Andrej Karpathy’s course
• Geoff Hinton on Coursera
• Andrew Ng on Coursera
• Yaser Abu-Mostafa’s course

Syllabus

Below you can find an outline of the course. Slides and references will be posted as the course proceeds.

Lecture Videos

The course may be recorded this year. John also gave a lecture series at MLSS, and videos are available:

• Lecture 1: intro, derivative free optimization
• Lecture 2: score function gradient estimation and policy gradients
• Lecture 3: actor critic methods
• Lecture 4: trust region and natural gradient methods, open problems

Related Materials

Courses

• Dave Silver’s course on reinforcement learning / Lecture Videos
• Nando de Freitas’ course on machine learning
• Andrej Karpathy’s course on neural networks

Relavent Textbooks

• Sutton & Barto, Reinforcement Learning: An Introduction
• Szepesvari, Algorithms for Reinforcement Learning
• Bertsekas, Dynamic Programming and Optimal Control, Vols I and II
• Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming
• Powell, Approximate Dynamic Programming

Misc Links

• A collection of deep learning resources

Previous Offerings

An abbreviated version of this course was offered in Fall 2015.