Article Source

• Decision Transformer; Reinforcement Learning via Sequence Modeling

Reinforcement Learning via Sequence Modeling

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Speaker

Aditya Grover is a research scientist in the Core ML team at Facebook AI Research, a visiting postdoctoral researcher at UC Berkeley, and an incoming assistant professor of computer science at UCLA. His research centers around foundations of probabilistic machine learning for unsupervised representation learning and sequential decision making, and is grounded in applications at the intersection of physical sciences and climate change. His research has been recognized with a best paper award, several research fellowships (Microsoft Research, Lieberman, Google-Simons-Berkeley, Adobe), a best undergraduate thesis award, and the ACM SIGKDD doctoral dissertation award. He is also a recipient of the Gores Award – Stanford’s highest university-level distinction in teaching for faculty and students. Aditya received his PhD from Stanford and bachelors from IIT Delhi, both in computer science.