• Home

  • Documents

  • Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy...
31
Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL Deep Learning at Supercomputer Scale | NIPS Workshop

Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

  • Upload
    others

  • View
    19

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Deep Reinforcement Learning at ScaleTimothy Lillicrap

Research Scientist, DeepMind & UCL

Deep Learning at Supercomputer Scale | NIPS Workshop

Page 2: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

What is Reinforcement Learning?

Supervised Learning Reinforcement Learning

Fixed dataset Data depends on actions taken in environment

Page 3: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Formalizing the Agent-Environment Loop

Environment

Actions

Observations

RewardsAgent

Neural Network(s)

Page 4: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Advantage Actor-Critic (A3C)

Mnih et al., ICML 2016

Page 5: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

A Single Trial (with Advantage Actor-Critic)

Mnih et al., ICML 2016

Time

Page 6: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Combating Variance: Advantage Actor-Critic

Mnih et al., ICML 2016

Page 7: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Scaling Reinforcement Learning (A3C)

Actor / LearnerParameter

ServerGradients

Parameters

Mnih et al., ICML 2016

Page 8: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Scaling Reinforcement Learning

Replay BufferLearner(s)

Actors

Parameters Experience+

Initial Priorities

Updated Priorities

Experience

Horgan et al., 2017 & Schaul et al. 2015

Page 9: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Off-policy Actor-Critic for Continuous Actions

Lillicrap et al., ICLR 2016

Page 10: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Distributional Distributed DDPG (D4PG)

Page 11: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 12: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 13: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 14: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 15: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 16: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Page 17: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

Distributional Distributed DDPG (D4PG)

Page 18: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

https://docs.google.com/file/d/0B_lYSIeNJrtQSlpBdjNpY3JDMDQ/preview
https://docs.google.com/file/d/0B_lYSIeNJrtQUTJINGpZUUNJbk0/preview
Page 19: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Hoffman, Barth-Maron et al., 2017

https://docs.google.com/file/d/1km6q0aT8VyZ-z-xfbMRxKdlaCuZ1Axtl/preview
https://docs.google.com/file/d/1hl_pL2bFStlwMjSvvKibuCXYtcwLC_yB/preview
Page 20: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Huang et al., Nature, 2016

Playing Go with Deep Networks and Planning

Use environment model in order to plan!

Page 21: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Huang et al., Nature, 2016

Training Policy and Value Networks

Page 22: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Huang et al., Nature, 2016

Planning with an Environment Model & MCTS

Page 23: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Huang et al., Nature, 2016

Planning with an Environment Model

Page 24: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 25: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

z

Page 26: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 27: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 28: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 29: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 30: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Silver, Schrittwieser, Simonyan, et al. Nature, 2017

Playing Go with Without Human Knowledge

Page 31: Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Questions?


Deep Reinforcement Learning: Q-Learning - Svetlana …slazebni.cs.illinois.edu/spring17/lec17_rl.pdf · Deep Reinforcement Learning: Q-Learning Garima Lalwani Karan Ganju Unnat Jain

Deep Reinforcement Learning: Q-Learning - Svetlana …slazebni.cs.illinois.edu/spring17/lec17_rl.pdf · Deep Reinforcement Learning: Q-Learning Garima Lalwani Karan Ganju Unnat Jain

Documents
Thinking While Moving: Deep Reinforcement Learning in ... While...Thinking While Moving: Deep Reinforcement Learning in Concurrent Environments. Thinking While Moving: Deep Reinforcement

Thinking While Moving: Deep Reinforcement Learning in ... While...Thinking While Moving: Deep Reinforcement Learning in Concurrent Environments. Thinking While Moving: Deep Reinforcement

Documents
Deep Reinforcement Learning and Control

Deep Reinforcement Learning and Control

Documents
Deep Reinforcement Learning

Deep Reinforcement Learning

Science
Deep Reinforcement Learning in System Optimization · arXiv:1908.01275v2 [cs.LG] 7 Aug 2019. Deep Reinforcement Learning in System Optimization Figure 1. Reinforcement learning. By

Deep Reinforcement Learning in System Optimization · arXiv:1908.01275v2 [cs.LG] 7 Aug 2019. Deep Reinforcement Learning in System Optimization Figure 1. Reinforcement learning. By

Documents
Hierarchical Deep Reinforcement Learning: Integrating ... · Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction ... expected future intrinsic (controller)

Hierarchical Deep Reinforcement Learning: Integrating ... · Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction ... expected future intrinsic (controller)

Documents
Deep Reinforcement Learning with Double Q-Learning

Deep Reinforcement Learning with Double Q-Learning

Documents
Deep Reinforcement Learning - University of Waterloo · PDF file–CS885: Deep Reinforcement Learning •Instructor: Pascal Poupart •Spring 2018 •Deep Q Networks, Recurrent deep

Deep Reinforcement Learning - University of Waterloo · PDF file–CS885: Deep Reinforcement Learning •Instructor: Pascal Poupart •Spring 2018 •Deep Q Networks, Recurrent deep

Documents
Reinforcement and deep reinforcement learning for wireless

Reinforcement and deep reinforcement learning for wireless

Documents
Intro to Deep Reinforcement Learning

Intro to Deep Reinforcement Learning

Data & Analytics
Deep Reinforcement Learning - Cross Entropy

Deep Reinforcement Learning - Cross Entropy

Documents
Reinforcement Learning: CNNs and Deep Q Learning

Reinforcement Learning: CNNs and Deep Q Learning

Documents
From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Documents
Deep Learning and Reinforcement Learning Workflows in A.I. · Deep Learning and Reinforcement Learning Workflows in A.I. Jon Cherrie Software Engineering Manager Deep Learning Toolbox

Deep Learning and Reinforcement Learning Workflows in A.I. · Deep Learning and Reinforcement Learning Workflows in A.I. Jon Cherrie Software Engineering Manager Deep Learning Toolbox

Documents
Deep Learning and Reinforcement Learning

Deep Learning and Reinforcement Learning

Technology
Agente Sonic. Deep reinforcement learning

Agente Sonic. Deep reinforcement learning

Documents
Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Documents
Deep Reinforcement Learning from Human Preferencespapers.nips.cc/...deep-reinforcement-learning-from-human-preferences.pdf · Deep Reinforcement Learning from Human Preferences Paul

Deep Reinforcement Learning from Human Preferencespapers.nips.cc/...deep-reinforcement-learning-from-human-preferences.pdf · Deep Reinforcement Learning from Human Preferences Paul

Documents
A Deep Reinforcement Learning Chatbot

A Deep Reinforcement Learning Chatbot

Data & Analytics
Deep Reinforcement Learning An Introduction

Deep Reinforcement Learning An Introduction

Technology
Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Documents
Deep Reinforcement Learning of Video Games...the eld of machine learning and reinforcement learning, in particular deep reinforcement learning. 1.2.1 Architectural neural network design

Deep Reinforcement Learning of Video Games...the eld of machine learning and reinforcement learning, in particular deep reinforcement learning. 1.2.1 Architectural neural network design

Documents
Deep Reinforcement Learning that Matters

Deep Reinforcement Learning that Matters

Documents
DEEP FEATURE EXTRACTION FOR SAMPLE-EFFICIENT REINFORCEMENT … · Deep Reinforcement Learning Il deep reinforcement learning (DRL) `e una branca dell’apprendimento per rinforzo

DEEP FEATURE EXTRACTION FOR SAMPLE-EFFICIENT REINFORCEMENT … · Deep Reinforcement Learning Il deep reinforcement learning (DRL) `e una branca dell’apprendimento per rinforzo

Documents
10703 Deep Reinforcement Learning

10703 Deep Reinforcement Learning

Documents
Survey of Deep Reinforcement Learning for Motion Planning ... · Reinforcement Learning Autonomous vehicles Fig. 1: Web of Science topic search for ”Deep Reinforcement Learning”

Survey of Deep Reinforcement Learning for Motion Planning ... · Reinforcement Learning Autonomous vehicles Fig. 1: Web of Science topic search for ”Deep Reinforcement Learning”

Documents
Tutorial: Deep Reinforcement Learning · 2018-09-10 · Tutorial: Deep Reinforcement Learning David Silver, Google DeepMind. Outline Introduction to Deep Learning Introduction to

Tutorial: Deep Reinforcement Learning · 2018-09-10 · Tutorial: Deep Reinforcement Learning David Silver, Google DeepMind. Outline Introduction to Deep Learning Introduction to

Documents
Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Documents
Hangzhou Deep Learning Meetup-Deep Reinforcement Learning

Hangzhou Deep Learning Meetup-Deep Reinforcement Learning

Technology
Deep Reinforcement Learning for Robotics

Deep Reinforcement Learning for Robotics

Documents