View
10
Download
0
Category
Preview:
Citation preview
Learning in sequential environments
Raia HadsellStaff Research Scientist, DeepMind
raiahadsell.com
Scaling deep reinforcement learning towards the real world:
part 1: learning sequential tasks without forgettingpart 2: learning to navigate in complex worlds
Raia Hadsell 2017
EnvironmentAgent
Reinforcement Learning
OBSERVATIONS
ACTIONS
REWARD?
Raia Hadsell 2017
○ Maximizing Qπ(s,a) over possible policies gives the optimal
action-value function and the Bellman equation:
○ Basic idea:
■ Approximate →
■ Apply the Bellman Equation as an iterative update
Value Iteration
Raia Hadsell 2017
○ Use a neural network for Q(s,a; )
○ Train end-to-end from raw pixels
End-to-End Reinforcement Learning
Raia Hadsell 2017
but.. a network for every task?
Raia Hadsell 2017
one network for all?
Raia Hadsell 2017
Catastrophic forgetting
● Well-known phenomenon● Especially severe in Deep RL
Raia Hadsell 2017
Catastrophic forgetting
https://www.youtube.com/watch?v=Fh_zNpdc0Xs
Raia Hadsell 2017
Catastrophic forgetting
https://www.youtube.com/watch?v=yk_sW4x6zb0https://www.youtube.com/watch?v=V4oT1Ei-8_khttps://www.youtube.com/watch?v=LjFGy4BxOL8
Raia Hadsell 2017
An illustration
Task B
*
Task A
SGD
EWC
L2
Raia Hadsell 2017
Elastic Weight Consolidation
Task B
*
Task A
Elastic Weight Consolidation (EWC):
Constrain important parameters
to stay close to their old values
Continual learning in the brain:
Synaptic consolidation reduces
the plasticity of synapses that are
vital to previous tasks.
SGD
EWC
L2
Raia Hadsell 2017
Elastic Weight Consolidation
Implement constraint as a quadratic penalty
that is applied while training on B, but not
uniformly - rather, should be greater for
important parameters of Task A.
Posterior distribution
contains exactly this,
but is intractable. Task B
*
Task A
SGD
EWC
L2
Raia Hadsell 2017
Estimate posterior with Gaussian.
Mean: parameter vector *A
Diagonal precision given by approximation
of the Fisher Information F.
Elastic Weight Consolidation
Task B
*
Task A
SGD
EWC
L2
Raia Hadsell 2017
Elastic Weight Consolidation
Task B
*
Task A
SGD
EWC
L2
Raia Hadsell 2017
Experiment: Permuted MNIST
Random, fixed permutations of MNIST dataset.
Train a multilayer, fully-connected network with ReLus until convergence
We compare SGD, L2 regularisation, and EWC.
Perm A Perm B Perm C
Raia Hadsell 2017
Experiment: Permuted MNIST
Raia Hadsell 2017
Experiment: Permuted MNIST
Raia Hadsell 2017
Experiment: Permuted MNIST
Raia Hadsell 2017
Experiment: Permuted MNIST
Raia Hadsell 2017
Let’s try something harder...
Sequential reinforcement learning tasks (10 Atari games)
Random ordering with extended game play on each task, multiple returns
Unknown task boundaries
Regular testing of all 10 games
Single network with fixed capacity
Raia Hadsell 2017
Experiment: Atari 10
Forget-Me-Not1 allows labeling of data
segments, used for
● EWC regularisation
● Task-specific replay buffers used for
DDQN2
● Task-specific bias and gains at each
network layer
Fisher estimated at each task
boundary and EWC penalty is updated
[1] The forget-me-not process, Milan et al., NIPS 2016[2] Deep reinforcement learning with double q-learning, Hasselt et al., AAAI 2016
https://www.youtube.com/watch?v=Ry2WRcnwsYU
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia
Clopath, Dharshan Kumaran, Raia Hadsell
Overcoming catastrophic forgetting in neural networks
PNAS 2017arxiv.org/abs/1612.00796
Raia Hadsell 2017
Learning to navigate in complex mazes
Raia Hadsell 2017
Navigation mazesGame episode:
1. Random start2. Find the goal (+10)3. Teleport randomly4. Re-find the goal (+10)5. Repeat (limited time)
+10 +1
Raia Hadsell 2017
Navigation mazesGame episode:
1. Random start2. Find the goal (+10)3. Teleport randomly4. Re-find the goal (+10)5. Repeat (limited time)
+10 +1
Variants:● Static maze, static goal● Static maze, random goal● Random maze
Observations: RGB, velocityActions: 8
Raia Hadsell 2017
Navigation mazesGame episode:
1. Random start2. Find the goal (+10)3. Teleport randomly4. Re-find the goal (+10)5. Repeat (limited time)
3600 steps/episode
Variants:● Static maze, static goal● Static maze, random goal● Random maze
Observations: RGB, velocityActions: 8
Raia Hadsell 2017
Navigation mazes Game episode:
1. Random start2. Find the goal (+10)3. Teleport randomly4. Re-find the goal (+10)5. Repeat (limited time)
3600 steps/episode
10800 steps/episode
3600 steps/episode
Variants:● Static maze, static goal● Static maze, random goal● Random maze
Observations: RGB, velocityActions: 8
Raia Hadsell 2017
The vast and meaningless silence of an agent exploring...
Given: Sparse rewards
Wanted:Spatial knowledge
1e7
I have been here before! I know where to go!
1e7
Why is learning navigation via reinforcement learning hard?
Raia Hadsell 2017
Given:Sparse rewards
Wanted:Spatial knowledge
1. Accelerate reinforcement learning through auxiliary losses➔ Stable gradients help learning, even if unrelated to reward
2. Drive spatial knowledge through choice of auxiliary tasks:● Depth prediction● Loop closure prediction
Why is learning navigation via reinforcement learning hard?
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
enc
xt
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
enc
xt
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
enc
xt rt-1 {vt, at-1}
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
4. RL: Asynchronous advantage actor critic (A3C)
enc
xt rt-1 {vt, at-1}
Mnih et al. (2016)
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
4. RL: Asynchronous advantage actor critic (A3C)
5. Aux task 1: Depth predictors
enc
Depth (D1 )
xt rt-1 {vt, at-1}
Depth (D2 )
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
4. RL: Asynchronous advantage actor critic (A3C)
5. Aux task 1: Depth predictors
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
4. RL: Asynchronous advantage actor critic (A3C)
5. Aux task 1: Depth predictor
6. Aux task 2: Loop closure predictor enc
Loop (L)
Depth (D1 )
xt rt-1 {vt, at-1}
Depth (D2 )
Raia Hadsell 2017
Nav agent ingredients:
1. Convolutional encoder and RGB inputs
2. Stacked LSTM
3. Additional inputs (reward, action, and velocity)
4. RL: Asynchronous advantage actor critic (A3C)
5. Aux task 1: Depth predictor
6. Aux task 2: Loop closure predictor
7. For analysis: Position decoder
enc
Loop (L)
Depth (D1 )
xt rt-1 {vt, at-1}
Depth (D2 ) Position
Raia Hadsell 2017
details..
Raia Hadsell 2017
more details.. policy gradient:
depth prediction from visual features:
depth prediction from LSTM features:
loop prediction from LSTM features:
Raia Hadsell 2017
Experiments
xt rt-1 {vt, at-1}
enc
xt
enc enc
Loop (L)
Depth (D1 )
a. FF A3C c. Nav A3C d. Nav A3C +D1D2L
xt rt-1 {vt, at-1}
enc
xt
b. LSTM A3C
Depth (D2 )
Raia Hadsell 2017
+10 +1
Results on large maze with static goal
https://www.youtube.com/watch?v=zHhbypmKaj0
Raia Hadsell 2017
Should depth be an input? Or a target?
rgbdt rt-1 {vt, at-1}
enc enc
Depth (D1 )
rgbt rt-1 {vt, at-1}
Depth (D2 )
Answer: the dense, non-noisy gradients from depth as a target are more helpful
Raia Hadsell 2017
Results with Random Goal locations
Is the agent remembering the goal
location?
● Mean time to first goal find of episode:
14.0 sec
● Mean time to subsequent goal finds:
7.2 sec
● Not as impressive for large mazes:
15.4 sec vs 15.0 sec.
small
large
Raia Hadsell 2017
Latency to goal (as the agent returns)
● Trajectories of the Nav A3C+D+L agent in the I-maze and random goal maze over the course of one episode
● Value function and goal finding (red lines) are shown
Raia Hadsell 2017
Position decoding
● Trajectories of the Nav A3C+D+L agent in the random goal maze
● Position likelihoods are overlaid (predicted from LSTM hiddens)
● Initial uncertainty gives way to accurate position estimation.
enc
���� L
D1
xt rt-1 {vt, at-1}
D2Position
Raia Hadsell 2017
Results in random mazessmall
large
https://www.youtube.com/watch?v=EKXQAjoNdGM
Thank you!raiahadsell.com
Piotr Mirowski Razvan Pascanu
Fabio Ross Andy Hubert Laurent Koray Dharsh Misha Andrea
Learning to navigate in complex environments
ICLR2017arxiv.org/abs/1611.03673
Recommended