• Home

  • Documents

  • FeUdalNetworks for Hierarchical Reinforcement Learning
26
FeUdal Networks for Hierarchical Reinforcement Learning by Artem Bachysnkyi Computational Neuroscience Seminar University of Tartu 3 May 2017

FeUdalNetworks for Hierarchical Reinforcement Learning

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: FeUdalNetworks for Hierarchical Reinforcement Learning

FeUdal NetworksforHierarchicalReinforcementLearning

byArtem BachysnkyiComputationalNeuroscienceSeminar

UniversityofTartu3May2017

Page 2: FeUdalNetworks for Hierarchical Reinforcement Learning

Reinforcementlearning

The basic reinforcement learning model consists of:

• a set of environment and agent states S • a set of actions A of the agent• policies of transitioning from states to actions• rules that determine the scalar immediate reward of a

transition • rules that describe what the agent observes.

Page 3: FeUdalNetworks for Hierarchical Reinforcement Learning

ATARIgames

Page 4: FeUdalNetworks for Hierarchical Reinforcement Learning

Standart approach

• useanaction-repeatheuristic,whereeachactiontranslatesintoseveralconsecutiveactionsintheenvironment

• notapplicableinnon-Marcovian environmentsthatrequirememory• can’tlearnontheweakrewardsignal

Page 5: FeUdalNetworks for Hierarchical Reinforcement Learning

Feudalreinforcementlearningintuition

• levelsofhierarchywithinanagentcommunicateviaexplicitgoals• goalscanbegeneratedinatop-downfashion• goalsettingcanbedecoupledfromgoalachievement

Page 6: FeUdalNetworks for Hierarchical Reinforcement Learning

Manager-Workermodel

Manager:• setsgoalsatalowertemporalresolution

Worker:• operatesatahighertemporalresolution

• producesprimitiveactions• followsthegoalsbyanintrinsicreward

Page 7: FeUdalNetworks for Hierarchical Reinforcement Learning

Mainproposals

• aconsistent,end-to-enddifferentiablemodel• approximatetransitionpolicygradientupdatefortrainingtheManager• useofgoalsthataredirectionalratherthanabsolute• dilatedLSTMfortheManagerRNNdesign

Page 8: FeUdalNetworks for Hierarchical Reinforcement Learning

FuN modeldescription

Page 9: FeUdalNetworks for Hierarchical Reinforcement Learning

FuN modeldescription

ℎ",ℎ# – internalstates𝑈% – workersoutput𝜙 – maps𝑔% into𝑤%𝜋 – vectorofprobabilitiesoverprimitiveactions

𝑠% – latentstaterepresentation𝑔% – goalvector𝑥% – observationfromtheenvironment𝑧% – sharedintermediaterepresentation

Page 10: FeUdalNetworks for Hierarchical Reinforcement Learning

Learning

Learningsteps:1. receivesanobservationfromtheenvironment2. selectanactionfromafiniteset3. theenvironmentrespondswithanewobservationandascalar

reward4. theprocesscontinuesuntiltheterminalstateisreached

Page 11: FeUdalNetworks for Hierarchical Reinforcement Learning

LearningBadidea:

trainfeudalnetworkend-to-endusingapolicygradientalgorithmoperatingontheactionstakenbytheWorker

Goodidea:independentlytrainManagertopredictadvantageousdirectionsinstatespaceandtointrinsicallyrewardtheWorkertofollowthesedirections

Page 12: FeUdalNetworks for Hierarchical Reinforcement Learning

Theagentsgoal

Maximizethediscountedreturn

where

Theagent’sbehaviour isdefinedbyitsaction-selectionpolicyπ.FuN producesadistributionoverpossibleactions.

Page 13: FeUdalNetworks for Hierarchical Reinforcement Learning

Managersupdaterule

where

– valuefunctionestimatefromtheinternalcritic

– cosinesimilarity

– advantagefunction

Page 14: FeUdalNetworks for Hierarchical Reinforcement Learning

Workersintrinsicreward

where

𝑐 – horizon

Page 15: FeUdalNetworks for Hierarchical Reinforcement Learning

TheWorkerspolicy

Advantageauthorcritic

Advantagefunction

Page 16: FeUdalNetworks for Hierarchical Reinforcement Learning

Architecturedetails

𝑓012310%– ConvolutionalNeuralNetwork:1. 168x8filters,stride42.324x4fil- ters ofstride23.fullyconnectedlayerhas256hiddenunits*eachlayerisfollowedbyarectifiednon-linearity

𝑓"40531 – anotherfullyconnectedlayer𝑓#266 – standardLSTM𝑓"266 – dilatedLSTM

Page 17: FeUdalNetworks for Hierarchical Reinforcement Learning

FuN modeldescription

Page 18: FeUdalNetworks for Hierarchical Reinforcement Learning

DilatedLSTMStateofthenetworkwith𝑟 separategroupsofsub-states

Attime𝑡 wecanindicatewich groupofcoresisupdated

Ateachtimesteponlythecorrespondingpartofthestateisupdatedandtheoutputispooledacrossthepreviouscoutputs.ThisallowsthergroupsofcoresinsidethedLSTM topreservethememoriesforlongperiods.

*Intheexperimentsr=10.

Page 19: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:ATARI

Page 20: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:Montezuma’srevenge

https://www.youtube.com/watch?v=_zbg9rs5QZY

Page 21: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:Montezuma’srevenge

Page 22: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:Non-matchandT-maze

Page 23: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:Watermaze

Page 24: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:transitionpolicygradient

Page 25: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:Temporalresolution

Page 26: FeUdalNetworks for Hierarchical Reinforcement Learning

Experiments:DilateLSTMagentbaseline


Hierarchical Deep Reinforcement Learning: Integrating ... · Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction ... expected future intrinsic (controller)

Hierarchical Deep Reinforcement Learning: Integrating ... · Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction ... expected future intrinsic (controller)

Documents
Model Minimization in Hierarchical Reinforcement Learning

Model Minimization in Hierarchical Reinforcement Learning

Documents
Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Documents
Hierarchical Object Detection with Deep Reinforcement Learning

Hierarchical Object Detection with Deep Reinforcement Learning

Data & Analytics
Hierarchical Reinforcement Learning with …papers.nips.cc/paper/8421-hierarchical-reinforcement...these challenging tasks [6]. In addition, Hierarchical Reinforcement Learning (HRL)

Hierarchical Reinforcement Learning with …papers.nips.cc/paper/8421-hierarchical-reinforcement...these challenging tasks [6]. In addition, Hierarchical Reinforcement Learning (HRL)

Documents
Hierarchical Reinforcement Learning for Course Recommendation …lfs.aminer.cn/misc/moocdata/publications/AAAI19-zhang-et... · 2019. 4. 3. · Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning for Course Recommendation …lfs.aminer.cn/misc/moocdata/publications/AAAI19-zhang-et... · 2019. 4. 3. · Hierarchical Reinforcement Learning

Documents
FeUdal Networks for Hierarchical Reinforcement Learningproceedings.mlr.press/v70/vezhnevets17a/vezhnevets17a.pdf · FeUdal Networks for Hierarchical Reinforcement Learning ager and

FeUdal Networks for Hierarchical Reinforcement Learningproceedings.mlr.press/v70/vezhnevets17a/vezhnevets17a.pdf · FeUdal Networks for Hierarchical Reinforcement Learning ager and

Documents
Hierarchical Reinforcement Learning with Unlimited Recursive ...Hierarchical reinforcement learning architecture RGoal • In RGoal, an agent's subgoal settings are similar to subroutine

Hierarchical Reinforcement Learning with Unlimited Recursive ...Hierarchical reinforcement learning architecture RGoal • In RGoal, an agent's subgoal settings are similar to subroutine

Documents
Hierarchical Reinforcement Learning for Self-Driving

Hierarchical Reinforcement Learning for Self-Driving

Documents
Hierarchical Reinforcement Learning ... - xiaojingzi.github.io · Hierarchical Reinforcement Learning for Course Recommendation in MOOCs Jing Zhang1,2, Bowen Hao1,2, Bo Chen1,2, Cuiping

Hierarchical Reinforcement Learning ... - xiaojingzi.github.io · Hierarchical Reinforcement Learning for Course Recommendation in MOOCs Jing Zhang1,2, Bowen Hao1,2, Bo Chen1,2, Cuiping

Documents
Hierarchical Reinforcement Learning with Parameters · 2020-05-19 · Hierarchical Reinforcement Learning with Parameters Maciej Klimek deepsense.ai Piotr Miło´s University of Warsaw

Hierarchical Reinforcement Learning with Parameters · 2020-05-19 · Hierarchical Reinforcement Learning with Parameters Maciej Klimek deepsense.ai Piotr Miło´s University of Warsaw

Documents
Hierarchical Reinforcement Learning - WSUtaylorm/14_580/Beiyu.pdf · 2014. 4. 15. · Reinforcement Learning Hierarchical Reinforcement Learning: To discouver and exploit hierarchical

Hierarchical Reinforcement Learning - WSUtaylorm/14_580/Beiyu.pdf · 2014. 4. 15. · Reinforcement Learning Hierarchical Reinforcement Learning: To discouver and exploit hierarchical

Documents
Data-Efficient Hierarchical Reinforcement Learning · 2019-02-19 · Data-Efficient Hierarchical Reinforcement Learning Ofir Nachum Google Brain ofirnachum@google.com Shixiang Gu

Data-Efficient Hierarchical Reinforcement Learning · 2019-02-19 · Data-Efficient Hierarchical Reinforcement Learning Ofir Nachum Google Brain [email protected] Shixiang Gu

Documents
Hierarchical Bayesian Methods for Reinforcement Learning · Hierarchical Bayesian Methods for Reinforcement Learning. My Research: ... Intuitive theories ... Power grid control

Hierarchical Bayesian Methods for Reinforcement Learning · Hierarchical Bayesian Methods for Reinforcement Learning. My Research: ... Intuitive theories ... Power grid control

Documents
Intrinsically Motivated Hierarchical Reinforcement Learning

Intrinsically Motivated Hierarchical Reinforcement Learning

Documents
Mechanisms of Hierarchical Reinforcement …ski.clps.brown.edu › papers › FrankBadre12.pdfAdvance Access publication June 21, 2011 Mechanisms of Hierarchical Reinforcement Learning

Mechanisms of Hierarchical Reinforcement …ski.clps.brown.edu › papers › FrankBadre12.pdfAdvance Access publication June 21, 2011 Mechanisms of Hierarchical Reinforcement Learning

Documents
Learning (Part II) Hierarchical Reinforcement

Learning (Part II) Hierarchical Reinforcement

Documents
Hierarchical Reinforcement Learning for Air-to-Air Combat

Hierarchical Reinforcement Learning for Air-to-Air Combat

Documents
Reinforcement Learning of Hierarchical Skills on the Sony Aibo …web.eecs.umich.edu/~baveja/Papers/icdl06.pdf · Reinforcement Learning of Hierarchical Skills on the Sony Aibo robot

Reinforcement Learning of Hierarchical Skills on the Sony Aibo …web.eecs.umich.edu/~baveja/Papers/icdl06.pdf · Reinforcement Learning of Hierarchical Skills on the Sony Aibo robot

Documents
Hierarchical Imitation and Reinforcement Learning · 2018. 6. 1. · Hierarchical Imitation and Reinforcement Learning Ziebart et al.,2008;Syed & Schapire,2008;Ho & Ermon, 2016) assumes

Hierarchical Imitation and Reinforcement Learning · 2018. 6. 1. · Hierarchical Imitation and Reinforcement Learning Ziebart et al.,2008;Syed & Schapire,2008;Ho & Ermon, 2016) assumes

Documents
Hierarchical Reinforcement Learning Ersin Basaran 19/03/2005

Hierarchical Reinforcement Learning Ersin Basaran 19/03/2005

Documents
Recent Advances in Hierarchical Reinforcement Learningprlt.dei.polimi.it/upload/5/5c/Hrl.pdf · Recent Advances in Hierarchical Reinforcement Learning Authors: ... Artificial Intelligence

Recent Advances in Hierarchical Reinforcement Learningprlt.dei.polimi.it/upload/5/5c/Hrl.pdf · Recent Advances in Hierarchical Reinforcement Learning Authors: ... Artificial Intelligence

Documents
Hierarchical Deep Reinforcement Learning: Integrating ......Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation Tejas D. Kulkarni DeepMind,

Hierarchical Deep Reinforcement Learning: Integrating ......Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation Tejas D. Kulkarni DeepMind,

Documents
Introduction to Hierarchical Reinforcement Learning

Introduction to Hierarchical Reinforcement Learning

Documents
Hierarchical Approaches to Reinforcement Learning Using ... · the decision space (hierarchical reinforcement learning). In this work we ask whether we can use ideas introduced in

Hierarchical Approaches to Reinforcement Learning Using ... · the decision space (hierarchical reinforcement learning). In this work we ask whether we can use ideas introduced in

Documents
Latent Space Policies for Hierarchical Reinforcement Learning

Latent Space Policies for Hierarchical Reinforcement Learning

Documents
Hierarchical Reinforcement Learning Explains Task Interleaving … · 2020. 11. 5. · Hierarchical reinforcement learning model for task interleaving Introduction How long will you

Hierarchical Reinforcement Learning Explains Task Interleaving … · 2020. 11. 5. · Hierarchical reinforcement learning model for task interleaving Introduction How long will you

Documents
Hierarchical Deep Reinforcement Learning through Scene Decomposition … · 2020. 8. 19. · Hierarchical Deep Reinforcement Learning through Scene Decomposition for Autonomous Urban

Hierarchical Deep Reinforcement Learning through Scene Decomposition … · 2020. 8. 19. · Hierarchical Deep Reinforcement Learning through Scene Decomposition for Autonomous Urban

Documents
Recent Advances in Hierarchical Reinforcement Learningincompleteideas.net/rlai.cs.ualberta.ca/papers/barto03recent.pdf · Recent Advances in Hierarchical Reinforcement Learning Andrew

Recent Advances in Hierarchical Reinforcement Learningincompleteideas.net/rlai.cs.ualberta.ca/papers/barto03recent.pdf · Recent Advances in Hierarchical Reinforcement Learning Andrew

Documents
A Non-Strict Hierarchical Reinforcement Learning for Interactive

A Non-Strict Hierarchical Reinforcement Learning for Interactive

Documents