29
An Introduction to Reinforcement Learning (RL) and RL Brain Machine Interface (RL-BMI) Aditya Tarigoppula www.joefrancislab.com SUNY Downstate Medical Center

An introduction to reinforcement learning (rl)

  • Upload
    pauldix

  • View
    6.392

  • Download
    0

Embed Size (px)

DESCRIPTION

Slides from Aditya Tarigoppula's talk at NYC Machine Learning on December 13th.

Citation preview

Page 1: An introduction to reinforcement learning (rl)

An Introduction to Reinforcement Learning (RL) and RL Brain Machine Interface (RL-BMI)

Aditya Tarigoppula www.joefrancislab.com

SUNY Downstate Medical Center

Page 2: An introduction to reinforcement learning (rl)

Outline

RL Examples

Environment

Value functions

Optimality

Methods for attaining optimality

DP MC TD

BMI & RL-BMI

Eligibility Traces

START / END

Page 3: An introduction to reinforcement learning (rl)

RL Examples

Stanford Autonomous Helicopterhttp://heli.stanford.edu/

Reinforcement Learning Brain Machine InterfaceJoe Francis Lab.

http://heli.stanford.edu/
Page 4: An introduction to reinforcement learning (rl)

Environment model - Markov decision process 1) States ‘S’

2) Actions ‘A’

3) State transition probabilities.

4) Reward

5)

Deterministic, non-stationary policy

RL Problem: The decision maker, ‘agent’ needs to learn the optimum policy in an ‘environment’ to maximize the total amount of reward it receives over the long term.

1'

'},,|'1Pr{' s

assPaatsstssta

ssP

10

...32

21

...321

trtrtrtR

TtrtrtrtrtR

as :

Page 5: An introduction to reinforcement learning (rl)

• Agent performs the action under the policy being followed.

• Environment is everything else other than the agent.

assP '

Page 6: An introduction to reinforcement learning (rl)

Value Functions: State Value Function

State – Action Value Function

},|{

},|{),(

10

aassrE

aassREasQ

ttktk

k

ttt

)]'([),(

}|{

}|{)(

''

'

20

1

sVRPas

ssrrE

ssREsV

ass

s

ass

a

tktk

kt

tt

Page 7: An introduction to reinforcement learning (rl)

Optimal Value Function:

Optimal Policy – A policy that is better than or equal to all the other policies is called Optimal policy.

(in the sense of maximizing expected reward)

Optimal state value function

Optimal state-action value function

Bellman optimality equation

)(max)(* sVsV

),(max),(* asQasQ

},|)','(max{),(

},|)'({max)(

*1

*

*1

*

aassasQrEasQ

aasssVrEsV

tta

t

ttta

Page 8: An introduction to reinforcement learning (rl)

At time = tAcquire Brain State

DecoderAction Selection (trying to execute an optimum action)

Action executedAt time = t +1Observe reward Update the decoder

t

t+1

EXAMPLE

Page 9: An introduction to reinforcement learning (rl)

8.0Pr

1.0Pr 1.0Pr

0Pr

EXAMPLE

))](*),((*1.0))(*...

)...,((*1.0))(*),((*8.0[)(

332

211

sVasRsV

asRsVasRsV

1S

2S3S

4S

Prof. Andrew Ng, Lecture 16, Machine learning

Page 10: An introduction to reinforcement learning (rl)

Outline

Environment

Value functions

Optimality

Methods for attaining optimality

DP MC TD

BMI & RL-BMI

Eligibility Traces

START / END

We're here !

RL Examples

Page 11: An introduction to reinforcement learning (rl)

Solution Methods for RL problem◦ Dynamic Programming (DP) – is a method for optimization of

problems which exhibit the characteristics of overlapping sub problems and optimal substructure.

◦ Monte Carlo method (MC) - requires only experience--sample sequences of states, actions, and rewards from interaction with an environment.

◦ Temporal Difference learning (TD) – is a method that combines the better aspects of DP (estimation) and MC (experience) without incorporating the ‘troublesome’ aspects of both.

Page 12: An introduction to reinforcement learning (rl)

Dynamic ProgrammingPolicy Evaluation

Page 13: An introduction to reinforcement learning (rl)

Dynamic ProgrammingPolicy Improvement

)())(,( ' sVssQ

*1 *

10 ...... VVVEIIEIE

o

E – Policy Evaluation I – Policy Improvement

Page 14: An introduction to reinforcement learning (rl)

Policy Iteration Value Iteration

Replace entire section with

DYNAMIC

PROGRAMMING

)]'([max)( ''

' sVRPsV ass

s

ass

a

Page 15: An introduction to reinforcement learning (rl)

Monte Carlo Vs. DP

◦ The estimates for each state are independent. In other words, MC methods do not "bootstrap“.

◦ DP includes only one-step transitions

whereas the MC diagram goes all the

way to the end of the episode.

◦ The computational expense of estimating the value of a single state is less when one requires the value of only a subset of the states.

Page 16: An introduction to reinforcement learning (rl)

Monte Carlo Policy Evaluation

Every visit MC First visit MC

Page 17: An introduction to reinforcement learning (rl)

-> Without a model, we need Q value estimates.-> All state-action pairs should be visited.-> Exploration techniques 1) Exploring starts 2) e-greedy Policy

Next SlideMONTE

CARLO

Page 18: An introduction to reinforcement learning (rl)

As promised, this is the “NEXT SLIDE” !

MONTE

CARLO

Page 19: An introduction to reinforcement learning (rl)

Temporal Difference Methods◦ Like MC, TD methods can learn directly from raw experience

without a model of the environment's dynamics. Like DP, TD methods update estimates are based in part on other learned estimates, without waiting for a final outcome (they bootstrap).

)]()([)()( 11 ttttt sVsVrsVsV

Page 20: An introduction to reinforcement learning (rl)

TD(lambda)

trace decay parameter

Bias decreases

Variance Increases

Bias –Variance Tradeoff

Intuition: start with large ‘lamda’ and then decrease over time

Page 21: An introduction to reinforcement learning (rl)

SARSA

Q Learning

Difference

Page 22: An introduction to reinforcement learning (rl)

Outline

Environment

Value functions

Optimality

Methods for attaining optimality

DP MC TD

BMI & RL-BMI

Eligibility Traces

START / END

We're here !

RL Examples

Page 23: An introduction to reinforcement learning (rl)

Eligibility Traces

OR

Page 24: An introduction to reinforcement learning (rl)

Outline

Environment

Value functions

Optimality

Methods for attaining optimality

DP MC TD

BMI & RL-BMI

Eligibility Traces

START / END

We're here !

RL Examples

Page 25: An introduction to reinforcement learning (rl)

Online/closed loop RL-BMI architecture

),(),(

))]([max(_

actionsQasQ

tsiQindexoutputaction

titt

tanh(.)

reward

traceeerrTDdelta

asQasQrerrTD ttttt

_*_

),(),(*_ 11

‘delta’ used for updating the weights through back-propagation

NEURALSIGNAL

Page 26: An introduction to reinforcement learning (rl)

Scott, S. H. (1999). "Apparatus for measuring and perturbing shoulder and elbow joint positions and torques during reaching." J Neurosci Methods 89(2): 119-27.

BM I

SET UP

Page 27: An introduction to reinforcement learning (rl)

Autonomous Helicopter (Stanford Uni) http://heli.stanford.edu/papers/iser04-invertedflight.pdf

Position , orientation, velocity and angular velocity ),,,,,,,,,,,(

zyxzyx ),,,,,,,(

zyx

S1 S2

a1 a2R1 R2

Dynamics Dynamics

DynamicsRandom Gen

Dynamics

Page 28: An introduction to reinforcement learning (rl)

Actor-Critic Model

http://drugabuse.gov/researchreports/methamph/meth04.gif

http://drugabuse.gov/researchreports/methamph/meth04.gif
http://drugabuse.gov/researchreports/methamph/meth04.gif
Page 29: An introduction to reinforcement learning (rl)

References Reinforcement Learning: An Introduction

Richard S. Sutton & Andrew G. Barto Prof. Andrew Ng’s machine Learning Lectures http://heli.stanford.edu Dr. Joseph T. Francis

www.joefrancislab.com Prof. Peter Dayan Dr. Justin Sanchez Group

http://www.bme.miami.edu/nrg/


Reinforcement Learninglxmls.it.pt/2019/rl-intro.pdf · I Seq2seq reinforcement learning: Bandit structured prediction, actor-critic neural seq2seq learning I O -policy/counterfactual

Reinforcement Learninglxmls.it.pt/2019/rl-intro.pdf · I Seq2seq reinforcement learning: Bandit structured prediction, actor-critic neural seq2seq learning I O -policy/counterfactual

Documents
Offline Reinforcement Learning and Offline Multi-Task RL

Offline Reinforcement Learning and Offline Multi-Task RL

Documents
Reinforcement Learning - Yisong Yue · Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents 1 Reinforcement Learning Introduction to RL. Markov Decision Processes

Reinforcement Learning - Yisong Yue · Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents 1 Reinforcement Learning Introduction to RL. Markov Decision Processes

Documents
Introduction to Reinforcement Learning - Inriachercheurs.lille.inria.fr/~lazaric/Webpage/EC-RL_Course15_files/... · EC-RL Course Introduction to Reinforcement Learning ... Optimal

Introduction to Reinforcement Learning - Inriachercheurs.lille.inria.fr/~lazaric/Webpage/EC-RL_Course15_files/... · EC-RL Course Introduction to Reinforcement Learning ... Optimal

Documents
Tutorial: Deep Reinforcement Learning€¦ · Reinforcement Learning in a nutshell RL is a general-purpose framework for decision-making I RL is for an agent with the capacity to

Tutorial: Deep Reinforcement Learning€¦ · Reinforcement Learning in a nutshell RL is a general-purpose framework for decision-making I RL is for an agent with the capacity to

Documents
Unsupervised Methods For Subgoal Discovery During Intrinsic …rafati.net/slides/KEG-2019-slides.pdf · 2019. 11. 26. · Reinforcement Learning Reinforcement learning (RL) is learning

Unsupervised Methods For Subgoal Discovery During Intrinsic …rafati.net/slides/KEG-2019-slides.pdf · 2019. 11. 26. · Reinforcement Learning Reinforcement learning (RL) is learning

Documents
A2-RL: Aesthetics Aware Reinforcement Learning for Image … · 2018-03-13 · A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping Debang Li 1;2, Huikai Wu , Junge Zhang

A2-RL: Aesthetics Aware Reinforcement Learning for Image … · 2018-03-13 · A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping Debang Li 1;2, Huikai Wu , Junge Zhang

Documents
Quantum Reinforcement Learning - arXiv · quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state

Quantum Reinforcement Learning - arXiv · quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state

Documents
Soar-RL: Reinforcement Learning and Soar Shelley Nason

Soar-RL: Reinforcement Learning and Soar Shelley Nason

Documents
A Unifying Framework for Computational Reinforcement Learning Theory Lihong Li Rutgers Laboratory for Real-Life Reinforcement Learning (RL 3 ) Department

A Unifying Framework for Computational Reinforcement Learning Theory Lihong Li Rutgers Laboratory for Real-Life Reinforcement Learning (RL 3 ) Department

Documents
MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement ...lasa.epfl.ch/teaching/lectures/ML_Phd/Slides/RL-GradientMethods.pdf · MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement

MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement ...lasa.epfl.ch/teaching/lectures/ML_Phd/Slides/RL-GradientMethods.pdf · MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement

Documents
RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real · 2020. 6. 29. · RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real Kanishka Rao1, Chris Harris1, Alex Irpan1,

RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real · 2020. 6. 29. · RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real Kanishka Rao1, Chris Harris1, Alex Irpan1,

Documents
Markov Games and Multi-Objective Reinforcement …/rl in...Dynamical SystemsBellman’s Principle of OptimalityReinforcement Learning Multi-Objective Reinforcement Learning Theorem(Cont’d)

Markov Games and Multi-Objective Reinforcement …/rl in...Dynamical SystemsBellman’s Principle of OptimalityReinforcement Learning Multi-Objective Reinforcement Learning Theorem(Cont’d)

Documents
Multi-Agent Reinforcement Learning - Game Theory Polimi · 2019-12-03 · Introduction to Multi-Agent Reinforcement Learning Reinforcement Learning MARL vs RL MARL vs Game Theory

Multi-Agent Reinforcement Learning - Game Theory Polimi · 2019-12-03 · Introduction to Multi-Agent Reinforcement Learning Reinforcement Learning MARL vs RL MARL vs Game Theory

Documents
Reinforcement Learning with Wasserstein Distance ...Keywords: Reinforcement Learning, Wasserstein distance 1. Introduction and Motivation Reinforcement learning (RL) is a formalism

Reinforcement Learning with Wasserstein Distance ...Keywords: Reinforcement Learning, Wasserstein distance 1. Introduction and Motivation Reinforcement learning (RL) is a formalism

Documents
Deep Reinforcement Learning - David Silver · Reinforcement Learning: AI = RL RL is a general-purpose framework for arti cial intelligence I RL is for an agent with the capacity to

Deep Reinforcement Learning - David Silver · Reinforcement Learning: AI = RL RL is a general-purpose framework for arti cial intelligence I RL is for an agent with the capacity to

Documents
RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Realopenaccess.thecvf.com/content_CVPR_2020/papers/Rao_RL... · 2020-06-07 · RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Realopenaccess.thecvf.com/content_CVPR_2020/papers/Rao_RL... · 2020-06-07 · RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

Documents
Novelty Search for Deep Reinforcement Learning Policy ... · Reinforcement learning (RL) problems o›en feature deceptive local optima, and learning methods that optimize purely

Novelty Search for Deep Reinforcement Learning Policy ... · Reinforcement learning (RL) problems o›en feature deceptive local optima, and learning methods that optimize purely

Documents
MDPs and RL Outline Reinforcement Learningcs188/sp11/slides/SP11 cs188 lectur… · Reinforcement Learning ! Reinforcement learning: ! Still assume an MDP: ! A set of states s ∈

MDPs and RL Outline Reinforcement Learningcs188/sp11/slides/SP11 cs188 lectur… · Reinforcement Learning ! Reinforcement learning: ! Still assume an MDP: ! A set of states s ∈

Documents
Meger ősítéses Tanulás = Reinforcement Learning (RL)

Meger ősítéses Tanulás = Reinforcement Learning (RL)

Documents
[RL輪読会]Distral: Robust Multitask Reinforcement Learning

[RL輪読会]Distral: Robust Multitask Reinforcement Learning

Data & Analytics
TEXPLORE: Temporal Difference Reinforcement Learning for ... · Reinforcement Learning Could be used for learning and adaptation on robots Value function RL has string of positive

TEXPLORE: Temporal Difference Reinforcement Learning for ... · Reinforcement Learning Could be used for learning and adaptation on robots Value function RL has string of positive

Documents
Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Documents
Sequence-to-Sequence Reinforcement Learning Seq2seq RL for

Sequence-to-Sequence Reinforcement Learning Seq2seq RL for

Documents
Evaluating Reinforcement Learning Algorithms in ... · with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub- eld within machine learning

Evaluating Reinforcement Learning Algorithms in ... · with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub- eld within machine learning

Documents
Introduction to Reinforcement Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_3_rl_intro.pdf · Introduction to Reinforcement Learning ... •Structure of RL algorithms

Introduction to Reinforcement Learningrll.berkeley.edu/deeprlcourse/f17docs/lecture_3_rl_intro.pdf · Introduction to Reinforcement Learning ... •Structure of RL algorithms

Documents
Distributed Reinforcement Learning with ADMM-RL › docs › fy19osti › 74798.pdfDistributed Reinforcement Learning with ADMM-RL Peter Graf, Jen Annoni, Chris Bay, Devon Sigler,

Distributed Reinforcement Learning with ADMM-RL › docs › fy19osti › 74798.pdfDistributed Reinforcement Learning with ADMM-RL Peter Graf, Jen Annoni, Chris Bay, Devon Sigler,

Documents
Reinforcement Learning (RL)

Reinforcement Learning (RL)

Documents
Reinforcement Learning Dealing with Complexity and Safety in RL

Reinforcement Learning Dealing with Complexity and Safety in RL

Documents
Reinforcement Learning - Home | College of Engineeringweb.engr.oregonstate.edu/~xfern/classes/cs434/slides/RL-1.pdf · Reinforcement Learning in a nutshell ... – Analogous to solving

Reinforcement Learning - Home | College of Engineeringweb.engr.oregonstate.edu/~xfern/classes/cs434/slides/RL-1.pdf · Reinforcement Learning in a nutshell ... – Analogous to solving

Documents