Course basics CSE 190: Reinforcement Learning: …cseweb.ucsd.edu/~gary/190-RL/Lecture_2.pdfCSE 190: Reinforcement Learning: An Introduction Acknowledgment: A good number of these

CSE 190: Reinforcement Learning:An Introduction

Acknowledgment:Acknowledgment:A good number of these slidesA good number of these slides are cribbed from Rich Suttonare cribbed from Rich Sutton

22CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture 22

Course basics• The website for the class is linked off my homepage.

• Grades will be based on programming assignments, homeworks, andclass participation.

• Homeworks will be turned in, but not graded, as we will discuss theanswers in class in small groups. Turning it in means I can see that youare holding up your end of the conversation (this is a major part ofclass participation!)

• Programming assignments will be graded

•• AnyAny email sent to me about the course should have “CSE 190” in thesubject line!


Course goals• After taking this course you should:

• Understand what is unique about Reinforcement Learning• Understand the tradeoff between exploration and exploitation• Be conversant in Markov Decision Problems (MDPs)• Know the various solution methods for solving the RL problem:

• Dynamic programming (value iteration, policy iteration, etc.)• Monte Carlo• TD learning

• Know what an eligibility trace is• Be aware of several well-known applications of RL• Be able to read papers in the field and understand 75% of each paper.


Last Time• Difference from other forms of learning:

• Learning by interaction with environment, which leads to• The exploration/exploitation tradeoffexploration/exploitation tradeoff:

• An agent learning by interacting with the environment must:•• Exploit its knowledgeExploit its knowledge to maximize reward•• Explore the environmentExplore the environment to ensure that its knowledge is correct

• The agent must “try everything” while favoring, over time, the mostrewarding actions.

• Elements of RL:• a policy• a reward function• a value function• optionally, a model of the environment.


Last Time• Elements of RL:

• a policy: a mapping from states to actions, possibly stochastic• a reward function: given as part of the problem• a value function: A prediction of reward from a state• optionally, a model of the environment.

• This should (almost) all be familiar from your programmingassignment.


Last Time: Elements of RL• a policy:

• A mapping from states to actions:

• Stochastic policy:

• a reward function:• Specified in the environment, not in the agent!

• Usually a scalar value at each state

• a value function:• A mapping from states to expected total rewards from this state if

we follow policy

• Written:

• A model of the environment:• Something that tells us what to expect if we take an action in a

state:

i.e., the probability of getting to state s’ from state s if we take action a.

! (s) = a! (s,a) = P(a | s)

V ! (s)!

Pss 'a


Elements of RL

• A model of the environment supports planning throughsimulating the future (if I do this, then he’ll do that…)

• In general, RL agents can span the gamut from reactive tofar-sighted.


Example 1: Tic-tac-toe• Since one can always play to a draw, let’s assume an

imperfect opponent.

• Reinforcement learning approach:• Set up a table, V[si], i=1…n, where n is the number of possible states

of the board, and si is a state

• Each entry of V[] is an estimate of the probability of a win from thatstate: the value of that state

• Assume we always play X’s - Initialize V as:

• V[si = a state with three X’s in a row] = 1

• V[si = a state with three O’s in a row] = -1

• V[si = all other states] = 0.5


Example 1: Tic-tac-toe• Reinforcement learning approach:

• Set up a table, V[si], i=1…n, where n is the number ofpossible states of the board, and si is a state

• Play many games against our imperfect opponent tolearn values of states

• How do we play? We need a policy.• Let’s use one called !-greedy

• For each move, ! of the time, we pick a move uniformlyat random from the possible moves.

• Otherwise, we pick the move that gets us to the statewith the highest value: V[si] (greedy).


Example 1: Tic-tac-toe• !-greedy policy (details):

• The ! case is exploration.

• The other (greedy, i.e., take the highest value state)case is exploitation

! (si ) =sk randomly, uniformly over states sk reachable from si , with probability "argmax

kV (sk ) over states sk reachable from si , with probability 1-"

#$%

&%


Example 1: Tic-tac-toe


Example 1: Tic-tac-toe• How do we update previous values?

where is the state reached after the opponent’s move and is a learning rate.

• This is called a temporal difference method because it usesvalues of states from two different time steps.

• It provably converges to the optimal policy.• Why do I say that updating the value estimates changes the

policy?

V (si ) = V (si ) +![V ( "s ) #V (si )]

!s!


Example 1: Tic-tac-toe

• Why does updating the V’s change "?

• Because the policy is a function of V:

• As we get better value estimates, we make better choices.

V (si ) = V (si ) +![V ( "s ) #V (si )]

! (si ) =sk randomly, uniformly over states sk reachable from si , with probability "arg max

kV (sk ) over states sk reachable from si , with probability 1-"

#$%

&%


Example 2: n-armed bandit

• n = 10 possible actions

• Each Q*(ai) is chosen randomly from a normal distrib.: Q*(ai) ~N (0,1)

• each rt is also normal: rt~N (Q*(ai),1)• Estimate Q-values using a

running average:

• Policy is !-greedy

• Use 1000 plays

• repeat the whole thing 2000 times and average the results

• Hopefully, this is what you did for your programmingassignment!

Qt (a) =r1 + r2 + ...+ rka

ka


Example 2: n-armed bandit• We compare three policies: greedy, !-greedy with ! = 0.1

and 0.01. Note that greedy never explores.


Example 2: n-armed bandit• We compare three policies: greedy, !-greedy with ! = 0.1 and 0.01.

Note that greedy never explores.

• And this is your programming assignment!


Another approach to exploration:The Softmax policy

P(a) = eQt (a )

!

eQt (b )

!

b=1

n"



• P(a) is the probability of taking action a• Qt(a) is the current estimate of Q*(a)• The higher Qt(a) is, the more likely we will choose action a.• # is the temperature: this is “annealed” (starts “hot” and

slowly “cools” over trials).• As #$0, the policy becomes deterministic.

P(a) = eQt (a )

!

eQt (b )

!

b=1

n"



• Why use this?• Note that !-greedy keeps exploring randomly long after it

has learned what the right values are.• Softmax quickly gives states or actions with better evidence

higher priority, while still exploring.

P(a) = eQt (a )

!

eQt (b )

!

b=1

n"


Incremental estimation of Q’s

• Recall (what you should have implemented in yourprogramming assignment:) the sample average method forestimating Qk(a):

• Can we do this incrementally (without storing all of therewards)?

• Where k is the number of times we have tried action a

Qk+1(a) = Qk (a) +1

k +1rk+1 !Qk (a)[ ]

Qt (a) =r1 + r2 + ...+ rka

ka


Incremental estimation of Q’s

• Note that this is a common form of update rules:

New Estimate =Old Estimate + step-size[target-Old Estimate]

• Note in the version above, the step size changes over time.

Qk+1(a) = Qk (a) +1

k +1rk+1 !Qk (a)[ ]


Tracking nonstationary problems

• Choosing Qk to be a sample average is appropriate in astationary problem, i.e., when none of the Q*(a) change over time.• When the Q*(a) change over time, this is called a non-stationary

problem.• In this case, the following is better:

• Where % is a constant between zero and one.

Qk+1(a) = Qk (a) +! rk+1 "Qk (a)[ ]

Qk+1(a) = (1!" )Qk (a) +"rk+1

Qk+1(a) = (1!" )kQ0 (a) + "(1!" )k! i ri

i=1

k

#


Tracking nonstationary problems

• The rule for non-stationary problems:

• I.e., a recency-weighted exponential average.

Qk+1(a) = Qk (a) +! rk+1 "Qk (a)[ ]

Qk+1(a) = (1!" )kQ0 (a) + "(1!" )k! i ri

i=1

k

#


Optimistic Initial Values

• All of the methods above are biased by the initial values,

although, in the one we just looked at, the dependence on

Q0 clearly decreases over time:

• This is bad: more parameters to pick

• This is good: prior knowledge can be incorporated into the

initial Q-values.

Qk+1(a) = (1!" )kQ0 (a) + "(1!" )k! i ri

i=1

k

#



• One way to bias the model is to use optimistic initial values.• Suppose instead of 0 for the n-armed bandit, we used +5.

• Now everything looks good, especially to a greedy method!

• So this encourages exploration:



• This looks good, but why does the greedy method oscillate

in the beginning?

Pause for effect…


Chapter 3: The ReinforcementLearning Problem

Objectives of what I will talk about from this chapter:

• describe the RL problem we will be studying for theremainder of the course

• present idealized form of the RL problem for which wehave precise theoretical results;

• introduce key components of the mathematics: valuefunctions and Bellman equations;

• describe trade-offs between applicability and mathematicaltractability.


Chapter 3: The ReinforcementLearning Problem

Objectives of what I will talk about from this chapter:

• describe the RL problem we will be studying for theremainder of the course

• present idealized form of the RL problem for which wehave precise theoretical results;

• introduce key components of the mathematics: valuefunctions and Bellman equations;

• describe trade-offs between applicability and mathematicaltractability.


The Agent-Environment Interface

Agent and environment interact at discrete time steps : t = 0, 1, 2, K Agent observes state at step t : st !S produces action at step t : at ! A(st ) gets resulting reward : rt+1 !" and resulting next state : st+1

t. . . st a

rt +1 st +1t +1a

rt +2 st +2t +2a

rt +3 st +3 . . .t +3a


Policy at step t, ! t : a mapping from states to action probabilities ! t (s,a) = probability that at = a when st = s

The Agent Learns a Policy

• Reinforcement learning methods specify how theagent changes its policy as a result of experience.

• Roughly, the agent’s goal is to get as much reward asit can over the long run.


Getting the Degree of Abstraction Right• Time steps need not refer to fixed intervals of real time.• Actions can be low level (e.g., voltages to motors), or high

level (e.g., accept a job offer), “mental” (e.g., shift in focus ofattention), etc.

• States can low-level “sensations”, or they can be abstract,symbolic, based on memory, or subjective (e.g., the state ofbeing “surprised” or “lost”).

• An RL agent is not necessarily like a whole animal or robot.• Rewards are in the agent’s environment because the agent

cannot change it arbitrarily - otherwise, it could simplyreward itself and call it a day!

• The environment is not necessarily unknown to the agent,only incompletely controllable.


Goals and Rewards

• Is a scalar reward signal an adequate notion of agoal?—maybe not, but it is surprisingly flexible.

• A goal should specify whatwhat we want to achieve, nothowhow we want to achieve it.

• A goal must be outside the agent’s direct control—thusoutside the agent.

• The agent must be able to measure success:• explicitly;

• frequently during its lifespan.


The reward hypothesis

• The reward hypothesis:

All of what we mean by goals and purposes can be thoughtof as the maximization of the cumulative sum of areceived scalar signal (reward)

• A sort of null hypothesis.

• Probably ultimately wrong, but so simple we have todisprove it before considering anything morecomplicated


Returns

Suppose the sequence of rewards after step t is: rt+1, rt+2 , rt+3,…What do we want to maximize?

In general, we want to maximize the expected return, E Rt{ }, for each step t.

Episodic tasksEpisodic tasks: interaction breaks naturally intoepisodes, e.g., plays of a game, trips through a maze.

Rt = rt+1 + rt+2 +!+ rT ,where T is a final time step at which a terminal stateterminal state is reached,ending an episode.


Returns for Continuing Tasks

Continuing tasks: interaction does not have natural episodes.

Instead, we use the Discounted return: Discounted return:

Rt = rt+1 + ! rt+2 + !2rt+3 +! = ! krt+ k+1,

k=0

"

#where ! , 0 $ ! $ 1, is the discount rate.This ensures that the expected reward converges.

shortsighted 0 !" # 1 farsighted


An Example

Avoid failure: the pole falling beyonda critical angle or the cart hitting end oftrack.

reward = +1 for each step before failure! return = number of steps before failure

As an episodic task where episode ends upon failure:

As a continuing task with discounted return:reward = !1 upon failure; 0 otherwise" return = ! # k , for k steps before failure

In either case, return is maximized by avoiding failure foras long as possible.


Another Example

Get to the top of the hillas quickly as possible.

reward = !1 for each step where not at top of hill" return = ! number of steps before reaching top of hill

Return is maximized by minimizing number of stepsto reach the top of the hill.


A Unified Notation• In episodic tasks, we number the time steps of each

episode starting from zero.

• We usually do not have to distinguish betweenepisodes, so we write instead of for the stateat step t of episode j.

• Think of each episode as ending in an absorbing statethat always produces reward of zero:

• We can cover all cases by writing

st st , j

Rt = ! krt+ k+1,k=0

"

#where ! can be 1 only if a zero reward absorbing state is always reached.


The Markov Property

• By “the state” at step t, the book means whateverinformation is available to the agent at step t about itsenvironment.

• The state can include immediate “sensations,” highlyprocessed sensations, and structures built up over timefrom sequences of sensations.

• Ideally, a state should summarize past sensations so as toretain all “essential” information, i.e., it should have theMarkov Property:

Pr st+1 = !s ,rt+1 = r st ,at ,rt , st"1,at"1,…,r1, s0 ,a0{ } = Pr st+1 = !s ,rt+1 = r st ,at{ }for all !s , r, and histories st ,at ,rt , st"1,at"1,…,r1, s0 ,a0 .


Markov Decision Processes

• If a reinforcement learning task has the MarkovProperty, it is basically a Markov Decision Process(MDP).

• If state and action sets are finite, it is a finite MDP.

• To define a finite MDP, you need to give:• state and action sets

• one-step “dynamics” defined by transition probabilities:

• reward probabilities:

Ps !sa = Pr st+1 = !s st = s,at = a{ } for all s, !s "S, a "A(s).

Rs !sa = E rt+1 st = s,at = a, st+1 = !s{ } for all s, !s "S, a "A(s).


Recycling Robot

An Example Finite MDP

• At each step, robot has to decide whether it should (1) activelysearch for a can, (2) wait for someone to bring it a can, or (3) go tohome base and recharge.

• Searching is better but runs down the battery; if runs out of powerwhile searching, has to be rescued (which is bad).

• Decisions made on basis of current energy level: high, low.• Reward = number of cans collected


Recycling Robot MDP

S = high,low{ }A(high) = search, wait{ }A(low) = search, wait, recharge{ }

Rsearch = expected no. of cans while searchingRwait = expected no. of cans while waiting Rsearch > Rwait


Value Functions

State - value function for policy ! :

V! (s) = E! Rt st = s{ } = E! " krt+k +1 st = sk =0

#

$% & '

( ) *

Action- value function for policy ! :

Q! (s, a) = E! Rt st = s, at = a{ } = E! " krt+ k+1 st = s,at = ak= 0

#

$% & '

( ) *

• The value of a state is the expected return starting fromthat state; depends on the agent’s policy:

• The value of taking an action in a state under policy! is the expected return starting from that state, takingthat action, and thereafter following ! :


Bellman Equation for a Policy !

Rt = rt+1 + ! rt+2 + !2rt+3 + !

3rt+4!

= rt+1 + ! rt+2 + ! rt+3 + !2rt+4!( )

= rt+1 + ! Rt+1

The basic idea:

So: V! (s) = E! Rt st = s{ }= E! rt+1 + "V st+1( ) st = s{ }

Or, without the expectation operator:

V ! (s) = ! (s,a) Ps "sa Rs "s

a + #V ! ( "s )$% &'"s(

a(


More on the Bellman Equation

V ! (s) = ! (s,a) Ps "sa Rs "s

a + #V ! ( "s )$% &'"s(

a(

This is a set of equations (in fact, linear), one for each state.The value function for ! is its unique solution.

Backup diagrams:

for V ! for Q!


Gridworld• Actions: north, south, east, west; deterministic.• If would take agent off the grid: no move but reward = –1• Other actions produce reward = 0, except actions that move agent out

of special states A and B as shown.

State-value function for equiprobable random policy;& = 0.9


Golf

• State is ball location• Reward of –1 for each stroke

until the ball is in the hole• Value of a state?• Actions:

• putt (use putter)• driver (use driver)

• putt succeeds anywhere onthe green


• For finite MDPs, policies can be partially ordered:

• There are always one or more policies that are better than orequal to all the others. These are the optimal policies. Wedenote them all ! *.

• Optimal policies share the same optimal state-valuefunction:

• Optimal policies also share the same optimal action-valuefunction:

! " #! if and only if V ! (s) "V #! (s) for all s $S

Optimal Value Functions

V !(s) = max"V " (s) for all s #S

Q!(s,a) = max"Q" (s,a) for all s #S and a #A(s)

This is the expected return for taking action a in state sand thereafter following an optimal policy.


Optimal Value Function for Golf

• We can hit the ball farther with driver than withputter, but with less accuracy

• Q*(s,driver) gives the value or using driver first,then using whichever actions are best


Bellman Optimality Equation for V*

V !(s) = maxa"A(s )

Q#!

(s,a)

= maxa"A(s )

E rt+1 + $V!(st+1) st = s,at = a{ }

= maxa"A(s )

Ps %sa

%s& Rs %s

a + $V !( %s )'( )*

The value of a state under an optimal policy must equalthe expected return for the best action from that state:

The relevant backup diagram:

V* is the unique solution of this system of nonlinear equations.5252CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture 22

Bellman Optimality Equation for Q*

Q!(s,a) = E rt+1 + " max#a Q!(st+1, #a ) st = s,at = a{ }= Ps #s

a Rs #sa + " max

#aQ!( #s , #a )$

%&'

#s(

The relevant backup diagram:

Q* is the unique solution of this system of nonlinear equations.


Why Optimal State-Value Functionsare Useful

Any policy that is greedy with respect to V* is an optimal policy.

Therefore, given V*, one-step-ahead search produces the long-term optimal actions.

E.g., back to the gridworld:


What About Optimal Action-ValueFunctions?

Given , the agent does not evenhave to do a one-step-ahead search:

Q*

!"(s) = argmaxa#A (s)

Q"(s, a)


Solving the Bellman OptimalityEquation

• Finding an optimal policy by solving the BellmanOptimality Equation requires the following:• accurate knowledge of environment dynamics;• we have enough space and time to do the computation;• the Markov Property.

• How much space and time do we need?• polynomial in number of states (via dynamic programming

methods; Chapter 4),• BUT, number of states is often huge (e.g., backgammon has

about 1020 states).

• We usually have to settle for approximations.• Many RL methods can be understood as

approximately solving the Bellman OptimalityEquation.


Summary• Agent-environment interaction

• States• Actions• Rewards

• Policy: stochastic rule for selectingactions

• Return: the function of futurerewards agent tries to maximize

• Episodic and continuing tasks• Markov Property• Markov Decision Process

• Transition probabilities• Expected rewards

• Value functions• State-value function for a policy

• Action-value function for a policy

• Optimal state-value function

• Optimal action-value function

• Optimal value functions

• Optimal policies

• Bellman Equations

• The need for approximation

END

Documents

Course basics CSE 190: Reinforcement Learning: …cseweb.ucsd.edu/~gary/190-RL/Lecture_2.pdfCSE 190: Reinforcement Learning: An Introduction Acknowledgment: A good number of these