Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
CSE 190: Reinforcement Learning:An Introduction
Chapter 4: Dynamic Programming
Acknowledgment:Acknowledgment:A good number of these slidesA good number of these slides are cribbed from Rich Suttonare cribbed from Rich Sutton
22CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Administrivia
•• AnyAny email sent to me about the course should have “CSE190” in the subject line!
33CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Goals for this chapter
• Overview of a collection of classical solution methods forMDPs known as dynamic programming (DP)
• Show how DP can be used to compute value functions, andhence, optimal policies
• Discuss efficiency and utility of DP
44CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Last Time: Value Functions
State - value function for policy ! :
V! (s) = E! Rt st = s{ } = E! " krt+k +1 st = sk =0
#
$% & '
( ) *
Action- value function for policy ! :
Q! (s, a) = E! Rt st = s, at = a{ } = E! " krt+ k+1 st = s,at = ak= 0
#
$% & '
( ) *
• The value of a state is the expected return starting fromthat state; depends on the agent’s policy:
• The value of taking an action in a state under policy! is the expected return starting from that state, takingthat action, and thereafter following ! :
55CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Last Time: Bellman Equation for aPolicy !
Rt = rt+1 + ! rt+2 + !2rt+3 + !
3rt+4!
= rt+1 + ! rt+2 + ! rt+3 + !2rt+4!( )
= rt+1 + ! Rt+1
The basic idea:
So: V! (s) = E! Rt st = s{ }= E! rt+1 + "V st+1( ) st = s{ }
Or, without the expectation operator:
V ! (s) = ! (s,a) Ps "sa Rs "s
a + #V ! ( "s )$% &'"s(
a(
66CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Last Time: More on the BellmanEquation
V ! (s) = ! (s,a) Ps "sa Rs "s
a + #V ! ( "s )$% &'"s(
a(
This is a set of equations (in fact, linear), one for each state.The value function for ! is its unique solution.
Backup diagrams:
for V ! for Q!
77CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Last Time: Bellman OptimalityEquation for V*
V !(s) = maxa"A(s )
Q#!
(s,a)
= maxa"A(s )
E rt+1 + $V!(st+1) st = s,at = a{ }
= maxa"A(s )
Ps %sa
%s& Rs %s
a + $V !( %s )'( )*
The value of a state under an optimal policy must equalthe expected return for the best action from that state:
The relevant backup diagram:
V* is the unique solution of this system of nonlinear equations.88CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Last Time: Bellman OptimalityEquation for Q*
Q!(s,a) = E rt+1 + " max#a Q!(st+1, #a ) st = s,at = a{ }= Ps #s
a Rs #sa + " max
#aQ!( #s , #a )$
%&'
#s(
The relevant backup diagram:
Q* is the unique solution of this system of nonlinear equations.
99CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
This Time
• How to solve these equations using iteration
• Can solve for optimal V*
• But often it is faster to evaluate and improve the policy
• Alternating figuring out V! and improving !
1010CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Evaluation
State - value function for policy ! :
V ! (s) = E! Rt st = s{ } = E! " krt+ k+1 st = sk=0
#
$%&'
()*
Bellman equation for V ! :
V ! (s) = ! (s,a) Ps "sa Rs "s
a + #V ! ( "s )$% &'"s(
a(
— a system of S simultaneous linear equations
Policy Evaluation: for a given policy !, compute the state-value function V !
Recall:
1111CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Iterative Methods
V0 !V1 !!!Vk !Vk+1 !!!V "
Vk+1(s)! " (s,a) Ps #sa Rs #s
a + $Vk ( #s )%& '(#s)
a)
a “sweep”
A sweep consists of applying a backup operation to each state.
A full policy-evaluation backup:
1212CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Iterative Methods
V0 !V1 !!!Vk !Vk+1 !!!V "
Vk+1(s)! " (s,a) Ps #sa Rs #s
a + $Vk ( #s )%& '(#s)
a)
a “sweep”
A sweep consists of applying a backup operation to each state.
A full policy-evaluation backup:
1313CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Iterative Policy Evaluation
1414CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
• An undiscounted episodic task
• Nonterminal states: 1, 2, . . ., 14;
• One terminal state (shown twice as shaded squares)
• Actions that would take agent off the grid leave stateunchanged
• Reward is –1 until the terminal state is reached
1515CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
• Note here that the actions are deterministic, so thisequation:
• Becomes:
Vk+1(s)! " (s,a) Ps #sa Rs #s
a + $Vk ( #s )%& '(#s)
a)
Vk+1(s)! " (s,a) Rs #sa + $Vk ( #s )%& '(
a)
1616CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
• And…it is undiscounted, so this:
• Becomes:
Vk+1(s)! " (s,a) Rs #sa + $Vk ( #s )%& '(
a)
Vk+1(s)! " (s,a) Rs #sa +Vk ( #s )$% &'
a(
1717CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(s)!" (s,UP) Rs #sUP +Vk ( #s )$% &' +
" (s,RIGHT ) Rs #sRIGHT +Vk ( ##s )$% &' +
" (s,DOWN ) Rs #sDOWN +Vk ( ###s )$% &' +
" (s,LEFT ) Rs #sLEFT +Vk ( ####s )$% &'
Vk+1(s)! " (s,a) Rs #sa +Vk ( #s )$% &'
a(
1818CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(s)! 0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ]
1919CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
• For state 4, for example, we have:
Vk+1(4)! 0.25UP "1+Vk (terminal)[ ] +0.25RIGHT "1+Vk (5)[ ] +0.25DOWN "1+Vk (8)[ ] +0.25LEFT "1+Vk (4)[ ]
2020CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(s)! 0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ]
2121CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(s)! 0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ] +0.25 "1+Vk ( #s )[ ]
2222CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(4)! 0.25UP "1+Vk (terminal)[ ] +0.25RIGHT "1+Vk (5)[ ] +0.25DOWN "1+Vk (8)[ ] +0.25LEFT "1+Vk (4)[ ]
2323CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
A Small Gridworld
Vk+1(4)! 0.25UP "1+ 0[ ] +0.25RIGHT "1+ "1[ ] +0.25DOWN "1+"1[ ] +0.25LEFT "1+"1[ ]= "1.75
2424CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Iterative PolicyEvaluationfor the SmallGridworld
! = equiprobable random action choices
2525CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Iterative PolicyEvaluationfor the SmallGridworld! = equiprobable random action choices
But look what happensBut look what happensif these values are usedif these values are usedto make a new policy!to make a new policy!
(note - this won(note - this won’’t alwayst alwayshappen!)happen!)
Exercise for the reader:Exercise for the reader:What are the values of the statesWhat are the values of the states
under the optimal policy?under the optimal policy?
2626CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Improvement
Suppose we have computed for a deterministic policy !.V !
For a given state s, would it be better to do an action ? a ! " (s)
Q! (s,a) = E! rt+1 + "V! (st+1) st = s,at = a{ }
= Ps #sa
#s$ Rs #s
a + "V ! ( #s )%& '(
The value of doing a in state s is:
It is better to switch to action a for state s if and only if Q! (s,a) >V ! (s)
2727CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Improvement Cont.
!" (s) = argmaxaQ" (s,a)
= argmaxa
Ps !sa
!s# Rs !s
a + $V " ( !s )%& '(
Do this for all states to get a new policy !" that is greedy with respect to V " :
Then V !" #V "
2828CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Improvement Cont.
What if V !" = V " ?
i.e., for all s #S, V !" (s) = maxa
Ps !sa
!s$ Rs !s
a + %V " ( !s )&' () ?
But this is the Bellman Optimality Equation.So V !" = V # and both " and !" are optimal policies.
2929CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Iteration
! 0 "V !0 "!1 "V !1 "!! * "V * "! *
policy evaluation policy improvement“greedification”
3030CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy Iteration
3131CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Jack’s Car Rental
• $10 for each car rented (must be available when request rec’d)
• Two locations, maximum of 20 cars at each
• Cars returned and requested randomly• Poisson distribution, n returns/requests with prob "ne-"/n! (where " is
the expected number)
• 1st location: average requests = 3, average returns = 3
• 2nd location: average requests = 4, average returns = 2
• Can move up to 5 cars between locations overnight at $2/car.
• States, Actions, Rewards?
• Transition probabilities?
3232CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Jack’s Car Rental
Note this makes sense - location 2 on average loses 2 cars per day.
3333CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Jack’s CR Exercise
• Suppose the first car moved is free• From 1st to 2nd location
• Because an employee travels that way anyway (by bus)
• Suppose only 10 cars can be parked for free at eachlocation• More than 10 cost $4 for using an extra parking lot
• Such arbitrary nonlinearities are common in realproblems
3434CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Policy iteration: Can we do better?• Each iteration involves policy evaluation, which is itself an iterative
process
• It looks like from the previous example that policy evaluation mayconverge long after the greedy policy based on the values hasconverged.
• Can we skip steps somehow?
• Yes: policy evaluation can be stopped early and under most cases,convergence is still guaranteed!
• A very special case: Stopping after oneone sweepsweep of policy evaluation.
• This is called value iteration
3535CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Value Iteration
Vk+1(s)! " (s,a) Ps #sa Rs #s
a + $Vk ( #s )%& '(#s)
a)
Recall the full policy-evaluation backup:
Vk+1(s)!maxa
Ps "sa Rs "s
a + #Vk ( "s )$% &'"s(
Here is the full value-iteration backup:
Note how this combines policy improvement and evaluation.It is simply the Bellman optimality equation turned into anupdate equation!In practice, often policy evaluation (sum) is performedseveral times between policy improvement (max) sweeps.
3636CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Value Iteration Cont.
3737CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Gambler’s Problem
• Gambler can repeatedly bet $ on a coin flip
• Heads he wins his stake, tails he loses it• Initial capital # {$1, $2, … $99}
• Gambler wins if his capital becomes $100loses if it becomes $0
• Coin is unfair• Heads (gambler wins) with probability p = .4
• States, Actions, Rewards?
!n
n!e"!
3838CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Gambler’s Problem Solution
3939CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Herd Management
• You are a consultant to a farmer managing a herd of cows
• Herd consists of 5 kinds of cows:• Young
• Milking
• Breeding
• Old
• Sick
• Number of each kind is the State
• Number sold of each kind is the Action
• Cows transition from one kind to another
• Young cows can be born
4040CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Asynchronous DP
• All the DP methods described so far require exhaustivesweeps of the entire state set.
• Asynchronous DP does not use sweeps. Instead it workslike this:• Repeat until convergence criterion is met:
• Pick a state at random and apply the appropriate backup
• Still need lots of computation, but does not get lockedinto hopelessly long sweeps
• Can you select states to backup intelligently? YES: anagent’s experience can act as a guide.
4141CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Generalized Policy Iteration
Generalized Policy Iteration (GPI): any interaction of policy evaluation and policy improvement, independent of their granularity.
A geometric metaphor forconvergence of GPI:
4242CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Efficiency of DP
• To find an optimal policy is polynomial in the number ofstates…
• BUT, the number of states is often astronomical, e.g., oftengrowing exponentially with the number of state variables(what Bellman called “the curse of dimensionality”).
• In practice, classical DP can be applied to problems with afew million states.
• Asynchronous DP can be applied to larger problems, and isappropriate for parallel computation.
• It is surprisingly easy to come up with MDPs for which DPmethods are not practical.
4343CSE 190: Reinforcement Learning, LectureCSE 190: Reinforcement Learning, Lecture on Chapteron Chapter 44
Summary
• Policy evaluation: backups without a max
• Policy improvement: form a greedy policy, if only locally
• Policy iteration: alternate the above two processes
• Value iteration: backups with a max
• Full backups (to be contrasted later with sample backups)
• Asynchronous DP: a way to avoid exhaustive sweeps
• Generalized Policy Iteration (GPI)
• Bootstrapping: updating estimates based on otherestimates END