Biosight: Quantitative Methods for Policy Analysis : Dynamic Models

Day 3: Dynamic Models

Day 3 NotesHowitt and Msangi 1

Understand Bellman’s Principle of Optimality and the basic Dynamic programming problem

Have your cake and eat it too – an example

Solve the DP using Chebychev Polynomial approximation

Apply concepts to Senegal livestock model

Evaluate changes in the DP optimal solution


Introduction to Dynamic Programming◦ Solving a simple problem with analytical methods

Value Function Iteration◦ Illustrate with “cake-eating” problem

Function Approximation◦ Chebychev nodes◦ Chebychev polynomials

Collocation example◦ Solving for polynomial terms with simple example

DP Senegal Livestock Model◦ (S)DP Solution


Fundamental Dynamic Program◦ State variable: x◦ Control variable: c◦ Discount factor: β


1

( , )

( , )

t t

t t t

Max f x csubject tox g x c+ =

Bellman’s “principle of optimality”◦ Dynamic problem and equation of motion:

◦ Infinite horizon:

◦ Where, β is the discount factor V(.) is the value function, defined as maximum utility for a

T-period problem, given initial starting stocks and conditions


( ) ( ) ( ){ }1 1( ) max , ,t

t t t t t t tcV x f c x V x x g x cβ + += + =

( ){ } ( ){ }( ) max maxc c

V x c V x x x c c V x cα αβ β+ += + = − = + −

Assumptions◦ Value functions that are continuous in the

controls and state variables.◦ Value functions that are concave in the controls

and state variables.◦ A decision-maker who optimizes the sum of

expected discounted value.



The Marie-Antoinette problem

“Let them eat cake…”

The cake eating example: CakeEatingDP_Analytical_Day3.gms

Using the generic DP previously used,

we can derive two key conditions that hold at the optimal solution:

1. “Euler” equation:

2. ‘Benveniste-Scheinkman’ condition:


( ) ( ) ( ){ }1 1( ) max , ,t


( ) ( ) ( )( )0 , , ,c t t c t t c t tf c x g x c V g x cβ= + ⋅ ⋅

( ) ( ) ( )( )( ) , , ,x t x t t x t t x t tV x f c x g x c V g x cβ= + ⋅ ⋅

Closed-form example of the cake-eating problem, written as:

Conversely to show that it is an infinite-horizon problem:

Next, calculate marginal rate of substitution between the current and future period:


( ) ( ){ }1 1( ) maxt

t t t t t tcV x u c V x x x cβ + += + = −

11

1( ) ( ) . .t

t t t tt

U u c s t x x cβ∞

−+

=

= = −∑c

( )( ), 1

1

( ) tt t

t

u cMRS

u cβ++

′=

′c

Using a simple utility function: , the problem would become:

And, after dropping the time sub-script, we obtain:


( ) ( )t tu c c α=

( ){ }1 1( ) max ( )t

t t t t t tcV x c V x x x cα β + += + = −

( ){ } ( ){ }( ) max maxc c


Backward recursion example of the cake-eating problem (“no tomorrow”):

We expect carry-over value is zero, so you eat all of the cake in the last period. Using this knowledge, combined with the function to used to determine carry-over value from the previous period, and using the Euler condition (first-order conditions), we define optimal consumption as:


( ){ }1 0( ) maxc

V x c V x cα β= + −

11

1 11 1

*

1 1x xc

α

α α

ββ β

−

− −= =

+ +

Using the optimal consumption function and substituting back into the Bellman equation and simplifying with algebra, we obtain:

By defining , the value function can be written as:

Next, solve the Bellman equation problem, then take the first-order conditions (w.r.t. consumption), to acquire the Euler condition, which yields:


( )11

1

2 ( ) 1V x xαα

αβ −−

= + ⋅

( )11

1

11 αα

β −−

+ = Θ

2 1( )V x xα= Θ ⋅

( ) ( ) ( )( ) ( )

11

11

1 11 1

11

1 11 1

x xc x cα

α

α α

ββ

β β

−

−

− −

⋅Θ= ⋅Θ ⋅ − = =

+ ⋅Θ + ⋅Θ

Substituting this function back into the Bellman equation, to acquire the maximized value, we acquire:

By defining , and using algebra to simplify, the value function can be written as:


( ) ( )( ) ( )

1 11 1

* *3 2

1 1

( )1 1

x xV x c V x c xα α

α α

αβ β

β β− −

= + − = + − + ⋅Θ + ⋅Θ

( )( )11

1

1 21 αα

β −−

+ ⋅Θ = Θ

3 2( )V x xα= Θ ⋅

From the previous results, we see the carry-over value functions have the following ‘equation of motion’:

Using this, we can simulate forward, from a starting value to reach convergence. We infer that:

Which we can substitute into the equation of motion and solve for the steady-state parameter value:


( )1

11

11s sα

α

β −−

− Θ = + ⋅Θ

1s s−Θ = Θ = Θ

11

11 α

αβ −

− Θ = −

Substituting the parameter value into our value function, yields:

From this, we can define the infinite-horizon Bellman equation as:


11

1( ) 1V x x xα

αα αβ −

−

∞ = Θ⋅ = − ⋅

( ){ }11

1( ) max 1

cV x c x cα

α αα β β −−

= + ⋅ − ⋅ −

Using backward-recursion, we can obtain the closed-form solution to the infinite-horizon value function:

Additionally, we can obtain the function that determines consumption as a function of the current state:

Derived directly from closed-form solution of the DP problem

Observed habits◦ Consumption and stock


11

1( ) 1V x xα

ααβ −

−

∞ = − ⋅

( ) ( ) ( )( )1

11 1 1

1 1 111

11

11

1 11 11

x x xc x α

α α αα

α

αβ

ββ β ββ

−

− − −−

−

−= = = = −

+ ⋅Θ ++ ⋅ − −

Solution properties◦ Solve Bellman equation in GAMS◦ Use the derived “policy function” to calculate consumption over

time

Verify first-order conditions hold for:

◦ First-order condition with respect to consumption (Euler equation)

◦ Envelope condition w.r.t. state variable (Benveniste-Scheinkmancondition)

◦ All implying the condition below holds over optimal path


( ){ }( ) maxc

V x c x c αα β= + ⋅Θ⋅ −

( ) 110 c x c ααα α β −−= − ⋅ ⋅Θ ⋅ −

( ) ( ) ( ) ( )1 11 1( ) ( )x xV x V x x x x xα αα αβ α α β β− −− −+ += ⋅ → ⋅Θ⋅ = ⋅ ⋅Θ ⋅ → = ⋅

( )1

1x x αβ+ −= ⋅

Steps for value function iteration:◦ Set a convergence tolerance of ◦ Make an initial guess, call it V, for the value function

at each possible state. The Contraction Mapping Theorem guarantees convergence for any starting value.

◦ Compute the value function using V, call it TV.◦ Compute ◦ Check if If true, solution has converged. If false, update V with TV and go to (c). Repeat until

convergence.

Discretizing the state-space Chebychev polynomial approximation


Functional fixed point equation◦ Domain contains an infinite number of points◦ Use approximation and interpolation methods:

Select a degree for the basis functions, n Require n conditions (equations), by selecting a series n of interpolation

nodes, Require that and are equal


1

ˆ ( ) ( )n

j jj

f x c xφ=

=∑

nxf f̂

( ) ( ) ( )1

ˆn

i j j i ij

f x c x f xφ=

= =∑

Chebychev Nodes◦ Nodes:

◦ Chebychev nodes are nearly optimal – in that they span the state space in the most efficient way


0.5cos , 1,2,...,2 2i

a b b a n ix i nn

π+ − − + = + ∀ =

Chebychev Polynomials◦ Select our basis functions using Chebychev

Polynomials Order i is defined over [-1,1], where we can

map [a,b] to [-1,1] by:

Polynomial has the recursive formulation:


( )( )

21

x az

b a−

= −−

1 2( ) 2 ( ) ( )j j iT z zT z T z− −= −

Evaluate a Chebychev polynomial of order iby:◦ Evaluate the first 2 order polynomials◦ For order i>1 we use these values and the recursive

formula◦ Evaluate the resulting polynomial

Polynomial interpolation matrix has typical element:


( )( )0.5 1cosij

n i jn

πφ

− + − =

Figure 1. Graph of the Chebychev Polynomial Terms over its Domain

-1.5

-1

-0.5

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5

x

Phi(x

)

phi1

phi2

phi3

phi4

phi5

Day 3 Notes 23Howitt and Msangi

Generalize approximation methods and how they solve a function equation (specifically, the Bellman equation)

Find a function that satisfies a known function g, the relationship:

Approximate using a linear combination of n basis function:


:[ , ]f a b

( ), ( ) 0 [ , ]g x f x for x a b= ∈f

( ) ( )1

ˆn

i j j ij

f x c xφ=

=∑

Replaced an infinite-decision problem with a finite-dimension specification◦ Will not be able to exactly satisfy approximated

function, so we will specify a convergence criteria◦ Applying Chebychev nodes to the Bellman equation to

approximate the value function:

where is the coefficient for the ith polynomial term and is defined over the [-1,1] interval given by the mapping.


( )( ) ( )i ii

V x c M xφ=∑

( )( )i M xφic

“Chubby-chev” – eating cake example: CakeEatingDP_ChebyApprox_Day3.gms

General problem:

rewritten as:

Instead of backward recursion, we might choose to use the collocation method for a numerical solution

“Value function iteration”◦ Chebychev polynomial◦ Bellman equation


( ) ( ) ( ){ }1 1( ) max , ,t


( ){ } ( ){ }( ) max maxc c


Polynomial approximation:

where j refers to the order of the polynomial, and coefficients are defined by to avoid notation conflicts in previous examples

Values of each polynomial term are evaluated with respect to the state variable (x)

A 3rd order Chebychev polynomial approximation will be written as:


( )( )( ) j jj

V x a M xφ=∑

ja

( )jφ ⋅

( )( ) ( )( ) ( )( ) ( )( )1 1 2 2 3 33

( ) j jj

V x a M x a M x a M x a M xφ φ φ=

= Φ = + +∑

We chose 3 points over the domain of the state variable, defined by the interval

Solve the Bellman equation for current-period values of the state variable to obtain a sequence of maximized values:

Where each solution yields a single scalar numerical value.

Then, equate the polynomial approximation terms at the computation ‘nodes’ (which we selected, not Chebychev):


( )1 2 3, ,x x x,low upx x

( ){ }( ){ }( ){ }

1 1 1

2 2 2

3 3 3

( ) max

( ) max

( ) max

c

c

c

V x c V x c v

V x c V x c v

V x c V x c v

α

α

α

β

β

β

= + − =

= + − =

= + − =

( )( ) ( )( ) ( )( )( )( ) ( )( ) ( )( )( )( ) ( )( ) ( )( )

1 1 1 2 2 1 3 3 1 1

1 1 2 2 2 2 3 3 2 2

1 1 3 2 2 3 3 3 3 3

a M x a M x a M x v

a M x a M x a M x v

a M x a M x a M x v

φ φ φ

φ φ φ

φ φ φ

+ + =

+ + =

+ + =

Recognizing the results as a linear system, we can define as:

Simplifying to:

or

Approximate an implicit function by choosing a proper set of basis functions◦ Solve for the vector of polynomial coefficients by solving

the inverse problem


( )( ) ( )( ) ( )( )( )( ) ( )( ) ( )( )( )( ) ( )( ) ( )( )

1 1 2 1 3 1 1 1

1 2 2 2 3 2 2 2

3 31 3 2 3 3 3

M x M x M x a vM x M x M x a v

a vM x M x M x

φ φ φ

φ φ φ

φ φ φ

=

11 21 31 1 1

12 22 32 2 2

13 23 33 3 3

a va va v

φ φ φφ φ φφ φ φ

=

=a vΦ

1−=a vΦ

Decision to pump groundwater◦ Current marginal net revenue◦ Future net value of groundwater◦ Pumping costs◦ Probability of future recharge

Expressed as:

subject to

where dt , Ht and Rt are respectively the annual pumping, the height of groundwater and recharges to the aquifer, and Pi are the probabilities of a given level of recharge in a future year

Later – look at the stochastic (SDP) version


( ) ( )1 11

max ,t

t t i t td t if d P f d Hβ

∞

+ ++

+∑ ∑

1t t t iH H d R+ = − +

Ferlo region livestock stocking model (Hein, 2009)◦ Decision variable: long-term stocking density Other assumptions

◦ Livestock sold at time t, ◦ Size of livestock herd in Tropical Livestock Units

(TLU), ◦ Livestock feed on fodder , which is produced on

the land depending on rain and Rainfall Use Efficiency (RUE)

Current profits defined as:


tSL

tTLUtF

trtRUE

( )0 1t t tSL SLπ α α= −

Ferlo region livestock stocking model (Hein, 2009)◦ Decision variable: long-term stocking density Other assumptions

◦ Livestock sold at time t, ◦ Size of livestock herd in Tropical Livestock Units (TLU), ◦ Livestock feed on fodder , which is produced on the land

depending on rain and Rainfall Use Efficiency (RUE)

Current profits defined as:

and are the intercept and slope of the livestock market demand function


tSLtTLU

tFtr tRUE

( )0 1t t tSL SLπ α α= −

0α 1α

RUE indicates the effectiveness to transfer rain to biomass

where are scaling parameters

R is the average rainfall

is the long term stocking density:

where H is grazed land area

Stocking density represents intensiveness of grazing, but does not account for variation of spatial distribution


( ) ( )( )2 22 3 42 2t t t t t tRUE r r SR r Rr vα α α µ µ= + − − − +

, ,vα µ

tSRt

tTLUSR

H=

Fodder production is a product of RUE and land area:

Land production limit depends on TLU feeding requirements (2,500 kg/ha):

TLU changes depend on how many units are sold, plus a growth factor. Growth assumed to follow a logistical function form, with shape parameter :


t tF RUE H=

MAX tt

FTLUPh

=

λ

1 1 tt t tMAX

t

TLUTLU TLU SLTLU

λ+

= − −

SDP Solution◦ Solve using Chebychev polynomial approximation of value

function◦ Define 4 nodes to evaluate the value function

approximation State space (TLU) is [300,600] and maps to [-1,1] interval by:

Transformation back to [300,600]=[L,U] interval calculated by:

◦ Now define interpolation matrix using the recursive formula


2 1ˆ cos , for 1,...,2jjx j nn

π − = =

( )( )ˆ2

jj

x L U U Lx

+ −=

1,

2,

, 1, 2,

1ˆ

ˆ2 3

j

j j

k j j k j k j

xx k

φ

φ

φ φ φ− −

=

=

= − ∀ ≥

Education

Biosight: Quantitative Methods for Policy Analysis : Dynamic Models