Economics 2010c: Lectures 9-10Bellman Equation in Continuous Time

David Laibson

9/30/2014

Outline Lectures 9-10:

9.1 Continuous-time Bellman Equation

9.2 Application: Merton’s Problem

9.3 Application: Stopping problem

9.4 Boundary condition: Value Matching

9.5 Boundary condition: Smooth Pasting

10.1 Stopping problem revisited

10.2 Solving second order ODE’s

10.3 Stopping problem resolved

1 Continuous-time Bellman Equation

Let’s write out the most general version of our problem. Begin with equationof motion of the state variable:

= ( )+ ( )

Note that depends on choice of control .

Using Ito’s Lemma, derive continuous time Bellman Equation:

( ) = ( ∗ ) +

+

( ∗ ) +

1

2

2

2( ∗ )2

∗ = ( ) = optimal value of control variable

• Note that value function is a second order partial differential equation(PDE).

• is the ‘dependent’ variable and and are the ‘independent’ variables.

• To solve this PDE need ‘boundary conditions,’ since many solutions exist.

1.1 Terminal condition.

• Suppose problem ‘ends’ at date

• Then we know that

( ) = Ω( ) ∀

• Can solve the problem using techniques analogous to backwards induction

1.2 Stationary ∞−horizon problem.

• Now value function doesn’t depend on

() = ( ∗) + ( ∗) 0 +1

2( ∗)2 00

• We have a second-order ordinary differential equation (ODE).

• In general a large class of functions are consistent with this ODE.

• To pin down a solution we need to know something about the economicsof the value function

• This will provide constraints that pick out a single solution.

2 Application: Merton’s consumption problem

• Consumer has CRRA utility: () = 1−1−

• Consumer has two assets.

• Risk free: return

• Equity: return + and proportional variance 2

• Consumer invests asset share in equities and consumes at rate

= [( + )− ]+

so, ( ) = [( + )− ] and ( ) =

• Bellman Equation and Ito’s Lemma:

( ) = max

()+( )

max

(()+

"

+

( ) +

1

2

2

2( )2

#

)

• For this problem, doesn’t depend directly on (why?):

() = max

(()+

"

[( + )− ] +

1

2

2

2()2

#

)

• Boundary condition: () = 1−1− So,

1−

1− = max

½()+

∙−[( + )− ]−

2−−1()2

¸¾

• First-order conditions:

−− 2µ

2

¶−−1()2 = 0

0() = − = −

• Simplification implies: =

2

= −1

• Plugging this back into last equation on previous page implies:

−1 =

+

Ã1− 1

!Ã +

2

22

!

• Interesting case: = 1.

• = So ' 005

• = 2

= 0061×(016)2

= 234!

• How many households place 2.34 times their wealth (including humancapital) in the stock market?

3 Application: Stopping problem

• Every instant the firm decides whether to continue and get instantaneousflow payoff, ( ) or to stop and get termination payoff, Ω( )

• We’ll assume that ( ) is increasing in

• Continuous time value function is given by (limit as ∆→ 0):

( ) = maxn( )∆+ (1 + ∆)−1 (0 0)Ω( )

o

• Assume that

= ( )+ ( )

• The solution to this problem is a stopping rule

if ∗() continueif ≤ ∗() stop

• Motivation: Irreversibly closing a production facility.

• Distinguish continuation region and stopping region of state space.

• The stopping region is the set of points h i such that () ≤ ∗()

• In continuation region, use Ito’s Lemma to characterize the value function:

( ) = ( )+( )

= ( )+

"

+

( ) +

1

2

2

2( )2

#

• We need to solve this partial differential equation.

4 Value Matching

Recall that Ω( ) is the termination payoff.

One set of boundary conditions is

lim→∗()

( ) = Ω(∗() )

for all boundary points, h∗() i.

This is referred to as a value matching condition (continuity of the value func-tion at the boundary).

Heuristic proof. There exists a neighborhood (∗ ∗ + ) in which:

( ) = ( )∆+ (1 + ∆)−1 (0 0)

= ( )∆+ (1 + ∆)−1" (+

√∆ 0) +Ω(−

√∆ 0)

2

#

=1

2lim

↓∗() ( ) +

1

2Ω(∗() ) +

= lim↓∗()

( ) +1

2

"Ω(∗() )− lim

↓∗() ( )

#+

This implies that

lim↓∗()

( ) = lim↓∗()

( ) +1

2

"Ω(∗() )− lim

↓∗() ( )

#

which implies

lim↓∗()

( ) = Ω(∗() )

5 Smooth Pasting

We could now apply backwards induction techniques if we knew the “freeboundary” ∗().

To pin down the free boundary, we need another boundary condition, which isderived from optimization and called smooth pasting:

(∗() ) = Ω(

∗() )

Heuristic proof. Suppose slopes to left and right of stopping point are givenby and + where ≥ 0 This is a convex kink. If you stop now you getpayoff Ω If you wait another instant and then stop, you get payoff:

∆+ (1 + ∆)−1"Ω−

√∆

2+(+ )

√∆

2

#

So the net payoff of waiting is

∆+ (1 + ∆)−1"Ω−

√∆

2+(+ )

√∆

2

#−Ω

= ∆+ (1 + ∆)−1"Ω+

√∆

2

#− Ω

Multiply through by (1+∆) simplify and remove terms of at least order ∆

to find:

(1 + ∆)∆+

"Ω+

√∆

2

#− (1 + ∆)Ω =

√∆

2

Intuition: At the boundary, the agent should be indifferent between continuationand stopping. If value functions don’t smooth paste at ∗(), then stopping at∗() can’t be optimal. Better to stop an instant later. If there is a (convex)kink at the boundary, then the gain from waiting is in

√∆ and the cost from

waiting is in ∆ So there can’t be a kink at the boundary. Hence:

(∗() ) = Ω(

∗() )

How would you rule out concave kinks?

Outline Lectures 9-10:

9.1 Continuous-time Bellman Equation

9.2 Application: Merton’s Problem

9.3 Application: Stopping problem

9.4 Boundary condition: Value Matching

9.5 Boundary condition: Smooth Pasting

10.1 Stopping problem revisited

10.2 Solving second order ODE’s

10.3 Stopping problem resolved

6 Stopping Problem Revisited

Assume that a continuous time stochastic process () is an Ito process,

= +

You might imagine that is the price of a commodity produced by this firm.While in operation, the firm has flow profit

() =

Assume that the firm can always costlessly (permanently) exit the industry andrealize termination payoff

Ω = 0

Intuitively, this firm will have a stationary stopping rule,

if ∗ continueif ≤ ∗ stop (exit)

Let 0 = (+∆) and let represent the interest rate. So

() = maxn∆+ (1 + ∆)−1 (0) 0

oIn the stopping region,

() = 0

In the continuation region,

() = ∆+ (1 + ∆)−1 (0)

(1 + ∆) () = (1 + ∆)∆+ (0)

()∆ = (1 + ∆)∆+ (0)− ()

Multiply out and let ∆→ 0 Terms of order 2 = 0

() = +( ) (*)

Now substitute in for ( ) using Ito’s Lemma:

( ) =

"

+

( ) +

1

2

2

2( )2

#

=

" 0 +

2

2 00

#

Substituting this expression into equation (*), we find

() = +

" 0 +

2

2 00

#

which is a second-order ordinary differential equation,

= + 0 +2

2 00

What are our boundary conditions?

Value matching:

(∗) = 0

Smooth pasting:

0(∗) = 0

As → ∞ the option value of exiting goes to zero, so converges to thevalue associated with the policy of never exiting the industry. Hence,

lim→∞

()1

³+

´ = 1We’ll derive this equation later. We could also have written

lim→∞ 0() =

1

7 Solving second order ODE’s

To solve the differential equations that come up in economics, it is helpful torecall a few general results from the theory of differential equations.

Consider a generic second order ordinary differential equation:

00() +() 0() +() () = ()

This equation is referred to as the “complete equation.” Note that (),(), and (), are given functions. We are trying to solve for withindependendent variable .

Now consider a “reduced equation” in which () is replaced by 0.

00() +() 0() +() () = 0

Solving this reduced differential equation will enable us to solve the completeequation. We begin by characterizing the solution of the reduced equation.

Theorem 7.1 Any solution, (), of the reduced equation can be expressedas a linear combination of any two solutions of the reduced equation, 1 and2 that are linearly independent.

() = 11() + 22()

Note that two solutions are linearly independent if there do not exist constants1 and 2 such that

11() +22() = 0 for all

Theorem 7.2 The general solution of the complete equation is the sum of anyparticular solution of the complete equation and the general solution of thereduced equation.

8 Stopping problem resolved

Recall the differential equation that characterizes the continuation region

= + 0 +2

2 00 (complete equation)

Consider the reduced equation,

0 = − + 0 +2

2 00 (reduced equation)

Our first challenge is to find solutions of this equation. Consider the class,

To confirm that this is in fact a solution, differentiate and plug in to find

0 = − + +2

22

This implies that

0 = −+ +2

22

Apply quadratic formula,

=−±

q2 + 22

2

Let + represent the positive root and let − represent the negative root. Anysolution to the reduced equation can be expressed

++ + −

− (general solution to the reduced equation)

The general solution to the complete equation can be expressed as the sumof a particular solution to the complete differential equation and the generalsolution to the reduced equation.

To find a particular solution, consider the payoff function of the policy “neverleave the industry.” The value of this policy is

Z ∞0

−()

Note that

[()] = ∙(0) +

Z

0()

¸=

∙(0) +

Z

0[+ ()]

¸=

∙(0) + +

Z

0()

¸= (0) +

The value of the policy “never leave” is derived with integration by parts:

Z ∞0

−() =Z ∞0

− [(0) + ]

=(0)

+

"−1−

#∞0

−Z ∞0−1−

=1

Ã(0) +

!Hence, our (candidate) particular solution takes the form

() =1

Ã+

!

Confirm that this is a solution to the complete differential equation.

·"1

Ã+

!#= + · 1

+2

2· 0 X

We now can draw all of these pieces together. First, we have our generalsolution to the reduced equation

++ + −

−

where the roots are given by

+ − =−±

q2 + 22

2

Second, we have our particular solution:

1

Ã+

!

So the general solution of the complete equation is

() =1

Ã+

!+ +

+ + −−

Finally, we have boundary conditions.

Value matching:

(∗) = 0

Smooth pasting:

0(∗) = 0

We know

lim→∞ 0() =

1

which implies + = 0 Value matching and smooth pasting imply

(∗) =1

Ã∗ +

!+ −

−∗ = 0 (1)

0(∗) =1

+ −−

−∗ = 0 (2)

Equation (1) implies

−−∗ = −1

Ã∗ +

!

Plugging this expression into equation (2) implies

1

− −

Ã∗ +

!= 0

Hence,

∗ =1

−−

We have,

− =−−

q2 + 22

2

Hence,

∗ = − 2

+q2 + 22

−

0

Some interesting special cases (see problem set):

∗(=0) = − √2

0

lim→∞∗ = −∞lim

→−∞∗ = 0

lim→0

∗ =

(− if ≥ 0

0 if 0

)lim→∞

∗ = −∞

lim→0

∗ =

(−∞ if ≥ 02

2 if 0

)lim→∞∗ = 0

Illustrative calibration:

∗(=0) = − √2

= − √2× 02

= −5

Wait until the gets 5 standard deviations below the break-even threshold( = 0) before shutting down.

Finally, now that we have our solution, it is easy to calculate the option valueof stopping:

() = ()− 1

Ã+

!Substituting in for () yields

() =

⎧⎨⎩ −− if ≥ ∗

−1³+

´if ∗

⎫⎬⎭So as →∞ the option value of stopping goes to zero.

Outline Lectures 9-10:

9.1 Continuous-time Bellman Equation

9.2 Application: Merton’s Problem

9.3 Application: Stopping problem

9.4 Boundary condition: Value Matching

9.5 Boundary condition: Smooth Pasting

10.1 Stopping problem revisited

10.2 Solving second order ODE’s

10.3 Stopping problem resolved