Suresh P. Sethi Gerald. L. Thompsonsethi/Prosper/...xk: state variable at time k. uk: control variable at time k. λk: adjoint variables, respectively, at time k, ∆xk:= xk+1 −xk:

OPTIMAL CONTROL THEORY: APPLICATIONS TO

MANAGEMENT SCIENCE AND ECONOMICS

(SECOND EDITION, 2000)

Suresh P. Sethi

Gerald. L. Thompson

Springer

Chapter 1 – p. 1/37

CHAPTER 1

WHAT IS OPTIMAL CONTROL THEORY?



• Dynamic Systems: Evolving over time.




• Time: Discrete or continuous; Optimal way to control a

dynamic system.





dynamic system.

• Prerequisites: Calculus, Vectors and Matrices, ODE and

PDE.





dynamic system.

• Prerequisites: Calculus, Vectors and Matrices, ODE and

PDE.

• Applications: Production, Finance, Economics, Marketing

and others.


BASIC CONCEPTS AND DEFINITIONS

• A dynamic system is described by state equation:

x(t) = f(x(t), u(t), t), x(0) = x0, (1)

wherex(t) is state variable,u(t) is control variable.

• The control aim is to maximize the objective function:

J =

∫ T

0

F (x(t), u(t), t)dt+ S[x(T ), T ]. (2)

• Usually the control variableu(t) will be constrained as

follows:

u(t) ∈ Ω(t), t ∈ [0, T ], (3)


BASIC CONCEPTS AND DEFINITIONS CONT.

• Sometimes, we consider the following constraints:

(1) Inequality constraints (mixed)

g(x(t), u(t), t) ≥ 0, t ∈ [0, T ] . (4)

(2) Constraints involving only state variables (pure)

h(x(t), t) ≥ 0, t ∈ [0, T ]. (5)

(3) Terminal state

x(T ) ∈ X, (6)

whereX is called thereachable setof the state variable at

time T.


FORMULATIONS OF SIMPLE CONTROL MODELS

• Example 1.1 A Production-Inventory Model.

We consider the production and inventory storage of a given

good in order to meet an exogenous demand at minimum

cost.


THE PRODUCTION-INVENTORY MODEL

State Variable I(t) = Inventory Level

Control Variable P (t) = Production Rate

State Equation I(t) = P (t) − S(t), I(0) = I0

Objective Function Maximize

n

J =

R T

0−[h(I(t)) + c(P (t))]dt

oState Constraint I(t) ≥ 0

Control Constraints 0 ≤ Pmin ≤ P (t) ≤ Pmax

Terminal Condition I(T ) ≥ Imin

Exogenous S(t) = Demand Rate

Functions h(I) = Inventory Holding Cost

c(P ) = Production Cost

Parameters T = Terminal Time

Imin = Minimum Ending Inventory

Pmin = Minimum Possible Production Rate

Pmax = Maximum Possible Production Rate

I0 = Initial Inventory Level



• Example 1.2 An Advertising Model.

We consider a special case of the Nerlove-Arrow advertising

model.


THE ADVERTISING MODEL

State Variable G(t) = Advertising Goodwill

Control Variable u(t) = Advertising Rate

State Equation G(t) = u(t) − δG(t), G(0) = G0

Objective Function Maximize

J =

R∞

0[π(G(t)) − u(t)]e−ρtdt

State Constraint · · ·

Control Constraints 0 ≤ u(t) ≤ Q

Terminal Condition · · ·

Exogenous Function π(G) = Gross Profit Rate

Parameters δ = Goodwill Decay Constant

ρ = Discount Rate

Q = Upper Bound on Advertising Rate

G0 = Initial Goodwill Level



• Example 1.3 A Consumption Model.

We consider a problem of an agent’s consumption of his

wealth over time in a way that maximizes his consumption

utility over his lifetime.


THE CONSUMPTION MODEL

State Variable W (t) = Wealth

Control Variable C(t) = Consumption Rate

State Equation W (t) = rW (t) − C(t), W (0) = W0

Objective Function Max

n

J =

R T

0U(C(t))e−ρtdt + B[W (T )]e−ρT

oState Constraint W (t) ≥ 0

Control Constraint C(t) ≥ 0

Terminal Condition · · ·

Exogenous U(C) = Utility of Consumption

Functions B(W ) = Bequest Function

Parameters T = Terminal Time

W0 = Initial Wealth

ρ = Discount Rate

r = Interest Rate


HISTORY OF OPTIMAL CONTROL THEORY

• Calculus of Variations.

• Brachistochrone problem: path of least time

• Newton, Leibniz, Bernoulli brothers, Jacobi, Bolza.

• Pontryagin et al.(1958): Maximum Principle.


FIGURE 1.1 THE BRACHISTOCHRONE PROBLEM


NOTATION AND CONCEPTS USED

• “=” “is equal to” or “is defined to be equal to” or “is

identically equal to”

• “ :=” “is defined to be equal to,”

• “≡” “is identically equal to,”

• “≈” “is approximately equal to.”

• “⇒” “implies”

• “∈” “is a member of.”


NOTATION AND CONCEPTS USED CONT.

Let y be ann-component column vector andz be anm-component row vector, i.e.,

y =

y1

y2

...

yn

= (y1, . . . , yn)T andz = (z1, . . . , zm),

y := dy/dt andz := dz/dt are defined as

y =dy

dt= (y1, · · · , yn)T andz =

dz

dt= (z1, . . . , zm),

Whenn = m, we can define the inner product

zy = Σni=1ziyi. (7)


NOTATION AND CONCEPTS USED CONT.

More generally, ifA = aij =

a11 a12 · · · a1k

a21 a22 · · · a2k

...... · · ·

...

am1 am2 · · · amk

is anm× k matrix andB = bij is ak × n matrix, we define

the matrix productC = cij = AB, which is anm× n matrix

with components

cij = Σkr=1airbrj. (8)


DIFFERENTIATING VECTORS AND MATRICES

W.R.T. SCALARS

• Let f : E1 → Ek be ak-dimensional function of a scalar

variablet. If f is a row vector, then

df

dt= ft = (f1t, f2t, · · · , fkt), a row vector.

• If f is a column vector, then

df

dt= ft =

f1t

f2t

...

fkt

= (f1t, f2t, · · · , fkt)T , acolumn vector.


DIFFERENTIATING SCALARS W.R.T. VECTORS

• If F (y, z) is a scalar function ofn-dimensional column

vectory andm-dimensional row vectorz, n ≥ 2, m ≥ 2,

then thegradientsFy andFz are defined, respectively, as

Fy = (Fy1, · · · , Fyn

), a row vector, (9)

and

Fz =

Fz1

...

Fzm

, a column vector, (10)


DIFFERENTIATING VECTORS W.R.T. VECTORS

If f : En × Em → Ek is ak-dimensional vector functionf

either row or column,k ≥ 2, i.e.,

f = (f1, · · · , fk) or f = (f1, · · · , fk)T ,

wherefi = fi(y, z) depends on columny ∈ En and row

z ∈ Em, n ≥ 2,m ≥ 2, thenfz denotesk ×m matrix:

fz =

∂f1/∂z1, ∂f1/∂z2, · · · ∂f1/∂zm

∂f2/∂z1, ∂f2/∂z2, · · · ∂f2/∂zm

...... · · ·

...

∂fk/∂z1, ∂fk/∂z2, · · · ∂fk/∂zm

= ∂fi/∂zj,

(11)



CONT.

fy will denote thek × n matrix

fy =

∂f1/∂y1 ∂f1/∂y2 · · · ∂f1/∂yn

∂f2/∂y1 ∂f2/∂y2 · · · ∂f2/∂yn

...... · · ·

...

∂fk/∂y1 ∂fk/∂y2 · · · ∂fk/∂yn

= ∂fi/∂yj.

(12)

Matricesfz andfy are known asJacobianmatrices. Note that

fz = fTz = fzT = fT

zT ,

where byfTz we mean(fT )z andnot (fz)

T .Chapter 1 – p. 20/37


CONT.

Applying the rule (11) toFy in (9) and the rule (12) toFz in (10),

respectively, we obtainFyz = (Fy)z to be then×m matrix

Fyz =

Fy1z1Fy1z2

· · · Fy1zm

Fy2z1Fy2z2

· · · Fy2zm

...... · · ·

...

Fynz1Fynz2

· · · Fynzm

=

∂2F

∂yi∂zj

, (13)



CONT.

andFzy = (Fz)y to be them× n matrix

Fzy =

Fz1y1Fz1y2

· · · Fz1yn

Fz2y1Fz2y2

· · · Fz2yn

...... · · ·

...

Fzmy1Fzmy2

· · · Fzmyn

=

∂2F

∂zi∂yj

. (14)


PRODUCT RULE FOR DIFFERENTIATION

• Let g be ann-component row vector function andf be an

n-component column vector function of ann-component

vectorx, Then

(gf)x = gfx + fTgTx = gfx + fTgx. (15)

If g = Fx, then

(gf)x=(Fxf)x=Fxfx + fTFxx = Fxfx + (Fxxf)T . (16)


VECTOR NORM AND NEIGHBORHOOD

• Thenormof anm-component row or column vectorz is

defined to be

‖ z ‖=√

z21 + · · · + z2

m. (17)

• NeighborhoodNz0of a point, e.g.,

Nz0= z| ‖ z − z0 ‖< ε , (18)

whereε > 0 is a small positive real number.


LITTLE-O NOTATION AND NORM OF A FUNCTION

• A functionF (z) : Em → E1 is said to be of the ordero(z),

if

lim‖z‖→0

F (z)

‖ z ‖= 0.

• Thenormof anm-dimensional row or column vector

functionz(t), t ∈ [0, T ], is defined to be

‖ z ‖=

[

Σmj=1

∫ T

0

z2j (τ)dτ

]

1

2

. (19)


SOME SPECIAL NOTATION

• The concepts ofleft andright limits

x(t−) = limε↓0

x(t− ε) andx(t+) = limε↓0

x(t+ ε). (20)

• Discrete-time models (in Chapter 8-9)

xk: state variable at timek.

uk: control variable at timek.

λk: adjoint variables, respectively, at timek,

∆xk := xk+1 − xk: difference operator.

xk∗ anduk∗: quantities along an optimal path.


SOME SPECIAL NOTATION CONT.

• bang function

bang[b1, b2;W ] =

b1 if W < 0,

undefined if W = 0,

b2 if W > 0.

(21)

• sat function

sat[y1, y2;W ] =

y1 if W < y1,

W if y1 ≤W ≤ y2,

y2 if W > y2.

(22)


SOME SPECIAL NOTATION CONT.

Impulse Control

imp(x1, x2, t) = limε→0

∫ t+ε

t

F (x, u, τ)dτ. (23)

If the impulse is applied only at timet, then we can calculate the

objective function as

J =

∫ t

0

F (x, u, τ)dτ + imp(x1, x2, t) (24)

+

∫ T

t

F (x, u, τ)dτ + S[x(T ), T ].


CONVEX SET

• A setD ⊂ En is aconvex setif for each pair of points

y, z ∈ D, the entire line segment joining these two points is

also inD, i.e.,

py + (1 − p)z ∈ D, for eachp ∈ [0, 1].

• Givenxi ∈ En, i = 1, 2, . . . , l, we definey ∈ En to be a

convex combinationof xi ∈ En, if there existspi ≥ 0 such

thatl

∑

i=1

pi = 1 andy =l

∑

i=1

pixi.


CONVEX HULL

• Theconvex hullof a setD ⊂ En is

coD :=

l∑

i=1

pixi:

l∑

i=1

pi = 1, pi ≥ 0, xi ∈ D, i = 1, 2, . . . , l

.


CONCAVE FUNCTION

• ψ : D → E1, is concave, if for each pair of pointsy, z ∈ D

and for allp ∈ [0, 1],

ψ(py + (1 − p)z) ≥ pψ(y) + (1 − p)ψ(z).

• If ≥ is changed to> for all y, z ∈ D with y 6= z, and

0 < p < 1, thenψ is called astrictly concave function.

• If ψ(x) is a differentiable function on the interval[a, b], then

it is concave, if for each pair of pointsy, z ∈ [a, b],

ψ(z) ≤ ψ(y) + ψx(y)(z − y).


FIGURE 1.2 A CONCAVE FUNCTION


CONCAVE AND CONVEX FUNCTIONS

• If the functionψ is twice differentiable, then it isconcave, if

at each point in[a, b],

ψxx ≤ 0.

In casex is a vector,ψxx needs to be a negative definite

matrix.

• If ψ : D → E1 defined on a convex setD ⊂ En is a concave

function, then the negative of the functionψ, i.e.,

−ψ : D → E1, is aconvex function.


AFFINE AND HOMOGENEOUS FUNCTIONS

• ψ : En → E1 is said to beaffine, if ψ(x) − ψ(0) is linear.

ψ : En → E1 is said to behomogeneous of degree one, if

ψ(bx) = bψ(x), whereb is a scalar constant.

• Saddle Pointψ : En × Em → E1. Formaxx miny, a point

(x, y) ∈ En × Em is called asaddle pointof ψ(x, y), if

ψ(x, y) ≤ ψ(x, y) ≤ ψ(x, y) for all x ∈ En andy ∈ Em.

Note also that

ψ(x, y) = maxx

ψ(x, y) = minyψ(x, y).

Similarly, for minx maxy . Just reverse the inequalities.Chapter 1 – p. 34/37

FIGURE 1.3 A SADDLE POINT


LINEAR INDEPENDENCE AND RANK OF A MATRIX

• A set of vectorsa1, a2, . . . , an fromEn is said to belinearly

dependentif there exist scalarspi not all zero such that

n∑

i=1

piai = 0. (25)

• If the only set ofpi for which (1.25) holds is

p1 = p2 = · · · = pn = 0, then the vectors are said to be

linearly independent.


LINEAR INDEPENDENCE AND RANK OF A MATRIX

CONT.

• Therank (or more precisely the column rank) of anm× n

matrixA, written rank(A), is the maximum number of

linearly independent columns inA.

• An m× n matrix is of full rank if

rank(A) = n.


Documents

Suresh P. Sethi Gerald. L. Thompsonsethi/Prosper/...xk: state variable at time k. uk: control variable at time k. λk: adjoint variables, respectively, at time k, ∆xk:= xk+1 −xk: