Electronic copy available at: http://ssrn.com/abstract=1087405

The Machine Maintenance and Sale Age Model ofKamien and Schwartz Revisited1

October 13, 2006

Alain BensoussanDirector, International Center for Decision and Risk Analysis

School of Management, The University of Texas at Dallas2601 N. Floyd Rd., Richardson, TX 75080

alain.bensoussan@utdallas.eduand

Suresh P. SethiDirector, Center for Intelligent Supply Networks

School of Management, The University of Texas at Dallas2601 N. Floyd Rd., Richardson, TX 75080

sethi@utdallas.edu

Abstract

In this paper, we revisit and clarify the celebrated machine maintenance andsale age model of Kamien and Schwartz (KS) involving a machine subject tofailure. KS formulate and solve the problem as a deterministic optimal controlproblem with the probability of the machine failure as the state variable. Thus,they obtain deterministic optimal maintenance and sale date. We study two un-derlying stochastic models with known and random machine modes, and clarifythe relationship between the resulting value functions to that of KS. In particular,our maintenance and sale date decisions, when the machine is in operation, areprecisely the ones obtained from the deterministic solution of KS. We explain whythat is so. Moreover, we provide a sufficient condition for an optimal maintenanceand sale date policy that is missing in KS. We describe many applications of theKS model in areas other than that of machine maintenance. We conclude thepaper with extensions of the KS problem that are stochastic control problems noteasily solvable or not at all solvable as deterministic problems.

Key Words. optimal control, stochastic control, dynamic programming, variationalinequality, maintenance and replacement

1To appear in Management Science

Electronic copy available at: http://ssrn.com/abstract=1087405

1 Introduction

Over thirty-five years ago, Morton I. Kamien and Nancy L. Schwartz (1971a) wrote a

seminal paper on optimal maintenance and sale age of a machine subject to failure. In

their model, the value of the machine’s output, when in operation, is independent of its

age, but its natural failure rate, called the hazard rate, increases with age. Preventive

maintenance can be applied to make the actual failure rate lower than the machine’s

natural failure rate. The machine while operational can be sold at any time, and a

failed machine cannot be repaired, so it must be junked. At the time of the sale, the

machine owner collects the salvage value of the machine. If the machine fails before it

could be sold, then the owner collects a junk value, assumed to be lower than the salvage

value at any time. The problem is to determine the optimal sale age of the machine

and the optimal preventive maintenance until it fails or it is sold, whichever comes first.

Their objective was to maximize the present value of the expected net returns from the

machine.

While their problem is clearly a stochastic optimal control problem, Kamien and

Schwartz (KS hereafter) formulated it as a deterministic optimal control problem with

the cumulative probability of the machine failing by time t as the state variable. Their

formulation requires the expected profit to be expressed in terms of the state and the

control variables, namely, the cumulative failure probability and the rate of preventive

maintenance, respectively.

Since the publication of the KS paper in Management Science in 1971, there have

been many applications of the KS framework to various areas such as pricing of durable

new products, pricing and advertising policies under the threat of a rival’s entry into the

market, innovation under rivalry, and optimal cheating behavior under various situations.

We will discuss some of these briefly in Section 4, after we have completed our analysis

of the KS model. For further details and additional references, the reader is referred to

Feichtinger and Hartl (1986), Sethi and Thompson (2000), Dockner et al. (2000), and

Sethi and Bass (2003).

1

In each one of these references, like the KS model, a deterministic optimal control

problem is formulated and solved. No one has formulated and solved the underlying

stochastic problems using stochastic dynamic programming. These problems, which in-

volve an optimal stopping time (i.e., the sale date or the machine failure time, whichever

comes first), the dynamic programming method amounts to solving a variational inequal-

ity (VI). In this paper, we formulate the KS problem as a stochastic control problem

involving a stopping time, derive the corresponding VI, and show how its solution re-

lates to as well as completes the solution obtained by KS. Furthermore, we will also

indicate some extensions of the KS problem, which cannot be reduced to at least sim-

ple deterministic optimal control problems, but can be naturally addressed by the VI

method.

In formulating the machine maintenance and sale age problems as a stochastic control

problem, it is natural to assume that the machine operator observes the machine mode

– up or down – at each instant of time. If he does not, then how can he know when

the machine has failed so that he does not have to maintain it any longer and can junk

it to collect the junk value? On the other hand, the KS paper is silent on the issue

of the observability of the machine mode. As a matter of fact, since KS solves only a

deterministic control problem, what is known at time t is only the state variable – the

cumulative probability that the machine has failed by that time. The machine mode is

a stochastic process, and it cannot be directly observed in a deterministic formulation.

What the KS model does is to write the expected profit, that is to be maximized, in

a way so that the cost of maintenance at time t is charged and revenue collected only

when the machine is up at that time, and the junk value at the time of the machine

failure or the salvage value at the optimally determined sale date is collected, whichever

comes first. Thus, the KS objective function represents the expected profit from these

actions that can be carried out only by some one who is able to observe the realization

of the machine mode over time. In this sense, we could say that the machine mode is

indirectly observed in the KS framework. This is articulated further in Section 3.2.

Moreover, KS solve their optimal control problem for a given deterministic sale date,

2

and then use the first-order condition to characterize an optimal sale date. They do

not even raise the issue of why a deterministic sale date T is appropriate in a stochastic

problem.

As it turns out, the KS solution agrees with the solution of the VI formulated in this

paper. This is not a general result, however, and we will show that it happens in this

specific case on account of a certain property of the (random) stopping time at which the

machine is retired (junked or salvaged as the case may be). It is this property, proved

in this paper, that allows them to use a deterministic sale date in their formulation of

the problem. Once this is done, we also show that a deterministic maintenance policy

for an operating machine follows.

In addition, our formulation obtains some other new results. We note that KS do

not prove the existence of an optimal control, they provide a sufficiency result only for

the preventive maintenance to be optimal for any given fixed sale date of the machine,

and they provide a necessary condition for the optimal sale date of the machine if

it has not failed by that time. But they do not provide the sufficiency for the full

problem of determining the optimal sale date simultaneously with the optimal preventive

maintenance. We derive a new condition (Theorem 2.1), which provides the existence

as well as the sufficiency of an optimal solution of the full problem.

Before proceeding to the next section, let us summarize the contributions we make

in this paper.

1) The KS problem of machine maintenance and sale age is a stochastic control

problem with a (random) stopping time for which the appropriate methodology is

that of VI. We derive and solve the VI for the KS problem.

2) KS formulated their problem as a deterministic optimal control problem to obtain

the optimal maintenance policy with a deterministic sale date T, and then opti-

mized over T to obtain the optimal sale date T ∗. Even so, their solution agrees

with our solution using VI. We then show rigorously that this occurs in this case

because of a specific property of the random machine retirement time, namely that

3

this stopping time can be shown to be the earlier of a deterministic sale date T

and the random machine failure time.

3) The VI methodology provides sufficiency condition for the existence of a unique

optimal maintenance and sale date policy.

4) Even though the KS model begins with a working machine, the probability state of

the machine evolves over time. Thus, the KS model involves a value function with

an arbitrary probability. In order to provide a meaning for their value function,

we formulate another underlying stochastic control problem with a random initial

machine mode. We also relate the value function of this problem and that of the

KS model to the value function obtained by VI.

5) We provide some extensions of the KS problem that cannot be reduced easily or

at all to deterministic control problems, but are solvable by the VI methodology.

In relating the KS model to the stochastic control literature, it is clear that the op-

timal maintenance part of the KS model (i.e., without the consideration of the optimal

sale age) is a very special case of the piecewise deterministic control problems formulated

by Davis (1993). In piecewise deterministic control problems, the system evolves in a

deterministic fashion between any two jump times, and at any of these jump times, the

deterministic law of motion switches from one mode to another. Furthermore, the main-

tenance part of the KS model is a particular special case of the piecewise deterministic

control problem in which there is no state variable and only one possible switch between

modes (from a working machine to a failed one), and therefore it can be reduced to

precisely the standard deterministic optimal control problem solved by KS. This con-

nection to the stochastic control literature is also pointed out by Dockner et al. (2000).

In this paper, on the other hand and for the first time, we formulate and solve the full

KS problem including the determination of the optimal sale age as a VI problem.

The plan of the paper is as follows. In Section 2, we formulate the VI for the

underlying stochastic problem with full observations. We provide the solution of the

4

problem, along with the necessary and sufficient conditions for optimality in Section 2.1.

We give a simple example in Section 2.2 to illustrate the results obtained in Section 2.1.

In Section 3.1, we formulate a stochastic control problem with a random initial machine

mode. In Section 3.2, we relate the KS model to the two stochastic problems formulated

in Sections 2 and 3.1. In Section 4, we briefly review some applications of the KS model

to a variety of problems reported in the literature. Section 5 concludes the paper with

suggestions for some extensions of the KS model requiring the VI methodology. The

Appendix provides technical proofs.

2 The Stochastic Control Problem with Full

Observation and the Variational Inequality

In this section, we formulate the optimal maintenance and sale age model of KS as a

stochastic control problem. The problem is to find optimal preventive maintenance on a

machine subject to failure, and an optimal planned sale date of the machine if it is still

operating at the time.

We begin with defining the stochastic process representing the mode of the machine,

whether operational or not, over time. The value 1 will denote a functioning machine and

the value 0 will mean a failed machine. Let (Ω,F ,P) denote the underlying probability

space. We will consider a Markov process Xx,t(s), t ≤ s ≤ Z, with values in 0, 1describing the mode of the machine at time s ≥ t ≥ 0, with its initial mode Xx,t(t) = x.

This process is a controlled process, as it will depend on the maintenance performed on

a functioning machine over time. The nature of this dependence will be specified shortly

hereafter.

Since the maintenance can be carried out only on a functioning machine, let us define

the sigma algebra

F st = σ(X1,t(τ), t ≤ τ ≤ s).

Also, let us define the random failure time θx,t = t as follows:

θ0,t = t

5

and

θ1,t = infs > t|X1,t(s) = 0.

It is clear that θ1,t is an F st -stopping time.

Let us now introduce the stochastic control process U(s), called the preventive main-

tenance rate, adapted to F st . For convenience, we assume 0 ≤ U(s) ≤ 1. Since U(s) is

adapted to X1,t(s), and X1,t(s) is either 0 or 1, U(s) can take only two values for each

s. Since a failed machine cannot be repaired, there is no point maintaining it, and so we

can set U(s) = 0 when X1,t(s) = 0. Then, we only need to define u(s) as the value U(s)

takes when X1,t(s) = 1. We can therefore express U(s) as

U(s) = u(s)1Is<θ1,t ,

where we note that u(s) is deterministic, as long as the machine is up.

We are now ready to specify the Markov process Xx,t(s), t ≤ s,≤ Z, t ≥ 0. In

absence of any maintenance, a functioning machine fails according to its natural failure

rate, also called the hazard rate in the reliability literature. Let us denote this rate by

h(s) satisfying

h(s) ≥ 0, h′(s) ≥ 0. (1)

This failure rate can be lowered by performing preventive maintenance rate u(s), 0 ≤u(s) ≤ 1, on a functioning machine. This reduction in the failure rate is assumed to be

proportional, so that a maintained machine has the failure rate h(s)(1− u(s)). We can

now describe the process Xx,t(s) as follows:

X0,t(s) = 0, t ≤ s ≤ Z, (2)

P (X1,t(s + δ) = 0|X1,t(s) = 1) = h(s)(1− u(s))δ, (3)

P (X1,t(s + δ) = 1|X1,t(s) = 1) = 1− δh(s)(1− u(s)), (4)

P (X1,t(s + δ) = 0|X1,t(s) = 0) = 1, (5)

where δ > 0 is a small increment in time. Equation (2) states that a failed machine

at time zero will remain failed thereafter. Equation (3) says that the probability of

6

a machine, that is functioning at time s, will fail in the small interval (s, s + δ] is

h(s)(1− u(s))δ. Clearly this controlled rate is lower than the natural failure rate h(s) if

u(s) > 0. Equation (4) describes what happens with the remaining probability. Equation

(5) says that a failed machine at time s remains failed thereafter.

The distribution of θ1,t ≥ t is given by

P (θ1,t ≤ s) = Ft(s) = 1− e−∫ s

th(τ)(1−u(τ))dτ . (6)

Its density is given by

ft(s) = Ft(s) = h(s)(1− u(s))e−∫ s

th(ξ)(1−u(ξ))dξ. (7)

We can restate (7) as the differential equation

Ft(s) = h(s)(1− u(s))(1− Ft(s)), Ft(t) = 0. (8)

It may be worthwhile to emphasize that this is a differential equation defined on the time

interval [t, Z], with the initial condition that the machine is in the functioning mode at

time t.

Let us now define the sale date of the machine – a decision variable – as a stopping

time Θ ≥ t with respect to the filtration generated by X1,t(s), s ≥ t. In other words,

the event Θ ≤ s ∈ F st .

In order to define the profit function, let R > 0 denote the constant rate of revenue

produced by a functioning machine at any given time, let S(s) denote the resale value of

a functioning machine at time s, and let J ≥ 0 be the junk value of the failed machine

at any given time. As in KS, we assume

S ′(s) < 0, 0 ≤ J ≤ S(s) ≤ R/r, (9)

where r represents the discount rate.

The cost of reducing the failure rate is given by M(u)h with u denoting the mainte-

nance rate, where

M(0) = 0, M ′(u) > 0, M ′′(u) > 0, 0 ≤ u ≤ 1. (10)

7

As in KS, the machine’s owner envisions a distant but finite horizon Z. We can now

write the value function V (1, t) of the stochastic control problem under consideration as

follows:

V (1, t) = maxt≤Θ≤Z

0≤u(·)≤1

E

[∫ Θ∧θ1,t

te−r(s−t)R−M(u(s))h(s)ds

+e−r(Θ∧θ1,t−t)(J1Iθ1,t≤Θ + S(Θ)1Iθ1,t>Θ)]. (11)

Here we use the notation 1IΓ as the indicator function of a subset Γ ⊂ Ω. Note also that

Θ ∧ θ1,t can be referred to as the retirement time of the machine. In other words, the

machine is retired at time Θ if it is still functioning, or at the failure time θ1,t if it fails

before its planned sale date.

The first term inside the integral is the discounted rate of profit, which is integrated

from any given t to the machine salvage time Θ or machine failure time θ1,t, whichever

comes first. The second term represents the present value of the junk value or the salvage

value of the machine, whichever is applicable. Thus, we can see that (11) defines the

value function V (1, t) representing the expected value of all future cash flows, associated

with an optimal policy and discounted to time t, for a functioning machine at time

t, 0 ≤ t ≤ Z.

Analogous to (11) and for the sake of completeness, we can also define V (0, t). Then,

in view of θ0,t = t, we have

V (0, t) = J. (12)

Here the interpretation is that if we begin with a failed machine at the initial time t, we

can immediately junk it to collect J.

Standard dynamic programming arguments (see Appendix) allow us to show that

V (1, t) satisfies the following VI:

V (1, t) ≥ S(t), V (1, Z) = S(Z), (13)

Vt(1, t)− rV (1, t) + R + h(t)(J − V (1, t))− h(t) minu

[M(u) + (J − V (1, t))u] ≤ 0, (14)

8

[V (1, t)− S(t)] [Vt(1, t)− rV (1, t) + R + h(t)(J − V (1, t))

−h(t) minuM(u) + (J − V (1, t))u] = 0. (15)

It is insightful to provide an economic interpretation of the relations (13)-(15). The

first inequality in (13) requires that the value function of a functional machine at any

given time be at least as great as the salvage value of the machine at that time. If this

were not the case at any time t, then it should be obvious that the optimal solution

would be to salvage the machine at that time. The second equation in (13) provides

the boundary condition at time Z. It merely states the fact that if the machine was not

salvaged before time Z and if it did not fail before that time, then it will necessarily be

salvaged at that time. Thus, the value function V (1, Z) of a functioning machine at time

Z must equal to its salvage value S(Z) at that time. Relation (14) allows for salvaging

the machine at an optimal time. In other words, the profit obtained with the option

of salvaging at any time is at least as large as the profit without the option, i.e., when

the only choice is that of maintaining the machine at that time. Hence the inequality

in (14). Finally, (15) says that at least one of (13) or (14) must hold as an equality at

each instant. The idea being that at each instant, either it is optimal to maintain, or it

is optimal to salvage, or both actions are equally desirable. In the first case (14) holds

as an equality, in the second case (13) holds as an equality, and in the third case both

(13) and (14) hold as equalities.

To summarize, the VI allows for a simultaneous optimization of a continuous control

like the machine maintenance and a stopping time like the retirement time of the ma-

chine. In the next section, we use VI to characterize the optimal solution. In particular,

we see that relations (13), (14), and (15) lead to an optimal deterministic time T ∗.

2.1 Optimal Solution

With the VI in hand, we can define the optimal maintenance and the optimal sale date

of the machine as follows:

9

i) At any time s, if X1,t(s) = 0, then the machine is already junked, and therefore

there is no need to define any maintenance or the sale date.

ii) At any time s, if X1,t(s) = 1 and

V (1, s) = S(s), (16)

then sell the machine. These relations define the optimal retirement time as the

stopping time

Θ∗1,t = θ1,t ∧ T ∗, (17)

where

T ∗ = inf s ≥ t|V (1, s) = S(s) . (18)

iii) At any time s, if X1,t(s) = 1 and V (1, s) > S(s), keep the machine and apply the

preventive maintenance at the rate u∗(s), which is given by

M ′(u∗(s)) + J − V (1, s) = 0 if 0 < u∗(s) < 1,

u∗(s) = 0 if M ′(0) + J − V (1, s) > 0,

u∗(s) = 1 if M ′(1) + J − V (1, s) < 0.

(19)

Thus, the solution of the VI reduces to solving for two deterministic quantities u∗(·)and T ∗. The random character of the solution is incorporated in the random failure time

θ1,t, whose probability distribution depends on u∗(·). Moreover, we will see in Section 5

that u∗(·) and T ∗ can be obtained as the solution of the deterministic optimal control

problem formulated by KS.

An important consequence of (19) is that the optimal maintenance expenditure is

nonincreasing as the machine becomes older. Such policies are often used in practice,

and they arise from the reasonable assumptions made on the hazard rate, the salvage

value, and the maintenance cost function in (1), (9), and (10), respectively.

To specify T ∗, we look for a C1 solution for V (1, t) on (0, Z), in view of the presence

of the term Vt(1, t) in (14) and (15). Then using (14) and (15), we can write the following

10

relations that determine T ∗ and u∗(T ∗) :

M ′u∗(T ∗) + J − S(T ∗) = 0 if 0 < u∗(T ∗) < 1,

u∗(T ∗) = 0 if M ′(0) + J − S(T ∗) > 0,

u∗(T ∗) = 1 if M ′(1) + J − S(T ∗) < 0,

and

R−M(u∗(T ∗))h(T ∗)+J(1−u∗(T ∗)h(T ∗))−[r+(1−u∗(T ∗))h(T ∗)]S(T ∗) = −S ′(T ∗) (20)

if T ∗ < Z. If not, then T ∗ = Z.

Assume an interior solution 0 < T ∗ < Z. Then the solution of the nonlinear VI

(13)-(15) will satisfy

V (1, t) = S(t), T ∗ ≤ t ≤ Z,

Vt − rV + R + h(t)(J − V )− h(t) minu[M(u) + (J − V )u] = 0, 0 ≤ t < T ∗,

V (1, T ∗) = S(T ∗),

(21)

provided that

S ′(t)− rS(t) + R + h(t)(J − S)− h(t) minu[M(u) + (J − S)u] ≤ 0, T ∗ ≤ t ≤ Z,

V (1, t) ≥ S(t), 0 ≤ t < T ∗.(22)

The second relation in (22) will be satisfied if

S ′(t)−rS(t)+R+h(t)(J−S(t))−h(t) minu

[M(u)+(J−S(t))u] ≥ 0, 0 ≤ t ≤ T ∗. (23)

To see this, let

z(t) = S(t)− V (1, t).

Then from the second relation in (21) and (23), we have

z′(t)− [r + h(t)(1− u∗(t))]z(t) ≥ 0, 0 ≤ t ≤ T ∗ and z(T ∗) = 0,

11

where u∗(t) minimizes M(u) + (J − S(t))u for each t ∈ [0, T ∗]. This implies that z(t) ≤0, 0 ≤ t ≤ T ∗, which is the second relation in (22).

We have now proved the following result providing a sufficiency condition under

which an optimal solution exists, and have characterized the optimal policy.

Theorem 2.1 If

S ′(t)− rS(t) + R + h(t)(J − S(t))− h(t) minu

[M(u) + (J − S(t))u] (24)

is decreasing on (0, Z), positive at t = 0, and negative at t = Z, then there is a unique

time T ∗ ∈ (0, Z) that satisfies (20). Then, the optimal retirement time Θ∗1,t, of the

machine is given by (17), and the optimal maintenance policy u∗(s), s ∈ [t, Θ∗1,t), is

given by (19).

In order to explain this result, let us first provide an interpretation of (20) as the

necessary first-order condition for T ∗ to be the optimal sale date of the machine. Con-

sider keeping the machine to time T ∗ + δ. The first two terms in (20) when multiplied

by δ give the incremental net cash inflow (revenue − cost of preventive maintenance),

to which is added the junk value J multiplied by the probability [1−u∗(T ∗)]h(T ∗)δ that

the machine fails during the short time δ. From this, we subtract the third term which

is the sum of loss of interest ρS(T ∗)δ on the resale value and the loss of the entire resale

value, when the machine fails, with probability [1 − u∗(T ∗)]h(T ∗)δ. Thus, the LHS of

(20) represents the marginal benefit of keeping the machine to T ∗ + δ. On the other

hand, the RHS term −S ′(T ∗)δ is the decrease in the resale value from T ∗ to T ∗ + δ.

Hence, equation (20) determining the optimal sale date is the usual economic condition

equating marginal benefit to marginal cost.

With the interpretation of (20) as the first-order necessary condition for T ∗ to be

optimal, we can see that the conditions specified for (24) in Theorem 2.1 give us the

existence of a unique T ∗ along with sufficiency of this T ∗ to be the one that maximizes

the objective function. As we will see in the following example, the decreasing property

of (24) is the second-order condition for T ∗ to be a maximum.

12

2.2 An Example

In this section, we solve a simple example to illustrate the results obtained in Theo-

rem 2.1. Let h(t) = h > 0 and M(u) = αu2/2. Assume S(0) − J ≤ α. Then from

(19),

u∗(t) =S(t)− J

α,

which is always less than or equal to one because of our assumptions in (9). Then,

minu

(M(u) + (J − S(t))u) = −α

2u2(t) = −1

2

(S(t)− J)2

α.

Let us consider

ψ(t) = S ′(t)− rS(t) + R + h(J − S) +h

2

(S − J)2

α. (25)

Then the conditions in Theorem 2.1 amount to

S ′′(t)− (r + h)S ′(t) +h

α(S − J)S ′(t) ≤ 0, (26)

S ′(0)− rS(0) + R + h(J − S(0)) +h

2

(S(0)− J)2

α> 0, (27)

S ′(Z)− rS(Z) + R + h(J − S(Z)) +h

2

(S(Z)− J)2

α< 0. (28)

Thus, ψ(T ∗) = 0, with ψ(t) defined in (25), represents the first-order condition (24).

The condition (26) at T ∗ gives the second-order condition for T ∗ to be a local maximum.

Conditions (26)-(28) imply the conditions specified in Theorem 2.1 for the example.

3 Relationship with the KS Model

With T denoting a deterministic sale date to be determined, KS formulated the problem

as the following deterministic optimal control problem:

max0≤T≤Z

0≤u(·)≤1

∫ T

0e−rs[R−M(u(s))h(s)(1−F0(s))+JF0(s)]ds+e−rT S(T )(1−F0(T )), (29)

13

subject to

F0(s) = (1− u(s))h(s)(1− F0(s)), F0(0) = 0. (30)

Note that the term F0(s)/(1 − F0(s)) is known as the hazard rate in the reliability

literature. Thus from (30), we see that the hazard rate at time s is (1 − u(s))h(s). As

F0(0) = 0 in (30), KS assumed that the machine is operational at time zero. In defining

the objective function (29), KS stated that the various cash flow possibilities – revenue,

maintenance cost, junk value and salvage value – are multiplied by their probabilities

and discounted.

KS used Pontryagin’s Maximum Principle to first solve the problem for a fixed num-

ber T ; see Pontryagin, et al. (1962), Kamien and Schwartz (1991), or Sethi and Thomp-

son (2000). Then they find the optimal sale date T ∗ by finding a maximum of the profit

function (29) with respect to T.

Since the underlying problem is stochastic, the sale date of the machine in general is a

stopping time Θ as formulated in Section 2. But KS did not provide any justification for

using a deterministic T instead. Yet, as we know from Section 2, the solution obtained

by KS agrees with the solution obtained by the VI method. In what follows, we will

provide a rigorous answer for why that is so.

The answer lies in a specific property of the machine retirement time Θ∧ θ1,t proved

in Lemma A.1 in the Appendix. There we show that we can express

Θ ∧ θ1,t = T ∧ θ1,t (31)

for some deterministic T. With this, we can easily see that if we replace Θ by T in (11)

and set t = 0, then (11) reduces to the KS objective function (29); see Appendix A.3. In

other words, given the property (31), the KS objective function is precisely the expected

profit to be maximized.

Next, we take up the question of interpreting the meaning of the deterministic control

problem (29)-(30), when the initial F0(0) 6= 0. In this case, the initial machine mode is

random. Of course, as soon as the machine operator starts operating the machine, he

will observe whether the machine is up or down. One could also interpret a machine

14

with a non-zero initial probability state as one that breaks down with probability F0(0)

as soon as we begin to operate the machine. With this interpretation, such an immediate

breakdown can happen only at the beginning.

Before we formulate the stochastic control problem underlying the deterministic con-

trol problem (29)-(30) with F0(0) 6= 0 in Section 3.1, let us define the general value

function associated with the KS problem, given the probability state Φ at an initial

time t, as

K(Φ, t) = max0≤T≤Z

0≤u(·)≤1

∫ T

te−r(s−t)[R−M(u(s))h(s)(1− Φt(s)) + JΦt(s)]ds

+e−r(T−t)S(T )(1− Φt(T )), (32)

subject to

Φt(s) = (1− u(s))h(s)(1− Φt(s)), Φt(t) = Φ. (33)

Here, to distinguish it from (8) for clarity, we have used the notation Φt, in place of Ft,

when the initial condition Ft(t) = Φ 6= 0. With this notation, (29) becomes K(0, 0).

3.1 The Stochastic Control Problem with Random Initial Ma-chine Mode

We begin with a random machine mode ξ, which takes value 0 with probability Φ and

value 1 with probability 1 − Φ at an initial time t. We assume that ξ and X1,t(s) are

independent. We set

Xξ,t(s) = X1,t(s)1Iξ=1 + X0,t(s)1Iξ=0 = X1,t(s)1Iξ=1,

and define the sigma algebra

Gst = F s

t ∪ σ(ξ).

Also, we define the Gst -stopping time

θξ,t = θ1,t1Iξ=1 + t1Iξ=0.

15

In order to formulate the stochastic control problem, we must define an admissible

control. We defined this to be U(s), which is Gst -measurable and which takes a determin-

istic value u(s) on the set θξ,t > s. Indeed on this set, ξ = 1 and θ1,t > s. Moreover, if

Θ is a Gts-stopping time, we can use Lemma A.1 in the Appendix to obtain

Θ ∧ θξ,t = t1Iξ=0 + Θ ∧ θ1,t1Iξ=1

= t1Iξ=0 + T ∧ θ1,t1Iξ=1,

where T is deterministic with T ≥ t. Therefore,

Θ ∧ θξ,t = T ∧ θξ,t. (34)

Next we obtain the distribution of θξ,t as follows:

P (θξ,t ≤ s) = P (ξ = 0) + P (θ1,t ≤ s ∩ ξ = 1)= Φ + (1− Φ)[1− e−

∫ s

t(1−u(τ))h(τ)dτ ]

= 1− (1− Φ)e−∫ s

t(1−u(τ)h(τ)dτ).

Now consider the problem

W (Φ, t) = maxt≤T≤Z

0≤u(s)≤1

E

[∫ T∧θξ,t

te−r(s−t)R−M(u(s))h(s)ds

+e−r(T∧θξ,t−t)(J1Iθξ,t≤T + S(T )1Iθξ,t>T )],

= ΦJ + maxt≤T≤Z

0≤u(s)≤1

E1Iξ=0

[∫ T∧θ1,t

te−r(s−t)R−M(u(s))h(s)ds

+e−r(T∧θ1,t−t)(J1Iθ1,t≤T + S(T )1Iθ1,t>T )],

= ΦJ + (1− Φ)V (1, t)

= ΦV (0, t) + (1− Φ)V (1, t) = EV (ξ, t). (35)

Equation (35) relates the value function W (Φ, t) of the stochastic control problem

with the initial probability state Φ to the value functions V (0, t) and V (1, t) of the fully

observed problem of Section 2. In other words, it says that if we only know the initial

16

probability Φ of the machine mode at time t, then the best we can do is to obtain its

junk value J with probability Φ and the fully observed value of the functioning machine

with probability (1− Φ). This result makes intuitive sense.

In the next section, we relate the value functions K(0, t), K(Φ, t), W (Φ, t), and

V (1, t) that were introduced in Sections 2 and 3.1.

3.2 Relating K(Φ, t), W (Φ, t), and V (1, t).

If we initialize the time at t instead of 0 in (29) and replace F0(s) by Ft(s) in (30), we

get

K(0, t) = max0≤T≤Z

0≤u(·)≤1

∫ T

te−r(s−t) [R−M(u(s)h(s))(1− Ft(s))

+JFt(s)]ds + e−r(T−t)S(T )[1− Ft(T )]. (36)

Now consider the problem (32) and (33) with 0 < Φ < 1. Let us do the following change

of variable in (36):

Ft(s) =Φt(s)− Φ

1− Φ.

Then,

Ft(s) =Φt(s)

1− Φand 1− Ft(s) =

1− Φt(s)

1− Φ. (37)

Substituting (37) in (36), we get

K(0, t) = max0≤T≤Z

0≤u(·)≤1

∫ T

te−r(s−t)

[R−M(u(s)h(s))1− Φt(s)

1− Φ

+JΦt(s)

1− Φ

]ds + e−r(T−t)S(T )

[1− Φt(T )

1− Φ

].

Since 0 < Φ < 1, we can multiply both sides by 1−Φ, and in view of (23), we obtain

K(0, t)(1− Φ) = K(Φ, t). (38)

In particular, when t = 0, we have

K(0, 0)(1− Φ) = K(Φ, 0). (39)

17

The relation (39) has a straightforward interpretation. It says that if we begin the

KS problem with a machine state Φ, then the value function K(Φ, 0) is equal to the

value K(0, 0), obtained by KS, times the probability 1 − Φ that the machine is in the

working condition.

Next we turn to relate K(Φ, t) and V (1, t). For this, first we integrate (33) to obtain

1− Φt(s) = (1− Φ)e−∫ s

t(1−u(τ))h(τ)dτ . (40)

We can now rewrite (32) as

K(0, t) =K(Φ, t)

1− Φ

= max0≤T≤Z

0≤u(·)≤1

∫ T

te−

∫ s

t(r+h(τ)(1−u(τ)))dτ [R−M(u(s)h(s)) + J(1− u(s))h(s)] ds

+ e−∫ T

t(r+(1−u(τ))h(τ))dτS(T )

. (41)

From (41), we can easily derive the following result relating the value function K(Φ, t)

to the value function V (1, t) obtained in Section 2.

Theorem 3.1 For any Φ, 0 ≤ Φ ≤ 1, we have

K(Φ, t) = V (1, t)(1− Φ) = K(0, t)(1− Φ). (42)

Proof. In Appendix A.3, we prove K(0, 0) = V (1, 0). Similarly, we can prove K(0, t) =

V (1, t). Then from (41), the theorem follows. 2

From (42), we see that K(Φ, t) is the maximum expected profit that we can derive

from a working machine at time t, times the probability that the machine is up at time

t. It is also insightful to examine the definition of K(Φ, t) in (32). To begin with, it is

obvious that K(Φ, T ) = S(T )(1 − Φt(T )) is the expected salvage value at time T, i.e.,

the money received for a working machine at time T, an event that takes place with

probability 1 − Φt(T ). Thus, the last term in (32) is the value K(Φ, T ) discounted to

18

time t. The first term inside the integral is the discounted revenue collected when the

machine is up. The second term JΦt(s) inside the integral represents the junk value

times the failure probability rate. When discounted and integrated, it provides the total

expected discounted junk value of the machine over the interval (t, T ]. Thus, if we begin

at time t with Φt(t) = Φ, then K(Φ, t) represents the total value of the expected cash

flows in the interval (t, T ], discounted to time t.

Note that K(Φ, t) does not include the expected junk value ΦJ of a failed machine

at time t. On the other hand, we see from (42) and (35) that

W (Φ, t) = ΦJ + K(Φ, t) (43)

does. This means that in the stochastic model with random initial machine mode, a

failed machine at the initial time t is immediately junked and its junk value collected

with probability Φ, whereas a working machine results in future cash flows according to

the KS model. Note that K(Φ, t) is not multiplied by 1−Φ in (43), since the KS model

also does not directly observe the machine mode. This is consistent with the relation

K(Φ, t) = V (1, t)(1− Φ) obtained in Theorem 3.1.

In order to see that the inclusion of the term ΦJ in W (Φ, t) given in (43) does not

result in double-counting of the junk value, let us set r = 0 in (32) for simplifying the

exposition. Then, the total expected junk value and the salvage value included in (32)

is ∫ T

tJΦt(s)ds + S(T )(1− Φt(T )) = −ΦJ + Φt(T )J + S(T )(1− Φt(T )). (44)

It is therefore clear that K(Φ, t) includes the junk value with probability Φt(T )−Φ and

the salvage value with probability 1−Φt(T ), and it does not include the expected junk

value ΦJ of an immediate breakdown at the initial time t.

It should be obvious that the value function W (Φ, t) represents the maximum ex-

pected value of total cash flows – consisting of revenues, maintenance costs, junk value

and salvage value – over the interval [t, T ∗] and discounted to time t, given only that

Φt(t) = Φ. Here T ∗ is the optimal sale time for a machine that has not failed by that

time. In this definition, the machine mode is not known at time t or thereafter. Still it

19

is assumed that the preventive maintenance is applied only while the machine is func-

tional, the junk value is collected if and when the machine fails, and a salvage value is

collected if the machine is still functioning at time T ∗. The best way to think of this

is that the machine owner decides on the optimal policy without the knowledge of the

machine mode, and gives the policy to a machine operator who executes the policy with

the full knowledge of the machine mode. It is in this sense that we meant in the intro-

duction that the machine mode is indirectly observed by the machine owner in the KS

framework.

Finally, we show that the necessary condition derived by KS for the sale date T ∗ to

be optimal is the same as (20). KS take the derivative of (29) with respect to T and set

it to zero in order to obtain the first-order condition for an optimal T ∗. Their condition

is that T ∗ satisfies (20).

We should note, however, that KS do not show that there exists a T ∗. Also, they

do not provide a sufficient condition for T ∗ and u∗(·) to be optimal. They do have a

sufficient condition for u∗(·) to be optimal for a given fixed T, however.

In Theorem 2.1 in Section 2, we have provided a sufficient condition for the existence

of an optimal sale date T ∗. Moreover, the VI formulation proves that T ∗ and u∗(·)obtained in Section 2 are optimal.

4 Various Applications of the KS Model

Since the publication of the KS model, the KS framework has been applied to a variety

of different problems. In this section, we can only provide a brief selected review of this

literature, since a complete review would require another full-length paper.

The first application that we discuss was made by Kamien and Schwartz (1971b)

themselves to a problem of limit pricing. They studied a situation of a seller who is

aware that his pricing policy will affect the probability of entry by competing suppliers.

The hazard rate in this model is assumed to be a nondecreasing function of product

20

price p(t) at time t. Thus,

F0(t) = h(p(t))(1− F0(t)), F0(0) = 0,

where h(·) ≥ 0 and h′(·) ≥ 0. This gives us the state equation corresponding to (30).

Kamien and Schwartz (1971b) consider the objective to be that of maximizing the

expected discounted total profit. This completes the description of the optimal control

problem, which clearly corresponds to the maintenance part of the KS model. Kamien

and Schwartz (1971b) show that the optimal pre-entry price tends to fall as the discount

rate drops, the market growth rate rises, the post-entry profit probabilities decline, or

certain non-price barriers to entry fall.

The limit pricing model of Kamien and Schwartz (1971b) was extended by Bour-

guignon and Sethi (1981), where the hazard rate at time t depends not only on the

product price but also on the rate of advertising and on F0(t) in a nonlinear fashion.

The nature of dependence on F0(t) characterizes potential entrants to be aggressive,

cautious, or neutral.

In another paper, Kamien and Schwartz (1972) apply the KS model to the timing

of introducing a new product and its pricing under rivalry. Factors taken into account

by the firm are the increasing cost of compressing the product development period, the

reduction of profit opportunities with prolongation of the development period, and the

probability of rival innovation and imitation that affect the potential reward available to

the firm. The single important uncertainty facing the firm is when a rival will introduce

the same or similar item. The rival may introduce its product before or after the firm

does. Let T denote the product development period or the product introduction time

of the firm. The firm assumes a constant hazard rate associated with the rival’s entry

prior to its planned introduction at time T. Furthermore, the firm believes that the

post-innovation entry probability, conditional on the previous absence of rival entrants,

depends on the price or profitability of the new product. The higher the price, the more

attractive entry may appear to potential rivals, and thereby the shorter the monopoly

period may be. Thus, the hazard rate is a constant h prior to time T and a nondecreasing

21

convex function h(p(t)) of the product price p(t), t ≥ T. The returns to the firm from

the introduction of a new product at time T depend not only on the magnitude of T,

but also on its price policy during the monopoly period and the rival’s introduction

date. With specification of these returns, it is possible to write the expression for the

firm’s expected profit. The problem of the firm is to choose the introduction time T

and the product price p(t) during the monopoly period to maximize the present value

of expected returns less the present value of development costs given the entry behavior

described above.

This problem is a slight variation of the KS model. In the KS model, maintenance

is optimized prior to machine failure or salvage, whichever comes first. Whereas in

Kamien and Schwartz (1972), the price is to be optimized for the possible monopoly

period subsequent to the firm’s product introduction time T. Of course, the planned

time T of innovation is also to be decided. But it could happen that a rival innovates

at a random time before this planned T, in which case the firm receives its reward

as an imitator from time T on. On the other hand, if the firm innovates before the

rival, it reaps the monopoly returns depending on the price it sets. The price it sets

determines the hazard rate of the rival’s entry subsequent to T. Once the rival enters

at a random time, the monopoly period ends, and the firm receives less than monopoly

returns thereafter.

Kamien and Schwartz (1972) show that the inability of the firm to appropriate all

the rewards tends to retard development, but also (by assumption) the firm expects to

receive a larger reward if it innovates rather than if it imitates, so the rush to increase the

likelihood of collecting the innovator’s reward tends to accelerate product development.

The authors combine these two influences in their simple optimal control model, and

study the circumstances under which each of the effects will dominate.

Feichtinger (1982), Mehlmann (1985), and Reinganum (1981,1982) have continued

this line of research. Feichtinger (1982) and Mehlmann (1985) consider rival firms en-

gaged in a race for technological breakthrough. Each firm’s completion time is modeled

by a hazard rate dynamics depending on the intensity of the firm’s research effort. Rein-

22

ganum (1981,1982) considers the probability of successfully winning the race for priority

by any given date to be a function of accumulating relevant knowledge by that time.

Mehlmann (1988) points out that Reinganum’s formulation reduces to that of Feichtinger

(1982) and Mehlmann (1985). Reinganum shows that the game admits a unique dif-

ferentiable subgame perfect equilibrium. This means that each firm may choose its

equilibrium research effort level independent of accumulated knowledge. However, there

exists other equilibria depending on the probability state variable.

Sethi and Bass (2003) applied the KS model to derive the optimal profit-maximizing

price of a durable new product over time. The sales rate dynamics depends on the

product price and on the unsold portion of the market. Specifically, the hazard rate

(i.e., the probability of a purchase by a new customer) increases as the price decreases

in a linear fashion. That is,

F0(t) = (1− F0(t))D(p(t)), F0(0) = 0,

where D(p) is the downward-sloping demand function. Sethi and Bass solve the problem

explicitly, and show that both the price and the sales rate decline over time for finite

horizon problems with or without discounting. In the discounted infinite horizon case,

the price remains constant over time.

There are a number of papers dealing with new product pricing based on the Bass

model. While they do not directly apply the KS model, they do formulate the hazard

rate dynamics (which is more complicated than in the KS model), and a deterministic

optimal control problem to obtain the optimal price over time. This literature is surveyed

in Krishnan, Bass, and Jain (1999).

Luhmer (2000) considers a principal who delegates the job of maintaining a system

to an agent with a higher discount rate than his own. The life of the system is modelled

using the hazard rate dynamics. The first-best problem is to maximize the sum of

principal’s and agent’s profits. The problem is a KS variant in the sense that the utility

from maintaining the system is captured only in form of postponing the terminal loss,

and maintenance cost function is time dependent. Luhmer also considers the second-best

23

problem when the agent’s effort is not observed.

Attempts have also been made to extend the KS framework to include replacement

of machines. Dogramaci and Fraiman (2004) have considered the chain of replacement

problem with the KS model as the basic model, but with the restriction that replacements

can happen only at a set of discrete time instants decided a priori; see also Tapiero (1973).

This gives rise to intervals of no activity, since the machines would almost certainly fail at

times other than the given discrete replacement opportunity times. Indeed, Dogramaci

(2005) has formulated a chain of replacement and hibernation intervals to be considered

in an optimal solution.

Another stream of literature based on the KS model follows the paper of Sethi (1979),

where he obtains the optimal pilfering policy of a thief or a shoplifter. The hazard rate

at time t is the conditional probability density of the shoplifter getting caught at that

time, given that he has not been caught yet, and it depends on how aggressively he

pilfers at that time. With the shoplifter’s loot as his gain and with a one-shot (or lump

sum) penalty of getting caught together with a continuous punishment thereafter repre-

senting the shoplifter’s costs, it is easy to write down the shoplifter’s objective function,

which he intends to maximize. Sethi solves the resulting optimal control problem for

different kinds of thieves depending on their attitudes toward risk. He also derives some

implications for crime control agencies.

Feichtinger (1983) extends the Sethi (1979) model to a differential game between a

thief and the police. The police’s control variable is the rate of law enforcement that

also goes into the hazard rate dynamics to increase the probability of catching the thief.

They show a Nash equilibrium in which law enforcement rate increase monotonically over

time, but the thief’s pilfering rate can be in one of three cases: increasing, decreasing or

constant.

A number of other papers with very interesting titles have appeared following the

Sethi (1979) paper. The titles include: on remuneration patterns for medical ser-

vices (Hartl and Mehlmann, 1986), optimal slidesmanship at conferences (Hartl and

Jørgensen, 1990), on the dynamics of extramarital affairs (Jørgensen, 1992), on Petrarch’s–

24

Canzoniere: rational addition and amorous cycles (Feichtinger, Jørgensen, and Novak,

1999), and optimal blood consumption by vampires (Hartl, Mehlmann, and Novak,

1992).

5 Extensions and Concluding Remarks

In this paper, we have provided an analysis of the two stochastic problems underlying the

well-known KS machine maintenance and sale age model. KS formulated their problem

as a deterministic optimal control problem. Yet their optimal solution is the same as

those in the two stochastic formulations. While this is not a general result, we show

why it is so in this case. This also helps clarify the KS model. In addition, we obtain

conditions for the existence and sufficiency of the optimal solution. Finally, we relate

the value functions of the three different formulations of the problem.

Since the KS problem is a stochastic control problem with optimal stopping, the

appropriate method of its solution is via variational inequalities (VI). We derive the

variational inequality for the KS problem and obtain its solution. VI is also a powerful

method that can be used to deal with many extensions of the KS problem. In what

follows, we describe some of these extensions. Some of them allow for a deterministic

formulation in terms of a set of nested optimal control problems that are very hard to

analyze. But others do not permit a transformation to deterministic control problems,

and these must be treated by VI.

The first set of extensions deal with incorporating machine replacements. We have

already mentioned the works of Dogramaci and Fraiman (2004) and Dogramaci (2005),

who incorporate the possibility of replacements in the KS problem only at a pre-

determined set of time instants. They are able to use the KS framework to treat the

resulting problem. However, if we must replace machines at their optimal retirement

time so as to avoid any interval of non-activity, then a VI formulation is needed.

The second set of extensions deal with introducing additional machine modes in

which the machine performance has deteriorated. In these partial modes, machines

25

produce at lower rates than when they are fully operational. It is possible in these

problems to formulate a nested set of deterministic optimal control problems. By this

we mean a KS type formulation in which the junk value is replaced by an expected value

in terms of the value functions of one or more KS models. A corresponding KS model

kicks in at the random time at which the current working machine changes to a partial

working mode. Furthermore, these models may also be nested, in turn, depending on the

number of partial modes and the structure of the transition probability matrix. Since

these value functions cannot be obtained in close forms, the nested formulation is rather

intractable. Moreover, the proof of the stopping time property required to justify the

nested deterministic formulation is also not straightforward. On the other hand, this set

of extensions can be formulated as VI and rigorously studied. Moreover, if we include

the possibility of repair by taking down the machine from operation for a period of

time, then the problem cannot be reduced even to a set of nested deterministic control

problems.

Lastly, we discuss the probability of introducing another state variable in a machine

maintenance and sale age model. For example, if we introduce a production rate as

another control variable in the KS model, which would give rise to a state variable called

inventory, we would once again have a problem that cannot be reduced to a deterministic

problem, but is solvable using VI. An interesting example of an additional state variable

occurs in Harris and Vickers (1995). They formulate a stochastic differential game played

between an exporting and an importing country. The exporting country E owns a finite

amount of exhaustible resources, which the importing country I uses. Country I seeks to

invent a cheaper, perfect substitute technology making it independent of the imported

resource. Country E’s control is the extraction rate, whereas country I’s control is R&D

effort resulting in a hazard rate dynamics for the probability of a successful invention.

Country E aims to maximize its expected profit from the sale of its resource, given that

the profit rate falls to zero subsequent to country I’s random invention time. Country

I maximizes expected consumer surplus from the use of the resource less the cost of its

R&D program. While this model uses a KS type formulation for the hazard rate, it also

26

has the remaining resource as another state variable. Because of the presence of this

additional state variable, this problem cannot be cast into a deterministic optimal control

problem, since the stopping time also depends on the amount of the remaining resource.

Since there is no optimal stopping time to be determined in this problem, Harris and

Vickers use dynamic programming for their analysis. If we were to incorporate a time

at which country I may decide to abandon its R&D effort prior to innovation, either

because it has become too expensive to continue investing in R&D or too unlikely to

result in a successful innovation, then once again we must use VI to study the problem.

Acknowledgement

We are grateful to Ali Dogramaci, Gerhard Sorger, Paul Zipkin, the AE, and the re-

viewers of this paper for their constructive comments on earlier drafts of this paper.

References

[1] Bensoussan, A. and J.-L. Lions (1982). Applications of Variational Inequalities in

Stochastic Control, North Holland, Amsterdam.

[2] Bourguignon, F. and Sethi, S.P. (1981). Dynamic Optimal Pricing and (Possibly)

Advertising in the Face of Various Kinds of Potential Entrants, Journal of Economic

Dynamics and Control, 3, 119-140.

[3] Davis, M.H.A. (1993). Markov Models and Optimization, Chapman & Hall, London,

UK.

[4] Dockner, E.J., S. Jørgensen, N.V. Long, and G. Sorger (2000). Differential Games in

Economics and Management Science, Cambridge University Press, Cambridge, UK.

[5] Dogramaci, A. (2005). Hibernation Durations for Chain of Machines with Main-

tenance Under Uncertainty, Chapter 14 in Optimal Control and Dynamic Games:

27

Applications in Finance, Management Science and Economics, C. Deissenberg and R.

Hartl (Eds.), Springer, New York, NY, 231-238.

[6] Dogramaci, A. and N.M. Fraiman (2004). Replacement Decisions with Maintenance

Under Uncertainty: An Imbedded Optimal Control Model, Operations Research, 52,

5, 785-794.

[7] Feichtinger, G. (1982). Optimal Policies for Two Firms in a Non-cooperative Differ-

ential Game in Optimal Control Thoery and Economic Analysis, G. Feichtinger (Ed.),

North-Holland, Amsterdam, 373-398.

[8] Feichtinger, G. (1983). A Differential Games Solution to a Model of Competition

between a Thief and the Police, Management Science, 29, 686-699.

[9] Feichtinger, G. and R.F. Hartl (1986). Optimale Kontrolle Okonomischer Prozesse:

Anwendungen des Maximumprinzips in den Wirtschaftswissenschaften, Walter De

Gruyter, Berlin.

[10] Feichtinger, G., Jørgensen, S., and Novak, A. (1999). Petrarch’s Canzoniere: Ratio-

nal Addiction and Amorous Cycles, Journal of Mathematical Sociology, 23, 3, 225-240.

[11] Harris, C. and J. Vickers (1995). Innovation and National Resources: A Dynamic

Game with Uncertainity, RAND Journal of Economics, 26, 418-430.

[12] Hartl and Jorgensen 1990 Hartl, R.F. and S. Jørgensen (1990). Optimal slides-

manship in conferences with unpredictable chairmen, Optimal Control Application &

Methods, 11, 143-155.

[13] Hartl, R.F. and A. Mehlmann (1986). On remuneration patterns for medical ser-

vices, Optimal Control Applications & Methods, 7, 185-193.

[14] Hartl, R.F., A. Mehlmann, and A. Novak (1992). Cycles of fear: optimal periodic

blood-sucking rates for vampires, Journal of Optimization Theory and Applications,

75, 3, 559-568.

28

[15] Jørgensen, S. (1992). The dynamics of extramarital affairs, in Dynamic Economic

Models and Optimal Control, G. Feichtinger (Ed.), North-Holland, Amsterdam, 239-

266.

[16] Kamien, M.I. and N.L. Schwartz (1971a). Optimal Maintenance and Sale Age for

a Machine Subject to Failure, Management Science, 17, 427-449.

[17] Kamien, M.I. and N.L. Schwartz (1971b). Limit Pricing and Uncertain Entry,

Econometrica, May, 441-454.

[18] Kamien, M.I. and N.L. Schwartz (1972). Timing of Innovations Under Rivalry,

Econometrica, 40, 1, 43-60.

[19] Kamien, M.I. and N.L. Schwartz (1991). Dynamic Optimization: The Calculus

of Variations and Optimal Control in Economics and Management, Second Edition,

North Holland, New York, NY.

[20] Krishnan, T.V., F.M. Bass, and D.C. Jain (1999). Optimal Pricing Strategy for

New Products, Management Science, 45, 12, 1650-1663.

[21] Luhmer, A. (2000). A Principal-Agent Problem in Continuous Time in Optimiza-

tion, Dynamics, and Economic Analysis: Essays in Honor of Gustav Feichtinger, E.J.

Dockner, R. F. Hartl, M. Luptacik and G. Sorger (Eds.), Physica-Verlag, New York,

151-162.

[22] Mehlmann, A. (1985). Differentialspiele – Die Analyse Dynamischer Konfliktsitua-

tionen, Hain Scientific, Konigstein, Germany.

[23] Mehlmann, A. (1988). Applied Differential Games, Plenum Press, New York, NY.

[24] Øksendal, B. and A. Sulem (2005). Applied Stochastic Control of Jump Diffusions,

Springer, Berlin.

[25] Pontryagin, L.S., V.G. Boltyanski, R.V. Gamkrelidze, and E.F. Mischenko (1962).

The Mathematical Theory of Optimal Processes, Wiley - Interscience, New York, NY.

29

[26] Reinganum, J.F. (1981). Dynamic Games of Innovation, Journal of Economic The-

ory, 25, 21-41.

[27] Reinganum, J.F. (1982). A Dynamic Game of R&D: Patent Protection and Com-

petitive Behavior, Econometrica, 50, 671-688.

[28] Sethi, S.P. (1979). Optimal Pilfering Policies for Dynamic Continuous Thieves,

Management Science, 25, 6, 535-542.

[29] Sethi, S.P. and F.M. Bass (2003). Optimal Pricing in a Hazard Rate Model of

Demand, Optimal Control Applications and Methods, 24, 183-196.

[30] Sethi, S.P. and G.L. Thompson (2000). Optimal Control Theory: Applications to

Management Science and Economics, Second Edition, Kluwer Academic Publishers,

Boston, MA.

[31] Sethi, S.P. and Q. Zhang (1994). Hierarchical Decision Making in Stochastic Man-

ufacturing Systems, in series Systems and Control: Foundations and Applications,

Birkhuser Boston, Cambridge, MA.

[32] Tapiero, C.S. (1973). Optimal Maintenance and Replacement of a Sequence of Ma-

chines and Technical Obsolescence, Opsearch, 19, 1-13.

A Appendix

A.1 Derivation of the VI for the Value Function Defined in(11)

For convenience of notation, let

π(u(s), s) = R−M(u(s))h(s). (45)

From (11), we have

V (1, t) ≥ maxt+δ≤Θ≤Z

0≤u(·)≤1

E

[∫ Θ∧θ1,t

te−r(s−t)π(u(s), s)ds

30

+e−r(Θ∧θ1,t−t)(J1Iθ1,t<Θ + S(Θ)1Iθ1,t>Θ)]. (46)

Since 1Iθ1,t>t+δ + 1Iθ1,t≤t+δ = 1, 1Iθ1,t≤t+δ1Iθ1,t>Θ = 0, and Θ ∧ θ1,t = θ1,t under the event

[θ1,t ≤ t + δ], we have

V (1, t) ≥ maxt+δ≤Θ≤Z

0≤u(·)≤1

E

1Iθ1,t>t+δ

[∫ Θ∧θ1,t

te−r(s−t)π(u(s), s)ds

+e−r(Θ∧θ1,t−t)(J1Iθ1,t≤Θ + S(Θ)1Iθ1,t>Θ)]

+ 1Iθ1,t≤t+δ

[∫ θ1,t

te−r(s−t)π(u(s), s)ds + e−r(θ1,t−t)J

]. (47)

Since t + δ ≤ Θ ∧ θ1,t on [θ1,t > t + δ], we can break up the first integral on the RHS

from t to t + δ and from t + δ to Θ ∧ θ1,t, and rearrange the terms to write

V (1, t) ≥ maxt+δ≤Θ≤Z

0≤u(·)≤1

E

1Iθ1,t>t+δ

∫ t+δ

te−r(s−t)π(u(s), s)ds

+1Iθ1,t≤t+δ

[∫ θ1,t

te−r(s−t)π(u(s), s)ds + e−r(θ1,t−t)J

]

+e−rδ1Iθ1,t>t+δ

[∫ Θ∧θ1,t

t+δe−r(s−(t+δ))π(u(s), s)ds

]

+ e−r(Θ∧θ1,t−(t+δ))[J1Iθ1,t≤Θ + S(Θ)1Iθ1,t>Θ

]. (48)

In order to further simplify (48), we note that in the first two terms inside the max,

the intervals of integration [t, t + δ] and [t, θ1,t], under the event [θ1,t ≤ t + δ], are of

length δ or smaller. Also, E1Iθ1,t≤t+δ = h(t)(1 − u(t))δ. Thus, by discarding the higher

than first-order terms in δ, we can rewrite these first two terms inside the max as

[π(u(t), t) + Jh(t)(1− u(t))]δ.

As for the second two terms inside the max of (48), we observe that if θ1,t > t + δ, then

θ1,t = θ1,(t+δ). Also, e−rδ ≈ 1− rδ. Thus, these second two terms can be expressed as

(1− rδ)E

1Iθ1,t>t+δ

[∫ Θ∧θ1,(t+δ)

t+δe−r(s−(t+δ))π(u(s), s)ds

]

+ e−r(Θ∧θ1,(t+δ)−(t+δ))[J1Iθ1,t(t+δ)≤Θ + S(Θ)1Iθ1,t(t+δ)>Θ

]

31

= (1− rδ)E

∫ Θ∧θ1,(t+δ)

t+δe−r(s−(t+δ))π(u(s), s)ds

+ e−r(Θ∧θ1,(t+δ)−(t+δ))(J1Iθ1,t(t+δ)≤Θ + S(Θ)1Iθ1,t(t+δ)>Θ)|θ1,t > t + δ

·[1− h(t)(1− u(t))δ]. (49)

32

We can now rewrite (48) as

V (1, t) ≥ max0≤u≤1

[π(u, t) + Jh(t)(1− u)] δ

+(1− rδ)[1− h(t)(1− u)δ] maxt+δ≤Θ≤Z

0≤u(·)≤1

E

∫ Θ∧θ1,(t+δ)

t+δe−r(s−(t+δ))π(u(s), s)ds

+ e−r(Θ∧θ1,(t+δ)−(t+δ))(J1Iθ1,t(t+δ)≤Θ + S(Θ)1Iθ1,t(t+δ)>Θ)|X1t(t + δ) = 1

. (50)

Note that the second max term in (50) is V (1, t + δ). So we have

V (1, t) ≥ max0≤u≤1

[π(u, t) + Jh(t)(1− u)] δ

+ [1− rδ − h(t)(1− u)δ]V (1, t + δ)≥ max

0≤u≤1[π(u, t) + Jh(t)(1− u)] δ

+ [1− rδ − h(t)(1− u)δ](V (1, t) + Vt(1, t)δ) . (51)

Subtracting V (1, t) from both sides, dividing by δ, and then letting δ → 0, we obtain

0 ≥ max0≤u≤1

π(u, t) + Jh(t)(1− u)[r + h(t)(1− u)]V (1, t) + Vt(1, t).

Substituting for π(u, t) and rearranging terms, we obtain

Vt(1, t)− rV (1, t) + R + h(t)(J − V (1, t))− h(t) min0≤u≤1

[M(u) + (J − V (1, t))u] ≤ 0. (52)

Thus, we have derived (14). The boundary condition (13) are obvious. Relation (15) is

simply an either-or statement. See, e.g., Bensoussan and Lions (1982), Sethi and Zhang

(1994), and Øksendal and Sulem (2005) for further details on variational inequalities

and their applications.

A.2 Proof of a Deterministic Sale Date

Lemma A.1 Let Θ be an F st -stopping time, then necessarily

Θ ∧ θ1,t = T ∧ θ1,t

for some deterministic T.

33

Proof. Let us define

T = infω∈Ω

Θ(ω).

Also for any ε > 0, define ωε ∈ Ω such that

Θ(ωε) ≤ T + ε.

Since 1IΘ<Θ(ωε) is FΘ(ωε)t -measurable, we have 1IΘ≤Θ(ωε) = constant on the set θ1,t >

Θ(ωε). It is then necessary that 1IΘ≤Θ(ωξ) = 1 on the set θ1,t > Θ(ωξ). Therefore,

Θ ≤ T + ε on the set θ1,t > T + ε. Let us now consider a sequence εn ↓ 0. Then for

any ω ∈ θ1,t > T, there exists N(ω) such that θ1,t > T + εn, ∀n ≥ N(ω); it follows

therefore that Θ(ω) ≤ T + εn, and thus Θ(ω) ≤ T. This implies that Θ(ω) = T. This

proves that Θ = T on the event θ1,t > T.On the other hand, if θ1,t ≤ T, we have θ1,t ≤ Θ, and thus

Θ ∧ θ1,t = θ1,t = T ∧ θ1,t.

This completes the proof. 2

A.3 Derivation of the KS Objective Function

In view of Lemma A.1, we can replace the stopping time Θ by a deterministic T in the

formulation of V (1, t) in (11). In order to relate it to the KS value function K(0, 0), we

consider initial time t = 0 using the notation in (45), we can now write

V (1, 0) = max0≤T≤Z

0≤u(·)≤1

E

[∫ T∧θ1,0

0e−rsπ(u(s), s)ds + e−r(T∧θ1,0)(J1Iθ1,0≤T + S(T )1Iθ1,0>T )

]

= max0≤T≤Z

0≤u(·)≤1

∫ T

0f0(τ)

∫ τ

0e−rsπ(u(s), s)dsdτ +

∫ ∞

Tf0(τ)

∫ T

0R−rsπ(u(s), s)dsdτ

+EJe−rθ1,01Iθ1,0<T + ES(T )e−rT 1Iθ1,0>T

= max0≤T≤Z

0≤u(·)≤1

∫ T

0

∫ T

sf0(τ)e−rsπ(u(s), s)dτds +

∫ T

0

∫ ∞

Tf0(τ)e−rsπ(u(s), s)dτds

∫ T

0Je−rsP (θ1,0 = s)ds + S(T )e−rT (1− F0(T ))

34

= max0≤T≤Z

0≤u(·)≤1

∫ T

0[F0(T )− F0(s)]e

−rsπ(u(s), s)ds +∫ T

0[1− F0(T )]e−rsπ(u(s), s)ds

+∫ T

0Je−rsF0(s)ds + S(T )e−rT (1− F0(T ))

= max0≤T≤Z

0≤u(·)≤1

∫ T

0e−rsπ(u(s), s)(1− F0(S)) + JF0(S)ds + e−rT S(T )(1− F0(T )),

which is precisely the KS objective function (29), denoted as K(0, 0) just before the

beginning of Section 3.1.

35