Inventory processes:Quasi-regenerative property,

performance evaluation and sensitivityestimation via simulation

Georg Pflug†

Reuven Y. Rubinstein††

† Department of Statistics and Decision Support Systems, University ofVienna,

1010 Vienna, Austria††Faculty of Industrial Engineering and Management

Technion – Israel Institute of Technology, Haifa 32000, Israel

Abstract

We consider a single-commodity, discrete-time, multiperiod (s, S)-policy inventory model with backlog. The cost function may containholding, shortage, and fixed ordering costs. Holding and shortagecosts may be nonlinear.

We show that the resulting inventory process is quasi-regenerative,i.e. admits a cycle decomposition and indicate how to estimate theperformance by Monte Carlo simulation. By using a conditioningmethod, the push-out technique and the change-of-measure method,estimates of the whole response surface (i.e. the steady-state perfor-mance in dependence of the parameters s and S) and its derivativescan be found. Estimates for the optimal (s, S) policy can be calculatedthen by numerical optimization.

Keywords. Inventory, Sensitivity Analysis, Stochastic Optimization,Response Surface Simulation, Change-of-measure Method.

1 Introduction

This paper deals with the inventory process in a classical single-commodity,discrete-time, multiperiod inventory model with backlog. By backlog wemean that the requested demand is backlogged when the system is out ofstock, and filled as soon as adequate replenishment arrives. As in standardsources on inventory theory [15], p. 56 (and following pages) or [17], p. 52,we refer to

1. on-hand inventory as the inventory physically on the shelves,

2. net inventory as the on-hand inventory minus the amount backlogged,

3. inventory position as the on-hand inventory minus backlogs plus onorder. On-order is the inventory already requisitioned but not yet re-ceived.

Furthermore we assume that

1. The lead time is either a nonnegative constant r or a nonnegative ran-dom variable R.

2. Demands occur at integer times.

3. The demand sizes Yj are i.i.d. random variables, which have a densityf , bounded by some constant C: f(u) ≤ C.

4. Inventory control is based on the inventory position Nt, not on the netinventory Lt.

5. Each time the inventory position falls to or below the reorder point s,a replenishment order is placed to raise the inventory up to the levelS; otherwise no order is placed. In other words, the quantity to beordered is the difference between S and the inventory position at thetime the order is placed.

6. Each order causes fixed costs K. In addition, a convex function ϕ(x)describing the holding (x ≥ 0) and shortage cost (x < 0) is given. Astandard choice is the broken linear function

ϕ(x) = c1x+ + c2x

− =

c1x, if x ≥ 0

c2|x|, if x < 0,(1)

wherex+ = max{x, 0}, x− = −min{x, 0}.

1

The position process {Nt} can be written recursively as:

Nt+1 =

{Nt − Yt+1, if Nt − Yt+1 ≥ sS, if Nt − Yt+1 < s,

(2)

where N0 = S.As is well known ([15], page 14), this process is regenerative with the

cycle length

τ = min{t : S −t∑

j=1

Yj ≤ s}. (3)

If the lead time is positive or random, then the process Lt is not regener-ative, but quasi-regenerative (see Appendix A for definition).

S

s

¾ -r ¾ -r

¾ -(quasi-)cycle

Figure 1: A typical trajectory of the process Lt (solid line). The process Nt

is shown by the dashed line.

For convenience, we reparameterize the problem and consider S and

Q = S − s

as the new parameters.The performance function is the steady-state cost per time

`(S, Q) = ES,Q{ϕ(L)}+K

ES,Q{τ} , (4)

2

with L being the steady state distribution of Lt and τ is the cycle time. Hereand later ES,Q denotes the expectation, when the control values are S andQ. If no confusion is possible, we write simply E for the expectation.

For purposes of Monte Carlo estimation, processes which allow unbiasedestimates in finite time are highly preferred. Let us say that a process Xt,which admits a stationary distribution allows a (quasi-) cycle decomposition,if there are stopping times Ti such that for all integrable φ

E{φ(X)} =E{∑Ti+1−1

t=Tiφ(Xt)}

E{Ti+1 − Ti} . (5)

where X is distributed according to the stationary distribution of Xt.The purpose of this paper is to derive (quasi-) cycle decompositions for

the objective (4) and its derivatives w.r.t. S and Q. This problem was alsoaddressed in [5], [6], [7].

The closed form of the objective is rather complex and not easy to handleanalytically. In particular, it involves the knowledge of the t-fold convolutionf ∗t of the density f . Therefore we consider the problem of estimation ofthe objective and its derivatives w.r.t. S and Q efficiently by simulation(without calculating f ∗t). Moreover, we show how to estimate the wholefunction `(S, Q) and its derivatives (sensitivities) from a fixed set of randomdeviates.

Methods for linking optimization with simulation are well known (seeoverviews in [3], [4], [12]). Among them, there is perturbation analysis [2],[8] and the score function method [13]. Moreover, diffusion approximationmay be used to simplify the model ([9]). In this paper, we show how theresponse function and its derivative can be efficiently estimated. Our workanswers the question L’Ecuyer ([10], page 1380), how to estimate derivativeswith respect to threshold parameters using a score function approach. Thereis no need to use finite differences as suggested by L’Ecuyer. Related workby Fu and Hu([5], [6], [7]) provides alternative estimators using perturbationanalysis and conditional Monte Carlo.

The approach works well for renewal situations. Karlin [11] was the firstto notice the close connection between inventory control and renewal theory.We illustrate in section 2 how derivative estimates can be found in suchsituations. Not only that a (push-out) score function estimate exists (see[14] for a description of this method), we also propose a new estimate byconditioning which shows very small variance (see section 2.2).

Notice that we treat here the continuous optimization problem (s, S ∈ R).The discrete optimization problem (s, S ∈ Z) may also solved by Markovpolicy iteration (see [19] and the references therein).

3

The paper is organized as follows. After the section 2 on derivative esti-mation , we present in section 3 the simple case of no lead time. The caseof positive, but constant lead time is treated in section 4. Section 5 dealswith random lead times. Methods of Monte Carlo estimation are presentedin section 6.

2 Derivative estimation in renewal situations

Consider a simple renewal situation, where Yi is a sequence of i.i.d. renewalepochs, each with distribution function F . Renewals take place at times0,Y1,Y2, . . ., where Yt =

∑ti=1 Yi. Set Y0 = 0. Let τ(Q) be the number of

renewals in the interval [0, Q), where Q > 0. τ(Q) has the representation

τ(Q) =∞∑

t=0

I{Yt<Q} (6)

and clearly, no pathwise derivative w.r.t. Q can exist (Here IA denotes theindicator function of the set A). However, there are several ways of findinggood estimates for ∇QE{τ(Q)}. In the following subsections 2.1 and 2.2 wediscuss such methods.

2.1 Push-out and Change-of-measure method

Recall that the change-of-measure allows to use the observations under onedensity as if they would have come from another density: If a random variableW1 has density f1 and another random variable W2 has density f2 such thatf1(u) = 0 implies that f2(u) = 0, then for arbitrary integrable function φ

E{φ(W2)} = E{φ(W1)f2(W1)

f1(W2)}.

For the calculation of ∇QE{τ(Q)} we can proceed as follows. Let Y Qj =

Yj/Q and YQt = Yt/Q. The density of Y Q

j is Qf(Qu). Fix some referenceparameter point Q0. We have

E{∞∑

t=1

I{Yt<Q}} = E{∞∑

t=1

I{YQt <1}}

= E{∞∑

t=1

I{YQ0t <1}

t∏

j=1

Qf(QY Qj )

Q0f(Q0YQ0j )

}. (7)

In (7) the parameter Q, called the structural parameter was pushed out fromthe indicator function to the density f to become a distributional parameter.

4

Now, the expression may be differentiated provided some mild regularityconditions hold, if the density f is differentiable with derivative f ′:

∇QE{∞∑

t=1

I{Yt<Q}}

= E{∞∑

t=1

I{YQ0t <1}∇Q

t∏

j=1

Qf(QY Qj )

Q0f(Q0YQ0j )

}

= E{∞∑

t=1

I{YQ0t <1}

t∏

j=1

Qf(QY Q0j )

Q0f(Q0YQ0j )

t∑

j=1

f(QY Q0j ) + Q2f ′(QY Q0

j )

Qf(QY Q0j )

}

= E{∞∑

t=1

I{YQ0t <1}SQ(Y Q0

1 , . . . , Y Q0t )}. (8)

Here

SQ(Y Q01 , . . . , Y Q0

t ) =t∏

j=1

Qf(QY Q0j )

Q0f(Q0YQ0j )

t∑

j=1

f(QY Q0j ) + Q2f ′(QY Q0

j )

Qf(QY Q0j )

is the score function. Since the structural parameter was pushed out tobecome a distributional parameter and the change-of-measure method wasused, the method is called the push-out-change-of-measure method.

2.2 Differentiation via conditioning

Alternatively to the previous method, one can take a closer look to the struc-ture of expression to be differentiated w.r.t. Q. For the following, differen-tiability of the density is not required.

Recall that F is the distribution function and f is the density of Yi. Wehave

∇QE{I{Yt<Q}} = ∇QE{I{Yt−1+Yt<Q}}= ∇QE{E[I{Yt<Q−Yt−1}|Yt−1]}= ∇QE{F (Q− Yt−1)} = f(Q− Yt−1).

Using this result, we get for τ given by (6)

∇QE{τ} = ∇QE{ ∞∑

t=0

I{Yt<Q}

}

= E{ ∞∑

t=1

f(Q− Yt−1)

}

= E{T } (9)

5

where

T =∞∑

t=0

f(Q− Yt). (10)

Here, we used the convention Y0 =∑0

j=1 Yj = 0 and the fact that f(u) = 0for u ≤ 0.

The estimate (10) has low variance as will be demonstrated by an exam-ple:

Example.Let Yi be distributed according to the Exponential(1) distribution. Then

Yt has Erlang density ut−1

(t−1)!e−u. Consider first the number of renewals τ(Q).

It has expectation 1 + Q, since

E{τ(Q)} = E{ ∞∑

t=0

I{Yt<Q}

}

= 1 +∞∑

t=1

∫ Q

0

ut−1

(t− 1)!e−u du

= 1 +∫ Q

0

∞∑

t=1

ut−1

(t− 1)!e−u du

= 1 +∫ Q

0du = 1 + Q. (11)

We calculate now the variance of τ(Q).

E{τ 2} = E{ ∞∑

s=0

∞∑

t=0

I{Ys<Q}I{Yt<Q}

}

= E{ ∞∑

s=0

∞∑

t=0

I{Ymax(s,t)<Q}

}

= E{ ∞∑

t=0

(2t + 1)I{Yt<Q}

}

= 1 + 3∫ Q

0e−u du +

∫ Q

0

∞∑

t=2

(2t + 1)ut−1

(t− 1)!e−u du

= 1 + 3∫ Q

0

∞∑

t=1

ut−1

(t− 1)!e−u du + 2

∫ Q

0

∞∑

t=2

ut−1

(t− 2)!e−u du

= 1 + 3∫ Q

0du + 2

∫ Q

0u du

= 1 + 3Q + Q2

Finally, the variance of τ(Q) is

Var(τ(Q)) = 1 + 3Q + Q2 − (Q + 1)2 = Q.

6

Let us now turn to the variance of the derivative estimate T =∑∞

t=0 e−(Q−Yt)I{Yt<Q}.To check the correctness, let us first calculate the expectation.

E{T } = E{∞∑

t=0

e−Q+YtI{Yt<Q}} = e−Q∞∑

t=0

E{eYtI{Yt<Q}}}

= e−Q[1 +∞∑

t=1

∫ Q

0eu ut−1

(t− 1)!e−u du = e−Q[1 +

∞∑

t=1

Qt

t!]

= e−Q · eQ = 1. (12)

As to the variance, we first set V = eQT − 1. The expectation of V iseQ − 1. The second moment is

E{V 2} = E{∞∑

t=0

∞∑

s=1

eYt+YsI{Yt<Q}I{Ys<Q}}

= E{∞∑

s=0

e2YsI{Ys<Q} + 2∞∑

s=1

∞∑

k=1

eYs+Ys+kI{Ys<Q}I{Ys+k<Q}}

= E{∞∑

s=0

e2YsI{Ys<Q} + 2∞∑

s=1

∞∑

k=1

e2YsI{Ys<Q}E[∞∑

k=1

eYs+k−YsI{Ys+k−Ys<Q−Ys}|Ys]}

= E{∞∑

s=0

e2YsI{Ys<Q} + 2∞∑

s=1

∞∑

k=1

e2YsI{Ys<Q}∞∑

k=1

(Q− Ys)k

k!}

= E{∞∑

s=0

e2YsI{Ys<Q} + 2∞∑

s=1

e2YsI{Ys<Q}[eQ−Ys − 1]}

= E{2∞∑

s=0

eQeYsI{Ys<Q} −∞∑

s=1

e2YsI{Ys<Q}}

= 2∞∑

s=1

eQ Qs

s!−

∞∑

s=1

∫ Q

0e2u ut−1

(t− 1)!e−u du

= 2eQ[eQ − 1]−∫ Q

0eu du

= 2eQ[eQ − 1]− 1

2[e2Q − 1] =

3

2e2Q − 2eQ + 1.

Finally, the variance of T is

Var(T ) = e−2Q(E(V 2)− [E(V )]2) = e−2Q 1

2e2Q =

1

2.

This is a remarkable fact: The basis variable τ has a variance whichincreases with the mean number of observations, i.e. with Q, its variance isproportional to Q. The estimate of the derivative T has constant expectationand constant variance!

7

This illustrates the quality of the proposed estimate.It is important to notice that the change-of-measure method may also

be applied to the just defined estimate. Changing the measure may furtherreduce the variance. For an example, see section 6.1.

3 Case 0: Zero lead time

To begin with, let us first consider the case of zero lead time: In this caseLt = Nt and therefore Lt is regenerative.

By (2) we have for every cycle

Lt = S − Yt if t ≤ τ

where

Yt =t∑

j=1

Yj; Y0 = 0,

and

τ = min{t : Yt ≥ Q} =∞∑

t=0

I{Yt<Q}. (13)

The objective function (4) is

`(S, Q) =ES,Q{∑τ−1

t=0 ϕ(S − Yt)}+ K

ES,Q{τ} =ES,Q{X0}+ K

ES,Q{τ} (14)

where

X0 =τ−1∑

t=0

ϕ(S − Yt). (15)

Our main concern is to find cycle decompositions of the gradient of theobjective function (14) with respect to S and Q. To this end it is sufficientto find expressions for the derivatives of ES,Q{τ} and ES,Q{X0} w.r.t S andQ separately.

Since τ is independent of S we have

∇SES,Q{τ} = 0.

By Lemma 2 of the Appendix B we get

∇SES,Q{X0} = E{τ−1∑

t=0

ϕ′(S − Yt)} = E{Z0}

8

with

Z0 =τ−1∑

t=0

ϕ′(S − Yt) (16)

For the differentiation with respect to the stopping level Q, we refer tosection 2, where exactly the same model was discussed. By the change-of-measure function approach, we get

∇QE{τ} = E{∞∑

t=1

I{YQ0t <1}SQ(Y Q0

1 , . . . , Y Q0t )}.

Analogously, we have

∇QES,Q{X0} = E{∞∑

t=1

ϕ(S − YQ0t )SQ(Y Q0

1 , . . . , Y Q0t )}

and by the quotient rule we have

∇Q`(S, Q) =∇QES,Q{X0} · ES,Q{τ} − [ES,Q{X0}+ K]∇QES,Q{τ}

E2S,Q{τ}

If we use conditioning, we get (as in section 2.2) the estimate

T =τ∑

t=0

f(Q− Yt−1). (17)

Using Lemma 3 of Appendix B, we obtain for X0 given by (15)

∇QE{X0} = ∇QE{ ∞∑

t=0

ϕ(S − Yt)I{Yt<Q}

}

=∞∑

t=1

∇QE{ϕ(S − Yt)I{Yt<Q}

}

=∞∑

t=1

E {ϕ(S − Yt)|Yt = Q} f ∗t(Q)

=∞∑

t=1

ϕ(S −Q)f ∗t(Q)

= ϕ(S −Q) · ES,Q

{τ∑

t=1

f ∗t(Q− Yt−1)

}

= ϕ(S −Q) · ES,Q{T }. (18)

Here f ∗t denoted the t-fold convolution of f .

9

The resulting expression for ∇Q` is therefore

∇Q`(S, Q) =∇QES,Q{X0} · ES,Q{τ} − [ES,Q{X0}+ K]∇QES,Q{τ}

E2S,Q{τ}

=ES,Q{T }ES,Q{τ}

[ϕ(S −Q)− ES,Q{X0}+ K

ES,Q{τ}

].

4 Case 1: Constant positive lead time

In this case every order being placed arrives after r > 0 units of time.Let T0(= 0), T1, . . . be the regeneration times of the inventory position

process {Nt}. At first glance one might think that {Lt} is also regenerativewith regeneration times T0 = T0 + r, T1 = T1 + r, . . .. This is, however,typically not the case. {Lt} at the i + 1 cycle depends on the demandsYTi+1, YTi+2, . . . governing the i-th regenerative cycle of the process {Nt},and therefore is not regenerative in the usual sense. Exception is the casewhere the demand distribution is exponential (lack of memory). In this case{Lt} is regenerative with cycle length equal to the time elapsed between twoconsecutive upward jumps of the process {Lt}. However, the process Lt isquasi-regenerative in the sense of Definition 2 of Appendix A. This meansthat it allows a quasi-cycle decomposition in the sense of (5), namely (20)below.

The following fundamental relationship gives a good insight in the struc-ture of the process

Lt = Nt−r −t∑

i=t−r+1

Yi, for t ≥ r (19)

between {Nt} and {Lt}, where r is the constant lead time. (19) is easy tosee: Suppose that no demand happens at times t−r+1, t−r+2, . . . , t. Thenthe inventory at hand Lt would just be the position Nt−r, since all ordersarrived. If there is demand at times t− r + 1, t− r + 2, . . . , t, this may havetriggered new orders, but these orders did not yet arrive. Therefore nothinghas to be added to Nt−r, only the meantime demand has to be subtracted.This relation appears already in Zipkin [20], which is a general reference forthe stochastic lead time case.

Using (19), we prove in Appendix C that the steady-state expected per-formance function `(S, Q) = ES,Q{ϕ(L)}+ K

ES,Q{τ} can be expressed as

`(S,Q) =ES,Q{∑τ−1

t=0 ϕ(S − Yr − Yt)}+ K

ES,Q{τ} =ES,Q{X1}+ K

ES,Q{τ} (20)

10

where Yr =∑r

i=1 Yi is an auxiliary random variable, independent of Yj, j ≥ 0

and such that Yj has the same distribution as Yj. In (20) we have introduced

X1 =τ−1∑

t=0

ϕ(S − Yr − Yt). (21)

Let us now turn to the derivatives. As before,

∇SES,Q{τ} = 0

and

∇SES,Q{X1} = E{τ−1∑

t=0

ϕ′(S − Yr − Yt)} = ES,Q{Z1}

where

Z1 =τ−1∑

t=0

ϕ′(S − Yr − Yt). (22)

We derive now an expression for ∇QES,Q{X1}. Arguing as in (18), using

Lemma 3 of Appendix B and the independence between Yr and Yt, we get

∇QES,Q{X1} = ∇QES,Q

{ ∞∑

t=0

ϕ(S − Yr − Yt)I{Yt<Q}

}

=∞∑

t=1

∇QES,Q

{ϕ(S − Yr − Yt)I{Yt<Q}

}

=∞∑

t=1

ES,Q

{ϕ(S − Yr − Yt|Yt = Q

}f ∗t(Q)

=∞∑

t=1

ES,Q

{ϕ(S −Q− Yr)

}f ∗t(Q)

= ES,Q{ϕ(S −Q− Yr)}ES,Q

{τ∑

t=1

f(Q− Yt−1)

}

= ES,Q{V1} · ES,Q{T }, (23)

whereV1 = ϕ(S −Q− Yr) (24)

and T is given by (10). Putting together (9) and (23) we get

∇Q`(S, Q) =ES,Q{T }[ES,Q{V1}ES,Q{τ} − ES,Q{X1} −K]

[ES,Q{τ}]2 . (25)

11

5 Case 2: Stochastic lead time

In a random lead time situation, a highly undesirable effect may occur: Itmay happen, that an order, which is placed later, arrives in fact earlier thananother one. Zipkin [20] discusses assumptions under which such ”overtakingof orders ” cannot happen. In this paper, we require a stronger condition byintroducing the following additional rule:

No-overtaking-rule:No order can be placed, if another order is pending.Thus the order time is either the time, when the inventory position falls belows (when no order is pending) or the time of delivery (when at this time theinventory at hand is below s). In either case, the amount of the order is suchthat the inventory will be filled up to size S.

For a constant lead time r, such an additional rule is not necessary, sincenever ”order overtaking” may take place. Notice therefore, that the no-overtaking-rule with lead time R = r (constant) leads to a different processthan the case 1, considered in section 3. The difference is illustrated inFigures 2 and 3. However, the difference is rather small, if the lead time R isreasonably shorter than the cycle time τ and therefore the probability thatthe inventory falls below s during a pending order is small.

The difference between the processes with and without the no-overtaking-rule is illustrated in Figures 2 and 3.

S

s

¾ -R1

¾ -R2

Figure 2: Process without the overtaking rule: Two orders are pending at

12

the same time.

S

s

¾ -R1

¾ -R2

¾ -(quasi-)cycle

Figure 3: Process with no-overtaking-rule: During a pending order, no neworder can be made.

It turns out, that the process {Lt} can be represented as one component ofa vector process, which is Markovian, ergodic and regenerative (see AppendixF for a proof). The (quasi-)regenerative times are the times when ordersarrive. At the beginning of the quasi-cycle, the value of Lt is S − YR, with

YR =R∑

i=1

Yi,

where R is an independent copy of R and Yi are independent copies of Yi. Anew order is placed at time σ of the cycle, where

σ = min{t ≥ 0 : YR + Yt ≥ Q}, (26)

with Yt as before. If YR ≥ Q, then σ = 0 and an order is immediately placedat time 0 of the quasi-cycle. The order placed at time σ arrives at time σ+Rand this is the time, when the quasi-cycle ends. Let Yt =

∑ti=1 Yi (with Yi

independent copies of Yi) denote the demands appearing between placementand delivery of the order in the actual cycle.

13

The length of the quasi-cycle is τ = σ + R and the performance `(S, Q)is

`(S,Q) =ES,Q{∑τ−1

t=0 ϕ(Lt)}ES,Q{τ}

=ES,Q{∑∞

t=0 ϕ(S − YR − Yt)I{YR+Yt<Q}}ES,Q{∑∞

t=0 I{YR+Yt<Q} + R}

+ES,Q{∑R−1

j=0 ϕ(S − YR − Yσ − Yj)}+ K

ES,Q{∑∞t=0 I{YR+Yt<Q} + R}

=ES,Q{∑σ−1

t=0 ϕ(S − YR − Yt)}ES,Q{σ + R}

+ES,Q{∑R−1

j=0 ϕ(S − YR − Yσ − Yj)}+ K

ES,Q{σ + R}=

ES,Q{X2}+ ES,Q{X3}+ K

ES,Q{σ + R} . (27)

Here

X2 =∞∑

t=0

ϕ(S − YR − Yt)I{YR+Yt<Q} (28)

X3 =R−1∑

j=0

ϕ(S − YR − Yσ − Yj) (29)

and Yσ is the cumulative demand between the beginning of the cycle and the(random) time σ. If σ = 0, then Yσ = 0. Always YR + Yσ ≥ Q.

As to the derivatives, we have

∇SES,Q{σ + R} = 0,

∇SES,Q{X2} = ES,Q{Z2}with

Z2 =σ−1∑

t=0

ϕ′(S − YR − Yt) (30)

and∇SES,Q{X3} = Z3

with

Z3 =R−1∑

j=0

ϕ′(S − YR − Yσ − Yj). (31)

14

Further

∇QES,Q{σ + R} = ES,Q{σ∑

t=1

f(Q− YR − Yt−1)}

and

∇QES,Q{X2} = ϕ(S −Q)ES,Q{∞∑

t=1

f ∗t(Q− YR)}

= ϕ(S −Q)ES,Q{σ∑

t=1

f(Q− YR − Yt−1)}

= ϕ(S −Q)ES,Q{V2},

where

V2 =σ∑

t=1

f(Q− YR − Yt−1). (32)

We rewrite X3 as

X3 =∞∑

t=1

R−1∑

j=0

ϕ(S − YR − Yt − Yj)[I{YR+Yt−1<Q} − I{YR+Yt<Q}

].

and calculate

∇QES,Q{X3} = ES,Q{R∑

j=1

ϕ(S −Q− Y0 − Yj)∞∑

t=1

ES,Q{f ∗(t−1)(Q− YR)}

−ES,Q{R∑

j=1

ϕ(S −Q− Yj)}ES,Q{∞∑

t=1

f ∗t(Q− YR)}

= ES,Q{R∑

j=1

ϕ(S −Q− Y0 − Yj)σ+1∑

t=2

ES,Q{f(Q− YR − Yt−2)}

−ES,Q{R∑

j=1

ϕ(S −Q− Yj)}ES,Q{σ∑

t=1

f(Q− YR − Yt−1)}

= ES,Q{V3},

where Y0 has the same distribution as Yt and is independent of everythingelse and

V3 =R∑

j=1

ϕ(S −Q− Y0 − Yj)σ+1∑

t=2

f(Q− YR − Yt−2)

−R∑

j=1

ϕ(S −Q− Yj)σ∑

t=1

f(Q− YR − Yt−1). (33)

15

Notice that the foregoing formulas are only correct under the no-overtaking-rule. If overtaking would be allowed, the times of delivery would not dissectthe process into quasi-cycles, since the future development would then de-pend heavily on the sizes and times of the pending orders.

6 Monte Carlo Estimation

As we have seen, the estimation of the performance function `(S,Q) requiresthe estimation of the following quantities:

• Case 0: ES,Q{τ} (see (13)), ES,Q{X0} (see (15))

• Case 1: ES,Q{τ}, ES,Q{X1} (see (21))

• Case 2: ES,Q{σ + R} (see (26)), ES,Q{X2} (see (28)), ES,Q{X3} (see(29)).

The sensitivity estimation involves in addition the estimation of

• Case 0: ES,Q{T } (see (10), ES,Q{Z0} (see (16))

• Case 1: ES,Q{Z1} (see (22)), ES,Q{V1} (see (24))

• Case 2: ES,Q{Z2} (see (30)), ES,Q{V2} (see (32)), ES,Q{V3} (see (33)).

The basic estimation of the mentioned quantities is straightforward: E{X}is estimated by 1

N

∑Ni=1 X(i), where X(i) are independent copies of X.

The push-out method allows to estimate the performance and the sen-sitivities different values of parameters (S,Q) using just one single samplepath (one single set of random deviates).

A glance at the definitions of the needed quantities reveals that they areof the same form:

E{X} = E{∞∑

t=0

ψ(S − Yt −Z1)I{Yt+Z2<Q}} (34)

where Yt =∑t

j=1 Yj and Z1,Z2 are some random variables which are inde-pendent of Yj.

Trivially an unbiased estimate of (34) is

∞∑

t=0

ψ(S − Yt −Z1)I{Yt+Z2<Q}. (35)

It is clear that the same random variables Z1,Z2 and Yj may be used in(35) for estimating the values for different values of S and Q. However, the

16

stopping time min{t : Yt + Z2 ≥ Q} determines the number of observationsneeded and this number varies with varying Q.

The push-out (see section 2.1) estimate does not have this drawback:Let Q0 be a reference parameter. Let Y Q0

j = Yj/Q0 and ZQ02 = Z2/Q0.

The density of Y Q0j is Q0f(Q0u). Let k(u) be the density of Z2. Set

XQ0 =∞∑

t=0

ψ(S −Q0YQ0t −Z1)I{YQ0

t +ZQ02 <1} (36)

where YQ0t =

∑tj=1 Y Q0

j . It is clear that XQ0 is an unbiased estimate forEQ0{X}. For a different Q, an unbiased estimate is

XQ =∞∑

t=0

ψ(S −QYQt −Z1)I{YQ

t +ZQ2 <1}. (37)

Now, one may use the same random variables Y Q0 as in (36), change themeasure and get

XQ0,Q =

=∞∑

t=0

ψ(S −QYQ0t −Z1)I{YQ0

t +ZQ02 <1}

Qk(QZQ02 )

Q0k(Q0ZQ02 )

t∏

j=1

Qf(QY Q0j )

Q0f(Y Q0j )

=∞∑

t=0

ψ(S − Q

Q0

Yt −Z1)I{Yt+Z2<Q0}Qk( Q

Q0Z2)

Q0k(Z2)

(Q

Q0

)t t∏

j=1

f( QQ0

Yj)

f(Yj)(38)

as an unbiased estimate of EQ{X}. The advantage of (38) compared to (37)is that the former uses for all Q the same stream of random numbers. Thechoice of the reference parameter Q0 controls the computational effort versusthe precision: If Q0 is chosen small, the stopping time is small and only fewobservations are needed. However, the variance is large. If Q0 is chosen large,then the variance becomes small on the expense of a larger sample needed.

6.1 Numerical Illustration

We illustrate our approach with the following example. We want to findthe function Q 7→ ∇QES,Q{τ} (see (9)), for Exponential(1) distributed Yi.We know that ES,Q{τ} = Q + 1 (compare (9)). The crude estimate for∇QES,Q{τ} is

T (1)(Q) =τ−1∑

t=0

f(Q− Yt) =τ−1∑

t=0

exp(Yt −Q). (39)

17

For the change-of-measure method, we ”push-out” the parameter Q in (39.Let Q0 be the reference parameter YQ

t = Yt/Q. Let τ0 = min{t : Yt ≥ Q0}(which is independent of Q). The change-of-measure estimate is

T (2)(Q) =τ0−1∑

t=0

f(Q− Q

Q0

Yt)

(Q

Q0

)t t∏

j=1

f( QQ0

Yj)

f(Yj)

=τ0−1∑

t=0

(Q

Q0

)t

exp(Yt −Q). (40)

0 0.5 1 1.5 2 2.50

0.5

1

1.5

0 0.5 1 1.5 2 2.50

0.5

1

1.5

Figure 4: Estimate of the function ∇QES,Q{τ} for exponential(1) demands.From the theoretical consideration, we know that ∇QES,Q{τ} = 1 for all Q

(see (12)). The left picture shows the estimate T (1)(Q) (39), the rightpicture shows the push-out estimate T (2)(Q) (40), both based on 100

replications and the reference parameter Q0 = 1. The push-out is evidentlybetter.

We have used the conditioning method and the push-out estimate withreference point Q0 = 3 to estimate the whole response surfaces for the caseK = 1, c = 2, r = 2 (deterministic) and Exponential(1) demands. Sincethe true values are known in this case (see Appendix E), we used them ascomparison. The estimates were based on 100 replications only.

5 10 15 20 25 3010

20

30

1

1.5

2

2.5

3

3.5

4

4.5

5

10 Q

10 S

5 10 15 20 25 3010

20

30

1

1.5

2

2.5

3

3.5

4

4.5

5

10 Q

10 S

18

Figure 5: The estimated response surface (S, Q) (left) and the true curve`(S,Q) (right).

5 10 15 20 25 3010

20

30

−3

−2

−1

0

1

2

3

10 Q

10 S

5 10 15 20 25 3010

20

30

−3

−2

−1

0

1

2

3

10 Q

10 S

Figure 6: The estimated response surface ∇S`(S, Q) (left) and the truecurve ∇S`(S,Q) (right).

5 10 15 20 25 3010

20

30

−2

−1

0

1

2

3

10 Q

10 S

5 10 15 20 25 3010

20

30

−2

−1

0

1

2

3

10 Q

10 S

Figure 7: The estimated response surface ∇Q`(S, Q) (left) and the truecurve ∇Q`(S, Q) (right).

510

1520

2530

5

10

15

20

25

30

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2

10 S

10 Q

510

1520

2530

510

1520

2530

−0.3

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

10 S

10 Q

Figure 8: The difference curves ∇S`(S, Q)−∇S`(S, Q) (left) and

∇Q`(S, Q)−∇Q`(S, Q) (right).

19

6.2 Stochastic Optimization

We mention briefly how to use the presented estimated to solve the prob-lem of optimal (s, S). We illustrate here the Case 1 (Case 2 is similar).To estimate ES,Q{τ}, ES,Q{X1}, ES,Q{T }, ES,Q{Z1}, ES,Q{V1} we take N

replicated independent quasi-cycle observations to get estimates ES,Q{τ},ES,Q{X1}, ES,Q{Z1}, ES,Q{V1}. Based on them we estimate the objective by

(S, Q) =ES,Q{X1}ES,Q{τ}

and its derivatives by

∇S`(S,Q) =ES,Q{Z1}ES,Q{τ}

and

∇Q`(S,Q) =ES,Q{T }[ES,Q{V1}ES,Q{τ} − ES,Q{X1} −K]

[ES,Q{τ}]2respectively (see (25)).

The estimates may be used in the stochastic approximation method.It uses the recursion

(Sn+1

Qn+1

)=

(Sn

Qn

)− 1

n

(∇S`(Sn, Qn)

∇Q`(Sn, Qn)

)(41)

For each step n, a new independent set of estimates (∇S`, ∇Q`) is taken.The number of replicates N per estimate may be set to N = 1 and stillconvergence holds. For the convergence properties of procedure (41) to theoptimal (S∗, Q∗) see for instance [12].

7 Appendix

Appendix A: Regenerative and Quasi-regenerative Processes.Definition 1.(see [16]) A (possibly vector valued) stochastic process Xt

is called is called regenerative, if there is a sequence of increasing stoppingtimes {Tk} such that

• the sequence Tk+1 − Tk is i.i.d. with finite expectation,

• the processes XTk+t; 0 ≤ t ≤ Tk+1 − Tk and XT`+t; 0 ≤ t ≤ T`+1 − T`

have the same distribution and are independent for k 6= `.

20

Definition 2. A process Xt is called quasi-regenerative, if there existsa process Xt, such that the pair (Xt, Xt) is a regenerative ergodic Markovprocess.

Lemma 1. If Xt is quasi-regenerative, then a ratio formula holds, i.e. forevery measurable function, which is integrable with respect to the stationarydistribution π of (X, X), we have for all i ≥ 1

E{φ(X)} =∫

φ(u) dπ(u) =E{∑Ti+1−1

t=Tiφ(Xt)}

E{Ti+1 − Ti} . (42)

Proof. This is a special case of the general regenerative ratio formula,see e.g. [16], p. 26. 2

Appendix B: Piecewise differentiability.Definition 3. The function φ is called piecewise regularly differentiable, ifthere is a function φ′ such that for all h > 0, there is a set Ah such that forsufficiently small h

|φ(s + h)− φ(s)− h · φ′(s)| ≤ Ch2 + ChI{s∈Ah}

with∫

I{s∈Ah} ds ≤ Ch, for some constant C not depending on h.Example. The function ϕ(x) = c1x

+ + c2x− is piecewise regularly dif-

ferentiable with derivative ϕ′(x) = c1I{x>0} − c2I{x<0}.Lemma 2. Suppose that φ is piecewise regularly differentiable. Then, if

the pair of random variables (Z1,Z2) has a bounded density l(z1, z2) and Z2

is bounded, then

∇SE{φ(S −Z1)Z2} = E{φ′(S −Z1)Z2}.

Proof.

|E{φ(S + h−Z1)Z2} − E{φ(S −Z1)Z2} − hE{φ′(S −Z1)Z2}|≤

∫ ∫|φ(S + h− z1)− φ(S − z1)− hφ′(S − z1)|z2l(z1, z2) dz1 dz2

≤ Ch2∫ ∫

z2l(z1, z2) dz1 dz2 + Ch∫ ∫

I{S−z1∈Ah}z2l(z1, z2) dz1 dz2

= O(h2)

2

Lemma 3. Let X,Y be two random variables and let Y have the densityf . Then

∇yE{XI{Y <y}

}= E {X|Y = y} · f(y).

21

Proof. Let F (x|y) be the conditional cdf of X given that Y = y. Wehave

∇yE{XI{Y <y}

}= ∇y

∫ y

−∞

∫ ∞

−∞x dF (x|u) f(u)du =

∫ ∞

−∞x dF (x|y) f(y)

= E {X|Y = y} · f(y).

2

Appendix C: Proof of equation (20).Let N be distributed according to the stationary distribution of {Nt}.

Then, due to the regenerative property of {Nt}, for every fixed a,

ES,Q{ϕ(N − a)} =ES,Q{∑τ−1

t=0 ϕ(Nt − a)}ES,Q{τ}

=1

ES,Q{τ}ES,Q

{ ∞∑

t=0

ϕ(S − Yt − a)I{Yt<Q}

}

Now let a = Yr =∑r

j=1 Yj, where Yj have the same distribution as Yj,but are independent of them. Then

ES,Q{ϕ(N − Yr)} =ES,Q{∑τ−1

t=0 ϕ(Nt − Yr)}ES,Q{τ}

=1

ES,Q{τ}ES,Q

{ ∞∑

t=0

ϕ(S − Yt − Yr)I{Yt<Q}

}

On the other hand, by the fundamental formula (19) and by the fact that∑tj=t−r+1 Yj is independent of Nt−r, we have

ES,Q{ϕ(L)} = ES,Q{ϕ(N − Yr)},

where Yr has the same distribution as∑t

j=t−r+1 Yj, but is independent of N .Appendix D: Analytical expressions for Case 1.Let f be the density of the demands Y , let f ∗t be its t-fold convolution

and F ∗t the corresponding distribution function.The expressions presented in section 3 have the following analytical form:

ES,Q{τ} = 1 +∞∑

t=1

F ∗t(Q)

ES,Q{T } =∞∑

t=1

f ∗t(Q)

22

ES,Q{X1} =∞∑

t=1

∫ ∞

0

∫ Q

0ϕ(S − v − u)f ∗t(u) du f ∗r(v) dv

ES,Q{Z1} =∞∑

t=1

∫ ∞

0

∫ Q

0ϕ′(S − v − u)f ∗t(u) du f ∗r(v) dv

ES,Q{V1} =∫ ∞

0ϕ(S −Q− v)f ∗r(v) dv.

Appendix E: Specialization to exponential demands.Suppose now that the demand sizes Yi follow an exponential distribution

with expectation 1 and that the cost function ϕ(u) is given as in (1), that isϕ(u) = c1u

+ + c2u−. Without loss of generality we may assume that c1 = 1,

i.e. we will consider the function ϕ(u) = u+ + cu−.For this special case, we present explicit expressions of the objective func-

tion `(S, Q) in (20) for the Case 1 (constant lead time). Formulas for Case 0can be found in [11], pp. 280 - 285.

Since the Yt have Erlang density ut−1

(t−1)!e−u, we know from (11) that

ES,Q{τ} = 1 + Q.

We shall now calculate the numerator of the objective function ` in (20),that is

ES,Q{X1} = ES,Q

{ ∞∑

t=0

ϕ(S − Yr − Yt)I{Yt<Q}

}. (43)

Conditioning (43) on the event {S − Yr = A} we get after evaluating theintegrals

ES,Q{X1|S − Yr = A} = A+ +1

2[A+]2 − 1

2[(A−Q)+]2

+c{A− − 1

2[A−]2 +

1

2[(A−Q)−]2}.

Let g(u) be the density of Yr. Then, integrating with distribution ofS − Yr, we get the unconditional expectation

ES,Q{X1} =∫ S

0(S − u)g(u) du +

1

2

∫ S

0(S − u)2g(u) du− 1

2

∫ s

0(s− u)2g(u) du

+c{∫ ∞

S(u− S)g(u) du− 1

2

∫ ∞

S(u− S)2g(u) du +

1

2

∫ ∞

s(u− s)2g(u) du.}

Assume, for instance, that the lead time r = 2. In this case Y2 has Erlangdensity g(u) = ue−u and after some algebra, we find

23

ES,Q{X1} =1

2S2 − S − 2− 1

2s2 + 2s− e−S + e−s(s + 3)− ce−S + ce−s(s + 3)

= (1 + Q)(S − 2)−Q2/2− (c + 1)e−S + (c + 1)e−(S−Q)(S −Q + 3).

Taking (11) into account, the objective function (20) can be written as

`(S, Q) =ES,Q{X1}+ K

ES,Q{τ}

= S − 2− Q2

2(Q + 1)+

K + (c + 1)e−(S−Q)(S −Q + 3)− (c + 1)e−S

1 + Q.

The derivatives of `(S,Q) can be easily found.Table 1 presents the optimal values s∗ and S∗ for different parameter

settings of c and K, found by solving the nonlinear optimization program

minS,Q

`(S, Q)

numerically. The first number in Table 1 represents the optimal value S∗

while the second number is the optimal value s∗ = S∗ −Q∗.

24

The optimal values S∗ and s∗ for differentparameter settings of c and K and lead time r = 2.K=1 K=2 K=3 K=4 K=5

c=2 2.92 / 1.21 3.30 / 0.92 3.60 / 0.73 3.87 / 0.57 4.10 / 0.42c=3 3.31 / 1.65 3.68 / 1.37 4.00 / 1.19 4.26 / 1.05 4.50 / 0.93c=4 3.60 / 1.97 3.98 / 1.70 4.30 / 1.52 4.56 / 1.39 4.80 / 1.28c=5 3.84 / 2.22 4.22 / 1.96 4.53 / 1.79 4.80 / 1.65 5.04 / 1.55c=6 4.04 / 2.43 4.42 / 2.17 4.73 / 2.00 5.00 / 1.87 5.24 / 1.77c=7 4.21 / 2.61 4.59 / 2.35 4.90 / 2.18 5.17 / 2.05 5.41 / 1.95c=8 4.35 / 2.76 4.73 / 2.51 5.04 / 2.34 5.31 / 2.21 5.56 / 2.11

Table 1

Appendix F: Quasi-regenerative property in case 2.We may represent the process Lt as the last component of a four dimen-

sional Markov process (ct, τt, Dt, Lt), where ct is the cycle time, τt (if not ∞)denotes the cycle time of the next delivery and Dt is the amount of nextdelivery.

Let (Yt) and (Rt) be i.i.d. sequences. The evolution of (ct, τt, Dt, Lt)follows the following dynamics:

ct =

{0 if ct−1 + 1 = τt−1

ct−1 + 1 if ct−1 + 1 < τt−1

τt =

∞ if Lt−1 − Yt > sct + Rt if Lt−1 − Yt ≤ s and τt−1 = ∞τt−1 if Lt−1 − Yt ≤ s and ct−1 + 1 < τt−1 < ∞Rt if Lt−1 − Yt ≤ s and ct−1 + 1 = τt−1 < ∞

Dt =

0 if Lt−1 − Yt > sDt−1 if Lt−1 − Yt ≤ s and ct−1 + 1 < τt−1

S − Lt−1 + Yt if Lt−1 − Yt ≤ s and ct−1 + 1 = τt−1

Lt =

{Lt−1 − Yt if ct−1 + 1 < τt−1

Lt−1 − Yt + Dt−1 if ct−1 + 1 = τt−1

Let Ti be the sequence of stopping times Ti = inf{n > Ti−1 : τt =ct}. With this set of stopping times, The process Lt is quasi-regenerativein the sense Definition 2, since LTi

is distributed as S − YR and the othercomponents, namely cTi

, τTiand DTi

depend on LTibut not on the past.

25

References

[1] Asmussen, S. Applied Probability and Queues. J.Wiley & Sons: NewYork, 1987.

[2] Bashyam, Sr.; Fu, M. Application of perturbation analysis to a class ofperiodic review (s,S) inventory systems. Naval Research Logistics Quar-terly 1994, 41, 47–88.

[3] Ermoliev, Y.M.; Gaivoronski, A.A. Stochastic programming techniquesfor optimization of discrete event systems. Annals of Operations Re-search 1992, 39, 1–41.

[4] Fu M. C. Optimization via Simulation - a Review. Annals of OperationsResearch 1994, 53, 199–247.

[5] Fu M., C. Sample Path Derivatives for (s,S) Inventory Systems. Opera-tions Research 1994, 42, 351–364.

[6] Fu M.C.; Hu J.Q. (s,S) Inventory Systems with Random Lead Times:Harris Recurrence and its Implications in Sensitivity Analysis. Proba-bility in the Engineering and Informational Sciences 1994, 8, 355–376.

[7] Fu M.C.; Hu J.Q. Conditional Monte Carlo: Gradient Estimationand OPtimization Applications. Kluwer Academic Publishers: Norwell,Mass., 1997.

[8] Glasserman, P. Gradient Estimation via Perturbation Analysis. KluwerAcademic Publishers: Norwell, Mass., 1991.

[9] Glasserman, P.; Liu, T.W. Corrected diffusion approximation for mul-tistage production-inventory systems. Mathematics of Operations Re-search 1997, 12 (1), 187–201.

[10] L’Ecuyer P. A unified view of the IPA, SF and LR Gradient EstimationTechniques. Management Science 1990, 36, 1364–1383.

[11] Karlin S. The Application of Renewal Theory to the Study of Inven-tory Policies. Studies in the Mathematical Theory of Inventory and Pro-duction; Arrow K.J., Karlin S., Scarf H., Eds.; Stanford UniversityPress,1958; 270 – 297.

[12] Pflug, G. Ch. Optimization of Stochastic Systems: The Interface Be-tween Simulation and Optimization. Kluwer Academic Publishers: Nor-well, Mass, 1996.

26

[13] Rubinstein, R.Y. Monte Carlo Optimization: Simulation and Sensitivityof Queueing Networks. J. Wiley & Sons: New York, 1986.

[14] Rubinstein, R.Y. Sensitivity analysis of discrete event systems by the“push-out” method. Annals of Operations Research 1992, 39, 229–251.

[15] Sahin, I. Regenerative Inventory Systems: Operating Characteristics andOptimization. Springer-Verlag: Berlin, 1986.

[16] Shedler,G.S. Regeneration and Networks of Queues. Springer Verlag:New York, 1987.

[17] Tijms, H.C. Stochastic Models: An Algorithmic Approach. John Wiley& Sons: New York, 1994.

[18] Zheng, Y.S. On properties of stochastic inventory systems. ManagementScience 1992, 38, 87–103.

[19] Zheng, Y.S.; Federgruen A. Finding optimal (s,S) policies is about assimple as evaluating a single policy. Operations Research 1991, 39, 654–665.

[20] Zipkin, P. Stochastic Leadtimes in Continuous-Time Inverntory Models.Naval Research Logistics Quarterly 1986, 33, 763 – 774.

27