Scheduling in a multi-class series of queues with deterministic service times

Queueing Systems 24 (1996) 83-99 83

Scheduling in a multi-class series of queues with deterministic service times

R e m a H a r i h a r a n a, M a g d i S. M o u s t a f a b and Shaler S t idham, Jr. c,,

a A T&TLaboratories, Room 1K-220, 101 Crawfords Corner Road, P. O. Box3030, Holmdel, NJ07733-3030, USA

E-mail: [email protected] b Mathematics Unit, Department of Science, The American University of Cairo,

P. O. Box2511, 113, Sharia Kasr El A ini, Cairo, Egypt E-mail: [email protected]

e Department of Operations Research, CB 3180, Smith Building, University of North Carolina, Chapel Hill, NC27599-3180, USA

E-mail: [email protected]

We consider a problem of scheduling in a multi-class network of single-server queues in series, in which service times at the nodes are constant and equal. Such a model has potential application to automated manufacturing systems or packet-switched communication networks, where a message is divided into packets (or ceils) of fixed lengths. The network is a series-type assembly or transfer line, with the exception that there is an additional class of jobs that requires processing only at the first node (class 0). There is a holding cost per unit time that is proportional to the total number of customers in the system. The objective is to minimize the (expected) total discounted holding cost over a finite or an infmite horizon. We show that an optimal policy gives priority to class-0 jobs at node 1 when at least one of a set ofm - 1 inequalities on partial sums of the components of the state vector is satisfied. We solve the problem by two methods. The first involves formu- lating the problem as a (discrete-time) Markov decision process and using induction on the horizon length. The second is a sample-path approach using an interchange argument to establish optimality.

Keywords: multi-class queue, scheduling, series of queues, network of queues, deterministic service times

1. I n t r o d u c t i o n

In this p a p e r we cons ider a p r o b l e m o f op t ima l schedul ing in an open n e t w o r k

o f queues in series wi th mul t ip le c u s t o m e r classes. C u s t o m e r s are divided in to

* The research of this author was supported by the National Science Foundation under Grant No. DDM-8719825. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

J.C. Baltzer AG, Science Publishers

84 R. Hariharan et al. / Scheduling in a series o f queues

classes according to their routes. The scheduling problem considered is which class of customers to process next at the first node of the network. The network that we consider in this paper consists of m nodes and m + 1 classes of customers. Class-0 customers require service at node 1 only. Class-k customers (k = 1, . . . , m) require service at nodes k, k + 1, ..., m, in that order. We assume that the service times are constant and are equal to one time unit at every node for all classes. We also assume that customers of each class arrive according to a discrete-time process with stationary independent increments which is independent of the number of customers in the network. There is a holding cost per unit time that is proportional to the total number of customers in the system. The objective is to minimize the (expected) total discounted holding cost over a finite or an infinite horizon.

The model with constant service times has potential applications to B-ISDN communication networks with A TM switches, in which messages are divided at the source node into packets (cells) of fixed length (number of bytes). Since the processing times at the nodes and links in a telecommunication network are directly proportional to the lengths of the packets, all packets require the same service time at each queue (assuming the communication channels all operate at the same trans- mission speed and with constant overhead). Other possible applications include automated cellular manufacturing systems, in which highly reliable machines per- form tasks of(nearly) constant duration.

The special case of a two-node series network (without the restriction to constant service times) has been considered by several authors, including Harrison [6], Harrison and Wein [7], Chen, Yang, and Yao [2], and Moustafa [11]. In this case there are two classes of customers, one requiring service only at node 1 and the other at both nodes 1 and 2. Using methods proposed by Harrison [6], Harrison and Wein [7] study the heavy-traffic behavior of the system, which is approximated by a corresponding Brownian network. They characterize optimal scheduling policies for the Brownian network and then use this information to suggest "good" policies for the original network. One example of a "good" policy is one that gives preference to the customers that require processing only at node 1 (class-0 customers), unless the number of customers at node 2 drops below a certain threshold, in which case preference is given at node 1 to customers requiring processing at both nodes (class-1 customers). An intuitive rationale for such a policy is that class-0 customers have the shortest total processing time. Hence scheduling such a customer will reduce the overall holding cost at the fastest possible rate. The only incen- tive for deviating from this rule is to avoid possible starvation of the server at the second node; thus, a class-1 customer is served in order to replenish the queue at node 2 whenever it drops below the threshold. Other papers on network scheduling that deal with Brownian approximations of a multiclass queueing network include Wein [15,16], Harrison and Wein [8], Laws and Louth [9], Wein [17], and Chevalier and Wein [3].

Chen et al. [2] analyze the two-node network in which arrival and service processes are modelled as counting processes with controllable stochastic intensities.

R. Hariharan et al. / Scheduling in a series o f queues 8 5

They show that a stationary policy (Markovian) policy is optimal and then use uni- formization and dynamic programming (backwards induction) to show that the optimal scheduling decisions are characterized by a switching curve. Moustafa [11] uses a Markov-decision-process model to analyze the two-node network with Poisson arrivals and with both exponential and constant service times, demonstrat- ing, that an optimal scheduling policy has certain properties of monotonicity. In the case of constant and equal service times for both classes at both nodes, he shows that an optimal policy always gives preference to a class-0 customer at node 1 if the number of customers at node 2 is at least two (cf. the threshold heuristic proposed in [7]). When the number of customers at node 2 equals zero or one, the optimal scheduling rule depends on the relative numbers of class-0 and class-1 customers at node 1. Unlike the previous references, Moustafa's model allows external arrivals at node 2. Hariharan [4] (see also Stidham and Weber [14]) extends Moustafa's results for the constant-service-time case to a network ofm nodes in series, showing that an optimal policy gives preference to a class-0 customer at node 1 when at least one of m - 1 partial-sum inequalities on the number of customers at nodes 2 through m is satisfied. This reduces the problem to one in which only a finite set of values for the number of customers at nodes 2 through m need to be considered.

In the present paper we strengthen the results in [4] for the case m > 2, by present- ing a sample-path argument which does not require the Markovian assumptions of [4]. (Hariharan [4] contains a sample-path proof for m = 2). The basic model is introduced in detail in section 2. For completeness and purposes of comparison, we present (in section 3) a proof for the Markovian case based on dynamic programming (essentially the proof from [4], which has not appeared previously in the pub- lished literature), and then present the sample-path proof, which is based on a pairwise-interchange argument, in section 4. (This proof has previously appeared in an unpublished technical memorandum [5]. For a comprehensive survey of sample-path arguments in control of queues, see Liu et al. [10].)

2. The model

Consider a network of m nodes in series. Each node j has a single server (j = 1 ,2 , . . . , m). The classes of jobs are defined according to their routes. Class-0 jobs require service at node 1 only. Class-k jobs (k = 1 ,2 , . . . , m) require service at nodes k, k + 1 , . . . , m (in that order). The service times at each node are constant and equal. Without loss of generality we adopt this service time as our time unit, called a slot. Thus, in each time slot t (t = 0, 1, . . .) , exactly one customer is served at each nodej = 1 ,2 , . . . , m (provided that the class to which the server is assigned has at least one customer in its queue at the beginning of the time slot). We assume that the random vectors of the number of arrivals in each class in successive time slots are i.i.d, and distributed as A = (A(0),A(1), . . . ,A(m)), where A(i) denotes the number ofclass-i arrivals. Customers of all classes incur holding cost at rate 1 while

86 R. Hariharan et al. / Scheduling in a series of queues

in the system. Future costs are discounted; the one-period discount factor is/3. The objective is to minimize the expected total discounted holding cost over a finite or infinite horizon.

Since all the jobs at node j , j = 2, 3, . . . , m, have the same remaining route and holding-cost rates, they can be regarded as equivalent to class-j jobs. Hence, for each node j >/2, it is sufficient to keep track of the total number of jobs at that node and the only scheduling decision to be made is at node 1, where the system control- ler has to choose between class-0 and class-1 jobs.

Let x = (x0, x l , . . . , Xm) denote the state of the system, where x0 is the number of class-0 jobs at node 1, Xl is the number of class-1 jobs at node 1, and Xk, k = 2, 3, . . . , m, is the number of jobs at node k (which we shall call level-k jobs). To describe the effects of the scheduling decision, it will be convenient to introduce the following notation. Let

do(x) := -e01(x0 > 0),

di(x) := (-ei+ei+l)l(xi > 0 ) , i = 1 , 2 , . . . , m - 1,

din(x) := - e m l ( x m > 0), (1)

where ei is the unit m-vector with a one in the ith component and zeroes elsewhere. Thus, di(x) measures the effect on the state vector x of the service of a level-/job, i = 0 ,1 , . . . ,m. For 2<<.j<<.k<~m, define d(/,k)(x):= ~ik=jdi(x). Now define the state-transformation operators Tix as follows:

Tox := x + do(x) + d(E'm)(x) , (2)

T1X :-~- X -'[- dl(x) -[- d(2'm)(x) . (3)

Thus, if the server is assigned to class i (i = 0, 1) in state x, then the state of the system at the beginning of next service slot, without the addition of new arrivals, is Tix.

3. M a r k o v decis ion model

Let V(x) be the minimal expected total/3-discounted cost (/3 < 1) over an infinite horizon starting from state x. Then V(x) satisfies the optimality equation

m

V(x) = ~ xi + min{ U(Tox), U(TlX) } , (4) i=0

where U(x) =/3E[V(x + A)]. We now characterize certain sufficient conditions to be met by the state vector

x under which it is optimal to serve customers of class 0 at node 1. To establish these properties for the infinite-horizon problem, we use successive approximations

R. Hariharan et al. / Scheduling in a series o f queues 87

(value iteration) and in the process show that they are also valid for the finite-horizon case.

Let V~ (x) be the minimal expected total/3-discounted cost over n periods (time slots) starting in state x, n/> 1 (V0 = 0). We can express the value functions Vn recur- sively as follows:

m

Vn(x) = Z x i + min{Un(Tox), Un(T1x)}, n>~l , (5) i=0

where Un(x) = fiE[ Vn-1 (x + A)]. It follows from the theory of Markov decision processes (see, eg., Schfil [13], Puterman [12], Bertsekas [1]) that Vn(x) ~ V(x) and Un(x) ~ U(x) asn --* c~.

We now state the main theorem of this section. It shows that it is optimal to serve class-0 customers at node 1 in all but a finite set of values of (x2, x3 , . . . , Xm), gener- ated by the intersection ofm - 1 partial-sum inequalities.

T H E O R E M 1

An optimal policy is to serve class-O customers at node 1 when the state x is in the set X0 := {x = (xo, X l , . . . , Xm):xo~l a n d ~-'~=2 Xi >>~ l for some l E {2, 3 , . . . , m} }.

The proof of Theorem 1 depends on the following two lemmas.

L E M M A 1

Given x such that x0 >~ 1 and Xl/> 1, if

V n ( y - eo) - Vn~V - e l -F e2) ~ 0 , for all y>>.x,

then

U n + l ( X - Co) - Un+l (X - el + e2) ~<0.

(6)

(7)

L E M M A 2 p For j>~i>~2, let X = {x: y~k=lXj_k>/p for all p E {1, 2, . . . , j -- i-- 1} and

)-]~=0 xj+k/> l + 1 for some l E {0, 1 ,2 , . . . , m --j}}. If II0 satisfies

Vn(xWei) - Vn (x+e j )~O, f o r a l l j > i~>2 a n d x E X , (8)

for n = 0, then Vn and U, also satisfy (8) for n/> 1.

Lemma 1 follows directly from the definition of Un. We shall defer the proof of Lemma 2 until after the proof of Theorem 1.

Proof of Theorem 1 To prove that the policy stated in Theorem 1 is optimal, it suffices by the optimal-

ity equation to show that U(Tox) <<. U(TlX) for all x 6 X0. To show this, it suffices to show


Un(Tox)<<.Un(TlX), x E X O , n ~ l , (9)

since Un "+ U pointwise as n ~ c~. We will prove this by induct ion on n. Since a non-idl ing policy is opt imal (see Mous ta fa [11]), the policy stated in Theo rem 1 is opt imal when Xl = 0. Therefore, it suffices to consider the case when xl >i 1.

Fo r each x, we have Ul ( Tox) ~ UI ( Tl x) trivially, since Vo( Tox) = Vo( Tx x) = O. Let x E X0 with xl f> 1. I t follows f rom (2) and (3) that

Tox = y - eo ,

Tl x --- y - el + e2 ,

where y := x + d (z,m) (x). Note that Y0 = x0/> 1 and Yl = Xl ~> 1. N o w assume tha t (9) holds for some n ~> 1. We shall show that this implies tha t

A : = V n ( z - e 0 ) - V n ( Z - e l + e 2 ) < < . 0 , f o ra l l z>~y=x+d(a 'm) (x ) . (10)

Then, using L e m m a 1, it will follow that

Un+ x ( Tox ) = Un+ l (y - eo ) <~ Un+ l (y - e l + e 2 ) --- Un+ l ( Zl X ) (11)

and the inductive step will be complete. To prove (10), fix z>~y. Let ~ = (z - el + e2)i and Y'i = (Y - el + e2)i. Observe

tha t y is the state of the system that results after including the effect of the service at nodes 2, 3 , . . . , m, on state x (without including the effect of the service at node 1 or adding any new arrivals to the system). Thus, the total number of jobs at nodes 2, 3 , . . . , l, in state y will be one less than the number of jobs at nodes 2, 3 , . . . , l in the cor responding state x, ifxl t> 1 (since there is a job that leaves this set of nodes f rom node l and there is no job entering this set of nodes, as the effect of the service at node 1 is not taken into account) and will be the same as the number of jobs at nodes 2 , . . . , l, in state x ifxz - 0 (since there is no job either entering or leaving this set of nodes). It follows that

1 l

i=2 i - 2

Therefore, since by serving a class-1 cus tomer at node 1 we add a job to the node set {2, 3 , . . . , l},

l 1 l l

Z y ' i = E y i + 1>~ ~ _ ~ x i - 1 + 1 = E x i > ~ l . /--2 i=2 i=2 i=2

Now, since ~li= 2 ~. >~ ~_,ti= 2 ~, action 0 is opt imal in state z - el + e2 at stage n, by the induct ion hypothesis. Re turn ing to (10), we use action 1 in state z - e0 to get an upper bound on A. For za ~> 1, clearly

R. Hariharan et al. / Scheduling in a series of queues 89

A <~ -- 1 + U,,(z eo - el -b e2 q- d(2'm)(z)) - Un(z - eo - el q- e2 -b d(2'm)(z))

= -1~<0 .

Fo r z2 = 0,

A<. - 1 + U,(z - e0 - el + e2 + d(3'm)(z))

- Un(z - eo - el + e2 - e2 n u e3 --b d(3'm)(z))

= - 1 + Un(~ + e2) - U,(~ + e3),

where ~ := z - e0 - el + d (3,m) (z). No te that l i> 3, since z2 = 0. Moreover , since ~--~l-3 ~r l l l z2 . . . . 0, we have y : 0 and hence Z-,k=0 3+k )--~i=3 z'i >/~--~i=3 zi 1 >i ~~i=3Yi -- 1

l >. l 1) 1 ~>l 2. Thus ~ satisfies the assumpt ions for = ~--~i=2 Y i - 1 ~- ( Z / = 2 x i - - -

L e m m a 2 with i = 2 and j = 3 , so that U , ( ~ , + e 2 ) - U n ( ~ + e 3 ) ~ < 0 . Thus, A~< - 1 ~<0.

Thus we have established that (10) holds for z2~>0. Hence, it follows f rom L e m m a 1 that (11) holds for n + 1, thus complet ing the inductive step. [ ]

P r o o f o f L e m m a 2 Proceeding by induction, observe that U1 satisfies (8) by the defini t ion of U1

and the fact tha t V0 - 0. The inductive step has two parts.

(i) I f Un satisfies (8), then Vn satisfies (8).

(ii) I f Vn satisfies (8), then Un+l satisfies (8).

But (ii) follows directly f rom the definit ion of Un. Thus it suffices to prove (i). At this point , we m a k e the following observations. Let A := Vn (x + el) - Vn (x + ej). I f it is op t imal to serve class-a customers (a = 0, 1) in state x + ej, then using the same act ion in state x + ei gives an upper bound on Vn (x + ei) and hence on A. We define d as follows:

d := ~ -e0 if the opt imal decision in state x + ej is 0,

L - e l + e2 if the opt imal decision in state x + ej is 1.

Thus,

A<~ U,,(x + ei + d + d(2'm)(x + el)) - U~(x + ej + d + d(2'm) (x + ej))

= Un(x + ei + d + d(2'i-1)(x) - (ei - el+l) + d(i+lj-1)(x)

- (ej -ej+l 1(/+ 1 ~<m))l(xj > 0) + d(i+l'm)(x)) -- U n ( x q- ej q- d q- d(2 ' i -1) (x) - (ei - e i+l) l (x i > O) + d ( i + l j - 1 ) ( x )


- (ej -iej+l l ( j -I- 1 ~<m)) + d(J+l'm)(x))

= Un(y + ei - (ei - ei+l)l(xi : 0)) - Un(y q- ej - (ej - e j+l) l (x j : 0)) ,

wherey := x + d + d(2'm)(x). N o w assume tha t Un satisfies (8). L e t j > i >/2 and x E X. It suffices to show tha t

A ~< 0. We divide the p roo f into the following four cases.

Case I. X i = 0 and xj = 0 Note that , since xj = 0, it mus t be the case that j < m, since i f j = m, then the

inequali ty )-'~=0 xj+k i> l + 1 cannot be satisfied. Thus,

A = gn(X q- el) - - g n ( X q- ej) <~ Un(y q- e i+ l ) -- Un(y q- e]+l) �9

N o t e tha t for a n y p E { 1 , 2 , . . . , j - i - 1}, p p - 1

~ Y j + l - k = ~ Xy-k + l(xy_p > O) - l ( x j > O) k=l k=O

p-1

= ~ Xj -k -~- l (X j -p > 0 ) , k=l

p -1 since xj = 0. I f Xj_p = 0, then ~ k = l xj-k = )-'~=1 xj-k >>-p, and hence ~ = 1 yj+l-k >~p. Ifxj_p > 0, then

P

Z Yj+ l - k ~-~ k=l

�9 1 X Also, if Y~'~k=0 j+k

l -1

-~Yj+l+k : k=O

by the induct ion A~<O.

p - 1

Z xj-k + l > ~ p - 1 + 1 = p . k=l

i> 1 + 1, then

1

~-~Xy+k + l(xy~> 1 ) - l(x./+l >i 1)~>(l + 1 ) - 1 = l , k=l

hypothesis , since xj = O. Thus, by the induct ion hypothesis ,

Case 2. xi > 0 and xj = 0 Again, note that since xj = 0, j < m. Thus,

A = Vn(x + ei) - Vn(x + ej) <<. U~(y + ei) - - Un(y Jr" e j + x ) .

Fol lowing the same arguments as in Case 1, we conclude that )-~=1Yj+l-k >>-P, for 1-1 all p E { 1 , 2 , . . . , j - i}, and )-'~g=0 yj+l+k >t l. Thus, by the induct ion hypothesis ,

A~<0.

Case 3. xi = 0 and xj > 0 In this case, A~< Un(y + eg+l) - Un(y + ej). I f i + 1 = j , A = 0; otherwise, note

tha t for a n y p E { 1 , 2 , . . . , j - i - 2},

R. ttariharan et al. / Scheduling in a series of queues 91

P P

E yj-k ~ E Xj-k + l(Xj-p-I > O) -- l(xj-1 > O) k = l k = l

P

= E Xj-k "1- l(Xj-p-1 ~" O) -- 1, k = l

1 since xj-1 = ~k=l xj-k ) 1, by the induction hypothesis. Since ~ = 1 xj_~ >~p, for p E { 1 , 2 , . . . , j - i - 1},ifx]_p_l = 0then

P P

) ~yj-k = ) ' ~ ~j-k + xj_~_l - 1 k = l k = l

p + l

= ~ j _ , , - l~>(p+ 1 ) - I = p . k = l

Ifxj-p_l > 0, then P P

Eyy_k~- Exj- l~+ l - l>~p. k = l k = l

�9 l >_ Also, l fY~= 0 xj+k,- l + 1, then ! !

EYJ +k ---- E xj+k + l(xj_, >~ 1) - l(xj+l ~> 1) k=0 k=O

/> ( l+ 1) + 1 - l ( x j + l 1>1)t>1+ 1,

by the induction hypothesis. Thus, again by the induction hypothesis, A ~ 0.

Case 4. xi > 0 and xj > 0 In this case, A<~Un(y+ei)-Un(y+ej). Note that for any p E { 1 , 2 , . . . ,

j - i - 1}, P P

Z Y J -k = E Xj-k + l(Xj-p-1 > O) -- l ( x j - 1 > O) k = l k = l

P /> '~" xj-k + l(xj_p-1 > O) - 1.

k ~ l

If xj-p-i > 0, then ~ = t Y]-k ~> ~ = 1 X j - k >~p, by the induction hypothesis. If xj-p-i = O, then

p p p + l

Zyj-k>~ EXy-k-- I = E X j - k - - 1 . k = l k = l k = l


For p E {1,2 , . . . , j - i - 2}, )-'~+11Xj-k ~>p + 1 by the induction hypothesis, and therefore ~ = 1 yj-k ~>p. Ifp = j - i - 1, then

j - i j- i-1

~--~Xj-k ~- Xi'~- ~ xj-k>/l + ( j - - i - - 1 ) = j - - i = P + l . k=l k=l

Also, if ~,tk=oXj+k>~l + 1 then, following the same argument as in Case 3, Y~=0 Yj+k 1> l + 1. Thus, again by the induction hypothesis, A ~< 0.

This completes the proof of the lemma. []

R E M A R K S

In Theorem 1, note that we have actually shown that V,,(z-e0) - Vn(z - el q- e2) ~< - 1, thus showing that every optimal policy must serve class-0 jobs in states x such that ~ = 2 xk >t/. Note that such a dominance condition is not necessary to establish that serving class-0 jobs in such states is an optimal policy.

Lemma 2 gives us conditions on the state-vector x under which having an extra customer at node i instead of at node j ( / > i), the number of customers at all other nodes being the same, does not result in an additional cost. This might seem counter intuitive at first. However, the condition x ~ X implies that the extra job at node j in state x + ej is blocked by the jobs at nodesj , j + 1 , . . . , m, and i + 1 , . . . , j - 1 such that the following statements hold:

(a) When this e~tra job reaches the terminal node (node m), there is at least one other job at the terminal node (refer to the condition that ~ = 1 xj+k ~> l for some l ~ {0, 1 , . . . , m - j } ) , so that the extra job continues to stay at node m.

(b) The nodes in between i and j are sufficiently loaded such that, at least until the extra job at node i (state x + e~) reaches the end node, there is a job (other than the extra job from nodej , starting in state x + ej) at node m (refer to the condition that ~-~ffc=l Xj-k>~P for all p E {1,2 , . . . , j - i - 1}). This ensures that the extra job (from state x + ej) is not served at node m until the extra job at node i (from state x + ei) reaches node m, so that the extra jobs in the two states leave the system at the same time.

Note that the inequality (8) may not be valid without this condition. For example, consider a 5-node problem. Let i = 2 and let j -- 5. Now consider the states x + ei ----- (2, 1, 1,0, 0, 0) and x + ej = (2, 1,0, 0, 0, 1). Suppose there are no arrivals. Then the sample path followed by the process, until the system empties, starting in state x + e~, if we choose to serve class 0 in the first two stages and class 1 in the third (there would be no jobs to serve at node 1 after the third stage) is

(2,1,1,0,0,0) ~ (1, 1,0, 1,0,0) ~ (0, 1,0,0, 1,0)

( 0 , 0 , 1 , 0 , 0 , 1 ) ~ ( 0 , 0 , 0 , 1 , 0 , 0 ) ~ ( 0 , 0 , 0 , 0 , 1 , 0 ) ~

R. Hariharan et al. / Scheduling in a series o f queues 93

(0,0,0,0,0, 1) --~ (0,0,0,0,

resulting in a total holding cost the sample path followed is

(2,1,0,0,0,1) 4 ( 1 , 1 , 0 , 0 ,

0,0),

of 14. On the other hand, starting in state x + ej,

0,0) (0, 1,o, 0,0, 0)

(0,0, 1,0,0,0) ~ (0,0,0, 1,0,0) ~ (0,0,0,0, 1,0)

(0,0,0,0,0, 1) ~ (0,0,0,0,0,0)

resulting in a holding cost of 11. Now, suppose the system is sufficiently loaded. For example, let

x + ej = (2, 1, 1, l, 0, 2) and x + ei = (2, l, 0, 1, 1,2). That is, x satisfies the conditions mentioned in Lemma 2. Following the same sequence of decisions as in the previous example, the resulting sample path until the system empties, starting in state x + ei is

(2,1,1,1,0,2) ~ (1, 1,0,1, 1, 1) --. (0, 1,0,0, 1, 1)

(0,0, 1,0,0, 1) --~ (0,0,0, 1,0,0) -~ (0,0,0,0, 1,0) --~

(o,o,o,o,o,1) (o,0,0,o,o,o) with a total holding cost of 20. The corresponding path starting in state x + ej

would be

(2,1,0,1,1,2) ~ (1,1,0,0,1,2) ~ (0,1,0,0,0,2) ---,

(0,0, 1,0,0, 1) ~ (0,0,0, 1,0,0) ~ (0,0,0,0, 1,0) --+

(0,0,0,0,0, 1) --~ (0,0,0,0,0,0)

with a total holding cost of 20. These observations suggest the possibility of a sample-path proof of Theorem 1,

which is the subject of the next section.

4. Sample-path proof

In this section we present a sample-path proof for the result given in the previous section. The assumptions of stationary and independent increments for the arrival process are no longer necessary. The arrival process for each class may be complete- ly general as long as the number of jobs arriving in any time unit does not depend on the state of the system or the control policy in effect. In this context, a p o l i c y is defined as a sequence of non-anticipative decision rules for choosing the action at

94 R. Hariharan et al. / Scheduling in a series o f queues

each time point t = 0, 1 ,2, . . . , where each decision rule may be history and/or time dependent.

Let x(t) := {x0(t), Xl(t), x2(t) , . . . , Xm(t )} denote the state of the system at time t = 0, 1 , . . . , where xo(t) denotes the number of class-0 jobs in the system, xl(t) denotes the number of class-1 jobs at node 1, and xk(t) denotes the total number of jobs at node k, k = 2, 3 , . . . , m.

Once again the holding cost per unit time is proportional to the total number of jobs in the system, and without loss of generality we assume the constant of propor- tionality equals 1. The total discounted cost over a horizon of length T is

T m

t=0 i=0

We shall prove that, for any realization of the arrival process, the total discounted cost over a finite or infinite horizon is minimized by the policy described in Theorem 1.

T H E O R E M 2

If a given non-idling policy schedules a class-1 job when x E )to, then there is another policy which schedules a class-0 job in the current period and has a smaller holding cost over every horizon. This result holds for every realization of the arrival process.

Proof The proof uses the following lemma. We state the lemma here and defer the proof

until after the theorem.

L E M M A 3

For any fixed policy and any realization of the arrival process, the total discounted cost over a finite or infinite horizon is equal for the starting states x + ei andx + ej, (j~>i~>2) if

n

~ '~Xj -k )n for all n E {1,2, . . . , j - i - 1} k=l

and

l

xj+t >i l + 1 for some l e {0, 1 ,2 , . . . , m - j} k=0

as long as the same class of job is served at node 1 in the two states.

Consider an arbitrary realization of the arrival process. Without loss of generality, we denote the current time period as t = 0. Consider a sample path $1 obtained

R. Harihar an et al. / Scheduling in a series of queues 95

by serving a class-1 job at t = 0 and following an arbitrary policy thereafter. Let 7-/~ (T) be the beta-discounted holding cost up to time T following this sample-path. Let y( t ) = { y o ( t ) , y l ( t ) , y 2 ( t ) , . . . , y m ( t ) } be the state of the system at time t. Therefore,

T m

t=0 j=0

We shah now construct a sample path 82 by interchanging the decisions at t = 0 and t =/C in the sample-path $1, where/C is the first time the given policy serves a class-0 customer if/C < 00. If/C = c~, then the decisions along the two sample paths differ only at time 0. Let z(t) = {zo(t), Zl (t), z 2 ( t ) , . . . , Zm(t)} denote the state of the system along this alternate sample path at time t. Let 7-/~(T) be the/3-discounted holding cost up to time T following sample path 82. Therefore,

T m

~-,[2fl( Z) = E /~t E zj( t) . t=O j=0

Obviously, since we start in the same state, zj(0) -- yj(O) for j = 0, 1 , . . . , m. Also, since the given policy serves a class-1 job in the current period whereas the alternate policy serves a class-0job, we have

z0(1) =y0(1) - 1,

Zl(1) =y l (1 ) + 1,

z2(1) = y / ( 1 ) - 1,

zi(1) = y i (1) , i/>3,

v " t - 2 z ~0 ( l(zt > 0)/>l 1). andY'~-2zi+2(1)>>.l - 1 (since y~'~-0Zzi+2(1)/> z.~i=0 i + 2 1 , } - -

We now make the following observation. Suppose,

zo(t) = yo(t) - 1,

Zl(t) : yl(t) + 1,

zj(t) = yy( t ) , 2<<.j<~j* - 1,

z i . ( t ) = yj .( t) - 1

zy(t) ---- yj(t) j ~.j + 1,

where j* E {2, 3, m}, .~l-j* zj.+i(t)/> l - j * + 1 and i �9 . . , z_~i=0 , Y'~k=lyj*-k(t)>>-t for all i E {1,2, . . . ,j* - 2}. If a class-1 job is served in both states z(t) and y( t ) , then we consider the following two cases.

96 R. Hariharan et al, / Scheduling in a series of queues

zj,(t)>~l In this case, the transfer of jobs due to service completion is the same in states

z(t) and y(t) for all the nodes. Hence at time t + 1, the difference in the number of jobs in states z(t) and y(t) remains the same as at time t at each node. (Since the arrival processes are identical, the jobs added at each node due to external arrivals are identical in both states.) Thus, we conclude that

zo(t + 1) = y o ( t + 1) - 1,

z l ( t + 1) = y l ( t + 1) + 1,

z j ( t + l ) = y j ( t + l ) , 2<. j<~j*- l ,

zj .( t+ 1) =yj . ( t+ 1 ) - 1,

zj(t + l) = yj(t + l) , j>~j* + l ,

i - 1)~>l Y~k=lYj*-k( t+ 1)/>iforal l i E (1 ,2 , . . . ,j* 2} andX7 't-j*z..~i=0 zJ*+i(t + - j * + 1.

zj,(t) = 0 Note that in this case, there is no job transferred from node j* to j* + 1 in state

z(t) whereas a job is transferred in state y(t). This causes the difference between the number of jobs at node j* + 1 in states y(t) and z(t) to increase by one. Also since E/k=0 gj*_i(;) ~ k for every k e ( 1 ,2 , . . . , }, z:*_l (t) = Yj*-I (t) t> 1. Hence there is one job transferred from node j* - 1 to node j* in both states z(t) and y(t). However, there is no job leaving node j* in state z(t). Thus the difference in the number of jobs between states z(t) and y(t) at node j* decreases, so that

zo(t+ 1) = y 0 ( t + 1) - 1,

zl(t + 1) = yl(t + 1) + 1,

zj( t+ l ) = y j ( t + l) , 2<~j<~j*,

z j .+1( t+l )=yj*+l( t+l ) - 1,

zj(t + l) = yj(t + l) , j~ j* + l ,

i "" ~ i = 0 Zj*+l+i(t) >~ l for Y]~k=lYj*+l-k(t+ 1)~>i for all i e (1,2,. ,j* - 1} and l _ j ,

somel E {j*,j* + 1 , . . . ,m} .

Thus, from the above observation, we conclude that, for t ~</E,

zo(t) = yo(t) - 1,


Zl(t) = yl( t) + 1,

zi(t) =y i ( t ) , 2 ~ i < ~ j * - 1,

zj.(t) = yj.(t) - 1,

zi( t)---yi( t) , i>~j*+l ,

i ~

Y'~k=lyj.-k(t)>>.lforalliE{1,2, �9 �9 �9 ,J*-2}and~--'t-J~zj*+i(t)>~l-j*+lforsomez_.,i=o j * E { 2 , 3 , . . . , m } . (Note that j* is no t the same for every t. However , j'* E {2, 3 , . . . , m} for any t ~/~). Moreover , for t = / C + 1,

zo(t) = y0(t),

zl(t) = y,( t )

z2(t) = y2(t) + 1,

z i ( t ) = y i ( t ) , 3<~i<~j*- l ,

z j , ( t ) = y j , ( t ) - l ,

zi(t) =y i ( t ) , i>/j* + 1,

i t-j" , _ j , Y~.k=lYj*-k( t )>~i foral l iE{1,2 , . . . , j*- -2}andy~i=oZj+i( t )>~l + 1 for some

j* E {2, 3 , . . . , m } . Thus, clearly for T ~</C + 1

Fo r T > / C + 1, using L e m m a 1 on states z(/C + 1) andy(/C + 1), we can claim tha t the total holding costs in periods subsequent to /C + 1 are identical a long the two sample paths. Hence,

for all T > 0. [ ]

Proof o f Lemma 3 The p r o o f uses induct ion on the hor izon length T. It is clear that the l emma is

t rue for T = 1, since the total number of jobs is equal in the two states. F r o m the above discussion, we note that the states at t = 1 can be represented

as y + ee and y + ej,, (i ~< i' ~< i + 1 a n d j ~<j' ~<j + 1), where


n

y~yj,_k>/n for all n E {1,2, . . . , f - i t - 1} k=l

and

l ~yj,+k>~l+ 1 for some l E { 0 , 1 , . . . , m - j ' } . k=l

We shall henceforth refer to these conditions as Conditions A and B, respectively. Now assume that the costs along the two sample paths are equal up to time t and the states along the two sample paths at time t are given as x(t) + ei and x(t) + ej for some 2 ~< i ~<j ~< m, where x(t) satisfies conditions A and B.

Our induction argument is complete if we show that the state of the system along the two sample paths can be represented by x(t + 1) + ee and x(t + 1) + ej, respectively for some i t a n d f (2 ~< i t ~<f ~<m), where x(t + 1) satisfies conditions A and B. However, this follows from essentially the same argument as used in the proof of Lemma 2 in the previous section. []

It is important to note that, in contrast to the results derived from the Markov decision model, these results are free of any assumption regarding the arrival process as long the arrival process is independent of the state of the system and the control mechanism in effect.

References

[1] D. Bertsekas, Dynamic Programming and Optimal Control, Vol. II (Athena Scientific, Belmont, Massachusetts, 1995).

[2] H. Chen, P. Yang and D. Yao, Control and scheduling in a two-station queueing network: optimal policies and heuristics, Queueing Systems 18 (1994) 301-332.

[3] P.B. Chevalier and L.M. Wein, Scheduling networks of queues: heavy-traffic analysis of a multistation closed network, Oper. Res. 41 (1993)743-758.

[4] R. Hariharan, Routing and scheduling in queueing networks, Ph.D. Dissertation, Department of Operations Research, University of North Carolina, Chapel Hill (1991).

[5] R. Hariharan, Scheduling in a multi-class queueing network: A sample-path approach, AT&T Bell Laboratories Technical Memorandum (1992).

[6] J.M. Harrison, Brownian models of queueing networks with heterogeneous customer populations, in: Stochastic Differential Systems, Stochastic Control Theory and Applications, eds. W. Fleming and P.L. Lions (Springer, New York, 1988) pp. 147-186.

[7] J.M. Harrison and L.M. Wein, Scheduling networks of queues: heavy-traffic analysis of a simple open network, Queueing Systems 5 (1989) 265-280.

[8] J.M. Harrison and L.M. Wein, Scheduling networks of queues: heavy-traffic analysis of a two- station closed network, Oper. Res. 38 (1990) 1052-1064.

[9] C.N. Laws and G.M. Louth, Dynamic scheduling of a four-station queueing network, Prob. Eng. Inform. Sci. 4 (1990) 131-156.


[10] Z. Liu, P. Nain and D. Towsley, Sample-path methods in the control of queues, Queueing Systems 21 (1995) 293-336.

[11] M.S. Moustafa, Scheduling in a multi-class network, Ph.D. dissertation, Program in Operations Research, North Carolina State University, Raleigh (1987).

[12] M. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, New York, 1994).

[13] M. Sch~il, Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal, Z. Wahrscheinlichkeitstheorie verw. Gerb. 32 (1975) 179-196.

[14] S. Stidham and R. Weber, A survey of Markov decision models for control of networks of queues, QueueingSystems 13 (1993)291-314.

[15] L.M. Wein, Optimal control of a two-station Brownian network, Math. Oper. Res. 15 (1990) 215-242.

[16] L.M. Wein, Scheduling networks of queues: Heavy-traffic analysis of a two-station network with controllable inputs, Oper. Res. 38 (1990) 1065-1078.

[17] L.M. Wein, Scheduling networks of queues: Heavy traffic analysis of a multistation network with controllable inputs, Oper. Res. 40 (1992)$312-$334.

Documents

Scheduling in a multi-class series of queues with deterministic service times