1106 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 50, … Collection/TAC/2005... · 2007-05-04 · 1106 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 50, NO. 8, AUGUST 2005 Workload

1106 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 50, NO. 8, AUGUST 2005

Workload Models for Stochastic Networks: ValueFunctions and Performance Evaluation

Sean P. Meyn, Fellow, IEEE

Abstract—This paper concerns control and performance evalu-ation for stochastic network models. Structural properties of valuefunctions are developed for controlled Brownian motion (CBM)and deterministic (fluid) workload-models, leading to the followingconclusions: Outside of a null-set of network parameters, the fol-lowing hold.

i) The fluid value-function is a smooth function of the ini-tial state. Under further minor conditions, the fluid value-function satisfies the derivative boundary conditions thatare required to ensure it is in the domain of the extendedgenerator for the CBM model. Exponential ergodicity ofthe CBM model is demonstrated as one consequence.

ii) The fluid value-function provides a shadow function foruse in simulation variance reduction for the stochasticmodel. The resulting simulator satisfies an exact large de-viation principle, while a standard simulation algorithmdoes not satisfy any such bound.

iii) The fluid value-function provides upper and lowerbounds on performance for the CBM model. This fol-lows from an extension of recent linear programmingapproaches to performance evaluation.

Index Terms—Computer network performance, networks,optimal control, routing, scheduling, simulation, singular optimalcontrol, stochastic systems.

I. INTRODUCTION

SEVERAL new approaches to performance evaluation andoptimization for stochastic network models have been pro-

posed over the past decade. Although the motivation and goalsare diverse, a common thread in each of these techniques isthe application of an approximate value function, or Lyapunovfunction.

i) In [1]–[5], linear programs are constructed to find aLyapunov function that will provide upper and lowerbounds on steady-state performance. These Lyapunovfunctions are quadratic, or piecewise-quadratic, sincein each of these papers the cost criterion is a linearfunction of the queue-length process.

ii) In [6] and [7], a quadratic or piecewise-quadraticLyapunov function is proposed as an initialization invalue-iteration or policy-iteration for policy synthesis.An example of a Lyapunov function is the associatedfluid value-function [see (42)]. The numerical results

Manuscript received february 9, 2004; revised April 23, 2005. Recommendedby Associate Editor Y. Wardi. This paper is based upon work supported bythe National Science Foundation under Award ECS 0217836 and Award DMI0085165.

The author is with the Coordinated Science Laboratory and the Departmentof Electrical and Computer Engineering, the University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA (e-mail: [email protected]).

Digital Object Identifier 10.1109/TAC.2005.852564

reported in [7] show significant improvements whenthe algorithm is initialized with a quadratic Lyapunovfunction. When initialized with the fluid value-func-tion, convergence is nearly instantaneous. Relatedideas are explored in [8].

iii) The variance-reduction techniques developed in[9]–[11] are similarly based on the application ofa Lyapunov function. In [10], it is found that thesimulator based on the fluid value-function results indramatic variance reductions for the models consid-ered. Results from one such experiment are shown inFig. 5.

Much of this research has been numerical and algorithmic innature. In particular, there has been little theory to explain thedramatic numerical results reported in [7] and [10]. The purposeof this paper is to provide a theoretical foundation for analysisof these algorithms. These results are based primarily on newbounds on the dynamic programming equations arising in con-trol formulations for fluid and stochastic network models.

The main results of this paper concern workload modelsfor the network of interest. This viewpoint is motivated byrecent results establishing strong solidarity between workloadmodels and more complex queueing model in buffer-coor-dinates [12]–[19]. In particular, under appropriate statisticalassumptions and appropriate assumptions on the operatingpolicy, a Gaussian or deterministic workload model is obtainedas a limit upon scaling the equations for a standard stochasticqueueing model (see [20]–[22] and [18]).

The starting point of this paper is similar to the approach usedin [23] and [24] to establish positive recurrence for a reflecteddiffusion. The authors first construct a Lyapunov function for adeterministic fluid model, and then show that the same functionor a smoothed version serves as a Lyapunov function for thediffusion (i.e., condition (V2) of [25] and [26] is satisfied.)

This paper considers both controlled fluid models and con-trolled Brownian motion (CBM) models for the workloadprocess. The main conclusions are as follows.

i) Structural properties of the fluid value-function aredeveloped: under mild conditions, the value-function is

, and its directional derivatives along the boundariesof the “constraint region” are zero, when the directionis taken as the particular reflection vector at this pointon the boundary (see Theorem 4.2.)

ii) These structural properties imply that the fluid value-function is in the domain of the extended generatorof the CBM workload model. As one corollary, it isshown in Theorem 4.7 that the stochastic workload

0018-9286/$20.00 © 2005 IEEE

MEYN: WORKLOAD MODELS FOR STOCHASTIC NETWORKS 1107

process satisfies a strong form of geometric ergodicityfor the policies considered.

iii) It is shown in Theorem 5.3 that the smoothed estimatorof [10] based on the function satisfies an exact largedeviation principle. It is argued that the standard sim-ulation algorithm does not satisfy any such bound.

iv) The linear-programming approaches of [1]–[5] are ex-tended based on the structural results obtained in The-orem 4.2.

To provide motivation, we begin in Section II with a descrip-tion of various models for a simple two-station network. A shortsurvey of control and performance evaluation techniques is pre-sented, focusing on this single example.

General workload models are described in Section III, alongwith basic properties of the associated value functions. Thesestructural results are refined and applied in Section IV to providestability results for the model, Section V contains applicationsto simulation, and examples illustrating linear-programming ap-proaches are described in Section VI. Conclusions and topics offuture research are included in Section VII.

II. ILLUSTRATIVE EXAMPLE

The Kumar–Seidman–Rybko–Stolyar (KSRS) networkshown in Fig. 1 will be used to develop terminology and illus-trate the issues to be considered in the remainder of this paper.Further details regarding fluid models, random walk models,Brownian models, and the relationships between them may befound in [12], [27], [14], [16], [10], [28], [17], [18], [29], and[8].

A. Models for the Queue-Length Process

The network shown in Fig. 1 has two stations, and fourbuffers. A convenient model is described in discrete-timeas follows. We let denote the queue-length process, and

denote the allocation process, both evolving on .1 Theallocation process is subject to linear constraints

U (1)

where denotes a vector of zeros, and inequalities be-tween vectors are interpreted component-wise. The 2 4 con-stituency matrix is given by

Let denote the service and arrival processes at the var-ious buffers. For each , and each time , thedistribution of is assumed to be Bernoulli, and the distri-bution of is supported on . We assume moreover thatE . The topology shown in Fig. 1 implies thatis of the specific form

1Throughout the paper we use boldface to denote a vector-valued function oftime, either deterministic or stochastic.

Fig. 1. KSRS network.

The common mean of the arrival and service variables areexpressed in the usual notation, E E

.The routing matrix for this model is defined by if

customers departing buffer immediately enter buffer . Hence,in this example, and otherwise.

For a given initial queue-length vector , the queuelength at future times is defined recursively via

(2)

where . This is a versionof the controlled random walk (CRW) model considered in [10]and [17].

The fluid model associated with (2) is the linear system

where E E isthe initial condition, and is the cumulative allocation processevolving on . The fluid allocation process is subject to thelinear constraints

U (3)

In the fluid model the lattice constraints on the queue-lengthprocess and allocation process are relaxed. Each is deterministicand Lipschitz continuous as a function of time.

These processes are differentiable a.e., since they are Lip-schitz continuous. Letting denote the rightderivative of the allocation process, the fluid model can beexpressed as the state–space model

(4)

The allocation-rate vector is constrained as in (1), withU for all .

B. Workload Models

Throughout much of this paper, we restrict attention to re-laxations of the fluid model based on workload, and analogousworkload models to approximate a CRW stochastic networkmodel.

The two workload vectors associated with the two stations inthe KSRS model are defined as follows, where for

:


The two load parameters are given by ,or , and we let

. The workload vectors are used to construct theminimal draining time for the network, thereby solving thetime-optimal control problem. Assuming that the load conditionholds, , it is known thatfor each initial condition , and we have the followingexplicit representation:

(5)

A cumulative allocation process is called time-optimal if theresulting state trajectory satisfies in minimumtime . One solution to the time-optimal control problemwhen is given by , where

. The optimal state trajectorytravels toward the origin in a straight line. This policy and re-lated policies for a stochastic model are examined in [30].

The central motivation for consideration of the workload vec-tors is that they provide the following definition of the work-load process. We let denote the 2 4 matrix whose rows areequal to the workload vectors . Equivalently,

. Given any state tra-jectory , the workload process is defined by for

.We have the following simiple representation of in terms

of the two-dimensional drift vector defined as ,and the two-dimensional cumulative idleness process, definedby . For a given initial condition

, and any

The idleness process is nonnegative and has nondecreasingcomponents since satisfies (3).

A relaxation of the fluid model is obtained on relaxing con-straints on the idleness process. Consider the two-dimensionalmodel satisfying and

(6)

The idleness process is assumed nonnegative, with nonde-creasing components, but its increments may be unbounded. Wesay that is admissible if it is a measurable function of time sat-isfying (6) with nondecreasing, , and restricted to

.Relaxations of the stochastic CRW model are developed in

[17] and [19], following the one-dimensional relaxation intro-duced in [31]. Here, we consider the CBM model based onthrough the introduction of an additive disturbance: The two-di-mensional process with initial condition satisfiesthe equations

(7)

where is again assumed nonnegative, with nondecreasingcomponents, and is restricted to . The process is atwo-dimensional Brownian motion with covariance .

C. Control

In constructing policies to determine the idleness processesfor either the fluid or CBM workload model, we restrict to thefollowing affine policies introduced in [17]. For a given convex,polyhedral region , the corresponding affine policy isdefined for the fluid model as follows: For each initial condition

we take , and for

minimal element satisfying

The trajectory is continuous and piecewise linear on ,and exhibits a jump at time if . The work-con-serving policy is simply the affine policy with . Themain result of [17] shows that an optimal policy for the CBMmodel is approximated by an affine policy when the state islarge.

Consider, for example, the convex polyhedron

(8)

where , and are given. Suppose that, and consider the initial condition for

some . Then, , and subse-quently, the trajectory is piecewise-linear on .

The definition of the affine policy is identical for the CBMmodel, except of course the trajectories are not piece-wise linear. A general construction is provided in Section III-B.

Let denote the -norm on . A typical objec-tive function in the queueing theory literature is the steady–statecost for the CRW model, defined as

E (9)

This is independent of for “reasonable” policies, includingthe average-cost optimal policy [6], [32]. The optimal policyis stationary and Markov [the action is a function of thecurrent state ], so that is a time-homogeneous Markovchain [6].

Given this cost function on buffer-levels, the effective costfor the two-dimensional relaxation is defined as the value of thelinear program

(10)

That is, is the cost associated with the “cheapest” statesatisfying the given workload values. Further motivation is

provided in [13], [17], [16], [10], and [28]. The effective cost ispiecewise linear: Letting denote the extreme points in thedual of the linear program (10), we havefor .


Fig. 2. Level sets of the effective cost for the KSRS model in Cases I and II, respectively, under the work-conserving policy.

Fig. 3. Level sets of the value function J for the KSRS model.

For a given affine policy the associated value function is de-fined by

It follows from the definition that the dynamic programmingequation holds

(11)

Suppose that is differentiable at some within the interior of. Then on setting , dividing each side of (11) by ,

and letting we obtain the differential dynamic program-ming equation

(12)

The fluid value function is constructed in Proposition 2.1 basedon (12) in the following two special cases.

Case I and

Case II and

In each case, the effective cost is continuous,piecewise linear, and strictly linear on each of the three regions

shown in Fig. 3. Level sets of and are

shown in Figs. 2 and 3, respectively. The level sets shown inFig. 3 are smooth since is .

Proposition 2.1: Suppose that the two-dimensional work-load relaxation on is controlled using the work-conservingpolicy. Then, in each of Cases I and II, the value function

is and piecewise-quadratic, and purely quadraticwithin each of the regions . Hence,

for , where in Case I thematrices are given by

(13)

Proof: The property is established for general work-load models in Theorem 4.2. Computation of is performed asfollows: The value function satisfies the dynamic programmingequation (12), or equivalently

It follows that for each . A single constraint on each ofis obtained on considering on the two boundaries

of . Finally, the property implies thatand . Combining these linear constraints yields(13).

In Case I, the effective cost is monotone, so that foreach . In this case, the work-conserving policy is pathwise op-timal for the fluid model, in the sense that is minimalfor each . It follows that the value function is convex in thiscase, which is seen in the plot at left in Fig. 3.

In Case II, the monotone region for the effective cost is givenby W , as shown at right in Fig. 2. In this case,


Fig. 4. Simulation of a stochastic workload model using an affine policy. In each case, the constraint regionR was defined by (8) with � = 0:9. The constant �was defined from left to right by � = 0; 25, and1. The sample paths of (AAA;SSS) and the initial condition W (0) = (150;0) were identical in each of the threeexperiments.

a pathwise optimal solution exists, but it is defined by the affinepolicy (8) using and , so that W . Inparticular, for an initial condition of the form ,the optimal trajectory is given by , and

for .Under the optimal policy in Case II, the value function is

purely quadratic on W

(14)

We now turn to the CBM model (7). In Case I, we find thatthe work-conserving policy in which is again pathwiseoptimal. This policy defines a reflected Brownian motion onwith reflection normal at each boundary.

In Case II, a pathwise optimal solution does not exist. Theoptimal solution evolves in a region that is a strict subsetof . In the discounted-cost case, it is shown in [17] that theconstraint region is approximately affine, in the sense that forlarge values of the region is approximated by an affinedomain of the form (8) with . Theorems 4.6 and 4.7and numerical results contained in [17] strongly suggest that thisresult carries over to the average-cost optimal control problem.

A formula for the parameter in this approximation is alsoobtained in [17]. Again in Case II, suppose that for some ,and we have and

. Then, [17, eq. (43)] gives

(15)

with , and

These are the diffusion parameters for the height process definedby .

Although (15) is in general an approximation, it defines theaverage cost optimal policy in some specific cases Consider theone-dimensional model of [33], and the multidimensional ex-amples treated in [34] where an affine policy with parametersdefined using a version of (15) is precisely average-cost optimal.

Fig. 4 illustrates the role of the affine shift parameter in asimulation using a CRW approximation of the CBM model.The constraint region used in the figure at left is similar to thatdefining the optimal policy for the fluid model in an exampleconsidered in [28].

A policy constructed for the workload model must finallybe translated to form a useful policy for the physical network.Given an affine policy of the form described here using theconstraint region defined in (8), and a vector ofsafety-stock values, a policy for the four-buffer queueing net-work is described as follows:

Serve at Station I if and only if or

and (16)

An analogous condition holds at Station II.The safety-stock values are necessary to prevent idleness at

each station as evolves in , so that the work-load process mimics the CBM model with constraint region .

D. Performance Evaluation

Given a Markov policy for the network such as (16), the con-trolled process is a time-homogeneous Markov chain. Welet denote its unique steady-state distribution, when it exists.In this case we have for a.e.,

. The steady-state distribution is not easily computed,so we seek bounds or approximations for the steady-state cost

.The most common approach to performance evaluation of a

scheduling policy is through simulation. The standard estimatorof is defined by the sample-path average

(17)

This is consistent in the sense that a.s., for a.e.,initial condition . However, it is well known that longsimulation runlengths are required for an accurate estimate ofwhen the system load is near unity (see [35], [36], and Section Vfor general results pertaining to the CBM model.)

A version of the smoothed estimator of the form consideredin [10], [11] is described as follows. For any function ondefine EE . The steady-state mean of

is zero, provided is absolutely integrable w.r.t. . Thesmoothed estimator is then defined as the sequence of asymp-totically unbiased estimates given by

(18)


Fig. 5. Estimates of the steady-state customer population in the KSRS model as a function of 100 different safety-stock levels using the policy (16). Two simulationexperiments are shown, where in each case the simulation runlength consisted of N = 200 000 steps. The left-hand side shows the results obtained using thesmoothed estimator (18); the right-hand side shows results with the standard estimator (17).

The function is an example of a shadow function, since itis meant to eclipse the function to be simulated [10]. This is aspecial case of the control variate method [37], [38]. However,the homogeneous structure of the workload model is exploitedto obtain a highly specialized estimator based on the fluid value-function and the dynamic programming (12) for .

Consider the four dimensional queueing model (2) controlledusing the policy (16) under the assumptions of Case I, and let

denote the fluid value function for usingthe same constraint region . We choose as the

, piecewise-quadratic function given by for.

Bounds on the summand in (18) are obtained as follows. Bythe mean value theorem applied to we have for some

, with

This identity combined with Lipschitz continuity of the functionon , and the second-moment assumption on , implies

the following bound: Writing E ,we have for some and all

The policy (16) is designed to enforce when-ever lies in , and when this holds the dynamic pro-gramming (12) implies that .The policy is also designed to maintain . Conse-quently , and this is precisely the motivationfor the smoothed estimator.

Shown in Fig. 5 are estimates of the steady-state customerpopulation in Case I for the family of policies (16), indexed bythe safety-stock level , with . Shown at left areestimates obtained using the smoothed estimator (18) based onthis function , and the plot at right shows estimates obtainedusing the standard estimator (17). This and other numerical re-sults presented in [10] show that significant improvements in es-

timation performance are possible with a carefully constructedsimulator.

We now develop theory for general workload models to ex-plain these numerical findings.

III. CONTROLLED BROWNIAN MOTION MODEL

In Section III-A, a general fluid model is constructed in whichthe state process is defined as a differential inclusion on .This is the basis of the general controlled Brownian motionworkload model defined in Section III-B, and developed in theremainder of this paper.

A. Workload for a Fluid Model

The state process for the fluid model has continuous samplepaths, and evolves on . For simplicity, we do not treat bufferconstraints here [17], [28]. Rate constraints on the sample pathsare specified as follows. It is assumed that a bounded, convex,polyhedral velocity set V is given, and that satisfies

V (19)

The right derivative exists for a.e., sinceis Lipschitz continuous. This velocity vector is interpreted as

a control, subject to the constraint V for .It is assumed that the fluid model is stabilizable. That is, for

each initial condition , one can find evolving in Vand such that for . We infer that Vunder the stabilizability assumption.

Since V is a convex polyhedron containing the origin, it canbe expressed as the intersection of half-spaces as follows. Forsome integer , vectors , andconstants

V (20)

These vectors generalize the workload vectors constructed inthe KSRS model.

To obtain a relaxation, we fix an integer , and relax thelinear constraints in the definition (20) for . It is assumedthat the vectors are linearly independent.

Workload Relaxation for the Fluid Model: The differentialinclusion satisfying the following state-space and rate con-straints:


i) for each ;ii) V for each , where

V

For a given initial condition , we say that is admis-sible if it is a measurable function of time, and satisfies theseconstraints with . An admissible solution need not becontinuous since V is in general unbounded.

Let denote the matrix whose rows are equal to .The workload process and idleness process are defined, respec-tively, by

The following properties follow from the definitions (see [16, p.188]), and demonstrate that is a generalization of the relax-ation (6) for the KSRS model. The bounds in (21) are decoupledsince the workload vectors are assumed lin-early independent.

Proposition 3.1: The following hold for each initial condi-tion , and each admissible trajectory starting from .

i) The workload process is subject to the decoupledrate constraints

(21)

ii) The idleness process is nondecreasing: Setting, we have

(22)

iii) The workload process is constrained to the workloadspace

W (23)

On constructing control solutions for we restrict to the work-load process, where control amounts to determining the idlenessprocess . For this purpose, suppose that a cost function is given,

. We assume that is piecewise linear, and defines anorm on . The associated effective cost W is definedfor W as the solution to the convex program

(24)

Just as was seen in the KSRS model examined in Section II, theeffective cost is piecewise linear, of the form

W (25)

where . The region on which ismonotone is denoted

W W W

(26)

We let W denote a continuous function such thatan optimizer of (24) for each W. Then, given a desir-

able solution on W, an admissible solution is specified by

We consider exclusively affine policies to determine the idle-ness process . For the general workload model, we again fix apolyhedral region W containing the origin with nonemptyinterior. Under these assumptions, there exist nonnegative con-stants and vectors such that

(27)

The following additional assumption is assumed so that we canconstruct a minimal process on in analogy with the construc-tion in Section II: For each there exists a “minimalelement” satisfying

i) ii) If and

then

The existence of the pointwise projection requires some as-sumptions on the set for dimensions 2 or higher [16].

The minimal process starting from an initial condition Wis then defined by

(28)

It is represented as the solution to the ODE

(29)

where the idleness rate is expressed as the state-feedback law, with

(30)

The mapping defines the affine policy with respect to the re-gion .

We conclude with some properties of the projection, and re-sulting properties of the feedback law . The th faceis defined by

(31)

We assume without loss of generality that each of these sets isof dimension .

Proposition 3.2: Suppose that the set is given in (27), andthat the pointwise projection exists. Then, thefollowing hold.

i) For each , the projection is the uniqueoptimizer of the linear program


ii) For each face with , there is aunique satisfying

and

iii) The feedback law W defined in (31) sat-isfies the following properties: for

. Otherwise, if for somewe have , and for , then

for , where is defined in ii).

Proof: Let be an optimizer of the linear pro-gram in i). We have since and, hence, also

. It follows that is the unique optimizerof this linear program.

We prove part ii) by contradiction: Fix , andsuppose that in fact and for some

.Consider , with for . For ,

we consider the open ball centered at given by, where denotes the Euclidean

norm. Then, from the definition (31), there exists suchthat whenever we have for .

Fix with , and define

We have by construction, and the vectorssatisfy

. Hence, by reducing if necessary, we may assumethat each of these vectors lies in . We conclude that

, which is only possible if . Thisviolates our assumptions, and completes the proof of ii).

Part iii) is immediate from ii) and definition (30).

B. Brownian Workload Model

The stochastic workload model considered in the remainderof this paper is defined as follows.

Controlled Brownian-Motion Workload Model: The contin-uous-time workload process obeys the dynamics

W (32)

The state process , the idleness process , and the disturbanceprocess are subject to the following.

i) is constrained to the workload space W [see (23)].ii) The stochastic process is a drift-less, -dimensional

Brownian motion with covariance .iii) The idleness process is adapted to the Brownian mo-

tion , with , and for.

A stochastic process satisfying these constraints is calledadmissible.

The definition of an affine policy for the CBM model requiressome effort. We begin with the following refinement of the de-terministic fluid model defined in Section III-A through the in-troduction of an additive disturbance: Fix a continuous function

satisfying , and consider the controlledmodel

(33)

A deterministic process on W is again called admissible if(33) holds for some nondecreasing idleness process , with

.We say that is the minimal solution on the region , with

initial condition W, if: i) is admissible; and ii)for all , and any other admissible solution with

initial condition . A minimal solution is a particular instanceof a solution to the Skorokhod problem (e.g., [39]–[41].) Thereflection direction on the th face is given by , the th stan-dard basis vector in , where is defined in Proposition 3.2.Under general conditions it is known that the Skorokhod map isa Lipschitz-continuous functional of , and the initial condition

(see [42]).The following result is a minor extension of ([16, Th. 3.10]).

The -norm in Proposition 3.3 ii) is given by

where denotes the Euclidean norm.Proposition 3.3: Suppose that the set is given in (27), and

that the pointwise projection exists. Then, thefollowing hold.

i) The minimal process exists for each continuous dis-turbance , and each initial condition W.

ii) There exists a fixed constant satis-fying the following: For any two initial conditions

W, and any two continuous disturbanceprocesses , the resulting workload processes

obtained with the respective ini-tial conditions and disturbances satisfy the followingbounds for :

iii) For each W , and .

for

Proposition 3.3 leads to the following definition of the affinepolicy for (32) with respect to a domain :

the minimal process on with (34)

The resulting idleness process is random since is assumedto be a Gaussian stochastic process. The controlled processis a time-homogeneous, strong Markov process. The strong


Markov property follows from the sample-path construction of[40].

It will be convenient below to introduce the extended gen-erator for the Markov process under an affine policy. Astochastic process adapted to some filtration iscalled a martingale if E for each

. It is called a local-martingale if there exists an increasingsequence of stopping times such thatis a martingale, for each , and a.s. as .Throughout this paper, we assume that is the fil-tration generated by .

A measurable function is in the domain of ifthere is a measurable function such that, for eachinitial condition , the stochastic process defined in(35) for is a local-martingale

(35)

In this case, we write .The following version of Itô’s formula (37) is used to identify

a large class of functions within the domain of . This result willbe applied repeatedly in the treatment of value functions thatfollows. Given a function W , the differential generatoris expressed

(36)

where is the weightedLaplacian.

Theorem 3.4: Suppose that the set is given in (27), and thatthe pointwise projection exists. Suppose that

is continuously differentiable, is uniformlyLipschitz continuous on compact sets, and that exists fora.e., . Then, the following hold.

i) For each initial condition

(37)

ii) Suppose that , and in addition thefollowing bounary conditions hold:

(38)

where is defined in (30). Then, is in the domain ofand .

iii) If the conditions of ii) hold, and in addition

E

then is a martingale for each initial condition.

Proof: When is , then i) is given in [43, Th. 2.9, p.287]. In the more general setting given in the theorem, the func-tion may be approximated uniformly on compacta by func-tions so that is simultaneously approximated in forany compact set ; see [44, Th. 1, p. 122], and the exten-sions in [45]–[47].

Suppose now that the assumptions of ii) hold. The assumptioninterior is imposed to ensure that does not vanish on

. Itô’s formula then gives the following representation:

(39)

The local martingale property is immediate since is driftlessBrownian motion.

We have for each , and each initial

E E

The right-hand side is finite under the conditions of iii), and themartingale property then follows from the representation (39)[43].

IV. VALUE FUNCTIONS

We now develop structural properties of the value functionsassociated with the fluid and CBM models. At the same time, weestablish a strong form of geometric ergodicity for the process

under an affine policy.It will be useful to introduce a scaling parameter to

investigate the impact of variability

W (40)

in which the idleness process is determined by the affine policy(34). We restrict to affine domains of the form

(41)

where is the constant used in (40), is a constantvector, and . This is motivated by affine approxi-mations of optimal policies of the form obtained in [17] andreviewed in Section II-C.

We consider two processes on the domain : The minimalprocess satisfying (40); and also the deterministic minimalsolution on defined in (28). When we wish to emphasizethe dependency on and the initial condition W we denotethe workload processes by , , respectively.

Let denote the steady-state mean of . Whenwe drop the subscript so that . The value functions

considered in this section are defined for the fluid and CBMworkload models, respectively, by

(42)

E (43)


where W and . We again suppress dependency onwhen .

Consider for the moment the special case . Assumingthat the integral in (43) exists and is finite for each , we obtainfrom the Markov property the following representation for each

:

E

It then follows that the stochastic process defined in (44) isa martingale for each initial condition W

(44)

Hence, is in the domain of the extended generator, and solvesthe dynamic programming equation

(45)

This is also called Poisson’s equation, and the function isknown as the relative value function [48]. General conditionsunder which both the steady-state mean and the integral in(43) are well defined are presented in Theorem 4.5.

We begin in Section IV-A with the simpler fluid model. Theproof of Proposition 2.1 can be generalized to conclude that thevalue function is continuous and piecewise-quadratic in thegeneral fluid model considered here (see e.g., [49, Sec. 4.2]).Our goal is to show that is a smooth solution to the dynamicprogramming equation (12), which is equivalently expressed

(46)

where the differential generator for the fluid model is defined inanalogy with (36) via

W (47)

To show that is smooth we differentiate the workloadprocess with respect to the initial condition

. Let denote the maximal open set such thatand are each for . Generally, since and theprojection are each piecewise linear on , it follows that theset can be expressed as the union

(48)

where each of the sets is an open polyhe-dron. Note that and the sets will in general depend upon

.For example, consideration of the KSRS model with, we see from Fig. 2 that and are both linear

on each of the sets shown in Fig. 3. The openpolyhedron is also contained insince and on . This reasoning leads toa representation of the form (48) when .

We list here some key assumptions imposed in the results thatfollow.

A1) For each , the set has nonempty interior,satisfies , and the pointwise projection

exists.A2) for each .

A3) .

A. The Fluid Value Function

In our consideration of the fluid value function, we requiresome structural properties for . The following scaling propertyfollows from the definitions.

Lemma 4.1: for each, and .

Theorem 4.2 establishes several useful properties of the fluidvalue function. To illustrate the conclusions and assumptions weagain turn to the KSRS model. Consider first the case in which

, the cost function is , andthe constraint region W . Assumption A2) doesnot hold: The integral in A2) is nonzero for any nonzero initialcondition on the diagonal in . The value function is givenby , which is not on .This explains why A2) is needed in Theorem 4.2 i).

To see why A3) is required in Theorem 4.2 iii), we take; and .

Assumptions A1) and A2) hold, and the value function maybe explicitly computed

with

Although smooth, we have andalong the lower boundary of . This is possible since the modelviolates A3).

We note that the value function for the discrete-time con-strained linear quadratic regulator problem is also smooth andpiecewise quadratic under general conditions [50], [51].

Theorem 4.2: Under A1), we have for eachW , and the following scaling property holds:

(49)

If, in addition, A2) holds, then for each , the followinghold.

i) The function is piecewise-quadratic,, and its gradient is globally Lipschitz contin-

uous on .ii) The dynamic programming equation (46) holds on

.ii) The boundary conditions (38) hold for , where

is defined in (30) with respect to the region . If inaddition A3) holds, then the function does not vanishon .

Proof: The scaling property (49) follows from Lemma4.1.

The dynamic programming equation ii) is simply the funda-mental theorem of calculus; see (11) and surrounding discus-sion.


To establish the smoothness property in (i) we define for

(50)

whenever the gradient with respect to exists. Letdenote the derivative of with respect to . The

matrix-valued function is constant on each of the connectedcomponents of . The gradient (50) exists whenever

for a given , and by the chain rule

Under A2), we have for each , and a.e.,.

We conclude that, for each W, the gradient (50) existsfor a.e., . Lipschitz continuity of inthen leads to the representation

(51)

where the minimal draining time for the fluid model is the piece-wise linear function

W

The range of integration in (51) is finite since for.

The proof of i) is completed on establishing Lipschitz con-tinuity of the right-hand side of (51). For a given initial condi-tion , let denote the sequenceof regions in [defined in (48)] so thatenters regions sequentially. We set , let

denote the exit time from , and set for. From the forgoing, we see that the gra-

dient can be expressed as the finite sum

Each of the functions is piecewise linear and contin-uous on under A1) and A2), and this implies that isLipschitz continuous.

To see iii), fix , and . Then,interior for each .

Consequently, applying ii), we have for

This combined with continuity of and establishes theboundary property . It is obvious from (30)that on the boundary under A3).

B. Regeneration and Ergodicity

A stationary version of the workload process can be con-structed using the shift-coupling technique [52]. Suppose thatthe Brownian motion is defined on the two-sided interval ,with , and construct a process on , initialized at

time . For a given initial condition , this is defined onthe interval with initial condition , andnoise process . Suppose thatall of the processes are initialized at . Then, for any fixed, it can be shown that is nondecreasing

in for . Consequently, the following limit exists withprobability one:

(52)

It follows from the main result of [24] that the controlled processis ergodic, so that is a.s. finite.

The following generalization of Lemma 4.1 allows us to re-strict to . Its proof is immediate from the scaling formula

.

Lemma 4.3: for each, and .

As one consequence of Lemma 4.3 we see that the steady-state cost scales linearly, for . Lemma 4.3 com-bined with Proposition 3.3 also implies a weak form of stability.

Proposition 4.4: If A1) holds, then the minimal process onwith satisfies, for each W

E (53)

Proof: From Proposition 3.3 we obtain the bound, for eachW

(54)

If , then for each . Combiningthe bound (54) with Lemma 4.3, we conclude that, for any

E E

An application of [53, Cor. 37.12] shows that E is fi-nite for each . Conclusion (53) is obtained on set-ting .

The limit (53) is precisely the form of fluid-scale stabilityassumed in [27] to formulate criteria for moments in a stochasticnetwork. For a given function , and for a pair ofprobability distributions on we define

Define for W, and letdenote the Markov transition group for with .

The main result of [23] and [24] establishes positive recur-rence of the reflected diffusion . The following result extendsthese results by establishing polynomial moments, and a poly-nomial rate of convergence of the underlying distributions.

Theorem 4.5: Suppose that (A1) holds. Then, the minimalprocess on with satisfies the following.

i) The limiting process exists, and its marginal dis-tribution on is the unique steady-state distributionfor .


ii) The steady-state distribution has finite moments, andfor each

W.iii) There is a compact set W s.t., for each integer

, there is a finite constant satisfying

E W (55)

where denotes the first entrance time to .Proof: Analogs of i) and ii) are obtained in [27, Th. 6.3].

Although this paper concerns networks with renewal input pro-cesses, the proof in the present setting is identical. Part iii) is[27, Prop. 5.3].

Theorem 4.5 can be strengthened under A1)–A3) since wemay then construct a Lyapunov function based on the fluid valuefunction.

Proposition 4.6: Suppose that A1)–A3) hold. Then, withequal to the minimal process on with .

i) The fluid value function is in the domain of the ex-tended generator, and , where

is defined a.e., on by

(56)

ii) The function is in the domain of theextended generator, and for some ,and a compact set

iii) The function is in the domain of the ex-tended generator for each . There existssuch that the following bound holds for each

: For finite constants

Proof: Theorem 4.2 iii) implies that satisfies theboundary conditions required in Theorem 3.4 ii) and, hence,

is in the domain of . Each of the functions considered inii) and iii) also satisfies the conditions of Theorem 3.4 ii) sincethese properties are inherited from .

The bounds on in ii) and iii) follow from the identity, and straightforward calculus.

We say that is -uniformly ergodic if for some, and all W

See [26], [3], [54], and [55] for background.Theorem 4.7: Suppose that A1)–A3) hold with . Then,

the steady-state mean satisfies , andthe minimal process is -uniformly ergodic for each

.Proof: -uniform ergodicity follows from [25, Th. 6.1]

and Proposition 4.6 iii).

The identity is obtained as follows. We havefrom Theorem 3.4 the representation

Setting and taking expectations gives

E

E

C. The Relative Value Function

The main result of this section is Theorem 4.9, which pro-vides a bound on the error betweenthe relative value function and the fluid value function. We firstobtain bounds on based on the dynamic programming (46),along with an analog of Theorem 4.2.

Theorem 4.8: Under A1) we have for allW , and the following scaling property holds:

(57)

Moreover, for each , the following hold.

i) For some constant , and all

W

ii) If is another solution to Poisson’s equation (45) thatis bounded from below, then there is a constant suchthat on .

Proof: The scaling property (57) follows directly fromLemma 4.3 and (43). Without loss of generality, we restrict to

in the remainder of the proof.To prove i), note first that Theorem 4.5 iii) implies that

is ‘ -regular, with bounding function ’ [26], [56], whereas before. This means that for each there

exists such that for any set with positive -measurethere is satisfying for all

E (58)

Specializing to it then follows from [48, Th. 3.2] thata solution to Poisson’s equation exists with quadratic growthsatisfying for each

E E

Letting and applying Theorem 4.5 ii) then gives, with defined in (43), so that is also bounded

by a quadratic function of .


Let be any compact set with nonempty interior. Theoptional sampling theorem implies thatis a martingale, and it can be shown using Theorem 4.5 and(58) that it is uniformly integrable. Consequently, we obtain theexpression

E E

(59)

On setting we obtain the lower bound,, completing the proof of i).

The uniqueness result ii) is given in [6, Th. A3]. Althoughstated in discrete time, Section VI of [6] contains a roadmapexplaining how to translate to continuous time.

Note that the identity (57) implies that. Similarly, (49) implies that

, as , and it fol-lows that as since .This is a version of the error bounds explored in [6] and [32](see also [8].) The following result provides a substantialstrengthening of this asymptotic bound.

Theorem 4.9: Under A1)–A3) the value functions anddiffer by a function with linear growth

Proof: We have noted that is a martingale, and isa martingale by Theorem 3.4 iii). Consequently,

is a martingale, and can be expressed for by

(60)

Let W denote the compact set found in Theorem 4.5iii). The martingale property for implies the representation(59), and combining (55) with Itô’s formula (37) we obtain theanalogous expression for . On subtracting, we obtain foreach initial condition

E

(61)

To complete the proof, observe that the function isbounded and E has linear growth by Theorem 4.5 iii)with .

An analysis of the differential generator reveals that theinvariant distribution is of a computable, exponential formunder certain conditions on the CBM model and the polyhedralregion . Unfortunately, these conditions are fragile (see [57],[58], and the references therein.) Computational methods areinvestigated in [59].

Given the complexity of computation of the invariant distri-bution, we may turn to simulation in order to evaluate a givenpolicy.

V. SIMULATION

In this section, we describe how Theorem 4.2 provides theo-retical justification for the construction of a smoothed estimatorof the form described in Section II-D.

It is unlikely that one would be interested in simulating aCBM model. However, we expect that the bounds obtained herewill have analogs in a discrete queueing model.

The following definitions are all generalizations of those usedin our treatment of the KSRS model. The standard estimator isagain defined as the sample-path average

(62)

The time-average variance constant (TAVC) is given by

E (63)

The limit in (63) exists and is finite under the assumptions ofTheorem 4.7, and has the alternative representation in terms ofthe mean quadratic-variation of the martingale

E (64)

This follows from [26, Th. 17.5.3], which establishes a ver-sion of (63) for any discrete-time geometrically-ergodic Markovchain. The proof in continuous time is identical [54].

To investigate the impact of variability and load we considera second scaling of the CBM model

(65)

The idleness process is again determined by the affine policyusing the region defined in (41). The parameteris interpreted as network load, and again determines vari-ability of the model. The corresponding deterministic processon with drift has value function

Proposition 5.1: Suppose that A1)–A3) hold, and thatfor all . Then, the TAVC for the process defined in (65) is

nonzero, and satisfies

Proof: This follows from the following scaling arguments.Consider first a simple spatial scaling: Let denote the processwith arbitrary. For each , the process


, is a version of (65) with ,and . Consequently

Consider next a temporal-spatial scaling: If is the solutionto (65) on , then , is also a solutionto (65) with unchanged, and variability parameter .This combined with the definition (63) leads to the identity

Combining these bounds gives the desired relation.Finally, positivity of the TAVC follows from the representa-

tion (64).To investigate the potential for variance reduction we con-

sider the following generalization of the smoothed estimator of[10]:

(66)

where is defined in (56) when and . Forarbitrary , we set

(67)

which is precisely as before, . This definitionis entirely analogous to the shadow function used in (18) for theCRW model.

In the next result, we demonstrate that the order of the TAVCis not improved in the smoothed estimator, unless is con-stant on .

Proposition 5.2: Suppose that A1)–A3) hold. Then, the fol-lowing hold for the process defined in (65).

i) The smoothed estimator is consistent. That is

ii) When and , the TAVC is given by themean quadratic-variation of the martingale defined in(60): For each

E (68)

This is zero if and only if is constant on , inwhich case .

iii) If for all , then the TAVC again satisfies,

Proof: Consistency follows from Theorem 4.7 (recall that.) The representation (68) is obtained just

as (64) is obtained for the standard estimator.

The scaling result iii) follows from (67) and the same argu-ments used in the proof of Proposition 5.1.

While the conclusion of Proposition 5.2 iii) is negative, innumerical experiments such as illustrated in Fig. 5 we have seendramatic variance reductions.

An alternative way to evaluate a simulation algorithm isthrough confidence bounds. Given we are interested inbounding the error probability P . Typically,one seeks a “large deviation” bound so that this probabilitydecays to zero exponentially fast as . However, evenfor the simple M/M/1 queue it is shown in [11] that the largedeviations principle fails.

The story is very different when using the smoothed esti-mator. The proof of Theorem 5.3 follows from [55, Ths. 6.3and 6.5] combined with Theorem 4.7. The “rate function”is the convex dual of the usual log-moment generating function.Without loss of generality, we restrict to .

Theorem 5.3: Suppose that the CBM model on satisfiesA1)–A3). If is not constant on , then the smoothed es-timator (66) satisfies the following exact large deviations prin-ciple. There is , and functions

, such that for each and initialcondition

P

An analogous bound is obtained for P .

VI. LINEAR PROGRAM BOUNDS

We now show that smoothness of the fluid value-functionleads to an algorithmic approach to performance approximation,based on the linear-programming approaches of [1]–[5], and [8].

Suppose that and the region satisfy A1)–A3). Letdenote open, connected polyhedral regions sat-

isfying the following: The function given in (56) is con-stant on each is linear on , and .

We consider a family of continuous, piecewise linear func-tions . It is assumed that each of these func-tions is linear on each of the sets . Consequently, the as-sumptions of Theorem 4.7 hold: Lettingdenote the associated value functions, and setting

, we obtain the identity for each. These identities are interpreted as equality constraints in the

linear program.The variables in the linear program are defined for

, by

E

We have several constraints.

a) Mass constraints: for each , and .b) Region constraints: For example, if

within region .c) Value function constraints: For some constants

and vectors


we have the representations for any

Letting , weobtain from Theorem 4.7, for each

(69)

d) Objective function: There is such that.

We illustrate this construction using the CBM workloadmodel for the KSRS network. We assume that the covariancematrix satisfies , and that .

Consider first the affine policy (34) with W. The cost function restricted to is the

same in Cases I and II, and the common value function shownin (14) is purely quadratic on . Consequently, in this case wehave , and

(70)

Consider now the minimal process on W in Case I.The function is not constant, so it is not obvious that wecan compute exactly using these techniques when W.

To construct an LP, we restrict to the following spec-ifications: , with as shownin Fig. 3, and , with and

.We thus obtain the following constraints.

a) Mass constraints: .b) Region constraints: We have for all since

W . Moreover, on considering the structure ofthe sets we obtain, , and .In addition, there are numerous symmetry constraints.For example, and sinceand .

c) Value function constraints: The value function is apure quadratic. In fact, if is any linearfunction on W, then

W

We conclude from Theorem 4.7 that EE . The value function

is given in (13). The identity (69) then impliesthe equality constraint,

.d) Objective function: In Case I, we have

E

We conclude with results from one numerical experiment inCase I, using parameters consistent with the values used in the

simulation illustrated in Fig. 5 for the controlled random walkmodel (2). The first-order parameters were scaled as follows:

where is chosen so that , and. The effective cost was similarly scaled,

for .The random variables used in (2) were taken

mutually independent, with the variance of each random vari-able equal to its mean. The steady-state covariance is approxi-mated by

where . This is exact when E.

The linear program constructed for a CBM model is designedto approximate the CRW model: was taken equal to the co-variance matrix given previously, and .Solving the resulting linear program then gives the followingbounds on the steady-state cost, and corresponding occupationprobabilities:

E

P (71)

Although these bounds apply to the CBM model, they roughlyapproximate the estimated value of E for theCRW model obtained in the simulation illustrated in Fig. 5 [10].

For the same parameters, with the regionwe obtain from (70)

E

The steady-state mean for the process restricted to the regionis strictly greater than the upper bound (71) obtained for the

minimal process on W. This is consistent with the fact that theminimal process on W is optimal whenever is monotone.

VII. CONCLUSION

The fluid value-function has several desirable properties thatprovide new algorithms for simulation and performance boundsin stochastic networks, and improved motivation for the appli-cation of recently proposed algorithms.

There are several immediate extensions to be considered.

i) In [7], the fluid value-function is proposed as an initial-ization in value iteration to compute optimal policies.It would be worthwhile to revisit this approach for bothCBM and CRW network models.

ii) We have not yet considered examples to test the linearprogramming techniques on CRW network models.

iii) In many applications, a linear cost function is notappropriate. For example, it may be desirable toobtain bounds on the moment generating functionE for .


iv) Many of the ideas in this paper may find extension toentirely different areas. In particular, the ODE methodcommonly used in the analysis of stochastic approx-imation is completely analogous to the use of a fluidmodel in the stability analysis of a stochastic network[60]. We are currently investigating the potential forvariance reduction in stochastic approximation, and re-lated learning algorithms.

ACKNOWLEDGMENT

This project grew out of on-going discussions with S. Hen-derson on simulation. In particular, the property of the valuefunction was first conjectured by Henderson. V. Borkar also pro-vided much inspiration in discussions on large deviations andon generalizations of Itô’s formula. R. Williams provided nu-merous suggestions for clarifying the presentation. All of thissupport is gratefully acknowledged.

REFERENCES

[1] S. Kumar and P. R. Kumar, “Performance bounds for queueing networksand scheduling policies,” IEEE Trans. Autom. Control, vol. 39, no. 8, pp.1600–1611, Aug. 1994.

[2] D. Bertsimas, I. Paschalidis, and J. N. Tsitsiklis, “Optimization of multi-class queueing networks: Polyhedral and nonlinear characterizations ofachievable performance,” Ann. Appl. Probab., vol. 4, pp. 43–75, 1994.

[3] P. R. Kumar and S. P. Meyn, “Duality and linear programs for stabilityand performance analysis queueing networks and scheduling policies,”IEEE Trans. Autom. Control, vol. 41, no. 1, pp. 4–17, Jan. 1996.

[4] E. Schwerer, “A linear programming approach to the steady-state anal-ysis of Markov processes,” Ph.D. dissertation, Stanford Univ., Stanford,CA, 1997.

[5] J. R. Morrison and P. R. Kumar, “New linear program performancebounds for queueing networks,” J. Optim. Theory Appl., vol. 100, no.3, pp. 575–597, 1999.

[6] S. P. Meyn, “The policy iteration algorithm for average reward Markovdecision processes with general state space,” IEEE Trans. Autom. Con-trol, vol. 42, no. 12, pp. 1663–1680, Dec. 1997.

[7] R.-R. Chen and S. P. Meyn, “Value iteration and optimization of mul-ticlass queueing networks,” Queueing Syst.: Theory Appl., vol. 32, no.1–3, pp. 65–97, 1999.

[8] M. H. Veatch, “Approximate dynamic programming for networks: Fluidmodels and constraint reduction,” submitted for publication, 2005.

[9] S. G. Henderson and P. W. Glynn, “Approximating martingales for vari-ance reduction in Markov process simulation,” Math. Oper. Res., vol. 27,no. 2, pp. 253–271, 2002.

[10] S. G. Henderson, S. P. Meyn, and V. B. Tadic, “Performance evaluationand policy selection in multiclass networks,” Discrete Event Dyn. Syst.:Theory Appl., vol. 13, no. 1–2, pp. 149–189, 2003.

[11] S. P. Meyn, “Large deviation asymptotics and control variates for simu-lating large functions,” Ann. Appl. Probab., 2005, to be published.

[12] J. M. Harrison and R. J. Williams, “Brownian models of open queueingnetworks with homogeneous customer populations,” Stochastics, vol.22, no. 2, pp. 77–115, 1987.

[13] J. M. Harrison and J. A. Van Mieghem, “Dynamic control of Browniannetworks: State space collapse and equivalent workload formulations,”Ann. Appl. Probab., vol. 7, no. 3, pp. 747–771, 1997.

[14] J. M. Harrison, “Brownian models of open processing networks:Canonical representations of workload,” Ann. Appl. Probab., vol. 10,pp. 75–103, 2000.

[15] M. Bramson and R. J. Williams, “Two workload properties for Browniannetworks,” Queueing Syst.: Theory Appl., vol. 45, no. 3, pp. 191–221,2003.

[16] S. P. Meyn, “Sequencing and routing in multiclass queueing networks.Part II: Workload relaxations,” SIAM J. Control Optim., vol. 42, no. 1,pp. 178–217, 2003.

[17] M. Chen, C. Pandit, and S. P. Meyn, “In search of sensitivity in networkoptimization,” Queueing Syst.: Theory Appl., vol. 44, no. 4, pp. 313–363,2003.

[18] H. Chen and D. D. Yao, Fundamentals of Queueing Networks: Perfor-mance, Asymptotics, and Optimization, ser. Stochastic Modeling andApplied Probability. New York: Springer-Verlag, 2001.

[19] S. P. Meyn, “Dynamic safety-stocks for asymptotic optimality in sto-chastic networks,” Queueing Syst., Aug. 2005, to be published.

[20] M. I. Reiman, “Open queueing networks in heavy traffic,” Math. Oper.Res., vol. 9, pp. 441–458, 1984.

[21] H. Chen and A. Mandelbaum, “Stochastic discrete flow networks: Dif-fusion approximations and bottlenecks,” Ann. Probab., vol. 19, no. 4, pp.1463–1519, 1991.

[22] R. J. Williams, “Diffusion approximations for open multiclass queueingnetworks: Sufficient conditions involving state space collapse,”Queueing Syst.: Theory Appl., vol. 30, no. 1–2, pp. 27–88, 1998.

[23] P. Dupuis and R. J. Williams, “Lyapunov functions for semimartingalereflecting Brownian motions,” Ann. Appl. Probab., vol. 22, no. 2, pp.680–702, 1994.

[24] R. Atar, A. Budhiraja, and P. Dupuis, “On positive recurrence of con-strained diffusion processes,” Ann. Probab., vol. 29, pp. 979–1000, 2001.

[25] S. P. Meyn and R. L. Tweedie, “Stability of markovian processes III:Foster-Lyapunov criteria for continuous time processes,” Adv. Appl.Probab., vol. 25, pp. 518–548, 1993.

[26] , Markov Chains and Stochastic Stability. London, U.K.:Springer-Verlag, 1993.

[27] J. G. Dai and S. P. Meyn, “Stability and convergence of moments formulticlass queueing networks via fluid limit models,” IEEE Trans.Autom. Control, vol. 40, no. 11, pp. 1889–1904, Nov. 1995.

[28] M. Chen, R. Dubrawski, and S. P. Meyn, “Management of demand-driven production systems,” IEEE Trans. Autom. Control, vol. 49, no.5, pp. 686–698, May 2004.

[29] M. H. Veatch, “Fluid analysis of arrival routing,” IEEE Trans. Autom.Control, vol. 46, no. 8, pp. 1254–1257, Aug. 2001.

[30] J. G. Dai and G. Weiss, “A fluid heuristic for minimizing makespan injob shops,” Oper. Res., vol. 50, no. 4, pp. 692–707, 2002.

[31] N. Laws, “Dynamic routing in queueing networks,” Ph.D. dissertation,Cambridge Univ., Cambridge, U.K., 1990.

[32] S. P. Meyn, “Sequencing and routing in multiclass queueing networks.Part I: Feedback regulation,” SIAM J. Control Optim., vol. 40, no. 3, pp.741–776, 2001.

[33] L. M. Wein, “Dynamic scheduling of a multiclass make-to-stock queue,”Oper. Res., vol. 40, no. 4, pp. 724–735, 1992.

[34] I.-K. Cho and S. Meyn, “Optimization and the price of anarchy in adynamic newsboy model,” submitted for publication, 2005.

[35] S. Asmussen, “Queueing simulation in heavy traffic,” Math. Oper. Res.,vol. 17, pp. 84–111, 1992.

[36] W. Whitt, “Planning queueing simulations,” Manage. Sci., vol. 35, pp.1341–1366, 1994.

[37] P. Glynn and R. Szechtman, “Some new perspectives on the methodof control variates,” in Monte Carlo and Quasi-Monte Carlo Methods2000: Proceedings of a Conference Held at Hong Kong Baptist Univer-sity, Hong Kong SAR, China, K. Fang, F. Hickernell, and H. Niederreiter,Eds. Berlin, Germany: Springer-Verlag, 2002, pp. 27–49.

[38] S. G. Henderson and B. L. Nelson, Eds., Handbooks in OperationsResearch and Management Science, XII. ser. Simulation. Cambridge,MA: Elsevier, 2005.

[39] M. I. Reiman and R. J. Williams, “A boundary property of semimartin-gale reflecting Brownian motions,” Probab. Theory Related Fields, vol.77, no. 1, pp. 87–97, 1988. (correction in 80 (1989), no. 4, 633).

[40] L. M. Taylor and R. J. Williams, “Existence and uniqueness of semi-martingale reflecting Brownian motions in an orthant,” Probab. TheoryRelated Fields, vol. 96, no. 3, pp. 283–317, 1993.

[41] P. Dupuis and K. Ramanan, “Convex duality and the Skorokhodproblem. I, II,” Probab. Theory Related Fields, vol. 115, no. 2, pp.153–195, 1999. 197-236.

[42] P. Dupuis and I. Hitoshi, “On Lipschitz continuity of the solution map-ping to the Skorokhod problem, with applications,” Stoch. Stoch. Rep.,vol. 35, no. 1, pp. 31–62, 1991.

[43] S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization andConvergence. New York: Wiley, 1986.

[44] N. V. Krylov, Controlled Diffusion Processes, ser. Applications of Math-ematics. New York: Springer-Verlag, 1980, vol. 14. translated from theRussian by A. B. Aries.

[45] , “On a proof of Itô’s formula,” Trudy Mat. Inst. Steklov., vol. 202,pp. 170–174, 1993.

[46] R. Chitashvili and M. Mania, “Generalized Itô formula and derivation ofBellman’s equation,” in Stochastic Processes and Related Topics (Sieg-mundsberg, 1994), ser. Stochastics Monograph. Yverdon, France:Gordon and Breach, 1996, vol. 10, pp. 1–21.


[47] H. Föllmer and P. Protter, “On Itâ’s formula for multidimensionalBrownian motion,” Probab. Theory Related Fields, vol. 116, no. 1, pp.1–20, 2000.

[48] P. W. Glynn and S. P. Meyn, “A Liapounov bound for solutions of thePoisson equation,” Ann. Probab., vol. 24, no. 2, pp. 916–931, 1996.

[49] G. Weiss, “A simplex based algorithm to solve separated continuouslinear programs,” Dept. Statist., Univ. Haifa—Mount Carmel, Haifa, Is-rael, Tech. Rep. 31 905, 2001.

[50] D. Q. Mayne, “Control of constrained dynamic systems,” Eur. J. Control,vol. 7, pp. 87–99, 2001.

[51] F. Borrelli, Constrained Optimal Control of Linear and Hybrid Systems,ser. Lecture Notes in Control and Information Sciences. Berlin, Ger-many: Springer-Verlag, 2003, vol. 290.

[52] H. Thorisson, Coupling, Stationarity, and Regeneration, ser. Probabilityand its Applications. New York: Springer-Verlag, 2000.

[53] L. C. G. Rogers and D. Williams, Diffusions, Markov Processes, andMartingales, ser. Cambridge Mathematical Library. Cambridge, U.K.:Cambridge Univ. Press, 2000, vol. 2.

[54] D. Down, S. P. Meyn, and R. L. Tweedie, “Exponential and uniformergodicity of Markov processes,” Ann. Probab., vol. 23, no. 4, pp.1671–1691, 1995.

[55] I. Kontoyiannis and S. P. Meyn, “Spectral theory and limit theorems forgeometrically ergodic Markov processes,” Ann. Appl. Probab., vol. 13,pp. 304–362, 2003.

[56] S. P. Meyn and R. L. Tweedie, “Generalized resolvents and Harris re-currence of Markov processes,” Contemp. Math., vol. 149, pp. 227–250,1993.

[57] J. M. Harrison and M. I. Reiman, “On the distribution of multidimen-sional reflected Brownian motion,” SIAM J. Appl. Math., vol. 41, no. 2,pp. 345–361, 1981.

[58] J. M. Harrison and R. J. Williams, “Brownian models of feedforwardqueueing networks: Quasireversibility and product form solutions,” Ann.Appl. Probab., vol. 2, no. 2, pp. 263–293, 1992.

[59] J. G. Dai and J. M. Harrison, “Reflected Brownian motion in an orthant:Numerical methods for steady-state analysis,” Ann. Appl. Prob., vol. 2,pp. 65–86, 1992.

[60] V. S. Borkar and S. P. Meyn, “The O.D.E. method for convergence ofstochastic approximation and reinforcement learning,” SIAM J. ControlOptim., vol. 38, no. 2, pp. 447–469, 2000.

[61] presented at the INFORMS Applied Probability Conf., New York, Jul.2001.

Sean P. Meyn (S’85–M’87–SM’95–F’02) received the B.A. degree in mathe-matics (summa cum laude) from the University of California, Los Angeles, in1982, and the Ph.D. degree in electrical engineering from McGill University,Montreal, QC, Canada, in 1987.

He subsequently held a two year postdoctoral fellowship at the AustralianNational University, Canberra. He is currently a Professor with the Departmentof Electrical and Computer Engineering, and a Research Professor in the Co-ordinated Science Laboratory at the University of Illinois, Urbana-Champaign.He is a coauthor, along with R. Tweedie, of the monograph Markov Chains andStochastic Stability (New York: Springer-Verlag, 2002). His research interestsinclude stochastic processes, optimization, complex networks, and informationtheory.

Dr. Meyn received jointly with R. Tweedie the 1994 ORSA/TIMS Best Pub-lication In Applied Probability Award.

Documents

1106 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 50, … Collection/TAC/2005... · 2007-05-04 · 1106 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 50, NO. 8, AUGUST 2005 Workload