Embed Size (px)
Citation preview
Optimal and Hierarchical Controls in Dynamic
Stochastic Manufacturing Systems
A Review
S. P. Sethi, H. Yan, H. Zhang, and Q. Zhang
[Pick the date]
To be cited as: S. P. Sethi, H. Yan, H. Zhang, and Q. Zhang , Optimal and Hierarchical Controls in Dynamic Stochastic Manufacturing Systems: A Review @http://www.informs.org/Pubs/ITORMS/Archive/Volume-3/No.-2-Sethi-et-al
1
School of Management, The University
of Texas at Dallas
Richardson, TX 75083-0688, USA
Department of Systems Engineering
and Engineering Management
Chinese University of Hong Kong
Shatin, Hong Kong
Institute of Applied Mathematics
Academia Sinica, Beijing, 100080, China
Department of Mathematics
University of Georgia, Athens, GA 30602, USA
Abstract
Most manufacturing systems are large and complex and operate in an uncertain environment. One
approach to managing such systems is that of hierarchical decomposition. This paper reviews the
research devoted to proving that a hierarchy based on the frequencies of occurrence of different types
of events in the systems results in decisions that are asymptotically optimal as the rates of some events
become large compared to those of others. The paper also reviews the research on stochastic optimal
control problems associated with manufacturing systems, their dynamic programming equations,
existence of solutions of these equations, and verification theorems of optimality for the systems.
Manufacturing systems that are addressed include single machine systems, flowshops, and jobshops
producing multiple products. These systems may also incorporate random production capacity and
demands, and decisions such as production rates, capacity expansion, and promotional campaigns. The
paper concludes with a review of computational results and areas of applications.
2
Table of Contents Abstract ......................................................................................................................................................... 1
1. Introduction .............................................................................................................................................. 3
Part I: DISCOUNTED-COST MODELS .............................................................................................................. 6
2. Optimal Control under the Discounted Cost Criterion ............................................................................. 6
2.1 Optimal control of single or parallel machine systems....................................................................... 7
2.2 Optimal control of dynamic flowshops ............................................................................................. 14
2.3 Optimal control of dynamic jobshops ............................................................................................... 20
3. Hierarchical Controls under Discounted Cost Criterion ......................................................................... 30
3.1 Hierarchical controls for single or parallel machine systems ........................................................... 30
3.2 Hierarchical controls for flowshops .................................................................................................. 35
3.3 Hierarchical controls for jobshops .................................................................................................... 41
3.4 Hierarchical controls for production-investment models ................................................................ 46
3.5 Hierarchical controls for other multilevel models ............................................................................ 51
3.6 Computational results ....................................................................................................................... 54
4 Optimal Control with the Long-Run Average Cost Criterion .................................................................... 58
4.1 Optimal control of single or parallel machine systems..................................................................... 59
4.2 Optimal control of dynamic flowshops ............................................................................................. 69
4.3 Optimal control of dynamic jobshops ............................................................................................... 74
5 Hierarchical Controls under the Long-Run Average Cost Criterion ......................................................... 79
5.1 Hierarchical controls of single or parallel machine systems under symmetric deterioration and
cancelation rates ..................................................................................................................................... 79
5.2 Hierarchical control of single or parallel machine systems without deterioration and cancelation
rates ........................................................................................................................................................ 88
5.3 Hierarchical controls of dynamic flowshops ..................................................................................... 94
5.4 Hierarchical controls of dynamic jobshops ..................................................................................... 105
5.5 Markov decision processes with weak and strong interaction....................................................... 112
6 Risk-Sensitive Hierarchical Controls ....................................................................................................... 119
6.1 Risk-sensitive hierarchical controls with discounted costs ............................................................. 119
6.2 Risk-sensitive hierarchical controls with long-run average costs ................................................... 124
7 Extensions and Concluding Remarks ..................................................................................................... 130
Bibliography .......................................................................................................................................... 132
3
1. Introduction
Most manufacturing firms are large, complex systems characterized by several decision subsystems, such as finance personnel, marketing, and operations. They may have a number of plants and warehouses and produce a large number of different products using a wide variety of machines and equipment. Moreover, these systems are subject to discrete events such as construction of new facilities, purchase of new equipment and scrappage of old, machine setups, failures, and repairs, and new product introductions. These events could be deterministic or stochastic. Management must recognize and react to these events. Because of the large size of these systems and the presence of these events, obtaining exact optimal policies to run these systems is nearly impossible both theoretically and computationally.
One way to cope with these complexities is to develop methods of hierarchical decision making for these systems. The idea is to reduce the overall complex problem into manageable approximate problems or subproblems, to solve these problems, and to construct a solution for the original problem from the solutions of these simpler problems.
There are several different (and not mutually exclusive) ways by which to reduce the complexity. These include decomposing the problem into problems of smaller subsystems with a proper coordinating mechanism; aggregating products and subsequently disaggregating them; replacing random processes with their averages and possibly other moments; modeling uncertainties in the production planning problem via diffusion processes; and so on. Development of such approaches for large, complex systems was identified as a particular fruitful research area by the Committee on the Next Decade in Operations Research (1988), as well as by the Panel on Future Directions in Control Theory, see Fleming (1988). A great deal of research has been conducted in the areas of Operations Research, Operations Management, Systems Theory, and Control Theory. For their importance in practice, see the surveys of the literature by Libosvar (1988), Rogers, Evans, Plante, and Wong (1991), and Bitran and Tirupati (1993), a bibliography complied by Bukh (1992), and books by Stadtler (1988) and Switalski (1989). Some other references on hierarchical systems are Simon (1962), Mesarovic, Macko, and Takahara (1970), Smith and Sage (1973), Singh (1982), and Auger (1989). It should be noted, however, that most of them concern deterministic systems.
Each approach mentioned above is suited to certain types of models and assumptions. The approach we shall first discuss is that of modeling uncertainties in the production planning problem via diffusion processes. The idea is initiated by Sethi and Thompson (1981a), Sethi and Thompson (1981b), and Bensoussan, Sethi, Vickson, and Derzko (1984). Because controlled diffusion problems can often be solved (see Ghosh (1993), Ghosh, Arapostathis, and Marcus (1997), Harrison and Taksar (1983), and Harrison, Sellke, and Taylor (1983)), one uses the controlled diffusion models to approximate stochastic manufacturing systems. Kushner and Ramachandran (1989) begin with a sequence of systems whose limit is a controlled diffusion process. It should be noted that the traffic intensities of the systems in sequence converge to the critical intensity of one. They show that the sequence of value functions associated with the
4
given sequence converges to the value function of the limiting problem. This enables them to construct a sequence of asymptotic optimal policies defined to be those for which the difference between the associated cost and the value function converges to zero as the traffic intensity approaches its critical value. The most important application of this approach concerns the scheduling of networks of queues. If a network of queues is operating under heavy traffic, that is, when the rate of customers entering some of the stations in the network is very close to the rate of service at those stations, the problem of scheduling the network can be approximated by a dynamic control problem involving diffusion processes. The optimal policies that are obtained for the dynamic control problem involving diffusion approximation are interpreted in terms of the original problem A justification of this procedure based on simulation is provided in Harrison and Wein (1989), Harrison and Wein (1990), and Kumar and Kumar (1994), for example. See also the survey on fluid models and strong approximations by Chen and Mandelbaum (1994) for related research. Furthermore, Krichagina, Lou, Sethi, and Taksar (1993) and Krichagina, Lou, and Taksar (1994) apply this approach to the problem of controlling the production rate of a single product using a single unreliable machine in order to minimize the total discounted inventory/backlog costs. They imbed the given system into a sequence of systems in heavy traffic. Their purpose is to obtain asymptotic optimal policies for the sequence of systems that can be expressed only in terms of the parameters of the original system.
Before concluding our discussion of this approach, we should emphasize that so far the approach does not provide us with an estimate of how much the policies constructed for the given original system deviate from the optimal solution, especially when the optimal solution is not known, which is most often the case. As we shall see later, the hierarchical approach under consideration in this survey enables one to provide just such an estimate in many cases.
The next approach we shall discuss is that of aggregation-disaggregation. Bitran, Hass, and Matsuo (1986) formulate a model of a manufacturing system in which uncertainties arise from demand estimates and forecast revisions. They consider first a two-level product hierarchical structure, which is characterized by families and items. Hence, the production planning decisions consist of determining the sequence of the product families and the production lot sizes for items within each family, with the objective of minimizing the total cost. Then, they consider demand forecasts and forecast revisions during the planning horizon. The authors assume that the mean demand for each family is invariant and that the planners can estimate the improvement in the accuracy of forecasts, which is measured by the standard deviation of forecast errors. Bitran, Hass, and Matsuo (1986) view the problem as a two-stage hierarchical production planning problem. The aggregate problem is formulated as a deterministic mixed integer program that provides a lower bound on the optimal solution. The solution to this problem determines the set of product families to be produced in each period. The second-level problem is interpreted as the disaggregate stage where lot sizes are determined for the individual product to be scheduled in each period. Only a heuristic justification has been provided for the approach described. Some other references in the area are Bitran and Hax (1977) , Hax and Candea (1984), Gelders and Van Wassenhove (1981), Ari and Axsåter (1988), and Nagi (1991).
5
Lasserre and Mercé (1990) assume that the aggregate demand forecast is deterministic, while the detailed level forecast is nondeterministic within known bounds. Their aim is to obtain an aggregate plan for which there exists a feasible dynamic disaggregation policy. Such an aggregate plan is called a robust plan, and they obtain necessary and sufficient conditions for robustness; see also Gfrerer and Zäpfel (1994).
Finally we consider the approach of replacing random processes with their averages and possibly other moments, see Sethi and Zhang (1994a). The idea of the approach is to derive a limiting control problem which is simpler to solve than the given original problem. The limiting problem is obtained by replacing the stochastic machine capacity process by the average total capacity of machines and by appropriately modifying the objective function. The solution of this problem provides us with longer-term decisions. Furthermore, given these decisions, there are a number of ways by which we can construct short-term production decisions. By combining these decisions, we have created an approximate solution of the original, more complex problem.
The specific points to be addressed in this review are results on the asymptotic optimality of the constructed solution and the extent of the deviation of its cost from the optimal cost for the original problem. The significance of these results for the decision-making hierarchy is that management at the highest level of the hierarchy can ignore the day-to-day fluctuation in machine capacities, or more generally, the details of shop floor events, in carrying out long-term planning decisions. The lower operational level management can then derive approximate optimal policies for running the actual (stochastic) manufacturing system.
While the approach could be extended for applications in other areas, the purpose here is to review models of a variety of representative manufacturing systems in which some of the exogenous processes, deterministic or stochastic, are changing much faster than the remaining ones and to apply the methodology of hierarchical decision making to them. We are defining a fast changing process as a process that is changing so rapidly that, from any initial condition, it reaches its stationary distribution in a time period during which there are few, if any, fluctuations in the other processes.
In what follows we will review the applications of the approach in Markov decision problems and stochastic manufacturing problems, where the objective function is to minimize the total discounted cost, or a long-run average cost, or a risk-sensitive criterion. Also reviewed are the research on dynamic programming equations, existence of their solutions, and verification theorems of optimality for single machine systems, flowshops, and jobshops producing multiple products. The review is divided into three parts: Part I-Discounted cost model; Part II-Average cost model; and Part III-Risk-sensitive controls.
Part I consists of Section 2 and Section 3. In Section 2, we review the existence of solutions to the dynamic programming equations associated with stochastic manufacturing systems with the discounted cost criterion. The verification theorems of optimality and the characterization of optimal controls are also given. Section 3 is devoted to reviewing the results on open-loop
6
and/or feedback hierarchical controls that have been developed and shown to be asymptotically optimal for the systems. The computational issues are also included in this section. Part II consists of Section 4 and Section 5. In Section 4, we review the existence of solutions to the ergodic equations corresponding to stochastic manufacturing systems with the long-run average cost criterion. Also reviewed are the verification theorems of optimality and the characterization of optimal controls. Section 5 surveys hierarchical controls for single machine systems, flowshops, and jobshops under the long-run average cost criterion. Markov decision processes with weak and strong interactions are also included. Part III contains Section 6, which is concerned with risk-sensitive controls. In particular, we focus on the applications of the idea of hierarchical control in the risk-sensitive framework. Finally, we conclude the review in Section 7.
Part I: DISCOUNTED-COST MODELS
One of the most important cost criteria is that of the total discounted cost. In this part we will discuss optimal and hierarchical controls of stochastic manufacturing systems with convex cost under the discounted cost criterion. In Section 2, we consider the theory of optimality for these systems. In Section 3, the theory will be used in deriving hierarchical controls. Moreover, the hierarchical controls will be proved to be nearly optimal.
2. Optimal Control under the Discounted Cost Criterion
The class of convex production planning models is an important paradigm in the operations
management/operations research literature. The earliest formulation of a convex production planning
problem in a discrete-time framework dates back to . They were interested in obtaining a production
plan over a finite horizon in order to satisfy a deterministic demand and minimize the total discounted
convex costs of production and inventory holding. Since then the model has been further studied and
extended in both continuous-time and discrete-time frameworks by a number of researchers, including
Johnson (1957), , Veinott (1964), Adiri and Ben-Israel (1966), Sprzeuzkouski (1967), Lieber (1973), and
Hartl and Sethi (1984). A rigorous formulation of the problem along with a comprehensive discussion of
the relevant literature appears in Bensoussan, Crouhy, and Proth (1983).
Extensions of the convex production planning problem to handle stochastic demand have been analyzed mostly in the discrete-time framework. A rigorous analysis of the stochastic problem has been carried out in Bensoussan, Crouhy, and Proth (1983). Continuous-time versions of the model that incorporate additive white noise terms in the dynamics of the inventory process were analyzed by and . html
Earlier works that relate most closely to problems under consideration here include Kimemia and Gershwin (1983), Akella and Kumar (1986), Fleming, Sethi, and Soner (1987), Sethi, Soner,
7
Zhang and Jiang (1992), and Lehoczky, Sethi, Soner, and Taksar (1991). These works incorporate piecewise deterministic processes (PDP) either in the dynamics or in the constraints of the model. Fleming, Sethi, and Soner (1987) consider the demand to be a finite state Markov process. In the models of Kimemia and Gershwin (1983), Akella and Kumar (1986), Sethi, Soner, Zhang, and Jiang (1992a) and Lehoczky, Sethi, Soner, and Taksar (1991), the production capacity rather than the demand for production is modeled as a stochastic process. In particular, the process of machine breakdown and repair is modeled as a birth-death process, thus making the production capacity over time a finite state Markov process.
Here we will discuss the optimality of single/parallel machine systems, N-machine flowshops, and general jobshops.
2.1 Optimal control of single or parallel machine systems
Akella and Kumar (1986) deal with a single machine (with two states: up and down), single product
problem. They obtained an explicit solution for the threshold inventory level, in terms of which the
optimal policy is as follows: Whenever the machine is up, produce at the maximum possible rate if the
inventory level is less than the threshold, produce on demand if the inventory level is exactly equal to
the threshold, and not produce at all if the inventory level exceeds the threshold. Recently, Feng and
Yan (1999) extend the Akella-Kumar model to the case where the exogenous demand forms a
homogeneous Poisson flow. They prove the optimality of the threshhold control type for this extended
Akella-Kumar model. When their problem is generalized to convex costs and more than two machine
states, it is no longer possible to obtain explicit solutions. Using the viscosity solution technique, Sethi,
Soner, Zhang, and Jiang (1992) investigate this general problem. They study the elementary properties
of the value function. They show that the value function is a convex function and that it is strictly convex
provided the inventory cost is strictly convex. Moreover, it is shown to be a viscosity solution to the
Hamilton-Jacobi-Bellman (HJB) equation and to have upper and lower bounds each with polynomial
growth. Following the idea of Thompson and Sethi (1980), they define what are known as the turnpike
sets in terms of the value function. They prove that the turnpike sets are attractors for the optimal
trajectories and provide sufficient conditions under which the optimal trajectories enter the convex
closure in finite time. Also, they give conditions to ensure that the turnpike sets are non-empty.
To more precisely state their results, we need to specify the model of a single/parallel machine manufacturing system. Following the notation given by Sethi, Soner, Zhang, and Jiang (1992),
let , , , and m(t) denote, respectively, the inventory level, the production rate,
the demand rate, and the machine capacity level at time . We assume that
, , is a constant positive vector in Rn+. Furthermore, we assume
that is a Markov process with a finite space .
8
We can now write the dynamics of the system as
(2.1)
Definition 2.1 A control process (production rate) is called admissible
with respect to the initial capacity m if
(i)
is adapted to the filtration with , the
-field generated by m(t)
(ii)
for all for some positive vector .
Let denote the set of all admissible control processes with the initial condition m(0)=m.
Definition 2.2 A real-valued function on is called an admissible feedback
control, or simply feedback control, if
(i)
for any given initial , the equation
has a unique solution;
(ii)
.
Let and denote the surplus cost and the production cost function, respectively. For
every , , and m(0)=m, let the cost function of the system be defined as
follows:
9
(2.2)
where is the given discount rate. The problem is to choose an admissible control
that minimizes the cost function . We define the value function as
(2.3)
We make the following assumptions on the cost functions and .
Assumption 2.1 is a nonnegative, convex function with h(0)=0. There are positive
constants C21, C22, C23, and , such that
Assumption 2.2 is a nonnegative function, c(0)=0, and is twice differentiable.
Moreover, is either strictly convex or linear.
Assumption 2.3 is a finite state Markov chain with generator Q, where Q=(qij),
is a(p+1) x (p+1) matrix such that for and . That
is, for any function on ,
With these three assumptions we can state the following theorem concerned with the property of the
value function , and proved in Fleming, Sethi, and Soner (1987).
10
Theorem 2.1 (i) For each m, is convex on Rn, and is strictly convex if is so. (ii) There exist positive constants C24, C25, and C26 such that for each m
where and are the power indices in Assumption 2.1.
We next consider the equation associated with the problem. To do this, let
where is given in Definition 2.1, and `` " between and represents the inner product
of the vectors and . Then, the HJB equation is written formally as follows:
(2.4)
for , , where is the partial derivative of with respect to .
In general, the value function v may not be differentiable. In order to make sense of the HJB equation (2.4), we consider its viscosity solution. In order to give the definition of the viscosity
solution, we first introduce the superdifferential and subdifferential of a given function on Rn.
Definition 2.3 The superdifferential and the subdifferential of any
function on Rn are defined, respectively, as follows:
Based on Definition 2.3, we give the definition of a viscosity solution.
11
Definition 2.4 We say that is a viscosity solution of equation (2.4) if the following holds:
(i)
is continuous in and there exist C27>0 and such that
;
(ii)
for all ,
(iii)
for all ,
For further discussion on viscosity solutions, the reader is referred to Fleming and Soner (1992) or Sethi
and Zhang (1994a).
Lehoczky, Sethi, Soner, and Taksar (1991) prove the following theorem.
Theorem 2.2 The value function defined in (2.3) is the unique viscosity solution to the HJB equation (2.4).
Remark 2.1 If there is a continuously differentiable function that satisfies the HJB equation (2.4), then it is a viscosity solution, and therefore, it is the value function.
Furthermore, we have the following result.
12
Theorem 2.3 The value function is continuously differentiable and satisfies the HJB equation (2.4).
For the proof, see Theorem 3.1 in Sethi and Zhang (1994a).
Next, we give a verification theorem.
Theorem 2.4 (Verification Theorem) Suppose that there is a continuously differentiable function
that satisfies the HJB equation (2.4). If there exists , for which the
corresponding satisfies (2.1) with , , and
almost everywhere in t with probability one, then and is optimal,
i.e.,
For the proof, see Lemma H.3 of Sethi and Zhang (1994a).
Next we give an application of the verification theorem. With Assumption 2.2, we can use the verification theorem to derive an optimal feedback control for n=1. From Theorem 2.4, an optimal feedback control u*(x,m) must minimize
for each (x,m). Thus,
when c''(u) is strictly positive, and
13
when c(u)=cu for some constant .
Recall that is a convex function. Thus u*(x,m) is increasing in x. From a result on differential equations (see Hartman (1982)),
has a unique solution x*(t) for each sample path of the capacity process. Hence, the control given above
is the optimal feedback control. From this application, we can see that the point such thatvx(x,m)=-c'(z )
is critical in describing the optimal feedback control. So we give the following definition.
Definition 2.5 The turnpike set is defined by
Next we will discuss the monotonicity of the turnpike set. To do this, define to be
such that i0<z<i0+1. Observe that for ,
Therefore, x(t) goes to monotonically as , if the capacity state m is absorbing. Hence,
only those , for which , are of special interest to us.
In view of Theorem 2.1, if is strictly convex, then each turnpike set reduces to a singleton, i.e.,
there exists xm such that , .
If the production cost is linear, i.e., c(u)=cu for some constant c, then xm is the threshold
inventory level with capacity m. Specifically, if , and if
(full available capacity).
14
Let us make the following observation. If the capacity m>z, then the optimal trajectory will move toward the turnpike set xm. Suppose the inventory level is xm for some m and the capacity increases to m1>m; it then becomes costly to keep the inventory at level xm, since a lower inventory level may be more desirable given the higher current capacity. Thus, we expect
. Sethi, Soner, Zhang, and Jiang (1992) show that this intuitive observation is true. We state their result as the following theorem.
Theorem 2.5 Assume to be differentiable and strictly convex. Then
where .
2.2 Optimal control of dynamic flowshops
We consider a manufacturing system producing a single finished product using N machines in tandem
that are subject to breakdown and repair. We are given a Markov chain
, where mj(t), j=1,...,N, is the capacity of the jth machine at time t.
We use ui(t) to denote the input rate to the jth machine, j=1,...,N, and xj(t) to denote the number of
parts in the buffer between the jth and (j+1)th machines, j=1,...,N-1. Finally, the surplus is denoted by
xN(t). The dynamics of the system can then be written as follows:
(2.5)
where
Since the number of parts in the internal buffers cannot be negative, we impose the state constraints
, j=1,...,N-1. To formulate the problem precisely, let
denote the state constraint domain, and let be a finite Markov chain with the state space
15
, where ( ). For
, , j=1,...,N, let
(2.6)
and for , let
=
(2.7)
The matrix A has an inverse denoted as A-1. Let the -field .
Definition 2.6 We say that a control is admissible with respect to the initial value
and if
(i)
is an -adapted process;
(ii)
for all ;
(iii)
the corresponding state process with for
all .
Let denote the set of all admissible controls.
16
Definition 2.7 A function is call an admissible feedback control, or simply feedback control, if
(i)
for any given initial , the equation
has a unique solution with and ;
(ii)
.
The problem is to find an admissible control that minimizes the cost function
(2.8)
where defines the cost function of surplus and production rate , is the initial
value of , and is the discount rate. The value function is then defined as
(2.9)
We impose the following assumptions on the Markov process
and the cost function :
17
Assumption 2.4 is a nonnegative jointly convex function that is strictly convex in
either or or both. For all and , j=1,...,p, there exist
constants C28 and such that
Assumption 2.5 The capacity process is a finite state Markov chain with the following
infinitesimal generator Q:
for some and any function on .
The problem of the flowshop with internal buffers and the resulting state constraints is much more complicated. Certain boundary conditions need to be taken into account for the associated HJB equation. Optimal control policy can no longer be described simply in terms of some hedging points.
Lou, Sethi, and Zhang (1994) show that the optimal control policy for a two-machine flowshop with linear costs of production can be given in terms of two switching manifolds. However, the switching manifolds are not easy to obtain. One way to compute them is to approximate them by continuous piecewise-linear functions as done by Van Ryzin, Lou, and Gershwin (1993) in the absence of production costs.
To rigorously deal with the general flowshop problem under consideration, we write the HJB equation in terms of the Directional Derivatives (HJBDD) at inner and boundary points. So, we first give the notion of these derivatives and some related properties of convex functions.
Definition 2.8 A function , is said to have a directional derivative
along the direction if the following limit exists, i.e.,
18
If a function is differentiable at , then exists for every and
A continuous convex function defined on a convex domain is differentiable almost
everywhere and has a directional derivative both along any direction at any inner point of
and along any admissible direction (i.e., such direction that for some
) at any boundary point of . Note that is the
set of admissible directions at . We can formally write the HJB equation in terms of directional derivative (HJBDD) for the problem as
(2.10
)
Similar to Theorem 2.1, Presman, Sethi, and Zhang (1995) prove the following theorem.
Theorem 2.6 (i) The value function is convex and continuous on S and satisfies the condition
(2.11)
for some positive constant C29 and all .
(ii) The value function satisfies the HJBDD equation (2.10) for all .
19
Furthermore, by introducing an equivalent deterministic problem to the stochastic problem, Presman, Sethi, and Zhang (1995) give the verification theorem along with the existence of the optimal control.
Theorem 2.7 (i) The optimal control exists, is unique, and can be
represented as a feedback control, i.e., there exists a function such that for any , we have
where is the optimal state process, the solution of (2.5) for , with .
(ii) If some continuous convex function satisfies (2.10) and the growth condition (2.11) with
, then . Moreover, if there is a feedback control
providing the infimum in (2.10) for , then , and
is an optimal feedback control.
(iii) Assume that is strictly convex in for each fixed . Let denote the minimizer function of the right-hand side of (2.10). Then,
has a solution , and is the optimal control.
Remark 2.2 The method of dynamic programming plays an important role in solving Markovian
stochastic optimal control problems. A typical approach for treating a dynamic programming equation is
to use the notion of viscosity solutions introduced in Section 2.1. The viscosity solution approach is used
in the case of the two-machine flowshop analyzed by Lou, Sethi, and Zhang (1994). Since the inventory
in the internal buffers between the two machines is required to be nonnegative, the technique of the
viscosity solution with state space constraints developed by Soner (1986) is needed in their case.
Remark 2.3 Presman, Sethi, and Suo (1997) study the N-machine flowshop with limited buffers. They show that Theorem 2.6 and Theorem 2.7 also hold in the limited buffer case.
20
2.3 Optimal control of dynamic jobshops
We consider a manufacturing system producing a variety of products in demand using machines in a
general network configuration, which generalizes both the parallel and the tandem machine models.
Each product follows a process plan--possibly from a number of alternative process plans--that specifies
the sequence of machines it must visit and the operations performed by them. A process plan may call
for multiple visits to a given machine, as is the case in semiconductor manufacturing. See Lou and Kager
(1989) , Srivatsan, Bai, and Gershwin (1994), Uzsoy, Lee, and Martin-Vega (1996), and Yan, Lou, Sethi,
Gardel, and Deosthali (1996). Often the machines are unreliable. Over time they break down and must
be repaired. A manufacturing system so described will be termed a dynamic jobshop.
Now we give the mathematically precise description of a jobshop suggested by Presman, Sethi, and Suo (1997a) as a revision of the description by Sethi and Zhang (1994a). First we give some definitions.
Definition 2.9 A manufacturing diagraph is a graph , where is a set of ,
vertices, and is a set of ordered pairs called arcs, satisfying the following properties:
(i)
There is only one source, labeled 0, and only one sink, labeled Nb+1, in the digraph.
(ii)
No vertex in the graph is isolated.
(iii)
The digraph does not contain any cycle.
Remark 2.4 Condition (ii) is not an essential restriction. Inclusion of isolated vertices is merely a
nuisance. This is because an isolated vertex is like a warehouse that can only ship out parts of a
particular type to meet their demand. Since no machine (or production) is involved, its inclusion or
exclusion does not affect the optimization problem under consideration. Condition (iii) is imposed to
rule out the following two trivial situations: (a) a part of type i in buffer i gets processed on a machine
without any transformation and returns to buffer i, and (b) a part of type i is processed and converted
back into a part of type j, , and is then processed further on a number of machines to
be converted back into a part of type i. Moreover, if we had included any cycle in our manufacturing
system, the flow of parts that leave buffer i only to return to buffer i would be zero in any optimal
solution. It is unnecessary, therefore, to complicate the problem by including cycles.
21
Definition 2.10 In a manufacturing digraph, the source is called the supply node and the sink represents the customers. Vertices immediately preceding the sink are called external buffers, and all others are called internal buffers.
In order to obtain the system dynamics from a given manufacturing digraph, a systematic procedure is required to label the state and control variables. For this purpose, note that our
manufacturing digraph contains a total of Nb+2 vertices including the source, the sink,
m internal buffers, and Nb-m external buffers for some integer m and Nb, ,
.
Theorem 2.8 We can label all the vertices from 0 to Nb+1 in a way so that the label numbers of the vertices along every path are in a strictly increasing order, the source is labeled 0, the sink is labeled Nb+1, and the external buffers are labeled m+1, m+2,...,Nb.
The proof is similar to Theorem 2.2 in Sethi, and Zhou (1994).
With the help of Theorem 2.8, one is able to formally write the dynamics and the state constraints associated with a given manufacturing digraph. To do this, we require the following definitions.
Definition 2.11 For each arc (i,j), , in a manufacturing digraph, the rate at which parts in buffer i are converted to parts in buffer j is labeled as controluij. Moreover, the control uij associated with the arc (i,j) is called an output of i and an input to j. In particular, outputs of the source are called primary controls of the digraph. For each arc (i, Nb+1), i=m+1,...,Nb, the demand for products in buffer i is denoted by zi.
In what follows, we shall also set
for a unified notation suggested in Presman, Sethi, and Suo (1997a). While zi and ui,j for are
not controls, we shall for convenience refer to ui,j, , , as controls. In
this way, we can consider the controls as an (Nb+1) x (Nb+1) matrix of the following form.
22
The set of all such controls is written as , i.e.,
Before writing the dynamics and the state constraints corresponding to the manufacturing digraph
containing Nb+2 vertices consisting of a source, a sink, m internal buffers, and Nb-m external
buffers associated with the Nb-m distinct final products to be manufactured (or characterizing a
jobshop), we give the description of the control constraints. We label all the vertices according to
Theorem 2.8. For simplicity in the sequel, we shall call the buffer whose label is i as buffer i, i=1,2,...,Nb.
The control constraints depend on the placement of the machines, and the different placements on the
same digraph will give rise to different jobshops. In other words, a jobshop corresponds to a unique
digraph, whereas a digraph may correspond to many different jobshops. Therefore, to uniquely
characterize a jobshop using graph theory, we need to introduce the concept of a placement of
machines, or simply a placement. Let , where denotes the total number of
arcs in .
Definition 2.12 In a manufacturing digraph , a set is called a
placement of machines 1,2,...,N, if is a partition of ,
namely, , for , and .
A dynamic jobshop can be uniquely specified by a triple , which denotes a
23
manufacturing system that corresponds to a manufacturing digraph along with a
placement of machines .
Consider a jobshop , let uij(t) be the control at time t associated with arc (i,j),
. Suppose we are given a stochastic process on
the standard probability space with mn( t) representing the capacity of the nth
machine at time t, n=1,...,N. The controls uij(t) with , n=1,...,N, , should satisfy the following constraints:
(2.12)
where we have assumed that the required machine capacity pij (for unit production rate of type j from
part type i) equals 1, for convenience in exposition. The analysis in this paper can be readily extended to
the case when the required machine capacity for the unit production rate of part j from part i is any
given positive constant.
We denote the surplus at time t in buffer i by xi(t), . Note that if xi(t)>0, i=1,...,Nb, we have an inventory in buffer i, and if xi(t)<0, i=m+1,...,Nb, we have a shortage of finished product i. The dynamics of the system are, therefore,
(2.13)
with . Since internal buffers provide
inputs to machines, a fundamental physical fact about them is that they must not have shortages. In
other words, we must have
(2.14)
Let
24
and
The relation (2.13) can be written in the following vector form:
(2.15)
where is the corresponding linear operator with
, and . Let
. Furthermore, let be the boundary of , and the interior .
Let us illustrate a jobshop by the following simple example.
Example 2.1. In Figure 2.1, we have four machines , two distinct products, and
five buffers. Each machine has capacity mi(t) at time t, and each product
j=1,2 has demand zj. As indicated in the figure, known as the state variables are associated with the buffers. More specifically, xi denotes the inventory/backlog of
part type . Control variables represent production rates. More specifically, u 1 and u2 are the rates at which raw parts coming from outside are converted to part types 1 and 5, respectively, and u3, u4, u5 and u6 are the rates of conversion from part types 3,1,1, and 2 to part types 4, 2, 4 and 3, respectively. Thus,
Therefore, the system dynamics associated with Figure 2.1 is
25
(2.16)
and the process must satisfy the capacity constraints
(2.17)
For i=1,2,3, buffer i is between two machines and is known as an internal buffer, the state constraints
are
(2.18)
for our example. The remaining buffers 4 and 5 are external buffers. x4(t) and x5 (t) are called surpluses
with positive values meaning inventories and negative values meaning backlogs.
We are now in the position to formulate our stochastic optimal control problem for the jobshop
defined by (2.13)-(2.15). For , let
and for and ,
26
Definition 2.13 We say that a control is admissible with respect to the initial state vector
and , if
(i)
is an -adapted measurable process with ;
(ii)
for all ;
(iii)
the corresponding state process for all .
Remark 2.5 The condition (iii) is equivalent to , .
Let denote the set of all admissible control with respect to and the
machine capacity vector . The problem is to find an admissible control that minimizes the cost function
(2.19)
where defines the cost of inventory/shortage and the production cost, is the initial state, and
is the initial value of .
The value function is then defined as
27
(2.20)
We impose the following assumptions on the random process
and the cost functions throughout this section.
Assumption 2.6 is a nonnegative and convex function. Further, for all
and , there exist constants and such that
Assumption 2.7 Let for some given integer . The capacity
process , , is a finite state Markov chain with generator Q=(qkk') such that
if and . Moreover, Q is irreducible.
We use to denote our control problem. Presman, Sethi, and Suo (1997a) prove the following theorem.
Theorem 2.9 The optimal control exists, and can be represented as a
feedback control, i.e., there exists a function such that for any we have
where is the optimal state process--the solution for (2.15) for
with . However, if is strictly convex in
, then the optimal feedback control is unique.
28
Now we consider the Lipschitz property of the value function. It should be noted that unlike in the case without state constraints, the Lipschitz property in our case does not follow directly. The reason for this is that in the presence of state constraints, a control which is admissible
with respect to is not necessarily admissible for when
.
Theorem 2.10 The value function is convex and continuous, and satisfies the condition
(2.21)
for some positive constant and all .
Because the problem of the jobshop involves state constraints, we can write the HJBDD for the problem as in Section 2.2:
(2.22)
Theorem 2.11 (Verification Theorem) (i) The value function satisfies equation
(2.22) for all . (ii) If some continuous convex function satisfies (2.22) and
the growth condition (2.21) with , then . Moreover, if
there exists a feedback control providing the infimum in (2.22) for ,
then , and is an optimal feedback control. (iii) Assume
that is strictly convex in for each fixed . Let denote the minimizer function of the right-hand side of (2.22). Then,
has a solution , and is the optimal control.
29
Remark 2.6 The HJBDD (2.22) coincides at inner points of with the usual dynamic programming equation for convex PDP problems. Here PDP is the abbreviation of piecewise deterministic processes introduced by Vermes (1985) and Davis (1993). The HJBDD gives at
boundary points of , a boundary condition in the following sense. Let the restriction of
on some l-dimensional face, 0<l<N, of the boundary of S be differentiable at an
inner point of this face. Note that this restriction is convex and is differentiable almost
everywhere on this face. Then there is a vector such that
for any admissible direction at . It follows from the continuity of the value function that
+
=
(2.23
)
The boundary condition on given by (2.23) can now be interpreted as follows. First, the optimal
control policy on the boundary has the same intuitive explanation as in the interior. The important
difference is that we now have to worry about the feasibility of the policy. What the boundary condition
accomplishes is to shape the value function on the boundary of in such a way that the unconstrained
optimal policy is also feasible.
According to (2.22), optimal feedback control policies are obtained in terms of the directional derivatives of the value function.
Note now that the uniqueness of the optimal control follows directly from the strict convexity
of function in and the fact that any convex combination of admissible controls for
any given is also admissible. For proving the remaining statements of Theorem 2.10 and Theorem 2.11, see Presman, Sethi, and Suo (1997a).
Remark 2.7 Presman, Sethi, and Suo (1997a) show that Theorem 2.9, Theorem 2.10, and Theorem 2.11 also hold when the systems are subject to lower and upper bound constraints on work-in-process.
30
3. Hierarchical Controls under Discounted Cost Criterion
In this section, the problems of hierarchical production planning with the discounted cost is discussed.
We present asymptotic results for hierarchical production planning in manufacturing systems with
machines subject to breakdown and repair. We make use of the singular perturbation methods to
reduce the original problems into simpler problems and then we describe a procedure to construct a
control, derived from the solution to the simpler problems, for the original systems. The simpler
problems turn out to be nothing but limiting problems obtained by averaging the given stochastic
machine capacities and modifying the objective function in a reasonable way to account for the
convexity of the cost function. Therefore, by showing that the associated value function for the original
systems converge to the value functions of the limit systems, we can construct controls for the original
systems from the optimal control of the limit systems. It turns out that the controls so constructed are
asymptotically optimal as the fluctuation rate of the machine capacities goes to infinity. Furthermore,
error estimates of the asymptotic optimality are provided in terms of their corresponding cost functions.
Here we will discuss hierarchical controls in single/parallel machine systems, flowshops, jobshops, and production-investment systems. Finally some computational results are given.
3.1 Hierarchical controls for single or parallel machine systems
Sethi and Zhang (1994b) and Sethi, Zhang, and Zhou (1994) consider a stochastic manufacturing system
with surplus (state) and production rate (control) satisfying
(3.1)
where is the constant demand rate and is the initial surplus .
Let denote the machine capacity process of our
manufacturing system, where is a small parameter to be specified later. Then the production
rate must satisfy for some positive vector . We
consider the cost function with and defined by
31
(3.2)
where is the discount rate, is the cost of surplus, and is the cost of production. The
problem is to find a control with , that minimizes
. We make the following assumptions on the machine capacity process and the
cost function on production rate and the surplus.
Assumption 3.1 and are convex. Furthermore, for all , there exist constants C31
and such that
and
Assumption 3.2 Let , where is an (p+1) x (p+1) matrix such that
with if and , for The capacity
process is a finite state Markov process governed by , that is,
for any function on .
Assumption 3.3 The Q(2) is weakly irreducible, i.e., the equations
32
have a unique solution . We call to be the equilibrium distribution of
Q.
Remark 3.1 Jiang and Sethi (1991) and Khasminskii, Yin, and Zhang (1997) consider a model in which the irreducibility Assumption 3.3 can be relaxed to incorporate machine state processes with a generator that consists of several irreducible submatrices. In these models, some jumps are associated with a fast process, while others are associated with a slow process; see Section 5.5.
Definition 3.1 We say that a control is admissible if
(i)
is an adapted measurable process;
(ii)
for all .
We use to denote the set of all admissible controls with the initial condition
. Then our control problem can be written as follows:
(3.3
)
Similar to Theorem 2.1, we can show that the value function is convex in for each m.
The value function satisfies the dynamic programming equation
33
=
(3.4)
in the sense of viscosity solutions.
Sethi, Zhang, and Zhou (1994) consider a control problem in which the stochastic machine
capacity process is averaged out. Let denote the control space
Then we define the control problem as follows:
(3.5
)
Sethi and Zhang (1994b) and Sethi, Zhang, and Zhou (1994) construct a solution of from a solution
of and show it to be asymptotically optimal as stated below.
Theorem 3.1 (i) There exists a constant C32 such that
(ii) Let denote an -optimal control. Then is asymptotically optimal, i.e.,
(3.6)
(iii) Assume in addition that is twice differentiable with , the function
is differentiable, and constants C33and exist such that
34
Then, there exists a locally Lipschitz optimal feedback control for . Let
(3.7)
Then, is an asymptotically optimal feedback control for with the
convergence rate of , i.e., (3.6) holds.
Remark 3.2 Part (ii) of the theorem states that from an -optimal open-loop control of the limiting
problem, we can construct an -optimal open-loop control for the original problem. With further
restrictions on the cost function, Part (iii) of the theorem states that from the -optimal feedback
control of the limiting problem, we can construct an -optimal feedback control for the original
problem.
Remark 3.3 It is important to point out that the hierarchical feedback control (3.7) can be shown to be a
threshold-type control if the production cost is linear. Of course, the value of the threshold
depends on the state of the machines. For single product problems with constant demand, this means
that production takes place at the maximum rate if the inventory is below the threshold, no production
takes place above it, and production rate equals the demand rate once the threshold is attained. This is
also the form of the optimal policy for these problems as shown, e.g., in Kimemia and Gershwin (1983),
Akella and Kumar (1986), and Sethi, Soner, Zhang, and Jiang (1992a). The threshold level for any given
machine capacity state in these cases is also known as a hedging point in that state following Kimemia
and Gershwin (1983). In these simple problems, asymptotic optimality is maintained as long as the
threshold, say, goes to 0 as Thus, there is a possibility of obtaining better policies than
(3.7) that are asymptotically optimal. In fact, one can even minimize within the class of threshold
policies for the parallel-machines problems discussed in this section.
Remark 3.4 Gershwin (1989) constructs a solution for by solving a secondary optimization problem and conjectures his solution to be asymptotically optimal. Sethi and Zhang (1994b), Remark (6.3) prove the conjecture. It should be noted, however, that the conjecture cannot be extended to include the simple two-machine example in Gershwin (1989) with one flexible and another inflexible machine. The presence of the inflexible machine requires aggregation of
some products at the level of and subsequent disaggregation in the construction of a
solution for . Also note that Gershwin, Caramanis, and Murray (1988), in their simulation study, consider hierarchical systems that do not involve aggregation and disaggregation.
35
Soner (1993) and Sethi and Zhang (1994c) consider in which depends on
the control variable . They show that under certain assumptions, the value function converges to the value function of a limiting problem. Moreover, the limiting problem can be
expressed in the same form as except that the equilibrium distribution
are now control-dependent. Thus in Assumption 3.3 is now
replaced by for each i; see also (3.31). Then an asymptotically optimal control for
can be obtained as in (3.7) from the optimal control of the limiting problem. As yet, no convergence rate has been obtained in this case.
An example of in a one-machine case with two (up and down) states is
Thus, the breakdown rate of the machine depends on the rate of production , while the
repair rate is independent of the production rate. These are reasonable assumptions in practice.
3.2 Hierarchical controls for flowshops For manufacturing systems with N machines in tandem, Sethi, Zhang, and Zhou (1992a) obtain a limiting
problem. Then they use a near-optimal control of the limiting problem to construct an open-loop control
for the original problem, which is asymptotically optimal as the rate of fluctuation in the machine states
goes to infinity. To illustrate, let us consider a two-machine flowshop treated in Sethi, Yan, Zhang, and
Zhou (1993). Let the Markov process generated by an
irreducible generator denote the machine capacity process. Here we assume that is a
finite state Markov process with the state space . Furthermore, we assume
that Q is weakly irreducible with the stationary distribution .
36
We denote the number of parts in the buffer between the first and the second machine called
work-in-process as and the difference of the real and planned cumulative
productions called surplus at the second machine as . Let denote the state constraint domain and let z denote the constant demand rate. Then,
Definition 3.2 We say that a control is admissible with respect to
if:
(i)
is adapted to ;
(ii)
;
(iii)
for all .
We use to denote the class of all admissible controls with the initial condition
and . Then, our control problem can be written as follows:
(3.8
)
37
Remark 3.5 The case of a finite capacity internal buffer, in which the state constraint domain is S=[0,M]
x R1 for some finite number M, is treated in Sethi, Zhang, and Zhou (1992b) and Sethi, Zhang, and Zhou
(1997). Recently, based on the Lipschitz continuity of the value function given by Presman, Sethi, and
Suo (1997), Sethi, Zhang and Zhang (1999c) give the hierarchical control for the N-machine flowshop
with limited buffers.
For , let denote the set of the deterministic measurable controls
such that for all , i=1,2 and j=1,2,...,p. We define the limiting problem
where is the equilibrium distribution of Q.
It can be shown that there exists an , such that for all and , we have
(3.9)
Similar to Theorem 3.1, for a given , we describe the procedure of constructing an asymptotic
optimal control of the original problem beginning with any near-optimal
control of the limiting problem . First we focus on the open-loop control. Let us
fix an initial state . Let , where
is an -optimal control for , i.e.,
38
Because the work-in-process level must be nonnegative, unlike in the case of parallel machine systems,
the control
may not be admissible. Thus, we need to modify the process given in Section 2.1. Let us define
a time as follows:
We define another control process as follows: For ,
(3.10)
It is easy to check that . Let
=
=
(3.11)
and let be the corresponding trajectory defined as
39
Note that . However, may not be in S for some
. To obtain an admissible control for , we need to modify so that the state
trajectory stays in S. This is done as follows. Let
(3.12)
Then, for the control constructed (3.10)-(3.12) above, it is shown in Sethi,
Yan, Zhang, and Zhou (1993) that
(3.13)
Moreover, the case of more than two machines is treated in Sethi, Zhang, and Zhou (1992a).
Next we consider a construction of asymptotically optimal feedback controls. The problem is addressed by Sethi, and Zhou (1996a), Sethi, and Zhou (1996b). They construct such controls for
in (3.8) with
(3.14)
where c1, c1+ and c1
- are given nonnegative cost coefficients and and
. In order to illustrate their results, we choose a simple situation in which each
of the two machines has a capacity m when up and 0 when down, and has a breakdown rate
and the repair rate . Furthermore, we shall assume that the average capacity of
each machine is strictly larger than the demand z. It is easy to see that the optimal control of the
corresponding limiting (deterministic) problem is:
40
(3.15)
From this, Sethi, and Zhou (1996a), Sethi, and Zhou (1996b) construct the following asymptotically
optimal feedback control:
(3.16
)
where as ; see Figure 3.1.
41
Note that the optimal control (3.15) of uses the obvious bang-bang and singular controls to go to
(0,0) and then stay there. In the same spirit, the control in (3.16) uses bang-bang and singular controls to
approach . For a detailed heuristic explanation of asymptotic optimality, see
Samaratunga, Sethi, and Zhou (1997); for a rigorous proof, see Sethi, and Zhou (1996a), Sethi, and Zhou
(1996b).
Remark 3.6 The policy in Figure 3.1 cannot be termed a threshold-type policy, since there is no
maximum tendency to go to when the inventory level x1 (t) is below and
In fact, Sethi, and Zhou (1996a), Sethi, and Zhou (1996b) show that a threshold-type policy, known also as a Kanban policy (see also Section 7), is not even asymptotically optimal when c1 > c2
+. Also, it is known that the optimal feedback policy for two-machine flowshops involve switching manifolds that are much more complicated than the
manifolds and required to specify a threshold-type policy. This implies that in the discounted flowshop problems, one cannot find an optimal feedback policy within the
class of threshold-type policies. While and could still be called hedging points, there is no notion of optimal hedging points insofar as they are used to specify a feedback policy. See Samaratunga, Sethi, and Zhou (1997) for a further discussion on this point.
Remark 3.7 Fong and Zhou (1997) study the hierarchical production policies in stochastic two-machine flowshop with finite buffers. Using a constraint domain approximation approach developed by Fong and Zhou (1996), they construct controls for the original problem from an optimal control of the limiting problem in a way similar to (3.15)-(3.16). Finally, they show that
the constructed controls are asymptotically optimal. When in the two-machine case, and in one general N-machine flowshop case, the problem of how to construct near-optimal feedback controls remain open.
3.3 Hierarchical controls for jobshops
Sethi, and Zhou (1994) consider hierarchical production planning in a general manufacturing system
given in Section 2.3. For the jobshop , let be the control at time t associated with
arc (i,j), . Suppose we are given a stochastic process
on the standard probability space with
42
representing the capacity of the nth machine at time t, n=1,...,N, where is a small
parameter to be precisely specified later. The controls with , n=1,...,N, ,
should satisfy the following constraints:
(3.17)
where we have assumed that the required machine capacity pij (for unit production rate of type j from
part type i) equals 1, for convenience in exposition. The analysis in this paper can be readily extended to
the case when the required machine capacity for the unit production rate of part j from part i is any
given positive constant.
We denote the surplus at time t in buffer i by , . Note that if
, i=1,...,Nb, we have an inventory in buffer i, and if , i=m+1,...,Nb, we have a shortage of finished product i. The dynamics of the system are therefore
(3.18)
with .
Let
and
Similar to Section 2.3, we write the relation (3.18) in the following vector form:
(3.19)
43
Definition 3.3 We say that a control is admissible with respect to the initial state vector
and , if
(i)
is an -adapted measurable process with ;
(ii)
for all ;
(iii)
the corresponding state process for all .
Remark 3.8 The condition (iii) is equivalent to , .Let
denote the set of all admissible control with respect to and the machine
capacity vector . The problem is to find an admissible control that minimize the cost
function
(3.20)
where defines the cost of inventory/shortage, is the production cost, is the initial state,
and is the initial value of .
The value function is then defined as
(3.21)
44
We impose the following assumptions on the random capacity process
and the cost functions and throughout this
section.
Assumption 3.4 Let for some given integer , where
, with mjk, k=1,...,N denoting the capacity of the kth machine, j=1,...,p.
The capacity process is a finite state Markov chain with the infinitesimal
generator , whereQ(1)=(qij(1)) and Q(2)=(qij
(2)) are matrices such that
if , and for r=1,2. Moreover, Q(2) is irreducible and, without any loss of generality, it is taken to be the one that satisfies
Assumption 3.5 Assume that Q(2) is weakly irreducible. Let denote the equilibrium
distribution of Q(2), that is, is the only nonnegative solution to the equations
(3.22)
Assumption 3.6 and are convex functions. For all and , there exist
constants C34 and such that
We use to denote our control problem
45
(3.23
)
In order to obtain the limiting problem, we consider the class of deterministic controls defined below.
Definition 3.4 For , let denote the set of the following measurable controls
with , , n=1,...,N, and the corresponding solution of
the system
(3.24
)
satisfies for all .
The object of the limiting problem is to choose a control that minimizes
We write (3.24) in the vector form
We use to denote the limiting problem and derive it as follows:
46
Based on the Lipschitz continuity of the value function given in Section 2.3, Sethi, and Zhou
(1994) prove the following theorem, which says that the problem is indeed a limiting
problem in the sense that the value function of converges to the value
function of . Furthermore, the theorem also gives the corresponding convergence rate.
Theorem 3.2 For each , there exists a positive constant C35 such that for all
and sufficiently small , we have
Based on Presman, Sethi, and Suo (1997a), which is related to the Lipschitz continuity of the value function for the general jobshop subject to lower and upper bound constraints on work-in-process, Sethi, Zhang and Zhang (1999d) also show that Theorem 3.2 is true for a general jobshop system with limited buffers.
3.4 Hierarchical controls for production-investment models Sethi, Taksar, and Zhang (1992b) incorporate an additional capacity expansion decision in the model
discussed in Section 3.1. They consider a stochastic manufacturing system with the inventory/backlog or
surplus and production rate that satisfy
(3.25)
where denotes the constant demand rate and is the initial surplus level. They assume
and for some , where is the machine
capacity process described by (3.27). The specification of involves the instantaneous
purchase of some given additional capacity at some time , , at a cost of K, where
47
means not to purchase it at all; see Sethi, Taksar, and Zhang (1994a) for an alternate model in
which the investment in the additional capacity is continuous. Therefore, their control variable is a pair
of a Markov time and a production process over time.
They consider the cost function defined by
(3.26)
where is the initial capacity and is the discount rate. The problem is to find
an admissible control that minimizes .
Define and as two Markov processes with state spaces
and , respectively. Here,
denotes the existing production capacity process and denotes the capacity process of the system if it were to be supplemented by the additional new capacity at time t=0.
Define further a new process as follows: For each -Markov time ,
(3.27
)
Here m2 denotes the maximum additional capacity resulting from the investment in the new capacity.
We make the following assumptions on the cost function and the process .
Assumption 3.7 is a nonnegative jointly convex function that is strictly convex in
either or or both. For all and , there exist constant C35
and such that
48
Assumption 3.8 and are Markov processes with
generators and , respectively, whereQ1=(q(1)ij) and Q2=(q(2)
ij) are matrices such
that if and for k=1,2. Moreover, Q1 and Q2 are both
irreducible.Let , ,
and .
Definition 3.5 We say that a control is admissible if
(i)
is an -Markov time;
(ii)
is -adapted and for .
We use to denote the set of all admissible controls . Then the problem is:
We use to denote the value function of the problem and define an auxiliary value function
to be K plus the optimal cost with the capacity process with the initial capacity
and no future capital expansion possibilities. Then the dynamic programming equations
are as follows:
49
(3.28
)
(3.29
)
Let and denote the
equilibrium distributions of Q1 and Q2, respectively. We now proceed to develop a limiting problem. We
first define the control sets for the limiting problem. Let
and
Then and
Definition 3.6 We use to denote the set of the following controls (admissible controls for the limiting problem):
(i)
a deterministic time ;
(ii)
a deterministic U(t) such that for , and for
, .
Let
50
and let
We can now define the following limiting optimal control problem:
(3.30)
Let denote the value functions for . Let denote any
admissible control for the limiting problem where
We take
Then the control is admissible for .
The following result is proved in Sethi, Taksar, and Zhang (1992b).
Theorem 3.3 (i) There exists a constant C36 such that
51
(ii) Let be an -optimal control for the limiting problem and let
be the control constructed above. Then, is asymptotically optimal
with error bound , i.e.,
3.5 Hierarchical controls for other multilevel models
Sethi and Zhang (1992b), and Sethi and Zhang (1995a) extend the model in Section 3.1 to incorporate
promotional or advertising decisions that influence the product demands. Zhou and Sethi (1994)
demonstrate how workforce and production decisions can be decomposed hierarchically in a stochastic
version of the classical HMMS model Holt, Modigliani, Muth, and Simon (1960). Manufacturing systems
involving preventive maintenance are studied by Boukas and Haurie (1990), Boukas (1991), Boukas,
Zhang, and Zhou (1993), and Boukas, Zhu and Zhang (1994). The maintenance activity involves
lubrication, routine adjustment, etc., which reduce the machine failure rates. The objective in these
systems is to choose the rate of maintenance and the rate of production in order to minimize the total
discounted cost of surplus, production, and maintenance.
In this section, we shall only discuss the model developed in Sethi and Zhang (1995a), we consider the case when both capacity and demand are finite state Markov processes constructed from generators that depend on the production and promotional decisions,
respectively. In order to specify their marketing-production problem, let as in
Section 3.1 and , for a given , denote the capacity process and the demand process, respectively.
Definition 3.7 We say that a control is admissible, if
(i)
is right-continuous having left-hand limit (RCLL);
(ii)
52
is , and satisfies
, and for all .
We use to denote the set of all admissible controls. Then our control problem can
be written as follows:
(3.3
1)
where by , we mean that the Markov process has the
generator .
We use to denote the admissible control space
53
(3.3
2)
Let denote an optimal open-loop control. We construct
Then , and it is asymptotically optimal, i.e.,
Similarly, let denote an optimal feedback control for .
Suppose that is locally Lipschitz for each . Let
and
Then the feedback control is asymptotically optimal for , i.e.,
54
We have described only the hierarchy that arises from a large and a small . In this case, promotional decisions are obtained under the assumption that the available production capacity is equal to the average capacity. Subsequently, production decisions taking into account the stochastic nature of the
capacity can be constructed. Other possible hierarchies result when both and are small or when is
large and is small. The reader is referred to Sethi and Zhang (1995a) for details on these.
3.6 Computational results Gershwin and associates ( Connolly, Dallery, and Gershwin (1992), Van Ryzin, Lou, and Gershwin (1993),
Violette (1993) and Violette and Gershwin (1991) ) have carried out a good deal of computational work
in connection with manufacturing systems without state constraints. Such systems include single or
parallel machine systems described in Section 3.1, Section 3.2, and Section 3.3 as well as no-wait
flowshops (or flowshops without internal buffers) treated in Kimemia and Gershwin (1983).
Darakananda (1989) has developed a simulation software called Hiercsim based on the control
algorithms of Gershwin, Akella, and Choong (1985), and Gershwin (1989). It should be noted that
controls constructed in these algorithms have been shown under some conditions to be asymptotically
optimal by Sethi and Zhang (1994b) and Sethi, Zhang, and Zhou (1994).
One of the main weaknesses of the early version of Hiercsim for the purpose of this review was its inability to deal with internal storage, see also Violette and Gershwin (1991). Bai (1991) and Bai and Gershwin (1990) developed a hierarchical scheme based on partitioning machines in the original flowshop or jobshop into a number of virtual machines each devoted to single part type production. Violette (1993) developed a modified version of Hiercsim to incorporate the method of Bai and Gershwin (1990). Violette and Gershwin (1991) perform a simulation study indicating that the modified method is efficient and effective. We shall not review it further, since the procedure based on partitioning of machines is unlikely to be asymptotically optimal.
As we indicated in Section 3.2, Sethi, and Zhou (1996b) have constructed asymptotically
optimal hierarchical controls , given in (3.16) with switching manifolds depicted in Figure 3.1, for the two-machine flowshop defined by (3.8) and (3.14). Samaratunga, Sethi, and Zhou (1997) have compared the performance of these hierarchical controls (HC) to that of optimal control (OC) and of two other existing heuristic methods known as Kanban Control (KC) and Two-Boundary Control (TBC). Like HC, KC is a two parameter policy defined as follows:
(3.33)
Note that KC is a threshold-type policy. TBC is a three-parameter policy developed by Lou and van Ryzin
(1989). Because it is much more complicated than HC or KC and because its performance is not
significantly different from HC as can be seen in Samaratunga, Sethi, and Zhou (1997), we shall not
discuss it any further in this survey.
55
In what follows, we provide the computational results obtained in [Samaratunga, Sethi, and Zhou (1997)] for the problem (3.8) and (3.14) with
Then we discuss the results.
In Table 3.1, different initial states are selected and the best parameter values are computed for these different initial states for HC and KC; note from Remark 3.6 that in general there are no parameter values that are best for all possible initial states. In the last row, the initial state (2.70, 1.59) is such that the best hedging point for HC and KC are (2.70,1.59). Table 3.2 uses the parameter values obtained in Table 3.1 in the row with the initial state (0,0).
Samaratunga, Sethi, and Zhou (1997) analyze these computational results and provide the following comparison of OC and KC.
Initial State
(X1,X2)
Control Policy
HC KC OC Cost
Cost Parameters Cost Parameters
(0,50) 771.45 (0.00,1.00) 771.45 (0.00,1.00) 770.31
(0,20) 252.72 (3.51,1.52) 253.53 (0.00,3.00) 231.38
(0,10) 150.77 (3.00,2.00) 151.85 (0.00,3.22) 101.13
(0,5) 132.08 (2.34,2.06) 132.16 (2.29,1.81) 69.11
(0,0) 132.76 (2.75,1.58) 132.76 (2.75,1.58) 66.56
(0,-5) 288.17 (3.75,1.50) 288.17 (3.75,1.50) 239.45
(0,-10) 617.27 (4.25,1.25) 617.27 (4.25,1.25) 590.67
(0,-20) 1469.54 (1.00,0.00) 1469.54 (1.00,0.00) 1466.54
(20,20) 414.78 (1.00,1.00) 414.98 (0.50,2.50) 406.96
(10,10) 194.74 (2.33,2.36) 194.74 (2.33,2.35) 165.71
56
(5,5) 136.80 (2.79,1.64) 136.8 (2.49,1.79) 84.49
(5,5) 267.87 (4.98,1.22) 267.87 (4.98,1.22) 214.46
(10,-10) 586.04 (6.41,0.72) 586.04 (6.41,0.72) 539.86
(20,-20) 1420.34 (1.00,0.00) 1420.34 (1.00,0.00) 1411.65
(2.70,1.59) 129.46 (2.70,1.59) 129.46 (2.70,1.59) 65.39
Note: Simulation Relative Error <= 2%, Confidence Level = 95%. Comparison is carried out for the same machine failure breakdown sample paths for all policies. OC is obtained from a Markov decision process formulation of the problem.
Table 3.1 Comparison of Control Policies with Best Threshold Values for Various Initial States.
Initial Inventory
(X1,X2)
Control Policy
HC Cost KC Cost OC Cost
(0,50) 771.45 794.96 770.31
(0,20) 252.78 269.12 231.38
(0,10) 150.94 156.79 101.13
(0,5) 132.31 132.31 69.11
(0,0) 132.76 132.76 66.56
(0,-5) 288.34 288.34 239.45
(0,-10) 617.85 617.85 590.67
(0,-20) 1471.18 1471.18 1466.54
(20,20) 415.03 415.03 406.96
(10,10) 194.83 194.83 165.71
(5,5) 136.82 136.82 84.49
57
(5,-5) 270.75 270.75 214.46
(10,-10) 583.85 583.85 539.86
(20,-20) 1426.58 1426.58 1411.65
Note: Simulation Relative Error <= 2%, Confidence Level = 95%. Comparison is carried out for the same machine failure breakdown sample paths. Therefore, the relative comparison is free of statistical uncertainty. Thresholds values used for HC as well as KC are (2.75,1.58) obtained from the (0,0) initial inventory row of Table 3.1.
Table 3.2 Comparison of Control Policies with Threshold Values (2.75, 1.58) for HC and KC.
HC vs. OC
In Table 3.1 and 3.2, the cost of HC is quite close to the optimal cost, if the initial state is sufficiently removed from point (0,0). Moreover, the farther the initial (x1,x2) is from point (0,0), the better the approximation HC provides to OC. This is because the hedging points are close to point (0,0), and hierarchical and optimal controls agree at points in the state space that are further from (0,0) or further from hedging points. In these cases, transients contribute a great deal to the total cost and transients of HC and OC agree in regions far away from (0,0).
HC vs. KC
Let us now compare HC and KC in detail. Of course, if the initial state is in a shortage situation (
), then HC and KC must have identical costs. This can be easily seen in Table 3.1 or Table 3.2 when initial (x1, x2) = (0, -5), (0, -10), (0, -20), (5, -5), (10, -10) and (20, -20). On the other hand, if the initial surplus is positive, cost of HC is either the same as or slightly smaller than the cost of KC, as should be expected. This is because, KC being a threshold-type policy,
the system approaches even when there is large positive surplus, implying higher inventory costs. In Table 3.1 and Table 3.2, we can see this in rows with initial (x1, x2) = (0, 5), (0,
10), (0, 20), and (20, 20). Moreover, by the same argument the values of for KC must not be larger than those for HC in Table 3.1. Indeed, in cases with large positive surplus, the value
of for KC must be smaller than that for HC. Furthermore, in these cases with positive surplus, the cost differences in Table 3.2 must be larger than those in Table 3.1, since Table 3.2 uses hedging point parameters that are best for initial (x1,x2) = (0,0). These parameters are the same for HC and KC. Thus, the system with an initial surplus has higher inventories in the
58
internal buffer with KC than with HC. Note also that if the surplus is very large, then KC in order
to achieve lower inventory costs sets = 0, with the consequence that its cost is the same as that for HC. For example, this happens when the initial (x1,x2) = (0,50) in Table 3.1. As should be expected, the difference in cost for initial (x1,x2) = (0,50) in Table 3.2 is quite large compared to the corresponding difference in Table 3.1.
Part II: AVERAGE-COST MODELS
In this part we review another important cost criterion, namely, the long-run average cost criterion.
The criterion presents a stark contrast to the discounted-cost criterion. The discounted-cost criterion considers near-term costs to be more important than costs occuring in the long term. In fact, the discounted-cost criterion all but ignores the distant future costs. The long-run cost criterion on the other hand considers only the long-term costs to be important.
In Section 4, we consider the theory of optimal control of stochastic manufacturing problems with the long-run average cost criterion. We use the vanishing discount approach for the purpose of this paper. Section 5 is concerned with hierarchical controls for long-run average cost problems.
4 Optimal Control with the Long-Run Average Cost Criterion
Beginning with Bielecki and Kumar (1988), there has been a considerable interest in studying the
problem of convex production planning in stochastic manufacturing systems with the objective of
minimizing long-run average cost. Bielecki and Kumar (1988) dealt with a single machine (with two
states: up and down), single product problem with linear holding and backlog costs. Because of the
simple structure of their problem, they were able to obtain an explicit solution for the problem, and thus
verify the optimality of the resulting policy. They showed that the so-called hedging point policy is
optimal in their simple model. It is a policy to produce at full capacity if the surplus is smaller than a
threshold level, produce nothing if the surplus is higher than the threshold level, and produce as much
as possible but no more than the demand rate when the surplus is at the threshold level.
Sharifnia (1988) dealt with an extension of the Bielecki-Kumar model with more than two machine states. Liberopoulos and Caramanis (1995) showed that Sharifnia's method for evaluating hedging point policies applies even when the transition rates of the machine states
59
depend on the production rate. Liberopoulos and Hu (1995) obtained monotonicity of the threshold levels corresponding to different machine states. Srivastsan and Dallery (1998) generalized the Bielecki-Kumar problem to allow for two products. They limited their focus to only the class of hedging point policies and attempted to partially characterize an optimal solution within that class. Bai and Gershwin (1990) and Bai (1991) use heuristic argument to obtain suboptimal controls in two-machine flowshops. In addition to nonnegative constraints on the inventory levels in the internal buffer, they also consider inventory level in this buffer to be bounded above by the size of the buffer. Moreover, Srivatsan, Bai, and Gershwin (1994) apply their results to semiconductor manufacturing (jobshop). All of these papers, however, are heuristic in nature, since they do not rigorously prove the optimality of the policies for their extensions of the Bielecki-Kumar model.
Presman, Sethi, Zhang, and Zhang (1998a) extend the hedging point policy to the problem of two part types by using the potential function related to the dynamic programming equation of the two part type problem. Using a verification theorem for the two part type problem, they prove the optimality of the hedging point policy. Furthermore, by the Bielecki and Kumar result, Sethi and Zhang (1999) prove the optimality of the hedging point policy for a general n-product problem under a special class of cost functions. For the deterministic model, Sethi, Zhang and Zhang (1996) provide optimal control policies and the optimal value.
The difficulty in proving the optimality for cases rather than these special cases lies in the fact that when the problem is generalized to include convex cost and multiple machine capacity levels, explicit solutions are no longer possible. One then needs to develop appropriate dynamic programming equations, existence of their solutions, and verification theorems for optimality. In this section, we will review some works related to this.
4.1 Optimal control of single or parallel machine systems
We consider an n-product manufacturing system given in Section 2.1. For any , define
(4.1)
where is the surplus process corresponding to the production process with
, and and are given in Section 2.1. Our goal is to choose so
as to minimize the cost function .
60
Except Assumption 2.1 in Section 2.1 on the production cost functions , we assume the
production capacity process and the surplus cost function to satisfy the following:
Assumption 4.1 is a nonnegative convex function with h(0)=0. There are positive
constants C41, C42, and such that
Moreover, there are constants C43 and such that
Assumption 4.2 m(t) is a finite state Markov chain with generator Q, where Q=(qij),
is a (p+1) x (p+1) matrix such that for and . We assume that
Q is weakly irreducible. Let be the equilibrium distribution vector of m(t).
Assumption 4.3 The average capacity .A control is called stable if the condition
(4.2)
holds, where is the surplus process corresponding to the control with
and is defined in Assumption 4.1. Let denote
the class of stable controls.
We will show that there exists a constant , independent of the initial condition
, and a stable Markov control policy such that
61
is optimal, i.e., it minimizes the cost defined by (4.1) over all , and furthermore,
(4.3)
where is the surplus process corresponding to with .
Moreover, for any other (stable) control ,
(4.4)
Since we use the vanishing discount approach to treat our problem, we provide the required results for
the discounted problem. First we introduce a corresponding control problem with the cost discounted at
a rate . For , we define the expected discounted cost as
The value function of the discounted problem is defined as
(4.5)
In order to study the long-run average cost control problem using the vanishing discount approach, we
must first obtain some estimates for the value function . To do this, we give the following
auxiliary lemma. Its proof is given in Sethi, Suo, Taksar and Zhang (1997).
Lemma 4.1 For any , , there exist a constantC43 and a control
policy , such that for ,
62
where
and , is the surplus process corresponding to the control policy and initial condition
.
This lemma leads to the following result proved in Sethi, Suo, Taksar and Zhang (1997).
Theorem 4.1 For any , there exist a constant C44 and a control
policy such that for ,
where
and is the surplus process corresponding to the control policy and initial condition
.
With Theorem 4.1 in hand, Sethi, Suo, Taksar and Zhang (1997) prove the following theorem.
Theorem 4.2 (i) There exists a constant such that is bounded. (ii) The function
(4.6)
is convex in . It is locally uniformly bounded, i.e., there exists a constant C 45>0 such that
63
for all , .
(iii) is locally uniformly Lipschitz continuous in , with respect to , i.e., for
any X>0, there exists a constantC46>0, independent of , such that
for all and all .
The HJB equation associated with the long-run average cost optimal control problem as formulated above takes the following form
(4.7)
where is a constant, is a real-valued function, known as the potential function or the
relative value function, defined on , is the partial derivative of the relative
value function with respect to the state variable , and denotes the inner product
Without requiring that is C1, it is convenient to write the HJBDD for our problem. The
corresponding HJBDD equation can be written as
(4.8)
Let denote the family of real-valued functions defined on such that
(i)
is convex;
(ii)
64
has polynomial growth, i.e., there are constants and C47>0 such that
A solution to the HJB or HJBDD equation is a pair with a constant and .
The function is called the potential function for the control problem, if is the minimum long-
run average cost. From Theorem 4.2 follows the next result.
Theorem 4.3 For , the following limits exist:
(4.9)
Furthermore, is a viscosity solution to the HJB equation (4.7).
Using results from convex analysis, Sethi, Suo, Taksar and Zhang (1997) prove the following theorem.
Theorem 4.4 defined in Theorem 4.3 is a solution to the HJBDD equation (4.8).
Remark 4.1 When there is no cost of production, i.e., , Veatch and Caramanis (1997) introduce the following differential cost function
where m=m(0), is the optimal value, and is the surplus process corresponding to the
optimal production process with . The differential cost function is used in the
algorithms to compute a reasonable control policy using infinitesimal perturbation analysis or direct
computation of average cost; see Caramanis and Liberopoulos (1992), and Liberopoulos and Caramanis
65
(1995). They prove that the differential cost function is convex and differentiable in . If
n=1, h(x1)=|x1| and , we know from Bielecki and Kumar (1988) that
(4.10)
This means that the differential cost function is the same as the potential function given by (4.9).
However so far, (4.10) has not been established in general. Now we state the following verification
theorem proved by Sethi, Suo, Taksar and Yan (1998).
Theorem 4.5 Let be a solution to the HJBDD equation (4.8). Then the following holds.
(i)
If there is a control such that
(4.11)
for a.e. with probability one, where is the surplus process corresponding to the
control , and
(4.12)
then
(ii)
66
For any , we have , i.e.,
(iii)
Furthermore, for any (stable) control policy , we have
(4.13)
In the remainder of this section, let us consider the single product case, i.e., n=1. For this case, Sethi,
Suo, Taksar and Zhang (1997) prove the following result.
Theorem 4.6 For andV(x,m) given in (4.9), we have
(i)
V(x,m) is continuously differentiable in x.
(ii)
is a classical solution to the HJB equation (4.7).
Let us define a control policy via the potential function as follows:
(4.14)
if the function is strictly convex, or
67
(4.15)
if c(u)=cu. Therefore, the control policy satisfies (i) of Theorem 4.5.
From the convexity of the potential function , there are
such that
and
The control policy can be written as
Then we have the following result.
Theorem 4.7 The control policy defined in (4.14) and (4.15), as the case may be, is optimal.
Proof. By Theorem 4.5, we need only to show that
But this is implied by Theorem 4.6 and the fact that is a stable control. Remark 4.2 When c(u) =0, i.e., there is no production cost in the model, the optimal control policy can be
68
chosen to be the so-called hedging point policy, which has the following form: there are real numbers xk, k=1,...,m, such that
In particular, if h(x) =c1x++c2x
- with and , we obtain the
special case of Bielecki and Kumar (1988). This will be reviewed next.
The Bielecki-Kumar Case: Bielecki and Kumar (1988) treated the special case in which
h(x)=c1x++c2x-, c(u)=0, and the production capacity is a two-state birth-death Markov
process. Thus, the binary variable takes the value one when the machine is up and zero when the machine is down. Let 1/q1 and 1/q0 represent the mean time between failures and the mean repair time, respectively. Bielecki and Kumar obtain the following explicit solution:
where
Remark 4.3 When the system equation is governed by the stochastic differential equation
(4.16)
where , are suitable function and is a standard Brownian motion, Ghosh,
Arapostathis, Marcus (1993), Ghosh, Arapostathis, and Marcus (1997) and Basak, Bisi and Ghosh (1997)
have studied the corresponding HJB equation and established the existence of their solutions and the
existence of an optimal control under certain conditions. In particular, Basak, Bisi and Ghosh (1997)
allow the matrix to be of any rank between 1 and n.
69
Remark 4.4 For n=2 and , Srivastsan and Dallery (1998) limit their focus to only the class of hedging point policies and attempt to partially characterize an optimal solution within this class.
Remark 4.5 Abbad, Bielecki, and Filar (1992) and Filar, Haurie, Moresino, and Vial (1999) consider the perturbed stochastic hybrid system whose continuous part is described by the following stochastic differential equation
where is continuous in both arguments, A is an n x n matrix, and is a Brownian motion.
The perturbation parameter is assumed to be small. They prove that when tends to zero, the optimal solution of the perturbed hybrid system can be approximated by a structured linear program.
4.2 Optimal control of dynamic flowshops
We consider the same dynamic system as the one given in Section 2.2, i.e., we consider the system
defined by (2.5). But here we do not minimize the total discounted cost. Instead our problem is to find
an admissible control that minimizes the long-run average cost function
(4.17)
where defines the cost of surplus and production given in Section 2.2, is the state
process corresponding to the control with , and is the initial value of
.
Before beginning the discussion of this problem, we give some requirements on the cost
function and the machine capacity process . In place of Assumptions 2.4 and 2.5 of Section 2.2, we impose the following assumptions on the process
throughout this section.
70
Assumption 4.4 The Markov process is strongly irreducible and has the
stationary distribution , . Let . Assume that
Remark 4.6 Assumption 4.4 is not needed in the discounted case. It is clear that this assumption is necessary for the finiteness of the long-run average cost in the case when
goes to as .As in Section 2.2, we use to denote the set
of all admissible controls with respect to and . Let denote the minimal expected cost, i.e.,
(4.18)
For writing the HJBDD equation for our problem, we first introduce some notation. Let denote the
family of real-valued functions defined on such that (i) is convex for any
; (ii) there exists a function such that for any and any ,
Consider now the following equation:
(4.19)
where is a constant, and denotes the directional derivative of function
along the direction . For the definition of the directional derivatives, see Section 2.2.
71
Presman, Sethi, and Zhang (1999a) prove the following verification theorem.
Theorem 4.8 Assume (i) with satisfies (4.19); (ii) there exists a
constant function for which
(4.2
0)
and the equation , has for any initial condition (
, a solution such that
(4.21)
Then is an optimal control. Furthermore, does not
depend on and and it coincides with . Moreover, for any T>0,
=
=
(4.22
)
Note that Theorem 4.8 explains why equation (4.19) is the Hamilton-Jacobi-Bellman equation for our
problem and why the function from Theorem 4.8 is called a potential function.
The next question is the existence of a solution to (4.19), that is, is there a pair that satisfies (4.19)? To get the existence, we use the vanishing discount approach. Consider
the corresponding control problem with the cost discounted at the rate . For
, we define the expected discounted cost as
72
Define the value function of the discounted cost problem as
Just as in Section 4.1, we need a technical lemma that states the finiteness of the rth moment of the time from any given state to any other state.
Lemma 4.2 For any and , there exists a
control policy , such that for any ,
(4.23)
where
and is the surplus process corresponding to the control policy and the
initial condition .
Because it is very tricky to construct the desired control, we give here only the outline of the
construction procedure. We begin by modifying the process in such a way that the modified average capacity of any machine is larger than the modified average capacity of the machine that follows it, and that the modified average capacity of the last machine is larger than z. Then we alternate between these two policies described below. In the first policy, the production rate at each machine is the maximum admissible modified capacity. In the second
73
policy, we stop producing at the first machine and have the maximum possible production rate at the other machines under the restriction that the content of each buffer k,
, is not less than yk. The first policy is used until such time when the content
of the first buffer exceeds the value y1 and the content of each buffer k, , exceeds the value a+yk for some a>0. At that time we switch to the second policy. We use the second policy until such time when the content of the last buffer drops to the level yN. After that we
revert to the first policy, and so on. Using this alternating procedure, it is possible to specify and provide an estimate for it. For a complete proof, see Presman, Sethi, and Zhang (1999a).
Based on this lemma, Presman, Sethi, and Zhang (1999a) give the next two theorems which are concerned with the solution of (4.19).
Theorem 4.9 There exists a sequence with as such that for
:
where .
Let be the derivative of
at the point where the derivative exists.
Theorem 4.10 In our problem, (i) does not depend on . (ii) The pair
defined in Theorem 4.9 satisfies (4.19) on So, where So is the interior of the setS.
(iii) If there exists an open subset of S such that , where is the boundary of ,
and is uniformly equi-Lipschitzian on ,
then the pair defined in Theorem 4.9 is a solution to (4.19).
74
Remark 4.7 Note that the statement (i) of Theorem 4.10 follows from Theorem 4.9. The statement (ii) of Theorem 4.10 follows from Theorem 4.9 and a simple limiting procedure.
4.3 Optimal control of dynamic jobshops
We consider the dynamic jobshop given by (2.13)-(2.15) in Section 3.3, but the Markov process
modeling the machine capacities is denoted by without parameter . Correspondingly, the
concept of the admissible control is modified as follows.
Definition 4.1 We say that a control is admissible with respect to the initial state
vector and , if
(i)
is an -adapted process with ;
(ii)
;
(iii)
the corresponding state process for all , where
(4.24)
with the initial condition
Let denote the set of all admissible controls with initial conditions , and
. Here our problem is to find an admissible control that minimizes the long-run
average cost function
75
(4.25)
where defines the cost of surplus and production and is the initial value of .
We impose the following assumptions on the process and
the cost function throughout this section.
Assumption 4.5 Let for some integer , where
. The machine capacity process is a finite state Markov chain with the following infinitesimal generator Q:
for some and any function on . Moreover, the Markov process is
strongly irreducible and has the stationary distribution , .
Assumption 4.6 Let and for
. Here pn represents the average capacity of the machine n, and n(i,j) is the number of machine placed on the arc (i,j). Assume that there exist
such that
(4.26)
76
(4.27)
and
(4.28)
Assumption 4.7 is a non-negative, jointly convex function that is strictly convex in either or
or both. For all and , there exist constants C
and such that
Let denote the minimal expected cost, i.e.,
(4.29)
In order to get the Hamilton-Jacobi-Bellman equation for our problem, similar to Section 4.2, we
introduce some notation. Let denote the family of real-valued functions defined on
such that
(i) is convex for any ;
(ii) there exists a function such that for any and any
Write (4.24) in the vector form
77
with . Consider now the following equation:
(4.30)
where is a constant, . We have the following verification theorem due to Presman,
Sethi, and Zhang (1999b).
Theorem 4.11 Assume (i) with satisfies (4.30); (ii) there exists
for which
(4.3
1)
and the equation , has for any initial condition (
, a solution such that
(4.32)
Then is an optimal control. Furthermore, does not
depend on and and it coincides with . Moreover, for any T>0,
=
78
=
(4.33
)
Next we try to construct a pair of which satisfies (4.30). To get this pair, we use the
vanishing discount approach. Consider a corresponding control problem with the cost discounted at the
rate . For , we define the expected discounted cost as
Define the value function of the discounted cost problem as
Theorem 4.12 There exists a sequence with as such that for
:
where, .
Theorem 4.13 (i) In our problem, does not depend on , and (ii) the pair
defined in Theorem4.12 is a solution to (4.19).
Remark 4.8 Assumption 4.6 is not needed in the discounted case. It is clear that this condition is
necessary for the finiteness of the long-run average cost in the case when tends to
as . Theorem 4.13 states in particular that this condition is also
79
sufficient.The proof of Theorems 4.12 and 4.13 is based on the following lemma obtained in Presman, Sethi, and Zhang (1999b).
Lemma 4.3 For any and , there exists a
control policy such that for any
(4.34)
where
and is the surplus process corresponding to the control policy and the
initial condition .
5 Hierarchical Controls under the Long-Run Average Cost Criterion
In this section, the results on hierarchical controls with the long-run average cost criterion are reviewed. Hierarchical controls for stochastic manufacturing systems including single/parallel machine system, the flowshops, and the general jobshops are discussed. For each model, the corresponding limiting problem is given, and the optimal value of the original problem is shown to converge to the optimal value of the limiting problem. Also constructed are asymptotic optimal controls for the original problem by using a near-optimal control of the limiting problem. The rate of convergence and error bounds of the constructed controls are provided.
5.1 Hierarchical controls of single or parallel machine systems under symmetric
deterioration and cancelation rates
Let us consider a manufacturing system whose system dynamics satisfy the differential equation
(5.1)
80
where with ai>0 and . The
attrition rate ai represents the deterioration rate of the inventory of the finished product type i when
, and it represents a rate of cancelation of backlogged orders when . We
assume symmetric deterioration and cancelation rates for product i only for convenience in exposition.
It is easy to extend our results when a+i>0 denotes the deterioration rate and a-
i>0 denotes the order
cancelation rate.
Let , , denote a Markov process generated by
, where is a small parameter and , is an
(m+1) x (m+1) matrix such that for and for . We
let represent the machine capacity state at time t.
Definition 5.1 A production control process is admissible, if
(i)
;
(ii)
, k=1,2,...,n, for all .
We denote by the set of all admissible controls with the initial condition .
Definition 5.2 A function defined on is called an admissible feedback control or simply a feedback control, if
(i)
for any given initial surplus and production capacity, the equation
81
has a unique solution;
(ii)
the control defined by
.
With a slight abuse of notation, we simply call , a feedback control when no ambiguity
arises. For any , define the expected long-run average cost
(5.2)
where is the surplus process corresponding to the production process in
with , and and are given as in Section 2. The problem is to obtain
that minimizes . We formally summarize our control
problem as follows:
Here we assume that the production cost function and the surplus cost function
satisfy Assumption 3.1, and the machine capacity process satisfies Assumptions 3.2 and 3.3. Furthermore, similar to Assumption 4.3, we also assume
82
Assumption 5.1 .
As in Fleming and Zhang (1998), the positive attrition rate implies a uniform bound for
. In view of the fact that the control is bounded between 0 and m, this implies
that any solution to (5.1) must satisfy
=
(5.3)
Thus under the positive deterioration/cancelation rate, the surplus process remains bounded.
The average cost optimality equation associated with the average-cost optimal control problem
in , as shown in Sethi, Zhang, and Zhang (1997), takes the form
=
(5.4)
where is the potential function of the problem . The analysis begins with the
proof of the boundedness of . Sethi, Zhang, and Zhang (1997) prove
Theorem 5.1 The minimum average expected cost of is bounded in , i.e., there exists a constant C>0 such that
83
In order to construct open-loop and feedback hierarchical controls for the system, one derives
the limiting control problem as . As in Sethi, Zhang, and Zhou (1994), consider the enlarged control space
Then define the limiting control problem as follows:
The average-cost optimality equation associated with the limiting control problem is
(5.5
)
where is a potential function for . From Sethi, Zhang, and Zhang (1997), we know
that there exist and such that (5.5) holds. Moreover, is the limit of
as .
Armed with Theorem 5.1, one can derive the convergence of the minimum average expected
cost as goes to zero, and establish the convergence rate.
Theorem 5.2 There exists a constant C such that for all ,
84
This implies in particular that .
Here we only outline the major steps in the proof. For the detailed proof, the reader is referred
to Sethi, Zhang, and Zhang (1997). The first step is to prove by constructing
an admissible control of from the optimal control of the limiting problem , and by estimating the difference between the state trajectories corresponding to these two
controls. Then one establishes the opposite inequality, namely, , by
constructing a control of the limiting problem from a near-optimal control of and then using Assumptions 2.1.
The following theorem concerning open-loop controls is proved in Sethi, Zhang, and Zhang (1997).
Theorem 5.3 (Open-loop control) Let be an
optimal control for , and let
Then , and is asymptotically optimal for , i.e.,
for some positive constant C.
We next consider feedback controls. We begin with an optimal feedback control for , which with a slight abuse of notation is denoted as
. This is obtained by minimizing the right-hand side of (5.5), i.e.,
85
We then construct the control
(5.6)
which is clearly feasible (satisfies the control constraints) for . Furthermore, if each is
locally Lipschitz in , then the system
(5.7)
has a unique solution, and therefore , is also an admissible feedback
control for . According to Lemma J.10 of Sethi and Zhang (1994a), there exists an such that
is weakly irreducible for . Let denote the
equilibrium distribution of , i.e.,
(5.8)
Sethi, Zhang, and Zhang (1997) prove the following result, but only in the single product case.
Theorem 5.4 (Feedback control) Let n=1. Assume (A.2.1) and (A.9.3) and that the feedback
control of the limiting problem is locally Lipschitz in x. Furthermore, suppose that for each
, the equation
86
(5.9)
has a unique solution , called the threshold. Moreover, suppose that for ,
(5.10)
and for
(5.11)
where . Then the feedback control given in (5.6) is asymptotically optimal, i.e.,
where .
Since there are several hypotheses, namely (5.9)-(5.11), in Theorem 5.4, it is important to provide at least an example for which these hypotheses hold. Below we provide such an example and at the same time illustrate the ideas of constructing the asymptotically optimal controls.
Example 5.1. Consider the problem
87
with and the generator for to be
This is clearly a special case of the problem formulated in this section. In particular,
Assumptions 3.3 and 3.4 hold and . The limiting problem is
where we use . Let us set the function
(5.12)
to be a feedback control for . Clearly the cost associated with (5.12) is zero. Since zero is the lowest
possible cost, our solution is optimal and . Furthermore, since is locally Lipschitz in x and
satisfies hypotheses (5.9)-(5.11), Theorem 5.4 implies that
is an asymptotically optimal feedback control for .
Remark 5.1 It is possible to obtain other optimal feedback controls for Example 5.1. It is also possible to provide examples with nonzero production cost, for which Lipschitz feedback controls satisfying (5.9)-(5.11) can be obtained, and their optimality asserted by a verification theorem similar to Theorem 4.3 in Sethi, Suo, Taksar, and Yan (1998).
88
Remark 5.2 A similar averaging approach is introduced in Altman and Gaitsgory (1993) and Altman and Gaitsgory (1997), Nguyen and Gaitsgory (1997), Shi, Altman and Gaitsgory (1998), Nguyen (1999) and references there in. They consider a class of nonlinear hybrid systems in which the parameters of the dynamics of the system may jump at discrete moments of time, according to a controlled Markov chain with finite states and action spaces. They assume that the unit of the length of intervals between the jumps is small. They prove that the optimal solution of the hybrid systems governed by the controlled Markov chain can be approximated by the solution of a limiting deterministic optimal control problem.
5.2 Hierarchical control of single or parallel machine systems without deterioration
and cancelation rates
Consider a manufacturing system whose system dynamics is given by (5.1) with ak=0, k=1,...,n, that is,
(5.13)
We assume that the machine capacity process is a finite-state Markov process given in Section
3.1. Now we define the set of admissible controls as follows:
Definition 5.3 We say that a control is admissible, if
(i)
is an adapted measurable process;
(ii)
and .
We use to denote the set of all admissible controls with the initial condition .
Definition 5.4 A function defined on is called an admissible feedback control, or simply a feedback control, if
(i)
for any given initial surplus and production capacity i, the equation
89
has a unique solution;
(ii)
the control defined by .
With a slight abuse of notation, we simply call a feedback control when no ambiguity arises.
For any , define
where is the surplus process corresponding to the production rate with
. Our goal is to choose so as to minimize the cost
. We formally summarize the control problem as follows:
(5.14
)
We make the same assumptions on the surplus cost function and the production cost function
, and the Markov chain as in Section 5.1. Without a positive inventory
deterioration/cancelation rate for each product type, the system state may no longer remain
bounded. Thus the approach presented in Section 5.1 cannot be used for the system given by (5.13).
Here we will use the vanishing discount approach to treat the problem. In order to derive the dynamic
programming equation for the average cost control problem formulated above and carry out an
asymptotic analysis of the minimum long-run average expected cost for (5.14), we introduce a related
90
control problem with the cost discounted at a rate . For , let
denote the cost function, i.e.,
where satisfies (5.13), i.e., the surplus process corresponding to the production rate
. Then, our related control problem can be written as (3.3) in Section 3.1.
In order to study the long-run average cost control problem by using the vanishing discount
approach, we must first obtain some estimates for the value function (
). Sethi and Zhang (1998) prove the following result.
Theorem 5.5 There exist constants and such that
(i)
is bounded;
(ii)
For and , the function
is convex in ;
(iii)
91
For and , is locally Lipschitz continuous in x, i.e., there
exists a constant C independent of and , such that
for and all , where is given in Assumption 3.2.
Corresponding to the dynamic programming equation (3.4) associated with the discounted cost case, we
can formally write the dynamic programming equation associated with the long-run average cost case as
(5.15
)
for any and , where is a constant and is a real-valued function on
.
Lemma 5.1 The dynamic programming equation (5.15) has a unique solution in the following
sense: If and are viscosity solutions to (5.15), then
.
For the proof, see Theorem G.1 in Sethi and Zhang (1994a).
Remark 5.3 While the proof of Theorem G.1 in Sethi and Zhang (1994a) works for the
uniqueness of , it cannot be adapted to prove whether and are
equal. Clearly, if is a viscosity solution to (5.15), then for any constant C,
is also a viscosity solution to (5.15). However, we do not need
92
to be unique for the purpose of this paper.According to (i) and (iii) of Theorem 5.5,
for a fixed and any subsequence of , there is a further subsequence denoted
by , such that
(5.16)
and
=
(5.17)
for all . Furthermore, we have the following lemma proved in Sethi and Zhang
(1998).
Lemma 5.2 There exists a constant such that for
(i)
given in (5.17) is convex and is locally Lipschitz continuous in , i.e., there is
constant C such that
for all and ;
(ii)
is a viscosity solution to the dynamic programming equation (5.15);
(iii)
93
there exists a constant C such that for all and ,
From (ii) of Lemma 5.2, for each subsequence of , denoted by , the limit of
with some function is a viscosity solution to the dynamic programming
equation (5.15). With the help of Lemma 5.1, we get that , the limit of does not
depend on the choice of the subsequence . Thus, we have the following lemma.
Lemma 5.3 converges to , as .
Multiplying both sides of (5.15) at i by , i=0,1,...,p defined in Assumption 2.3 and summing over i, we have
keeping in mind that . Then we let to obtain
(5.18)
This procedure will be justified in the next theorem. Note that
94
where is taken by in . Thus, (5.18)
is also equivalent to
(5.19)
which is the dynamic programming equation for the following deterministic problem:
(5.20
)
where
With the help of (i) of Theorem 5.5, (i) of Lemma 5.2 and Lemma 5.3, Sethi and Zhang (1998) derive the
convergence of the minimum average expected cost as goes to zero.
Theorem 5.6 , the minimum average cost of the problem defined in (5.14), converges to
, the minimum average cost of the limiting control problem defined in (5.20), i.e.,
5.3 Hierarchical controls of dynamic flowshops
We are given a stochastic process on the standard
probability space , where is the capacity of the kth machine at
time t, and is a small parameter to be specified later. We use to denote the input rate to the
95
kth machine, k=1,...,N, and to denote the number of parts in the buffer between the kth and
(k+1)th machines, k=1,...,N-1. We assume a constant demand rate z. The difference between cumulative
production and cumulative demand, called surplus, is denoted by . If , we have
finished good inventories, and if , we have a backlog.
The dynamics of the system can then be written as follows:
(5.21)
where and ak>0 are constants. The attrition rate ak represents the deterioration rate of
the inventory of the part type k when (k=1,...,N-1), and it represents a rate of cancelation
of backlogged orders for finished goods when . We assume symmetric deterioration and
cancelation rates for finished good N only for convenience in exposition. It would be easy to extend our
results if a+N>0 had denote the deterioration rate and a-
N>0 had denoted the order cancelation rate.
Equation (5.21) can be written in the following vector form:
(5.22)
where A and B are given in Section 2.2, and . Since the number of parts in the
internal buffers cannot be negative, we impose the state constraints , k=1,..., N-1. To
formulate the problem precisely, let denote the state
constraint domain, for , let
(5.23)
and for let
96
=
(5.24)
Let the sigma algebra . We now define the concept of
admissible controls.
Definition 5.5 We say that a control is admissible with
respect to the initial state vector if
(i)
is an -adapted measurable process;
(ii)
for all ;
(iii)
the corresponding state process for all .
The problem is to find an admissible control that minimizes the cost function
(5.25)
where defin[Aes the cost of inventory/shortage, is the production cost, and is the
initial value of . We impose the following assumptions on the random process
and the cost function and throughout this
section.
97
Assumption 5.2 Let for some given integer , where
, withmjk, k=1,...,N, denoting the capacity of the kth machine, j=1,...,p.
The capacity process is a finite state Markov chain with the infinitesimal
generator , whereQ(1)=(qij(1)) and Q(2)=(qij
(2)) are matrices such that
if , and for r=1,2. Moreover, Q(2) is irreducible and, without any loss of generality, it is taken to be the one that satisfies
Assumption 5.3 Assume that Q(2) is weakly irreducible. Let denote the
equilibrium distribution of Q(2). That is, is the only nonnegative solution to the equation
(5.26)
Furthermore, we assume that
(5.27)
Assumption 5.4 and are non-negative convex function. For all and
, j=1,...,p, there exist constants C and such that
and
98
We use to denote the set of all admissible controls with respect to and
. Let denote the minimal expected cost, i.e.,
(5.28)
We know, by Theorem 2.4 in [], that under Assumption 5.3, is independent of the
initial condition . Thus we will use instead of . We use to denote
our control problem, i.e.,
(5.29
)
As in Fleming and Zhang (1998), the positive attrition rate implies a uniform bound for .
Next we examine elementary properties of the relative cost known also as the potential function and
obtain the limiting control problem as . The HJBDD equation associated with the average-cost
optimal control problem in , as shown in Sethi, Zhang, and Zhang (1998), takes the form
(5.30)
99
where is the potential function of the problem ,
denotes the directional derivative of along the direction
, and for any
function on . Moreover, following Presman, Sethi, and Zhang (1999b), we can show that there
exists a potential function such that the pair is a solution of (5.30),
where is the minimum average expected cost for .
The analysis of the problem begins with the boundedness of proved in Sethi, Zhang, and Zhang (1999a).
Theorem 5.7 The minimum average expected cost of is bounded in , i.e., there exists a constant M1 >0 such that
Next we derive the limiting control problem as . Intuitively, as the rates of the machine
breakdown and repair approach infinity, the problem , which is termed the original problem, can be approximated by a simpler problem called the limiting problem, where the
stochastic machine capacity process is replaced by a weighted form. The limiting problem, which was first introduced in Sethi, Zhang, and Zhou (1994), is formulated as follows.
As in Sethi and Zhang (1994c), we consider the enlarged control space
such that , for all , j=1,...,p, and k=1,...,N, and the corresponding solution of the system
100
satisfy for all . Let represent the set of all these controls with
. The objective of this problem is to choose a control that minimizes
We use to denote the above problem, and will regard this as our limiting problem. Then we
define the limiting control problem as follows:
The average cost optimality equation associated with the limiting control problem is
(5.31)
where is a potential function for and is the directional
derivative of along the direction with
101
. From Presman, Sethi, and Zhang (1999a), we know that there exist and
such that (5.31) holds. Moreover, is the limit of as .
Hierarchical controls are based on the convergence of the minimum average expected cost as
goes to zero. Thus we will consider the convergence, as well as the rate of convergence. To do this, we
first give without proof the following lemma similar to Lemma C.3 of Sethi and Zhang (1994a)].
Lemma 5.4 Let
Then, for any bounded deterministic measurable process , , and , which is a
Markov time with respect to , there exists positive constants C and such that
for all and sufficiently small .
In order to get the required convergence result, we need the following auxiliary result, which is the key to obtaining the convergence result.
Lemma 5.5 For and any sufficiently small , there exist C>0,
and
such that for each j=1,...,N
102
(5.32)
and
(5.33)
where is the trajectory under
For the proof, see Sethi, Zhang, and Zhang (1999a). With the help of Lemma 5.5, Sethi, Zhang, and Zhang (1999a) give the following lemma.
Lemma 5.6 For , there exist , , , and
such that
(5.34)
and
(5.35)
where is the state trajectory under the control .
With Lemmas 5.4, 5.5 and 5.6, we can state the main result of this section proved in Sethi, Zhang, and Zhang (1999a).
103
Theorem 5.8 For any , there exists a constant such that for all sufficiently
small ,
(5.36)
This implies in particular that
Finally we give the procedure to construct an asymptotic optimal control.
Construction of an Asymptotic Optimal Control
Step I: Pick an -optimal control for , i.e.,
Let
Furthermore, let
and
Define
104
Then we get the control
This step can be called partial pathwise lifting.
Step II: Define
Then we get the control
This step can be called pathwise shrinking.
Step III: We choose
such that
105
This step can be called entire pathwise lifting.
Step IV: Set
and
Set
Sub-step n(n=2,...,N): Set
Then we get .
5.4 Hierarchical controls of dynamic jobshops
We consider the jobshop given in Section 3.3. The dynamics of the system are given by
We write this in the vector form as
106
=
=
(5.37)
where , and
and
Our problem is to find an admissible control that minimizes the cost function
(5.38)
where defines the cost of inventory/shortage, is the production cost, is the initial state,
and is the initial value of .
In place of Assumptions 3.4, 3.5 and 3.6 in Section 3.3 on the cost function and and
the machine capacity process , we impose the following assumptions on the random
process throughout this section.
Assumption 5.5 Let , and for
, that is, pn is the average capacity of the machine n, and n(i,j) is the number of the machine located on the arc (i,j). Furthermore, we assume that there exist
such that
107
(5.39)
(5.40)
and
(5.41)
We use to denote the set of all admissible controls with respect to and
. Let denote the minimal expected cost, i.e.,
(5.42)
In the case of the long-run average cost criterion used here, we know, by Theorem 2.4 in
Presman, Sethi, and Zhang (1999b), that under Assumption 5.5, is independent of
the initial condition . Thus we will use instead of . We use to denote our control problem, i.e.,
(5.43
)
As in Section 5.1, the positive attrition rate implies a uniform bound for .
Next we examine elementary properties of the potential function and obtain the limiting
control problem as .
108
The Hamilton-Jacobi-Bellman equation in the directional derivative sense with the average-cost
optimal control problem in , as shown in Sethi, Zhang, and Zhang (1999b), takes the form
=
(5.44)
where is the potential function of the problem , denotes
the directional derivative of along the direction , and
for any function on . Moreover, following
Presman, Sethi, and Zhang (1999b), we can show that there exists a potential function
such that the pair is a solution of (5.44), where is the minimum average
expected cost for .
First we can get the boundedness of .
Theorem 5.9 The minimum average expected cost of is bounded in , i.e., there exists a constant M1>0 such that
For the proof, see Sethi, Zhang, and Zhang (1999b).
Now we derive the limiting control problem as . As in Sethi and Zhou (1994), we give the following definition.
Definition 5.6 For , let denote the set of the following measurable controls
109
with
such that for all , j=1,...,p, and n=1,...,nc, and the
corresponding solutions of the following system
with (x1(0),...,xN(0))=(x 1,...,xN) satisfy for all .The objective of this problem is to
choose a control that minimizes
We use to denote the above problem, and will regard this as our limiting problem. Then we
define the limiting control problem as follows:
The average cost optimality equation associated with the limiting control problem is
110
(5.45)
where is a potential function for and is the directional
derivative of along the direction with .
From Presman, Sethi, and Zhang (1999b), we know that there exist and such that
(5.45) holds. Moreover, is the limit of as .
Now we are ready to discuss the convergence of the minimum average expected cost as goes to zero, and establish the corresponding convergence rate. First we give two lemmas which are used in proving the convergence.
Lemma 5.7 For and any sufficiently small , there exist ,
, and
such that for each n=1,...,nc and j=1,...,p,
(5.46)
and
(5.47)
111
where is the trajectory under
Lemma 5.8 For , there exist , and , and
such that
(5.48)
and
(5.49)
where is the state trajectory under the control .For the proof of Lemmas 5.7 and 5.8, see Sethi, Zhang, and Zhang (1999b).
Using these two lemmas, Sethi, Zhang, and Zhang (1999b) derive the following theorem.
Theorem 5.10 For any there exists a constant such that for all sufficiently
small
(5.50)
This implies in particular that
112
Remark 5.4 The theorem says that the problem is indeed a limiting problem in the sense
that the of converges to of . Moreover, it gives the corresponding convergence rate.
5.5 Markov decision processes with weak and strong
interaction
Markovian decision processes (MDP) have received much attention in the recent years because of their
capability in dealing with a large class of practical problems under uncertainty. The formulation of many
practical problems such as stock-option, allocation, queuing, and machine replacement, fits well in the
framework of Markov decision processes, see Derman (1970). In this section we present results that
provide a justification for hierarchical controls of a class of Markov decision problems. We focus on the
problem of a finite state continuous-time Markov decision process that has both weak and strong
interactions. More specifically, consider the model in which the state of the process can be divided into
several groups such that transitions among the states within each group occur much more frequently
than the transitions among the states belonging to different groups. By replacing the states in each
group by the corresponding average distribution, we can derive a limiting problem which is simpler to
solve. Given an optimal solution to limiting problem, we can construct a solution for the original
problem which is asymptotically optimal.
Let us consider a Markov decision process and a control process
such that , , where U is a set with finite elements.
Let , , denote the generator of such that
(5.51)
where ,Ak(u)=(akij(u))mk x mk with for and
,B(u)=(bij(u))m x m with , for and
, and is a small parameter. For each , let
113
, where mk is the dimension of Ak(u). The state space of is given by
, i.e.,
Example 5.2. Let us consider a failure-prone manufacturing system that consists of a two-machine flowshop. Each of the two machine has two states up, denoted by 1, and down, denoted by 0. Then, the system has four states, represented by
. Let s11=(1,1), s12=(0,1), s21=(1,0), and s22=(0,0). We consider u to be the rate of preventive maintenance, which is used to reduce the failure rate of machines and to improve the productivity of the system. The objective of the problem is to choose u to keep the average machine capacity at a reasonable level and, in the meantime, to avoid excessive costly preventive maintenance.
We consider the system in which the state of the first machine is changing more rapidly than that of the second one. A typical way of modeling the machine state process is to formulate the process (see Jiang and Sethi (1991), Sethi and Zhang (1994a), and Yin and Zhang (1997) ) as a Markov chain with the following generator:
The breakdown rate and repair rate for the first machine are and and
and for the second machine. The magnitude of represents the frequency of transitions of
between s11 and s12 or s21 and s22. For small , the transition of within either {s11,s12}
or{s21,s22} is much more frequently than that between the two groups {s11,s12} and {s21,s22}.
114
The matrix A(u) dictates the fast transition of the process within each group ,
and the matrix B(u) together with the average distribution of the Markov chain
generated by Ak(u) dictates the slow transition of between different groups. When is
small, the process has a strong interaction within any group , and has a weak interaction among these groups.
Let u=u(i) denote a function such that for all . We call u an admissible
control and use to denote all such functions. We make the following assumptions:
Assumption 5.6 For each , is irreducible, , and for each
, is irreducible, where is defined in (49).
Assumption 5.7 U is a finite set.
We consider the following problem:
(5.52)
Note that the average cost function is independent of the initial value x(0)=i, and so is the value
function. The DP equation for is
(5.53)
where is a function defined on .
Theorem 5.11 (i) For each fixed , there exists a pair that satisfies the DP equation
(5.53).
115
(ii) The DP equation (5.53) has a unique solution in the sense that for any other solution to
(5.53), and , for some constant K.
(iii) Let denote a minimizer of the right-hand side of (5.53). Then is
optimal, i.e., .
To proceed, we define the control set for the limiting problem. Motivated by Sethi and Zhang
(1994a), we consider a control set in which each control variable corresponds to a state in .
To be more precise, for each , let
The control set for the limiting problem is defined as , i.e.,
We define matrices , , and as follows. For each , let
, , where
with , and
(5.54)
116
For each , let denote the average distribution of the Markov chain generated by
for . Define and
with being mk dimensional column vector.
Using and Ie, we define another matrix as a function of :
(5.55)
Note that the kth row of depends only on Uk. Thus, . We write, with
a little abuse of notation, instead of
, for a function f defined on .
We next define the limiting problem. For , define
where is the average distribution of the Markov chain
generated by Ak(Uk). Let denote a class of functions , . For
convenience, we will call an admissible control for the limiting
problem termed .
We define the limiting problem with the long-run average cost.
117
(5.56)
It can be shown that is irreducible for each .
The DP equation for is
(5.57)
for some function h0(k).
Theorem 5.12 (i) There exists a pair that satisfies the DP equation (5.57). (ii) The DP equation (5.57) has a unique solution in the sense that for any other solution
to (5.57), and , for some constant K.
(iii) Let denote a minimizer of the right-hand side of (5.57). Then is optimal,
i.e., .
Remark 5.5 Note that the number of the DP equations for is equal to
, while the number of that for is only r. Since for each
, , it follows that . The difference between m and r could be very large for either large r or a large mk for some k. Since the computational effort involved in solving the DP equations depends largely on the number of the equations to be solved (cf.
Hillier and Lieberman (1989)), the effort in solving the DP equations for is substantially
less than that of solving for m-r large.
We construct a control as follows:
118
(5.58)
where IA is the indicator of a set A.
Theorem 5.13 Let denote an optimal control for and let be the
corresponding control constructed as in (5.58). Then, is asymptotically optimal and with error
bound , i.e.,
Remark 5.6 Filar, Haurie, Moresino, and Vial (1999) introduce diffusions in these models; see Remark 4.5. In their model, all jumps are associated with a slow process, while the continuous
part including the diffusions is associated with a small parameter (see Remark 4.5)
representing a fast process. As tends to zero, the problem is shown to reduce to a structured linear program. Its optimal solution can then be used as an approximation to the optimal
solution of the original system associated with a small .
Part III: RISK-SENSITIVE COST MODELS
In this part we consider robust production plans with risk sensitive cost criteria. This consideration is
motivated by the following observations. First, since most manufacturing systems are large and
complex, it is difficult to establish accurate mathematical models to describe these systems. Modeling
errors are inevitable. Second, in practice, an optimal policy for a subdivision of a big corporation is
usually not an optimal policy for the whole corporation. Optimal solutions with the usual cost criteria
may not be desirable in many real situations. An alternative approach is to consider robust controls. In
some manufacturing systems, it is more desirable to consider controls that are robust enough to
attenuate uncertain disturbances, such as modeling errors, and therefore achieve the system stability.
Robust control design is particularly important in manufacturing systems with unfavorable disturbances.
There are two kinds of system disturbances in the systems under consideration: (1) unfavorable internal
disturbances - usually associated with unfavorable machine capacity fluctuations; (2) unfavorable
external disturbances such as fluctuations in demand.
119
6 Risk-Sensitive Hierarchical Controls
The basic idea of the risk-sensitive control is to consider a risk-sensitive cost function that penalizes
heavily on costs associated with large values of state and control variables. Typically an exponential-of-
integral cost criterion is considered. Such cost functions penalize heavily, state trajectories and controls
which give large values to the exponent. Related literature on risk-sensitive and robust controls can be
found in Whittle (1990), Fleming and McEneaney (1995), Barron and Jensen (1989), and references
therein. For details of models discussed in this section, see Zhang (1995).
As the rate of fluctuation of the production capacity process goes to infinity, we show that the risk-sensitive control problem can be approximated by a limiting problem in which the stochastic capacity process can be averaged out and replaced by its average. We also show that the value function of the limiting problem satisfies the Isaacs equation of a zero-sum, two-player differential game. Then, we use a near optimal control of the limiting problem to construct a nearly optimal control for the original risk-sensitive control problem.
In Section 6.1 we consider risk-sensitive hierarchical controls with discounted costs. In Section 6.2 we study risk-sensitive hierarchical controls with average costs.
6.1 Risk-sensitive hierarchical controls with discounted costs
Consider the production system whose dynamic is given by
Let denote the risk-sensitive discounted cost function defined by
The problem is to find an admissible control that minimizes .
We now specify the production constraints. For each , let
120
With this definition, the production constraint at time t is
We assume the demand rate z(t) to be a bounded process independent of .
We say that a control is admissible if u(t) is a
-adapted measurable process and for all . Then our control problem can be written as follows:
Let . We consider the following control space:
and two control problems and defined as follows:
121
and
It can be seen below that when is small, can be approximated by and
can be approximated further by . Therefore, can be approximated by . Then,
a near optimal control for will be used to construct controls for that are nearly optimal.
Theorem 6.1 There exist constants and C such that, for ,
We show that can be approximated by and the value function of is a viscosity solution to the Isaacs equation of a zero-sum, two-player differential game. To simplify
the notation, we take and consider the following control problem .
122
Theorem 6.2 is a monotone increasing function of and
For each .
We write v0,0(x) as the value function of with the initial value x0=x. Note that
for any random variable . Let
such that and let denote a
compact subset of Rn. We consider functions ( ) that are right-continuous and
have left-hand limits. Let denote the metric space of such functions that is equipped with
the Skorohod topology .
We assume a.s. and for each and any ,
.
Theorem 6.3 v0,0(x) is the only viscosity solution to the following Isaacs equation:
123
Theorem 6.4 The following assertions hold: (i)
(ii) Let denote a stochastic open loop -optimal control
for , i.e.,
Let , where 1A denotes the indicator of a set A. Then,
and
(iii) Let denote a feedback
-optimal control for , i.e., Let
124
Assume that U(z,x) is locally Lipschitz in x, i.e., for some k>0,
Then, and
6.2 Risk-sensitive hierarchical controls with long-run average costs
In this section we consider a manufacturing system with the objective of minimizing the risk-sensitive
average cost criterion over an infinite horizon. The risk-sensitive approach has been applied to the so-
called disturbance attenuation problem; see, for example, Whittle (1990), Fleming and McEneaney
(1995), and references therein.
Let us consider a single product, parallel-machine manufacturing system with stochastic
production capacity and constant demand for its production over time. For , let ,
, and z denote the surplus level (the state variable), the production rate (the control
variable), and the constant demand rate, respectively. We assume ,
, , and z a positive constant. They satisfy the differential equation
(6.1)
where a>0 is a constant, representing the deterioration rate (or spoilage rate) of the finished product.
We let represent the maximum production capacity of the system at time t, where
is given in Section 3.1. The production constraints is given by the inequalities
125
Definition 6.1 A production control process is admissible if
(i)
is -progressively measurable;
(ii)
for all .
Let denote the class of admissible controls with . Let H(x,u) denote a cost
function of surplus and production. The objective of the problem is to choose to
minimize
(6.2)
where is the surplus process corresponding to the production process . Let
. A motivation for choosing such an exponential cost criterion is
that such criteria are sensitive to large values of the exponent which occur with small probability, for
example, rare sequences of unusually many machine failures resulting in shortages ( ). We
assume that the cost function H(x,u) and the production capacity process satisfy the following:
Assumption 6.1 is continuous, bounded, and uniformly Lipschitz in x.
Remark 6.1 In manufacturing systems, the running cost function H(x,u) is usually chosen to be of the
formH(x,u)=h(x)+c(u) with piecewise linear h(x) and c(u). Note that piecewise linear functions are not
bounded as required in (A.6.1). However, this is not important, in view of the uniform bounds on
and on for the initial state in any bounded set.
Assumption 6.2 Q is irreducible in the following sense: The equations
126
have a unique solution with . The vector is
called the equilibrium distribution of the Markov chain .Formally, we can write the associated
HJB equation as follows:
(6.3)
where is the potential function, denotes the partial derivative of with
respect to x, and for a function f on . Zhang (1995)
proves the following theorem.
Theorem 6.5 (i) The HJB equation (6.3) has a viscosity solution .
(ii) The pair satisfies the following conditions:For some constant C1 independent
of ,
(a)
, and
(b)
.
(iii) Assume that to be Lipschitz continuous in x. Then,
127
This theorem implies that in the viscosity solution is unique. Furthermore, Zhang
(1995) gives the following verification theorem.
Theorem 6.6 Let be a viscosity solution to the HJB equation in (6.3). Assume
that to be Lipschitz continuous in x. Let .Suppose
that there are , , andr*(t) such that
satisfying
(6.4)
a.e. in t and w.p. 1. Then, .
We next discuss the asymptotic property of the HJB equation (6.5) as . First of all, note that this HJB equation is similar to that for an ordinary long-run average cost problem except for the term involving the exponential factor. In order to get rid of such a term, we make use of the logarithmic transformation as in Fleming and Soner (1992).
Let . Define
128
With the logarithmic transformation, we have for each ,
The supremum is obtained at .
The logarithmic transformation suggests that the HJB equation is equivalent to an Isaacs equation for a two-player zero-sum dynamic stochastic game. The Isaacs equation is given as follows:
(6.5
)
where
for ; see Evans and Souganidis (1984) and Fleming and Souganidis (1989).
We consider the limit of the problem as . In order to define a limiting problem, we first define the control sets for the limiting problem. Let
and
For each , let be such that
129
and let denote the equilibrium distribution of . The next theorem says
that is irreducible. Therefore, there exists a unique positive for each . Moreover,
depends continuously on V. It can be shown that for each , is irreducible.
Theorem 6.7 Let be a sequence such that and . Then,
(i)
w0(x,k) is independent of k, i.e.,w0(x,k)=w0(x);
(ii)
w0(x) is Lipschitz; and
(iii)
(k0,w0(x)) is a viscosity solution to the following Isaacs equation:
k0=
(6.6)
Let
Note that , where is the sup norm. Moreover, since ,
where V=1 means vi(j)=1 for all i,j. Then the equation in (6.6) is an Isaacs equation
associated with a two-player, zero-sum dynamic game with the objective
130
subject to
where and are Borel measurable functions and and for
; see Barron and Jensen (1989).
One can show that
which implies the uniqueness of .
Finally, in order to use the solution to the limiting problem to obtain a control for the original problem, a numerical scheme must be used to obtain an approximate solution. The advantage of the limiting problem is its dimensionality, which is much smaller than that of the original
problem if the number of states in is large.
Let (U*(x),V*(x)) denote a solution to the limiting problem. As in Section 3.1, it is expected that the constructed control
is nearly optimal for the original problem (6.2). See Zhang (1995) for the proof of this result.
7 Extensions and Concluding Remarks
In this paper, we have reviewed various models of manufacturing systems consisting of completely
flexible machines for which a theory of optimal control has been developed and for which hierarchical
controls that are asymptotically optimal have been constructed. We have looked at systems in which
production capacity and/or demand are random, in which state constraints may or may not be present,
and in which there are reentrant flows and multiple hierarchical decision levels. We have considered for
different criteria. These are discounted costs, long-run average costs, risk-sensitive controls with
discounted costs and risk-sensitive controls with average costs.
131
While asymptotic optimal controls have been constructed for these systems under fairly general conditions, many problems remain open. We shall now describe some of them.
All of the results on the hierarchical controls in this survey assume the costs of inventory/shortage and of production to be separable. Lehoczky, Sethi, Soner, and Taksar (1991) assume a nonseparable cost and prove that the value function of the original problem converges to the value function of the limiting problem. Controls are constructed and are only conjectured to be asymptotically optimal.
With regards to systems with state constraints such as flowshops and jobshops discussed in Section 3 and Section 5, respectively, only asymptotic optimal open-loop controls are constructed in general. Because of the absence of the Lipschitz property of the constructed feedback controls, their asymptotic optimality is much harder to establish. It has only been done in Sethi and Zhou (1996a), Sethi and Zhou (1996b), Fong and Zhou (1996), and Fong and Zhou (1997) for a two- machine flowshop with a specific cost structure defined in (3.14). Generalization of their results to jobshops and to general cost functions represents a challenging research problem.
When the Markov processes involved depend on control variables, as they do in Soner (1993), Sethi and Zhang (1994c), and Sethi and Zhang (1995a), no error bounds are available for constructed asymptotic optimal controls. Estimation of these errors and extensions of the results to Markov processes depending as well on the state variables remain open problems.
In the case of the the long-run average cost criterion, all of models dealing with hierarchical controls except Section 5.2 have constructed open-loop hierarchical controls in presence of an attrition rate. Extensions of these results to the systems without an attrition term remain open. No hierarchical feedback controls that are optimal have been constructed. In case of optimal control of these ergodic systems, verification theorems have been obtained, but the existence of optimal controls is proved only in the single product case of Section 4.1. In other cases, even the existence of optimal controls remains an open issue.
An important class of manufacturing systems consists of systems that have machines which are not completely flexible, and thus involve setup costs and/or setup times, when switching from production of one product to that of another. Such systems have been considered by Gershwin (1986), Gershwin, Caramanis, and Murray (1988), Sharifnia, Caramanis, and Gershwin (1991), Caramanis, Sharifnia, Hu, and Gershwin (1991), Connolly, Dallery, and Gershwin (1992), Hu and Caramanis (1992), and Srivatsan and Gershwin (1990). They have examined various possible heuristic policies and have carried out numerical computations and simulations. They have not studied their policies from the viewpoint of asymptotic optimality. Sethi and Zhang (1995b) have made some progress in this direction.
Based on the theoretical work on hierarchical control of stochastic manufacturing systems, Srivatsan, Bai, and Gershwin (1994) have developed a hierarchical framework and describe its experimental implementation in a semiconductor research laboratory at MIT. It is expected that
132
such research would lead to the development of real-time decision making algorithms suitable for use in actual flexible manufacturing facilities; see also Caramanis, Sharifnia, Hu, and Gershwin (1991).
Finally, while error bounds have been obtained in some cases and not others, they do not
provide information on how small , the rate of slow and fast rates, have to be for asymptotic hierarchical controls to be acceptable in practice. This issue can only be investigated computationally, and there is much work remaining to be done in this regard.
Bibliography
1
M. Abbad, T. R. Bielecki, and J. A. Filar, ``Perturbation and Stability Theory for Markov Control
Problems", IEEE Trans. Auto. Contr., Vol. 37, (1992), 1421-1425.
2
I. Adiri and A. Ben-Israel, ``An Extension and Solution of Arrow-Karlin Type Production Models by
the Pontryagin Maximum Principle", Cahiers du Center d'Etudes de Recherche Opérationnelle,
Vol. 8, (1966), 147-158.
3
R. Akella and P. R. Kumar, ``Optimal Control of Production Rate in a Failure-Prone Manufacturing
System,'' IEEE Trans. Auto. Contr., Vol. AC-31, (1986), 116-126.
4
E. Altman and V. Gaitsgory, ``Control of a Hybrid Stochastic System", Systems Control Letters,
Vol. 20, (1993), 307-314.
5
E. Altman and V. Gaitsgory, ``Asymptotic Optimization of a Nonlinear Hybrid System Governed
by a Markov Decision Process", SIAM J. Contr. Optim., Vol. 35, (1997), 2070-2085.
6
E. A. Ari and S. Axsater, `` Disaggregation under uncertainty in hierarchical production planning,"
European Journal of Operational Research, Vol. 35, (1988), 182-186.
7
K. J. Arrow, S. Karlin, and H. Scarf, Studies in Mathematical Theory of Inventory and Production,
Stanford University Press, Stanford, CA, 1958.
8
133
P. Auger, Dynamics and Thermodynamics in Hierarchically Organized Systems, Pergamon Press,
Oxford, England, 1989.
9
S. Bai, Scheduling Manufacturing Systems with Work-in-Process Inventory Control, Ph.D. Thesis,
MIT Operations Research Center, Cambridge MA, 1991.
10
S. Bai and S. B. Gershwin, ``Scheduling Manufacturing Systems with Work-in-Process Inventory,''
Proc. of the 29th IEEE Conference on Decision and Control, Honolulu, HI, (1990), 557-564.
11
E. N. Barron and R. Jensen, ``Total Risk Aversion, Stochastic Optimal Control, and Differential
Games," Appl. Math. Optim., Vol. 19, (1989), 313-327.
12
G. K. Basak, A. Bisi and M. K. Ghosh, ``Controlled Random Degenerate Diffusions Under Long-
Run Average Cost," Stochastic and Stochastic Reports, Vol. 61, (1997), 121-140.
13
S. Bensoussan, M. Crouhy, and J.-M., Proth, Mathematical Theory of Production Planning, North-
Holland, New York, NY, 1983.
14
S. Bensoussan, S. P. Sethi, R. Vickson, and N. A. Derzko, ``Stochastic Production Planning with
Production Constraints,'' SIAM J. Contr. Optim., Vol. 22, (1984), 920-935.
15
T. R. Bielecki and J. A. Filar, ``Singularly Perturbed Markov Control Problem: Limiting Average
Cost,'' Annals of Operations Research, Vol. 28, (1991), 153-168.
16
T. R. Bielecki and P. R. Kumar, ``Optimality of Zero-Inventory Policies for Unreliable
Manufacturing Systems," Operations Research, Vol. 36, (1988), 532-546.
17
A. Bisi, E. Presman, S. P. Sethi, and H. Zhang, ``Optimality in Two-Machine Flowshop with Limited
Buffer," to appear in Annals of Operations Research, 1999a.
134
18
A. Bisi, S. P. Sethi, and H. Zhang, ``Existence of an Optimal Control in a Two-Machine Flowshop
with Limited Buffer," Working Paper, School of Management, The University of Texas at Dallas,
1999b.
19
G. Bitran, E. A. Hass, and K. Matsuo, ``Production Planning of Style Goods with High Set-up Costs
and Forecast Revisions", Operations Research, Vol. 34, (1986), 226-236.
20
G. Bitran and A. Hax, ``On the Design of Hierarchical Production Planning Systems", Decision
Sciences, Vol. 8, (1977), 28-54.
21
G. Bitran and D. Tirupati, ``Hierarchical Production Planning,'' Chapter 10 in Logistics of
Production and Inventory, S. C. Graves, A. H. G. Rinnooy Kan, and P. H. Zipkin (Eds.), Vol. 4 in the
Series Handbooks in Operations Research and Management Science, North-Holland,
Amsterdam, (1993), 523-568.
22
E. K. Boukas, ``Techniques for Flow Control and Preventive Maintenance in Manufacturing
Systems", Control and Dynamic Systems, Vol. 48, (1991), 327-366.
23
E. K. Boukas and A. Haurie, ``Manufacturing Flow Control and Preventive Maintenance: A
Stochastic Control Approach", IEEE Trans. on Automatic Control, Vol. 35, (1990), 1024-1031.
24
E. K. Boukas, Q. Zhang, and Q. Zhou, ``Optimal Production and Maintenance Planning of Flexible
Manufacturing Systems", Proc. of the 32nd IEEE Conference on Decision and Control, San
Antonio, TX, Dec. 1993, 15-17.
25
E. K. Boukas, Q. Zhu, and Q. Zhang, ``A Piecewise Deterministic Markov Process Model for
Flexible Manufacturing Systems with Preventive Maintenance", J. Optimization Theory and
Applications, Vol. 82, (1994), 269-286.
26
135
P. N. D. Bukh, ``A Bibliography of Hierarchical Production Planning Techniques, Methodology,
and Applications (1974-1991)", Technical Report, Institute of Management, University of Aarhus,
Aarhus, Demark, 1992.
27
M. Caramanis and G. Liberopoulos, ``Perturbation Analysis for the Design of Flexible
Manufacturing System Flow Controllers,'' Operations Research. Vol. 40, (1992), 1107-1125.
28
M. Caramanis and A. Sharifnia, ``Optimal Manufacturing Flow Controller Design,'' Int. J. Flex.
Manuf. Syst. Vol. 3, (1991), 321-336.
29
M. Caramanis, A. Sharifnia, J. Hu, and S. B. Gershwin, ``Development of a Science Base for
Planning and Scheduling Manufacturing Systems,'' Proc. of the 1991 NSF Design and
Manufacturing Systems Conference, The University of Texas at Austin, Austin, TX, Jan. 9-11,
(1991), 27-40.
30
H. Chen and A. Mandelbaum, ``Hierarchical Modeling of Stochastic Networks, Part I: Fluid
models, Part II: Strong Approximations," in Stochastic Modeling and Analysis of Manufacturing
Systems, D. D. Yao(ed.), Springer Series in Operations Research, Springer-Verlag, New York, NY,
1994.
31
Committee on the Next Decade in Operations Research, ``Operations Research: the Next
Decade,'' Operations Research, Vol. 36, (1988), 619-637.
32
S. Y. Connolly, Y. Dallery, and S. B. Gershwin, ``A Real-time Policy for Performing Setup Changes
in a Manufacturing System,'' Proc. of the 31st IEEE-CDC, Tucson, AZ, Dec. 16-18, 1992.
33
B. Darakananda, Simulation of Manufacturing Process under a Hierarchical Control Algorithm,
M.S. Thesis, MIT Department of EECS, Cambridge, MA, 1989.
34
M. H. A. Davis, Markov Models and Optimization, Chapman & Hall, New York, 1993.
35
136
F. Delebecque and J. Quadrat, ``Optimal Control for Markov Chains Admitting Strong And Weak
Interactions,'' Automatica, Vol. 17, (1981), 281-296.
36
C. Derman, Finite State Markovian Decision Processes, Academic Press, New York, NY, 1970.
37
Evans L. C. and P. E. Souganidis, ``Differential Games and Representation Formulas for Solutions
of Hamilton-Jacobi-Isaacs Equations," Indiana Univ. Math. Journal, Vol. 33, (1984), 773-797.
38
Y. Feng and H. Yan, Optimal production control in a discrete manufacturing system with
unreliable machines and random demands, Working Paper, Department of Systems Engineering
and Engineering Management, the Chinese University of Hong Kong, Hong Kong, 1999.
39
J. A. Filar, A. Haurie, F. Moresino, and J. P. Vial, ``Singularly Perturbed Hybrid Systems
Approximated by Structured Linear Programs", Proceedings of International Workshop on
Markov Processes and Controlled Markov Chains, August 22-28, 1999, Changsha, Hunan, China.
40
W. H. Fleming (Chair), ``Future Directions in Control Theory: A Mathematical Perspective,'' SIAM
Reports on Issues in the Mathematical Sciences, Philadelphia, PA, 1988.
41
W. H. Fleming and W. M. McEneaney, ``Risk-Sensitive Control on an Infinite Horizon," SIAM J.
Contr. Optim., Vol. 33 ,(1995), 1881-1921.
42
W. H. Fleming, S. P. Sethi, and H. M. Soner, ``An Optimal Stochastic Production Planning Problem
with Random Fluctuating Demand," SIAM J. Contr. Optim., Vol. 25, (1987), 1494-1502.
43
W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-
Verlag, New York, 1992.
44
W. H. Fleming and P. E. Souganidis, ``On the Existence of Value Functions of Two-Player, Zero-
Sum Stochastic Different Games," Indiana Univ. Math. Journal, Vol. 38, (1989), 293-314.
137
45
W. H. Fleming and Q. Zhang, ``Risk-Sensitive Production Planning of a Stochastic Manufacturing
System," SIAM J. Contr. Optim., Vol. 36, (1998), 1147-1170.
46
N. T. Fong and X. Y. Zhou, ``Hierarchical Production Policies in Stochastic Two-Machine Flowshop
with Finite Buffers," Journal of Optimization Theory and Applications, Vol. 89, (1996), 681-712.
47
N. T. Fong and X. Y. Zhou, ``Hierarchical Production Policies in Stochastic Two-Machine Flowshop
with Finite Buffers II: Feedback Controls," Working Paper, Department of Systems Engineering
and Engineering Management, Chinese University of Hong Kong, 1997.
48
L. F. Gelders and L. Van Wassenhove, ``Production Planning: A Review", European Journal of
Operational Research, Vol. 7, (1981), 101-110.
49
S. B. Gershwin, ``Stochastic Scheduling and Setups in a Flexible Manufacturing System,'' Proc. of
Second ORSA/TIMS Conference on Flexible Manufacturing Systems, Ann Arbor, MI, Aug., (1986),
431-442.
50
S. B. Gershwin, ``Hierarchical Flow Control: a Framework for Scheduling and Planning Discrete
Events in Manufacturing Systems,'' Proceedings of the IEEE, Special Issue on Dynamics of
Discrete Event Systems, Vol. 77, (1989), 195-209.
51
S. B. Gershwin, Manufacturing Systems Engineering, Prentice-Hall, Englewood Cliffs, NJ, 1994.
52
S. B. Gershwin, R. Akella, and Y. Choong, ``Short Term Production Scheduling of an Automated
Manufacturing Facility,'' IBM J. Res. Dev., Vol. 29, (1985), 392-400.
53
S. B. Gershwin, M. Caramanis, and P. Murray, ``Simulation Experience with a Hierarchical
Scheduling Policy for a Simple Manufacturing System,'' Proceedings of the 27th IEEE Conference
on Decision and Control, Austin, TX, Vol. 3, (1988), 1841-1849.
138
54
H. Gfrerer and G. Zäpfel, ``Hierarchical Model for Production Planning in the Case of Uncertain
Demand", Working Paper, Johannes Kepler Universität Linz, Linz, Austria, 1994.
55
M. K. Ghosh, A. Arapostathis and S. I. Marcus, ``Optimal Control of Switching Diffusions with
Applications to Flexible Manufacturing Systems," SIAM J. on Contr. Optim., Vol. 31, (1993), 1183-
1204.
56
M. K. Ghosh, A. Arapostathis, and S. I. Marcus, `` Ergodic Control of Switching Diffusions," SIAM
J. on Contr. Optim., Vol. 35, (1997), 1952-1988.
57
J. M. Harrison and M. I. Taksar, ``Instantaneous Control of Brownian Motion", Mathematics of
Operations Research, Vol. 8, (1983), 439-453.
58
J. M. Harrison, T. M. Sellke, and A. J. Taylor, ``Impulse Control of Brownian Motion",
Mathematics of Operations Research, Vol. 8, (1983), 454-466.
59
J. M. Harrison and L. M. Wein, ``Scheduling Networks of Queues: Heavy Traffic Analysis of a
Simple Open Network," Queuing Systems: Theory and Applications, Vol. 5, (1989), 265-280.
60
J. M. Harrison and L. M. Wein, ``Scheduling Networks of Queues: Heavy Traffic Analysis of a
Two-Station Closed Network," Operations Research, Vol. 38, (1990), 1052-1064.
61
R. F. Hartl and S. P. Sethi, ``Optimal Control Problems with Differential Inclusions: Sufficiency
Conditions and an Application to a Production-Inventory Model", Optimal Control Applications &
Methods, Vol. 5, (1984), 289-307.
62
P. Hartman, Ordinary Differential Equations. 2nd Edition, Birkhäuser-Verlag, Boston, MA, 1982.
63
139
A. Haurie and Ch. van Delft, ``Turnpike Properties for a Class of Piecewise Deterministic Control
Systems Arising in Manufacturing Flow Control,'' Annals of Operations Research, Vol. 29, (1991),
351-373.
64
A. C. Hax and D. Candea, Production and Inventory Management, Prentice-Hall, Englewood
Cliffs, NJ, 1984.
65
F. S. Hillier and G. L. Lieberman, Introduction to Operations Research, McGraw-Hill, New York,
NY, 1989.
66
C. C. Holt, F. Modigliani, J. F. Muth, and H. A. Simon, Planning Production, Inventories, and
Workforce, Prentice-Hall, Inc. Englewood Cliffs, NJ, 1960.
67
J. Hu and M. Caramanis, ``Near Optimal Setup Scheduling for Flexible Manufacturing Systems,''
Proc. of the Third RPI International Conference on Computer Integrated Manufacturing, Troy, NY,
May 20-22, 1992.
68
J. Jiang and S. P. Sethi, ``A State Aggregation Approach to Manufacturing Systems Having
Machines States with Weak and Strong Interactions,'' Operations Research, Vol. 39, (1991), 970-
978.
69
S. M. Johnson, ``Sequential Production Planning over Time at Minimum Cost", Management
Science, Vol. 3, (1957), 435-437.
70
R. Z. Khasminskii, G. Yin, and Q. Zhang, Constructing Asymptotic Series for Probability of Markov
Chains with Weak and Strong Interactions, Quart. Appl. Math. Vol. 38, (1997), 177-200.
71
J. G. Kimemia and S. B. Gershwin, ``An Algorithm for the Computer Control Production in
Flexible Manufacturing Systems,'' IIE Trans. Vol. 15, (1983), 353-362.
72
140
P. Koktovic, ``Application of Singular Perturbation Techniques to Control Problems,'' SIAM
Review, Vol. 26, (1984), 501-550.
73
E. Krichagina, S. Lou, S. P. Sethi, and M. I. Taksar, ``Production Control in a Failure-Prone
Manufacturing System: Diffusion Approximation and Asymptotic Optimality," The Annals of
Applied Probability, Vol. 3, (1993), 421-453.
74
E. Krichagina, S. Lou, and M. I. Taksar, ``Double Band Polling for Stochastic Manufacturing
Systems in Heavy Traffic," Mathematics of Operations Research, Vol. 19, (1994), 560-596.
75
S. Kumar and P. R. Kumar, Performance Bounds for Queuing Networks and Scheduling Policies,
IEEE Trans. Automatic Control, Vol. 39, 1994, 1600-1611.
76
H. J. Kushner and K. M. Ramachandran, ``Optimal and Approximately Optimal Control Policies
for Queues in Heavy Traffic", SIAM Journal on Control and Optimization, Vol. 27, (1989), 1293-
1381.
77
J. B. Lasserre and C. Mercé, Robust hierarchical production planning under uncertainty, Annals
of Operations Research, Vol. 26, 1990, 73-87.
78
J. Lehoczky, S. P. Sethi, H. M. Soner, and M. I. Taksar, ``An Asymptotic Analysis of Hierarchical
Control of Manufacturing Systems Under Uncertainty,'' Mathematics Operations Research, Vol.
16, (1991), 596-608.
79
G. Liberopoulos and M. Caramanis, ``Production Control of Manufacturing Systems with
Production Rate Dependent Failure Rates", IEEE Trans. Automatic Control, Vol. 38, (1993), 889-
895.
80
G. Liberopoulos and M. Caramanis, ``Dynamics and Design of a Class of Parameterized
Manufacturing Flow Controllers," IEEE Trans. Automatic Control, Vol. 40, (1995), 201-272.
81
141
G. Liberopoulos and J. Hu, ``On the Ordering of Optimal Hedging Points in a Class of
Manufacturing Flow Control Models", IEEE Trans. Automatic Control, Vol. 40, (1995), 282-286.
82
C. M. Libosvar, ``Hierarchies in Production Management and Control: a Survey,'' Technical
Report, LIDS-P-1734, Lab. for Infor. and Decision Systems, MIT, Cambridge, MA, (1988).
83
Z. Lieber, ``An Extension of Modigliani and Hohn's Planning Horizons results", Management
Science, Vol. 20, (1973), 319-330.
84
S. Lou and P. Kager, ``A Robust Production Control Policy for VLSI Wafer Fabrication", IEEE Trans.
on Semiconductor Manufacturing, Vol. 2, (1989), 159-164.
85
S. Lou, S. P. Sethi, and Q. Zhang, ``Optimal Feedback Production Planning in a Stochastic Two
Machine Flowshop,'' European Journal of Operational Research-Special Issue on Stochastic
Control Theory and Operational Research, Vol. 73, (1994), 331-345.
86
S. Lou and G. van Ryzin, ``Optimal Control Rules for Scheduling Job Shops,'' Annals of Operations
Research, Vol. 17, (1989), 233-248.
87
L. F. Martins, S. E. Shreve, and H. M. Soner, ``Heavy Traffic Convergence of a Controlled
Multiclass Queuing System," SIAM J. Contr. Optim., Vol. 34, (1996), 2133-2171.
88
M. D. Mesarovic, D. Macko, and Y. Takahara, Theory of Multilevel Hierarchical Systems,
Academic, New York, NY, 1970.
89
F. Modigliani and F. Hohn, ``Production Planning Over Time and the Nature of the Expectation
and Planning Horizon", Econometrica, Vol. 23, (1955), 46-66.
90
R. Nagi, Design and Operation of Hierarchical Production Management Systems, PH.D. Thesis,
System Research Center, University of Maryland, Baltimore, MD, 1991.
142
91
M.-T. Nguyen, Singular Perturbations in Deterministic and Stochastic Hybrid Control Systems: An
Averaging Approach, Doctoral Dissertation, University of South Australia, Adelaide, Australia,
1999.
92
M.-T. Nguyen and V. Gaitsgory, `` Asymptotic Optimization for a Class of Nonlinear Stochastic
Hybrid Systems on Infinite Time Horizon", Proceedings of the 1997 American Control Conference,
3587-3591.
93
R. G. Phillips and P. Koktovic, ``A Singular Perturbation Approach to Modeling and Control of
Markov Chains,'' IEEE Trans. Automatic Control, Vol. AC-26, (1981), 1087-1094.
94
E. Presman, S. P. Sethi, and W. Suo, ``Optimal Feedback Production Planning in Stochastic
Dynamic Jobshops," Mathematics of Stochastic Manufacturing Systems, Proceedings of the 1996
AMS-SIAM Summer Seminar in Applied Mathematics, G. Yin and Q. Zhang(Eds.), Lectures in
Applied Mathematics, American Mathematical Society, Providence, RI, 1997a, 235-252.
95
E. Presman, S. P. Sethi, and W. Suo, ``Optimal Feedback Production Planning in a Stochastic N-
Machine Flowshop with Limited Buffers," Automatica, Vol. 33, (1997), 1899-1903.
96
E. Presman, S. P. Sethi, and H. Zhang, ``Optimal Production Planning in a N-Machine Flowshop
with Long-Run Average Cost," Proceedings of the Indian Academy of Sciences (forthcoming),
1999a.
97
E. Presman, S. P. Sethi, and H. Zhang, ``Optimal Production Planning in a General Jobshop with
Long-Run Average Cost," Working Paper, School of Management, The University of Texas at
Dallas, 1999b.
98
E. Presman, S. P. Sethi, H. Zhang, and Q. Zhang, ``Optimality of Zero-Inventory Policies for an
Unreliable Manufacturing System Producing Two Part Types", Dynamics of Continuous, Discrete
and Impulsive Systems, Vol. 4, (1998a), 485-496.
143
99
E. Presman, S. P. Sethi, and Q. Zhang, ``Optimal Feedback Production Planning in a Stochastic N-
Machine Flowshop,'' Proc. of 12th World Congress of International Federation of Automatic
Control, Sydney, Australia, July 18-23 (1993).
100
E. Presman, S. P. Sethi, and Q. Zhang, ``Optimal Feedback Production Planning in a Stochastic N-
Machine Flowshop," Automatic, Vol. 31, (1995), 1325-1332.
101
D. F. Rogers, J. R. Evans, R. D. Plante, and R. T. Wong, ``Aggregation and Disaggregation
Techniques and Methodology in Optimization,'' Operations Research, Vol. 39, (1991), 553-582.
102
V. R. Saksena, J. O'Reilly, and P. Kokotovic, ``Singular Perturbations and Time-Scale Methods in
Control Theory: Survey 1976-1983,'' Automatica, Vol. 20, (1984), 273-293.
103
C. Samaratunga, S. P. Sethi, and X. Y. Zhou, ``Computational Evaluation of Hierarchical
Production Control Policies for Stochastic Manufacturing Systems,'' Operations Research, Vol.
45, (1997), 258-274.
104
S. P. Sethi, ``Some Insights into Near-Optimal Plans for Stochastic Manufacturing Systems,"
Mathematics of Stochastic Manufacturing Systems, G. Yin and Q. Zhang(Eds.), Lectures in
Applied Mathematics, Vol. 33, American Mathematical Society, Providence, RI, 1997, 287-315.
105
S. P. Sethi, H. M. Soner, Q. Zhang, and J. Jiang, ``Turnpike Sets and Their Analysis in Stochastic
Production Planning Problems,'' Mathematics of Operations Research, Vol. 17, (1992a), 932-950.
106
S. P. Sethi, W. Suo, M.I. Taksar and H. Yan, ``Optimal Production Planning in a Multi-Product
Stochastic Manufacturing System with Long-Run Average Cost," J. Discrete Event Dynamic Syst.,
Vol. 8, (1998), 37-54.
107
144
S. P. Sethi, W. Suo, M.I. Taksar and Q. Zhang, ``Optimal Production Planning in a Stochastic
Manufacturing System with Long-Run Average Cost," Journal of Optimization Theory and
Applications, Vol. 92, (1997), 161-188.
108
S. P. Sethi, M. I. Taksar, and Q. Zhang, ``Capacity and Production Decisions in Stochastic
Manufacturing Systems: An Asymptotic Optimal Hierarchical Approach,'' Prod. & Oper. Mgmt.,
Vol. 1, (1992b), 367-392.
109
S. P. Sethi, M. I. Taksar, and Q. Zhang, ``Hierarchical Decomposition of Production and Capacity
Investment Decision in Stochastic Manufacturing Systems,'' International Transactions in
Operations Research, Vol. 1, (1994a), 435-451.
110
S. P. Sethi and G. L. Thompson, ``Simple Models in Stochastic Production Planning,'' In Applied
Stochastic Control in Econometrics and Management Science, Bensoussan, A., Kleindorfer, P. and
Tapiero, C. (eds.). North-Holland, Amsterdam, (1981a), 295-304.
111
S. P. Sethi and G. L. Thompson, Optimal Control Theory: Applications to Management Science,
Martinus Nijhoff Publishing, Boston, MA, 1981b.
112
S. P. Sethi, H. Yan, Q. Zhang, and X. Y. Zhou, ``Feedback Production Planning in a Stochastic Two
Machine Flow Shop: Asymptotic Analysis and Computational Results,'' Int. J. Production Econ.,
Vol. 30-31, (1993), 79-93.
113
S. P. Sethi and H. Zhang, ``Hierarchical Production Planning in a Stochastic Manufacturing
Systems with Long-Run Average Cost: Asymptotic Optimality," Stochastic Analysis, Control,
Optimization and Applications, A Volume in Honor of W. H. Fleming, eds. W. McEneany, Y. Yin,
and Q. Zhang, Birkhäuser, Boston, 1998.
114
S. P. Sethi and H. Zhang, ``Average-Cost Optimality of Kanban Policies for a Stochastic Flexible
Manufacturing System," International Journal on Flexible Manufacturing Systems, Vol. 11,
(1999), 213-224.
115
145
S. P. Sethi, H. Zhang and Q. Zhang, ``A Note on Optimal Production Rates in a Deterministic Two-
Product Manufacturing System", Working Paper, School of Management, The University of
Texas at Dallas, 1996.
116
S. P. Sethi, H. Zhang and Q. Zhang, ``Hierarchical Production Control in a Stochastic
Manufacturing System with Long-Run Average Cost," Journal of Mathematical Analysis and
Applications Vol. 214, (1997), 151-172.
117
S. P. Sethi, H. Zhang and Q. Zhang, ``Minimum Average Cost Production Planning in Stochastic
Manufacturing Systems", Mathematical Models and Methods in Applied Sciences, Vol. 8, (1998),
1252-1276.
118
S. P. Sethi, H. Zhang and Q. Zhang, ``Hierarchical Production Control in a Stochastic N-Machine
Flowshop with Long-Run Average Cost," Working Paper, School of Management, The University
of Texas at Dallas, 1999a.
119
S. P. Sethi, H. Zhang and Q. Zhang, ``Hierarchical Production Control in Dynamic Stochastic
Jobshops with Long-Run Average Cost," Working Paper, School of Management, The University
of Texas at Dallas, 1999b.
120
S. P. Sethi, H. Zhang and Q. Zhang, ``Hierarchical Production Planning in a Stochastic N-Machine
Flowshop with Limited Buffers," Journal of Mathematical Analysis and Applications (to appear),
1999c.
121
S. P. Sethi, H. Zhang and Q. Zhang, ``Hierarchical Production Planning in a Stochastic Jobshop
with Limited Buffers," Working Paper, School of Management, The University of Texas at Dallas,
1999d.
122
S. P. Sethi and Q. Zhang, ``Asymptotic Optimality in Hierarchical Control of Manufacturing
Systems under Uncertainty: State of the Art," Operations Research Proceedings 1990, (W.
Buehler, G. Feichtinger, R. Hartl, F. Radermacher, and P. Staehly, Eds.), Springer-Verlag, Berlin,
(1992a), 249-263.
146
123
S. P. Sethi and Q. Zhang, ``Multilevel Hierarchical Controls in Dynamic Stochastic Marketing-
Production Systems," Proc. of 31st IEEE CDC, Tucson, AZ, Dec. 16-18, (1992b), 2090-2095.
124
S. P. Sethi and Q. Zhang, Hierarchical Decision Making in Stochastic Manufacturing Systems,
Birkhäuser Boston, Cambridge, MA, 1994a.
125
S. P. Sethi and Q. Zhang, ``Hierarchical Production Planning in Dynamic Stochastic Manufacturing
Systems: Asymptotic Optimality and Error Bounds," Journal of Mathematics Analysis and
Applications, Vol. 181, (1994b), 285-319.
126
S. P. Sethi and Q. Zhang, ``Asymptotic Optimal Controls in Stochastic Manufacturing Systems
with Machine Failures Dependent on Production Rates," Stochastics and Stochastics Reports,
Vol. 48, (1994c), 97-121.
127
S. P. Sethi and Q. Zhang, ``Multilevel Hierarchical Decision Making in Stochastic Marketing-
Production Systems," SIAM J. Contr. Optim., Vol. 33, (1995a), 528-563.
128
S. P. Sethi and Q. Zhang, ``Hierarchical Production and Setup Scheduling in Stochastic
Manufacturing Systems," IEEE Trans. Auto. Contr., Vol. 40, (1995b), 924-930.
129
S. P. Sethi and Q. Zhang, and X. Y. Zhou, ``Hierarchical Controls in Stochastic Manufacturing
Systems with Machines in Tandem," Stochastics and Stochastics Reports, Vol. 41, (1992a), 89-
118.
130
S. P. Sethi, Q. Zhang, and X. Y. Zhou, ``Hierarchical Production Planning in a Stochastic Flowshop
with a Finite Internal Buffer," Proc. of 31st IEEE Conference on Decision and Control, Tucson, AZ,
December 16-18, (1992b), 2074-2079.
131
S. P. Sethi, Q. Zhang, and X. Y. Zhou, ``Hierarchical Controls in Stochastic Manufacturing Systems
in th Convex Costs," Journal of Optimization Theory and Applications, Vol. 80, (1994), 303-321.
147
132
S. P. Sethi, Q. Zhang, and X. Y. Zhou, ``Hierarchical Production Controls in a Stochastic Two-
Machine Flowshop with a Finite Internal Buffer," IEEE Trans. on Robotics and Automation, Vol.
13, (1997), 1-13.
133
S. P. Sethi and X. Y. Zhou, ``Stochastic Dynamic Job Shops and Hierarchical Production Planning,"
IEEE Trans. Auto. Contr., Vol. 39, (1994), 2061-2076.
134
S. P. Sethi and X. Y. Zhou, ``Optimal Feedback Controls in Deterministic Dynamic Two-Machine
Flowshops," Operations Research Letters, Vol. 19, (1996a), 225-235.
135
S. P. Sethi and X. Y. Zhou, ``Asymptotic Optimal Feedback Controls in Stochastic Dynamic Two-
Machine Flowshops," in Recent Advances in Control and Optimization of Manufacturing
Systems, Lecture Notes in Control and Information Science, Vol. 214, G. Yin and Q. Zhang (eds.),
Springer-Verlag, New York, NY, 1996b, 203-216.
136
A. Sharifnia, ``Production Control of a Manufacturing System with Multiple Machine States,"
IEEE Trans. Auto. Contr. Vol. AC-33, (1988), 620-625.
137
A. Sharifnia, M. Caramanis, and S. B. Gershwin, ``Dynamic Setup Scheduling and Flow Control in
Manufacturing Systems," J. Discrete Event Dynamic Syst., Vol. 1, (1991), 149-175.
138
P. Shi, E. Altman, and V. Gaitsgory, ``On Asymptotic Optimization for a class of Nonlinear
Stochastic Hybrid Systems", Mathematical Methods of Oper. Res., Vol. 47, (1998), 229-315.
139
H. A. Simon, ``The Architecture of Complexity," Proc. of the American Philosophical Society, Vol.
106, (1962), 467-482; reprinted as Chapter 7 in Simon, H.A., The Sciences of the Artificial, 2nd
Ed., The MIT Press, Cambridge, MA, 1981.
140
M. G. Singh, Dynamical Hierarchical Control, Elsevier, New York, NY, 1982.
148
141
N. J. Smith and A. P. Sage, "An Introduction to Hierarchical Systems Theory," Computers and
Electrical Engineering, Vol. 1, (1973), 55-72.
142
H. M. Soner, ``Optimal Control with State Space Constraints II," SIAM J. Contr. Optim, Vol. 24,
(1986), 1110-1122.
143
H. M. Soner, ``Singular Perturbations in Manufacturing Systems," SIAM J. Contr. Optim., Vol. 31,
(1993), 132-146.
144
A. Y. Sprzeuzkouski, ``A Problem in Optimal Stock Management", Journal of Optimization Theory
and Applications, Vol. 1, (1967), 232-241.
145
N. Srivatsan, Synthesis of Optimal Policies in Stochastic Manufacturing Systems, Ph.D. Thesis,
MIT, Operations Research Center, Cambridge, MA, 1993.
146
N. Srivatsan, S. Bai, and S. B. Gershwin, ``Hierarchical Real-Time Integrated Scheduling of a
Semiconductor Fabrication Facility," in Computer-Aided Manufacturing/Computer-Integrated
Manufacturing, Part II of Vol. 61 in the Series Control and Dynamic Systems, Leondes, C.T.(ed.)
Academic Press, New York, (1994), 197-241.
147
N. Srivastsan and Y. Dallery,``Partial Characterization of Optimal Hedging Point Polices in
Unreliable Two-Part-Type Manufacturing Systems," Operations Research, Vol 46, (1998), 36-45.
148
N. Srivatsan and S. B. Gershwin, ``Selection of Setup Times in a Hierarchically Controlled
Manufacturing System," Proc. of 29th IEEE-CDC, Honolulu, HI, 1990, 575-581.
149
H. Stadtler, Hierarchische Produktionsplanung bei Losweiser Fertigung, Physica-Verlag,
Heidelberg, Germany, 1988.
150
149
M. Switalski, Hierarchische Produktionsplanung, Physica-Verlag, Heidelberg, Germany, 1989.
151
G. L. Thompson and S. P. Sethi, ``Turnpike Horizons for Production Planning,'' Management
Science. Vol. 26, 3, (1980), 229-241.
152
G. L. Thompson, S. P. Sethi, and J.-T. Teng, ``Strong Planning and Forecast Horizons for a Model
with Simultaneous Price and Production Decisions," European Journal of Operational Research,
Vol. 16, (1984), 378-388.
153
R. Uzsoy, C. Y. Lee, and L. Martin-Vega, ``A Review of Production Planning and Scheduling of
Models in the Semiconductor Industry, Part II: Shop Floor Control", IIE Transactions on
Scheduling and Logistic, Vol. 26, (1996).
154
G. J. Van Ryzin, X. C. Lou, and S. B. Gershwin, ``Production Control for a Tandem Two-Machine
System," IIE Transactions, Vol. 25, (1993), 5-20.
155
M. H. Veatch, Queuing Control Problems for Production/Inventory Systems, Ph.D. Thesis, MIT,
Operations Research Center, Cambridge, MA, 1992.
156
M. H. Veatch and M. C. Caramanis, `` Optimal Average Cost Manufacturing Flow Controllers:
Convexity and Differentiability," IEEE Trans. Auto. Contr., Vol. 44, (1999), 779-783.
157
A. F. Veinott, ``Production Planning with Convex Costs: A Parametric Study", Management
Science, Vol. 10, (1964), 441-460.
158
D. Vermes, Optimal Control of Piecewise Deterministic Markov Processes, Stochastics, Vol. 14,
1985, 165-207.
159
J. Violette, Implementation of a Hierarchical Controller in a Factory Simulation, M.S. Thesis, MIT
Department of Aeronautics and Astronautics, Cambridge, MA, 1993.
150
160
J. Violette and S. B. Gershwin, ``Decomposition of Control in a Hierarchical Framework for
Manufacturing Systems," Proc. of the 1991 Automatic Control Conference, Boston, MA, (1991),
465-474.
161
L. Wein, ``Optimal Control of a Two-Station Brownian Network," Mathematics of Operations
Research, Vol. 15, (1990), 215-242.
162
P. Whittle, Risk-Sensitive Optimal Control, Wiley, New York, 1990.
163
X. L. Xie, ``Hierarchical Production Control of a Flexible Manufacturing System," Applied
Stochastic Models and Data Analysis, Vol. 7, (1991), 343-360.
164
H. Yan, S. Lou, S. P. Sethi, A. Gardel, and P. Deosthali, ``Testing the Robustness of Two-Boundary
Control Policies in Semiconductor Manufacturing", IEEE Transactions on Semiconductor
Manufacturing, Vol. 9, (1996), 285-288.
165
H. Yan, G. Yin, and S. Lou, ``Using Stochastic Optimization to Determine Threshold Values for
Control of Unreliable Manufacturing Systems", Journal of Optimization Theory and Applications,
Vol. 83, (1994), 511-539.
166
G. Yin and Q. Zhang, Continuous-Time Markov Chains and Applications: A Singular Perturbation
Approach, Springer-Verlag, New York, NY, 1997.
167
Q. Zhang, `` Risk Sensitive Production Planning of Stochastic Manufacturing Systems: A Singular
Perturbation Approach," SIAM J. Contr. Optim., Vol. 33, (1995), 498-527.
