Multi-Level Coordination and Decision-Making in … · Multi-Level Coordination and Decision-Making in ... customer satisfaction is an important ... theoretic framework where agents

Wernz and Henry: Multi-Level Coordination and Decision-Making in Service Operations Service Science 1(4), pp. 270-283, © 2009 SSG

270

Multi-Level Coordination and Decision-Making in Service Operations

Christian Wernz, Andrew Henry

Grado Department of Industrial and Systems Engineering Virginia Polytechnic Institute and State University

Blacksburg, VA 24061, USA [email protected], [email protected]

ecisions in service operations are complex since they involve various interdependent decision-makers (agents) at different hierarchical levels, ranging from customer representatives over account managers

to top executives. We formulate a three-level maintenance service problem as a stochastic decision-making model, with an account manager, supervisor, and worker at each of the three levels. The proposed incentive mechanism aligns the interests of lower level agents with the goals of the top agent. The agents’ strategic interactions are analyzed game-theoretically and results show that a Pareto-efficient Nash equilibrium can be attained given certain organizational properties. Furthermore, we show that local information can be sufficient for organization-wide optimal decisions. In a final step, the three-level model is generalized for multi-level, i.e., multi-organizational-scale, systems.

Key words: multiscale decision theory; multi-level systems; game theory; organizational theory; incentives;

maintenance service operations; service science History: Received Nov. 13, 2009; Received in revised form Jan. 29, 2010; Accepted Feb. 5, 2010; Online

first publication Feb. 5, 2010 1. Introduction The service industry is the dominant post-industrial sector representing over 60% of GDP and providing jobs to more than one-third of the workforce worldwide (Duignan 2007). The service industry is multi-faceted and examples of specific industries include healthcare, entertainment, education, finance, and maintenance/repair. In this paper, we focus on decision challenges in a hierarchical maintenance service organization, but the insights gained by our analysis apply to other organizations in the service industry and beyond.

Most machines, such as escalators, elevators, photocopiers, and motor vehicles, require preventive maintenance and/or repairs to sustain their usability. Machine owners, particularly businesses, often outsource their maintenance service needs by committing to service contracts. Satisfaction of business customers with the maintenance service affects the likelihood of contract renewal (Zeithaml et al. 1996). Thus, customer satisfaction is an important goal for an account manager in the respective service organization. The goals of account managers can be in conflict with the goals of lower level employees, such as maintenance supervisors and maintenance workers. For instance, the supervisor, who is one hierarchical level below the account manager, may care primarily about costs when fulfilling service contracts, and not customer satisfaction. One further organizational level below, a maintenance worker may seek to get his work assignment done with the least amount of effort. The goals of supervisor and worker are in conflict with the account manager’s objective to satisfy the customer and to get the service contract renewed.

In this paper, we demonstrate how incentives in a three-level organization can be used to align the interests of lower level agents with the objectives of the highest level agent. We provide agents with decision-making rules in form of analytic equations, which allow them to easily make optimal decisions regarding incentives and their respective courses of action. We extend the three-level system to multiple levels and show that comparatively simple equations are sufficient for optimal decision making in large organizations with multi-organizational-scale agent interactions. In other words, we propose an organizational incentive system that aligns conflicting interests of decision-makers with top management priorities and provide all decision-makers with tools to identify their optimal choices.

The remainder of the paper is organized as follows: Section 2 reviews the relevant literature. In Section 3, the three-level maintenance service problem is described in detail. Section 4 introduces the mathematical three-level model and its notation. The game-theoretical analysis is conducted in Section 5. Section 6 illustrates the

D


271

mathematical findings with a numerical example. Section 7 extends the three-level model to a multi-level hierarchy. Results and key findings are summarized in Section 8.

2. Literature Review In this section, we discuss the relevant literature on multi-level systems, incentives, and goal alignment in organizations, principal-agent models, game theory, and service science. We briefly introduce models and approaches that have influenced our work and show in what ways our contribution is novel and extends the capabilities of prior methods.

Despite the ubiquity of hierarchical, multi-level systems such as service organizations, only few mathematical models that can derive analytical solutions exist. Motivated by this research opportunity, Wernz (2008) developed a multiscale decision-making model that recognizes and integrates organizational and temporal scales in hierarchical systems. Specifically, Wernz and Deshmukh (2010a) show how multiscale decision-making can bridge organizational-scales in hierarchical systems to improve agents’ decision-making capabilities and overall organizational performance. For advancements in multiscale decision theory and applications see work by Wernz and Deshmukh (2007a, 2007b, 2010c).

Prior to multiscale decision theory, Schneeweiss (1995, 2003) developed a framework for distributed decision-making. His model provides a unified notation for hierarchical agent interactions and has been applied to production planning (Heinrich and Schneeweiss 1986), scheduling (Schneeweiss and Schröder 1992), and supply chain management (Schneeweiss and Zimmer 2004), among others. For the second edition of his monograph, Schneeweiss (2003) included a chapter on service operation, which documents the relevance and timeliness of service science as a research domain. Schneeweiss’ distributed decision-making model is suitable for systems with up to three hierarchical levels; however, solutions for multi-level problems have not been published.

A prominent hierarchical decision-making model in the production planning domain is from Hax and Meal (1975), see also Bitran et al. (1981) and additionally Hax and Candea (1984). In hierarchical production planning (HPP), levels are connected through a sequential, top-down decision process. The higher level’s decision is passed down in the form of constraints affecting the lower level’s decision realm. The hierarchical interactions in our model are realized through mutual influences on rewards and transition probabilities recognizing top-down and bottom-up interactions. HPP has been successfully applied to challenges in production planning (de Kok 1990, Lin et al. 2006); for an overview see McKay et al. (1995). A service science equivalent to HPP does not yet exist and the paper at hand presents a first contribution towards a multi-organizational-scale service science framework.

In systems engineering, Mesarovic et al. (1970) investigate conceptual and formal aspects of multi-organizational-scale systems and deduce a mathematical theory of coordination in hierarchies. We build our model on Mesarovic et al.’s agent interaction concept: success of the supremal decision unit (superior agent) depends on the performance of the infimal decision unit (subordinate agent). For similar approaches see Pappas et al. (2000) and a survey paper by Sandell et al. (1978). However, none of these systems engineering models includes the possibility of a strategic, i.e., game-theoretic, interaction between agents. Moreover, Mesarovic et al. (1970) rely on dynamic control theory, which requires continuous time system equations as inputs. Particularly for real-world systems with human decision-makers, system equations and their parameters are difficult to obtain. Thus, we chose to model the system dynamics via discrete transition probabilities and Markov decision processes. The model presented in this paper is a one-period, two-stage model; for multi-period extensions see Wernz (2008) and Wernz and Deshmukh (2009, 2010c).

Multi-agent systems (MAS) represent an alternative to our analytical model. MAS are typically studied by computer scientists, particularly in the area of distributed artificial intelligence (Weiss 1999), but have also found applications in engineering, particularly industrial engineering (Krothapalli and Deshmukh 1999, Middelkoop and Deshmukh 1999, Anussornnitisarn et al. 2005). Most MAS are simulation-based. In contrast, we derive results analytically and obtain closed-form solutions, providing deeper insights and simple, but still optimal, decision rules.

Game theory has been used to model hierarchical agent interactions. Kim et al. (2003) propose a game-theoretic framework where agents schedule maintenance down-time in a competitive energy market. Murthy and Asgharizadeh (1999) model maintenance service contract negotiation; similarly, Jackson and Pascual (2008) derive optimal contract negotiation mechanisms for aging machines. Deng and Papadimitriou (1999) developed a game-theoretic model where hierarchically arranged decision-makers, with potentially conflicting objective functions, solve a linear program. Anandalingam and Apprey (1991) presented a mathematical programming approach to resolve a multi-level conflict problem. Geanakoplos and Milgrom (1991) conducted a team-theoretic analysis of


272

decision-makers in hierarchies. Team theory (Marschak and Radner 1972) is similar to game theory; the difference being that agents no longer have conflicting interests since they receive identical rewards.

Stackelberg games, where one agent (leader) makes a decision before a second agent (follower), have been used to analyze hierarchical, two-level decision problems (Chang and Ho 1981, Nie et al., 2006, van Hoesel, 2008). Ho (1983) describes a Stackelberg game with incentives for multiple time periods. Luh et al. (1984) extended Ho’s control-theoretic analysis to three levels. In our one-period model, agents make their decisions in two stages: in stage one, agents decides on the incentives they offer to their respective subordinate agent. Agents commit to their incentive shares sequentially. In stage two of the decision process, agents make their choices about which action to take knowing the incentive share coming from their superior agent. Agents’ choices about their actions are neither observable nor communicated to the other agents. Thus, the agents’ interaction in the second stage is modeled as a game with simultaneous decisions.

In principal-agent (PA) theory, incentives play a key role in motivating subordinates. An agent works for a principal – typically the owner and beneficiary of the enterprise – and the agent’s actions affect the principal’s rewards. To motivate good work, a principal may offer a share of its reward to the agent; see van Ackere (1993), Klock (2004), and Zhang et al. (2007) for incentive models in the service industry. In our model, we assume, a reward-dependent incentive payment from the superior to the subordinate agent; see Song and Chen (2006) for a similar model. Itoh (1991) developed a PA model with one principal and two agents. Laffont (1990) and Tirole (1986) examined a three-level hierarchy, consisting of a principal, supervisor, and agent. Their work focuses on contract design and collusion across levels of the organization.

Siemsen et al. (2007) examined the use of individual and group incentives. Individual and group incentives are used to motivate effort, knowledge sharing, and helping in groups. The authors’ concept of outcome linkages, which is the dependency of one employee’s task outcome on another’s, can also be found in our model formulation.

To represent the mutual influence of agents on each other, we use dependency graphs (Dolgov and Durfee 2004) initially proposed by computer scientists. Dependency graphs are based on ideas from multi-agent influence diagrams (Koller and Milch 2001) and other graphical models (Kearns et al. 2001, Guestrin et al. 2003), which originated from decision analysis (Howard and Matheson 2005). Dependency graphs capture and graphically illustrate how agents influence each other through rewards and changes in transition probabilities.

Wernz and Deshmukh (2007a, 2007b) developed a two-agent model that the paper on hand extends to a three-agent, and later, multi-agent model. Our paper showcases the capability of multiscale decision theory to solve a challenge in the service science domain. In addition, novel theorems are presented that contribute to multiscale decision theory by showing that compact, closed-form solutions can be derived for hierarchical chains of multiple agents. In general, multiscale decision theory informs organizational designers on how to resolve conflicts of interest between agents in organizations. The theory also provides decision-makers with tools to determine optimal incentive levels and Pareto-optimal Nash equilibrium choices. Multiscale decision theory is currently being applied to healthcare systems, financial networks, climate change, homeland security, and technology management.

3. Motivational Example We consider an enterprise that produces, installs, and maintains escalators. The maintenance/service division of this enterprise is a three-level hierarchical organization consisting of an account manager, maintenance supervisor, and maintenance worker.

The worker is responsible for preventive maintenance, e.g., lubrication of key bearings and gears. Additionally, the worker is tasked with inspecting the escalators for problems, such as abnormal wear, which could jeopardize the equipment’s performance. During the visual inspection, the worker makes the (internal) choice to conduct either a thorough inspection or a superficial inspection. The inspection decision of the worker will result, with a certain probability, in the escalator being under serviced or well serviced. Given a thorough inspection the outcome of a well serviced escalator is more likely than an under serviced one, but both states are possible for either of the worker’s decisions. A well serviced escalator is associated with a larger effort by the worker. A superficial inspection is more likely to result in an under serviced state with low effort expended by the worker. The worker seeks to dispense as little effort as possible and thus prefers the superficial inspection over the thorough inspection.

The maintenance supervisor is in charge of the maintenance operation. She is tasked with setting the daily job schedule for the worker. When choosing the schedule, the supervisor must decide between a heavy or light workload for her subordinate. A heavy workload requires the worker to visit and service a large number of jobsites in a day. A heavy workload has a high probability of causing the worker to rush, not spending enough time at each job. The worker having to rush likely results in future malfunction of the escalator, which leads to low service


273

satisfaction of the customer. In contrast, a light workload level allows the worker to spend more time with each customer, which leads with greater probability to high service satisfaction. Again, both states (high and low service satisfaction) are possible given either decision (light and heavy workload). The supervisor is evaluated on a cost per job basis, and her goal is to keep costs low. Therefore, even though service satisfaction will likely suffer, the decision for a heavy workload is preferred by the supervisor as the expected cost is much lower than with the light workload. A well serviced escalator increases the probability of high service satisfaction and high cost; in contrast, an under serviced escalator increases the probability of low service satisfaction and low cost.

At the top level of the enterprise’s maintenance/service division is the account manager. The manager is responsible for managing the relationships with the escalator customer. The manager receives a commission for customers renewing their maintenance service contracts. The service satisfaction outcome influences the probability of contract renewal. High service satisfaction increases the probability of contract renewal while low service satisfaction reduces the probability. The manager considers a reward-sharing incentive to align the supervisor’s goals with his own.

Figure 1 Agent Interdependencies


274

Figure 1 summarizes the agents’ interdependencies with respect to actions, rewards, transition probabilities and outcomes. 4. Model and Notation Manager, supervisor, and worker are represented by agents A1, A2, and A3 in the following general model formulation. All three agents are confronted with choosing one of two possible actions from their respective set of actions { }1 1 1

1 2,A A Aa a=A , { }2 2 21 2,A A Aa a=A , { }3 3 3

1 2,A A Aa a=A . (1)

In the example introduced in Chapter 3, these decisions correspond to {thorough inspection, superficial inspection} of the worker (agent A3) and {high workload, low workload} of the supervisor (agent A2). The manager (agent A1) does not make a decision; yet, the general model includes the possibility of a decision by the top-level agent.

Depending on its decision, each agent will move to one of two possible states { }1 1 1

1 2,A A As s=S , { }2 2 21 2,A A As s=S , { }3 3 3

1 2,A A As s=S (2)

with probabilities ( )1 1 1|A A A

i mp s a , ( )2 2 2|A A Aj np s a , ( )3 1 1|A A A

k op s a , { }, , , , , 1,2i j k m n o∈ . (3)

Specifically, the transition probabilities for agent A1 are ( )1 1 1 1

1 1 1|A A A Ap s a α= , ( )1 1 1 12 1 1| 1A A A Ap s a α= − ,

( )1 1 1 12 2 2|A A A Ap s a α= , ( )1 1 1 1

1 2 2| 1A A A Ap s a α= − , (4)

with 10 1Amα≤ ≤ . The transition probabilities for agent A2 and A3 are denoted accordingly with superscripts A2

and A3. Each agent receives a state-dependent reward

( )1 1 1A A Ai ir s ρ= , ( )2 2 2A A A

j jr s ρ= , ( )3 3 3A A Ak kr s ρ= . (5)

These probabilities and rewards describe the agents’ state transitions and rewards prior to their interaction with one another. As explained in Section 3, agents affect each other’s rewards and transition probabilities through the state to which they move. Agent A1 gives an incentive to agent A2, which depends on agent A1’s reward; likewise, agent A2 passes a share of its reward on to agent A3. The incentive payments are the top-down influence in the hierarchical agent interaction. Agents also affect each other bottom-up. Agent A3 influences the probability of agent A2 moving to a particular state, and agent A2 likewise influences the transition probability of agent A1.

Figure 2 Dependency Graph for Three-Level Model

transition probability

reward

cost of influence

A2

A3

A1


275

We graphically represent the agent interaction through a dependency graph (Figure 2), which builds upon a notation by Dolgov and Durfee (2004). Solid and dashed line arrows indicate that agents influence transition probabilities and rewards of other agents. The dotted line indicates a cost associated with providing a reward.

The reward arrow emanating from agent A1 means that agent A1 makes an incentive payment to agent A2. Agent A1’s incentive to agent A2 has an indirect effect on agent A3, which is illustrated through an arrow emanating from the reward arrow between agent A1 and A2. Also, agent A2 directly affects the reward of agent A3. Originating at the same point as the corresponding reward arrow, the dotted line arrow signifies a reward-dependent cost for the agent paying the incentive. The interpretation of the transition probability arrows is accordingly. Agents A1 and A2 are directly affected by their subordinates and agent A3 indirectly influences the transition probabilities of agent A1.

Since agent A1 pays a share of its reward to agent A2, its final reward for reaching a particular state is ( ) ( ) ( )1 1 1 1

11A A A Afinal i ir s r s b= ⋅ − with 10 1b≤ < . (6)

We refer to xb as the share coefficient. The final reward for agent A2 includes the incentive from agent A1 minus the incentive paid to agent A3, which is expressed as

( ) ( ) ( )( ) ( )2 1 2 2 2 1 11 2, 1A A A A A A A

final i j j ir s s r s b r s b= + ⋅ ⋅ − (7)

with share coefficient 2b ( 20 1b≤ < ). Consequentially, the final reward for agent A3 is

( ) ( ) ( ) ( ) ( )3 1 2 3 3 3 2 3 3 2 2 1 12 2 1 2, ,A A A A A A A A A A A A A

final i j k k final k j ir s s s r s b r r s b r s b b r s= + ⋅ = + ⋅ + ⋅ ⋅ . (8)

Next, we describe the effects on the agents’ transition probabilities. Starting at the bottom of the hierarchical chain, agent A3’s transition probability is not affected. One level up, the transition probability of agent A2 is affected by the state that agent A3 moves to: ( ) ( ) ( )2 2 2 3 2 2 2 2 3

2| , | ,A A A A A A A A Afinal j n k j n j kp s a s p s a f s s= + . (9)

The transition probability influence function 2f , which describes the effect on agent A2, is defined as

( ) 22 32

2

if,

ifA Aj k

c j kf s s

c j k=⎧

= ⎨− ≠⎩ . (10)

with 2 0c > . The constant xc is referred to as the change coefficient. The final transition probability of agent A1 is

( ) ( ) ( ) ( )( )1 1 1 2 3 1 1 1 1 2 2 31 2| , , | , 1 ,A A A A A A A A A A A A

final i m j k i m i j j kp s a s s p s a f s s f s s= + ⋅ +

(11)

with transition probability influence function 1f and change coefficient 1 0c > :

( ) 11 21

1

if,

ifA Ai j

c i jf s s

c i j=⎧

= ⎨− ≠⎩ . (12)

By combining (10)-(12), the final transition probability of agent A1 simplifies to ( ) ( ) ( ) ( )1 1 1 2 3 1 1 1 1 1 1

1 2 1 1 2| , , | 1 |A A A A A A A A A A Afinal i m j k i m i mp s a s s p s a c c p s a c c c= ± ⋅ ± = ± ± ⋅ . (13)

Since probability values must be within interval [ ]0,1 , and since change coefficients are positive, the following conditions must hold { }2 2 2 2

2 1 2 1 20 min , ,1 ,1A A A Ac α α α α< ≤ − − , (14)

( ) { }1 1 1 11 2 1 2 1 20 1 min , ,1 ,1A A A Ac c α α α α< ⋅ + ≤ − − . (15)

In order to create an interesting agent interaction, that requires agents to reason about a non-trivial decision strategy, we make the following assumptions: 1 1

1 2A Aρ ρ> , 2 2

1 2A Aρ ρ< , 3 3

1 2A Aρ ρ< (16)

1 2 3 1, ,2

A A Am n oα α α > (17)


276

Inequalities in (16) express that agent A1 prefers state 11As over 1

2As and agents A2 and A3 reversely prefer 2

Axs

over 1Axs . Expression (17) states that an action is linked to the state with the same index; in other words, there is a

corresponding action for every state, which is the most likely consequence of the respective action. This restriction circumvents redundant cases in the analysis, but does not limit the generality of the model.

Change coefficients 1c and 2c are intrinsic organizational properties and describe how strongly a superior agent depends on the performance of its subordinate. For details on the interpretation of change coefficients see work by Wernz (2008), and Wernz and Deshmukh (2010a). Wernz and Deshmukh (2007a) have analyzed a model variation where change coefficients are selected by agents themselves. In a further study (Wernz and Deshmukh 2007b), an organizational designer selects share and change coefficients on behalf of the agents.

In our model, agents A1 and A2 only select their respective share coefficient 1b and 2b . Agent A2 first

commits to its share coefficient 2b followed by agent A1’s choice of share coefficient 1b . This sequence, in the first stage of the game, prevents free-riding of agent A2 (Wernz 2008). In the second stage of the game, agents simultaneously chose their actions according to the Nash equilibrium.

The following table summarizes the key variables and parameters. Table 1 Overview of Notation

Axja Actions Axjs States

( )Ax Axjr s ,

( )1,Ax Ax Ax

final j ir s s − reward function before, after interaction Axjρ rewards (before interaction)

( )|Ax Ax Axj np s a transition probability function unaffected by interaction

( )|Ax Ax Axj np s a , ( )1| ,Ax Ax Ax Ax

final j n mp s a s − transition probability function before, after interaction Axjα transition probabilities (before interaction)

( )1,Ax Axx i jf s s + influence function

xc change coefficient

xb share coefficient

5. Analysis: Agent Interaction and Optimal Decision-Making We assume that agents are risk-neutral and rational, i.e., agents maximize their expected utilities or, equivalently, their expected rewards. Agents calculate their and the other parties’ expected rewards and choose actions according to the Nash equilibrium. The expected rewards for agents A1, A2, and A3 are calculated as follows: ( ) ( ) ( )1 1 2 3 1 1 1 2 3 1 2 3| , , , , | , ,A A A A A A A A A A A A

final m n o final i i j k m n oi j k

E r a a a r s p s s s a a a= ⋅∑∑∑ (18)

( ) ( ) ( )2 1 2 3 2 1 2 1 2 3 1 2 3| , , , , , | , ,A A A A A A A A A A A A Afinal m n o final i j i j k m n o

i j kE r a a a r s s p s s s a a a= ⋅∑∑∑ (19)

( ) ( ) ( )3 1 2 3 3 1 2 3 1 2 3 1 2 3| , , , , , , | , ,A A A A A A A A A A A A A Afinal m n o final i j k i j k m n o

i j kE r a a a r s s s p s s s a a a= ⋅∑∑∑ (20)

with ( ) ( ) ( ) ( )1 2 3 1 2 3 1 1 1 2 3 2 2 2 3 3 3 30, , | , , | , , | , |A A A A A A A A A A A A A A A A A A

i j k m n o final i m j k final j n k kp s s s a a a p s a s s p s a s p s a= ⋅ ⋅ (21)


277

The following theorems determine the Nash equilibrium for the three agent interaction in stage two of the game. The Nash equilibrium for the three agent interaction is a strategy profile ( )1* 2* 3*, ,A A A

m n oa a a a∗ = for which

( ) ( ) ( )( )| | ,Ai Ai Aifinal final i iE r a E r a aμ μ

∗ ∗−≥ for all { }1,2,3i∈ and ( ) { }1,2iμ ∈ , (22)

where ( )ia μ∗− is the strategy profile excluding ( )

*Aiiaμ ; ( )iμ is the action index associated with agent Ai.

Theorem 1: Agent A1’s (strictly) dominant strategy is action 11Aa .

Proof: We find agent A1’s Nash equilibrium action by determining index m for which ( ) ( )1 1* 2* 3* 1 1 2* 3*| , , | , ,A A A A A A A A

final m n o final m n oE r a a a E r a a a−≥ , (23)

where { }1,2m− ∈ is the alternative index to the index of the equilibrium decision 1*Ama . Evaluating

( ) ( )1 1 2 3 1 1 2 31 2| , , | , ,A A A A A A A A

final n o final n oE r a a a E r a a a> (24)

for all permutations of { }, 1,2n o∈ leads to

( ) ( ) ( )1 1 1 11 2 1 2 11 1 0A A A A bα α ρ ρ+ − ⋅ − ⋅ − > . (25)

Condition (25) is always true since each of the factors in the inequality is strictly positive (as defined in (6), (16) and (17)). □

Notice that condition (25) only depends on data from agent A1 and not on any decisions or data from the subordinate agents.

Theorem 2: For a reward share of

( )2 2

2 11 1 1

1 1 22

A A

A Ab

cρ ρρ ρ−

≥−

, (26)

agent A2 (weakly) prefers action 21Aa over 2

2Aa .

Proof: We evaluate ( ) ( )2 1 2 3 2 1 2 3

1 1 1 2| , , | , ,A A A A A A A Afinal o final oE r a a a E r a a a≥ (27)

for { }1,2o∈ . Since we have not yet determined the Nash equilibrium action of agent A3, we consider both

possibilities 1o = and 2o = . Evaluation of (27) leads to result (26), which is independent of the action taken by agent A3. □

This theorem has determined the necessary condition for agents A2 to choose the cooperative actions 21Aa .

Notice that condition (26) only depends on aggregate reward data from agent A1 and A2. Transition probabilities of any agent and data from agent A3 do not affect agent A2’s choice.

In the first stage of the two-stage game, agent A1 selects its share coefficient 1b . Agent A1 selects the smallest coefficient that still guarantees cooperative behavior by its subordinate:

( )2 2

2 11 1 1

1 1 22

A A

A Ab

cρ ρρ ρ

∗ −=

−. (28)

We assume that data (rewards etc.) are such that agent A1 prefers choosing a share coefficient according to (28) over giving no incentive, i.e., 1 0b = . This assumption is referred to as the participation condition. For details on the participation conditions see Henry and Wernz (2010). At the next lower level of the hierarchical three-agent chain, Theorem 3 determines agent A3’s condition for cooperative behavior.

Theorem 3: For a reward share of

( )3 3

2 12 2 2

2 2 1

A A

A Ab

cρ ρρ ρ

−≥

− (29)

agent A3 (weakly) prefers action 31Aa over 3

2Aa .


278

Proof: From Theorems 1 and 2, we know that in the first stage of the game agent A1 chooses 11Aa and agent

A2 chooses 21Aa when offered a share coefficient 1b according to (26). To determine the strategy for agent A3, we

evaluate the Nash equilibrium condition ( ) ( )3 1 2 3 3 1 2 3

1 1 1 1 1 2| , , | , ,A A A A A A A Afinal finalE r a a a E r a a a≥ , (30)

which results in

( )( )

3 32 1

2 2 2 1 12 1 2 1 1 1 22 3

A A

A A A Ab

c b cρ ρ

ρ ρ ρ ρ−

≥− + −

. (31)

Term (31) contains 1b , which we have already determined in equation (28). By inserting (28) into (31), we get inequality (29). □

Again, notice that condition (29) only depends on local, aggregate reward information. Agent A1’s parameters do not play a role in agent A3’s decision problem. All agent A3 needs to know about agent A1 is that agent A1 is rational, i.e., chooses its dominant decision strategy and an optimal incentive level.

In the context of our example, we have shown that with appropriate incentives, the worker chooses a thorough inspection and the supervisor decides on a light workload, both of which increase the likelihood of contract renewal for the manager. The manager receives a higher expected reward that trickles down the hierarchical chain to the supervisor and worker, who choose the cooperative but initially unfavorable actions. Supervisor and worker knew that through the incentive payments originating from top of the hierarchy, they will be rewarded for their cooperative behavior.

6. Numerical Example In this section, we provide a numerical example to illustrate the model. The following table shows the example’s parameters and coefficients. Table 2 Input Data

Rewards Transition Probabilities Change Coefficients Manager (A1) 1 1

1 2900 100A Aρ ρ= = 1 11 20.7 0.3A Aα α= =

1 0.15c = 2 0.15c =

Supervisor (A2) 2 21 210 50A Aρ ρ= = 2 1

1 20.6 0.8A Aα α= =

Worker (A3) 3 31 21 1A Aρ ρ= − = 3 3

1 20.8 0.6A Aα α= =

Via knowledge of analytic results derived in Theorems 2 and 3, agents can determine their cost minimal and

cooperation-inducing share coefficients. Agent A1 and A2 will choose 1 1 6b = and 2 1 3b = , respectively. Figure 3 illustrates the results for various levels of change coefficients in a phase diagram. Each of the transition functions (28) and (29) divides the ( ),i ib c space into two phases. For ( ),i ib c values on or above the transition lines, the subordinate agent chooses the cooperative action. Below the transition line, influence and incentives are not large enough such that the subordinate agent responds with the initially preferred, non-cooperative action 2

Axa . Also,

notice that the range of possible values for the change coefficients is limited to 2 0.2maxc = (due to (14) and 1

1 0.7Aα = , 12 0.3Aα = ) and ( )1 2 0.15 0.26maxc c = ≈ (due to (15) and 1

2 0.8Aα = ). For a value of 2 2 0.2maxc c= = ,

the upper bound becomes ( )1 2 0.25max maxc c = .


279

Figure 3 Phase Diagrams

Supervisor (Agent A2) Worker (Agent A3)

2maxc

2c1c

( )1 2max maxc c

1b∗

2b∗

2b1b

( )1 2 0.15maxc c =

The expected rewards of agents for the example data are given in Table 3. The left column shows the expected

rewards for agents with optimal share coefficients and cooperative actions. The right column represents the no participation case where all share coefficients are zero and lower level agents choose their initially preferred, non-cooperative actions. Table 3 Results

Expected Rewards with Cooperation

Expected Rewards without Cooperation

Manager (A1) 597.0 540.8 Supervisor (A2) 92.9 43.2

Worker (A3) 46.6 0.2

7. Multi-Level Extension Most organizations have more than three levels. Overseeing the account manager could be a regional sales manager, who again is subordinate to a global sales representative, which in turn reports to the vice president of sales, who is subordinate to the chief executive officer. Recognizing the need for models that can address large organizations, we extend the three-agent model to a hierarchical chain of many agents. Despite the increased complexity, we can still derive analytic solutions that merely require local information for agents’ optimal parameter and decision choices. For models with multiple agents on the same hierarchical level, see Wernz and Deshmukh (2010a).

We assume a hierarchical chain of m agents Ax with { }1,2,...,x m∈ . All notation is extended according to the new index set. Except for the agents at the top and bottom of the chain, agents take on the role of both subordinates and superiors as they interact in both directions in the organizational hierarchy. The reward function can be expressed recursively as

( ) ( )( )

( )( ) ( )( ) ( ) ( )1 1111 1, , , 1A x A xAx A Ax Ax Ax

final x final xv v x v x v xr s s s r s b r b− −−−

⎡ ⎤= + ⋅ ⋅ −⎣ ⎦… (32)

with { }1,2,..., 1x m∈ − and state index function ( )xν . Equation (32) can also be expressed in explicit form:

( )( ) ( )( ) ( )11

11

xxAx Ax Ax Ai Aifinal j xv x v i

i j i

r r s r s b b−−

= =

⎡ ⎤⎛ ⎞= + ⋅ −⎢ ⎥⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

∑ ∏ . (33)

The reward for the last agent in the chain, agent Am, is according to (33) without the factor of ( )1 mb− , since agent Am does not make an incentive payment.


280

The transition probability of reaching a particular set of states given all agents’ actions is the multiplicative combination of all agents’ final transition probability functions. Thus, similar to equation (21) we get

( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( )( )1 2 1 2 1 1 11 1 2 1 2 11 2 1 2 1 1, , , , , , |A A Am A A Am A A A

mv v v m m vp s s s a a a p s a c c c c c cμ μ μ μ −⎡ ⎤= ± ± ± ± ⋅⎣ ⎦… … … …

( ) ( )( ) ( )( )( )

( )( )( ) ( ) ( )( )1 1 12 2 2

2 2 3 2 3 1 12 2 1 1| | |A m A m A mA A A Am Am Amm mv v m m v m mp s a c c c c c c p s a c p s aμ μ μ

− − −− −− −

⎡ ⎤ ⎡ ⎤± ± ± ± ⋅ ⋅ ± ⋅⎣ ⎦ ⎣ ⎦… … …

( ) ( )( ) ( ) ( )( )( )

111

1

| ,m m im

A jAx Ax Ax Ajjv x x j j

i xx j x

p s a f s sμ ν ν

−−++

== =

⎡ ⎤= +⎢ ⎥

⎣ ⎦∑∏ ∏ (34)

with action index function ( )xμ , as introduced earlier. In the analysis of the three-agent model in Section 5, we have seen that the decision and parameters of agent A3

do not influence transition function (27) that determines agent A1’s optimal incentive for agent A2. We will show that this result extents to multi-organizational-scale chains. In other words, we demonstrate that limited, local information can be sufficient for chain-wide optimal decisions.

Theorem 4: The optimal incentive levels for agent A2 through A(m-1) are

( ) ( )

( )1 1

* 2 1

2 1

A x A x

x Ax xx

bcρ ρ

ρ ρ

+ +−=

−. (35)

Proof: For the first two levels (agent A1 and A2), we receive the same results as before:

( ) ( )( ) ( ) ( )( )2 1 2 3 2 1 2 31 1 1 23 3| , , ,..., | , , ,...,A A A A Am A A A A Am

final finalm mE r a a a a E r a a a aμ μ μ μ≥ (36)

results in (26) and

( ) ( )( ) ( ) ( )( )3 1 2 3 4 3 1 2 3 41 1 1 1 1 24 4| , , , ,..., | , , , ,...,A A A A A Am A A A A A Am

final finalm mE r a a a a a E r a a a a aμ μ μ μ≥ (37)

in (29) for ( ) { }1,2xμ ∈ . For the next lower level, the evaluation of

( ) ( )( ) ( ) ( )( )4 1 2 3 4 5 4 1 2 3 4 51 1 1 1 1 1 1 25 5| , , , , ,..., | , , , , ,...,A A A A A A Am A A A A A A Am

final finalm mE r a a a a a a E r a a a a a aμ μ μ μ≥

(38)

yields

( ) ( )( )

4 42 1

3 2 2 3 33 2 2 2 1 2 13 2

A A

A A A Ab

c b cρ ρ

ρ ρ ρ ρ−

≥− − −

. (39)

Similar to the previous step, substituting *2b into (39) results in

( )4 4

2 13 3 3

3 2 1

A A

A Ab

cρ ρρ ρ

−≥

−. (40)

Transition function (40) is structurally identical to (29). The decision situation in both cases is also identical. Agents A3 and A4 both have superior agents that have been motivated to choose actions 1

Axa over the initially

preferred actions 2Axa . Result (40) repeats itself in a self-similar fashion for lower level agents. Due to the

similarity and repetition of the agents’ situation throughout the chain, we conclude that in general

( ) ( )( )( )

( )( ) ( ) ( )( )( )

( )( )1 1 2 1 1 21 11 1 1 1 1 22 2| ,..., , , ,..., | ,..., , , ,...,A x A x A x A k A x A xA Ak Am A Ak Am

final finalx m x mE r a a a a a E r a a a a aμ μ μ μ+ + + + + +

+ +≥

results in

( ) ( )

( )1 1

2 1

2 1

A x A x

x Ax xx

bcρ ρ

ρ ρ

+ +−≥

− for { }2,..., 1x m∈ − . (41)

and agents will choose the Pareto-efficient share coefficient according to (35). □ Agent Ax selects the optimal share coefficient value, such that agent A(x+1) chooses the cooperative action, and

merely needs to evaluate its own private information and the aggregate reward information of agent Ax, its immediate superior. Chain-wide optimal and cooperative behavior is possible with local interaction and information. To avoid free-riding, agents commit to the share coefficient in a bottom-up sequence, and – provided


281

that the participation constraint is met – all agents will choose non-zero share coefficients leading to chain-wide, cooperative behavior that is aligned with the goals of the top-most agent.

8. Conclusions This paper presents a three-agent, hierarchical decision-making model for a maintenance service operation. The model is extended for multi-level organizations with any number of hierarchical levels. The agent interaction is graphically represented through a dependency graph from which an algebraic formulation of the problem is derived. A game-theoretic analysis determines the agents’ equilibrium decision strategies. The model’s results provide guidelines for decision-makers in hierarchical organizations on (1) optimal incentive-levels, (2) information and communication requirements, and (3) optimal courses of actions. Furthermore, organizational designers are informed about organizational structures in which agents can coordinate incentives autonomously with little information and communication needs.

Provided that the participation condition is met, an agent’s optimal course of action is to communicate necessary information, share rewards, and select cooperative actions supporting its superiors. By exchanging limited and aggregated private information, agents can determine optimal, two-stage decision strategies. In stage one, agents choose optimal incentives (share coefficients); in stage two, agents select their best courses of action. As a result of their cooperative behavior, agents receive higher expected rewards compared to myopically selfish, i.e., non-cooperative, behavior.

In the context of our service operations example, the maintenance worker carries out a thorough inspection of the equipment, even though a superficial inspection seems more appealing at first. The maintenance supervisor sets a light workload level for the worker, which leads more likely to a high cost for the supervisor, but also to high customer satisfaction. The account manager benefits from high customer satisfaction through an increase in the customer’s likelihood to renew their service contract. A share of the manager’s higher expected reward is passed down to both agents providing incentives for them to behave cooperatively. References

Anandalingam, G., Apprey, V. 1991. Multi-Level Programming and Conflict Resolution. European Journal of Operational Research 51(2) 233-247.

Anussornnitisarn, P., Nof, S.Y., Etzion, O. 2005. Decentralized Control of Cooperative and Autonomous Agents for Solving the Distributed Resource Allocation Problem. International Journal of Production Economics 98(2) 114-128.

Bitran, G.R., Haas, E.A., Hax, A.C. 1981. Hierarchical Production Planning: A Single Stage System. Operations Research 29(4) 717-743.

Chang, T.-S., Ho, Y.-C. 1981. Incentive problems: A Class of Stochastic Stackelberg Closed-Loop Dynamic Games. Systems & Control Letters 1(1) 16-21.

de Kok, A.G. 1990. Hierarchical Production Planning for Consumer Goods. European Journal of Operational Research 45(1) 55-69.

Deng, X., Papadimitriou, C.H. 1999. Decision-Making by Hierarchies of Discordant Agents. Mathematical Programming 86(2) 417-431.

Dolgov, D., Durfee, E. 2004. Graphical Models in Local, Asymmetric Multi-Agent Markov Decision Processes. Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 2 956-963.

Duignan, B. 2007. Service Industry, Encylopedia Britannica. Last accessed: February 4, 2010, http://www.britannica.com/EBchecked/topic/535980/service-industry.

Geanakoplos, J., Milgrom, P. 1991. A Theory of Hierarchies Based on Limited Managerial Attention. Journal of the Japanese and International Economies 5(3) 205-225.

Guestrin, C., Koller, D., Parr, R., Venkataraman, S. 2003. Efficient Solution Algorithms for Factored MDPs. Journal of Artificial Intelligence Research 19 399-468.

Hax, A.C., Candea, D. 1984. Production and Inventory Management. Prentice Hall, Englewood Cliffs, NJ. Hax, A.C., Meal, H.C. 1975. Hierarchical Integration of Production Planning and Scheduling. North Holland,

Amsterdam. Heinrich, C.E., Schneeweiss, C. 1986. Multi-Stage Lot-Sizing for General Production Systems. Springer, Berlin. Henry, A., Wernz, C. 2010. Incentives in Multi-Organizational-Scale Service Enterprise Systems. Working Paper.


282

Ho, Y.-C. 1983. On Incentive Problems. Systems & Control Letters 3(2) 63-68. Howard, R.A., Matheson, J.E. 2005. Influence Diagrams. Decision Analysis 2(3) 127-143. Itoh, H. 1991. Incentives to Help in Multi-Agent Situations. Econometrica 59(3) 611-636. Jackson, C., Pascual, R. 2008. Optimal Maintenance Service Contract Negotiation with Aging Equipment. European

Journal of Operational Research 189(2) 387-398. Kearns, M., Littman, M., Singh, S. 2001. Graphical Models for Game Theory. Proceedings of the Uncertainty in

Artificial Intelligence Conference (UAI). Kim, J.-H., Park, J.-B., Park, J.-K., Kim, B.H. 2003. A New Game-Theoretic Framework for Maintenance Strategy

Analysis. IEEE Transactions on Power Systems 18(2) 698-706. Klock, B.A. 2004. Test and Evaluation Plan: Screener Effectiveness of Monetary/Non-Monetary Incentives.

Transportation Security Administration 60. Koller, D., Milch, B. 2001. Multi-Agent Influence Diagrams for Representing and Solving Games Proceedings of

the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI). Krothapalli, N.K.C., Deshmukh, A. 1999. Design of Negotiation Protocols for Multi-Agent Manufacturing Systems.

International Journal of Production Research 37(7) 1601-1624. Laffont, J.J. 1990. Analysis of Hidden Gaming in a Three-Level Hierarchy. Journal of Law, Economics, &

Organization 6(2) 301-324. Lin, J.T., Chen, T.L., Lin, Y.T. 2006. A Hierarchical Planning and Scheduling Framework for TFT-LCD Production

Chain. IEEE International Conference on Service Operations and Logistics, and Informatics. Luh, P., Chang, T.-S., Ning, T. 1984. Three-level Stackelberg Decision Problems. IEEE Transactions on Automatic

Control 29(3) 280-282. Marschak, J., Radner, R. 1972. Economic Theory of Teams. Yale University Press, New Haven. McKay, K.N., Safayeni, F.R., Buzacott, J.A. 1995. A Review of Hierarchical Production Planning and its

Applicability for Modern Manufacturing. Production Planning & Control 6(5) 384-394. Mesarovic, M.D., Macko, D., Takahara, Y. 1970. Theory of Hierarchical, Multilevel, Systems. Academic Press,

New York. Middelkoop, T., Deshmukh, A. 1999. Caution! Agent Based Systems in Operation. InterJournal of Complex

Systems 256. Murthy, D.N.P., Asgharizadeh, E. 1999. Optimal Decision Making in a Maintenance Service Operation. European

Journal of Operational Research 116(2) 259-273. Nie, P.-Y., Chen, L.-H., Fukushima, M. 2006. Dynamic Programming Approach to Discrete Time Dynamic

Feedback Stackelberg Games with Independent and Dependent Followers. European Journal of Operational Research 169(1) 310-328.

Pappas, G.J., Lafferriere, G., Sastry, S. 2000. Hierarchically Consistent Control Systems. IEEE Transactions on Automatic Control 45(6) 1144-1160.

Sandell, N., Varaiya, P., Athans, M., Safonov, M. 1978. Survey of Decentralized Control Methods for Large Scale Systems. IEEE Transactions on Automatic Control 23(2) 108-128.

Schneeweiss, C. 1995. Hierarchical Structures in Organizations: A Conceptual Framework. European Journal of Operational Research 86(1) 4-31.

Schneeweiss, C. 2003. Distributed Decision Making. Springer, Berlin. Schneeweiss, C., Zimmer, K. 2004. Hierarchical Coordination Mechanisms within the Supply Chain. European

Journal of Operational Research 153(3) 687-703. Schneeweiss, C.A., Schröder, H. 1992. Planning and Scheduling the Repair Shops of the Deutsche Lufthansa AG: A

Hierarchical Approach. Production and Operations Management 1(1) 22–33. Siemsen, E., Balasubramanian, S., Roth, A. 2007. Incentives That Induce Task-Related Effort, Helping, and

Knowledge Sharing in Workgroups. Management Science 53(10) 1533-1550. Song, Z., Chen, Z. 2006. Game Analysis of Gain-sharing Percentage for Technological Innovators. International

Conference on Management Science and Engineering, ICMSE '06. Tirole, J. 1986. Hierarchies and Bureaucracies: On the Role of Collusion in Organizations. Journal of Law,

Economics, & Organization 2(2) 181-214. van Ackere, A. 1993. The Principal/Agent Paradigm: Its Relevance to Various Functional Fields. European Journal

of Operational Research 70(1) 83-105. van Hoesel, S. 2008. An Overview of Stackelberg Pricing in Networks. European Journal of Operational Research

189(3) 1393-1402. Weiss, G. 1999. Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence. MIT Press,

Cambridge, MA.


283

Wernz, C. 2008. Multiscale Decision Making: Bridging Temporal and Organizational Scales in Hierarchical Systems. Dissertation, University of Massachusetts Amherst, MA.

Wernz, C., Deshmukh, A. 2007a. Decision Strategies and Design of Agent Interactions in Hierarchical Manufacturing Systems. Journal of Manufacturing Systems. 26(2) 135-143.

Wernz, C., Deshmukh, A. 2007b. Managing Hierarchies in a Flat World, Proceedings of the 2007 Industrial Engineering Research Conference.

Wernz, C., Deshmukh, A. 2010a. Multiscale Decision-Making: Bridging Organizational Scales in Systems with Distributed Decision-Makers. European Journal of Operational Research 202, 828-840.

Wernz, C., Deshmukh, A. 2010b. An Incentive-Based, Multi-Period Decision Model for Hierarchical Systems. Proceedings of the 3rd Annual Conference of the Indian Subcontinent Decision Sciences Institute Region (ISDSI).

Wernz, C., Deshmukh, A. 2010c. Multi-Time-Scale Decision Making. Working Paper. Zeithaml, V., Berry, L., Parasuraman, A. 1996. The Behavioral Consequences of Service Quality. The Journal of

Marketing 31-46. Zhang, X., Zhang, Z., Chen, Z. 2007. A Principal-Agent Problem IEEE International Conference on Grey Systems

and Intelligent Services (GSIS).

Dr. Christian Wernz is an Assistant Professor at the Grado Department of Industrial and Systems Engineering at Virginia Tech. He is the director of the Multiscale Decision Making Laboratory (MSDM Lab). Dr. Wernz received his Ph.D. in Industrial Engineering and Operations Research from the University of Massachusetts Amherst in 2008. He earned his B.S. (1999) and M.S.-MBA (2003) in Economics and Business Engineering (Diplom Wirtschaftsingenieur) at the Karlsruhe Institute of Technology (KIT, formerly University of Karlsruhe), Germany. In his dissertation, Dr. Wernz laid the foundation for multiscale decision theory. He continues his research on multiscale analysis and modeling of complex systems, cooperative and non-cooperative game theory, and multi-time-scale Markov decision processes. Together with his students at the MSDM Lab, Dr. Wernz is advancing the theory of multiscale decision-making and is modeling multiscale challenges in healthcare, service operations, technology management and climate change. Dr. Wernz is a member of INFORMS and IIE.

Andrew Henry is a Ph.D. student in Industrial and Systems Engineering at Virginia Teach. Before joining the ISE graduate program, Andrew graduated from Virginia Tech with a B.S. in ISE with a minor in Business. While working on his undergraduate degree, Andrew held several engineering internships and worked on an undergraduate research project investigating the influenza vaccine supply chain. More recently, Andrew worked on a consulting project with Meridium Inc. validating their software reliability suite. Andrew's research interests include game theory, decision theory, stochastic processes and reliability. His current research involves design and application of incentive mechanisms that align objectives in multiscale systems, particularly healthcare systems. Andrew is a member of the Institute for Operations Research and the Management Sciences (INFORMS), he is also president of the INFORMS student chapter at Virginia Tech.

Documents

Multi-Level Coordination and Decision-Making in … · Multi-Level Coordination and Decision-Making in ... customer satisfaction is an important ... theoretic framework where agents