Simulation Modelling Practice and Theory 20 (2012) 89–98

Contents lists available at SciVerse ScienceDirect

Simulation Modelling Practice and Theory

journal homepage: www.elsevier .com/ locate/s impat

Application of agent based modelling to aircraft maintenance manningand sortie generation

Adam MacKenzie a, J.O. Miller b,⇑, Raymond R. Hill b, Stephen P. Chambal b

a Air Force Analysis and Lessons Learned, 1570 Air Force Pentagon, Washington, DC, United Statesb Department of Operational Sciences, Air Force Institute of Technology, Wright-Patterson AFB, OH, United States

a r t i c l e i n f o a b s t r a c t

Article history:Received 26 August 2010Received in revised form 5 August 2011Accepted 7 September 2011Available online 13 October 2011

Keywords:Agent based modellingMilitary aircraft maintenanceExperimental design

1569-190X/$ - see front matter Published by Elsevidoi:10.1016/j.simpat.2011.09.001

⇑ Corresponding author. Address: AFIT/ENS, 29504943.

E-mail address: john.miller@afit.edu (J.O. Miller)

This research develops an agent based simulation model for application to the sortie gen-eration process, focusing on a single fighter aircraft unit. The simulation includes represen-tations of each individual maintainer within the unit, along with supervisory agents thatprovide direction in the form of dynamic task prioritization and resource assignment.Using a high-fidelity depiction of each entity, an exploration of the effects of differentmixes of skill levels and United States Air Force Specialty Codes (AFSCs) on sortie produc-tion is performed. Analysis is conducted using an experimental design with results pre-sented demonstrating the effects of maintenance manning decisions on the CombatMission Readiness (CMR) of a fighter unit.

Published by Elsevier B.V.

1. Introduction

Substantial time expenditures for both training and maintenance activities are required to ensure the constant readinessof operators and support personnel to support mission taskings within the United States Air Force. As with any complexorganization, metrics have been established for leaders to gauge progression and measure status of processes and systemscritical to mission accomplishment. A key metric used by leadership to gauge the Air Force’s instantaneous level of readinessto apply airpower is Combat Mission Readiness (CMR). Specifically, CMR is defined as ‘‘the minimum training required forpilots to be qualified and proficient in all of the primary missions tasked to their assigned unit and weapons system’’ [1].At its basest level, a pilot’s attainment of a full CMR rating depends on the completion of specific sets of training activities,chief among them the successful completion of a particular number of sorties over a defined period of time. Within the con-text of this document, the term ‘‘sortie’’ is defined as a single mission flown by an individual aircraft. ‘‘Sortie generation’’ andthe ‘‘sortie generation process’’ encapsulate the activities required to provide mission-ready aircraft to fly a specified sortie.These activities include various inspections, repair and preparation actions, all of which require the application of mainte-nance resources (manpower) to execute.

A previous analysis examined CMR in response to a tasking by the commander of United States Air Forces in Europe(USAFE) [2]. While not an exact measure of specific sortie generation levels, the CMR metric provides leadership a top-levelview of a unit’s overall readiness to execute their assigned mission at any given time. Lipina’s research developed a regres-sion model to determine the major factors driving CMR. His results showed that CMR depended in large part on availabilityof qualified aircraft maintenance manpower.

er B.V.

Hobson Way, WPAFB, OH 45433-7765, United States. Tel.: +1 937 255 6565x4326; fax: +1 937 656

.

90 A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98

This research develops an agent based simulation model for application to the sortie generation process, focused on a sin-gle fighter aircraft unit. Specifically, the research examines the potential implications of various manpower resourcing op-tions on a unit’s sortie production capacity. The simulation includes representations of each individual maintainer within theunit, along with supervisory agents that provide direction in the form of dynamic task prioritization and resource assign-ment. Using a high-fidelity depiction of each entity, an exploration of the effects of different mixes of skill levels and AirForce Specialty Codes (AFSCs) on sortie production is performed.

In the remainder of this paper we present a brief background on previous sortie generation modeling. Then we detail ourstudy methodology, results, and conclusions.

2. Background

The sortie generation problem is not new. There has been a host of research performed on the issue with objectivesspanning everything from general system observation and characterization to attempts to optimize one or more constituentsub-processes within the overall sortie generation process. These research efforts have employed many methods, includingdiscrete event simulation [3,4], Markov decision analysis [5] and neural networks [6]. Some of these efforts have even ad-dressed the specific issue of maintenance manning and its potential effects on sortie production and overall readiness [7].Regardless, the methodologies utilized follow a more traditional approach of decomposing the system under investigationand attempting to describe its behavior as the ‘‘sum of its parts’’, which has been shown to be ‘‘inadequate to model andanalyze’’ some large and complex systems [8]. In fact, research performed across multiple disciplines has shown that thesetraditional methods of system decomposition and subsequent reconstitution can prove not only inadequate but also canpotentially produce misleading results [9]. Kaegi et al. [8] further argue that in these situations, an Agent Based Model(ABM) has a ‘‘high potential to help realistically model large and complex systems’’.

3. An agent based sortie generation simulation

A simulation of the sortie generation process benefits from use of an ABM structure due to its identified complexity.While the process is arguably somewhat linear (inspect, fix, fuel, launch, catch, inspect, etc.) it is highly dependent on theindividuals involved in its execution. Decisions made by one central supervisory entity are subsequently interpreted andimplemented by subordinate entities, and then executed by additional sets of entities. Looking specifically at the inter-rela-tionships between maintenance personnel across a variety of skill levels and job specialties and their potential outputs interms of sortie production, an ABM provides a detailed individual-based perspective on the overarching process.

Our model focuses on the flight-line maintenance portion of the sortie generation process. We first discuss the data re-quired to capture various processes and how we incorporated these into our model. Then we describe the overall flow, goinginto details on the logic for each type of agent.

The model was developed using the NetLogo agent based modeling environment, developed at Northwestern University[10]. The development environment was selected based on ease of use and the system’s operating characteristics [11]. Onechief concern was the robustness of the random number generator, and the ability of the system to designate separate anddistinct random number seeds to specific processes in order to implement common random numbers as a variance reductiontechnique during the analysis period. Since the core NetLogo environment did not have the capability to track more than asingle random number seed at a time, an extension was developed to add this capability. The random number used withinthe environment is the Mersenne twister, proposed by Matsumoto and Nishimura [12]. An in-depth statistical evaluation[13] found the generator to perform as well as those used in a variety of commercial simulation software packages.

An analysis of the sortie generation process is a data intensive undertaking, even when examined at an aggregate or verytop level. As additional levels of detail were added to each of our agents, requirements for additional data increased. Primarysources included the Logistics Installations and Mission Support Enterprise View (LIMS-EV), and the Global Combat SupportSystem – AF Data Services (GCSS-AF), both web-based tools accessible via the Air Force Portal. Additional data was providedby USAFE/A9, Air Combat Command (ACC)/A4 and interviews with maintainers at both Shaw Air Force Base (AFB) and Spang-dahlem Air Base (AB). Table 1 lists key data requirements and their definitions.

A large number of discrepancies were noted during collection of the maintenance data from GCSS. See Mackenzie [14] formore details. Potential impacts due to inaccuracy of data within automated maintenance information systems have been dis-cussed in other studies [15]. Efforts were made to filter out inconsistencies that were readily apparent, but a key assumptionis that the remaining data used in our study is representative of true system behavior and performance.

While in some instances the raw data could be used directly (numbers of aircraft and personnel), in the majority of casesdistributions had to either be constructed or fitted to the gathered data using commercial tools such as Excel or the ArenaInput Analyzer. This was further delineated into those instances requiring the construction of simple distributions and thoserequiring the formulation of a more complex conditional structure. Distributions for break and abort rates are examples ofthe simpler case, where each rate was represented with a single theoretical distribution or empirical distribution function.Conversely, Work Unit Code (WUC) determination and fix rates provide an example of the more complex structure. Ratherthan utilizing a single distribution to characterize a general fix time for each break, our model stochastically determines aWUC for each break along with a specific AFSC set to work that WUC, and then utilizes a separate specifically fitted

Table 1Data requirements and definitions.

Data requirement Definition

Number of aircraft Number of aircraft modeledNumber of personnel Number of personnel modeledBreak rate % of sorties landing with major discrepancy requiring fixAbort rate % of sorties with issues preventing mission completionFix rates Time taken to fix aircraft (WUC/AFSC dependent)Work Unit Code (WUC)

determinationFailed aircraft system (WUC) for each break occurrence

AFSC assignment Determines job assignment of AFSCs (WUC dependent)Crew size determination Size of crew required for fix (AFSC/WUC dependent)Average Sortie Duration (ASD) Length of time each sortie lastsLearning curve Rate at which maintenance agents increase efficiency

attribute

A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98 91

theoretical distribution based on these conditions to calculate a fix time. This multi-tiered approach provides a more realisticportrayal of the process, both in terms of a more accurate depiction of manpower allocation (specific AFSCs) and a fix timeassociated with that allocation. A similar multi-tiered conditional structure is used for determination of crew sizes for eachjob.

The only exception to using historical data and a fitting process to either an empirical or a theoretical distribution was thedevelopment of the learning curve parameters. This data was obtained from interviews with senior maintenance personnelat Shaw AFB, and consisted of estimates of worker efficiency by both AFSC and skill level, along with an estimate of how longit would take to improve that efficiency to a level on par with the next highest skill level. ‘‘Efficiency’’ as an attribute wasboth discussed and modeled as a modifier to working speed. Specific details are included below in the discussion of themaintainer agent. Utilizing manpower availability estimates published in AFI 38-201 [16], these upgrade times and effi-ciency figures were used to form linear plots. The slope of each plot was then taken as the efficiency improvement learningcurve and used to calculate an efficiency improvement at the completion of every maintenance task. Essentially, as eachmaintenance agent works, the learning curve (slope of their efficiency improvement curve) is used to calculate the specificincrease in that agent’s efficiency attribute. This approach did not consider loss of learning due to missed training or lack oftask accomplishments. Fig. 1 depicts the learning curves calculated for the Avionics AFSC. Similar curves were constructedfor the remaining AFSCs.

Fig. 2 displays a top-level view of the modeled sortie generation process. Within the model, four separate agent typesinteract according to specific defined behaviors in order to accomplish the tasks making up various portions of the sortiegeneration process. The defined agent types are production supervisor, expediters, maintenance, and aircraft. The productionsupervisor agent provides general oversight and direction to the other agents. Expediter agents allocate personnel to theirassigned tasks and are broken down into crew chief, avionics and mechanical (electro-environmental and propulsion) spe-cialties. The maintenance agents serve as assignable resources and are further defined by their AFSC of crew chief, avionics,electro-environmental or propulsion; and skill level (three, five or seven-level with higher numbers representing increasedskill level). Each of the maintenance agents possess a learning curve which models their increase in efficiency (modeled as anincrease in working speed) over time. Finally, the aircraft agents are purely reactive and serve to keep track of sortie char-acteristics and some maintenance activities. The following section discusses the development of each of the above agents inmore detail.

4. Agent development

The power of an agent based model environment lies in its ability to codify an agent’s decision making processes intobehaviors and then observe the agents as they ‘‘autonomously’’ react and interact with other agents and their environment,potentially collectively producing ‘‘emergent behaviors’’ that might not otherwise have been either observed or predictedbased on other methods of analysis [17]. In the case of our sortie generation model, specific logic flows were mapped foreach agent type. Each flow was developed using the agent’s point of view and level of global ‘‘awareness’’, which varieddepending on the agent type. ‘‘Awareness’’ within the context of the model should be understood as one specific type ofagent’s level of knowledge of other entities within the model. For instance: the production supervisor agent was aware ofthe status of all aircraft, along with their flight schedules and whether maintenance actions (ongoing or pending) would pre-clude them from meeting a scheduled sortie assignment. This allowed this agent to execute his prioritization of maintenanceactivities and assign spare aircraft to specific sortie requirements as required. Conversely, each of the maintenance agentswas aware only of the task they had been assigned.

As defined in AFI 21-101 [18], the Production Supervisor (Pro Super) is responsible for directing ‘‘the overall maintenanceeffort of their unit.’’ As such, the Pro Super agent is the only one with true global awareness, and makes the majority of deci-sions within the simulation. These include job priorities as they arise, which aircraft are to be put into the flying schedule,how many and which aircraft are to be generated as spares, and when to begin work on what aircraft.

Fig. 1. Avionics learning curves.

Fig. 2. Modeled sortie generation process.

92 A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98

A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98 93

The Pro Super has two states: available/planning and Exceptional Release (ER) signoff. In the former, he is performingeach of the required decisions outlined above every time step. For the latter, he is considered unavailable while signingthe ER. As the individual with overall responsibility for maintenance execution, the Pro Supers’ signing of the ER ‘‘servesas certification. . .that the [aircraft] is safe for flight’’ [19] and is required prior to takeoff for each sortie. Fig. 3 depicts thePro Super agent logic.

Each of the three different specialty expediter agents has a reduced level of global awareness. All three are aware of tasksassigned to their specific specialties by the production supervisor as well as the current status, skill levels and efficiency val-ues of the maintenance agents that ‘‘belong’’ to them by virtue of their AFSC. Additionally, the crew chief expediter is awareof any impending aircraft landings, since aircraft tend to be recovered by crew chief personnel. Expediters also perform theallocation of maintenance technicians to tasks based on the task’s priority as assigned by the production superintendentagent. By definition, expediters ‘‘work for the Pro Super and manage, control and direct resources’’ in order to ‘‘ensure main-tenance is accomplished’’ [18]. Within their execution time step, each of the expediters scan for jobs in the system (taskingsfrom the production superintendent), and if any are found for the AFSCs they are responsible for (note above that each expe-diter is responsible for a specific subset of the overall mix of maintenance AFSCs) they proceed with their job assignmentlogic.

For expediters to assign a task to a group of maintainers, they must first determine if sufficient manpower is currentlyavailable. One key consideration is the number of maintainers required for each job. In many cases, specific tasks carry tech-nical order requirements for minimum crew sizes. While the current model does not include sufficient detail to capture theactual task level crew size requirements, this influence is captured by treating crew size as a random variable and drawingfrom an empirical distribution based on two years worth of data from Spangdahlem AB. Specifically the crew size is based onthe WUC of the job and the AFSC assigned to work the job. Depending on the priority of the task and the availability of per-sonnel working lower priority jobs, the expediter may pull personnel from lower priority jobs; delay the job; or work the jobwith the available (sub-optimal) manpower with a penalty on job completion time.

Additionally when assigning a job, expediters must determine when to allow training to occur, and whether or not train-ing (when allowed) will occur. This is determined based on the priority of the job (priority one jobs do not allow for training)and the skill level of the initially assigned team. Once a fully qualified (five and seven levels) team has been selected by theexpediter, a random draw is evaluated against the lowest efficiency value on the qualified team (discussed under maintaineragent). If the random number is lower, then training is permitted to occur and up to two three-levels are randomly selectedto be trained. The expediter agent logic is included in Fig. 4.

The individual maintenance agents have a minimum level of global awareness since the typical maintainer’s focus is onfixing, inspecting, or servicing an aircraft. In effect, each modeled maintainer resides in a ready pool until tasked to a job bytheir owning expediter. Logic to determine the nature, fix-time, crew size, etc. of the tasks is driven by random draws eval-uated against empirical or theoretical distributions derived from the data sources previously discussed. When assigned to ajob, each individual maintenance agent’s efficiency attribute is used to determine the speed at which it is accomplished. Anindividual agent’s efficiency value attribute varies from 0 (no skills) to 1 (highly skilled). This attribute is increased over timevia a learning curve function (like Fig. 1) developed through interviews with experienced maintainers both in the field and atthe Air Force Institute of Technology (AFIT). General maintenance agent logic flow is depicted in Fig. 5.

Fig. 3. Pro Super agent logic.

Fig. 4. Expediter agent logic.

Fig. 5. Maintenance (mx) agent logic.

94 A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98

Each aircraft agent is purely reactive and contains attributes to track completion of preflight inspections, sortie counts,and current status among other things, but does not employ any active decision making functions. Based on data obtainedfrom Spangdahlem AB, 22 aircraft were incorporated into the model. Break, abort and other data representing failures of theaircraft were gathered and utilized to construct empirical or theoretical distributions for use in describing the stochastic nat-ure of the various failure mechanisms.

The aircraft agent has two system states: Non-Mission Capable (NMC) and Fully Mission Capable (FMC). We do not in-clude a Partially Mission Capable (PMC) state since other model logic does not require a third state and this simplifies theoverall state space of the model. Fig. 6 provides the modeled flows for aircraft agents.

With logic for each of the agent types well defined, the final step in the model development was the design and imple-mentation of an overarching construct that established the agents, defined their environment and behaviors, and provided abackdrop to track their iterative execution. Specifically this included the addition of supporting logic and routines to manageprocesses such as shift change, input and output of data, and the establishment of a standard work week and flying schedule.As the model took shape, multiple assumptions had to be made in order to effectively scope the development effort. Theseassumptions included: flying window remains constant; sortie load retains a set weekly pattern; aircraft configuration is nota concern; scheduled maintenance is not modeled; and data used to determine underlying distributions is assumed to beaccurate and representative of the underlying real world systems. Throughout our model development, numerous tests wereconducted to verify the model coding and structure. In addition, subject matter experts were consulted to validate ourassumptions and logic and to observe the overall flow of our final model. The subject matter experts’ approval of the model’slogic and agent behavior sets, served to provide a top-level validation of the agent implementations. This, combined with the

Fig. 6. Aircraft agent logic.

A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98 95

previously mentioned debugging efforts, provided a verification that the model was indeed coded appropriately and from anagent behavioral standpoint was suitably valid to proceed with the experiment.

5. Experimental design and methodology

The key focus for our analysis was examining the effects of varied levels of maintenance manpower on CMR, measuredboth in terms of utilization and sortie production. Specific responses of interest were daily utilization rates for each AFSC andskill level; and sorties flown and cancelled per week. Factors identified for analysis were the number and skill level of per-sonnel within each AFSC. While it is acknowledged that different strategies for allocating personnel across shifts could gen-erate significantly different results, the vast number of potential combinations and approaches suggested limiting designcomplexity. For this research, four aggregated manning factors were utilized, one for each AFSC. Each of these had threelevels:

– Base: the typical manning profile based on Shaw AFB’s daily manning,– Reduced: a 10% loss of personnel, and– Increased: a 10% increase of personnel.

Rather than turn this analysis into an exploration for an optimum AFSC and skill mix for each shift in the face of changingmanning availability, both the reduced and increased cases were calculated in as straightforward a manner as possible. Un-der the assumption that a unit would maintain relatively the same proportions of skills and AFSCs on each shift regardless ofmanning changes, each shift and AFSC is evaluated individually. As an example, on day shift there are 30 crew chiefs in thebase case: 7 three-levels, 14 five-levels and 9 seven-levels. The reduced case then translates to a loss of three crew chiefs onthis shift. Using the original proportions of skill levels as a reference (23%, 46% and 30% for three, five, and seven-levelsrespectively), these proportions are applied to the reduced shift manning level of 27 crew chiefs. In this example, the reducedlevels of manning then equate to 6 three-levels, 13 five-levels and 8 seven-levels.

Despite this clear-cut approach at testing, the running of a complete 34 full factorial experiment with 81 design points(distinct simulation configurations) was estimated to take over fifteen days. However, since cross-utilization training wasnot modeled (there is no sharing of job taskings between AFSCs), it was believed that none of the identified responses wouldexhibit significant interactions between any of the experimental factors.

With the assumed lack of significant interactions in mind, a fractional factorial screening experiment was developed. Westarted by specifying a portion (fraction) of the previously identified 34 factorial experiment, reducing the computationalburden with fewer design points while still enabling us to examine (screen) the design space and obtain information onthe factors of interest. As suggested in Montgomery [20], we also simplified the factor space by including only the reducedand increased manning levels, with the base levels used as center points. This provided a means to identify curvature (non-linear response of a factor(s)) and test for lack of fit, while simultaneously minimizing the size and design complexity of theoverall experiment. The resulting screening experiment was a 24�1 design with 25 center points. This design enabled us toexamine each of our four factors at two levels (three with the centerpoints) while providing estimates of the main effectsconfounded only with the high-level 3-factor interactions. Since the high-level interactions were not assumed to be signif-icant, this gave us an indication of the significance of each of our four main factors without having to initially run the entirefull factorial experiment. Bottom line is that this experimental design allowed for insight into the likely system effects due tovariations in each of the identified factors while providing an indication of non-linear system response with a significantreduction in the number of simulation runs required.

For the execution of the experiment we selected a simulation replication length of 210 days. This equated to 7.5 ‘‘months’’of 28 calendar days (20 working days) each. This temporal abstraction was implemented in the interest of simplicity, as well

Table 2Screening test results.

Responses Significant factor(s) Responses Significant factor(s)

CC3 UTE CC, AV manning EE3 UTE EE manningCC5 UTE CC manning EE5 UTE CC, AV, EE manningCC7 UTE CC, AV manning EE7 UTE EE manningAV3 UTE AV manning JET 3 UTE CC manningAV5 UTE AV manning JET 5 UTE JET manningAV7 UTE AV manning JET 7 UTE N/ASorties/week N/A* Cancels/Week N/A*

Legend: CC – crew chiefs, AV – avionics, EE – electro-environmental, JET – propulsion.* P-value for significance (F-test) greater than 0.05, but less than 0.1.

96 A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98

as to more closely align with standard Air Force availability planning factors of 20.9 assigned days per month [16]. The figureof 7.5 months was selected based on a standard of 6 months for typical manning studies [21], with additional time added toallow for data truncation. Initial tests revealed no definitive warm-up period for the simulation, which was expected due tothe cyclical nature of the flying schedule implemented in our model. However, since time-keeping logic within the modelallowed for six days of flying the first simulated week, this entire week was truncated prior to commencement of any anal-ysis. A variance assessment was performed on the responses of interest, and it was determined that after 20 replications,variance remained relatively constant; thus, 25 replications per experimental treatment were used.

Table 2 depicts the results of the screening experiment, which were somewhat at odds with the original expectations.Namely, rather than individual AFSCs showing as significant, the results indicated a variety of interactions between variousdisparate AFSCs. However, upon further reflection, this became not only justifiable but an obvious potential system response.

First we examined the emergence of multiple significant factors for various individual responses. Looking specifically atthree-level crew chief (CC3) utilization (UTE) as an example, it is arguable that even though each AFSC works independently,a significant number of crew chief and avionics jobs might be needed in extended maintenance on broken aircraft, especiallyif either or both of these manning pools was affected in some manner. This would result in an elevated maintenance priorityfor aircraft slated to fly, which would then prohibit the completion of any training, the end result being a decrease in thethree-level utilization rate. These results allowed for a reduction in scope of the experiment to a 24 full-factorial, reducingthe overall testing to less than a quarter of the initial design.

6. Results and analysis

The running of the remainder of the 24 full-factorial experiment served to further solidify the assessments made on theinitial screening experiment. While certain AFSCs and skill levels exhibited multiple significant factors, each of these is easilyattributable to causes similar to those discussed above. Results from the 24 full-factorial experiment are included below inTable 3. Changes from the initial screening experiment results are indicated by italicized text within the table.

Similar to the initial findings discussed above, these results indicate a surprising number of statistically significant rela-tionships between individual AFSCs that were not predicted. An initial concern was that the results might be due to someinvalidity of the fundamental distributional assumptions for the analysis of variance. Other than some slight departures fromnormality in the tails of the analyzed residuals, however, the underlying assumptions of normally and independently distrib-uted errors with constant variance were verified to hold. Also, as one considers the effects of dynamic reprioritization ofmaintenance, it becomes easier to visualize that these are the apparent effects of immutable production requirements beinglevied upon a dynamic grouping of resources. In a real world sense this represents a unit’s production staff waging their dayto day battle of meeting the flying mission while simultaneously attempting to provide sufficient training opportunities tojunior troops. As available qualified resources become scarce, they are forced to sacrifice training in order to maintain levelsof production necessary to meet mission requirements.

Table 3Results of full 24 experiment.

Responses Significant factor(s) Responses Significant factor(s)

CC3 UTE CC, AV⁄ manning EE3 UTE EE manningCC5 UTE CC manning EE5 UTE CC, AV⁄, EE, JET⁄ manningCC7 UTE CC, AV⁄ manning EE7 UTE CC, AV, EE manningAV3 UTE AV manning JET 3 UTE AV, JET⁄ manningAV5 UTE AV, JET⁄ manning JET 5 UTE CC, JET manningAV7 UTE AV manning JET 7 UTE CC⁄ manningSorties/week AV manning Cancels/Week AV manning

Legend: CC – crew chiefs, AV – avionics, EE – electro-environmental, JET – propulsion.* P-value for significance (F-test) greater than 0.05, but less than 0.1.

0 5 10 15 20 25 300

5

10

15

20

25

Time

# C

ance

ls/W

eek

Weekly Cancels, Decreased Manning

Raw Streams

Column Average

0 5 10 15 20 25 300

5

10

15

20

25

Time

# C

ance

ls/W

eek

Weekly Cancels, Increased Manning

Raw Streams

Column Average

Fig. 7. Comparison of cancels per week.

A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98 97

Looking at a measure more directly related to sortie generation, Fig. 7 provides a graphical comparison of weekly cancel-lations over time between the reduced manning (top) and increased manning (bottom) scenarios. In the reduced case, noticethat the slope of the data’s average is relatively flat, indicating that maintenance is in ‘‘survival mode’’, essentially striving tomaintain one specific level of performance. Conversely, the increased manning scenario provides the capacity for training ofjunior personnel to occur with greater regularity, which results in a net increase in capacity as the unit’s overall average skilllevel increases. The end result is a decrease in overall cancellations per week as time goes by. Additional details on the anal-ysis performed can be found in Mackenzie [14].

7. Conclusions

The nature of the results presented in our study provide a level of fidelity unavailable from both current and past meth-odologies surveyed in terms of details on individual AFSCs and skill levels. Since many of the processes external to the core

98 A. MacKenzie et al. / Simulation Modelling Practice and Theory 20 (2012) 89–98

sortie generation process are abstracted out of our model, some might consider the specific utilization rates produced by themodel of little value. However, considering the model presents a best case scenario in which maintenance personnel are re-quired to deal only with the unscheduled maintenance items that crop up on a day to day basis, this model provides signif-icant insight into the relationships between specific AFSCs and skill levels and their effects both on changes in AFSCutilization as well as sortie production capacity. Our study clearly demonstrates that differing mixes of skills within individ-ual AFSCs can exert significant influence on a unit’s capacity and capability, and shows the merit of using an agent basedmodeling framework to capture many of the dynamic relationships that drive the complex processes involved in sortiegeneration.

Acknowledgments

The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position ofthe US Air Force, the Department of Defense, or the US Government.

References

[1] Department of the Air Force, Flying Operations (AFI 11-2F-16V1), 19 January 2007.[2] A.J. Lipina, Identifying Critical Factors Affecting Combat Mission Ready Status Among USAF Europe’s Aircrew, Graduate Research Project, School of

Engineering and Management, Air Force Institute of Technology, Wright Patterson AFB OH, June 2009.[3] P.D. Faas, Simulation of Autonomic Logistics System (ALS) Sortie Generation, MS Thesis, School of Engineering and Management, Air Force Institute of

Technology, Wright Patterson AFB OH, March 2003.[4] K. Iakovidis, Comparing F-16 Maintenance Scheduling Philosophies, MS Thesis, School of Engineering and Management, Air Force Institute of

Technology, Wright Patterson AFB OH, June 2005.[5] D.C. Dietz, Optimal Specialization and Allocation of Maintenance Manpower, PhD Dissertation, The Pennsylvania State University, University Park PA,

1991.[6] J.M. Dagg, An Exploratory Application of Neural Networks to the Sortie Generation Forecasting Problem, MS Thesis, School of Systems and Logistics, Air

Force Institute of Technology, Wright Patterson AFB OH, September 1991.[7] G.A. Gotz, R.E. Stanton, Modeling the Contribution of Maintenance Manpower to Readiness and Sustainability, Contract MDA903-85-C-0030, RAND

Corporation, Santa Monica, CA, 1986.[8] M. Kaegi, R. Mock, W. Kroger, Analyzing maintenance strategies by agent-based simulations: a feasibility study, Reliability Engineering and System

Safety 94 (9) (2009) 1416–1421.[9] G.V. Bobashev, D.M. Goedecke, F. Yu, J. Epstein, A hybrid epidemic model: combining the advantages of agent-based and equation-based approaches,

in: S.G. Henderson, B. Biller, M. Hsieh, J. Shortle, J.D. Tew, R.R. Barton (Eds.), Proceedings of the 2007 Winter Simulation Conference, IEEE Press,Piscataway, NJ, 2007, pp. 1532–1537.

[10] Wilinsky, Uri. Netlogo. Version 4.1RC3, Windows. Computer Software. Center for Connected Learning and Computer-Based Modeling, NorthwesternUniversity, Evanston, IL. <http://ccl.northwestern.edu/netlogo/>, 1999.

[11] Steven F. Railsback, Steven L. Lytinen, Stephen K. Jackson, Agent-based simulation platforms: review and development recommendations, Simulation82 (9) (2006) 609–623.

[12] Makoto Matsumoto, Takuji Nishimura, Mersenne Twister: A 623-dimensionally equidistributed uniform pseudo-random number generator, ACMTransactions on Modeling and Computer Simulation 8 (1) (1998) 3–30.

[13] L’Ecuyer, Pierre, Software for uniform random number generation: distinguishing the good and the bad, in: Proceedings of the 2001 Winter SimulationConference, vol. 1, December 2001, pp. 95–105.

[14] A.S. MacKenzie, An Exploration of the Effects of Maintenance Manning on Combat Mission Readiness Utilizing Agent based Modeling, MS Thesis,School of Engineering and Management, Air Force Institute of Technology, Wright Patterson AFB OH, March 2010.

[15] C.J. Dahlman, R. Kerchner, D.E. Thaler, Setting Requirements for Maintenance Manpower in the US Air Force, Contract F49642-01-C-0003, RANDCorporation, Santa Monica, CA, 2002.

[16] Department of the Air Force, Determining Manpower Requirements (AFI 38-201), 30 December 2003.[17] E. Bonabeau, Agent-based modeling: methods and techniques for simulating human systems, Proceedings of the National Academy of Sciences of the

United States of America 99 (10) (2002) 7280–7287.[18] Department of the Air Force, Aircraft and Equipment Maintenance Management (AFI 21-101), 29 June 2006.[19] Department of the Air Force, Maintenance Data Documentation (TO 00-20-2), 15 April 2007.[20] D.C. Montgomery, Design and Analysis of Experiments, seventh ed., John Wiley & Sons, Hoboken, NJ, 2009.[21] F. Juarez, LCOM Program Manager, Air Force Manpower Agency, Telephone Interview, 28 January 2010.