- Simulation of a Model Tank Gunnery Test.pdf

Embed Size (px)

Citation preview

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    1/84

    ARl TECHNICAL REPORTTR-79-A6LEVELSlomletiom of e Model Tenk Gammery Test

    byPaul W. Fingorman and George R. Wheaton

    C"O AMERICAN INSTITUTES FOR RESEARCH

    ~~ and 4C? G. Gary Boycan

    t, ARI Engagement Simulation Technical Area

    lrcl 1979

    Contract DAHC-19-76-C.O003Prepared for.LLJ __j

    C U.S. ARMY RESEARCH INSTITUTEB for the BEHAVIORAL and SOCIAL SCIENCES5001 Eisenhower AvemueAlexemdrl, Virginla 22333

    Approved for public relebse; distribution unlimited..79 08C06. 122/

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    2/84

    U. S. ARMY RESEARCH INSTITUTEFOR THE BEHAVIORAL AND SOCIAL SCIENCESA Field Operating Agency under the Jurisdiction of theDeputy Chief of Staff for Personnel

    WILLIAM L. HAUSERJOSEPH ZEIDNER Colonel, U S ArmyTechnical Director Commander

    NOTICES

    DISTRIBUTION: Primary distribution of this report ha s been made by ARI. Plom address correspondenceconcerning distribution of reports to: U. S. Army Research Institute for the Behavioral and Social Sciences.ATTN: PERt-P, 5001 Eisenhower Avenue, Alexandria. Virginia 22333.

    FINAL DISPOSITION: This report m ay be destroyed when it is no longer needed. Please do no t return it tothe U. S.Army Research Institute for the Behavioral and Social Sciencme.

    NOTE: The findings in this report are not to be construed as an official Department of the Army position,unless so designated by other authorized documents.

    4 ~.- - - - --.--C -

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    3/84

    UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE (UM Date auterve

    READ sVMUC'notisREPORT DOCUMENTATION PAGE BEFORE COMPLETDG FORMXi-R- - R 1, - OVTI ACCESSION NO. S. RECIPIENT'S CATALOG NUMBER

    4. TITLE (and Subtitle) S. TYPE Of REPORT & PERIOD COVERED/ Final ReportSIMULATION OF A MODEL TANK GUNNERY TEST,0 9/77 - 9/78

    . ONTRACT O R R NPaul W. Fingerman, George R jwhaton-.-G. Gary/BoycanP-04- AH 9AH 976C ;;09. PERFORMING ORGANIZATION NAME AND ADDRESS 10 ROGRAM ELEMENT, PROJECT. TASKAmerican Institutes for Research RF ' WOR IT U R

    1055Thoms JefersntNW2Q163743A78,{r

    WashinUnclasDCifiedI1. DISTROUTIONGSTICATMEN ADD51f.RESS

    UArv eeafrcublitleae; distriBuhaio anlimited.9

    AlxadraAn22333Se 7 -Sp 8

    IS. DSTUPPEMNTANTESET(fti eo t

    19. KEY WORDS (Continue an sever.. side It necessary a"s Identify by block number)Criterion-referenced testingsimulationtank gunnerycrew marksmanshipevaluation

    R ~ACT rCwt am re est~ d N neree m IdlenUtfy, b lc k number)is report describes the activities conducted during Phase 11 of a projectdevoted to the development of methods for assessing tank crev marksmanship

    performance. An earlier report on Phase I (ARl report TR 78-A24. AD A061 153)presented methods for the development of a criterion-referenced test of marks-manship. This effort led to the specification of a model livef ire test of tankcrew gunnery, together with scoring and test administration procedures fordetermining crew qualification. The present effort was aimed at evaluatingD JAIN 73 TO FINO . SL Unclassified (s. i *sS"O A

    .* S6CUm~rV CLASSIFICATION OF THIS PAGE (Whensnto Euslere

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    4/84

    UnclassifiedIECUREVV CLAW04PCATION OP THIS PAO~efbw DO& aftfo*

    - 0'"4h feasibility of assessing crew gunnery through the use of simulated,rather than livefired, teat exercises. Thirty-nine simulator devices wereevaluated; two Were identified as viable candidates for use in a simulated test.The analytic methodology is described, and a proposed simulated model test ispresented. The report concludes with a discussion of the evaluation proceduresthat are required if the simulated model test is to be considered for use as asubstitute for the livefire test..

    Unclassified*SMCUMnTY CLASSIFICATION Of THIS PAGE(Wfhen Data S0F0;0*

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    5/84

    ACKNOWLEDGMENTS

    The authors gratefully acknowledge the contributions ofthe many individuals who assisted in the research reportedherein. Lieutenant Colonel R. M. Boss6fman and Mr. Robert K.Bauer of the Directorate of Training Developments, U.S. ArmyArmor School at Fort Knox aided greatly in the conceptualiza-tion of Armor training and testing throughout the project.Staff Sergeant Rodney Caesar, of the Main Battle Tank Branch,Weapons Department, Armor School, deserves our special thanksfor his painstaking and insightful review of the candidatesimulators and their capabilities. His special efforts werecritical to the success of the project.

    iii

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    6/84

    SIMULATION OF A MODEL TANK GUNNERY TESTBRIEFRequirement:

    To develop a simulated version of a model livefire tankgunnery test that can be used to evaluate crew marksmanship.By using simulation to reduce testing costs, more engagementscan be added to increase confidence in the accuracy of crewqualification decisions and to provide more diagnostic infor-mation about crew training needs.Procedure:

    The feasibility of using simulation techniques as cost-effective alternatives to livefire testing was examined fora model livefire test of tank gunnery developed in Phase 1 ofthe project. The model test takes into consideration differenttypes of target engagements as well as the behaviors of theindividual crew members that are required during firing.The goal was to identify existing simulators or training devicesthat might best be used to simulate some or all of the modellivefire test exercises.

    The analysis consisted of several steps. Devices andsimulators were identified from a variety of sources and com-piled into an initial list of 39 candidates. These werescreened to cull out those failing to meet certain basicrequirements. The 14 remaining candidates were then evaluatedwith respect to the specific types of engagement conditionsthey could simulate. This information was then used to examinethree finalist devices with respect to the specific exercisesthat they could simulate, the exercises being drawn from themodel test and various versions of Table VIII. The quality ofsimulation was determined by considering the crew behaviorsin each exercise and assessing the extent to which each behaviorwould be produced in the simulated version. A simulated testwas then constructed that appeared promising enough to warrantempirical tryout and evaluation.

    Test evaluation issues were also studied. Developmentof a formal plan for empirically assessing the reliability andvalidity of the simulated marksmanship test was viewed asabsolutely essential.

    iVI4

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    7/84

    Findings:Two devices were identified that potentially could beused to simulate a test of crew marksmanship. One deviceconsists of the M2 .50 caliber machinegun affixed to an

    operational M60AlA0S tank by means of a Telfare mount. Itcan be used on a 1/2-scale or full-scale range facility. Theother device is the Tank Appended Crew Evaluation Device(TACED) which provides gun camera pictures of the crew's aimingperformance. Because of their complementary features, it isrecommended that both be used to support the simulated testingof livefire main gun engagements.The simulated test is a variant of the model livefiretest. There are 13 simulated main gun exercises and 15 live-fired machinegun exercises on which crews are tested duringdaylight and nighttime conditions. However, each of the simu-lated main gun engagements is repeated three times in an attemptto increase test reliability. The resulting 54 test exercises

    are designed to provide estimates of crew proficiency on the266 objectives comprising the domain of tank gunnery.The accompanying evaluation plan provides for an empiricalassessment of the test using one of two different experimentaldesigns. The pros and cons of each design are discussed asare practical considerations such as the requirements for thenumber of crews participating, and the advisability of differentshortcuts when implementing the evaluation study.

    Utilization of Findings:Crew proficiency in the use of tank weapons is a majorgoal of gunnery training and evaluation. Recently, however,the Army has decided to shift much of the ammunition allocated

    for training and evaluation from crew marksmanship to section,platoon, and higher element tactical gunnery. The simulatedmodel test of crew marksmanship can expedite this shift whileimproving evaluation of crew proficiency in operating thetank weapon system. To reach this objective the empiricalfield study recommended in the report should be carried out.

    vi

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    8/84

    TABLE OF CONTENTSSection PageBRIEF ......... ......................... . ivI. INTRODUCTION ...................... .. 1Purpose of the Research ............ 2II. ANALYSIS OF GUNNERY SIMULATORS .... .......... 4

    Step 1 - Compilation of Devices .. ......... 4Step 2 - Initial Screening of Devices ..... . 10Step 3 - Analysis of Engagement Conditions. .. 10Step 4 - Analysis of Marksmanship Exercises . . 13Step 5 - Analysis of Behavioral Elements . ... 14Step 6 - Specification of Simulated Tests . . . 16

    Test Exercises .... ............... . 17Simulators ..... ................. . 18Scoring....... ................... . 18Summary....... ................... . 20

    III. EVALUATION OF SIMULATED GUNNERY EXERCISES . . .. 21Goals of the Empirical Evaluation Study . . .. 21

    Objective 1: Assess reliability of thelivefire test ........ ......... 22Objective 2: Assess reliability of thesimulated test ........ . . 24

    Objective 3: Assess the'validity of thesimulated test..... ........ 24Objective 4: Explore different amounts ofsimulated testing .... ............. . 25

    Experimental Design ................ 26Non-replicated designs .. ........... . 26Replicated designs .... ............. . 32Number of subjects .... ............ . 34Reduced level of effort............... 35

    Implementation Guidelines .. ........... . 36REFERENCES ........ ...................... 41APPENDIX A - Devices Eliminated During InitialFeasibility Screening .. .......... . 44APPENDIX B - Engagement Conditions not Simulated on

    Specific Devices.... ............. . 48

    vii

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    9/84

    TABLE OF CONTENTS (cont'd.)Section PageAPPENDIX C - Evaluation of Three Simulators in Termsof Selected Main Gun Engagements....... 54APPENDIX D - Simulated Model Test of Tank CrewMarksmanship..... ............... ... 60APPENDIX E - Reliability and Validity: Implicationsfor Evaluating the Simulated Test .... 64

    viii

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    10/84

    LIST OF TABLESTable Description PageTable 1. Model Test of Crew Marksmanship: Part I.Main Gun ..... ................. 5Table 2. Model Test of Crew Marksmanship: Part II.

    Machinegun ....... ............... 6Table 3. Candidate Scaled-Range Devices. .. .... 8Table 4. Additional Candidate Devices ... ...... 9Table 5. Engagement Conditions and Levels ofConditions ..... ................ 11Table 6. Ordering Engagements for Control of Carry-

    Over Effects (Daylight Exercises of ModelTest)...... .................. . 27Table 7. Counterbalanced Arrangement for the Non-Replicated Design... ............ . 30Table B-i. Engagement Conditions Not Simulated onSpecific Devices ... ............ . 50Table C-1. Analysis of Main Gun MarksmanshipExercises: Model Test.............. 56Table C-2. Analysis of Main Gun MarksmanshipExercises: Table VIII .............. 57Table C-3. Analysis of Main Gun Marksmanship

    Exercises: Draft Revised Table VIII. . . 58Table D-1. Simulated Model Test of Tank Crew

    Marksmanship: Daylight Engagements . . . 61Table D-2. Simulated Model Test of Tank Crew

    Marksmanship: Nighttime Engagements. . . 62Table D-3. Crew Qualification Decisions . . . . . . 63

    ix

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    11/84

    I. INTRODUCTION

    The U.S. Army must maintain and document the readinessof its armored forces to move, shoot, and coordinate at everytactical level. The foundation for this readiness is thecapacity of the tank crew to use all of its weaponry to neu-tralize a vari-ety of threat targets.In order to evaluate their proficiency at neutralizingtargets, the U.S. Army requires its tank crews to qualifyannually in gunnery. The specific gunnery exercises that havebeen used to evaluate crew proficiency, together with the scor-ing systems and qualification standards that have been appliedto their performance, are defined by Table VIII in FM 17-12(1977) and FM 17-12-2 (1977). These exercises have been selected

    and developed on the basis of competent opinion and the judgmentof experienced armor personnel who, believing that comprehen-sive and exhaustive testing of crew capabilities is impossiblebecause of resource constraints, have attempted to distill theessence of gunnery into a manageable set of test exercises.But, test developers have included and excluded exercises frompast, present, and proposed gunnery tables without makingexplicit the rationales for doing so. Moreover, they haverelied on the costly use of full-caliber main gun ammunition,where simulation techniques might have been equally effective.In the past these costs have ranged from $101.00 for a HEP-TP-T round to $242.00 for a SABOT service round. The totalcost of the 27 main gun rounds allocated for one firing ofthe Table VIII described in FM 17-12-2 (1977) is about $4000;approximately $200,000 per battalion.In response to these problems the U.S. Army ResearchInstitute under U.S. Army Armor School sponsorship has supporteda program of research concerned with the development of cost-effective techniques for evaluating crew weapons proficiency.The need for such research was verified during Phase I of theproject (Wheaton, Fingerman, and Boycan, 1978). A review ofgunnery tables indicated that there were a number of redundantexercises on the one hand, while on the other, entire classesof exercises had been omitted from the tests. Nor were theexercises representative of the entire set of crew gunnerybehaviors. Therefore, one could have little confidence whenmaking general inferences about a crew's qualifications basedon its (limited and unrepresentative) test performance.To address this problem a set of systematic, analytictechniques was developed for constructing crew qualificationtests. The techniques were used to recommend exercises for

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    12/84

    inclusion in a model test of crew marksmanship. Emphasiswas given to the development of explicit testing rationales,together with associated scoring procedures, based on the issuesof test content and purpose. The objective agreed upon wasconstruction of a test that would be optimal for crew qualifi-cation but that would also serve a variety of other purposes.

    The 28-item test that was ultimately developed satisfiedthis objective and met a number of other criteria consideredessential to the design of an effective test. First, atleast one highly representative exercise was included fromeach major family of gunnery objectives. This step provideda basis for inference7 to be drawn about the quality of per-formance in each family and about proficiency in the gunnerydomain as a whole. Second, the exercises spanned the range offiring conditions under which engagements might occur. Third,the test exercises required the crew to demonstrate theirability to perform nearly all of the 112 crew behaviorsinvolved in tank gunnery. Most importantly, the resultingmarksmanship table served as a model example of the kind ofhighly effective test that could be developed in other settingswith other weapon systems, were the same explicit test construc-tion techniques used.PURPOSE OF THE RESEARCH

    The research described in this report represents thesecond phase of the larger program mentioned above. Thesecond phase of the effort examined the feasibility of usingsimulation techniques as cost-effective alternatives to live-fire testing. Impetus for the examination came from theArmy's recent decision to invest more of its ammunition allo-cation in the evaluation of section, platoon, and higherechelon tactical gunnery. If feasible, the use of simulationtechniques in assessing crew marksmanship would clearlyexpedite such a shift. At the same time, having drasticallyreduced crew qualification testing costs, more engagementscould be added to the crew marksmanship table to increaseconfidence in the accuracy of qualification decisions, and toprovide increased diagnostic information about crew strengthsand weaknesses.

    The remainder of this report describes the search forviable simulations of a marksmanship table and discusses thesteps required to evaluate their effectiveness as tests. Inthe next section candidate devices are identified and theprocess and results of a rigorous screening program aredescribed. The potential of the most useful devices for designof simulated marksmanship tests is then discussed. The lastmajor section deals with the complex issue of device evaluation.

    2

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    13/84

    Traditional concepts of reliability and validity are discussedin light of the nontraditional criterion-referenced marksman-ship table. Experimental designs and resource requirementsfor evaluation procedures are then described. This discussionis intended to be of direct benefit to those who must empiri-cally evaluate the feasibility of replacing livefire withsimulation alternatives.

    3

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    14/84

    II. ANALYSIS OF GUNNERY SIMULATORS

    Given a model livefire test of crew marksmanship (Tables1 and 2), the focus of the project shifted to the issue ofcost-effectiveness in marksmanship testing. Specifically, theeffectiveness of using simulation techniques in place of expen-sive livefire testing was examined.The goal was to identify existing simulators or trainingdevices that might be used to simulate some or all of themodel livefire exercises, and to develop a simulated versionof the test that could be substituted for livefire exercises,especially those involving the main gun. To accomplish thisobjective an analysis was undertaken consisting of six steps:Devices and simulators were identified and compiled into aninitial list of candidates. These were screened to cull outthose failing to meet certain basic requirements. The remaining

    devices were then evaluated with respect to the specific typesof engagement conditions that they could or could not simulate.This information was used to evaluate each of the still viablecandidates vis A vis each specific exercise in the model test,as well as exercises included in various versions of TableVIII. The quality with which each exercise was simulated wasdetermined by considering the crew behaviors involved in eachexercise, and assessing the extent to which each behaviorwould be reproduced in the simulated version. The final stepwas to construct simulated model tests that appeared promisingenough to warrant empirical tryout and evaluation. Each ofthe six steps is described in the remainder of this section.STEP 1 - COMPILATION OF DEVICES

    Candidate training devices and simulators were gleanedfrom several sources. Many types of relevant documentationwere surveyed including Chapter 18 of FM 17-12 (1977), DraftTC 17-12-7 (1976), TRADOC Pam 71-9 (1976), the Index andDescription of Army Training Devices (DOA Pam 310-12, 1972),and relevant articles in Armor magazine. These materialswere then reviewed and augmented during discussions withcognizant subject matter experts from the Devices Branch,Collective Training Division, Directorate of Training Develop-ments (DTD), and the Weapons Department, U.S. Army ArmorSchool (USAARMS), Fort Knox, Kentucky. The search resultedin the identification of some 39 devices that held varyinginitial degrees of promise for implementation of simulatedmarksmanship exercises.

    4

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    15/84

    Table 1. Modal Test of Crew Marksmanship: Pert 1. Main Gun

    Unit Tanks rew TCGunner Loadle, Drive,

    NO OFEXERCISE CONDITIONS ROUNDS AMMUNITION STANDARDI MOVING TANK Battlesighi 2 ITPOST Engagen 6 seconds.

    1 a0al Hit nl10sconds.Telescope2 TANK FRONT SHOT Bassight 2 HEAT TP.T Engagen 5 seconds.10011m Hit in 10 seconds.On thiemeGunner*& periscope3 MOVING TANK Precision 2 TPDST Engagen 10 seconds. --

    ITC) 1700Gml Hit in IS seconds.Raingefindeir HAT

    Day 4 MOVING TANK Precision 2 HA PT Engagen 10 seconds.Time 170Cma.al Hit in 1 seconids.Telecp___ _

    5 TANK IILHOIJETTE Precision 2 ITPDS-T Engagen 10 secondL200Gm Hit nIS seconds.Moving toea altGunner's periscope6 BUNKER/ Precision 2 HEPTP-T Engagesn 10 seconds.CREW WEAPON 2200mu Hit inISsconds

    11TC) * Movi toall ~ - __7 MOVING TRUCK Precision i 2 HEP-TP-T IEngage in 10 seconds.I 20Gm HltivilSacodMoving to ahaft1 MOVING TANK Rangearday 1o 2 HEAT.TP.T Hit within nisnuis ofsorect fire reaching releranced poaition.

    Stationery vehicle_______________ Telesope, flire -2 TROOPS Range card lay to 2 APERS Hlit ninnsnutes ofdirect fire reaching rae frne Position.

    Stationary vehicleGunner Periscope,infrared -_ _ _3 MOVING TANK Range card ay to 2 TPDST Hit within 5 ninuitsofIihC) direct fire reaching referena Psition.

    Tim Stationery vehicle4 TANK FRONT SHOT i ettlesigist 2 HEAT-TPhT EngaentO scod

    Staioer vehicle IHtwti SscnsGunner's Perope* infrared __S TROOPS Precision 2 APERS Engagen ISseconds.1700meiei Hitriithin 20 scds.Gunner'sprsoe lrS TANK FRONT SHOT biisgt2 TPOS.T Engage tO amd

    ITC On fteimoveHi mlhnIRangilefinder, flare _________ ________

    NOTES t. owring conduct of hie ie. target astoM i IsMso 'asfound I. NOdine soKcon4rsun lii Oflrd2 . Csrfius sonis 5 i. li In";he0eli southed=0in - peseointsre sin forasersunidn aotmm oo.endmond rou nsst rfife fin reundlts.3LCrew utis wieNOTsss4i. Threemarlsnun oundsuve bees llocated for wer~ir. end arecoflrneiirri liws roundsorder. one round or sirl).TheNewsxpensiveound HEP.TP I shrouldeusedorwnmsp purtosessnd tsehfghestinr'seisfyienh imrsunlfonITPDS.TI hould eunit for arol%5. ASn alternatilve. this mond mighi a ggemfiw Myvbe endasa iovNg Iruch wish i4EP6. Asen lernsis. is e f fthivrengere mar tm fired atabisker weith 4CCnd ire leasov7. ;= loeseen ed" rea to tire flinhtront iheaer efeirnst of~hsa Ila fire fontitnd or aying f ts min gunorirtin tofteechelrIi frlfheof the. irst ro.-nd.Auseondoumnieaat.nirnibefled w~thin5secondsf. first round 5i5s.a. Flise IllsiOnintioi marbe isolated usfsh ite lisfilluirris- ifrr.,asoiler iok.

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    16/84

    T"bl I Modl Test of Crew Mwknshisp: Plait 11.Machmeguan

    EXERCISE CONDITIONS ROUNDS STANDARD1ROOPS 300 m 100 Engagwitin 5 secondsIOn theamove Coax Obtain 3/Is coverageInfinity sight2 TRUCK 900 m s0 Engag within 6 seconds.---Miovinsgo a had Coax Obtain I tracer hit.Ranerfinder3 MOVING TRUCK 700Cm 50 Engage ithin 5 seconds.

    -On the mowe Coax Obtain I trae hit.0" Intl nity sight

    T1mKi 4 AIRCRAFT 2200Cm 100 Engage wthin 6seconds.Moving to.a halt Cal .50 Obtain I tracer hit.Tank Commander'sPeriscope5 TRUCK 6m m so Engagwithin 5seconds.On the move Coax Obtain I -race hit.Infinity sight

    a ROOP CARRIER 150C m 100 Engge ithin 6seconds.On the move Cal 10 Obtain Will coverage.Tank Commander's- ___ _____ periscope

    I MOVING TRUCK 1300 m 50 Engageaithin t0 seconds.Stationary vehicle coaxs Obtain I tracer hit.IGunner's periscope. I

    - j infrared. RCLO I2 TRUCK Sw r0m5 Engageithin 10 seconds.Stationary vehicle C .. Obtain I tracer hit.

    Metasop, infrared. FICILi3 TRUCK I 0m50 Engagewvithin 0 seconds.Moving to a halt ICo.. Obtain I tracer hit.AIRCRAFT Linfinity sight flare _4 AICR F 900Cm 100W Engagewithinl0econds.Movingtoahalt ICal .50 .Obtain 1 tracer hit.I r Tank Comniander'speicoe Infrared

    5 TROOPS I70C inC 10 Engage ithin 10 seconds.NWm On he move Coax Obtain 3s coverege.Tim Gunner's periscope,4- infrared6 MOVING TRUCK 300.s 50 I Engage ithin 10 seconds.Moving to ahalt Coax ,Obtain I tracer hit.Metaescope.nfrared

    7 MOIGTRUCK 500m 5 Engagewithin ICseconds.Moving to a halt Co.. Obtain 1 racemhtGunner's periscope,InfraredIU OVN AIRCRAFT 200. 10 Engge fiithin 10seconds,Moving to ahalt Cal .50 Obtain 1 reae hit.Tank Commalndsiperiscope, infrared* AIRCRAFT 20010m 100 Engage viithin 10seconds.Stationary vehicle Cal .50 Obtain I tracer hit.Tank Co n sndIr'speriscope, fisare

    NoTrES .Ourintsondiat of tisa ibe. enagm n dhit tis a reilid. Scrrne 5 har ..comoeisied isivt a vrety of Procedre.2.Aaanallierieie vEvariose .iiqis~rmv~elhie ieybe ssau3.Flare llumination viarybe epiade ih while light IvvffivsMv~rvinhl tvisenk.OayiqVl slavdarvts5iotie flan be irvi4 sarone einageiarit are iredby the ankrii.ivlkW

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    17/84

    To facilitate processing in subsequent stages of theanalysis, the candidates were assigned to one of three cate-gories, each of which contained a relatively homogeneous setof devices. In the first category were 25 devices, each con-sisting of a specific kind of subcaliber weapon, a particulartype of mount used to affix the weapon to an operationalM60AlA0S tank, and a scaled-range facility. In any specificinstance the latter was either a 1/60, 1/35, 1/20, or 1/2-scale range equipped with pop-up and knock-down scaled targets,or in the case of laser ranges, special retroreflectivenonknock-down targets. The devices comprising this firstcategory, and representing unique mixes of weapon, mount,and scaled range, are indicated in Table 3 by check marks.Empty cells correspond to cases clearly ruled out as viablecandidates, either because target effects would be too severe(e.g., for some of the larger caliber weapons fired on thesmaller ranges) or because the limits of accurate fire wouldbe exceeded (e.g., for smaller caliber weapons used on thelarger ranges). (As will be discussed shortly, cells withdouble check marks represent devices deemed potentially viablein a subsequent stage of the analysis.)

    None of the devices in the second category involve actuallivefiring, although roughly half do require range facilitiesof some kind (i.e., REALTRAIN, MILES, TACED, Stout, Dry Fire).The nine candidates represent a mixed bag of approaches tosimulation as suggested by the listing in Table 4. Some,such as the venerable Green Hornet, have been in the inventoryfor years. Others, such as MILES, are still undergoingdevelopment.The third and final category of devices consisted of fiveentries: the Unit Conduct-of-Fire Trainer, the Full CrewInteraction Simulator, the Tank Weapons Gunnery SimulationSystem, the Combat Training Theater (Subcaliber) and theCombat Training Theater (Laser). These candidates differfrom those in the first two categories by virtue of the factthat they are still in early conceptual or engineering stagesof development. Their implementation on even the most limitedbasis is not contemplated until some time in the 1980s. Thus,devices in this last set represent possibilities for the future.An unexpected outcome of the search for simulators wasthe large number of potentially relevant devices that wereuncovered. Most have been used as training devices for appli-

    cations ranging from classroom instruction in the fundamentalsof applying adjustment-of-fire procedures (e.g., the variousconduct-of-fire trainers) to practice of combined-arms tacticsin the field (e.g., REALTRAIN). Some, in addition to providingopportunities for practice of skills, have also been used to

    7

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    18/84

    Table &Candidate Scaled-Range Devices

    Weapons and Mounts 1/0 Scaled RangesB01 DVC-D 17-53 .22 Cal. In-Bore (90, 105mm) N/ v

    DVC-D 17-85.22 Cal. Mini-tankBracket/Rifle V VDVC-D 17-87 Brewster Device withM16 Rifle and .22 Cal. Rimfire Adapter ~ / VDVC-D 17-87 Brewster Device withM16 Rifle NVVV VDVC- D 17-87 Brewster Device with7.62mm Coaxial Machinegun V NVDVC-D 17-87 Brewster Device withM55 Laser Trainer NA/V NVDVC-D 17-88 Telfare Device XVVDVC-D 17-89 Wallace DeviceV

    BOl Cal. .50 In-Bore Device (90, 105mmn)V801 Riley 20mm In-Bore DeviceVB01 M55 Laser Trainer (Coaxial Mount) VNV/BOl 7.62mm Single Shot DeviceV [ V

    VInitial candidate device.V'Initial candidate deemed acceptable.

    8

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    19/84

    Table 4.Additional Candidate Devim

    InitialInitial CandidateDevice Candidate DeemedAcceptableChrysler Conduct-of-Fire Trainer VWiley 17-84 Conduct-of-Fire Trainer N/DVC-D 17-4 Conduct-of-Fire TankGunnery Trainer (Green Hornet) VDVC-D 17-94 Stout Device V/VREALTRAIN .Multiple Integrated Laser EquipmentSystem (MILES) VMain Gun Simulator VTank Appended Crew EvaluationDevice (TACED) V VDry Fire

    9

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    20/84

    test crew or crew member proficiency. These consist primarilyof the various subcaliber devices used in connection withscaled gunnery Tables I-VP (FM 17-12-2). The possibility ofusing any or all of the devices to simulate the model marks-manship exercises was evaluated in the next four phases of theanalysis.STEP 2 - INITIAL SCREENING OF DEVICES

    The feasibility of using any of the 39 devices to testcrew marksmanship was determined during technical discussionswith cognizant USAARMS personnel. Two subject matter experts,working independently, analyzed each device in detail. Theywere asked to screen out devices with hardware or design problemsthat would interfere with marksmanship testing (e.g., mountinstability, sight and weapon parallax).Fourteen of the 39 candidate devices were viewed as poten-

    tially useful. Nine of these were subcaliber devices fired onscaled ranges; they consisted of Brewster-mounted M55 lasersor 5.56mm or .22 caliber rifles used on various ranges, andthe Telfare-mounted .50 caliber device fired on a 1/2-scalerange. The nine candidates are indicated in Table 3 by doublecheck marks. The five additional candidates that were deemedviable are indicated by check marks in the second column ofTable 4. With two exceptions (MILES and TACED) these fivewere judged only marginally acceptable. They were carried intothe next phase of analysis, however, to provide as complete apicture as possible of different approaches to the simulationof crew marksmanship exercises. The shortcomings of devicesexcluded from further consideration are described in AppendixA.Virtually no documentation was readily available thatdescribed the functional characteristics or capabilities ofthe five futuristic devices. Therefore, formal evaluation ofthese devices was not attempted. (However, the Full Crew Inter-action Simulator (FCIS) and the Tank Weapons Gunnery SimulationSystem (TWGSS), judged informally, appeared promising.)

    STEP 3 - ANALYSIS OF ENGAGEMENT CONDITIONSDuring Phase 1 of the project a domain of tank gunneryjob objectives or tasks was defined. In keeping with anemphasis on crew marksmanship, these objectives defined allpossible ways that a variety of targets could be neutralizedwith the 105mm main gun, the 7.62mm coaxial machinegun, andthe .50 caliber machinegun weapons of the M60AlA0S tank system.Objectives were created by combining levels of all conditionsassociated with hypothetical engagements. The 11 conditionsand levels within specific conditions are listed in Table 5,

    10

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    21/84

    ii I EIL wSI UVIIL

    a- EE

    a' >f1l-

    li2 - KU.z

    8~ ~_ ________o ~I Ia I gg,.iS&jE

    0t~ -cr.~u I-__ _ . _ _I

    aJ t

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    22/84

    as adapted from earlier reports (Kraemer, Boldovici, and Boycan,1975; Wheaton, Fingerman, and Boycan, 1978). These conditionsand their associated levels were the focus of the third stageof analysis. Each of the 14 devices was considered with respectto each condition in an attempt to identify specific facetsof hypothetical engagements that could or could not be simu-lated.The results of this appraisal can be characterized intwo ways. First, each of the 11 engagement conditions can beconsidered individually, and discussed in terms of the devicesthat can or cannot provide for simulation. For example, theability of each of the candidates to represent various typesof (real or simulated) firing-vehicle motion can be described.These ancillary findings are presented in Appendix B (p. 48)where the focus is on those facets or conditions of an engage-ment that could not be simulated on the 14 candidate devices.For the purpose of pinpointing viable candidates, however,

    each device may be evaluated by scanning across the variousengagement conditions, thus revealing the dearee to which eachdevice suffers from relatively few or many deficiencies insimulation. These results (also discussed in detail in Appen-dix B) are summarized below.Based on their ability to simulate important dimensionsor conditions of tank crew marksmanship exercises, three poten-tially viable candidates were identified: the Telfare .50caliber device fired on a 1/2-scale range; the Brewster M16device used in conjunction with a 1/20-scale range; and theTank Appended Crew Evaluation Device (TACED) used on a 1/20-,1/2-, or full-scale range. Each of these devices providesfor less than full simulation, in the sense that each requiresan operational M60AIA0S tank and some form of (scaled) rangefacility. Because of this fact some of the costs associatedwith livefire testing must be borne when these devices are

    used. But this same relatively high degree of realism makesthese devices the ones of choice. They possess the greatestversatility in the conditions that can be simulated and pro-vide most information about crew performance.Versatility is an important consideration from a logisticspoint of view. It would clearly be impractical to run crewsthrough a number of different testing devices, each of whichwas specifically employed to simulate one or two particularkinds of engagements. One would simply be better off intrying to keep to a minimum the number of different devicesused to simulate a broad range of exercises. The three con-tenders were superior to their rivals in this respect. Theywere also superior in terms of exercising the driver, gunner,

    and tank commander, as well as the loader, provided thiscrewman was furnished with dummy rounds. Full-crew inter-action of this type is vital if the simulated test is toyield valid estimates of a crew's performance relative tocrew qualification standards.12

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    23/84

    STEP 4 - ANALYSIS OF MARKSMANSHIP EXERCISESInformation generated in the preceding step was used toevaluate the capabilities and limitations of the three devices

    still considered to be viable candidates. The analysisfocused on their potential for simulating specific gunneryexercises drawn from three different sources. The first setconsisted of the 28 engagements developed during Phase I ofthe project (Wheaton, et al., 1978). These exercises consti-tute a model livefire test of crew marksmanship inasmuch asthey are the end product of a systematic process of testdevelopment that stressed representativeness (by providingfor sampling of a wide variety of engagement conditions) andgeneralizability (by providing for coverage of virtually allcrew member behaviors). The second set included the 22 exer-cises comprising the Table VIII test currently used by theArmy to determine tank crew gunnery qualification (FM 17-12-2).The third set was made up of the 21 exercises contained in adraft revision of the current Table VIII that emphasizesmultiple engagements in tactical gunnery (Draft Change No.2, FM 17-12-2, 1978).

    Exercises from the latter two gunnery qualificationtables were included in order to provide as thorough an analy-sis as possible. Broadening the base of test exercises wasintended to make the evaluation less dependent on the spe-cific set of engagements under consideration. This expansionwas viewed as especially important to the extent that themodel test might include exercises representing rather unusualor infrequently encountered engagement conditions.The very first outcome of the analysis was the decisionto limit detailed evaluation of the candidate simulators tomain gun exercises only. The preceding examination of engage-ment conditions demonstrated clearly and convincingly thatneither coaxial nor .50 caliber machinegun exercises areparticularly well-suited to simulation with the three devices(or for that matter most of the others) under consideration.As a consequence, the recommended strategy for testing crewmarksmanship performance on these engagements is to inter-sperse them with simulated main gun exercises, but to ivefirethem. Given that the three remaining simulators require someoi of range facility, the livefiring of machinegun exerciseson these same or slightly larger ranges would seem more cost-effective than either foregoing such engagements entirely, or

    considering special-purpose devices that might be developedfor their simulation. Livefiring would be required for the 15machinegun exercises in the model test (see Table 2), or forthe 10 and nine machinegun engagements included, respectively,in the current and revised Table VIIIs.

    13

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    24/84

    The main gun exercises that underwent scrutiny are pre-sented in Appendix C (p. 54). Exercises from each of thethree gunnery tables are listed in Tables C-i to C-3 and aredescribed in terms of the 11 specific conditions of engagement.For exercises drawn from the model test (Table C-l) thesedescriptions are precise. They are less precise for engage-ments comprising the two Table VIII tests because the relevantdocumentation does not specify nor does the tactical testingphilosophy require that each engagement be carried out in aparticular way. (In this sense neither Table VIII test pro-vides an acceptable measure of crew marksmanship since choiceof firing mode, fire control instrument, etc. is ultimatelyleft to the discretion of the crew in response to the tacticalsituation.) To deal with the ambiguity inherent in these cases,all of the specific exercises that a particular engagementmight actually represent were listed as options in Tables C-2and C-3. All of these alternatives were evaluated in termsof how amenable they were to simulation.

    In summary, the results of the analysis are fairly clearcut. The Telfare-mounted .50 caliber device on a 1/2-scalerange apparently can be used to simulate any main gun exer-cise associated with the three gunnery tests. This is animpressive outcome considering the range and diversity ofexercises included in the model test and the two Table VIIIs.Next best is TACED. The device appears particularly well-suited to the simulation of daylight engagements and may becapable of handling a broad range of those fired at night.This nighttime use will depend ultimately on the device'sability to cope with engagements fired under infrared andlow levels of illumination. In contrast, the Brewster-mountedM16 device fired on the 1/20-scale range is clearly inferiorto the two preceding devices. Its primary weakness for test-ing crew marksmanship, is the inability of the tank commanderto provide range data in support of precision engagements.

    Given these findings, the Brewster M16 device was droppedfrom the final step in the analysis. (It should be kept inmind for the future, however, in the event that a rangefinderis developed for use on the 1/20-scale range). TACED andthe Telfare device emerged as the prime candidates and weresubjected to an analysis of the behaviors involved in crewgunnery.STEP 5 - ANALYSIS OF BEHAVIORAL ELEMENTS

    The final step in the analytic procedure was to obtainestimates of the quality of simulation provided by TACED andTelfare. This evaluation was the logical conclusion to aprocess that had begun with the identification of candidate

    14

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    25/84

    devices and had proceeded to evaluate them with respect toconditions of engagement and specific marksmanship exercises.The rationale for this last analytic activity was pre-dicated on the tank crew gunnery job-objective/behavioral-element domain specified during Phase 1 of the project. Thedomain consisted of a specification of the 266 ways in whichan M60AIA0S crew could neutralize targets. Correspondingly,each of these exercises was described in terms of 114 specificdriver, loader, gunner, and tank commander activities (seeAppendix A of Boldovici, Boycan, Fingerman, & Wheaton, 1979for detailed specification of the domain). Given this database, therefore, the obvious question to ask is how well thecandidate devices provide for simulation of each of thebehavioral elements comprising a given marksmanship exercise.The answer was provided by ratings on a three-point scale.One end of the scale signified that a given behavior could be

    performed in the simulator essentially in the same manner asit occurred in the actual livefire setting. At the other endlay judgments that the behavior in question was not representedin the simulator. Between these two extremes lay a gray areain which the behavior might occur in the simulator, but it wouldbe noticeably different in some respect from that occurringin the livefire situation. Thus, a judgment was made as towhether or not the behavior was at least functionally similarto that occurring during livefire. Further discussion of theconcept of functional similarity, as applied to displays andcontrols rather than behaviors, may be found elsewhere (Wheaton,Rose, Fingerman, Korotkin, & Holding, 1976). Suffice it tosay that in the present case few behaviors were judged tofall into this middle category.The scale was applied to the behavioral elements compris-ing each of the 13 main gun exercises in the model test ofmarksmanship. These exercises were developed to representvirtually all crew behaviors associated with main gun firingand, therefore, were representative of main gun engagementsat large. Consequently, exercises from the two Table VIIIswere not examined separately. Information obtained from themwould have been redundant with the more inclusive array ofbehavioral elements associated with the model test.The results can be reported succinctly for each device.Given the exercises listed in Table C-l, and replacing thetwo BEEHIVE engagements (#103, #81) with their designated

    alternates (#106, #69) on grounds of current policy, TACEDprovides for the simulation of all but two infrared engage-ments (subject to further device development). In theremaining 11 exercises virtually all. of the component crew

    15

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    26/84

    behaviors are reproduced faithfully. This outcome is notsurprising since the crew is in essence dry-firing an other-wise operational tank. The only exception (for TELFARE aswell) is that the loader requires two dummy rounds in orderto exhibit the required behaviors in the daylight HEP engage-ment (#97). This evaluation of TACED applies to the firingof an initial round; firing of subsequent rounds (e.g., BOTadjustment) cannot be simulated.

    Telfare can also be used to simulate the model exercises.The quality of simulation, however, is slightly degraded onfour exercises. In addition to the requirement mentionedabove for the daylight HEP exercise (#92), one crew behaviorcannot be simulated. In firing the .50 caliL c simulationof this HEP exercise, the tank commander cannot and would notapply aim off in order to achieve a target hit. The threeother cases (#43, #67, #113) involve precision or range-card-lay-to-direct-fire engagements of moving targets. In theseinstances the gunner does not lead the target in preciselythe manner he would if firing SABOT or HEAT. Consequently,three of his behaviors were judged to be only functionallyequivalent: "gunner applies lead in direction of target appa-rent motion", "gunner lays rangeline leadline at center oftarget vulnerability", and "gunner makes final precise lay".

    Few shortcomings were found in either device. Those thatwere uncovered did not appear serious. As a consequence, thecontent validity of a simulated marksmanship test based onthe model exercises and the Telfare or TACED approach wasjudged to be high. All of the main gun exercises could besimulated and virtually all of the underlying behavioral ele-ments were represented in a realistic manner. Given theseoutcomes the final step was to propose simulated crew marks-manship tests.STEP 6 - SPECIFICATION OF SIMULATED TESTS

    In specifying simulated tests of tank crew marksmanshipthat could actually be used to replace livefire testing, threeissues were considered. The first concerned the specificset of exercises upon which to base the test. The secondaddressed the choice of device, given that both Telfare andTACED had potential. The third and final issue was the scor-ing system that should be used to evaluate crew proficiency.Eacn of these issues is discussed below in the course ofelaborating the model simulated test.

    16

    . . ..

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    27/84

    Test exercises. One set of exercises on which to basethe simulated test consists of the 28 engagements used in thelivefire version of the model test (see Tables 1 and 2). Inimplementing this test one would simulate the 13 main gunengagements, using one of the devices for that purpose, andlivefire the 15 machinegun exercises, engaging appropriatetargets on a 1/2- or even full-scale range. All of theseexercises, however, were selected within the context of livefiretesting; as a consequence they represent a minimal set, chosenin light of resource and cost constraints. One could improvethe overall quality of the test by expanding the number ofengagements on which crew performance is evaluated. The pre-sumed increase in validity and/or reliability could be obtainedin the simulated testing environment for nominal increases incost. Accordingly, the strategy of simply duplicating thelivefire test was abandoned in favor of an alternative approachin which the greatly reduced cost of simulated testing per-mitted an assessment of crew performance based on a largernumber of exercises.

    Given the desirability of firing more engagements, alter-native ways of constructing the simulated test were considered.On the one hand, new exercises could be added to those in theexisting test to provide even better representation of theoverall gunnery domain. Toward this end the general samplingstrategies used to select the basic set of 28 exercises couldgain be pursued (Wheaton, et al., 1978). In theory, the con-tent validity of the resulting test would be greater than forone based on only 28 items; however, stability of performanceon any single engagement would be problematic since each engage-ment would only be fired once. Another alternative would beto repeat the basic set of 28 exercises some number of times.Such an approach would help stabilize estimates of crew per-formance on each engagement, where the systematic analyticprocedures had been used to choose good representatives foreach family. In theory, the reliability of the test would beimproved; but the breadth of coverage would remain the same.A third option, of course, would be to combine these twoapproaches. A choice among these options involves the resolu-tion of many complex and subtle issues and requires empiricalstudy (see Chapter III and Appendix E.) Because of possibleresource constraints the combined approach was not pursued.Since coverage of the domain was already considered to beadequate, the preferable option was to increase stability inestimates of performance on each engagement. This was accom-plished by replication of exercises as described below.

    17

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    28/84

    The simulated model test is composed of 54 exercises:each of the 13 simulated main gun engagements is replicatedthree times; the 15 machinegun engagements are livefiredonce. (The decision not to replicate the machinegun exerciseswas fairly arbitrary and based on the opinion that typicalcrews have little difficulty with these engagements; givensufficient resources they also could be replicated.) In anattempt to balance out carry-over and learning effects(discussed in detail below), three random orders of the sevendaylight main gun engagements were developed, a processrepeated for the six nighttime main gun engagements. Thelivefire machinegun exercises were then interspersed amongthe main gun engagements: during daylight firing, every thirdsimulated main gun engagement is followed by a livefire machine-gun exercise, providing for two of the latter in each of thethree replications; at night the machinegun engagements areinserted after two simulated main gun exercises, providingfor three livefire machinegun engagements in each replication.The simulated tank crew marksmanship test is nresented inAppendix D. Daylight engagements appear in Table D-1 whilethose fired at night may be found in Table D-2.

    Simulators. The second issue in specifying the simula-ted test was the choice of simulator: Telfare or TACED. Eachdevice has obvious strengths: Telfare allows the crew to put"steel on target," yielding target effects (i.e., knock-downs,sensings) that enable the crew to adjust subsequent fire ifnecessary; TACED permits the crew to go through all of itsrequired behaviors, and provides a hard copy record of theresulting proficiency as indicated by the consequent sightpicture. Both have weaknesses: Telfare, like any other weaponsystem, is subject to dispersion effects that may penalize thecrew in spite of perfect performance; TACED is of questionablevalue under low light conditions, and in no case can it simu-late firing of subsequent rounds (since the first "round"cannot be sensed).

    It is recommended that both devices be used to simulateand measure crew marksmanship, especially because of dispersioneffects. TACED can pinpoint dispersion by providing a recordof the final sight picture which may be compared to the strikeof the Telfare round. TACED can also verify Telfare targeteffects. The burden of actual simulation would fall on Telfare,while TACED would improve the accuracy of performance assessment.The main gun exercises in Tables D-1 and D-2 are consequentlysimulated by using the two devices conjointly.Scorin. The model simulated test is designed to provideinformation about crew marksmanship performance for a variety of

    18

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    29/84

    t lip

    purposes: qualification, training diagnosis, and motivation.Scoring procedures for these purposes are detailed below.Similar procedures for the livefire model test are detailed inWheaton, et al. (1978).Basically, the same scoring procedures used on the modellivefire test are to be used with the simulated version. Eachround fired is scored either as a hit or a miss, or as a mea-sured deviation from some idealized (e.g., center of target)aiming point. Various aspects of engagement time are measuredand recorded to reflect the speed with which crews engage tar-gets. Also recorded is the round with which a target hit isfirst obtained. These data, which are collected for each maingun and machinegun exercise in the test, constitute the mostbasic scores in a scoring system involving three hierarchicallevels. Scores at the second level are derived by applyingstandards of performance to the speed and accuracy data obtainedfor each engagement. The standards (shown in Tables 1 and 2)

    are used to identify engagements in which crew proficiency equalsor exceeds agreed upon levels (scored as a "GO") or fails todo so (scored as a "NO GO"). The patterns of "GO" and "NO GO"exercises together with the underlying performance data can beused for diagnostic purposes to identify behaviors and engage-ment conditions on which a crew may need remedial training.The highest level of the scoring system results in adecision about the crew's level of competence vis A vis theoverall domain of marksmanship exercises. Two aggregate scoresare developed, one representing competence on main gun engage-ments, the other indicating competence on machinegun engage-ments. The part scores are calculated and evaluated againsta level of competence specified separately for -Fch type of

    engagement.The three-category scheme used to determine crew qualifi-cation on the livefire test can be adopted for use on thesimulated version (Wheaton, et al., 1978). This scheme isproposed only tentatively. Final standards should ultimatelybe developed based on an evaluation/calibration study asdescribed in the next major section of this report. In thescheme adopted, three standards of competence are specifiedfor main gun and machinegun marksmanship. Crews successfullyperforming 92% of all the repetitions of either type ofengagement would be qualified on that type. Crews performing69% or fewer would be unqualified. Crews lying between thesetwo bounds would be viewed as marginally qualified. Theoverall qualification decisions that would be reached forcrews, falling into the three different zones for each of thetwo aspects of marksmanship, are portrayed in Table D-3.A detailed discussion of this and related scoring topics appearsin Wheaton et al., (1978).

    19

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    30/84

    The replication of main gun engagements in the proposedsimulated test may be used to make the standards for qualifi-cation even more stringent. For example, an additional cri-terion for qualification might be added to those describedabove: at least one of the three replications of every maingun exercise fired must be performed successfully (in additionto meeting the 92% requirement). Such an approach to qualifi-cation emphasizes that ever objective in the domain of tankgunnery is relevant, and that a "qualified" crew must becapable of performing all objectives. But it also recognizesthat there are problems in measuring crew performance becauseof the uncertainty introduced by dispersion, whether the wea-pon is a main gun or a subcaliber device (thus the requirementto successfully perform one out of three, rather than threeout of three replications). Finally, even if this more strin-gent criterion is not adopted for determining crew qualification,the idea of looking at replications of the same engagementcould be of great value in scoring for diagnosis of particularcrew strengths and weaknesses. Any exercise which is failedthree times by a particular crew would clearly signal an aspectof marksmanship on which additional training is required.

    Summary. The effort described above has resulted in thespecification of a model simulated test that, conceptually atleast, can be used to evaluate the marksmanship proficiencyof tank crews. Development of the test is of great potentialsignificance because of the promise it holds as a substitutefor livefire testing. At this point, however, an importantcaveat must be raised, and a proper note of caution sounded.The simulated marksmanship test, like its livefire forebearer,has been painstakingly developed to provide a valid assessmentof crew marksmanship. But the development process has beenentirely analytical, and to this point has proceeded in theabsence of any hard data. Such data are absolutely essentialbefore faith can be placed in the simulated test. The nextsection of the report describes why this is so and discussesthe approach that should be followed.

    20

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    31/84

    III. EVALUATION OF SIMULATED GUNNERY EXERCISES

    Preceding sections of this report have discussed why andhow one would test crew marksmanship using a set of simulatedexercises. The remaining issues are why and how one wouldevaluate such a test. While this section will focus on thespecific exercises comprising the model test, the material isgenerally applicable to evaluation of any simulated test.Why is an empirical evaluation necessary? The rationalesused in developing both the model tank gunnery test and itssimulated version are compelling; and at first glance it mightseem sufficient simply to accept these rationales and to beginemploying the simulated test. However, in practice it has beenamply demonstrated that even tests with compelling rationalesmay not, in fact, measure precisely that which they were intendedto measure. The simulated test is to be used in making decisionswith real-world consequences including whether or not crewsare qualified, and whether or not training programs are adequate.These decisions are too important to be made without high con-fidence in the test; it would simply be too risky to proceedsolely on an analytical basis. Thus, an empirical evaluationof the simulated test is required.

    GOALS OF THE EMPIRICAL EVALUATION STUDYIn designing the evaluation it is important to considerthe specific objectives of such a study. These objectives haveimplications for development of one or more designs capable of:

    answering the specific questions of interest, making explicitthe necessary controls for extraneous effects stating a prioriany assumptions to be made, and being sensitive -o the practi-cal constraints of experimentation in the real Army environ-ment.Before considering the list of detailed objectives, it isimportant to review the purposes of the model livefire and simu-lated gunnery exercises. These exercises were selected tomeasure the performance of tank crews exercising combat-relevant marksmanship skills. They were assembled into a testwhich was designed to: a) assess whether or not the crewsshould be considered "qualified," and b) provide diagnosticinformation on areas of performance in need of remedial training.

    It was also assumed in the development of these tests thatsome of the obtained performance information might be usefulin making judgments about combat effectiveness (Wheaton et al.,1978). The simulated test is to be used for these same purposes.Thus, the overall goal of the evaluation is to assess thevalidity of the simulated test vis & vis these purposes. Sincevalidity is constrained by reliability, the reliability of the

    21

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    32/84

    simulated test must also be assessed, and since scoring forqualification is criterion-referenced, scores on the simulatedtest must be calibrated to scores on the livefire test. Thesethree psychometric issues and the interplay among them arediscussed more fully in Appendix E. A subordinate objectiveis to evaluate means of enhancing performance assessmentthrough simulated testing. These broad goals may be trans-lated into the following list of specific objectives for theevaluation. These represent evaluation criteria against whichany simulated test must be considered.Objective 1: Assess reliability of the livefire test.The livefire test is the empirical criterion for establishingthe validity of the simulated test. Ideally, therefore, thelivefire test should be a perfect measure of (true-score) marks-manship proficiency. In fact, however, it is not. Despitehaving a strong claim for content and construct validity(Wheaton et al., 1978; Guion & Ironson, 1978) a test basedon the model set of livefire exercises is likely to involve

    substantial error of measurement. In particular, when assessingthe accuracy of gunnery (i.e., strike of the round on target)it is known that the weapon system itself adds considerablevariance. Thus, even when the gunner aims perfectly, thestrike of the round includes a dispersion component that isliterally random and that the crew members have no control over(Brodkin, 1958; Pfleger & Bibbero, 1969; Fingerman, 1978).Such random dispersion exists in addition to biased dispersioncomponents over which the tank crew does have control, includ-ing, for example, errors in zeroing or boresighting, and sys-tematic variations in performance among various lots ofammunition. Therefore, when accuracy is scored, whethersensing "target" or measuring the distance between the strikeof the round and the center of the target, random and nonrandomdispersion components produce errors of measurement with regardto the crew's true proficiency. When observers are used toscore hits, additional sources of error are introduced that mayfurther reduce the reliability of performance measurement. Inresponse to such problems perfect performance on the livefirecriterion-referenced test was characterized as 95% or better.In many ways this correction for dispersion and other kinds ofmeasurement error was gross, but some type of correction wasneeded.

    In the evaluation study more sophisticated methods forreducing error of measurement are required, since an accurateestimate of criterion performance is particularly important.The measures of livefire performance are to be used to evaluatethe validity of the simulated test, and the higher the reli-ability of the livefire criterion measure, the higher will bethe potential validity of the simulated test. Thus, it ismandatory that as many sources of measurement error as possible

    22

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    33/84

    be eliminated from the livefire test. Even fairly expensivemeans for getting accurate measurement of livefire performanceshould be considered. In addition to providing reliable datafor the validity assessment, estimates of the cost (in termsof measurement error) of traditional versus highly controlledmeasurement procedures in livefire testing will be a factorin determining the trade-off between simulated and livefiretesting.Three different kinds of reliability should be assessedfor the livefire test. First, there is the reliability of the

    qualification decision. If crews are to be labeled "qualified"or "not qualified" by the livefire test, then two kinds ofmeasurement error can be anticipated. In the first, crewswhose true proficiency is sufficient to make them qualified maynevertheless fail to pass enough exercises to be so classified;cases in which qualified crews fail the test are termed "falsenegatives." In the second, crews whose true proficiency is notsufficient to qualify might nevertheless pass enough exercisesto be classified as qualified; such cases are called "falsepositives." The probability of these two kinds of measurementerror may be predicted. Wheaton et al. (1978) provide anextensive discussion assuming a binomial distribution oferrors; others have used more complex assumptions (c.f.Hambleton, Swaminathan, Algina, & Coulson, 1978). At a mini-mum, the empirical evaluation should permit a test of thesetheoretical predictions of classification error. If the obtaineddata fit these theoretical models well, as determined by good-ness of fit tests for example, then little need be done toimprove the reliability of the livefire test. If good fitsare not obtained, then improved control of the test environmentis required, and corrections for attenuation of validitycoefficients (due to unreliability in the criterion) may berequired.

    In the discussion above, the only matter of concern iswhether or not crew performance can be classified reliably withrespect to some cutoff score, presumably expressed in terms ofthe proportion of exercises passed. However, a second kindof reliability which must be assessed is the reliability ofthe proportion-correct test score itself, independent of aparticular standard for qualification. Reasonably high reli-ability must be demonstrated for this score since the consis-tency of the qualification decision will be heavily dependentupon it. Similarly, a reliable proportion-correct score will beuseful in providing diagnostic information for training purposes.

    The third kind of reliability which must be consideredis the reliability of measurement on individual engagements;that is, the reliability of crew performance on the individualexercise in terms of gunnery accuracy and speed. Once again,the reliability of these measurements underlies the reliabilityof the qualification decision. In addition, if such detailedmeasurements are reliable, they will contain considerable

    23

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    34/84

    diagnostic information. It should be possible from suchinformation to provide detailed feedback to crews and to unittraining personnel on specific performance deficiencies.Objective 2: Assess reliability of the simulated test.The issues described above also apply to simulators. Sub-caliber devices are subject to dispersion effects; althoughsuch effects have not been well quantified they may be moresevere than for the main gun. Alternatively, the overall reli-ability of the simulated test may in fact turn out to be some-what better than that of the livefire test since certain sourcesof distraction will be missing. 1 In either case, interest inthe reliability of the simulated test revolves around reli-ability as a limiting factor on validity. As above, threekinds of reliability need to be assessed: a) reliability ofqualification decisions; b) reliability of the proportion-correct score; and c) reliability of measures of proficiencyfor individual engagements.Objective 3: Assess the validity of the simulated test.The final primary objective of the evaluation must be anassessment of the consistency with which the simulated and live-fire tests measure the performance of tank crews. Three kindsof validity are of concern. The first addresses the extentto which a qualification decision based on the simulated testis consistent with a qualification decision based on the livefiretest. It may be that inconsistencies in qualification decisionsbased on the simulated vis A vis the livefire test may beremoved by using different cutting scores on the simulatedtest, or even radically different scoring approaches. There-fore, more creative scoring procedures for the simulated testmay be investigated for their impact on validity. Accordingly,this portion of the validity assessment may necessitate anexamination of methods for test calibration. The issue isdiscussed in Appendix E.The second kind of validity concerns the proportion-correct score. For training-diagnostic and motivational pur-poses this score has many norm-referenced applications, leadingto an ordering of crews based on their performance. The con-sistency with which the simulated test orders crews vis A visthe livefire test must be determined.

    1Sources of distraction associated with livefire may sometimesbe relevant to gunnery proficiency assessment (e.g., theflash/bang effect). Nevertheless, removing them should tendto improve each crew's consistency of performance, and thusthe reliability of the simulated test. See Gagn4 (1954) foran earlier discussion of this topic.

    24

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    35/84

    The final form of validity is at the individual engagementlevel. For individual crew-diagnostic and feedback purposesone must be able to tell crews precisely what engagements theyhad difficulties with. If it can be shown that difficultieson particular simulated engagements are predictive of diffi-culties with particular livefire engagements, then the simulatedtest will serve this purpose quite well. The three kinds ofvalidity are arranged hierarchically, the validities of pro-portion-correct and individual engagement performance scoresclearly underlying the validity of qualification decisions.

    Considered jointly, the three objectives discussed abovedefine the primary purpose of the evaluation--to determine theeffectiveness of simulated exercises as substitutes for live-fire in crew gunnery testing. Other less critical questionsmight also be addressed in the same study. The next objectiveis one example.

    Objective 4: Explore different amounts of simulatedtesting. One of the chief advantages of simulated testing isreduced cost. This makes it possible to try to improve theprecision of test information by using a longer test (moreexercises) or by replicating a short test (giving it more thanonce) since in general, the longer a test the more reliableit will be.1 We have recommended replicating engagements twiceor even three times in order to obtain more precise informationfor qualification and performance diagnostic purposes. In thecourse of the evaluation cost trade-off functions could becomputed comparing improvements in reliability of the simulatedtest scores (and hence improvement in potential validity) tothe cost of additional replications.

    In order to reliaze any of these objectives, the evalua-tion must be carefully designed. In addition to considerationsof scientific rigor, the costs of the required field researchmust also be taken into account. The next section develops twocandidate experimental designs that have been developed tosatisfy the four objectives.2This is true assuming that item error variances are homoge-neous, measurement errors are uncorrelated, and item contentis homogeneous. Study of the relation between test lengthand reliability has generally followed two tracks, one theo-retical and the other empirical. In the theoretical analysesit is assumed that the test content is homogeneous, and thatthe additional items measure the same underlying factor as theoriginal items. Empirical evaluations have also generallyfocused on tests in which it is easy to assume content homo-geneity, and assumption that may not hold in a tank gunnerytest, even within more restricted item domains or families.Thus, in the present case improvement in reliability wasattempted with replications of the simulated exercises asopposed to the addition of (potentially non-homogeneous) items.

    25

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    36/84

    7|

    EXPERIMENTAL DESIGNThe evaluation can be based on either of two kinds ofexperimental designs. The first kind, termed "non-replicated,"is distinguished by the fact that each of the two tests, live-fire and simulated, is taken once by each crew. While thisdesign is inexpensive, it provides only the minimum of infor-mation required about reliability and validity. The secondkind, termed "replicated," involves administration of one orboth of the tests several times to each crew. For the increasedcost one gains considerably more information on test reliability,and is able to explore the value of increasing the length ofthe simulated test for (later) routine administration.Many other designs were considered initially. Some turnedout to be elaborations of the designs proposed below that pro-vide additional or more specific information on certain ques-

    tions. Others were rejected as inappropriate, even though theymay have frequently appeared in the psychometric literature.The reasons for some of these rejections will be raised in thediscussion of the proposed designs.Non-replicated designs. The non-replicated design is thesimpler of the two types. In an evaluation based on thisdesign, each tank crew would fire the 28 basic engagements ofthe simulated test once; they would then fire the same engage-ments once in the liveire test environment. Since neitherthe simulated nor the livefire exercises are replicated, test-retest and parallel-forms methods of assessing reliabilitycannot be employed (see Appendix E). Reliability would insteadbe assessed with one of the measures of internal consistency

    such as the split-half reliability coefficient or the alphacoefficient. Since the exercises comprising the test maynot be homogeneous (e.g., main gun and machinegun exercises),the test would be broken down into at least two componentsfor reliability assessment. Specifically, the reliabilityof the main gun and machinegun portions of the test would beassessed independently for both the simulated and the live-fire versions.The order in which the engagements are fired is importantbecause there may be large carry-over and learning effectsover the course of firing 28 exercises. The specific designemployed must control for these carry-over effects. One wayto do so is illustrated in Table 6 for the daylight portion

    of the test. The steps in deriving Table 6 were as follows.The main gun daylight exercises were randomly ordered as werethe machinegun daylight exercises; the daylight portion ofthe test was then constructed by alternately taking one main

    26

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    37/84

    Table 6.Ordering En-gagemnents for Control of Carry-Over Effect

    (Daylight Exercises of Model Test)Exercises: MG #6mg#3

    MG#2mg#5MG#3rng#6MG #4mg# IMG#Img#2MG#5mg #4MG#7

    MG - Main Gunmg machinegun

    27

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    38/84

    gun engagement and one machinegun engagement. 3 Thus anyadjacent main gun/machinegun pair is preceded by approximatelythe same amount of prior practice. The same procedure wouldbe followed for the nighttime portion of the simulated table.Main gun and machinegun engagements would be randomly orderedand then drawn in alternation to construct the actual test.Precisely the same order of engagements would be used forthe livefire and simulated tests.In executing this design, each tank crew should firethe daylight and nighttime portions of the simulated teston Day 1, and the daylight and nighttime portions of the live-fire test on Day 2. This is preferable to having a crew firesimulated and livefire portions on the same day, since inter-

    spersing simulated and livefire testing in this fashion wouldcontaminate estimates of reliability for each component. Onemight find in this non-preferred design, for example, thatthe reliability of the daylight and nighttime portions of thesimulated test differed. In this case one would be unableto determine whether the difference was due to daylight vs.nighttime conditions, or to the interspersing of livefireexercises. By presenting the simulated and livefire tests ontwo different days, one would hope to minimize learning orcarry-over effects from the simulated engagements to the live-fire engagements. Further, by keeping the delay intervalreasonably short (i.e., one day), it would still be reasonableto assume that the true proficiency of each crew has remainedthe same from one test to the other. If the two tests wereseparated by a longer period (e.g., a month), it would be moredifficult to assume true scre stability, a prerequisite tothe assessment of validity.

    The non-renlicated design is subject to the problem oftime-bounded inferences. Specifically, the (prior) simulatedtest is to be compared to the (posterior) livefire test; thatis, the simulated test is being used to predict the livefiretest score. Alternatively, one could present the livefiretest first, followed by the simulated test, and determinewhether the simulated test reproduced the measures of pro-ficiency established in the preceding livefire test. Thechoice is between prediction of livefire scores and "postdiction"3An even better procedure during the evaluation study would beto use several random orders, randomly assigned to crews.One approach would be to employ a modified graeco-latin squarearrangement (Myers, 1972). This procedure would permit anexamination of carry-over effects independent of specificengagement types (the ordinal-position-in-sequence-effect--see Myers, p. 282).

    4This assumption is only reasonable in some contexts. See thediscussion on crew experience that is provided later in thissection.

    28

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    39/84

    of livefire scores. Strictly speaking, neither ordering isentirely satisfactory. The goal is to determine the sub-stitutibility of the simulated test for the livefire test orits concurrent validity. With either ordering of the testscarry-over effects may occur. Such carry-over effects couldobscure the validity of the simulated test or, at the veryleast, make any attempt to calibrate th,! simulated testextremely difficult (see Appendix E for a discussion of testcalibration issues).

    It is not possible to have a literally concurrent test,that is, a simulated and livefire test at exactly the sametime. One approximation, however, would be what is known asa "revolving door design." Each engagement would be firedtwice in succession, once simulated and once livefire, thusinterweaving the two tests. One could also counterbalance,firing engagement #1 simulated, then engagement #1 livefire,then engagement #2 livefire, then engagement #2 simulated,and so on. However, as discussed above, the constant inter-spersing of livefire and simulated engagements would makeassessment of reliability extremely tenuous. Since the reli-ability issue is ultimately more critical than the calibra-tion issue, and perhaps more sensitive to the specifics ofexperimental design, the revolving door approach has beenrejected. The Day 1 - Day 2 approach, while potentiallyincreasing the difficulty of calibrating the simulated test,should minimize simulated-to-livefire carry-over effects andthus insure adequate assessment of reliability. One improve-ment in the basic design would be to split the sample crewsinto two groups. One group would receive the two tests inthe order: simulation followed by livefire; the other groupwould receive the two tests in reverse order. This designis layed out in Table 7. The symbolic notations in Table 7correspond to the variety of reliability and validitycoefficients which could be obtained. For eample, r refersto the reliability of the simulated test on Day 1.5 Ainotherexample is rL2; this refers to the reliability of the live-fire test on Day 2. By comparing, for example, r.1 to rs2,we may assess the change in reliability as an impact of carry-over effects; rsl is the reliability of the test with no test-specific prior experience, while rs2 is the reliability ofthe simulated test following a similar livefire test. Ifthese two coefficients are not significantly different, the-Actually there are four sets of reliability coefficientsrepresented by rsl. They are: the reliability of the maingun portion of the simulated test during daylight; the reli-ability of the main gun portion of the simulate test atnight; reliability of the machinegun portion of the testding daylight; and reliability of the machinegun portion ofthe test at night.

    29

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    40/84

    Tabl 7.Counterbulanced Arrangement for fthNon-Replicated Design

    Da y 1 Da y 2

    Group 1:

    (Simulated /Daylight Livefire /Daylightr1 Simulated /Night Vpre Livefire /Night }L 2Group 2:

    Livefire / Daylight Simulated /DaylightLi Livef re / Night OsSimulated/ Night

    30

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    41/84

    data may be pooled to gain a more precise estimate of reli-ability of the simulated test. If they are very different,the reliability estimate on Day 1 would be more instructivesince it would reflect performance in the absence of priortest-specific experience. Similar comparisons are possiblefor the reliability coefficients associated with the livefiretest.

    Two validity coefficients are also shown in Table 7.One, labeled Vpre, is for the simulated-test-before-livefiretest; the other, labeled V ot, is for the simulated-test-after-livefire test. Again, if these coefficients are of similarmagnitude the data may be pooled to get a more precise esti-mate of the validity of the simulated test. If they are verydifferent, however, subtle distinctions in their interpreta-tion may be appropriate. The V re coefficient is a measureof the predictive validity of tie simulated test, that is,how well the simulated test predicts subsequent livefire per-formance. The Vpost coefficient may also be interpreted asa predictive validity coefficient, that is, how well livefireperformance predicts simulated performance. In cases wherethe two coefficients differ, the former Vpre index wouldgenerally be preferred.

    If no significant carry-over or learning effects areobtained (i.e., rsl = rs 2, rLl = rL2, Vpre = Vpost) interpre-tation is straightforward. When the coefficients reveal sig-nificant carry-over effects, the fine structure of the datamay be revealing. For example, one may be able to determinethat some engagements reveal strong carry-over effects whileothers do not. Further separate analyses might focus onengagements which do and do not exhibit carry-over effects.

    If validity of the qualification decisions is deemedhigh, then computation of a calibration formula would followvia regression analysis. Rather than addressing whether ornot the two tests ordered crews consistently, this analysiswould determine whether or not the same qualification decisionswere dictated by the two scores (livefire and simulated).The intercept parameter from the regression analysis would,in fact, be the calibration value, and its significance couldbe tested using standard methods. If validity at the qualifi-cation-level were low, then calibration would be more complex,depending, for example, on whether validity using finer mea-sures of performance was low or high.

    The richness of even this simple design is illustratedby the large number of reliability and validity coefficientswhich would have to be considered. Fortunately, not all ofthese coefficients are independent, and a synthesis of theresults should be possible. In many cases simple inspection

    31

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    42/84

    of the pattern of results will enable one to make some generalstatements regarding the reliability and validity of the simu-lated marksmanship test. In other instances it may provenecessary to employ sophisticated analytical methods to achievea synthesis.If the estimated validity of the simulated test is high,no further comment is required. If the validity is low, twopossible problems should be examined. First, one shouldquestion whether the reliabilities obtained for the simulatedand livefire tests are sufficiently high. Were reliabilityon the livefire portion of the test low, then concerns regard-ing errors of measurement would be well founded, and if allcountermeasures were taken (see impiementation guidelines),little recourse would be available. The same would be truewere reliability of the simulated test found to be low. Thesecond possible problem, suggested by a low estimate ofvalidity but high reliability on both tests, is that duringdevelopment of the simulated test some critical componentsof the task having an impact on performance may have beenoverlooked. These components might be intrinsic to the task,such as particular behavioral elements required in the live-fire but not in the simulated environment, or others demandedby the simulation but not found in the livefire version.Alternatively, the critical components might be extrinsicfeatures of the livefire environment that are missing fromthe simulation (or vice versa). An example is the noise andrecoil (flash/bank) associated with service firing of themain gun. If the simulated and livefire tests are suspectedof differing in terms of such critical features, then thesefactors should be identified and a new version of the simu-lated test should be constructed where such distinctions areremoved. This step would improve the test's (content) validity.In the present case the problem of high reliability but lowvalidity is considered highly unlikely because of the effort

    made to insure the content validity of the simulated test(Guion & Ironson, 1978).Replicated designs. In this class of designs at leastone of the tests is administered more than once, several repli-cations being likely. The design is a good one because itcan be used to determine whether the theoretical gains asso-ciated with increasing test length are actually realized.Because of the cost of livefire testing it is unlikely thatthere would be interest in routine administration of a repli-cated livefire marksmanship test. However, for the evaluationstudy, such replications would be valuable.The chief problems with replicated designs are the sameas those found in non-replicated designs: carry-over effectsand time-bound inference considerations. Carry-over effects

    6This is true assuming that item error variances are homoge-neous, measurement errors are uncorrelated, and item contentis homogeneous.

    32

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    43/84

    are more complicated here since, in addition to engagement-to-engagement and simulated-to-livefire effects, there noware replication-to-replication carry-over effects. Individualengagement carry-over effects (e.g., across main gun andmachinegun exercises) can be dealt with in the same fashionas in the non-replicated designs (see Table 6) . The replica-tion-to-replication carry-over effect is more complicated,since it depends on the precise shape of the learning curve.In general, the best way to resolve this problem will be toconduct a pilot study. One group of subjects would be exposedrepeatedly to the simulated test (to study learning effectsin that context), and a second group would receive the live-fire test several times. Design of the pilot study isstraightforward. Engagement carry-over effects would becontrolled for as in Table 6. Replications should not imme-diately follow one another; a moderate delay between replica-tions is required, on the order of half a day to one day.Troops should have the same background, prior experience, andtraining as the troops to be employed in the main study. Mostimportantly, their experience immediately prior to the testshould be the same as that of crews to be used in the maintest (see the section below on implementation guidelines).Extensive prior training, e.g. Tables I through VII, is likelyto reduce learning effects across replications of the tests.

    Precise design of the main evaluation study will dependon results of the pilot study. These most likely will indi-cate that repeated administrations of the simulated testresult in improved performance but, hopefully, not to theextent that severe ceiling effects are encountered. If thepilot study covers an appropriate range of replications(e.g. up to seven or ten), the data might be ux3ed to selectthat number of replications which improves the reliabilityof the simulated test without leading to ceiling effects.The same number of repetitions would then be employed in theevaluation study. Performance scores would be collapsedacross replications to characterize crew proficiency.

    Results of the livefire pilot study could be somewhatdifferent. Because of the flash/bang effect, early livefireperformance is likely to suffer; as crews adapt to thiseffect, performance will improve rapidly. This will resultin a steep learning curve initially, with substantial improve-ment over the first two or three replications (as crewsadapt), followed by a slower continuing growth due to moretypical learning effects. The number of livefire replicationsto be used in the evaluation study could be chosen in thesame way as the number of simulated replications (i.e. thatnumber which leads to improved reliability but which precedesthe occurrence of ceiling effects). However, a two-stage

    33

  • 7/27/2019 - Simulation of a Model Tank Gunnery Test.pdf

    44/84

    rise in the livefire learning curve would complicate matters.A rapid change in scores over the first two or three replica-tions would not, in fact, reflect true proficiency. A moredesirable measure, therefore, might be a composite acrossreplications four to six where performance would still beimproving, but at a more typical rate. This measure wouldpresumably be more stable and hence more reliable.The possibility of carry-over effects from the simulatedtest to the livefire test and vice versa also exists in thereplicated design. The effect may again be dealt with bycounterbalancing treatment order across two groups of subjects.Repeated ad