Embed Size (px)
Citation preview
National Aeronautics and Space Administration National Aeronautics and Space Administration
Workload Measurement in Human Autonomy Teaming: How and Why ?
Jay Shively NASA-Ames Research Center
27 June 2016
https://ntrs.nasa.gov/search.jsp?R=20160008388 2018-07-15T13:58:44+00:00Z
Outline
• Meandmybiases– NASA-TLX– MIDAS
• Categoriesofmetrics– Subjec@ve– Objec@ve– Physiological– Computa@onal
• WorkloadandHumanAutonomyTeaming– Changesworkload
• Righttoolfortherightques@on– Assessment– Predic@on– Design
2
Jay
• WorkedwithSandyHartandtheTLXteamfrom1984un@labout1990• ManagedtheMan-MachineIntegra@onDesignandAnalysisSystem(MIDAS)
1990–1995• Bias?• Yes,butnotinthewayyoumightthink---Iknowwheretheskeletonsare
buried!!
3
NASA-TLX
• NASATaskLoadIndex(TLX)– Includessixsubjec@vedimensionsofglobalworkload– Firstvalida@onanddevelopmentstudiesdonebyHartandStaveland(1988)– StudieshaveusedNASA-TLXalongwithphysiologicalmeasures
• Borghinietal.(2012)conductedastudyassessingworkloadduringdrivingavehicleusingEEG,alongwithotherphysiologicaldata
• Par@cipantgivenNASA-TLXques@onnaireattheendofeachcondi@onforsubjec@veworkloadassessment
• Correla@onbetweenNASA-TLXscoresandphysiologicaldata– Some@meshasstrongesteffectsizeoutofotherworkloadmeasures– Sensi@vetobothtasktypeanddualtasking(Machews,Reinerman-Jones,Barber,&AbichIV,2015)
– Consideredtobemorefavorableforsubjectsascomparedtoothermeasuresofworkload(Caoetal.,2009)
4
NASA-TLXExample
NASA-TLXExample
SWAT
• DevelopedbyReid&Nygren(1988)
• Developsasingle,globalra@ngscalewithintervalproper@es(Rubioetal.,2004)
• Usedinavarietyoftasksegngs(Hendy,Hamilton,&Landry,1993)
BedfordScale
• Modifica@onoftheCooperHarperScale(Roscoe,1984)• Unidimensional• Usesadecisiontreeandexamineswhether:
– Thetaskcanbesuccessfullycompleted– Thelevelofworkloadexperiencedwastolerable– Thelevelofworkloadwassa@sfactorywithoutreduc@on
• Tapsintooperator’ssparementalcapacity• Currentlyveryfewstudieshaveuseditincontrolledsegngs• Moreokenusedinappliedsegngs• Notenoughdataonvalidityofthescaleavailable(NATOGuidelinesonHumanEngineeringTes@ngandEvalua@on,2001)
BedfordScale
ModifiedCooperHarperScale
• Usedmostokeninavia@on• Unidimensional• Usesdecisiontreera@ngscale,withascoreof1indica@ng“best”andascoreof10indica@ng“worst”
• Rela@velysensi@vetochangesinworkload(Wierwille&Connor,1983)andvarioustypesofworkload
• Dataiscollectedakerthetrial,par@cipantspooratrecallingpastmentalevents(Wierwille&Casali,1986)
• Limitedtomanualcontroltasks
ModifiedCooper-HarperScale
11
Cooper-GarrodEstateVineyards
Subjec@vePros:• Precygoodideaofoperator’sexperienceofworkload(Crabtree,Bateman,&Acton,1984)• Cheapandeasy(Stanton,Salmon&Walker,2007)• Yearsofuse=valida@on(?)• Goldstandard???• Diagnos@cwhencombinedwithobjec@vemeasures(Crabtree,Bateman,&Acton,1984)
Cons:• Phenomenonhastobeavailableforintrospec@on(seeSA)(Yeh&Wickens,1984)• Retrospec@ve,i.e.,notreal-@me• Memorial(pronetomemoryfailure?)(Muckler&Seven,1992)• Notcon@nuous,reflectaverageorpeak• Subjec@ve–NOTobjec@vedata 13
NASA-TLXBias• Almostanyofthese(orascale1–100)givesyouaprecygoodideaofoverall
workloadexperiencedbytheoperator• NOTreallydiagnos@c–Iknowofnosystemdesignevermodifiedbecauseof
toohigh“physicalworkload”• Individualdifferences“weigh@ngs”reducevariance–butmathema@callyhave
to!!• Nooneknowswhatthe“ownperformance”scalemeans–maybeSandy• Haveeffec@velybecomethegoldstandardagainstwhichothermetrics–such
asphysioorcomputa@onalmodelsarejudged
14
IssuesofSubjec@veMeasures
• Processingcharacteris.csarelost(Yeh&Wickens,1984)– Limitedinscope– Onlyprovidesscalarmeasures
• Dissocia.onsbetweensubjec.veandobjec.vemeasures(Yeh&Wickens,1988)
• Difficulttocompareresultsacrossscales(Gopher&Braune,1984)– Lackofformaltheoryforworkload– Subjec@vemeasurementscalesareinfluencedbyhowexperimenters
selectscalardimensionsforra@ng
Objec@ve
Metrics:• EmbeddedSecondarytasks• Naturallyoccurringsecondarytasks• WLprobes
16
WorkloadProbes
• Situa.onPresentAssessmentMethod(SPAM)– On-lineprobemethodthatcanmeasureworkload,inaddi@ontoSA(Stanton,Salmon,&Walker,2007)
– Readinesslatency:Timefromonsetof“ready”promptforquerytoanindividual’sresponsetothepromptactsasanindicatorforworkload
• Objec@ve(Stanton,Salmon,&Walker,2007)• Some@mesaccompaniedbyanauditorywarningsignal(Pierce,2012)
– Queriescanalsoaskoperatortoreportcurrentmentalworkload(Silvaetal.,2013)• Scale• Subjec@vera@ng
– Notintrusivetooperatorperformanceandworkload(Silvaetal.,2013;Keeleretal.,2015)
17
EmbeddedSecondaryTasks
• Operatorperformsaprimarytaskinaddi@ontoasecondarytask
• Awidevarietyofsecondarytaskshavebeenusedinstudies(Ogden,Levine,&Eisner,1979)
• RTonsecondarytasksokenshowsgreatestsensi@vitytoworkloadchanges
• Timees@ma@ontaskisalsosensi@ve,butcanbeintrusive(Wierwille,Rahimi,&Casali,1985)
• Changedetec@onalsosuccessfulandlessintrusive(Teo,Reinerman-Jones,&Szalma,2015)
Objec@ve
Pros:• Objec@vedata:RT,error
Cons:• Spacifictoeachimplementa@on• Lowdatarate• Canbedifficulttoimplement• Momentarymeasure(notcon@nuous)• Can’timplementinsomesitua@ons(realcockpit-can’taddsecondarytasks)
19
Physiological
Metrics:• Heart-rate(variability)• Eyegaze• GSR• Eye-blink• fMRI• EEG• fNIRS
20
PhysiologicalMeasures• EyeGaze
– Gold,Körber,Lechner,andBengler(2016)• Studytodeterminehowtrafficdensityandverbaltasksaffecttake-overperformanceinhighlyautomateddriving
– I.e.howmuch@medoesthedriverneedinordertoregaincontrolofanautomatedvehicleifasitua@onnecessitatesthis?Doestrafficdensityandaverbaltaskhaveanaffectonperformanceoftakingbackcontrol?
• Usedeyegazebehaviorasameasureofworkload– Lowerhorizontalgazedistribu@on(HGD)=Moreworkload(Wang,Reimer,Dobres,&Mehler,2014)
PhysiologicalMeasures• HeartRateVariability
– Decreaseinheartratevariabilitymayindicateanincreaseinmentalworkload(Mulder,1980)
– Strang,Best,andFunke(2014)• Studiedmentalworkloadofpar@cipantsinasimulatedtrainingexercise
involvingrealis@c,large-scaleair-combatscenarios.• Examinedtheabilityofheartratetopredictmentalworkload.• Somedatatosupportthatheartratemaybeabletopredictmental
workload,butthisrela@onshipisinconsistent.
PhysiologicalMeasures
• EyeBlink– Some@mesmeasuredthroughelectrooculography(EOG)(Veltman&Gaillard,1996)
– Lengthorfrequencyofblink– Notalwayssensi@vetochangesinworkload(Wierwille&Connor,1983)– Mightneedtobecombinedwithothereyetrackingtechniquestobemorereliable(Orden,Limbert,&Makeig,2001)
PhysiologicalMeasures• Func.onalMagne.cResonanceImaging(fMRI)
– Monitoringcerebralbloodflowvelocity(CBFV)
– AsCBFVincreasesintheprefrontalcortex,mentalworkloadincreases(Parasuraman&Caggiano,2005)
– Highlyconstrainedenvironment
– Limitswhatkindofac@vi@escanbeanalyzed(Warm,Parasuraman,&Machews,2008)
PhysiologicalMeasures
• Electroencephalogram(EEG)– Electrodesareplacedonthescalpovervariousbrainareas:
• Fz,F3,F4,Cz,C3,C4,Pz,P3,P4
– Differenttypesofbrainwaves• Alpha(7-14Hz)• Beta(14-30Hz)• Theta(4to7Hz)• Delta(upto4Hz)
– Asmentalworkloadincreases,alphawavesarereplacedbybetawaves,andfrontalthetawavesareincreased(Borghini,etal.,2012)
PhysiologicalMeasures
• GalvanicSkinResponse(GSR)– Measurementofresistanceofskin@ssuetoelectricalcurrent
– Measuredthroughpalms,inside/outsideofwrist,archoffoot,forehead,orfingers
– Suscep@bletoindividualdifferencesinresponse(Wierwille,1979)
– Foundtobeassociatedwithcogni@veworkload(Shietal.,2007)• MeanGSRincreasesascogni@veload
increases
PhysiologicalMeasures
• Func.onalNear-InfraredSpectroscopy(fNIRS)– Rela@velynewmeasure
– Monitorseleva@onofrSO2
– HigherrSO2levels=morecogni@velydemanding(Machews,Renierman-Jones,Barber,&AbichIV,2015)
– Notalwayssensi@veenoughtochanges,butdoescorrelatewellwithotherphysiologicalmeasuresofworkload(e.g.HR)(Teo,Reinerman-Jones,&Szalma,2015)
Physio
Pros:• Con@nuous• Poten@allyunobtrusive• FaceValidity–looksscien@fic(Levinetal.,2006)• Supplementssubjec@vemeasures(Wierwille&Eggemeier,1993)Cons:• Nota“pure”workloadsignal• Individualdifferences(Wierwille,1979)• Sensi@vetoexternalevents/sources• Poorcorrela@ontosubjec@vemetrics(goldstandard)
28
Computa@onal
Models• MIDAS• IMPRINT• OMAR• ACT-r*
HumanPerformanceModelinginAvia@on,Foyle,D.C.&Hooey,B.L.(2008)
29
MIDAS
• Man-machineIntegra@onDesignandAnalysisSystem(MIDAS)
• NASAAmesResearchCenterhumanperformancemodel(HPM)sokwaretool
• Predicthuman-systemperformanceandmodelhumanerror
• “What-if”analyses(Gore,2011)• 3-Drapidprototyping,dynamicsimula@on,andhuman
performancemodelingwiththeaimtoreducedesigncycle@me
• Linksavirtualhumantoacomputa@onalcogni@vestructurethatrepresentshumancapabili@esandlimita@ons(NASA,2016)
• Currentlyv5beingused
MIDAS
• Cogni@vecomponent– Perceptualmechanism– Memory– DecisionMaker– Responseselec@onarchitecture
• Outputsinclude:– Dynamicvisualrepresenta@ons– Timelines– Tasklists– Cogni@veloadsalongresourcechannels– Actual/perceivedS.A.– Humanerrorvulnerability– Humanperformancequality
MIDASArchitecture
Crew Station Design, Cockpit Geometry,
Display Control Layout, Cockpit Equipment
Functionality
Mission / Tasks, Flight Profiles, Waypoints,
Other Scenario Objects, Planned Operator Mission Activities
Operator Characteristics,
Cognitive Physical, Motor Response
Parameters
Inputs
Cockpit Design Editor
Equipment Editor
Route Editor
Activity Editor
Interactive Graphical Interface
Display Animation
Interactive Graphical Interface
Runtime Data Graphical Displays
Summary Data Graphical Displays
3D Graphical Display
ErgonomicAnalysis
ResultsReach,Fit,&OtherMIL-STDAnalysis,VisibilityandLegibility
MissionOperatorPerformanceMeasures
Accuracy,InfoFLow,ResponseTimes,
Ac@vityTraces,TaskLoadTimelines,
ResourceConflicts
Ac@vi@es,EquipmentStatus
Interactive Cockpit Design Tools
Simulation System Models & Tools
Jack
AnthropometricModel
VisionModels
SymbolicOperatorModel
Vision,Percep@on,Acen@on,WorldRepresenta@on,Decision,Scheduling,Task
Loading,MissionAc@vi@es,Motor
WorldModelVehicle,CockpitEquipment,FlightDynamics
User Interface User Interface Outputs
MIDAS
• AirMIDAS– Avia@onspecificversionofMIDAS– ModelsATCo– Switchesbetweencontrolstrategiesdependingonnumberofaircrakundercontrolandthecomplexityofmaneuverstheaircrakhavetoperform
IMPRINT
• ImprovedPerformanceResearchIntegra@onTool(IMPRINT)• DevelopedbytheU.S.ArmyResearchLab,HumanResearch&EngineeringDirectorate
• Sokwareisavailableforfreefor– U.S.governmentagencies– U.S.privateindustrywithU.S.governmentcontract– U.S.collegesanduniversi@esworkinginHSI
• Designedtosupportsystemperformancethroughlifecycle• Canhelpto
– Setrealis@csystemrequirements– Iden@fySoldier-drivenconstraintsonsystemdesign– Evaluatethecapabilityofavailablemanpowerandpersonneltoeffec@velyoperateandmaintainasystemunderenvironmentalstressors(U.S.Army,2016)
IMPRINTModules
Es@matethetypeofindividualswhowillbeavailabletooperateandmaintainthesystem
Es@matetheeffectofoperatorperformanceonsystemperformance,
including@me,accuracy,ormentalworkload
Es@matemaintenanceman-hoursrequiredtoacainacceptablesystem
availability
Es@matethemanpowerneededtocompletetherou@neandunplannedworkperformedbya
forceunit
Warfighter
Equipment
Missions
Forces
OperatorModelArchitecture(OMAR)
• Providesasimula@onenvironmentthatallowsformodelinghumanoperators,wheretheywork,andtheen@@esofthelargerworldthatarereflectedintheirworkplaces
• A“produc@onrule-basedexecu@veprocess”regulatesschedulingofcompe@ngtasks
• Emphasisondevelopingmul@ple-taskbehaviorsfrom“func@onalcenters”thatareopera@ngatthesame@mewithoutanexecu@veorcentralcontrol(Deutsch,1998)
UniqueCharacteris@csofOMAR
• S@mulidirectlyaffectproceduralmemory• “Func@on-specificprocedures”thatrepresent
specificbrainareascoordinatethecomple@onoftasks
• Resul@ngbehaviorsmaybeconsidered“intelligent”
• Taskcomple@onismediatedonapairwisebasisandnotthroughacentralexecu@ve.(Deutsch,1998)
Computa@onal
Pros:• Learnalotbyformalizingdescrip@onofyoursystem• Objec@ve(sortof–input,assump@ons,etc.)• Whatifques@onscanbeasked• Canmodandre-run• Consistent• Canbedonewithno@onalsystemCons:• Benefitmightlargelybeintheprocess(moreofmybias)• Needadetailedtaskanalysis/systemdesign• !@$#%input>!@#$output
Rela@onshipBetweenAutoma@on&Workload
• Automa@ondoesnotnecessarilyreduceworkload,justchangesit.• Automa@onchangesanoperator’srolefrommanuallycontrollingasystemto
monitoringtheautomatedsystem(Parasuraman&Riley,1997)
• Examples– Wiener(1989)
• Pilotresponsesweredividedwhenaskedwhetherworkloadwasdecreasedinamoreautomatedcockpit
– Warm,Dember,&Hancock(1996)• Monitoringtaskscanleadtounderarousalandincreasedmentalworkload
– Wiener&Curry(1980)• Althoughautoma@onmayreducemanualworkload,itmayincreaseoverallworkload
asaresultofincreasedmentalworkload.
Rela@onshipBetweenWLandAutoma@on
Workload Unpredictability
Competency
W U W U
C C
Increased Human Mgt (Adaptable) Increased Automation Mgt
(Adaptive)
Miller, C.A. & Parasuraman, R. (2007)
23 June 2016 41
APlaybook®ApproachtoDelega@on
• AmeansofDelega@on• Playscontainanimplicitgoal• Playsdefinea“template”ofplan/behavioralterna@ves—a“space”ofdelegatedplanningauthority
– “pre-compiled”withconvenientlabel– Supervisorcanfurtherconstrain/s@pulateasdesired–byreferencetoplaystructure
– Monitoringandinforma@onrepor@ngfacilitatedbysharedintentstructure
– Dynamic,real@merevisionandtuning=“callingsignals”
• Subordinatesresponsibleforbest-effortacemptswithinplayconstraints
A page from Alonzo Stagg’s 1927 Playbook
Playbook&HAT
• SingleOperatorcontrolofmul@ple,heterogeneousUAS(Simula@onsandflighttests)
– Toptenpre-definedPlays–fromoperators• Conveysupport• Troopsincontact• Reconanarea
– IncreasedPerformance– DecreasedWorkload
• HumanAutonomyTeaming– ReducedCrewComplimentinCommercialAvia@on– Onestepfurther–notjustdelega@on,butdiscussion,nego@a@on,jointproblemsolving
– Automa@on(andinterface)adaptsby(largely)pilot-directedcontext
Whymeasureworkload?
SystemLife-cycle• Design• Evalua@on(R&D)• Evalua@on(Opera@onal)• Embedded(adap@veautoma@on)
• WCFielde:WorkloadConsultantforFieldEvalua@on
Design
Environment:• Systemdoesn’texist• SME’smaybetangen@al• Non-real@meDecisions:• Rolesandresponsibili@es• Informa@onflow/displays• CrewsizeMetric:Computa@onalModels
Evalua@on(R&D)Environment:• Prototypesystem• Focusonothervariables• Real-@meQues@ons:• Workloadtoohigh/low• EffectofvariablesonWLMetrics:*• Subjec@ve• Objec@ve/secondary• Phsyio*Choicedependsonabilitytoinsert/iden@fysecondarytasks
Evalua@on(Opera@onal)
Environment:• System• Realusers• Real-@meQues@ons:• Workloadtoohigh/lowMetrics:• Subjec@ve• Physio(ifnon-intrusive)
Embedded(e.g.,Adap@veAutoma@on)
Environment:• System(WLevalispartofthesystem)• Realusers• Real-@meQues@ons:• Workloadtoohigh/lowMetrics:• Subjec@ve• Performance• Physio(ifnon-intrusive)
Summary
• ProsandConsofallapproaches• DrivenbytheQUESTION• Stronglyadviseusingabaceryofmeasurestoconvergeon“workload”
• Adaptablevs.Adap@veAutoma@on…
ReferencesBorghini,G.,Vecchiato,G.,Toppi,J.,Astolfi,L.,Maglione,A.,Isabella,R.,Caltagirone,C.,Kong,W.,Wei,D.,Zhou,Z.,Polidori,L.,
Vi@ello,S.,Babiloni,F.(2012).Assessmentofmentalfa@gueduringcardrivingbyusinghighresolu@onEEGac@vityandneurophysiologicindices.34thAnnualInterna1onalConferenceoftheIEEEEMBS.
Budiu,R.(n.d.).About.RetrievedJune07,2016,fromhcp://act-r.psy.cmu.edu/about/.Cao,A.,Chintamani,K.K.,Pandya,A.K.,&Ellis,R.D.(2009).NASATLX:[email protected]
researchmethods,41(1),113-117.Crabtree,M.S.,Bateman,R.P.,&Acton,W.H.(1984,October).Benefitsofusingobjec@[email protected]
ProceedingsoftheHumanFactorsandErgonomicsSocietyAnnualMee1ng(Vol.28,No.11,pp.950-953)[email protected],S.(1998).InterdisciplinaryFounda@onsforMul@ple-taskHumanPerformanceModelinginOMAR.Twen1ethAnnualMee1ng
oftheCogni1veScienceSociety.Fallahi,M.,Motamedzade,M.,Heidarimoghadam,R.,Soltanian,A.R.,&Miyake,S.(2016).Effectsofmentalworkloadonphysiological
andsubjec@veresponsesduringtrafficdensitymonitoring:afieldstudy.Appliedergonomics,52,95-103.Geddie,J.C.,Boer,L.C.,Edwards,R.J.,Enderwick,T.P.,&Graff,N.(2001).NATOGuidelinesonHumanEngineeringTes@ngand
Evalua@on(No.RTO-TR-021).NATOResearchandTechnologyOrganiza1on(France).Gold,C.,Körber,M.,Lechner,D.,Bengler,K.(2016).TakingOverControlFromHighlyAutomatedVehiclesinComplexTraffic
Situa@ons:TheRoleofTrafficDensity.HumanFactors,58(4),642-52.doi:10.1177/0018720816634226.Gopher,D.,&Braune,R.(1984).Onthepsychophysicsofworkload:Whybotherwithsubjec@vemeasures?.HumanFactors:The
JournaloftheHumanFactorsandErgonomicsSociety,26(5),519-532.Gore,B.F.(2011).Man–machineintegra@ondesignandanalysissystem(MIDAS)v5:Augmenta@ons,mo@va@ons,anddirec@onsfor
aeronau@[email protected](p.43-54).SpringerMilan.
References
Hart,S.G.,&Staveland,L.E.(1988).DevelopmentofNASA-TLX(TaskLoadIndex):[email protected],52,139-183.
Hendy,K.C.,Hamilton,K.M.,&Landry,L.N.(1993).Measuringsubjec@veworkload:whenisonescalebecerthanmany?.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,35(4),579-601.
Keeler,J.,Bagste,H.,Hallec,E.C.,Roberts,Z.,Winter,A.,Sanchez,K.,Strybel,T.Z.,&Vu,K.P.L.(2015).MayIInterrupt?TheeffectofSPAMProbeQues@onsonAirTrafficControllerPerformance.ProcediaManufacturing,3,2998-3004.
Lebiere,c.,Anderson,J.R.,Bothell,D.(2001).Mul@-TaskingandCogni@veWorkloadinanACT-RModelofaSimplifiedAirTrafficControlTask.ProceedingsoftheTenthConferenceonComputerGeneratedForcesandBehaviorRepresenta1on,2001.
Levin,S.,France,D.J.,Hemphill,R.,Jones,I.,Chen,K.Y.,Rickard,D.,...&Aronsky,D.(2006).Trackingworkloadintheemergencydepartment.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,48(3),526-539.
Machews,G.,Reinerman-Jones,L.E.,Barber,D.J.,&Abich,J.(2015).Thepsychometricsofmentalworkloadmul@[email protected]:TheJournaloftheHumanFactorsandErgonomicsSociety,57(1),125-143.
Moray,N.(1982)[email protected]:TheJournaloftheHumanFactorsandErgonomicsSociety,24(1),25-40.
Muckler,F.A.,&Seven,S.A.(1992).Selec@ngperformancemeasures:"Objec@ve"versus"subjec@ve"measurement.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,34(4),441-455.
NASA(2016).Man-machineIntegra@onDesignandAnalysisSystem(MIDAS).RetrievedonJune7,2016fromhcp://human-factors.arc.nasa.gov/groups/midas/.
Ogden,G.D.,Levine,J.M.,&Eisner,E.J.(1979).Measurementofworkloadbysecondarytasks.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,21(5),529-548.
References
Orden,K.F.,Limbert,W.,Makeig,S.,&Jung,T.P.(2001).Eyeac@vitycorrelatesofworkloadduringavisuospa@almemorytask.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,43(1),111-121.
Parasuraman,R.,Riley,V.(1997).HumansandAutoma@on:Use,Misuse,Disuse,Abuse.HumanFactors,39(2),230-253.Parasuraman,R.,&Caggiano,D.(2005).Neuralandgene@cassaysofhumanmentalworkload.Quan1fyinghumaninforma1on
processing,123-149.Pierce,R.S.(2012).TheeffectofSPAMadministra@[email protected]:TheJournaloftheHumanFactors
andErgonomicsSociety,54(5),838-848.Reid,G.B.,&Nygren,T.E.(1988).Thesubjec@veworkloadassessmenttechnique:Ascalingprocedureformeasuringmental
workload.Advancesinpsychology,52,185-218.Roscoe,A.H.(1984).Assessingpilotworkloadinflight.RoyalAircrakEstablishment.Bedford(England).Rubio,S.,Díaz,E.,Mar�n,J.,&Puente,J.M.(2004).Evalua@onofsubjec@vementalworkload:AcomparisonofSWAT,NASA-TLX,and
workloadprofilemethods.AppliedPsychology,53(1),61-86.Shi,Y.,Ruiz,N.,Taib,R.,Choi,E.,&Chen,F.(2007,April).Galvanicskinresponse(GSR)[email protected]'07
extendedabstractsonHumanfactorsincompu1ngsystems(pp.2651-2656).ACM.Silva,H.I.,Ziccardi,J.,Grigoleit,T.,Bagste,V.,Strybel,T.Z.,&Vu,K.P.L.(2013).AretheintrusiveeffectsofSPAMprobespresent
whenoperatorsdifferbyskilllevelandtraining?.InHumaninterfaceandthemanagementofinforma1on.Informa1onandinterac1ondesign(p.269-275).SpringerBerlinHeidelberg.
Stanton,N.,Salmon,P.M.,&Rafferty,L.A.(2013).Humanfactorsmethods:[email protected],Ltd.
Strang,A.J.,Best,C.,Funke,G.J.(2014).HeartRateCorrelatesofMEntalWorkloadinaLarge-ScaleAir-CombatSimula@onTrainingExercise.ProceedingsoftheHumanFactorsandErgonomicsSociety58thAnnualMee1ng.
Teo,G.,Reinerman-Jones,L.,Machews,G.,&Szalma,J.(2015).ComparisonofMeasuresUsedtoAssesstheWorkloadofMonitoringanUnmannedSysteminaSimula@onMission.ProcediaManufacturing,3,1006-1013.
U.S.Army(2016)[email protected],2016fromhcp://www.arl.army.mil/www/default.cfm?page=445.
Veltman,J.A.,&Gaillard,A.W.K.(1996).Physiologicalindicesofworkloadinasimulatedflighttask.Biologicalpsychology,42(3),323-342.
Wang,Y.,Reimer,B.,Dobres,J.,Mehler,B.(2014).Thesensi@vityofdifferentmethodologiesforcharacterizingdrivers’gazeconcentra@[email protected],227-237.doi:10.1016/j.trf.2014.08.003
Warm,J.S.,DemberW.N.,Hancock,P.A.(1996).Vigilanceandworkloadinautmatedsystems.Automa1onandhumanperformance:Theoryandapplica1ons,183-200.
Warm,J.S.,Parasuraman,R.,&Machews,G.(2008).Vigilancerequireshardmentalworkandisstressful.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,50(3),433-441.
Wiener,E.L.(1989).HumanFactorsofAdvancedTechnology(“GlassCockpit”)TransportAircrak.NASAContractorReport177528.Wiener,E.L.,Curry,R.E.(1980).Flight-deckautoma@on:promisesandproblems.Ergonomics23(10),995-1011.Wierwille,W.W.,&Connor,S.A.(1983).Evalua@onof20workloadmeasuresusingapsychomotortaskinamoving-baseaircrak
simulator.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,25(1),1-16.
References
Wierwille,W.W.,Rahimi,M.,&Casali,J.G.(1985).Evalua@onof16measuresofmentalworkloadusingasimulatedflighttaskemphasizingmedia@[email protected]:TheJournaloftheHumanFactorsandErgonomicsSociety,27(5),489-502.
Wierwille,W.W.,&Eggemeier,F.T.(1993).Recommenda@onsformentalworkloadmeasurementinatestandevalua@onenvironment.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,35(2),263-281.
Wierwille,W.W.(1979).Physiologicalmeasuresofaircrewmentalworkload.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety,21(5),575-593.
Yeh,Y.H.,&Wickens,C.D.(1984).Thedissocia@[email protected],Y.Y.,&Wickens,C.D.(1988).Dissocia@[email protected]:TheJournalof
theHumanFactorsandErgonomicsSociety,30(1),111-120.
