A Random Glance at the Flight Deck PilotsrsquoScanning Strategies and the Real-TimeAssessment of Mental WorkloadFrancesco Di NoceraMarco CamilliMichela TerenziUniversity of Rome ldquoLa Sapienzardquo Italy

ABSTRACT Based on previous research showing the usefulness of spatial statistics in de-tecting randomness in the distribution of eye fixations this study investigated the ocularbehavior of professional pilots engaged in a simulated flight The distribution of eyefixations was used as an indirect index of mental workload Eye movements were record-ed during the different phases (departure to landing) of a simulated flight and wereanalyzed using spatial statistics algorithms Results showed sensitivity of spatial disper-sion indices to variations in mental workload higher during departure and landinglower during climb and descend and the lowest during the cruise phase This findingprovides additional evidence of the utility of fixations distribution as a real-time measureof mental workload and consequently as a trigger for adaptive automation

Introduction

HUMAN FACTORS AND ERGONOMICS (HFE) RESEARCH CONTINUES TO DEMONSTRATE THAT

extreme levels of mental workload decrease an individualrsquos ability to react to incom-ing information and increase the likelihood of human error The analysis of mentalworkload has increasingly gained in popularity within the aviation domain (Kan-towitz amp Casper 1988 Wickens 2002) Several strategies have been devised to ob-tain a real-time measure of this construct (eg Lee amp Liu 2003 Wilson 2002) Suchreal-time measures would be extremely useful given that aircraft pilots experiencefluctuations of mental workload because of task demands that range from very highto very low The takeoff and landing phases as well as occasional unexpected eventsthat may occur during the flight generate the highest level of workload Indeed pi-lotsrsquo subjective ratings of overall workload were found to vary significantly acrossflight segments with the highest ratings occurring after departure and landing seg-ments (Hart Hauser amp Lester 1984) Departure and landing are also known to pro-duce the greatest number of changes in psychophysiological data because of theincreased level of cognitive demand placed on pilots during these important maneu-vers (Wilson 2002)

In contrast to departure and landing phases workload is minimal during the

271

ADDRESS CORRESPONDENCE TO Francesco Di Nocera Cognitive Ergonomics Laboratory Departmentof Psychology University of Rome ldquoLa Sapienzardquo Via dei Marsi 78ndash00185 Rome Italy E-mail dinocerauniroma1it Visit the JCEDM Online Companion at httpcedmwebexonecom

Journal of Cognitive Engineering and Decision Making Volume 1 Number 3 Fall 2007 pp 271ndash285 DOI101518155534307X255627 copy 2007 Human Factors and Ergonomics Society All rights reserved

cruise phase (particularly when the pilot is confined to a monitoring role by auto-mation) which generates boredom and a state of reduced alertness that makes it dif-ficult to handle emergencies (see Scerbo 2001 for an account on the relation betweenboredom workload stress and vigilance) The allocation of responsibility between pi-lot copilot navigator and automation is one of the key factors influencing mentalworkload and its real-time assessment might be useful to optimize functional alloca-tion as well as to implement workload-matched adaptive automation

Several candidate measures have been proposed for measuring mental work-load in real time Psychophysiological measures are among the best indices becausethey provide information on the ongoing functional state of the operator (Scerbo etal 2001) For example it is well known that some EEG bandwidths and componentsof the event-related potentials are sensitive to variations in mental workload Howev-er these indices are difficult to collect and analyze in real time and their reliabilityand stability are seldom assessed (see Di Nocera 2003) Moreover real-time filteringto obtain single-trial data are still an issue that is far from being solved

Among the different psychophysiological measures of operator load eye move-ments are relatively easy to collect in real operational environments Recent eye-tracking technology is also not intrusive and data are relatively easy to analyze Eyemovements however can be analyzed in different ways Measurements based on dif-ferent aspects of ocular behavior such as the number of fixations and their durationor the saccadesrsquo length and speed have been used to derive metrics of mental work-load However these techniques need post hoc analyses and are not well suited toshow real-time changes in the operatorrsquos functional state

Pupillary response has also been proposed as an index of the amount of cogni-tive processing (see Beatty 1982 for a review) Generally it has been reported (acrossdifferent tasks involving different cognitive processes) that increased processing loadevokes greater pupillary dilation responses For example pupil size recorded dur-ing digit-span recall tasks systematically increases following the presentation of eachadditional to-be-recalled digit (Granholm Asarnow Sarkin amp Dykes 1996) Al-though pupil size offers a spontaneous measure of workload recording this ocularresponse requires implementing several experimental controls such as keeping con-stant illumination of the experimental setting and the brightness of the stimuli Thisrepresents a critical aspect in many studies involving visual or ecological tasks in thehuman factorsergonomics (HFE) domain

Other spontaneous ocular measures sensitive to variations in mental workload arethe frequency and duration of eye-blinks They have been reported to be inverselycorrelated to mental load (Brookings Wilson amp Swain 1996 Hankins amp Wilson1998) but it is worth noting that different tasks can generate different patterns de-pending on the type of index employed Some indexes can be sensitive to visualdemands but insensitive to cognitive demands For example Wilson Fullenkampand Davis (1994) found that the durations of eye-blinks decreased in a visual track-ing task (which generates a minimum mental workload) whereas the durations ofeye-blinks remained stable in a more cognitively engaging task (a flight simulation)

A technique often used in HFE research that provides a different perspective from

272 Journal of Cognitive Engineering and Decision Making Fall 2007

other eye-movement methods is the analysis of visual patterns For example ItohHayashi Tsukui and Saito (1990) and Diez et al (2001) used this method for gath-ering information about the scanning strategies of pilots interacting with a Boeing 747simulator Usually these studies analyze the ocular activity within specific areas ofinterest (AOI) each one including a tool inspected by pilots during a simulated flightAlthough the scanpath is usually used to obtain qualitative information it can alsobe used in association with advanced computing techniques Particularly visual scan-ning randomness (or entropy) has been proposed as a measure of mental workload(Ephrath Tole Stephens amp Young 1980 Harris Glover amp Spady 1986 Tole Steph-ens Vivaudou Ephrath amp Young 1983)

In thermodynamics the concept of entropy is related to the quantity of disorderin a system (in this case the disorder in visual exploration) The rationale underly-ing this approach is that the exploration pattern becomes more stereotyped (ieless random) as workload increases In contrast as mental workload decreases therandomness of the pattern should increase Hilburn Jorna Byrne and Parasuraman(1997) corroborated this hypothesis in a study on air traffic controllersrsquo scanningstrategies However using entropy analysis Kruizinga Mulder and de Waard (2006)found an opposite pattern which suggests that more studies are needed in order tounderstand the relationship between mental workload and visual scanning

Using a different approach Di Nocera Terenzi and Camilli (2006) also founddivergent results They used a distance statistic indicator called the Nearest NeighborIndex (NNI Clark amp Evans 1954) to analyze eye fixations data Unlike the entropymeasure which ignores fixations outside the predefined regions of interest this ap-proach takes into account all fixations considering them all informative The follow-ing section summarizes this strategy for analyzing eye movement data

Analyzing Spatially Distributed DataThe measurement and description of pattern distribution were first addressed

in reference to plant and animal populations In forestry for example the positionsof trees in a forest form a point pattern in the plane Information about the distribu-tion of such points is relevant for investigating phenomena such as plant infectionsor growing patterns In the beginning the basic assumption was that individuals ofmost populations (eg plants animals or fossils) were distributed at random butit soon became clear that the randomness assumption was not appropriate The issuethen became to establish the degree of variation from random expectation as well asthe statistical significance of differences in the distribution pattern of two or more pop-ulations To achieve this goal Clark and Evans (1954) introduced the NNI which isthe ratio between the average of the observed minimum distances between points andthe mean random distance that one would expect if the distribution were randomFifty years later this index is still one of the most widely used distance statistics inagriculture paleontology and crime analysis (All deal with spatially arranged data)

As a first step the nearest neighbor distance or d(NN) should be computed asfollows

A Random Glance at the Flight Deck 273

d(NN) = min 1 le j le N j ne i

where min(dij) is the distance between each point and the point nearest to it and Nis the number of points in the distribution

This index is nothing more than the average of the nearest neighbor distancesThe second step is to compute the mean random distance or d(ran) that is the d(NN)one would expect if the distribution were random

d(ran) = 05

where A is the area of the region (the measurement unit of the index is related to theone used here) and N is the number of points

The final step is the actual computation of the NNI as follows

NNI =

Of course this ratio is equal to 1 when the distribution is random Values less than1 suggest grouping whereas values greater than 1 suggest regularity (ie the pointpattern is dispersed in a nonrandom way) Theoretically NNI lies between 0 (max-imum clustering) and 21491 (strictly regular hexagonal pattern)

Di Nocera et al (2006) applied this procedure to eye fixations (given that they arepoint patterns as well) and found this index to be sensitive to variation in mentalworkload showing a tendency toward randomness in the high-workload conditionThis is the opposite of what the entropy-based method would predict Howeverentropy studies have used ocular data within specific and static AOI whereas theDi Nocera et al (2006) study used ocular data gathered from a dynamic scene (par-ticipants were requested to play the Asteroids PC game in two difficulty conditions)within a convex hull defined by the outermost fixations in the distribution The highmental workload condition was obtained by preventing the use of the weapon todestroy the asteroids whereas the lowmoderate-workload condition consisted ofthe regular game allowing the use of the weapon

Considering the dynamic nature of the Asteroids game (the ship moves aroundin the screen area) it is possible that the different distributions of fixations that havebeen found were strategy-driven rather than workload-driven Indeed even if the twoversions of the game were geometrically equivalent (having the same number of as-teroids either between conditions throughout the game) avoiding the asteroids mightfavor a strategy aimed at spreading the fixations over a wide area whereas the shoot-ing condition might have been supported by a strategy based on focusing over theship and target positions

In order to address the role of these differences in the present study we appliedthe same rationale to investigate ocular behavior during interaction with a somewhatldquostaticrdquo visual scene To this aim a flight simulation task was used comprising both

d(NN)mdashmdashmdashd(ran)

AmdashmdashN

(dij)mdashmdashN

N

sumi =1

274 Journal of Cognitive Engineering and Decision Making Fall 2007

high workload (departure and landing) and low to moderate workload (climb cruiseand descent) phases Of course this was ldquostaticrdquo in the sense that in a flight deck thelocations of objects to monitor (namely the instruments) did not change over timeeven if the visual scene outside the cockpit did change

Method

Participants Ten instrument flight rules (IFR) -licensed pilots belonging to the Italianpolice (all males mean age = 4110 years SD = 482) volunteered for this study Pilotswere members of a special unit employed in critical law enforcement missions Par-ticipants had 594 to 2570 hours of flight experience (mean = 168160 SD = 75727)and had normal hearing and vision All the pilots received training with the simula-tion software prior to experimentation Performance was considered acceptable onlyif they completed an entire simulated flight with no errors

Apparatus Microsoft Flight Simulator 2004 was used as the task in this study Thedeck was that of the Beechcraft Baron 58 which is similar to the Partenavia P68 Observ-er used by police pilots The input device was a Trust GM-2600 Joystick Pilots werevectored by a simulated air traffic control (ATC) workstation with an experimenterusing the Microsoft Flight Simulator Navigator Audio communications were carriedout using TeamSpeak 2 an application that enables people to speak with one anoth-er over IP The CTRL key of the computer keyboard was used to activatedeactivatethe communication channel

Ocular activity recordings The Tobii ET17 eye-tracking system (see Figure 1) wasused for recording ocular activity This system allows the collection of ocular data with-out using invasive andor uncomfortable head-mounted instruments It uses nearinfrared diodes to generate reflection patterns on the cornea of the eyes A cameracollects these reflection patterns together with other visual information Image-processing algorithms identify relevant features including the eyes and corneal reflec-tion patterns Three-dimensional position in space of each eyeball and the gaze pointon the screen are then calculated Sampling frequency was 30Hz

Procedure Participants sat in a dark sound-attenuated room underwent a calibra-tion procedure for eye-tracking and received instructions for the execution of thetask They were asked to fly from Pratica di Mare (41deg 39prime 34N 12deg 26prime 43E) toCiampino (41deg 47prime 58N 12deg 35prime 42E) without the use of autopilot GPS and radiocommunication controls Meteorological conditions were kept constant (good weath-er) during the simulation The selected route is frequently used for operative missions

The simulation was paused at the end of the climb the cruise and the landingphases in order to administer the NASA Task Load Index (NASA-TLX Hart amp Stave-land 1988) The pilotsrsquo response times to ATC calls were recorded we made sure thatattention paid to the task was constant during the simulation and that no changes hadcontaminated performance in one or more phases An instrumental landing system(ILS) procedure was used in the final phase The total flight duration was about 38 min

A Random Glance at the Flight Deck 275

Data Analysis and ResultsThe pilotsrsquo response times to controllersrsquo calls were analyzed by a repeated-

measures ANOVA design using the flight phase as the factor (departure vs climb vscruise vs descent vs landing) Results showed no significant differences betweenflight phases in pilotsrsquo response times F4 36 = 64 p gt 05 (Table 1)

NASA-TLX scores were analyzed by a repeated-measures ANOVA design usingthe flight phase as the factor (departure amp climb vs cruise vs descent amp landing) Inthis case only three phases were used because it was not possible to pause the sim-ulation and administer the NASA-TLX between continuous phases such as departureand climb or descent and landing Results showed a significant effect of flight phaseF218 =1282 p lt 001 Duncan post hoc testing showed that cruise workload was ratedas significantly lower than that of the other two phases p lt 01 (see Figure 2)

The Nearest Neighbor Index was analyzed by a repeated-measures ANOVA de-sign using the flight phase as the factor (departure vs climb vs cruise vs descent vslanding) Results showed a significant effect of flight phase F436 = 2585 p lt 0001Duncan post hoc testing showed that values associated with departure and landingwere significantly different from those associated with cruise (p lt 01) Departure wasalso significantly different from climb (p lt 01) and descent (p lt 05) The cruise phasewas not significantly different from the climb (p = 16) but showed a tendency towardstatistical significance from the descent phase (p = 08) Figure 3 shows the NNI val-ues by flight phase

276 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 1 A pilot practicing with the simulation software displayed within the Tobii ET17eye-tracking system To the pilot the device looks like a standard 17-inch TFT displayThe only difference is the embedded video camera (at the bottom) and the infrared LEDspositioned around the camera and at the top of the display

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

other eye-movement methods is the analysis of visual patterns For example ItohHayashi Tsukui and Saito (1990) and Diez et al (2001) used this method for gath-ering information about the scanning strategies of pilots interacting with a Boeing 747simulator Usually these studies analyze the ocular activity within specific areas ofinterest (AOI) each one including a tool inspected by pilots during a simulated flightAlthough the scanpath is usually used to obtain qualitative information it can alsobe used in association with advanced computing techniques Particularly visual scan-ning randomness (or entropy) has been proposed as a measure of mental workload(Ephrath Tole Stephens amp Young 1980 Harris Glover amp Spady 1986 Tole Steph-ens Vivaudou Ephrath amp Young 1983)

In thermodynamics the concept of entropy is related to the quantity of disorderin a system (in this case the disorder in visual exploration) The rationale underly-ing this approach is that the exploration pattern becomes more stereotyped (ieless random) as workload increases In contrast as mental workload decreases therandomness of the pattern should increase Hilburn Jorna Byrne and Parasuraman(1997) corroborated this hypothesis in a study on air traffic controllersrsquo scanningstrategies However using entropy analysis Kruizinga Mulder and de Waard (2006)found an opposite pattern which suggests that more studies are needed in order tounderstand the relationship between mental workload and visual scanning

Using a different approach Di Nocera Terenzi and Camilli (2006) also founddivergent results They used a distance statistic indicator called the Nearest NeighborIndex (NNI Clark amp Evans 1954) to analyze eye fixations data Unlike the entropymeasure which ignores fixations outside the predefined regions of interest this ap-proach takes into account all fixations considering them all informative The follow-ing section summarizes this strategy for analyzing eye movement data

Analyzing Spatially Distributed DataThe measurement and description of pattern distribution were first addressed

in reference to plant and animal populations In forestry for example the positionsof trees in a forest form a point pattern in the plane Information about the distribu-tion of such points is relevant for investigating phenomena such as plant infectionsor growing patterns In the beginning the basic assumption was that individuals ofmost populations (eg plants animals or fossils) were distributed at random butit soon became clear that the randomness assumption was not appropriate The issuethen became to establish the degree of variation from random expectation as well asthe statistical significance of differences in the distribution pattern of two or more pop-ulations To achieve this goal Clark and Evans (1954) introduced the NNI which isthe ratio between the average of the observed minimum distances between points andthe mean random distance that one would expect if the distribution were randomFifty years later this index is still one of the most widely used distance statistics inagriculture paleontology and crime analysis (All deal with spatially arranged data)

As a first step the nearest neighbor distance or d(NN) should be computed asfollows

A Random Glance at the Flight Deck 273

d(NN) = min 1 le j le N j ne i

where min(dij) is the distance between each point and the point nearest to it and Nis the number of points in the distribution

This index is nothing more than the average of the nearest neighbor distancesThe second step is to compute the mean random distance or d(ran) that is the d(NN)one would expect if the distribution were random

d(ran) = 05

where A is the area of the region (the measurement unit of the index is related to theone used here) and N is the number of points

The final step is the actual computation of the NNI as follows

NNI =

Of course this ratio is equal to 1 when the distribution is random Values less than1 suggest grouping whereas values greater than 1 suggest regularity (ie the pointpattern is dispersed in a nonrandom way) Theoretically NNI lies between 0 (max-imum clustering) and 21491 (strictly regular hexagonal pattern)

Di Nocera et al (2006) applied this procedure to eye fixations (given that they arepoint patterns as well) and found this index to be sensitive to variation in mentalworkload showing a tendency toward randomness in the high-workload conditionThis is the opposite of what the entropy-based method would predict Howeverentropy studies have used ocular data within specific and static AOI whereas theDi Nocera et al (2006) study used ocular data gathered from a dynamic scene (par-ticipants were requested to play the Asteroids PC game in two difficulty conditions)within a convex hull defined by the outermost fixations in the distribution The highmental workload condition was obtained by preventing the use of the weapon todestroy the asteroids whereas the lowmoderate-workload condition consisted ofthe regular game allowing the use of the weapon

Considering the dynamic nature of the Asteroids game (the ship moves aroundin the screen area) it is possible that the different distributions of fixations that havebeen found were strategy-driven rather than workload-driven Indeed even if the twoversions of the game were geometrically equivalent (having the same number of as-teroids either between conditions throughout the game) avoiding the asteroids mightfavor a strategy aimed at spreading the fixations over a wide area whereas the shoot-ing condition might have been supported by a strategy based on focusing over theship and target positions

In order to address the role of these differences in the present study we appliedthe same rationale to investigate ocular behavior during interaction with a somewhatldquostaticrdquo visual scene To this aim a flight simulation task was used comprising both

d(NN)mdashmdashmdashd(ran)

AmdashmdashN

(dij)mdashmdashN

N

sumi =1

274 Journal of Cognitive Engineering and Decision Making Fall 2007

high workload (departure and landing) and low to moderate workload (climb cruiseand descent) phases Of course this was ldquostaticrdquo in the sense that in a flight deck thelocations of objects to monitor (namely the instruments) did not change over timeeven if the visual scene outside the cockpit did change

Method

Participants Ten instrument flight rules (IFR) -licensed pilots belonging to the Italianpolice (all males mean age = 4110 years SD = 482) volunteered for this study Pilotswere members of a special unit employed in critical law enforcement missions Par-ticipants had 594 to 2570 hours of flight experience (mean = 168160 SD = 75727)and had normal hearing and vision All the pilots received training with the simula-tion software prior to experimentation Performance was considered acceptable onlyif they completed an entire simulated flight with no errors

Apparatus Microsoft Flight Simulator 2004 was used as the task in this study Thedeck was that of the Beechcraft Baron 58 which is similar to the Partenavia P68 Observ-er used by police pilots The input device was a Trust GM-2600 Joystick Pilots werevectored by a simulated air traffic control (ATC) workstation with an experimenterusing the Microsoft Flight Simulator Navigator Audio communications were carriedout using TeamSpeak 2 an application that enables people to speak with one anoth-er over IP The CTRL key of the computer keyboard was used to activatedeactivatethe communication channel

Ocular activity recordings The Tobii ET17 eye-tracking system (see Figure 1) wasused for recording ocular activity This system allows the collection of ocular data with-out using invasive andor uncomfortable head-mounted instruments It uses nearinfrared diodes to generate reflection patterns on the cornea of the eyes A cameracollects these reflection patterns together with other visual information Image-processing algorithms identify relevant features including the eyes and corneal reflec-tion patterns Three-dimensional position in space of each eyeball and the gaze pointon the screen are then calculated Sampling frequency was 30Hz

Procedure Participants sat in a dark sound-attenuated room underwent a calibra-tion procedure for eye-tracking and received instructions for the execution of thetask They were asked to fly from Pratica di Mare (41deg 39prime 34N 12deg 26prime 43E) toCiampino (41deg 47prime 58N 12deg 35prime 42E) without the use of autopilot GPS and radiocommunication controls Meteorological conditions were kept constant (good weath-er) during the simulation The selected route is frequently used for operative missions

The simulation was paused at the end of the climb the cruise and the landingphases in order to administer the NASA Task Load Index (NASA-TLX Hart amp Stave-land 1988) The pilotsrsquo response times to ATC calls were recorded we made sure thatattention paid to the task was constant during the simulation and that no changes hadcontaminated performance in one or more phases An instrumental landing system(ILS) procedure was used in the final phase The total flight duration was about 38 min

A Random Glance at the Flight Deck 275

Data Analysis and ResultsThe pilotsrsquo response times to controllersrsquo calls were analyzed by a repeated-

measures ANOVA design using the flight phase as the factor (departure vs climb vscruise vs descent vs landing) Results showed no significant differences betweenflight phases in pilotsrsquo response times F4 36 = 64 p gt 05 (Table 1)

NASA-TLX scores were analyzed by a repeated-measures ANOVA design usingthe flight phase as the factor (departure amp climb vs cruise vs descent amp landing) Inthis case only three phases were used because it was not possible to pause the sim-ulation and administer the NASA-TLX between continuous phases such as departureand climb or descent and landing Results showed a significant effect of flight phaseF218 =1282 p lt 001 Duncan post hoc testing showed that cruise workload was ratedas significantly lower than that of the other two phases p lt 01 (see Figure 2)

The Nearest Neighbor Index was analyzed by a repeated-measures ANOVA de-sign using the flight phase as the factor (departure vs climb vs cruise vs descent vslanding) Results showed a significant effect of flight phase F436 = 2585 p lt 0001Duncan post hoc testing showed that values associated with departure and landingwere significantly different from those associated with cruise (p lt 01) Departure wasalso significantly different from climb (p lt 01) and descent (p lt 05) The cruise phasewas not significantly different from the climb (p = 16) but showed a tendency towardstatistical significance from the descent phase (p = 08) Figure 3 shows the NNI val-ues by flight phase

276 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 1 A pilot practicing with the simulation software displayed within the Tobii ET17eye-tracking system To the pilot the device looks like a standard 17-inch TFT displayThe only difference is the embedded video camera (at the bottom) and the infrared LEDspositioned around the camera and at the top of the display

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

d(NN) = min 1 le j le N j ne i

where min(dij) is the distance between each point and the point nearest to it and Nis the number of points in the distribution

This index is nothing more than the average of the nearest neighbor distancesThe second step is to compute the mean random distance or d(ran) that is the d(NN)one would expect if the distribution were random

d(ran) = 05

where A is the area of the region (the measurement unit of the index is related to theone used here) and N is the number of points

The final step is the actual computation of the NNI as follows

NNI =

Of course this ratio is equal to 1 when the distribution is random Values less than1 suggest grouping whereas values greater than 1 suggest regularity (ie the pointpattern is dispersed in a nonrandom way) Theoretically NNI lies between 0 (max-imum clustering) and 21491 (strictly regular hexagonal pattern)

Di Nocera et al (2006) applied this procedure to eye fixations (given that they arepoint patterns as well) and found this index to be sensitive to variation in mentalworkload showing a tendency toward randomness in the high-workload conditionThis is the opposite of what the entropy-based method would predict Howeverentropy studies have used ocular data within specific and static AOI whereas theDi Nocera et al (2006) study used ocular data gathered from a dynamic scene (par-ticipants were requested to play the Asteroids PC game in two difficulty conditions)within a convex hull defined by the outermost fixations in the distribution The highmental workload condition was obtained by preventing the use of the weapon todestroy the asteroids whereas the lowmoderate-workload condition consisted ofthe regular game allowing the use of the weapon

Considering the dynamic nature of the Asteroids game (the ship moves aroundin the screen area) it is possible that the different distributions of fixations that havebeen found were strategy-driven rather than workload-driven Indeed even if the twoversions of the game were geometrically equivalent (having the same number of as-teroids either between conditions throughout the game) avoiding the asteroids mightfavor a strategy aimed at spreading the fixations over a wide area whereas the shoot-ing condition might have been supported by a strategy based on focusing over theship and target positions

In order to address the role of these differences in the present study we appliedthe same rationale to investigate ocular behavior during interaction with a somewhatldquostaticrdquo visual scene To this aim a flight simulation task was used comprising both

d(NN)mdashmdashmdashd(ran)

AmdashmdashN

(dij)mdashmdashN

N

sumi =1

274 Journal of Cognitive Engineering and Decision Making Fall 2007

high workload (departure and landing) and low to moderate workload (climb cruiseand descent) phases Of course this was ldquostaticrdquo in the sense that in a flight deck thelocations of objects to monitor (namely the instruments) did not change over timeeven if the visual scene outside the cockpit did change

Method

Participants Ten instrument flight rules (IFR) -licensed pilots belonging to the Italianpolice (all males mean age = 4110 years SD = 482) volunteered for this study Pilotswere members of a special unit employed in critical law enforcement missions Par-ticipants had 594 to 2570 hours of flight experience (mean = 168160 SD = 75727)and had normal hearing and vision All the pilots received training with the simula-tion software prior to experimentation Performance was considered acceptable onlyif they completed an entire simulated flight with no errors

Apparatus Microsoft Flight Simulator 2004 was used as the task in this study Thedeck was that of the Beechcraft Baron 58 which is similar to the Partenavia P68 Observ-er used by police pilots The input device was a Trust GM-2600 Joystick Pilots werevectored by a simulated air traffic control (ATC) workstation with an experimenterusing the Microsoft Flight Simulator Navigator Audio communications were carriedout using TeamSpeak 2 an application that enables people to speak with one anoth-er over IP The CTRL key of the computer keyboard was used to activatedeactivatethe communication channel

Ocular activity recordings The Tobii ET17 eye-tracking system (see Figure 1) wasused for recording ocular activity This system allows the collection of ocular data with-out using invasive andor uncomfortable head-mounted instruments It uses nearinfrared diodes to generate reflection patterns on the cornea of the eyes A cameracollects these reflection patterns together with other visual information Image-processing algorithms identify relevant features including the eyes and corneal reflec-tion patterns Three-dimensional position in space of each eyeball and the gaze pointon the screen are then calculated Sampling frequency was 30Hz

Procedure Participants sat in a dark sound-attenuated room underwent a calibra-tion procedure for eye-tracking and received instructions for the execution of thetask They were asked to fly from Pratica di Mare (41deg 39prime 34N 12deg 26prime 43E) toCiampino (41deg 47prime 58N 12deg 35prime 42E) without the use of autopilot GPS and radiocommunication controls Meteorological conditions were kept constant (good weath-er) during the simulation The selected route is frequently used for operative missions

The simulation was paused at the end of the climb the cruise and the landingphases in order to administer the NASA Task Load Index (NASA-TLX Hart amp Stave-land 1988) The pilotsrsquo response times to ATC calls were recorded we made sure thatattention paid to the task was constant during the simulation and that no changes hadcontaminated performance in one or more phases An instrumental landing system(ILS) procedure was used in the final phase The total flight duration was about 38 min

A Random Glance at the Flight Deck 275

Data Analysis and ResultsThe pilotsrsquo response times to controllersrsquo calls were analyzed by a repeated-

measures ANOVA design using the flight phase as the factor (departure vs climb vscruise vs descent vs landing) Results showed no significant differences betweenflight phases in pilotsrsquo response times F4 36 = 64 p gt 05 (Table 1)

NASA-TLX scores were analyzed by a repeated-measures ANOVA design usingthe flight phase as the factor (departure amp climb vs cruise vs descent amp landing) Inthis case only three phases were used because it was not possible to pause the sim-ulation and administer the NASA-TLX between continuous phases such as departureand climb or descent and landing Results showed a significant effect of flight phaseF218 =1282 p lt 001 Duncan post hoc testing showed that cruise workload was ratedas significantly lower than that of the other two phases p lt 01 (see Figure 2)

The Nearest Neighbor Index was analyzed by a repeated-measures ANOVA de-sign using the flight phase as the factor (departure vs climb vs cruise vs descent vslanding) Results showed a significant effect of flight phase F436 = 2585 p lt 0001Duncan post hoc testing showed that values associated with departure and landingwere significantly different from those associated with cruise (p lt 01) Departure wasalso significantly different from climb (p lt 01) and descent (p lt 05) The cruise phasewas not significantly different from the climb (p = 16) but showed a tendency towardstatistical significance from the descent phase (p = 08) Figure 3 shows the NNI val-ues by flight phase

276 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 1 A pilot practicing with the simulation software displayed within the Tobii ET17eye-tracking system To the pilot the device looks like a standard 17-inch TFT displayThe only difference is the embedded video camera (at the bottom) and the infrared LEDspositioned around the camera and at the top of the display

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

high workload (departure and landing) and low to moderate workload (climb cruiseand descent) phases Of course this was ldquostaticrdquo in the sense that in a flight deck thelocations of objects to monitor (namely the instruments) did not change over timeeven if the visual scene outside the cockpit did change

Method

Participants Ten instrument flight rules (IFR) -licensed pilots belonging to the Italianpolice (all males mean age = 4110 years SD = 482) volunteered for this study Pilotswere members of a special unit employed in critical law enforcement missions Par-ticipants had 594 to 2570 hours of flight experience (mean = 168160 SD = 75727)and had normal hearing and vision All the pilots received training with the simula-tion software prior to experimentation Performance was considered acceptable onlyif they completed an entire simulated flight with no errors

Apparatus Microsoft Flight Simulator 2004 was used as the task in this study Thedeck was that of the Beechcraft Baron 58 which is similar to the Partenavia P68 Observ-er used by police pilots The input device was a Trust GM-2600 Joystick Pilots werevectored by a simulated air traffic control (ATC) workstation with an experimenterusing the Microsoft Flight Simulator Navigator Audio communications were carriedout using TeamSpeak 2 an application that enables people to speak with one anoth-er over IP The CTRL key of the computer keyboard was used to activatedeactivatethe communication channel

Ocular activity recordings The Tobii ET17 eye-tracking system (see Figure 1) wasused for recording ocular activity This system allows the collection of ocular data with-out using invasive andor uncomfortable head-mounted instruments It uses nearinfrared diodes to generate reflection patterns on the cornea of the eyes A cameracollects these reflection patterns together with other visual information Image-processing algorithms identify relevant features including the eyes and corneal reflec-tion patterns Three-dimensional position in space of each eyeball and the gaze pointon the screen are then calculated Sampling frequency was 30Hz

Procedure Participants sat in a dark sound-attenuated room underwent a calibra-tion procedure for eye-tracking and received instructions for the execution of thetask They were asked to fly from Pratica di Mare (41deg 39prime 34N 12deg 26prime 43E) toCiampino (41deg 47prime 58N 12deg 35prime 42E) without the use of autopilot GPS and radiocommunication controls Meteorological conditions were kept constant (good weath-er) during the simulation The selected route is frequently used for operative missions

The simulation was paused at the end of the climb the cruise and the landingphases in order to administer the NASA Task Load Index (NASA-TLX Hart amp Stave-land 1988) The pilotsrsquo response times to ATC calls were recorded we made sure thatattention paid to the task was constant during the simulation and that no changes hadcontaminated performance in one or more phases An instrumental landing system(ILS) procedure was used in the final phase The total flight duration was about 38 min

A Random Glance at the Flight Deck 275

Data Analysis and ResultsThe pilotsrsquo response times to controllersrsquo calls were analyzed by a repeated-

measures ANOVA design using the flight phase as the factor (departure vs climb vscruise vs descent vs landing) Results showed no significant differences betweenflight phases in pilotsrsquo response times F4 36 = 64 p gt 05 (Table 1)

NASA-TLX scores were analyzed by a repeated-measures ANOVA design usingthe flight phase as the factor (departure amp climb vs cruise vs descent amp landing) Inthis case only three phases were used because it was not possible to pause the sim-ulation and administer the NASA-TLX between continuous phases such as departureand climb or descent and landing Results showed a significant effect of flight phaseF218 =1282 p lt 001 Duncan post hoc testing showed that cruise workload was ratedas significantly lower than that of the other two phases p lt 01 (see Figure 2)

The Nearest Neighbor Index was analyzed by a repeated-measures ANOVA de-sign using the flight phase as the factor (departure vs climb vs cruise vs descent vslanding) Results showed a significant effect of flight phase F436 = 2585 p lt 0001Duncan post hoc testing showed that values associated with departure and landingwere significantly different from those associated with cruise (p lt 01) Departure wasalso significantly different from climb (p lt 01) and descent (p lt 05) The cruise phasewas not significantly different from the climb (p = 16) but showed a tendency towardstatistical significance from the descent phase (p = 08) Figure 3 shows the NNI val-ues by flight phase

276 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 1 A pilot practicing with the simulation software displayed within the Tobii ET17eye-tracking system To the pilot the device looks like a standard 17-inch TFT displayThe only difference is the embedded video camera (at the bottom) and the infrared LEDspositioned around the camera and at the top of the display

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

Data Analysis and ResultsThe pilotsrsquo response times to controllersrsquo calls were analyzed by a repeated-

measures ANOVA design using the flight phase as the factor (departure vs climb vscruise vs descent vs landing) Results showed no significant differences betweenflight phases in pilotsrsquo response times F4 36 = 64 p gt 05 (Table 1)

NASA-TLX scores were analyzed by a repeated-measures ANOVA design usingthe flight phase as the factor (departure amp climb vs cruise vs descent amp landing) Inthis case only three phases were used because it was not possible to pause the sim-ulation and administer the NASA-TLX between continuous phases such as departureand climb or descent and landing Results showed a significant effect of flight phaseF218 =1282 p lt 001 Duncan post hoc testing showed that cruise workload was ratedas significantly lower than that of the other two phases p lt 01 (see Figure 2)

The Nearest Neighbor Index was analyzed by a repeated-measures ANOVA de-sign using the flight phase as the factor (departure vs climb vs cruise vs descent vslanding) Results showed a significant effect of flight phase F436 = 2585 p lt 0001Duncan post hoc testing showed that values associated with departure and landingwere significantly different from those associated with cruise (p lt 01) Departure wasalso significantly different from climb (p lt 01) and descent (p lt 05) The cruise phasewas not significantly different from the climb (p = 16) but showed a tendency towardstatistical significance from the descent phase (p = 08) Figure 3 shows the NNI val-ues by flight phase

276 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 1 A pilot practicing with the simulation software displayed within the Tobii ET17eye-tracking system To the pilot the device looks like a standard 17-inch TFT displayThe only difference is the embedded video camera (at the bottom) and the infrared LEDspositioned around the camera and at the top of the display

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

A Random Glance at the Flight Deck 277

Figure 2 NASA-TLX mean scores by flight phase

Figure 3 Nearest Neighbor Index mean values by flight phase (NNI = 1 means random-ness in the point pattern NNI = 0 means grouping)

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

Neighbors in TimeA spatial pattern is the result of a process evolving over time so it can be used to

test some hypotheses about the time course of the process itself This paper does notdeal extensively with the quantitative aspects of the NNI pattern changes over time(ie spectral analysis) this topic needs to be addressed in a separate work The pri-mary aim of this section is to show only the differences over time of the index and toprovide some evidence of changes in the NNI pattern within the flight phases This iscritical for designing adaptive systems based on the real-time computation of the NNI

For illustrative purposes Figures 4andash4d (pages 279ndash280) show the changes ofthe pattern across the five flight segments (unsmoothed) for pilot 1 (Figure 4a) andpilot 2 (Figure 4b) and the respective periodograms (Figures 4c and 4d) Periodo-grams have been obtained after mean subtraction and detrending To obtain a clear-er picture of underlying periodicities spectral density estimates have been computedby smoothing the periodogram values with a weighted moving average (Parzen win-dow) The overall pattern is evident High-workload phases generate higher NNIvalues even within the same pilot The effect is much clearer for pilot 1 than for pilot2 clearly there are individual differences to take into account

Individual differences are also reflected in the periodograms The spectral analy-sis showed at least three peaks that may suggest the existence of an underlying ultra-dian rhythm (a pattern that repeats in less than a day) Indeed the time series clearlyshow a recurring pattern that could be considered as evidence of a cyclical alloca-tion of resources to the task at hand This is compatible with other studies (ConteFerlazzo amp Renzi 1995 Smith Valentino amp Arruda 2003) that have analyzed reac-tion times in prolonged vigilance tasks these studies show that cycles of relatively goodand poor performance tend to last 4 min or longer However experiments designedto study ultradian rhythms usually last for hours and make use of much longer series

The main limit of the present (brief) account is the fact that only 38 values (oneper each minute of flight) could have been used In less than 1 min there would notbe enough fixations to compute the index Generally it is possible to consider 50 pointsas the threshold For now this is anecdotal evidence but in the future with a largedatabase of studies it will be possible to obtain more specific information on thesample size needed to confidently compute the index from ocular data

It is also worth noting that the NNI ratio per se brings no information about itsstatistical significance Nevertheless it is possible to test the significance of the indexusing the Complete Spatial Randomness testing procedure (see Clark amp Evans 1954)Of course the hypothesis testing procedure is mandatory when a single NNI ratio isobtained on the basis of ldquoone-shotrdquo information (eg distribution of trees in an area)Instead in the present study the interest focused on the utility of the ldquorawrdquo index ina task providing thousands of points in different conditions It would be interestinghowever to use the outcome of the significance-testing procedure as a filter to retainor reject NNI values that are computed over short epochs (which include a limitednumber of pointsfixations) as in the previously described case

278 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

A Random Glance at the Flight Deck 279

Figure 4a and 4b Time series of the Nearest Neighbor Index (a = pilot 1 and b = pilot 2)

(a)

(b)

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

280 Journal of Cognitive Engineering and Decision Making Fall 2007

Figure 4c and 4d Periodograms (c = pilot 1 and d = pilot 2)

(c)

(d)

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

Discussion and Conclusions

Compared with other psychophysiological indices (eg event-related potentialsheart rate variability) that have been proposed as candidate measures for triggeringadaptive systems eye movements show many benefits They are insensitive to limbmovements (they can also be adjusted for head movements) not much training isnecessary for setting up the equipment (at least with the infrared-based system usedin this study) and the calibration procedure can be accomplished in a short timeFuture studies will need to compare various eye activity measures in order to deter-mine which measures (either alone or in combination) ultimately correlate best withworkload Among those measures we recommend including the spatial distribu-tion of fixations

Results of the present study confirmed that the NNI computed on eye fixations issensitive to variations in mental workload thus replicating previous findings and pro-viding additional support for the robustness of this index Moreover specific workload-related fixation patterns were found using eye movement data collected during theinteraction with a ldquostaticrdquo interface as opposed to the ldquodynamicrdquo one used previous-ly As expected higher NNI values were associated with high-workload phases (depar-ture and landing) The lack of significant results in some of the post hoc comparisonsshould not be considered an indication of a lack of sensitivity of the measure becausethat was presumably attributable to the small sample size available for this experi-ment Indeed the landing phase showed very high variability in the NNI values

Response times to ATC calls showed that pilots performed homogeneously dur-ing the entire simulation Showing that pilots have attended to ATC calls uniformly(and reasonably) during the simulation removed any doubt about their commitmentto the experimental task Nevertheless both NNI values and NASA-TLX scores pro-vided evidence of a variation in the amount of mental load This is not surprising asindividuals can allocate more cognitive resources to keep performance constant

Overall the evidence supports implementing NNI as a real-time measure of men-tal workload and as a trigger for automated systems One additional benefit of theproposed index is that it does not necessarily need extreme precision and high tem-poral resolution In fact the comparison is made between the actual distribution ofpoints and the expected random distribution of the same number of points The indexitself is a rough estimate of grouping and having more points (ie more than100ndash200)does not enrich its meaningfulness

As found by Di Nocera et al (2006) the direction of the NNI pattern diverges againfrom that expected on the basis of the entropy studies run by Harris et al (1986)Hilburn et al (1997) and Tole et al (1983) However it is questionable whether di-rect comparison between the two indices can be made Indeed one is based on tran-sitions between AOIs whereas the other is a ratio between point distributions overthe whole scene It is likely that scanpath and fixation distribution depend on the sametype of processes although they lead to different results The analyses that have beencarried out here however do not need to define specific areas of interest and thismight be a relevant feature in the applied domain

A Random Glance at the Flight Deck 281

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

The functional significance of the index can be summarized as follows Underhigh mental workload conditions more dispersed patterns may be attributable to astrategy aimed at optimizing promptness to incoming information Indeed as hy-pothesized by Smith Valentino and Arruda (2003) endogenous mechanisms thatcause organisms to automatically alternate their attention between focusing and cast-ing a wide net may have evolved This is also compatible with a finding reported byPelz and Canosa (2001) that individuals might use ldquolook-aheadrdquo fixations in antic-ipation of future tasks

The cyclical pattern shown in Figures 4andash4d seems to support this view At thisstage of research development it is impossible to address the basic mechanisms in-volved in the generation of this effect However the results provided in this papermay indicate a fluctuation of attentional resources From a logical standpoint wecould think of three possible strategies for resources allocation on demand contin-uous and cyclical The first would be a strategy to minimize unnecessary expendi-tures and allocate them only when required This strategy would strongly reduce thepromptness of the individual to react The second strategy would involve a continu-ous expenditure of attentional resources in order to position the individual to reactadequately however this is incompatible with the fact that cognitive resources arelimited The third considers the possibility of a cyclical allocation of attentional re-sources (a parsimonious strategy) so that a certain degree of promptness is always(cyclically) available to the individual

Such a cyclical rise and fall would then allow the individual to take advantage ofthe level of mental resources made available and to use that level as a starting pointfor voluntary resource management This is impossible to verify at this time but itwould be easy to design an experiment in which this effect is monitored by the intro-duction of occasional system faults As a consequence of a fault a change in the NNIpattern should be observed The direction of the NNI curve should invert if the de-mand for additional resources happens during the fall periods and its amplitudeshould increase if the additional resources are required during the rise periods Basi-cally when task demands are high it becomes mandatory to monitor everything inthe shortest time frame without wasting time (and fixations) on the same instrumentAfter all fixations are ldquopauses over informative regions of interestrdquo (Salvucci amp Gold-berg 2000 p 71)

Similar considerations have been made about the course of ocular inspectionof pictures Fixations are usually shorter when one starts to view a picture (high-workload condition) This phenomenon is not new It has already been reported byKahneman (1973) who found it puzzling because that is exactly the phase when oneneeds to gather more information and fixations should last longer However the typeof information one needs in the initial phases (or in the most difficult phases) maybe the difference Indeed structural more than semantic information may be extract-ed which can be accomplished with a few short fixations

This account is also compatible with recent findings Irwin and Zelinsky (2002)reported a continuous increase in fixation duration during the inspection time (overa15-fixation-long period) Unema Pannasch Joos and Velichkovsky (2005) found

282 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

A Random Glance at the Flight Deck 283

a shift (which is a function of the inspection time) from shorter fixations and saccadeswith longer amplitudes to longer fixations and shorter saccades The authors inter-preted this effect in terms of two different spatial representations underlying the earlyand late phases in picture viewing The focus of that work was on the ldquowhatrdquo andldquowhererdquo systems and their relation to the ldquoventralrdquo and ldquodorsalrdquo visual pathways(Ungerleider amp Mishkin 1982)

An extensive discussion of this topic is outside the scope of the present paperHowever Unema et al (2005) did report that the transition from short-fixationslarge-saccades to long-fixationsshort-saccades may suggest that ldquotwo qualitatively differ-ent competitive processes negotiate whether to keep fixating or to go on to the nextsalient objectrdquo (p 491)

Future studies run on trainee pilots could resolve any doubt about the adaptivenature of this strategy In this regard it is worth noting that some studies (Bunecke1987 Ephrath et al 1980) have shown that workload affects the duration of fixa-tions Others (Bellenkes Wickens amp Kramer 1997 Miller 1973) recorded shorterand more frequent fixations in expert operators (All these studies were run on air-craft pilots)

Another issue worth investigating in the aviation domain is the sensitivity of theindex to out-of-the-cockpit fixations Under visual flight rules for example the pilotoperates the aircraft by visual reference to the environment outside the cockpit In-struments will also be monitored occasionally allowing a comparison between thetwo distributions that could be obtained from the out-of-the-cockpit fixations andthe fixations to instruments Consequently the upperlower visual fields NNI ratiomay be correlated with mental workload

In conclusion the application of the Nearest Neighbor Index to eye fixation dataprovides a domain-independent measure that could eventually be used in operationalenvironments for gathering real-time information on operator load This is of criticalinterest in the aviation domain but can be easily extended to other critical tasks aswell including air traffic control and baggage screening

Acknowledgments

The authors thank Capt Pietro A Ferruggia for his assistance and support Allopinions expressed in this paper are those of the authors and do not necessarily reflectthose of the Italian Police Authorities The authors are also very grateful to Robert SBolia and Ericka Rovira for their comments on a preliminary version of this paper

References

Beatty J (1982) Task-evoked pupillary responses processing load and the structure of process-ing resources Psychological Bulletin 91 377ndash381

Bellenkes A H Wickens C D amp Kramer A F (1997) Visual scanning and pilot expertise Therole of attentional flexibility and mental model development Aviation Space and Environment-al Medicine 68 569ndash579

Brookings J B Wilson G F amp Swain C R (1996) Psychophysiological responses to changesin workload during simulated air traffic control Biological Psychology 42 361ndash377

Bunecke J L (1987) Quantifying some information processing requirements of the pilotrsquos instru-ment crosscheck Proceedings of the Human Factors Society 31st Annual Meeting (pp 1301ndash1305)Santa Monica CA Human Factors and Ergonomics Society

Clark P J amp Evans F C (1954) Distance to nearest neighbor as a measure of spatial relationshipsin populations Ecology 35 445ndash453

Conte S Ferlazzo F amp Renzi P (1995) Ultradian rhythms of reaction times in performance invigilance tasks Biological Psychology 39(2) 159ndash172

Diez M Boehm-Davis D A Hansberger J T Pinney M E Hansberger J T amp Schoppek W(2001) Tracking pilot interactions with flight management systems through eye movementsProceedings of the 11th International Symposium on Aviation Psychology Columbus The OhioState University

Di Nocera F (2003) On reliability and stability of psychophysiological indicators for assessingoperator functional states In G R J Hockey A W K Gaillard amp O Burov (Eds) Operatorfunctional state The assessment and prediction of human performance degradation in complex tasks(pp 162ndash173) Amsterdam IOS Press

Di Nocera F Terenzi M amp Camilli M (2006) Another look at scanpath Distance to nearestneighbour as a measure of mental workload In D de Waard K A Brookhuis amp A Toffetti(Eds) Developments in human factors in transportation design and evaluation (pp 295ndash303)Maastricht Netherlands Shaker Publishing

Ephrath A R Tole J R Stephens A T amp Young L R (1980) Instrument scan ndash Is it an indica-tor of the pilotrsquos workload Proceedings of the Human Factors Society 24th Annual Meeting (pp257ndash258) Santa Monica CA Human Factors and Ergonomics Society

Granholm E Asarnow R F Sarkin A J amp Dykes K L (1996) Pupillary responses indexcognitive resource limitations Psychophysiology 33 457ndash461

Hankins T C amp Wilson G F (1998) A comparison of heart rate eye activity EEG and subjec-tive measures of pilot mental workload during flight Aviation Space and Environmental Medi-cine 69 360ndash367

Harris R L Glover B L amp Spady A A (1986) Analytic techniques of pilot scanning behavior andtheir application (Technical Paper No 2525) Hampton VA NASA Langley Research Center

Hart S G Hauser J R amp Lester P T (1984) Inflight evaluation of four measures of pilot work-load Proceedings of the Human Factors Society 28th Annual Meeting (pp 945ndash949) Santa MonicaCA Human Factors and Ergonomics Society

Hart S G amp Staveland L E (1988) Development of NASA-TLX (Task Load Index) Results ofempirical and theoretical research In P A Hancock amp N Meshkati (Eds) Human mentalworkload (pp 139ndash183) Amsterdam Elsevier ScienceNorth Holland

Hilburn B Jorna P G Byrne E A amp Parasuraman R (1997) The effect of adaptive air trafficcontrol (ATC) decision aiding on controller mental workload In M Mouloua amp J Koonce(Eds) Human-automation interaction Research and practice (pp 84ndash91) Mahwah NJ Erlbaum

Irwin D E amp Zelinsky G J (2002) Eye movements and scene perception Memory for thingsobserved Perception amp Psychophysics 64 882ndash895

Itoh Y Hayashi Y Tsukui I amp Saito S (1990) The ergonomic evaluation of eye movement andmental workload in aircraft pilots Ergonomics 33 719ndash733

Kahneman D (1973) Attention and effort Englewood Cliffs NJ Prentice-HallKantowitz B H amp Casper P A (1988) Human workload in aviation In E Wiener amp D Nagel

(Eds) Human factors in aviation (pp157ndash187) San Diego CA Academic PressKruizinga A Mulder B amp de Waard D (2006) Eye scan patterns in a simulated ambulance dis-

patcherrsquos task In D de Waard K A Brookhuis amp A Toffetti (Eds) Developments in humanfactors in transportation design and evaluation (pp 305ndash317) Maastricht Netherlands ShakerPublishing

284 Journal of Cognitive Engineering and Decision Making Fall 2007

A Random Glance at the Flight Deck 285

Lee Y H amp Liu B S (2003) Inflight workload assessment Comparison of subjective and phys-iological measurements Aviation Space and Environmental Medicine 74 1078ndash1084

Miller J M (1973) Visual behavior changes of student pilots flying instrument approachesProceedings of the Human Factors Society 17th Annual Meeting (pp 208ndash214) Santa MonicaCA Human Factors and Ergonomics Society

Pelz J B amp Canosa R (2001) Oculomotor behavior and perceptual strategies in complex tasksVision Research 41 3587ndash3596

Salvucci D D amp Goldberg J H (2000) Identifying fixations and saccades in eye-tracking pro-tocols In Proceedings of Eye Tracking Research and Applications Symposium (pp 71ndash78) NewYork ACM Press

Scerbo M W (2001) Stress workload and boredom in vigilance A problem and an answer InP A Hancock amp P A Desmond (Eds) Stress workload and fatigue Human factors in trans-portation (pp 267ndash278) Mahwah NJ Erlbaum

Scerbo M W Freeman F G Mikulka P J Parasuraman R Di Nocera F amp Prinzel L J (2001)The efficacy of psychophysiological measures for implementing adaptive technology (Technical PaperNo 211018) Hampton VA NASA Langley Research Center

Smith K J Valentino D A amp Arruda J E (2003) Rhythmic oscillations in the performance ofa sustained attention task Journal of Clinical and Experimental Neuropsychology 25 561ndash570

Tole J R Stephens A T Vivaudou M Ephrath A R amp Young L R (1983) Visual scanning be-havior and pilot workload (NASA Contractor Report No 3717) Hampton VA NASA LangleyResearch Center

Unema P Pannasch S Joos M amp Velichkovsky B M (2005) Time-course of information pro-cessing during scene perception The relationship between saccade amplitude and fixationduration Visual Cognition 12 473ndash494

Ungerleider L G amp Mishkin M (1982) Two cortical visual systems In D J Ingle M A Goodaleamp R J W Mansfield (Eds) Analysis of visual behavior (pp 549ndash586) Cambridge MA MITPress

Wickens C D (2002) Situation awareness and workload in aviation Current Directions in Psycho-logical Science 11 128ndash133

Wilson G F (2002) An analysis of mental workload in pilots during flight using multiple psycho-physiological measures International Journal of Aviation Psychology 12 3ndash18

Wilson G F Fullenkamp P amp Davis I (1994) Evoked potential cardiac blink and respirationmeasures of pilot workload in air-to-ground missions Aviation Space and Environmental Medi-cine 65 100ndash105

Francesco Di Nocera is a research professor at the Department of Psychology University ofRome ldquoLa SapienzardquoVia dei Marsi 78 ndash 00185 Rome Italy (dinocerauniroma1it) He receivedhis Laurea in work and organizational psychology (1995) and his PhD in psychology (2001)from the University of Rome ldquoLa SapienzardquoHis current research interests are in cognitive ergo-nomics and cognitive neuroscience

Marco Camilli is a graduate student at the Department of Psychology of the University of RomeldquoLa SapienzardquoHis research activity is primarily devoted to the analysis of human mental work-load using eye-tracking technology

Michela Terenzi is completing her PhD at the Department of Psychology of the University ofRome ldquoLa Sapienzardquo Her research activity is primarily devoted to the investigation of humancognition during interaction with automated systems