lable at ScienceDirect

International Journal of Industrial Ergonomics 41 (2011) 677e687

Contents lists avai

International Journal of Industrial Ergonomics

journal homepage: www.elsevier .com/locate/ergon

Is SA shared or distributed in team work? An exploratory studyin an intelligence analysis task

Linda J. Sorensen*, Neville A. StantonFaculty of Engineering and the Environment, University of Southampton, Highfield, Southampton SO17 1 BJ, UK

a r t i c l e i n f o

Article history:Received 19 January 2011Received in revised form3 August 2011Accepted 4 August 2011Available online 3 September 2011

Keywords:Team situation awarenessSituation awareness measurementDistributed cognitionPropositional NetworksSocial Network Analysis

* Corresponding author. Tel.: þ44 2380599575; faxE-mail address: l.j.sorensen@soton.ac.uk (L.J. Soren

0169-8141/$ e see front matter � 2011 Elsevier B.V.doi:10.1016/j.ergon.2011.08.001

a b s t r a c t

This study compared two theoretical approaches to Situation Awareness (SA): the psychological school ofthought and the systems ergonomics school of thought, by assessing measurement of team SA withinthese frameworks. Two teams were assigned and organised into either a traditional Hierarchy or a Peer-to-Peer organisational structure in a single case study design. Measures derived from the psychologicaland systems ergonomics perspectives were applied to assess their sensitivity for assessing team SA. Nostatistically significant differences were found between the two teams when measures originating in thepsychological tradition were considered: differences were found, however, for measures originating inthe systems ergonomics tradition. Literature concerned with team SA reveals a lack of consensus withregards to explaining the nature of the phenomenon as well as its measurement. This paper argues fora debate in the field to clarify what constitutes appropriate measurement techniques for team SA andsuggests that these are taken from the systems ergonomics tradition, as suggested by the present studiesfindings.Relevance to industry: Teams are a major feature of most industrial applications of work, and maintaininggood situation awareness is important to successful performance. A method for examining the situationawareness of teams is proposed and compared with the individual models. Analysing the team asa functional unit of situation awareness is presented for future work.

� 2011 Elsevier B.V. All rights reserved.

1. Introduction

There is still considerable debate concerning the nature ofSituation Awareness (SA) in teams and as yet there is neitherconsensus nor any single measure developed to assess thephenomenon (Patrick et al., 2006). Reviewing the extensive litera-ture on SA identifies a number of conceptual issues which differ-entiate perspectives on SA. A recent paper by Stanton et al. (2010)presents three schools of thought on SA: the psychological, theengineering and the systems ergonomics schools of thought. Thepresent study examined two of these: the psychological and thesystems ergonomics approaches. Twomodelswere considered fromthese: the model of Shared SA which represents the psychologicalapproach, while the more recent model of Distributed SA takesa systems ergonomics perspective. In this paper the two schools ofthought and their associated models are discussed in terms of howeach explain SA, what they consider to be the unit of analysis for SAand how each approach measures SA, followed by an empiricalinvestigation with discussions and conclusions for team SA.

: þ44 7919156759.sen).

All rights reserved.

1.1. Explanations of SA

SA can be explained in terms of several aspects, two of which areconsidered here; as individual or as team SA. The psychologicalschool of thought considers SA as being contained entirely withinthe mind of the agent (Stanton et al., 2010). Endsley’s (1995) three-level model has receivedmost attention of the contributions withinthis approach. This model presents SA as consisting of three sepa-rate levels: perception, comprehension and projection (Endsley,1995). By perceiving the available elements in the environment(Level 1) and understanding these (Level 2) the individual canmakeprojections about the future (Level 3) and ultimately take actionsin-line with his or her predictions. This information processingapproach to describing SA provides an intuitive definition of theconcept (Salmon et al., 2006).

In contrast, the systems ergonomics school considers SA as anemergent property arising from people’s interactionwith the world(Stanton et al., 2006). Bubb (1988) defines systems ergonomics as“the application of system technics on ergonomical problems” (p.233); both the term and its sentiment are in wider use within thehuman factors and ergonomics community (Helander, 1997; Clegg,2000; Waterson, 2009). SA has been described as a systems

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687678

phenomenon (Salmon et al., 2008, 2009a). The approach arguesthat SA is distributed cognition, where the mind is situated in aninterdependent relationship with the world (Stanton et al., 2010).Stanton et al. (2006) proposed a theory of Distributed SA which isbased on three theoretical approaches: schemata, perceptual cycleand distributed cognition. The systems ergonomics approach doesnot discount the significance of the individual to the developmentof SA; however, the distributed model of SA considers that theindividual is simply one part of the system (Stanton et al., 2006).The individual holds genotype schemata which are activated by thetask which is being performed (Stanton et al., 2009b,c. Throughtask performance the phenotype schemata are created through theinteraction between people, the world and artefacts (Salmon et al.,2009a). In this approach it is assumed that SA does not residewithin the individual alone but within the system. In a similar way,Bedny and Meister (1999) argue that the individual is so closelycoupled to their environment that they cannot be analysed inisolation from it; as such, people and artefacts form a “jointcognitive system” (Hollnagel, 2001). This is echoed by Gorman et al.(2006) who consider SA an interaction-based phenomenon.

Salmon et al. (2008) argue that cognitive processes emerge fromand are distributed throughout this system. It is the interactionsbetween people and technology which enables distributed cogni-tion (Salmon et al., 2008, 2009b). Patrick and Morgan (2010)highlight that the individual needs to continuously extract andmake sense of its environment and argue that ‘the important point isthat the relevant awareness and comprehension of something in theenvironment is determined by the goals of the system that can bedecomposed both between and within people and artefacts’ (p. 5).Smith and Hancock (1995) accede to Neisser’s (1976) perceptualcycle model when considering SA. Accordingly, they argue thatinformation and action flow incessantly around the cycle, in that‘the environment informs the agent, modifying its knowledge.[and]. knowledge directs the agent’s activity in the environment.That activity samples and perhaps anticipates or alters the environ-ment, which in turn informs the agent’ (p. 141).

Endsley’s (1995) model provides an integrated and coherentdefinition of the phenomenon of the individual (Wickens, 2008).The definition is often favoured in the literature as it is easilyoperationalised by the three discrete levels of SA (Salmon et al.,2006; Ma and Kaber, 2005; Kaber et al., 2006). Within thepsychological school of thought and within the frames of Endsley’smodel, team SA is understood as Shared SA where team membersshare SA requirements for a task. Nofi (2000) states that Shared SAimplies that all team members understand a given situation in thesameway. A benefit of this approach is that if the team essentially is‘one person’ support can be aimed at the team as a whole throughthe use of shared interfaces and training. Yet Salas et al. (1995)argue, as do others, that team SA is more than the sum of itsparts (Masys, 2005; Salmon et al., 2009b,c). Therefore, simplyadding individual SA together to provide a measure of team SA isnot satisfactory (Gorman et al., 2006).

In contrast, Stanton et al. (2006) advocate the view that teammembers possess unique but compatible parts of system aware-ness, rather than share SA. They argue that compatible SA is theglue that holds the distributed system together (Stanton et al.,2006, 2009a). Individual team members enhance and updateeach other’s awareness through SA relevant transactions (Salmonet al., 2009a). These transactions may be interpreted in light oftheir specific tasks and goals (Salmon et al., 2008).

Stanton et al. (2006) define Distributed SA as ‘activated knowl-edge for a specific task, at a specific time within a system’ (p. 1291).This definition is considerably more difficult to operationalise thanthat given by Endsley (1995) as what is ‘activated knowledge’ mayinclude cognitive and behavioural processes across the system.

What constitutes ‘knowledge’ must for instance be separated outfrom mere data and information; however, such analysis havemerit as it enables analysis of what may have been ‘missing’ insituations where there has been a breakdown in team performance,such as in friendly fire incidents (Rafferty et al., 2010).

Salmon et al. (2008) clarify the definition of Distributed SA byexplaining that information held by the system becomes active atdifferent points in conjunction with the goals and tasks beingperformed and their associated constraints. As such, any individualhave different SA for the same situation, depending on their teamrole and tasks (Salmon et al., 2008). It is clear that defining theboundaries of the team or the system, as well as the individual partswithin it and their respective roles, requires effortful analysis.However, this would seem a fair reflection of the nature of teamdynamics and the complex environments they operate in.

Communication, as an SA transaction, connects and maintainsthe different parts of the distributed system. The model of team SAas distributed therefore views the system as a whole ‘by consider-ation of the information held by the artefacts and people and the wayin which they interact’ (Stanton et al., 2010, p. 34).

The differing explanations of the phenomenon, as outlinedabove, take different units of analysis as points of measurement.

1.2. Unit of analysis

The psychological approach emphasises cognitive capabilities ofthe individual that are necessary and sufficient to achieve SA. Sarterand Woods (1991) considered SA as a variety of cognitive pro-cessing activities which are critical to agile performance. A mentaltheory of the world, developed by the individual, supports anunderstanding of how parts fit together and of how future states ofthe world can be foreseen (Banbury et al., 2004). Artman (2000)emphasised the individual as an active intermediary in devel-oping SA and sees it as an ‘active construction of a situation model’(p. 1113). The psychological approach therefore takes the individualas the unit of analysis for SA.

In contrast, it is the system which is the unit of analysis in theDistributed SA framework and the systems ergonomics approach(Salmon et al., 2009c). Klir (1972) defined a system as ‘anarrangement of certain components so interrelated as to form a whole’(p. 1) while von Bertalanffy (1950) states in explaining the tenets ofthe General Systems Theory that ‘living systems are open systems,maintaining themselves in exchange of materials with [their] envi-ronment’ (p. 23). The model of Distributed SA is therefore foundedon ‘the notion that in order to understand behaviour in complexsystems it is more useful to study the interactions between parts in thesystem and the resultant emerging behaviour rather than the partsthemselves’ (Salmon et al., 2008, p. 369). This is similar toHollnagel’s (1993) argument that team behaviour should be ana-lysed at amacro level, e.g. by taking the environment and context ofthe team into account (Stanton et al., 2001). The two theoreticalapproaches described above suggest that there is still disagreementas to how team SA is best understood, either the sum of individualsor the team interaction as a whole. The different entities underanalysis in the psychological and systems ergonomics approachesto SA informed the development of diverse measurement tech-niques which are considered in the following.

1.3. Measurement of SA

Perhaps the most popular of measures within the psychologicalschool of thought is the Situation Awareness Global AssessmentTechnique (SAGAT) which is developed from Endsley’s three-levelmodel (Endsley et al., 1998). SAGAT is presented as an objectivemeasure of SA in individuals, although Annett (2002) argued that all

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687 679

knowledge is basedon subjective experience, casting somedoubtonwhether complete objectivity in self-reportingmeasures is possible.

Endsley et al. (1998) assert that measures of SA provide an indexof the ability of individuals to acquire and integrate informationfrom the environment. Measurement within the psychologicalapproach therefore seeks to determine, either through objective orsubjective measurement techniques, the extent of this ability in anindividual. The objectivity is claimed by the freeze-probe techniquewhich involves the simulation of any operation, such as air trafficcontrol, being frozen at a random point in time and specific ques-tions about the situation as it was before the freeze are presented(Endsley et al., 1998). A SAGAT score is calculated for each partici-pant after the simulation (Stanton et al., 2005). Endsley et al. (1998)argue that the main advantage of SAGAT is its provision of an indexacross the three levels of SA. An obvious disadvantage is that themeasure requires freezes to take place, disrupting natural taskperformance (Endsley et al., 1998; Endsley, 2000). Another criticismof the measure is that it is heavily reliant onmemory (Salmon et al.,2009c). Despite its origin as an individual measure of SA it is alsoapplied to assess team SA. Although heavily criticised when used toprovide a team measure, the SAGAT scores of the individuals in theteam are averaged to provide an overall team SA score (Salas et al.,1995; Masys, 2005; Salmon et al., 2008; Stanton et al., 2009a).

The Situation Awareness Rating Technique (SART) is alsoa popular measure within the psychological approach. SARTprovides an assessment of SA based on operators’ own subjectiveopinions (Taylor, 1990). It consists of 14 components which aredetermined in relation to their relevance to the task or environ-ment under study (Endsley et al., 1998). The operators are requiredto rate on a series of bipolar scales the degree to which theyperceive a demand on their resources, the supply of resourcesavailable to them and their understanding of the situation (Endsleyet al., 1998). The scales are combined to provide an overall measureof SA (Endsley et al., 1998).

Given that Distributed SA considers the joint cognitive system asa whole it is clear that measurement of SA within this theoreticalframeworkmust take a broader systems theoretical view. Kirlik andStrauss (2006) argue that a “comprehensive approach to SAmodellingand measurement requires techniques capable of representing anddecomposing both the technological and psychological contributions toSA” (p. 464). The aim here is to consider the interaction betweenindividuals and their environment to achieve a holistic picture ofthe SA contained in a system (Stanton et al., 2010). Kirlik and Strauss(2006) go on to state that “modelling SA as distributed is important inan engineering sense because only techniques capable of representingthe external contributors to SA are capable of analyzing and predictinghow technology design influences on SA” (p. 464). Social NetworkAnalysis (SNA) and Propositional Networks (PN) have been appliedas a way of describing Distributed SA as these are able to reveal theinformation which constitutes a systems knowledge, the relation-ships between the different pieces of information and the ways inwhich each component in the system utilises it (e.g. Stanton et al.,2008; Salmon et al., 2008; Houghton et al., 2006). These reflectthe ‘object-relation-subject’ patterns within the CDM and give aninsight into inherent knowledge structures of the system and theway in which these may be activated (Salmon et al., 2009b).Distributed SA is therefore represented in pieces of information andthe relationship between them (Salmon et al., 2009c; Stanton et al.,2010). A PN can reflect the entire systems SA by showing all theinformation containedwithin it as well as identifying individuals orartefacts within the system in detail. The latter approach enablesa consideration of compatible SA.

Theoretical deliberation of the phenomenon of SA has consid-ered whether it is best understood as a product or a process (Sarterand Woods, 1991; Endsley, 1995; Banbury et al., 2004; Patrick and

James, 2004). As all four measures from both the psychologicaland the systems ergonomics schools of thought considered theoverall SA attained within each of the conditions at the end of taskperformance, all can be understood as ‘product’ measures.However, the measures of Distributed SA have the potential toconsider both the product of SA and the process of achieving it byconsidering the emergence of SA through interactions withina system over time.

It is clear from the discussions above that the two schools ofthought, despite seeking to explain the same phenomenon, offerdifferent conceptions of the nature of SA. Consequently, if thepsychological approach provides the most appropriate theory ofteam SA then SAGAT and SART would be the more sensitivemeasure, and reversely, if the systems ergonomics approach offersthe most appropriate theory then PNs and SNA would prove themore insightful measure. The following hypotheses were thereforetested to ascertain which approach had the sensitivity required todistinguish between two different teams and explain thesedifferences:

Hypothesis 1. The measures derived from the psychologicaltradition of SAe SAGATand SARTewill reveal differences betweenthe two conditions, if SA is shared between team members.

Hypothesis 2. The measures derived from the systems ergo-nomics tradition of SA e SNA and PNs e will reveal differencesbetween the two conditions, if SA is distributed between teammembers.

2. Method

2.1. Participants

A sample of 34 was drawn from the University of Southampton’spostgraduate population. The participants were randomly dividedinto two groups, one Hierarchical organisational structure and onePeer-to-Peer organisational structure, with 17 participants in eachcondition. Both conditions had an identical mean age of 28(S.D. ¼ 5.52). In the Hierarchical condition there were 15 males and2 females, while in the Peer-to-Peer condition there were 12 malesand 5 females. Though there are fewer female participants thanwould be expected from the general student population thepurpose of this case study was to discover differences revealed bythe SA measures and as such the gender bias was not expected toimpact on the findings. Furthermore, students were selected asparticipants as a result of research which has shown that there is nodifference between using novices, such as students, and experts forsimple task measures such as those considered here (Walker et al.,2010). Ethical permission for the experiment was requested andgranted by the University of Southampton.

2.2. Design

The present study was of a between-subjects experimentaldesign. The between variable was organisation structure; Hierar-chical and Peer-to-Peer and participants were randomly assignedinto either of these. The Hierarchical condition had three layers, onecoordinating leader, four team leaders in themiddle and three teammembers reporting to each team leader as illustrated in Fig. 1.

The Peer-to-Peer organisational structure allowed all teammembers to interact with any other teammember, as seen in Fig. 2.Information had the potential to flow freely between teammembers and the group was self-managed.

The use of different organisational structures to design differentexperimental groups has also been reported elsewhere (Walker

Fig. 1. Hierarchical organisational structure.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687680

et al., 2009b,c). The dependent variables were SA, time and taskperformance.

2.3. Equipment

The study utilised the ELICIT software tool which allows for anorganisation of participants in two conditions while they performan intelligence analysis task (Ruddy, 2007) while the ELICIT LogAnalyzer (CCRP, 2009) was used to extract performance data. Acomputer room was set up which provided a computer, keyboard,mouse, desk and chair, for each of the participants and the studyleader. In addition headphones were supplied for the participants,allowing them to listen to a video instruction describing the soft-ware interface. Paper copies of the surveys were administratedwhile pens and sheets of paper were made available for partici-pants to make notes during the experiment.

2.4. Task

The participants were instructed to use information elementssupplied during the experiment to establish ‘who’, ‘what’, where

Fig. 2. Peer-to-Peer organisational structure.

and ‘when’ of an adverse attack. Once the correct solution wassupplied by either team the experiment ended.

In the Hierarchical condition the participants were divided intoone of the three team functions, cross-team coordinator, teamleader or team member. They were further grouped by topic toidentify either who, what, where or when of the adversary attack.In the self-managed Peer-to-Peer condition all team memberscontributed equally to the identification of who, what, where andwhen of the attack. Both groups were required to utilise theorganisational structure they were organised into to successfullycomplete the team task collaboratively. This was done by compilinginformation, posting it on relevant group web pages and sharing itwith relevant team members (Ruddy, 2007). Importantly, in theHierarchical condition access to information was constrained bythe team function to which a participant was allocated. In this wayonly team members in the “who” group could access informationrelated to the “who” of the attack, such as information shared withthem by other team members, information sent from the experi-ment software (so called ‘official’ information) and informationposted on the who-related website. The cross-team coordinatorhad access to information on all web pages and could communicatewith anyone. The Peer-to-Peer condition had no such constraintsand each team member could share information with anyone elseas well as utilise information posted on any web page. See Table 1for an overview of each condition’s specific access to information.

2.5. Procedure

The study used the procedure set out in Ruddy (2007),comprising the following steps:

� Participants were recruited and randomly assigned eitherHierarchical or Peer-to-Peer conditions;

� Participants welcomed and the experiments aims describedbriefly;

� A video was shown to demonstrate how the experiment soft-ware should be used;

� In the Hierarchical condition participants were at this pointrandomly assigned into one of four groups and team roles (i.e.either ‘who’, ‘what’, ‘when’ or ‘where’);

� Participants randomly assigned to the self-managed Peer-to-Peer group had access to all information;

� Familiarisation game. No talking allowed during or after thegame. Technical help given to any participant who has ques-tions about the experiment interfaces;

� A short break was given but no talking was allowed;� The experimental game was started. All interaction via textualmeans using the ELICIT software interface;

� Administration of experimental surveys;� Debriefing of participants.

Table 1Access to information.

Condition Availability of information

Who or What or When or Where

Who and What and When and Where

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687 681

2.6. Data reduction and analysis

A SAGAT questionnaire was administered and a score wascalculated (Endsley, 2000). The SAGAT probes were developed fromthe information elements provided in the game and categorisedinto the three levels of SA as described by Endsley’s (1995) model.The highest possible score was 21. Individual SAGAT scores werecalculated separately for each team member and a median for theteamwas obtained. In-line with the literature described above, theSAGAT score provides an indication of Shared SA in the twoconditions. The three levels of SA as measured by SAGAT wereinvestigated using a histogram to compare the Hierarchical andPeer-to-Peer conditions. In addition a SART questionnaire wasadministered and a median score was calculated for the team(Stanton et al., 2005; Funke and Galster, 2009). To compare differ-ence in mean rank of SAGAT and SART scores between the twogroups the non-parametric ManneWhitney U test was performedfor each score.

Distributed SAwasmeasured using the Critical DecisionMethod(CDM); which were analysed to produce PNs (Salmon et al., 2009b;Klein, 2000; Klein and Armstrong, 2005). Walker et al. (2010)described the process of data reduction and creation of PNs fromthe outputs of CDM transcripts, which was followed here; firstlya word frequency list was established from the CDM transcripts,and secondly, words with an insufficient frequency were discarded.This enables words which form the PNs to focus on the groupcontributions, not individuals (Walker et al., 2010). Walker et al.(2010) explained that by plotting a word frequency list in a graph,“the word frequency curve approximates to a form of Scree plot” (p.477). Drawing a line to where the curve flattens out provided a cut-off point for words of an individual nature, leaving the grouprelevant words with the higher frequencies. An inter-rater reli-ability test was performed which achieved 80% agreement.

Social Network Analysis (SNA) was used to examine the struc-ture of communications and reveal patterns that emerged in eachcondition, as has been done elsewhere (Walker et al., 2006, 2009a).In order to describe the PNs in a quantitative manner, SNA of thePNs’ diameter, density, Bavelas-Leavitt centrality and sociometricstatus, number of nodes and number of links between nodes wasperformed. Diameter measures the largest number of agents whichmust be traversed in order to travel from one node [or agent] toanother (Weisstein, 2008; Harary, 1994). As such the diameter ofa network gives an insight into how ‘big’ it is. Walker et al. (2009e)stated that “the maximum value for density is 1, indicating that allnodes are connected to each other” (p. 85e6).The density of

Fig. 3. SAGAT scores for the two

a network is therefore the proportion of all the ties observed in thenetwork and gives insight into the speed at which information canbe diffused (Walker et al., 2009e). The B-L Centrality statistic givesa centrality value for each node in the network by calculating ”themost central position in a pattern [which] is the position closest to allother positions” (Leavitt (1951) cited in Walker et al., 2009d, p. 18).Walker et al. (2009e) hypothesised that Hierarchical organisationalstructures would generally possess fewer highly central agentscompared with Peer-to-Peer organisational structures. Sociometricstatus was measured to identify the information concept mostfrequently occurring in either PN, while a simple count wasmade ofthe number of nodes and the links existing between them.

The structure of communication was thus examined andpatterns of qualitative differences were quantitatively investigated(Walker et al., 2009a). See Salmon et al. (2009b) for a furtherdiscussion of PNs as an analytical and representational tool forDistributed SA and Walker et al. (2006, 2009a) for further discus-sion of the use of SNA.

Performance was in addition measured to investigate differ-ences arising from the organisational constraints placed on them.Performance was measured in terms of sharing behaviours, howquickly either condition completed the task and whether theycorrectly identified the solution. It was expected that there wouldbe a difference in the time taken to complete the task, while it wasexpected that both conditions would complete the task success-fully. The following hypothesis was tested:

Hypothesis 3. The performance of the two conditions asmeasured by ELICIT will reveal differences between them in termsof time to complete, correct identification and sharing behaviours.

3. Results

The results of themeasures related to SAGAT, SART, SNA and PNsfor the two teams are briefly presented.

3.1. SAGAT

A median score of 12 was obtained for the Hierarchical condi-tion while a median of 13 was obtained for the Peer-to-Peercondition, neither score more than just over half of the maximumscore of 21, see Fig. 3. There were no statistically significantdifferences between the Hierarchical and Peer-to-Peer conditionson the overall SAGAT scale (U¼ 0.559, P¼N.S.). Participants in bothconditions therefore reported the same level of objective SA.

organisational structures.

Fig. 4. SAGAT score by the three SA levels for both organisational structures.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687682

The SAGAT scores obtained associatedwith the three levels of SAdescribed by Endsley (1995) were compared for the two conditions,illustrated in Fig. 4.

ManneWhitney U tests were performed for each of the threelevels of SA to determine whether the medians obtained for eachlevel were equal for the two conditions. No statistically significantresults were found for level 1 (U¼ 85.5, P¼ N.S.), level 2 (U¼ 119.5,P ¼ N.S.) or level 3 (U ¼ 128, P ¼ N.S.) when compared betweenHierarchical and Peer-to-Peer conditions.

3.2. SART

The median SART score achieved for the Hierarchical and Peer-to-Peer conditions was 4 and 5 respectively. No statisticallysignificant differences were found for the ManneWhitney ranksum test on the overall SART scale (U¼ 0.786, P¼ N.S.). Participantstherefore report the same relatively low level of subjective SA inboth conditions. Fig. 5 shows the spread of SART scores.

Themedian achieved by either condition on each of SART’s threedimensions were compared, see Fig. 6. No statistically significantdifferences were found between Hierarchy and Peer-to-Peerconditions when considering the test statistics of the Man-neWhitney rank sum test on either of the SART dimensions:demand (U¼ 142, P¼N.S.), understanding (U¼ 139.5, P¼ N.S.) andsupply (U ¼ 140, P ¼ N.S.).

The outcomes of the two measures were examined using theManneWhitney two-sample rank sum test. There were no

Fig. 5. Spread of

statistically significant differences between the overall SAGAT(U ¼ 120, P ¼ N.S.) and SART (U ¼ 129, P ¼ N.S.) scores. Similaritiesbetween SAGAT and SART were investigated further by subjectingthe three SA levels measured by SAGAT and the three maindimensions of SART (demand, understanding and supply) toSpearman’s test of correlation. No statistically significant correla-tionwas found between any of the three SAGAT levels and the SARTdimensions (P ¼ N.S.). Hence no difference was found between thetwo measures for either condition.

The findings above did not reveal any statistically significantdifferences between the Hierarchical and Peer-to-Peer conditionsin terms of the quantitative measures of SA derived from thepsychological school of thought. No support was therefore foundfor Hypothesis 1. In the following section the results with regards toHypothesis 2 are presented.

3.3. Propositional Networks

Frequency counts of words extracted from the CDM (Klein,2000; Klein and Armstrong, 2005) transcripts were performed.Cut-off points were identified for words which were to be includedin the PNs as nodes (onlywords appearing frequently are of interestto this team level analysis). For the Hierarchical condition the cut-off point was 4, hence no concepts mentioned fewer than 4 timeswere included, as illustrated in Fig. 7 (Salmon et al., 2009a,b,c). Forthe Peer-to-Peer condition the cut-off point was 5 individual cita-tions (see Fig. 8).

SART scores.

Fig. 6. SART score by the three SA dimensions for both organisational structures.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687 683

Fig. 9 depicts the PN created from the subject-relation-objectpatterns revealed in the CDM responses for the Hierarchicalcondition; Fig. 10 displays these results for the Peer-to-Peercondition.

The PNs show that although they contain many of the sameconceptual elements there are a number of concepts that areexclusive to one condition. For instance, “receive” exists only in theHierarchical condition, while “process” is unique to the Peer-to-Peer condition. The relationships between the concepts in any ofthe PNs are also qualitatively different, which reflect that theinformation available to the team is utilised in different ways. Forinstance, in the Hierarchical condition the information element“information” is directly connected to “attack” but only indirectlyconnected to “target” (through “attack”). This is reversed in thePeer-to-Peer condition.

Fig. 7. Frequency of words for the Hierarchical organisational structure.

3.4. Social Network Analysis

Applying network analysis to the pattern of communicationenables a quantitative probe of the qualitative findings given above.Table 2 shows the PNA statistics obtained for the PNs diameter,number of nodes, links between nodes, density, centrality andsociometric status.

As can be seen from the table above the Hierarchical PN wasdenser than the Peer-to-Peer PN. In both structures “Attack” wasthe node with highest sociometric status, although the highermean for sociometric status for the Peer-to-Peer PN indicated that“Attack” had greater connectivity in this condition. This means thatthe nodes which were connected to the “Attack” node referred to itmore frequently in the Peer-to-Peer PN than in the Hierarchical PN.

Fig. 8. Frequency of words for the Peer-to-Peer organisational structure.

Fig. 9. Propositional Network for the Hierarchical organisational structure.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687684

The Hierarchical and Peer-to-Peer PNs have 10 nodes incommon; however, each condition had a number of additionalnodes which were not shared. In the Peer-to-Peer PN there were 8additional nodes: team, share, receive, difficult, irrelevant, web-sites, when and where. These additional nodes refer to threethemes: team work, issues with information and source of infor-mation. The Hierarchical PN had 5 additional nodes: inbox, piece,factoid, find and process. These refer to searching for information.On all other metrics there are only small differences.

The findings from the PNs and SNA analyses revealed qualitativeand quantitative differences between the two conditions. Supportwas therefore found for Hypothesis 2.

3.5. Performance

The Peer-to-Peer condition achieved task completion in 2292 swhich was marginally faster than the Hierarchical condition, whichcompleted in 2440 s (i.e. the Peer-to-Peer condition was 2 min 28 sfaster than the Hierarchical condition).

Both conditions correctly identified the solution in the experi-ment trial. In the Hierarchical condition it was only the Commanderwho could make an identification attempt and this participant didcorrectly identify the solution. In the Peer-to-Peer condition therewere 4 identification attempts, of which 3 were successful.

Three types of sharing behaviours were measured; direct sharesbetween team members: posting on websites and pulling infor-mation from these websites.

There were greater instances of sharing in the Hierarchicalcondition (326) than in the Peer-to-Peer condition (119). Similarlythere was greater number of posts in the Hierarchical condition

(154) than in the Peer-to-Peer condition (131). However, there weregreater instances of pull, i.e. extracting information, in the Peer-to-Peer condition (747) than in the Hierarchical condition (167), seeTable 3.

These findings therefore reveal a small difference between thetwo conditions, specifically with regards to the patterns of sharingbehaviours the conditions displayed. Hypothesis 3 is thereforesupported. The findings are summarised in Table 4.

4. Discussion

This study aims to contribute to the ongoing debate aboutappropriate theory and measures to assess team SA. By contrastingtwo approaches to SA the discrepancy between them in terms oftheir explanation of what SA is, the unit which are subjected toanalysis and how the phenomenon are measured has been high-lighted. By applying quantitative and qualitative measures whichhave been developed within these approaches, this incongruity isfurther emphasised.

Within the psychological school of thought Shared SA is under-stood as shared SA requirements for team members (Endsley andRobertson, 2000; Endsley, 1995; Nofi, 2000). A difference wastherefore expected to be revealed between the Hierarchical andPeer-to-Peer conditions when analysing SA as Shared SA, asmeasured by SAGAT and SART. Specifically, it was expected that thePeer-to-Peer organisation, in which all team members share thesame teamrole and task responsibility,wouldobtain a higher SAGATscore than theHierarchical condition.However, thefindings for eachShared SA measure did not reveal any difference between the twoconditions. No support was therefore found for Hypothesis 1.

Fig. 10. Propositional Network for the Peer-to-Peer organisational structure.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687 685

SAGATand SARTaim to reveal the product of SA as the individualhas achieved it in task performance. These measures therefore ineffect consider each individual in isolation by estimating howmuchof the overall situation they were aware of at the time ofmeasurement. These estimates are then added together to give anoverall team score. Gorman et al. (2006) expressed concern thatsimply adding individual SA together to give team SA scores isunsatisfactory. The findings presented here emphasise that suchconcerns remain relevant. Stanton et al. (2010) argued that usingindividual SA measures to interpret team SA does not take intoaccount the wider environment of the individuals which is utilisedto aid task performance in the most efficient way, e.g. artefacts andother team members. While SAGAT and SART have been proven(Banbury et al., 2004; Endsley et al., 1998) to give valuable insightinto individual SA the findings presented here indicate that thesemay be less sensitive when applied to assess team SA.

The systems approach, in contrast to the notion of team SA asbeing shared, argues that SA is an emergent property of collabo-rative systems. The qualitative findings in the PNs reflect such

Table 2SNA statistics for Hierarchical PN and Peer-to-Peer PN.

Hierarchical Peer-to-Peer % difference

Diameter 2.0 2.0 0.00Number of nodes 15 18 16.67Links between nodes 26 28 7.15Density 0.53 0.41 22.65Centrality (mean) 8.91 8.33 6.51Sociometric status (mean) 3.63 4.44 18.25

systems in the differing patterns of interactions which emerged.The individual team member is only one part of this system andeach has awareness which is different but compatible to that ofother team members. According to Stanton et al. (2006), compat-ible SA holds the distributed system together. They further arguedthat Distributed SA is activated knowledge which is utilised fora particular task within the system. The PNs showed the relevantinformation contained within the two conditions and the relationallinks between them. These links showed how the informationelements were utilised within the teams. Both PNs thereforeexhibited Distributed SA. The PNs for the two conditions showedthat the two teams utilised their organisational structure indifferent ways to coordinate their efforts for successful taskcompletion. The PNs further showed that although they containmany of the same conceptual elements there were a number ofconcepts that were exclusive to one organisational structure,revealing qualitative differences between them.

The SNA data found that the Hierarchical PN was denser thanthe Peer-to-Peer PN. In contrast the Peer-to-Peer PN had a highermean sociometric status than the Hierarchical PN. The findings ofthe PNs and the SNA reveal qualitative (i.e. differences in the

Table 3Sharing behaviours.

Hierarchical Peer-to-Peer

Share 326 119Post 154 131Pull 167 747

Table 4Summary of main findings.

SAGAT SART PN SNA Performance

Hierarchy (Median) 12 4 5 nodes not shared withPeer-to-Peer condition.

15 nodes Greater instancesof direct sharing

Peer-to-Peer (Median) 13 5 8 nodes not shared withHierarchical condition

18 nodes Greater instancesof information pull

Difference No statistically significantdifference found

No statistically significantdifference found

Qualitative differencesfound between the PNs.

Hierarchy found to be denserthan the Peer-to-Peer PN

Difference found betweenconditions on sharing behaviours

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687686

concepts represented by the nodes) and quantitative (i.e. datashown in Table 2) differences between the two conditions. Theperformance measures which revealed differences between thetwo conditions’ sharing behaviours support the findings from thePNs and SNA.

The ability of the measures of Distributed SA to reveal a differ-ence between the two conditions provided support for Hypothesis2. The findings reported here therefore lend support to the notionof Distributed SA, expressed by Salmon et al. (2008), that under-standing behaviour in complex systems requires study beyond theindividual components of a system to consider also interactionsbetween them. Team behaviour should be analysed at a macro level(Hollnagel, 1993). The comparison of the two theoreticalapproaches to SA e Shared and Distributed SA e therefore showedthat Distributed SAwas a sensitivemeasure of team SA and this wasverified by the subtle differences in task performance.

5. Conclusion

This study compared the psychological and systems ergonomicsapproaches to SA and also measured team SA within these frame-works. The findings show differences in terms of how they explainthe phenomenon, the level of analysis andmethods for assessment.The unit of analysis in the psychological approach is the individual,whereas the entire system is analysed in the systems ergonomicsapproach. In the psychological approach the SA captured isconsidered a product, whilst it is considered as a process arisingfrom interaction in the systems ergonomics approach. ExplainingSA as either a cognitive construct residing in the mind of an indi-vidual, or as a systems phenomenon which emerges throughinteraction between individuals and artefacts within the system,naturally leads to different measurement techniques. No significantstatistical differences were found between the two different teamstructures when considering the scores obtained for SAGAT andSART (measures developed within the psychological school ofthought to assess individual SA). Both qualitative and quantitativedifferences were found, however, when applying SNA and PNs(measures developed within the systems ergonomics approach toassess team SA). These findings were also supported by differencesfound in the performance of the two conditions, specificallydifferent patterns of sharing and pulling behaviours. The resultspresented here emphasise the need to clarify the nature of team SAand the associated measures which are appropriate to assess thisparticular phenomenon.

Acknowledgements

This work from the Human Factors Integration Defence Tech-nology Centre was part-funded by the Human Sciences Domain ofthe UK Ministry of Defence Scientific Research Programme. Wewould like to express our gratitude to the ELICIT community, inparticular to Mary Ruddy of Parity Communications Inc. for herinstruction in the ELICIT software platform, experiment materialsand practical advise.

References

Annett, J., 2002. Subjective rating scales: science or art? Ergonomics 45 (14),966e987.

Artman, H., 2000. Team situation assessment and information distribution. Ergo-nomics 43 (8), 1111e1128.

Banbury, S.P., Croft, D.G., Macken, W.J., Jones, D.M., 2004. A cognitive streamingaccount of situation awareness. In: A Cognitive Approach to Situation Aware-ness: Theory and Application. Ashgate, Hampshire.

Bedny, G., Meister, D., 1999. Theory of activity and situation awareness. Interna-tional Journal of Cognitive Ergonomics 3 (1), 63e72.

von Bertalanffy, L., 1950. The theory of open systems in Physics and Biology. Science111 (2872), 23e29.

Bubb, H., 1988. System ergonomics as an approach to improve human reliability.Nuclear Engineering and Design 110 (2), 233e245.

CCRP, 2009. ELICIT Log Analyzer Retrieved August 24, 2010, from: http://www.dodccrp.org/html4/elicit.php.

Clegg, C.W., 2000. Sociotechnical principles for system design. Applied Ergonomics31 (5), 463e477.

Endsley, M., 1995. Toward a theory of situation awareness in dynamic systems.Human Factors 37 (1), 32e64.

Endsley, M., 2000. Theoretical underpinnings of situation awareness: a criticalreview. In: Situation Awareness Analysis and Measurement. Lawrence ErlbaumAssociates, Mahwah, NJ.

Endsley, M., Robertson, M.M., 2000. Situation awareness in aircraft maintenanceteams. International Journal of Industrial Ergonomics 26 (2000), 301e325.

Endsley, M., Selcon, S.J., Hardiman, T.D., Croft, D.G., 1998. A comparative analysis ofSAGAT and SART for evaluations of situation awareness. In: Proceedings of the42nd Annual Meeting of the Human Factors & Ergonomics Society. Presented atthe Annual Meeting of the Human Factors & Ergonomics Society. HumanFactors & Ergonomics Society, Chicago, IL.

Funke, G.J., Galster, S.M., 2009. The effects of cognitive processing load andcollaboration technology on team performance in a simulated command andcontrol environment. International Journal of Industrial Ergonomics 39,541e547.

Gorman, C.J., Cooke, N.J., Winner, J.L., 2006. Measuring team situational awarenessin decentralized command and control environments. Ergonomics 49 (12e13),1312e1325.

Harary, F., 1994. Graph Theory. Addison-Wesley, Reading, MA.Hollnagel, E., 1993. Human Reliability Analysis: Context and Control. Academic

Press, London.Helander, M.G., 1997. Forty years of IEA: some reflections on the evolution of

ergonomics. Ergonomics 40 (10), 952e961.Hollnagel, E., 2001. Extended cognition and the future of ergonomics. Theoretical

Issues in Ergonomics Science 2 (3), 309e356.Houghton, R.J., Baber, C., Stanton, N.A., Salmon, P.M., Stewart, R., Walker, G.H., 2006.

Command and control in emergency services operations: a social networkanalysis. Ergonomics 49 (12e13), 1204e1225.

Kaber, D., Perry, C., Segall, N., McClernon, C.K., Prinzel III, L.J., 2006. Situationawareness implications of adaptive automation for information processing inan air traffic control-related task. International Journal of Industrial Ergonomics36, 447e462.

Kirlik, A., Strauss, R., 2006. Situation awareness as judgement I: statistical modelingand quantitative measurement. International Journal of Industrial Ergonomics36 (2006), 463e474.

Klein, G., 2000. Cognitive task analysis of teams. In: Cognitive Task Analysis. Law-rence Erlbaum Associates, Mahwah, NJ, pp. 417e430.

Klein, G., Armstrong, A.A., 2005. Critical decision method. In: FactorsMethods, Human (Ed.), A Practical Guide for Engineering and Design. Ashgate,Aldershot.

Klir, G.J., 1972. Trends in General Systems Theory. Wiley-Interscience, New York.Ma, R., Kaber, D., 2005. Situation awareness and workload in driving while using

adaptive cruise control and a cell phone. International Journal of IndustrialErgonomics 35, 939e953.

Masys, A.J., 2005. A systemic perspective of situation awareness: an analysis of the2002 mid-air collision over Uberlingen, Germany. Disaster Prevention andManagement 14 (4), 548e557.

Neisser, U., 1976. Cognition and Reality: Principles and Implications for CognitivePsychology. Freeman, San Fransisco.

Nofi, A.A., 2000. Defining and Measuring Shared Situational Awareness. Centre forNaval Analyses, Alexandria.

L.J. Sorensen, N.A. Stanton / International Journal of Industrial Ergonomics 41 (2011) 677e687 687

Patrick, J., James, N., 2004. A task-oriented perspective of situation awareness. In:Banbury, S., Tremblay, S. (Eds.), A Cognitive Approach to Situation Awareness:Theory and Application. Ashgate & Town, Aldershot, UK.

Patrick, J., Ahmed, A., Halliday, P., 2006. Observational assessment of situationawareness, team differences and training implications. Ergonomics 49, 393e417.

Patrick, J., Morgan, P.L., 2010. Approaches to understanding, analysing and devel-oping situation awareness. Theoretical Issues in Ergonomics Science 11 (1e2),41e57.

Rafferty, L.A., Stanton, N.A., Walker, G.H., 2010. The famous five factors in teamwork:a case study of fratricide. Ergonomics 53 (10), 1187e1204.

Ruddy, M., 2007. The Experimental Laboratory for Investigating Collaboration,Information-sharing and Trust. (Report). Parity Communications Inc, Boston,MA.

Salas, E., Prince, C., Baker, D.P., Schrestja, L., 1995. Situation awareness in teamperformance: implications for measurement and training. Human Factors 37(1), 123e136.

Salmon, P.M., Stanton, N.A., Walker, G.H., Baber, C., Jenkins, D.P., McMaster, R.,Young, M.S., 2009a. What is really going on? Review of situation awarenessmodels for individuals and teams. Theoretical Issues in Ergonomics Science 9(4), 297e323.

Salmon, P.M., Stanton, N.A., Walker, G.H., Green, D., 2006. Situation awarenessmeasurement: a review of applicability for C4i environments. Applied Ergo-nomics 37 (2006), 225e238.

Salmon, P.M., Stanton, N.A., Walker, G.H., Jenkins, D.P., 2009b. Distributed SituationAwareness. Ashgate, Farnham.

Salmon, P.M., Stanton, N.A., Walker, G.H., Jenkins, D.P., Ladva, D., Rafferty, L.,Young, M.S., 2009c. Measuring situation awareness in complex systems:comparison of measures study. International Journal of Industrial Ergonomics39 (2009), 490e500.

Salmon, P.M., Walker, G.H., Jenkins, D.P., Baber, C., McMaster, R., 2008. Representingsituation awareness in collaborative systems: a case study in the energydomain. Ergonomics 51 (3), 367e384.

Sarter, N.B., Woods, D.D., 1991. Situation awareness: a critical but ill-definedphenomenon. The International Journal of Aviation Psychology 1 (1), 45e57.

Smith, K., Hancock, P.A., 1995. Situation awareness is adaptive, externally directedconsciousness. Human Factors 37 (1), 137e148.

Stanton, N.A., Ashleigh, M., Roberts, A.D., Xu, F., 2001. Testing Hollnagel’s contextualcontrol model: assessing team behaviour in a human supervisory control task.International Journal of Cognitive Ergonomics 5 (2), 111e123.

Stanton, N.A., Baber, C., Harris, D., 2008. Modelling Command and Control: EventAnalysis of Systemic Teamwork. Ashgate, Aldershot.

Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., Jenkins, D.P., 2005. HumanFactors Methods: A Practical Guide for Engineering and Design. Ashgate,Aldershot.

Stanton, N.A., Salmon, P.M., Walker, G.H., Jenkins, D.P., 2009a. Genotype andphenotype schemata as models of situation awareness in dynamic commandand control teams. International Journal of Industrial Ergonomics 39 (2009),480e489.

Stanton, N.A., Salmon, P.M., Walker, G.H., Jenkins, D.P., 2009b. Genotype andphenotype schema and their role in distributed situation awareness in collab-orative systems. Theoretical Issues in Ergonomics Science 10 (1), 43e68.

Stanton, N.A., Salmon, P.M., Walker, G.H., Jenkins, D.P., 2010. Is situation awarenessall in the mind? Theoretical Issues in Ergonomics Science 11 (1), 29e40.

Stanton, N.A., Stewart, D., Harris, D., Houghton, R.J., Baber, C., McMaster, R.,Salmon, P.M., et al., 2006. Distributed situational awareness in dynamicsystems: theoretical development and application of an ergonomics method-ology. Ergonomics 49, 1288e1311.

Taylor, R.M., 1990. Situational Awareness Rating Technique (SART): The Develop-ment of a Tool for Aircrew Systems Design (No. SEE N90-28972 23-53). AGARD-CP-478. NATO AGARD: International Organization, Neuilly Sur Seine, France, pp.3/1e3/17.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D.P., 2006. Event analysis ofsystemic teamwork (EAST): a novel integration of ergonomics methods toanalyse C4i activity. Ergonomics 49 (12), 1345e1369.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D.P., 2009a. Command andControl: The Sociotechnical Perspective. Ashgate, Hampshire.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D., 2009b. How can we supportthe commander’s involvement in the planning process? An exploratory studyinto remote and co-located command planning. International Journal ofIndustrial Ergonomics 39, 456e464.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D., 2009c. An evolutionaryapproach to network enabled capability. International Journal of IndustrialErgonomics 39, 303e312.

Walker, G.H., Stanton, N.A., Salmon, P., Jenkins, D., Rafferty, L., Bessell, K., 2009d.Beyond NEC: Aerosystems. Human Factors Integration Defence TechnologyCentre, Yeovil.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D.P., 2009e. Command andControl: The Sociotechnical Perspective. Ashgate, Farnham, UK.

Walker, G.H., Stanton, N.A., Salmon, P.M., Jenkins, D.P., Rafferty, L., Ladva, D., 2010.Same or different? Generalising from novices to experts in military commandand control studies. International Journal of Industrial Ergonomics 40 (5),473e483. doi:10.1016/j.ergon.2010.04.003.

Waterson, P., 2009. A systems ergonomics analysis of the Maidstone and TunbridgeWells infection outbreaks. Ergonomics 52 (10), 1196e1205.

Weisstein, E.W., 2008. GraphDiameter. FromMathWorldeAWolframWebResource.http://mathworld.wolfram.com/graphdiameter.html (accessed 24.08.10.).

Wickens, C.D., 2008. Situation awareness: review of Mica Endsley’s 1995 articles onSA theory and measurement. Human Factors 50, 397e403.