Design for pattern recognition in spatial awareness - Eurocontrol

CARE INO III

3D IN 2D PLANAR DISPLAY PROJECT

D2.4 DESIGN FOR PATTERN RECOGNITION IN SPATIAL AWARENESS

(LOT NO. 1, WP 2)

Reference : Edition Effective Date 29/08/08

Authors Organisation Signature

Stephen Gaukrodger Middlesex University

Fan Han Middlesex University

William Wong

Ifan Shepherd

Middlesex University

Middlesex University

D2.4_TEMPORAL DESIGN FOR PATTERN RECOGNITION IN SPATIAL AWARENESS.doc

DOCUMENT CONTROL

Copyright notice

© 2009 European Organisation for the Safety of Air Navigation (EUROCONTROL). All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of EUROCONTROL.

Edition history

Edition Nº

Effective date or status

Author(s) Reason

Acknowledgements

Name Location

Paola Amaldi Middlesex University

Bob Fields Middlesex University

Martin Loomes Middlesex University


Page 2 Effective Date : 06/01/09


Table of Contents

1. INTRODUCTION ................................................................................................. 1 2. WHAT IS EXPERTISE? ...................................................................................... 1

2.1 Context sensitivity..................................................................................................... 3

2.2 Cognitive limitations on human performance ........................................................... 3

2.2.1 Attention ............................................................................................................ 3

2.2.2 Working Memory ............................................................................................... 4

3. EXPERTISE IN ATC............................................................................................ 5 3.1 Situational Awareness .............................................................................................. 5

3.1.1 Levels of Situation Awareness .......................................................................... 6

4. SPATIAL REASONING AS AN ANALOGUE FOR TEMPORAL REASONING. 6 5. COGNITIVE PROCESSES IN EXPERTISE ........................................................ 7

5.1 Perception ................................................................................................................ 7

5.2 Comprehension ........................................................................................................ 9

5.3 Reasoning .............................................................................................................. 10

6. PATTERN RECOGNITION AND ATC .............................................................. 11 6.1 Pattern recognition and conflict detection............................................................... 12

6.2 The Spatial Temporal Design Framework .............................................................. 13

6.3 Analysing Visualisations ......................................................................................... 15

6.3.1 The current co-planar display.......................................................................... 15

6.3.2 AR Stack Manager .......................................................................................... 17

7. DESIGNING WITH PATTERN RECOGNITION IN MIND.................................. 19 8. REFERENCES .................................................................................................. 22




1. INTRODUCTION

Air Traffic Control (ATC) is a task that requires integrating complex spatial and temporal

situations with extensive procedural data to maintain a safe and efficient airspace. As such,

it is important that any future displays for ATC should make this integration as easy as

possible. Controllers are expert performers who can reason about a current situation state,

the way the situation is likely to change, and the actions they should take to ensure that the

future state is in accordance with their goals.

This paper will first explore what an expert is, what limitations experts face, and how these

limitations are circumvented. We will then look at the ways pattern recognition helps experts

in perceiving, understanding, and acting upon the environment.

We will look at how experts in other domains perceive spatial and temporal patterns and how

these patterns are used to build mental models. We will see that patterns are then used to

generate appropriate responses to the environment. This will lead to a review of evidence

regarding the perception and recognition of patterns in ATC, the form these patterns take,

and how they are used.

Finally, we will consider these patterns in terms of the spatial-temporal framework (Rozzi,

Wong, Amaldi, Woodward, & Fields, 2006) and how this framework can be used to inform

visualisation design.

2. WHAT IS EXPERTISE?

Ericcson and Lehmann (1996) define expertise as consistently superior performance on a

set of tasks for a domain.

Expertise appears to come, almost universally, from hard work. Experts normally take about

ten years to reach peak performance (Ericsson, Krampe, & Tesch-Romer, 1993; Hayes,

1981; Simon & Chase, 1973), and only then by constantly striving to improve (Ericsson &

Lehmann, 1996).

As such, there are few domains where expert performance is expected from participants.

Although professional athletes and musicians must demonstrate expert performance in order

to compete with their peers, most normal workers do not compete to the same degree, and

therefore do not need to be experts. Although some may attain expert levels, it is normally

not necessary for satisfactory performance.




While few companies are willing to spend years training their staff before they can even

begin to work, exceptions are often found in safety-critical situations. Fireground

commanders (Klein, Calderwood, & Clinton-Cirocco, 1986), pilots (Durso & Gronlund, 1999;

Endsley, 1989) and air traffic controllers(Durso & Gronlund, 1999) must be experts in order

to ensure safety and to react to unexpected events.

The age at which experts reach peak performance is closely related to the domain. In

vigorous sports, the band for peak performance is very narrow (Schulz & Curnow, 1988). In

cognitive areas, the highest achievements are by experts in their thirties (Lehman, 1953).

Although experts in a field typically attain their highest performance after about 10 years,

once this 10 year point is reached, actual performance is not well correlated with experience.

Measures of hours in chess competitions (Charness, Krampe, & Mayr, 1995) and of baseball

games in the major leagues (Schulz, Musa, Staszewski, & Siegler, 1994) only weakly

correlate with individual differences in player performance among skilled performers.

This weakness appears to be caused by the need for deliberate practice. Actual situations

provide relatively few opportunities for effective learning. Deliberate practice refers to

individual training activities designed to improve specific aspects of an individual’s

performance through repetition and successive refinement (Ericsson et al, 1993). To be

effective, this practice must be monitored with full concentration. This is effortful, and limits

the duration of daily training. Ericsson et al (1993) found a strong correlation between

deliberate practice and individual performance.

Salthouse (1991) argued that expertise in cognitive tasks consists of circumventing human

processing limitations. These limitations, working memory (Baddeley & Hitch, 1974; Miller,

1956) and attention (Ashcraft, 2006) in particular, may be improved by training (Olesen,

Westerberg, & Klingberg, 2004) but remain as major bottlenecks in cognition (Endsley,

1997).

It is possible to reduce the requirement for ten years’ experience by reducing the size of the

domain. Ericsson and Harris (1990) and Ericsson and Oliver (1989) showed that an

individual with little or no prior knowledge of chess could learn to recall briefly presented

chess positions at a level matching that of chess masters after only 50 hours of practice at

the task. Interestingly, the methods differed significantly – the novices focused on

perceptually salient pawns, while the masters recalled the critical pieces in the centre of the

chess board.




2.1 Context sensitivity

Unfortunately, expertise does not tend to transfer to other areas well (Eisenstadt & Kareev,

1979; Ericsson & Lehmann, 1996; Marchant, Robinson, Anderson, & Schadewald, 1991).

Expert brain surgeons do not perform better on ATC tasks than novice surgeons. The

reason is that each type of expertise requires phenomenal amounts of domain specific data,

gained through practice and experience, before it can be of use to a practitioner.

Even when domains are apparently similar, there is surprisingly little transfer. GO and

Gomoko are played on the same board and use the same pieces, but expert GO players

were no better at remembering briefly presented Gomoko displays than were novices.

Likewise, Gomoko experts were no better with their memory for GO displays (Eisenstadt &

Kareev, 1979). Experts in chemistry are no better at political science than novices (Voss,

Greene, Post, & Penner, 1983; Voss, Tyler, & Yengo, 1983).

Endsley (2006) points out that expertise in maintaining Situation Awareness (see below)

might not transfer to expertise in other domains. In fact, Endsley argues that expertise may

not transfer to novel situations within an expert’s own domain. This risk should be seriously

considered when designing tools and setting regulations that will alter the situation of which

an expert is expected to be aware.

2.2 Cognitive limitations on human performance

The main cognitive limitations, or bottlenecks, in human performance are attention and

working memory. Attentional limits determine how quickly we can acquire information, while

working memory limits the amount of information we can work with at one time (Miller, 1956).

2.2.1 Attention

Humans are finite beings that cannot attend to all things at once. Attention is used to focus

our mental capacities on selections of the sensory input so that the mind can successfully

process the stimulus of interest (Duchowski, 2000). Several theories of attention have

focused on where and what we pay attention to, but all agree that attention is a limited

resource (Duchowski, 2000; Wolfe, 2000; Yantis, 1993).

To circumvent this bottleneck, experts learn to direct their attention effectively. This direction

usually means alternating between top-down, goal-directed attention, and bottom-up, data-

driven processes (Endsley, 1997). In a data-driven condition, operators will perceive a

piece of information, interpret what it means then project what will happen next. This is

effective for building a mental model, but is often inefficient. By forming and working towards




goals based on current understanding or projections, individuals will look for data to confirm

or deny their assessments. The relationship between data-driven and goal-driven attention

is iterative. The data-driven search helps determine goals, while goals inform the search for

new data.

Whether goal or data driven, experts learn about where to look for information. Endsley

(2006) states that controllers learn to scan their radar maps efficiently, based on traffic

patterns in their sector. During a simulation, we observed an instructor tell a student that

giving direct routings was risky, as it shifted potential conflicts away from established

waypoints, disrupting established visual search patterns.

Although efficient visual search and goal-direction ensure that the attention bottleneck is

best used, an even better idea is to circumvent the bottleneck entirely. By automating

processes, attention can be diverted to novel situations that require more detailed

examination (Ashcraft, 2006; Posner & Snyder, 1975; Schneider & Shiffren, 1977; Shiffren &

Schneider, 1977).

Normally, automatic processing refers to some perceptual-motor task. The canonical

example is riding a bike. Peddling, balancing and steering originally required considerable

effort and constant access to perceptual and cognitive resources. With sufficient practice,

the task of riding the bike becomes automated and resources are freed up for other tasks –

talking, sight-seeing or daydreaming.

Automatic processes appear to be both faster and capable of running in parallel, producing

enormous efficiencies for practitioners. Secrist & Hartman (1993) trained participants to

recognise briefly presented, masked stimuli. Classification and detection accuracy were

comparable, suggesting that the trained subjects can classify a subject in the time it takes to

detect it.

In terms of expertise, this means that automatic processes largely avoid cognitive

bottlenecks, meaning that automating a process frees up resources for other, non-

automated tasks (Schneider & Shiffren, 1977; Shiffren & Schneider, 1977).

2.2.2 Working Memory

Working memory describes the volatile memory used for short-term processing. Baddeley

and Hitch (1974) argued that working memory consists of a central executive managing

resources that include a “visuo-spatial sketchpad,” and a “phonological loop.”




The two processes appear to work in parallel, with little interference caused by loading each

area simultaneously, but with considerable disruption when the one process is overloaded

(Baddeley & Hitch, 1974; Brooks, 1968; Logie, Zucco, & Baddeley, 1990).

Working memory is limited, and this limit appears fixed (Rouder et al., 2008). In general,

people can only remember as many digits as they can say in about 2s (Baddeley,

Thompson, & Buchanan, 1975). Experts can remember more items, but they do this by

using strategies, such as chunking (Miller, 1956). Chunking consists of combining multiple

items into a single chunk, and then remembering the chunk as a single item in short term

memory. This chunking is a typical example of a circumvention strategy. Experts use long

term memory to replace some of the functions of short term memory (Ericsson & Kintsch,

1995; Ward, A. Mark Williams, & Hancock, 2006). Numerous patterns are stored in long

term memory and these patterns are used to chunk stimuli as large, meaningful units. This

tactic is knows as Long Term Working Memory (LTWM) (Jodlowski & Doane, 2003)..

3. EXPERTISE IN ATC

If expertise is “consistently superior performance on a set of tasks for a domain” (Ericsson &

Lehmann, 1996), what examples of superior performance are found in the domain of ATC?

Expert controllers consistently produce safer and more efficient airspaces than novices.

They do this by scanning the airspace more efficiently; using this scanning process to build

and maintain a mental model that contains more relevant information; and using this mental

model to make decisions that are faster and better than a novice could make.

This expertise is closely related to Situation Awareness (SA) (Endsley, 2006)

3.1 Situational Awareness

Situation Awareness (SA) is a somewhat nebulous concept, coined by aviators and

controllers to describe their awareness of the world around them. It has since been

extended to other practitioners in domains where information is changing constantly and

decisions must be made in a timely manner. While these domain operators universally

understood the meaning, actually defining the term precisely has proven more difficult.

Endesly (1988) defines SA as

The perception of the elements in the environment within a volume of time and

space, the comprehension of their meaning and the projection of their status in the

near future




While SA may be difficult to define, it is extremely important. A review of National

Transportation Safety Board aircraft accidents over a four year period found that 88% of

accidents attributed to human error involved SA as a major causal factor (Endsley, 1995,

1997).

3.1.1 Levels of Situation Awareness

Endsley (1997) identifies three levels of SA; perception, comprehension and projection.

Perception, Level 1 SA, is used to acquire information. Without proper perception, the odds

of an error increase dramatically - Jones & Endsley (1996) found that 76% of SA errors in

pilots could be traced to errors in perception. The main bottleneck in perception is attention

– controllers are limited in the amount of information they can attend to at any moment. As

described above, experts alternate between goal-directed and data-directed attention, in

order to optimise their Level 1 SA (Endsley, 1997).

Comprehension, Level 2 SA, is an understanding of the perceptions from Level 1. It

encompasses how people combine, interpret, store and integrate multiple pieces of

information to arrive at an understanding of the situation and how it relates to their goals.

Endesly (2006) likens Level 2 SA to reading comprehension, where Level 1 SA means the

ability to read words.

Projection, Level 3 SA, is the ability to forecast future situation events and dynamics, and is

the highest degree of SA. If Level 1 SA is reading words, and Level 2 is reading

comprehension, then Level 3 is the ability to guess how the book will end. The ability to

project into the future is essential for evaluating courses of action – it is impossible to

evaluate the effects of something when you are unable to predict what those effects will be.

4. SPATIAL REASONING AS AN ANALOGUE FOR TEMPORAL REASONING

Researching time is harder than researching space. Interactive spatial displays can be

made by moving paper about, while interactive temporal displays normally require software

of some description. Fortunately, recent research has suggested that temporal reasoning

and spatial reasoning share similar mechanisms.

Schaeken et al (1996) showed that propositional temporal reasoning used the same types of

mental models as propositional spatial reasoning. Other research (Gentner, 2001) has

shown that we regularly use spatial metaphors when talking about temporal dimensions.

The way we use these metaphors in language, and the speed with which we comprehend




them, was taken to show that the spatial and temporal reasoning systems were at least

similar, and probably shared a common root in human thought.

Due to the dearth of research on temporal reasoning, it will be assumed that the lessons on

spatial reasoning can be cautiously applied to the temporal domain.

5. COGNITIVE PROCESSES IN EXPERTISE

The levels of SA provide a good framework for the process of spatial-temporal reasoning in

ATC. Controllers first acquire information, integrate it in to a mental model, then use the

model to reason about courses of action. This process is also found in a number of other

domains that involve spatial and temporal reasoning, and it is hoped that research from

these other areas may help us to understand how the process functions in ATC. Our focus

will be on how experts in these domains circumvent attentional and working memory

limitations.

5.1 Perception

When acquiring information, the primary limitation is attention. Experts circumvent the

attention bottleneck using automatic processes that appear to be heavily dependent on

pattern recognition. In a spatial display with numerous interacting objects it is time

consuming in the extreme to consider the relationship between every possible subset of

objects. For instance, if there are ten objects, then there are 45 possible pairs. Clearly,

efficient processes do not take every possible pairing into account.

Obvious as it may be, when we look at a graph, we do not consider the relationship between

every point. This is clearly demonstrated by considering how much easier it is to

comprehend a graph than a table. When perceiving a graph, the first step is pattern

recognition (Shah & Carpenter, 1995). Patterns recognised include whether there is a

straight or jagged line, if there are multiple lines, and whether lines are parallel, converging,

or intersecting. In all of these cases, the patterns are both visually salient and conceptually

meaningful. Converging lines may mean differing effects of an independent variable while

parallel lines mean a congruent effect. The degree of jaggedness in a line may indicate

measurement uncertainty while the underlying trend is simultaneously visible.

Perhaps the most critical lesson to be learned from pattern recognition in graphs is that it

can be flawed. Cleveland and McGill (1984) provided an example of an optical illusion that

leads to misinterpretation of a 2D graph (Figure 1). The illusion is so powerful that even with

foreknowledge it is difficult, if not impossible, to avoid.




Although it appears that the number of cats and dogs is converging, the difference is, in fact,

constant. Note that this effect disappears when the lines joining the points is removed. This

is because, when considering lines, people tend to judge the shortest distance between the

lines. By removing the lines, the focus is on the distance between the points.

This misinterpretation is extremely relevant to future ATC designs. Consider a free flight

scenario where aircraft trajectories are displayed as lines. If aircraft were to follow

trajectories of the types shown in figure 1, it would appear that they were converging, when

in fact their separation remains constant.

Figure 1 (a) Fictitious graph showing two curves that appear to converge. (b) The same data, presented without lines. By removing the lines, we can see that the

apparent convergence in (a) was illusory (Cleveland & McGill, 1984).

Expert chess players show a number of perceptual differences from novices. Most

importantly for ATC, chess experts use a larger field of view. Note that this does not mean

that experts search a larger area with serial eye movements – they actually see a larger area

with a single viewing. This allows them to extract information at a deeper structural level. At

the same time, experts process in parallel, where novices process in serial. This appears to

be a use of automaticity to avoid the attentional bottleneck. The parallel processing also

seems to depend on pattern recognition, with the experts processing important relationships

between pieces, rather than the pieces themselves.

Perception is not solely a bottom-up, data-driven process. As mentioned above, perception

can also be goal-driven and top-down. Schemas and mental models, described in the next

section, inform the operator about salient aspects of the situation and help to focus attention.

These schemas and mental models will also be shown to rely heavily on pattern recognition,

meaning that pattern recognition is important whether the scanning strategy is bottom-up or

top-down.




5.2 Comprehension

The next step in the process of reasoning is comprehension. A person wishing to reason

about a complex display must understand what their perceptions mean in terms of the

objects, constraints, and relationships depicted. According to SA theory, this means

integration into a mental model.

Mental models form the basis of spatial and temporal reasoning (Johnson-Laird, 1989;

Schaeken, Johnson-Laird, & d'Ydewalle, 1996). In order to make projections about the

future of the airspace, controllers must create a mental model of that airspace, and then

make inferences. They are best described as “mechanisms whereby humans are able to

generate descriptions of system purpose and form, explanations of system functioning and

observed system states, and predictions of future states” (Rouse & Morris, 1985).

Endsley (2006) lists the following advantages of good mental models:

• Knowledge of those aspects of the system relevant to the current situation. This is

critical for attending to and classifying information in the perceptual process. This

makes the process more efficient, particularly when there is a large amount of

information to process.

• Easier integration of elements to form an understanding of the situation. Without a

mental model it is difficult to understand the significance of something you have

perceived. Understanding is often a function of integrating multiple pieces of data,

and a good mental model provides a way of identifying and storing those data.

• Easier projection of future system states. The future is a function of the both the

current state and the rules and events within a system. In some cases, projections

are simply rule based – if X and Y are true, then Z will occur. In other cases the

system is more complex, allowing for a number of different evolutions. Accurate

projection is a function of the operators’ ability to judge the probabilities of each

outcome.

Over time, operators build up a library of prototypical situations, called schemas. These

schemas allow pattern matching between the current state of the model and the library of

schemas stored in long term memory. The patterns perceived need not be a direct match,

but rather need only be similar in critical cues to provide a near match.

Since the size of the mental model is limited by working memory, experts use pattern

recognition and schema to perform a procedure similar to chunking. The first example of

this was found in chess, where experts could remember briefly presented game positions far




more accurately than novices (Simon & Chase, 1973). This advantage disappeared when

the board positions were not possible in chess. Since then, memory for schema-typical

presentations has become a standard method for determining the level of expertise in many

areas. In all these cases, pattern recognition is regarded as the mechanism underpinning

the use of schemas.

5.3 Reasoning

In many cases, pattern recognition can greatly improve the efficiency of reasoning

processes. Instead of searching through an option space, experts can use the schemas that

comprise their mental model to generate appropriate solutions. If the first solution generated

is satisfactory then no further solutions need to be considered. This technique follows the

Recognition Primed Decision model (RPD), a form of Naturalistic Decision Making (Klein,

1989, 1993a, 1993b, 1997; Klein, Calderwood, & Clinton-Cirocco, 1986).

Fireground commanders also claim that they make few decisions – the procedure to be

taken follows naturally from the situation perceived (Klein, Calderwood, & Clinton-Cirocco,

1986). Instead of comparing the merits of numerous courses of action, they aimed to find a

solution that would work satisfactorily. This strategy, called satisficing (Simon, 1957) is a

common technique in situations where the costs of optimisation outweigh the benefits.

Klein et al (1986) found that commanders follow one of two paths – Simple Match or

Evaluate a Course of Action. If the situation is a Simple Match to a known situation, then a

stored procedure can be implemented. If the situation is more complex, the commander

considers a course of action, and evaluates probable outcomes. If the evaluation suggests

that the course of action is satisfactory then no other options are considered. If the course of

action is unsatisfactory, then another is considered. A Diagnosis step was added later, in

which operators attempt to find causal reasons for unexpected system behaviour (Kaempf,

Wolf, Thordsen, & Klein, 1992; Klein, 1993b; Klein, Orasanu, Calderwood, & Zsambok,

1993).

Kaempf et al (1992) estimated that in 78 instances of decision making in operational Navy

warfare incidents, 78% of the time, the operator adopted a course of action without any

deliberate evaluation. In a further 18% of cases, the evaluation was accomplished via

mental simulation. In only 4% of cases was any comparison of alternative courses made.

Likewise, Randel et al (1994) found that 93% of decisions by electronic warfare technicians

were made by serially evaluating potential courses of action, rather than by comparing

courses against each other.




Where correct perception leads automatically to correct action, incorrect perception can lead

to incorrect action. According to Endsley (1997):

Many human errors that are attributed to poor decision making

actually involve problems with the SA portion of the decision making

process as opposed to the choice portion of the process. Decision

makers make the correct decision for their perception, but that

perception is in error. This represents a fundamentally different

category of problem than a decision error in which the correct

situation is comprehended but a poor decision is made as to the

best course of action.

p 270.

A recent addition to the RPD model is Diagnosis. When an operator is uncertain about the

nature of a situation, they attempt to find an explanation for the situation, so that the situation

can be remedied. The RPD model (Klein, 1997; Klein, Orasanu, Calderwood, & Zsambok,

1993) suggests that decision makers will often spend more time and energy determining

what is happening and distinguishing between different explanations, than comparing

different courses of action. There are two major techniques of diagnosis. The first is feature

matching, in which “symptoms” are used to categorise a situation. The second is story

building, in which operators build a mental simulation to produce a causal explanation –

essentially, they ask “why would he do that?” The story they build from this question is often

a key part of the decision making process.

6. PATTERN RECOGNITION AND ATC

While many of the observations made of other domains also apply to ATC, there are also

specific observations that can be made regarding the work of controllers.

When perceiving the airspace, controllers’ visual search is like the experts in chess. Novice

controllers make many, closely spaced, fixations and hold them for long periods of time.

Conversely, expert controllers make few, well spaced, fixations and hold them for briefer

periods (Lim et al., 2006).

Most importantly, controllers do use schema. These are based around events (Seamster et

al (1993). In essence, the airspace is perceived as spatial-temporal events. Events are

schema typical arrangements, generated from patterns of objects. Controllers use these

events to hold the important information about several aircraft at once. This important

information is context dependant, but, for instance, it may be something like: D2.4_TEMPORAL DESIGN FOR PATTERN RECOGNITION IN SPATIAL AWARENESS.doc



“These two aircraft are following the same route, but the rear

aircraft is going to catch the front aircraft in two minutes, at

this point here.”

The event is a “catching and overtaking event” which only requires the controller to

remember:

• the aircraft involved • the place the event will occur • the time the event will occur

This is known as gist storage, and fits perfectly with schema theory mentioned above. The

schematic events act to:

• Make searching for information relevant to events easier, since the controllers can narrow their search to schema relevant information.

• Reduce the amount of information controllers must store in their mental models, since the controller can use LTWM instead of the visuo-spatial sketchpad.

• Make selection of responses easier since, according to RPD, correctly classifying an event should lead to generation of an appropriate procedure.

When reasoning, controllers use RPD (Seamster, Redding, Cannon, Ryder, & Purcell,

1993). The event schemas act as triggers for procedures. Triggers take the form of logical

conditionals, in which a series of preconditions lead to a conclusion. For instance “If an

aircraft is less than 30 miles from the sector boundary AND no other aircraft are around

THEN hand off the aircraft.” These triggers appear to be closely linked to the events

controllers use to chunk the airspace.

6.1 Pattern recognition and conflict detection

The statements above show how patterns can help with guiding scanning and with

comprehending and acting once scanning is complete, but how exactly do controllers

determine whether there is a conflict or not? Xu and Rantanen (2003) and Stankovic et al

(2006) deal in detail with theories of how controllers detect and classify impending conflicts.

They provide several different calculation methods for determining whether a conflict is likely

to occur. In both cases, the experiments dealt solely with situations in which only two aircraft

were present. Even in this limited situation, exact judgment requires considerable effort. It

would not be a simple task to perform this search in an airspace containing ten aircraft.

Loft et al (2007) apply pattern recognition to this task. Instead of using patterns to identify

schemas, they look for specific instances in memory. Instances differ from schemas in that

they contain specific episodic information, not merely relative information. Logan (1988)




developed a version of instance theory to explain skill acquisition. Just as operators build

libraries of schemas, they also build libraries of instances. When considering a response (in

the case of ATC, the response would be “deciding there is a conflict”), there is a race

between the algorithmic process of deciding and the pattern based retrieval from memory.

As the size of the instance library increases, the probability that retrieval will win the race

increases. Eventually, as the size of the library increases, the algorithm will be abandoned

and performance will be based entirely on memory. Like RPD, experts do not seem to

reason on-line. Instead, they recognise patterns of stimuli and recall the result automatically.

Instance theory need not conflict with schema theory. Saito (2000) shows how the two

theories can be reconciled to produce a model with far greater explanatory power.

Loft et al (2007) focused on determining concrete features that would identify an instance.

They suggested that participants used information about both static and dynamic features,

specifically:

• The speed of the aircraft

• The angle of the intersection

• The relative positions of the aircraft

This is one of the first attempts to apply instance theory to a dynamic domain. Previous

uses have worked around instances on static tasks like alphabetic arithmetic and perceptual

categorisation. In these cases, instances occur at a specific point in time. A conflict may

develop over a period of two minutes, and detection should be possible at any point. The

challenge for proponents of instance theory, and for design of displays to facilitate instance

recognition, is to determine exactly how a dynamic event can be described as an instance.

6.2 The Spatial Temporal Design Framework

Rozzi et al (2006) provide a framework for considering the way people reason about spatial-

temporal processes which may fill this need. The spatial-temporal framework provides a

way to describe human perception and modelling of complex, multi-dimensional processes

(Figure 2).




Figure 2 The spatial-temporal framework (Rozzi, Wong, Amaldi, Woodward, & Fields, 2006)

According to this framework purely spatial entity can be described in terms of its constituent

parts, the relationships between those parts and the rules that govern the parts and

relationships. When a spatial entity changes over time, the history and potential futures of

the object become more important. The intentions of agents interacting with the entity must

also be considered.

The cognitive process of crossing the road provides a concrete example of the importance of

intentionality. There are numerous spatial rules that appear to provide information about

whether it is safe to cross a road with oncoming traffic (Owen, Simpson, & Murray, 2003). If,

however, you are waiting to cross at a pedestrian crossing, the spatial dimensions of an

approaching vehicle are less important than the intent of the driver. Will the car stop to let

you cross? The spatial dimensions are still important, but now they are used to judge

intention, rather than to directly measure safety.

When the pedestrian judges these spatial relations to determine intention, they are

evaluating behaviour. Is the driver slowing down? Behaviour is reflected in spatial changes

over time.

In the context of this paper, the process is the event – a conflict, a descent, a hand-off – and

controllers use schemas or instances in memory to improve their use of attention and




working memory. The spatial-temporal framework gives one way that a schema or instance

might be structured.

Recall that Loft et al (2007) showed that the perception of conflicts relied on noticing the

speeds, relative angles and relative positions of aircraft. In this case, the process would be

the conflict event.

Categorising event types in terms of the framework will rely on precise, detailed

understanding of each task.

6.3 Analysing Visualisations

A good visualisation will make the elements of relevant schemas or instances salient. For

instance, a display that helped road crossing would emphasise information that indicated

stopping intention; is the car slowing down and is that rate of slow-down is consistent with

stopping?

Without further research, we can only speculate on the elements that comprise instances

and schemas in modern control work. Even with further research, we will only ever be able

to speculate about the elements that comprise instances and schemas in future control work.

Thus, any attempt to categorise current displays in terms of fitness for specific current or

future tasks would, largely, be speculation.

With the spatial-temporal framework, we can categorise different element types, and talk

about how well a visualisation provides them. For example, some displays provide good

information about spatial dimensions and little temporal information. Once future research

determines the elements of the schemas and instances in a task, we can then use the

framework to suggest visualisations that provide these elements.

In this section, we will examine some specific visualisations in the context of the spatial-

temporal framework and the elements that each display makes salient. These elements are,

in most cases, a function of the way the display is embodied. They are rarely an inherent

aspect of the visualisation technique.

6.3.1 The current co-planar display

Current displays are two dimensional, showing aircraft positioned on the screen according to

their latitude and longitude (Figure 3). Although the displays are highly configurable,

controllers commonly choose to display line that indicates heading and speed and a series

of trailing dots that show the recent history of the aircraft. Some controllers only use the




dots, since they provide sufficient information to infer the heading and speed, rendering the

line redundant.

Additional information – such as call-signs, altitude, and aircraft type – is displayed on a

label that is attached to the aircraft.

Controllers may also choose to show navigation points and common flight paths on the

display, although we have been informed that most controllers know their sectors well

enough that this information is not necessary.

Figure 3 A section of a current coplanar display

In terms of the spatial-temporal framework, the salient elements of the visualisation are

(Figure 4):

• The horizontal positions, velocities and headings of the aircraft (Object information)

• The positions of navigation points (Object information)

• The horizontal distances and closing rates of aircraft from each other and from

navigation points (Relationship information)

• The recent history of the aircraft (Past and present temporal information)

• Probable flight paths (Projection and intent information)

Vertical relationships, actual intent, and the constraints imposed by aircraft and the

environment must be remembered, deduced or inferred.




Figure 4 The salient aspects of current coplanar displays, in the context of the spatial-temporal framework.

6.3.2 AR Stack Manager

This visualisation shows aircraft in a holding stack. Holding stacks are procedural structures

that allow aircraft to wait safely. Holding aircraft follow an oval “racetrack” pattern at a

constant altitude. Controllers release waiting aircraft from the bottom of the stack and insert

new aircraft at the top. When an aircraft at the bottom is released the aircraft above

descend to fill the vacant space.

The critical tasks for controllers are releasing aircraft, inserting aircraft, and maintaining a

minimum safe vertical separation.

In this visualisation, the aircraft are displayed in 3D, 2D and 1D. The 3D visualisation shows

the position and heading of the aircraft in 3D space. The 2D view on the wall shows the

vertical position and one horizontal value, and the 1D view – the ruler – shows only the

vertical position. The lines on the wall and the ruler show the height the aircraft has been

assigned (For a detailed description of the stack manager see (Wong et al., 2008)




Figure 5 The AR Stack manager

In terms of the spatial-temporal framework the salient elements are (Figure 6):

• The 3D, 2D, and 1D positions and headings of the aircraft (Object information)

• The relative distances, particularly the relative vertical distances, of the aircraft

(Relationship information)

• The assigned altitude of the aircraft (Intent information)

This visualisation is only a prototype, and provides an excellent example of how the spatial-

temporal framework could be useful for interface analysis. If it were determined that

controllers did not need 3D and 2D information about aircraft, then we could remove the

components that display this information, leaving only the ruler. If it were found that

controllers need information about the recent past, then we could add the same history dots

as are found in the co-planar display.




Figure 6 The salient aspects of the stack manager in the context of the spatial-temporal framework.

7. DESIGNING WITH PATTERN RECOGNITION IN MIND

Pattern recognition is an integral component of the cognitive processes involved in cognitive

fields, and ATC in particular. Pattern recognition guides information search by using

schemas to suggest which information will be relevant. It increases the size and relevance

of mental models by building them from schemas stored in long term working memory. It

helps in decision making by allowing recognition primed decision making; potential

responses are generated and evaluated serially, rather than compared against each other.

Controllers do not scan the entire airspace constantly. Instead, they use schemas in LTWM

to suggest where to search. They look for larger patterns, using a larger field of view than

novices. These two uses of patterns make the best use of their limited attention resources.

A good design will make sure that schemas in memory continue to facilitate this process.

When scanning the airspace, controllers use patterns to access a stored library of instances.

These instances are used to judge the risk of conflicts, circumventing the need for complex

computation. It is likely that other, apparently complex tasks, use instance based methods

as well. If we are to design for expert performance, we need to ensure that it is easy for

controllers to learn these instances and that visualisations support their use. The spatial-




temporal framework provides a way to discuss the structure of instances, and a method to

compare designs in terms of the elements they make salient. Good design will be a

compromise between providing necessary information and avoiding clutter.

Controllers do not rely completely on the environment, but instead use mental models as an

essential step to SA. The mental models avoid the limitations of working memory by using

schema stored in long term memory to structure the model. This allows them to store more

information, and ensures that the information stored is more relevant.

The schemas in ATC are based on events, which are spatial-temporal entities. Events are

relationships between objects in the environment that change over time in some significant

way. The spatial-temporal framework provides a way to discuss and classify these events,

and helps us think about ways in which visualisations can aid recognition of events in a

control environment.

Finally, the decision making process in an expert controller is very different from the normal

meaning of the word. Controllers do not consider multiple alternatives and choose the best.

Instead, they generate possible courses of action, determine whether they are satisfactory,

and then either carry out that course of action or dismiss it and look at another alternative.

The processes that underlie this process, RPD, are based on schemas and pattern

recognition. For controllers to make good decisions quickly, they will need to be able to use

schemas and patterns to recognise the appropriate courses of action.

The effects of pattern recognition are summed up in table 1.

Task The contribution of pattern recognition

Scanning Schema-based patterns guide controllers in performing efficient visual search of the airspace

Detecting Allows controllers to replace algorithmic calculation with instance-based recall

Modelling Increases the size and relevance of the model by using LTWM to represent the airspace as a collection of event schemas

Response Selection Allows controllers to use RPD instead of reasoning about appropriate responses

Table 1 The contributions of pattern recognition to expert performance. When designing for experts, it is important to ensure that the designs facilitate recognition

of appropriate patterns

The spatial-temporal framework is a powerful tool for describing the elements of patterns,

and for classifying visualisations in terms of the elements that contribute to those patterns.




When designing a display to suit pattern recognition by experts, we can follow these steps:

Figure 7 A procedure for developing designs using pattern recognition and the spatial-temporal framework.

Controllers have numerous tasks, and these tasks will rely on different spatial-temporal

elements. If all elements for all tasks are displayed, we are likely to find the final result is an

extremely cluttered display. Designing a display to support all of these tasks will probably

require compromises, involving inferring some elements, and allowing controllers to filter the

displayed elements depending on the task they are performing at the time.

The analysis of pattern recognition provides a base for thinking about how controllers’

expertise allows them to circumvent cognitive limitations. The spatial-temporal framework is

a tool for describing these patterns and relating them to the design of displays. Proper

consideration of patterns and their spatial-temporal elements will allow us to design displays

that allow experts to circumvent cognitive limitations more effectively.

Future research will need to look at the spatial-temporal elements that comprise schemas

and instances, and consider how visualisations can best make these elements available to

controllers.




8. REFERENCES

Ashcraft, M. H. (2006). Cognition (4 ed.). Upper Saddle River, NJ: Pearson Education International.

Baddeley, A. D., & Hitch, G. (1974). Working memory. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47-89). New York: Academic Press.

Baddeley, A. D., Thompson, N., & Buchanan, M. (1975). Word length and the structure of short-term memory. Journal of Verbal Learning and Verbal Behaviour, 14, 575-589.

Brooks, L. R. (1968). Spatial and verbal components of the act of recall. Canadian Journal of Experimental Psychology, 22, 349-368.

Charness, N., Krampe, R. T., & Mayr, U. (1995). The importance of coaching in entrepreneurial skill domains: and international comparison of life-span chess skill acquisition. Paper presented at the Conference on the Acquisition of Expert Performance, Wakulla Springs, FL.

Cleveland, W. S., & McGill, R. (1984). Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American Statistical Association, 79(387), 531-554.

Duchowski, A. (2000). Eye-Based Interaction in Graphical Systems: Theory & Practice. Paper presented at the SIGGRAPH 2000, New Orleans.

Durso, F. T., & Gronlund, S. D. (1999). Situation Awareness. In F. T. Durso, R. S. Nickerson, R. W. Schvaneveldt, S. T. Dumais, D. S. Lindsay & M. T. H. Chi (Eds.), Handbook of Applied Cognition (pp. 281-314): John Wiley & Sons.

Eisenstadt, M., & Kareev, Y. (1979). Aspects of human problem solving: The use of internal representations. In D. A. Norman & D. E. Rumelhart (Eds.), Exploration in cognition. San Francisco: W. H. Freeman.

Endsley, M. (1988). Design and evaluation for situation awareness enhancement. Paper presented at the 32nd Annual Meeting of the Human Factors Society, 1988.

Endsley, M. (1989). Pilot situation awareness: The challenge for the training community. Paper presented at the Interservice/Industry Training Systems Conference (I/ITSC), Ft Worth, TZ.

Endsley, M. (1995). A taxonomy of situation awareness errors. In R. Fuller, N. Johnson & N. McDonald (Eds.), Human Factors in Aviation Operation: Proceedings of the Western European Association of Aviation 21st Conference. Dublin, Ireland: Avebury Aviation.

Endsley, M. (1997). The role of situation awareness in naturalistic decision making. In G. K. Caroline E. Zsambok (Ed.), Naturalistic Decision Making (pp. 269-283). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Endsley, M. (2006). Expertise and Situation Awareness. In K. A. Ericsson, N. Charness, P. J. Feltovich & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 633-651). New York: Cambridge University Press.

Ericsson, K. A., & Kintsch, W. (1995). Long term working memory. Psychological Review, 102, 211-245.

Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406.

Ericsson, K. A., & Lehmann, A. C. (1996). Expert and Exceptional Performance: Evidence of Maximal Adaptation to Task Constraints. Annual Review of Psychology, 47, 273-305.

EUROCONTROL. (2006). Medium-Term Conflict Detection. from http://www.eurocontrol.int/muac/public/standard_page/AUPNTmediumTermConfDet.html

Gentner, D. (2001). Spatial Metaphors in Temporal Reasoning. In M. Gattis (Ed.), Spatial Schemas and Abstract Thought. Boston, MA: Massachusetts Institute of Technology.

Hayes, J. R. (1981). The complete problem solver. Philadelphia, PA: Franklin Institute Press.



http://www.eurocontrol.int/muac/public/standard_page/AUPNTmediumTermConfDet.html

http://www.eurocontrol.int/muac/public/standard_page/AUPNTmediumTermConfDet.html


Jodlowski, M. T., & Doane, S. M. (2003, July 31-August 2). Event reasoning as a function of working memory capacity and long term working memory skill. Paper presented at the 25th Annual Meeting of the Cognitive Science Society, Boston, MA.

Johnson-Laird, P. N. (1989). Mental models. In M. I. Posner (Ed.), Foundations of Cognitive Science (pp. 469-499). Cambridge, MA: Lawrence Erlbaum Associates.

Jones, D. G., & Endsley, M. (1996). Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine, 67(6), 507-512.

Kaempf, G. L., Wolf, S., Thordsen, M. L., & Klein, G. (1992). Decision making in the AEGIS Combat Information Center. Task 1. Report prepared for the Naval Command, Control and Ocean Surveillance Center, San Diego, CA. (Draft Technical Report). Fairborn, OH: Klein Associates Inc.

Klein, G. A. (1989). Recognition-Primed Decisions. In W. B. Rouse (Ed.), Advances in Man-Machine Systems Research (pp. 47-92). Greenwich, CT: JAI Press Inc.

Klein, G. A. (1993a). Naturalistic Decision Making: Implications for Design (No. SOAR 93-1): Crew Systems Ergonomics Information Analysis Centre, Wright-Patterson AFB, USAF, OH.

Klein, G. A. (1993b). A Recognition-Primed Decision (RPD) Model of Rapid Decision Making. In G. A. Klein, J. Orasanu, R. Calderwood & C. E. Zsambok (Eds.), Decision Making in Action: Models and Methods. Norwood, NJ: Ablen Publishing Corp.

Klein, G. A. (1997). The Recognition-Primed Decision (RPD) Model: Looking back, looking forward. In G. K. Caroline E. Zsambok (Ed.), Naturalistic Decision Making (pp. 285-303). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid Decision Making on the Fireground. Paper presented at the Human Factors and Ergonomics Society 30th Annual Meeting.

Klein, G. A., Orasanu, J., Calderwood, R., & Zsambok, C. E. (Eds.). (1993). Decision Making in Action: Models and Methods. Norwood, NJ: Ablex Publishing Corp.

Lehman, H. (1953). Age and achievement. London: Oxford University Press. Lim, S. J., Shin, J. S., Jang, J. Y., Yoon, Y. S., Park, H. J., & Jung, H. C. (2006). Eye-

tracking and protocol analysis of expertise in air traffic control situation awareness. Paper presented at the 5th International Conference of the Cognitive Science, British Columbia, Canada.

Loft, S., Neal, A., & Humphreys, M. S. (2007). The development of a general associative learning account of skill acquisition in a relative arrival-time judgment task. Journal of Experimental Psychology: Human Perception and Performance, 33(4), 938-959.

Logan, G. D. (1988). Toward and instance theory of automatization. Psychological Review, 109, 376-400.

Logie, R. H., Zucco, G., & Baddeley, A. D. (1990). Interference visual short-term memory. Acta Psychologia, 75, 55-74.

Marchant, G., Robinson, J., Anderson, U., & Schadewald, M. (1991). Analogical transfer and expertise in legal reasoning. Organisational Behaviour and Human Decision Making, 48, 272-290.

Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63, 81-97.

Olesen, P., Westerberg, H., & Klingberg, T. (2004). Training of Working Memory in Children with ADHD. Journal of Clinical and Experimental Neuropsychology, 24(6), 781-791.

Owen, D. H., Simpson, G., & Murray, S. (2003). Jaywalking in virtual reality: optical information affecting the margin of safety. Paper presented at the International Conference of Perception and Action 12th Conference, Gold Coast, Australia.

Posner, M. I., & Snyder, C. R. R. (1975). Facilitation and inhibition in the processing of signals. In P. M. A. Rabbit & S. Dornic (Eds.), Attention and performance V (pp. 669-682). New York: Academic Press.




Randel, J. M., Pugh, H. L., Reed, S. K., Schuler, J. W., & Wyman, B. (1994). Methods for analyzing cognitive skills for a technical task. San Diego, CA: Navy Personnel Research and Development Center.

Rouder, J. N., Morey, R. D., Cowan, N., Zwilling, C. E., Morey, C. C., & Pratte, M. S. (2008). An assessment of fixed-capacity models of visual working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(16), 5975-5975.

Rouse, W. B., & Morris, N. M. (1985). On looking into the black box: Prospects and limits in the search for mental models" (No. DTIC #AD-A159080). Atlanta, GA: Center for Man-Machine Systems Research, Georgia Institute of Technology.

Rozzi, S., Wong, W., Amaldi, P., Woodward, P., & Fields, B. (2006). Design sketching for space and time. In A. Mark-Pjetersen, T. Clemensen, R. Orngreen, P. Campos & W. Wong (Eds.), Proceedings of the First Conference on Human Work and Interaction Design HWID 2006 (pp. 181-205). Funchal, Portugal.

Saito, A. (2000). Bartlett, Culture and Cognition: Routledge. Salthouse, T. A. (1991). Expertise as the circumvention of human processing limitation. In K.

A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise (pp. 286-300). Cambridge, England: Cambridge University Press.

Schaeken, W., Johnson-Laird, P. N., & d'Ydewalle, G. (1996). Mental models and temporal reasoning. Cognition, 60(3), 205-234.

Schneider, W., & Shiffren, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84, 1-66.

Schulz, R., & Curnow, C. (1988). Peak performance and age among superathletes: Track and field, swimming, baseball, tennis, and golf. Journal of Gerentology: Psychological sciences, 43, 113-120.

Schulz, R., Musa, D., Staszewski, J., & Siegler, R. S. (1994). The relationship between age and major league baseball performance: Implications for development. Psychology and Aging, 9(2), 274-286.

Seamster, T. L., Redding, R. E., Cannon, J. R., Ryder, J. M., & Purcell, J. A. (1993). Cognitive Task Analysis of Expertise in Air Traffic Control. The International Journal of Aviation Psychology, 3(4), 257-283.

Secrist, G. E., & Hartman, B. O. (1993). Situational awareness: the trainability of the near-threshold information acquisition dimension. Aviation, Space, and Environmental Medicine, 64, 885-892.

Shah, P., & Carpenter, P. A. (1995). Conceptual limitations in comprehending line graphs. Journal of Experimental Psychology, 124, 43-61.

Shiffren, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190.

Simon, H. A. (1957). Models of man: Social and rational. New York: John Wiley & Sons, Inc. Simon, H. A., & Chase, W. G. (1973). Skill in Chess. American Scientist, 61, 394-403. Stankovic, S., Raufaste, E., & Averty, P. (2006). Modeling Risk Judgments about Conflicts

between Planes in Air Traffic Controllers: Individual Differences and Expertise. Paper presented at the HCI Aero 2006, Seattle.

Voss, J. F., Greene, T. R., Post, T. A., & Penner, B. C. (1983). Problem solving skill in the social sciences. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research theory (Vol. 17, pp. 165-213). New York: Academic Press.

Voss, J. F., Tyler, S., & Yengo, L. (1983). Individual differences in the solving of social science problems. In R. Dillon & R. Schmeck (Eds.), Individual differences in cognition (pp. 205-232). New York: Academic Press.

Ward, P., A. Mark Williams, & Hancock, P. A. (2006). Simulation for performance and training. In K. A. Ericsson, N. Charness, P. J. Feltovich & R. R. Hoffman (Eds.), The




Cambridge handbook of expertise and expert performance (pp. 243-261). New York: Cambridge University Press.

Wolfe, J. M. (2000). Visual Attention. In K. K. DeValois (Ed.), Seeing (2nd ed., pp. 335-386). San Diego, CA: Academic Press.

Wong, B. L. W., Rozzi, S., Gaukrodger, S., Boccalatte, A., Paola, A., Fields, B., et al. (2008). Human-Centred Innovation: Developing 3D-in-2D Displays for ATC. In D. Vu N., A. Zellweger, D. George & J.-M. Garot (Eds.), Proceedings of ICRAT 2008, the Third International Conference on Research n Air Transportation (pp. Accepted for publication).

Xu, X., & Rantanen, E. M. (2003). Conflict detection in air traffic control: A task analysis, a literature review, and a need for future research. Paper presented at the the 12th International Symposium on Aviation Psychology, Dayton, OH.

Yantis, S. (1993). Stimulus-Driven Attentional Capture and Attentional Control Settings. Journal of Experimental Psychology: Human Perception and Performance, 19(3), 676-681.



Documents

Design for pattern recognition in spatial awareness - Eurocontrol