7

ASSESSING THE HUMAN VIGILANCE CAPACITY OF CONTROL ROOM OPERATORS

C.H.M Donald

Leaderware, South Africa

Current control systems and control room design make use of highly sophisticated technology which has led to the potential for ever greater service capabilities, yet this potential is often not fully realised in implementation. Ergonomic design improvements in areas such as layout, information display and control interfaces have been increasingly recognised and implemented to improve service delivery. Backed up by a system of policies and procedures to ensure standardised operation, they have led to high expectations of performance. Yet management can still be disappointed by the performance of these expensive technology systems. In addition, the performance of personnel within the control room environment can differ markedly, and simple errors of attention or oversights may occur despite intensive attention to the ergonomic interface. Central to all these issues is the frequently underrated point that operators are critical to effective system functioning. The identification of the kind of people required and the selection of these personnel is fundamental to any effective system implementation. Indeed, while technology is often taking over the role of people, the remaining critical components often become concentrated in fewer people who become even more indispensable to effective functioning.

From an engineering or management perspective we have come across three general perspectives on the role of operators within control room operations in the CCTV area. The first is that the involvement of operators is not significant because the technology can perform or be developed to perform any significant task required of an operator. A second and sometimes related perspective is that operator jobs are routine and mundane and that there are no special skills requirements from such a person. A third perspective, and one we favour, is that operators are an integral part of the control room system and are critical in realising the performance potential of the technology. In terms of this, we have a number of fundamental beliefs about the nature of the operator role and implications for the kind of person required. These include:

1. The surveillanceheview function of the control room operator is an active and not a passive activity. It requires active observation and continual analysis of images and information, it requires these to be put in context, and a continual evaluation of the status of the situation.

2. The operator must be capable of long term vigilance.

Even where the situation may consist of periods of more intensive focus or demands, ongoing concentration and the ability to detect issues are important.

3. Perceptual processing and recognition is important to outcomes. This goes beyond simple sensory recognition to being aware of the context of the situation and the identification of patterns within the displayed information.

4. The operator must be able to deal with separate sources of information and allocate attention dynamically according to the pattern or relevance of signals received.

CORE OPERATOR COMPETENCIES

There are many criteria that will influence CCTV operator effectiveness and these need to be addressed comprehensively as part of managing the control room and the selection and placement process. Specifically, assessing a candidate’s suitability for the position would include a review of qualifications, training, and experience. However, we believe the central core of competencies for a control room operator revolves around the concept of human vigilance. Vigilance in this context refers to a capacity for sustained effective attention when monitoring adisplay for critical signals or events to which the observer must respond. Incorporated into our view of vigilance is the ability to effectively observe, process and analyse signal information. The context of this human vigilance applied to the CCTV operator is illustrated in Figure 1.

Important to note in Figure 1 is the progress from basic physical vigilance requirements from the operator to more advanced perceptual processing of the information and ultimately the ability to apply this on a sustained basis. Physical and sensory aspects of vigilance are in fact only preconditions to the perceptual processing and observations skills which are essential to realise results. Further, these have to be applied on a sustained and consistent basis in order to be fully successful. Training, suitability of personality, ergonomics and other factors illustrated are essential to the realisation of these core competencies in the control room environment, but the vigilance of the operator provides the foundation for performance.

People in Control: An International Conference on Human Interfaces in Control Rooms, Cockpits and Command Centres: 21 - 23 June 1999, Conference Publication No. 463,O IEE, 1999

8

Management style

(Eg., high concentration levels, sustained

performance, consistency) (Eg.,Visual acuity colour differentiation)

Observation

(Eg., pattern recognition, cue identification, recognition of anomalies,

use of memory, ability to see information in context, and effective screening of

environment monitored environment

FIGURE 1: CCTV operator vigilance and determinants of performance

ASSESSING OPERATOR VIGILANCE

Traditional assessment methods of application form screening, interviewing, and testing are important in providing a broader perspective on the candidate but tend to fall well short in their capacity to evaluate vigilance. For instance, because a person may have experience in positions where vigilance was a requirement of the job, the same selection shortfalls would have been characterised in the other operation and there is no guarantee that they would have demonstrated the required levels in such a position. While a simulation would be appropriate, for example the identification of events within a CCTV street scene, this actively discriminates against people who do not have familiarity with this kind of content but who may well have excellent potential. For this reason, an exercise was developed to simulate the required core competencies we associated with the ability to sustain a high degree of vigilance and detection ability.

Developing the vigilance based exercises

During the development of a suitable assessment exercise, the Surveillance and Monitoring Assessment Exercise (SAMAE), to evaluate the core vigilance dimensions it was felt that a single exercise could not replicate all the competencies required. Three separate exercises were therefore developed addressing different aspects of the CCTV operator competency requirements. Two of these

could be seen as generic to control operators while the third examines images linked directly to the CCTV surveillance of people. Common to all the exercises are the measurement of the accuracy of identification, response time, and the level of false alarms (inappropriate identification of a signal). All these main indices can be plotted on a graph to provide a visual indication of variance in performance levels over time.

The images used within the exercises were of particular importance. Developing scenes that may suit particular industries or types of control rooms, would have high cost implications and generating incident conditions for all of these would not be feasible. Also, as discussed previously, the use of material which is specific to existing applications within an industry is unfamiliar and unfair to new applicants. It also does not measure actual potential. It was important, therefore, that the images used in the exercises allowed measurement of the core competencies but were generic enough to apply to a range of operations. Also, the evaluation of human vigilance in research has typically utilised a set of signals that are based on direct sensory recognition. These, for example, would include a flicker on a video screen, a flashing light, length of line segments, or digital values. It was felt that these types of signals did not address the more complex vigilance skills outlined in Figure 1. Specific images were therefore developed around the design requirements for each test.

The Scanning Exercise consists of a complex image that constantly moves across the screen. Elements within the image can change in a number of ways at any stage and it

9

needs to be continually compared to a standard image of the object position at the bottom of the screen. Elements of colour, positioning of objects relative to one-another, rotation, loss of information,. and the ability to evaluate the situation as a whole are incorporated into the exercise. The availability of a standard representation of the target helps by refreshing memory and keeping the person sensitive to requirements. The exercise evaluates rapid review and search behaviour and reflects the speed of information processing required for these activities. It also examines the level and consistency of sustained concentration over time. Quick recognition of essential details and the ability to home in on relevant points or characteristics would help the candidate's performance as would the ability to remain focussed over time and maintain attention to detail.

The Dynamic Attention Exercise evaluates a person's ability to effectively monitor and deal with a situation where attention must be allocated to number of independent objects. Candidates are required to constantly track and prioritise their attention between a set of geometric figures which move around on predetermined paths and which change at specific intervals. Changes in colour, perspective, speed and the visual presence of the objects need to be detected. Important aspects examined during the exercise include the ability to divide and allocate attention between multiple information sources and the use of peripheral vision to monitor the broader context of what is going on. Abilities which assist in the tasks include the sampling of all channels of information ie., not being diverted into following a limited area of focus and neglecting others, and efficient switching of attention between different aspects of a situation. Sensitivity to motion and differential speed is also examined as part of this.

The Incident Detection Exercise evaluates a person's accuracy in rapidly identifying and responding to representation of people's behaviour. Use is made of continual changes in the postures of figures of people in the exercise, deliberate distracters, and the intermittent showing of incident activities. Speed and levels of detection are important as is the ability to differentiate between incident and near incident activities. The ability to generalise a classification of behaviour within a category and apply it to all incidents within that classification are also relevant.

THE IMPACT OF ASSESSING CORE VIGILANCE COMPETENCIES

The utility of the SAMAE exercises measuring the core competencies was evaluated within a CCTV control room of a major international company - see Donald (1). This site had comprehensively addressed the measurement of

operator performance as a conscious part of establishing benchmarks of system effectiveness. CCTV operators were responsible for observing activities that indicated the potential theft of product (suspicious incidents) as well as violations of company policy, production procedures, and safety violations. The study looked at whether the results from the assessment exercises were consistently related to the levels of performance of the operators.

Measures were taken over an eighteen month period and all operators were included in the analysis. These operators had already been selected using standard company procedures. Two methods of sampling were made for this group when conducting the analysis of the relationship between assessment and performance. On one hand, the period for which the largest number of operators formed a consistent group was used. The second method took the eighteen month period and divided it into sections and then examined the consistency of the assessment/performance relationship over these periods. The sample during each of these periods changed slightly as turnover within the operation changed. In establishing the assessment/performance relationships, it was found that while accuracy, reaction time and false alarms of an exercise were relevant in themselves, the use of an efficiency measure combining accuracy and false alarms proved to be a better predictor of performance. The efficiency scores from all the exercises were also totalled to give an overall measure. In-depth information on the descriptions of variables and results are available in the source document (l).The exercises assessing the human vigilance competencies consistently and strongly relate to the key performance indicators on the job including the verified detection of potentially illegal activities. Correlations illustrating these relationships are shown in Table 1. These correlations are highly significant despite a small sample and are substantial when compared to the relationships generally reported for the use of standard selection instruments or procedures. Further analysis demonstrates that the relationships between the scanning and dynamic exercises and performance criteria are maintained across different time periods within this 18 month period with a slightly changing sample composition over this time. The incident detection exercise, which interestingly enough is the most "sensory" based, does not show the same consistency.

The utility of using personnel with high measured abilities of the core competencies is illustrated below in Table 2. Table 2 demonstrates the number of incidents detected by operators who scored in the top 50% of the sample on the exercises versus those scoring in the bottom 50%. Results show that those scoring in the top 50% on the exercises consistently detected more than double the number of confirmed incidents than those scoring in the bottom 50%.

TABLE 1. Correlations between SAMAE exercises and work performance measures

SAMAE Exercises

WORK PERFORMANCE MEASURES

Confirmed Suspicious Incidents Manager's Pre-training measure of Incidents Detected Detected Performance Rating non-verbal behaviour

recognition.

Total score .12** .76** .72** .59**

Scanning efficiency .78** .71** .61** .57**

Dynamic efficiency .55* .58* .59** 0.38

Incident efficiency 0.25 0.38 .45* .52**

Sample size 10 10 15 16 Note: * pe0.05 ** p<O.Ol

TABLE 2. SAMAE score groups and detection rates.

SAMAE Sample half. Incidents Test Score detected.

Scanning Top 50% 113

Bottom 50% 47 Effectiveness

~ ~

Top 50% 102 Dynamic Att. Effectiveness Bottom 50% 41

DISCUSSION

The scanning exercise shows the strongest relationship to job performance and appears to provide the highest utility of the three exercises. Consistent with the model in Figure 1, this exercise is also the one that requires the highest level of advanced perceptual processing and observation skills. The strong impact of the dynamic attention exercise which emphasises observation skills and the ability to allocate attention effectively among a number of objects also appears to have a high utility. On the other hand, the incident detection exercise which is the most "sensory" based of the three did not correlate well with overall job performance. However, the quick recognition of behaviour being measured in this exercise may have more specialised application in certain contexts, for example casino tables where quick actions are characteristic of the type of behaviour under surveillance.

High scores for all exercises also related to the effective identification of non-verbal behaviour in both pre- and

post-test training evaluation assessments that were conducted with the sample of operators. This appears to indicate that there is an underlying set of competencies within people as illustrated in Figure 1 that make them naturally more suited to control room based surveillance and detection. Training is likely to enhance people's ability, but people who have natural potential for the work appear likely to maintain the performance gap after training. This reinforces the need to have an appropriate selection instrument to identify people who will be able to operationalis? the training inputs more effectively than others.

Assessment of the core vigilance competencies can go further than providing an immediate selection and placement role. By providing a diagnostic function, the exercises can be used for examining current employees and gaining a perspective on performance issues. More effective positioning of personnel to suit their profiles is possible. Indeed, preliminary work in the casino industry, for example, indicates that while speed of processing is essential for staff conducting surveillance on gaming tables who may have to detect issues within a second or two, other surveillance roles in terms of slot machine floors and premises protection require a more detailed but less time intensive profile. Similar differences may exist between staff operating in a "live" context and those with review or audit roles. With an understanding of their capacity and strengths, staff can be assigned to control room functions which not only benefit them, but also allow greater effectiveness of the system. The same is likely to be true of control room functions in other industries where information processing or situational assessment based on monitoring of signals may require fairly unique processing demands.

Our research shows that the core vigilance competencies discussed and measured by the SAMAE exercises do in fact appear important to operator performance. It also

11

indicates that selection of appropriate personnel can play a major contribution in achieving the desired outputs from a control room based CCTV system. Further, to produce a comparable level of increase in system performance by investing in additional technology is likely to be prohibitively expensive and unrealistic. It is quite possible that even the best technology systems are not being fully utilised in terms of their capacity, or alternatively errors occurring in systems are not the fault of the technology but of inappropriate staffing of personnel. Given the capital outlays of modern control centres and the potential cost of failure for the safety and welfare of people, the role,and capacity of operators need to be recognised as critical in the delivery of a control room technology solution.

List of References

1 . Donald C.H.M., 1998, Technical report on the validity of the Surveillance and Monitoring Assessment Exercise (SAMAE), Report No 2/1998, Leaderware, Johannesburg.