at SciVerse ScienceDirect

Progress in Nuclear Energy 55 (2012) 93e101

Contents lists available

Progress in Nuclear Energy

journal homepage: www.elsevier .com/locate/pnucene

Human factors evaluation in nuclear power plant control rooms using a mobilesystem to support collaborative observation

Luiz Carlos Silva Junior a, Marcos Roberto da Silva Borges a, Paulo Victor Rodrigues Carvalho a,b,*

aGraduate Program in Informatics-NCE&IM, Cidade Universitária, Rio de Janeiro, RJ, BrazilbComissão Nacional de Energia Nuclear, Instituto de Engenharia Nuclear, Cidade Univerisitária, Ilha do Fundão, Rio de Janeiro, CEP 21945-970 Rio de Janeiro, RJ, Brazil

a r t i c l e i n f o

Article history:Received 1 September 2011Received in revised form3 November 2011Accepted 12 November 2011

Keywords:Mobile computer systemCollaborative observationHuman factors engineeringHuman-system interface design

* Corresponding author. Comissão Nacional de EEngenharia Nuclear, Cidade Univerisitária, Ilha do21945-970 Rio de Janeiro, RJ, Brazil. Tel.: þ55 21 2173

E-mail addresses: paulov@ien.gov.br, paulov@pq.c

0149-1970/$ e see front matter � 2011 Elsevier Ltd.doi:10.1016/j.pnucene.2011.11.007

a b s t r a c t

Nuclear industry has several regulations and guidelines for human factors engineering (HFE) that shouldbe applied in modernization of existing ones and in construction of new power plants. Their aim is toensure that human factors/ergonomics are applied during the design process, making the system morecontrollable, reliable and safe than it would be if human factors were not adequately incorporated. Timeconsuming and inadequate methods for data collection and analysis in complex settings are oftendescribed as a problem to apply human factors in the design of process control systems. To facilitate theuse of human factors/ergonomics in the design, we developed a mobile computer system to supportcollaborative observation of work teams to be used in complex environments. In this paper, we arguethat collaborative observation should be used in the work teams’ knowledge elicitation for humansystem interface design purposes, showing the advantages of the collaborative approach in comparisonto other non-collaborative ones. We describe the development of a mobile support system for collabo-rative observation tested in the evaluation of human system interfaces of a nuclear power simulator crewwhen operating a digital human-system interface. The results indicated that the mobile system stimu-lates collaboration among observers and the organization of overall human-system interface evaluation.

� 2011 Elsevier Ltd. All rights reserved.

1. Introduction

Nuclear power appears as an option for low cost and sustainableelectrical power source. There has been renewed interest inbuilding new nuclear power plants (NPP) in many countries asshown by the International Atomic Energy Organization e IAEAdata presented in Fig. 1. Most of these reactors have more than 20years of operation, as shown in Fig. 2, and their instrumentationand control (I&C) systems are now being modernized to use newdigital technology.

Therefore, several NPP vendors are in the process of submittingnew I&C and Human System Interface (HSI) for design certificationreview by the regulatory authorities of many countries, and theHuman Factors Engineering (HFE) program is an importantrequirement for design certification of new HSIs. The aim of HFEprogram is to insure that human system interfaces, procedures, andtraining effectively support NPP personnel tasks and foster safe and

nergia Nuclear, Instituto deFundão, Rio de Janeiro, CEP3835.npq.br (P.V.R. Carvalho).

All rights reserved.

efficient generation of power (O’Hara and Brown, 2004). The HFEprogram encompasses a comprehensive set of HFE activities thatinclude operating experience review; functional requirementsanalysis and function allocation; task analysis; staffing and quali-fications analysis; human reliability analysis; human-systeminterface design; procedure development; training programdevelopment; and a final, comprehensive, verification and valida-tion (V&V) program. The V&V is conducted with operators in plantsimulators specific to the plant design that has been built. In theseV&V process we need to understand operators’ interactions withthe human-system interface that shape groupwork activities. Fig. 3presents HFE key activities that should be considered in any HSImodernization program.

To perform adequate validation of control room interface designa human-centered approach (HCA) has to be used to exploit thetechnical innovations for the optimum humaneartifact interac-tions, aiming at improving the appropriateness of the technologicalsolutions (Hancock and Chignell, 1995). In HCA we have to under-stand human interactions in group work activities because theyplay a paramount role in problem solving and local decision-making processes, and give important clues to investigate thetacit knowledge that teams use during their work activity. Behindthese interactions we can find important reasoning mechanisms

Fig. 1. Reactors under construction worldwide e from: www.iaea.org.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e10194

that will drive people, according to their experience, to select andmanage their actions in context-dependent work situations. Theanalysis of these interactions using ethnographic methods(Carvalho et al., 2005, 2006; Carvalho, 2005) may lead to thediscovery whether the human-system interface can efficientlysupport team members’ communication, reasoning, decision-making, and action selection, the objective of the human factorsvalidation process. However, to perform a cognitive task analysisusing ethnographic methods based only in filed notes (paper andpencil) is an extremely time-consuming task (Crandall et al., 2006;Stanton et al., 2005; Carvalho et al., 2007).

Therefore, there is still a need for an effective, efficient andusable support tool to practitioners in the use of ethnographicmethods for activity analysis purposes. Such a tool has to providea lean, systematic and structured support to perform a cognitivetask analysis in a specific setting (control room layout, procedures,and people interactions) that could facilitate the job of practitionersin the various phases of analysis, from planning to the final reports.This paper describes the development of a mobile computerizedsupport tool to be used in collaborative observation: performed byseveral observers who can interact with each other. The tool wastested in the analysis of operators’ activities during a nuclear powerplant control room digital interface evaluation.

Fig. 2. Number of operating reactor

2. Human factors in hybrid and advanced NPPs control rooms

In most operating nuclear reactors, conventional control roomsinclude a large control board at the front that provides plantparameter indications and controls, as well as alarm tiles that lightup and provide audio indications to alert to abnormal plantconditions (see Fig. 4).

Conventional control room is usually staffed by two reactoroperators (RO) that are responsible for monitoring the controlboard and taking control actions, and a Foreman(a Senior ReactorOperator or SRO) who is responsible for maintaining broad situa-tion awareness and directing the activities of the ROs, and a shiftsupervisor, responsible for the overall operations conduct. In caseof plant disturbances, paper procedures, providing detailed step-by-step guidance, are used by the crew to respond to the event.Typically, the Foreman reads the procedure steps aloud to the twoROs who, based on the Foreman instructions, read off plant indi-cations and take control actions at the control board. The ROs callout plant parameter readings and control actions so that the wholecrew maintains shared situation awareness of plant conditions andabout operators’ actions (Vidal et al., 2009).

The new NPP control roomshave a compact digital design basedon computerized visualization systems (video display unitseVDUs,wall panels) and soft controls (mouse, keyboards, tracker balls). Inthis new work setting, operators will be seated at computer work-stations with multiple VDUs providing integrated displays of plantparameter information and control capability, as well as a dedicatedalarm VDU and a computerized procedure VDU. Digital controlrooms normally have a large wall-mounted display panel (LDP) atthe front (See Fig. 5) where the situations of main components aredisplayed. In these control rooms the RO and SCO have the all theinformation about plant processes in their screens, but they need tonavigate to find the adequate screen according to the plant situa-tion. In new control rooms operations are still conducted by one RO,who will be responsible for control actions in the primary(or reactor) side of the plant, and one SCO who will be responsiblefor control actions in the secondary (or turbine) side of the plant.A senior operator (SRO) will encompass the former activities of theForeman and Shift Supervisor (a staff reduction of 1 element).

by age e from: www.iaea.org.

Fig. 3. HFE activities that should be considered for any modernization program (from USNRC, 2002).

Fig. 4. Conventional control room based on analog instrumentation and controlsystems.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e101 95

Hybrid control rooms, as shown in Fig. 6, appeared in themodernization process of existing reactors, a continuouslychanging process, and poses new challenges for operators.

Recent human factors and ergonomics researchmakes clear thatchanges in control room design based on new technologicalpossibilities (types/levels of automation in a system, digital human-machine interfaces, operator support systems using artificialintelligence etc.) can impact operator monitoring and situationawareness, and teamwork performance (Parasuraman et al., 2000;Sheridan, 2002; Nachreiner et al., 2006; Carvalho et al., 2008, 2009,Vicente et al., 1997). The transition to a digital control room intro-duces opportunities for enhanced support (e.g., integrated infor-mation displays for greater situation awareness; improved alarmsystems for more effective directed attention; computerizedprocedures that reduce the mental overload associated withfollowing paper-based procedures). At the same time this situationintroduces potential for new operating problems. For example, theshift from a control board (an open environment) to individualworkstations (a more private workspace) may make it more diffi-cult for operators to maintain shared awareness of crew memberintentions and activities (O’Hara and Roth, 2005). Another exampleis presented in the control room in Fig. 6. In this control room, theoperators change their operation modes according to the plantsystem they have to control. For most of the plant systems, theoperators use conventional panels with analog indicators andpushbuttons, and for a small set of systems they use an entirelydigital environment, using VDUs, keyboard and mouse. Similarly,the eventual reduction in crew size after modernization (the fore-man is excluded in new control rooms team configuration), andresulting shift in workload distribution, might lead to higher(and perhaps unacceptable) physical or cognitive workload. In thisnew operating environment system failures and accidents emergein situations when operators do not understand the actual situa-tion, i.e., the systems do not provide the right information in theright moment to the operators who were not able to adapt theirbehaviors according to the actual situation demands and maketheir decisions in a safe way.

To overcome the problems that new control technology mayintroduce in the operation of new and existing NPPs, it is necessaryto emphasize design processes, based on cognitive requirementsthat facilitate the adaptation of the human cognition to the systemfunctions. The cognitive requirements for design can be defined asthe functional requirements of the system that will give criticalsupport to the cognitive activities of the operator throughout theexecution of his/her work. These requirements will be paramountfor the construction of information systems capable to enablepeople to achieve adequate situation awareness (enhance percep-tion possibilities, decision-making and action planning support).Therefore these cognitive requirements must be incorporated earlyin the design of displays, human-system interfaces, and must bepresent in the collaborative characteristics of the system.

To deal with the modernization of Brazilian NPPs control roomsand the digital control rooms of new plants, the Nuclear Engi-neering Institute (IEN) has initiated a comprehensive human

Fig. 5. Advanced control room based on digital instrumentation and control systems.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e10196

factors and ergonomics program to identify and address cognitiverequirements early in the design process (Santos et al., 2007).Considering that a full-scope (plant specific) simulator will beavailable later in the design process, early tests to understandoperators interactions in the new control rooms are being con-ducted using the Human System Interface Laboratory (HSIL) digitalcontrol room, which is driven by a dynamic simulator of a nuclearreactor of the same type used in Brazil (Pressurized Water Reactore PWR). Fig. 7 shows the HSIL control room.

3. Understanding cognition behind human work

To perform an adequate evaluation of human-system interfacethere is a need to understand how the cognitive system, composedby people, technology, and environment, actually operates.A cognitive system is a self regulated and adaptable system thatfunctions using knowledge about itself and the environment toplan and modify their actions (Hollnagel and Woods, 2005). Anadequate design of a cognitive system depends basically on: theexistence of a common vocabulary between their parts; the studyof the agents’ cognition during their actual activity, analyzing themanetechnology interactions in real environments rather thanstudies that analyze each system component apart.

Hutchins (1995) states that cognition is best understood asa distributed phenomenon. The theory of distributed cognitionseeks to understand the organization of cognitive systems by

Fig. 6. Hybrid control room. The operator has to use conventional buttons and indic

extending the scope of what is considered cognitive beyond theindividual and cover the interactions between people, and betweenpeople and the resources and materials of the work environment.The concept is very important for control rooms, since our aim is tounderstand the interrelationships between people and artifacts, aswell as groups of people, in control rooms.

Distributed cognition looks for cognitive processes, whereverthey may occur, depending on the functional relationships of theelements that participate together in the process. It is important tonote in this context that a process is not cognitive simply because ithappens in the brain, or ceases to be cognitive, simply because ithappens within interactions among many brains. In distributedcognition, it is expected to find a system that can dynamicallyconfigure itself to coordinate performing many functions.A cognitive process is delimited by the functional relationshipsbetween its elements and by their position in space. In the field,applying these principles, there are mainly three types of distri-bution of cognitive processes (Hollan et al., 2000):

� Cognitive processes may be distributed over the members ofa group.

� Cognitive processes may involve coordination betweeninternal and external structure (material or environmental).

� These processes can be distributed over time, so that theproducts of earlier events can transform the nature of laterevents.

To design human-system interfaces there is a need to understandthe system’s cognitive requirements that can be defined as func-tional requirementsof the systemthatwill give critical support to thecognitive activities of the operator throughout the execution of his/herwork. Thismeans that these requirementswill be paramount forthe construction of information systems capable at enabling peopleto achieve adequate situation awareness (enhance perceptionpossibilities, decision-making and action planning support).

These cognitive requirements or cognitive demands must beincorporated as early as possible in the design of HSIs, and arepresent in the collaborative characteristics of many cognitivesystems, such as control rooms (Christel and Kang,1992; Roth et al.,2004); military decision-making systems (Crandall et al., 2006); airtraffic control (Bentley et al., 1992; Harper et al., 1990). To discovercognitive requirements for design, we argue that is necessary tocarry out cognitive task analysis in actual or simulated work situ-ations using collaborative observation methods, understandinghow and why operators act, and the intrinsic human-system

ators to operate some systems and a digital HSI to operate other plant systems.

Fig. 7. HSIL control room. RO means Reactor operator, SCO Secondary system operator,SS Shift Supervisor.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e101 97

relations (human-organization, human-technology, and human-ehuman actions).

4. The collaborative observation approach

Collaborative observation is the combination of collaborationbetween subjects and observers to analyze work activities in fieldresearch, which is particularly suitable for the study of complexactivities. A straightforward combination between observers andobserved subjects is presented in Table 1. We can have a singleperson observing another person or a group, and a team observinga single person or a group.

Case A is the most common: an individual observing anotherindividual carrying out some task. This situation also refers toa number of observer/observed pairs in parallel. If the observationrefers to the same task, some variationmay occur between differentobservers/observed pairs. The results are quite dependent on thebackground and on the previous experience of the observer. Theremay also be some information loss during the observation. In thisscenario, the aggregation of information from different sources isnot an easy task and may show some inconsistencies.

Case B is possible but not common. In this scenario the loss ofinformation is potentially high. It is very difficult to a singleobserver to capture and understand all tasks and interactions,particularly if the activities are complex. A way to overcome thesedifficulties is to do the observation in several sessions.

Case C is normally used when the observation requires multipleperspectives over the same observer or task. In this scenario it isexpected that the information gathered and processed would bericher than that collect by a single observer. This can be considereda particular case of case D, assuming that the observers would meetand discuss about their findings before, during and after theobservation. This is not easy to do without the support of anappropriate process and a supporting tool.

Finally, case D is the situation that will be dealt with in thiswork. The observation of groups and their interaction being per-formed by a team of observers is a very challenging task. The hugeamount of information collected that needs to be organized and

Table 1Combination of ethnography studies possibilities.

Observer/Observed Individual Teams

Individual A BTeams C D

processed requires not only an organized process but also anappropriate supporting tool. The complexity of this alternative isa consequence of combined circumstances originated by themultiple, and perhaps conflicting, perspectives from the observers’part and the potentially high number of interactions among themembers of the observed group. We will try to avoid the loss ofrelevant information intrinsic to case B.

Even a collaborative observation can generate losses and insome cases conflicts. When people are observing a complex event,losses may occur due to several possibilities: the complexityof cognitive actions required, a geographically dispersed event (thesame observer cannot be in two places at the same time), or evenlack of attention from the observer. Redundancy and collaborationin these cases and an observational framework are essential.

We believe that the collaborative observation is absolutelynecessary for human system interface evaluation in nuclearindustry. Most of methods currently used were adequate fordescribing individual activities done in a well-defined sequence.However, there are many field studies (e.g. Carvalho et al., 2005;Vicente et al., 1997) showing that the work in NPP control roomsinvolves multiple and often conflicting (in goals and time) lines ofactivities, with many differences between the tasks described inprocedures and real work actions (how the tasks are actually done).Even in a rigid work setting like NPPs, the actual work in controlrooms is characterized by adaptations, improvisation and ad hocprocedure modifications, because the work demands and resourcesavailable rarely correspond to what was anticipated when the taskwas developed, thereby rendering the task description or opera-tional procedures unworkable (Hollnagel and Woods, 2005;Carvalho et al., 2007). Using the non-collaborative, one-to-one,observation procedure, it is very difficult for the observers tocapture the multiple actions pathways of real work activities,describing the many simultaneous tasks and tasks adaptations thatpeople have to do to cope with reality. Another difficulty for thetraditional observation methods is the collective/collaborativecharacteristic of the work done in a NPP control room, where mostof the activities are done in cooperation, involves multiple opera-tors who use many different cooperative mechanisms (Vidal et al.,2009). Therefore, for an adequate performance evaluation in NPPcontrol rooms, we need the collaborative observation - an obser-vation procedure in which many observers, in collaboration, areable to observe the activities of many subjects - and adequatecomputerized tools to support the observers’ tasks.

5. Mobile system to support collaborative observation

The direct observation of work activities approach presentssome problems, due the difficulty that observers have to captureand analyze the entire information of the social environment. Tofacilitate the observers work and further data analysis, the obser-vation preparation demands extensive planning and coordination,especially if the observation will be carried out by teams, becausepeople may have different perceptions and viewpoints about theobserved activities (Crandall et al., 2006; Vidal et al., 2009). Anotherlimitation is the need of a long time to capture and analyze whatoccurs in the field to get rich and detailed information (Millen,2000). In some cases, it is necessary a familiarization with thedomain to be studied. Moreover, it is difficult to use the techniquein complex and distributed settings due the huge number of vari-ables and situations to be observed. The technique also can presentrisks for the researchers, or to be impracticable if the presence ofthe researches/observers jeopardizes the work activities observed.Normally, there is the need of getting the permission for entrance inthe work settings and for registering the information, and also theacceptance and the assent from those who will be observed.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e10198

According to Guerlain et al. (2002), the evaluation of activitiesfrom a coordinated team involves the independent observation ofmultiples individuals. The analysis of these independent observa-tions requires that they should be coordinated, codified andcorrelated before a subjective evaluation can be performed. Thus,similar to other works (Carvalho et al., 2005, 2006; Hughes et al.,1997), Guerlain claims that a tool to support the informationexchange and the discussion among observers is desired. The toolthat supports the collaborative observation should facilitatecollaboration and interaction during and after the gathering of fielddata in order to identify the cognitive requirements with less effort.

In this work, the tool that will support the ethnographic methodaims at structuring the observation procedure (preparation phase),during the observation (capture phase), encouraging collaborationand interaction among specialists in the gathering of information inthe field, and after the observation, producing a consolidated fieldreport. The process is described in on Fig. 8.

The mobile support system for observation aims to providetechnological support for the observer in the preparation, capture,and analysis phases of a collaborative observation, specifically inthe following points:

� Preparation phase: Definition of tasks to the team, choice ofperspectives and key people, choice of observable variables,choice of registration methods.

� Capture phase: Collecting records and elements of observation,storage and cataloging, transcript of records.

� Analysis phase: Organization in the timeline and categorizedrecords.

The basic requirements for the development of the mobilecomputerized system to support collaborative observationapproach are:

� Allow the registration of those involved in ethnographic study(actors, operators, researchers and others involved).

Fig. 8. Structuring of ethnography

� Allow the registration of artifacts used by those involved inethnographic study (tools, systems and other objects that arethe subject of interactions).

� Allow the record of field characteristics and related graphics.� Allow the record of terminology and expressions used by thoseinvolved in field study.

� Allow the record of different types of interaction between theinvolved and/or artifacts previously registered(human � human, human � artifact, artifact � artifact).

� Allow the record of field notes, activities, events and distur-bances in the execution of tasks.

� Support the information sharing between observers during theimplementation of the ethnographic study.

� Support the need for communication between observers atcertain times of observation.

� Support the division of labor among observers during theethnographic study, including the analysis phase.

� Generate reports of events recorded: chronological order(timeline), reports by event type/category, reports about datacollection preparation.

The main goal of the system development was to design anartifact to centralize the preparation of the observations and theinformation collected in the field in a single device in an organizedand easily retrievable manner. The systemwas designed to supportbefore, during and after an ethnographic study in an attempt toleverage the work of the observer in his field research. A snapshotof artifact designed, the input devices, and one of the outputs aredepicted on Fig. 9.

6. Collaborative observation in HSI evaluation of NPP controlrooms

The collaborative observation was used in the final phase of theHFE program for control room and human/system interfacesevaluation for nuclear power plants (O’Hara et al., 1994), where the

and mobile system features.

Fig. 9. A snapshot of artifact, inputs devices and some outputs (the indications in the figure are in Portuguese because that is language in which the device was designed).

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e101 99

entire process is simulated. It requires a simulated work setting,a detailed experimental planning, including data acquisition,analysis systems such as computer logs (process state, processevents), operator log (human machine interface events, keyboard,mouse) and audio& video recorder (verbal protocols, communi-cation). The last evaluation phases of HFE program must be donewith the system to be used in the plant and comprises simulationsin test facilities, factory acceptance tests and commissioning testsin the plant site. These last phases are based on performanceevaluation of the operators’ activities. Activity is defined as the setof behaviors and resources used by the operators to do their work,and can be understood through observation of communications,interactions (with the system and other people), gestures andpostures. Qualified observers, using collaborative observation,must be able to discover patterns of behavior that are recognizableand repeated during work and to identify not only the task/actionsrelated to the prescribed work, but also side activities not formu-lated in the frame of the task description (Marmaras and Pavard,1997). The data obtained (direct observation, field notes) is theset of signals picked up by the operators in the information fieldand how they use these signals to manipulate the control roominterfaces. The aim is to discover, by further analysis of the data setobtained, how operators transform the interface information inactions and decisions.

To evaluate the mobile support tool for collaborative observa-tionwe performed an experiment using the tool in the HSIL controlroom. The aim is to verify whether the use of collaborativeapproach with the support tool can reveal the information needed,adding quality to the information obtained through the discussion(collaboration) of the information throughout the ethnographicprocess. Previous work of Crandall et al. (2006) and Carvalho et al.(2005) will guide the experiment towards the implementation of

methodological phases of observation and critical issues of what,who, when and how to observe.

The experiment was conducted at the Laboratory of Human-eSystem Interface (HSIL) of Nuclear Engineering Institute (IEN)located in Rio de Janeiro. The HSIL control room design (Fig. 7)includes the normal HSI resources (e.g., alarms, displays, softcontrols, Large Display Panel - LDP, computerized procedures)that are used in new (digital) plants. While the detailed content ofthe HSI resources (e.g., specific content of displays, specificalarms) will depend on the detailed plant design, the basic HSIdesign concepts and principles by which specific displays, navi-gation processes, procedures, alarms etc., are developed to applyin other plants too.

The goal of the ethnographic method application on HSIL wasthe performance evaluation of the operators during postulatedaccidents that may occur in a nuclear power plant in order toimprove the human computer interface design. We paid particularattention to the tasks dictated by the procedure manual and to theoperators’ actual activity. This experiment is part of a process tosearch for particular deficiencies in the support of operatorresponse to abnormal system states to redesign the operatorinterface to improve the graphical layout of information, navigationamong screens, alarm presentation, acknowledgment andresponse, and to integrate these with computer-based proceduresthat dynamically correspond with real-time system information.

The HSIL control room operating crew, who participated in thestudy, is composed of three operators e Shift Supervisor, ReactorOperator (RO) and the Secondary Circuit Operator (SCO). The ShiftSupervisor is an engineer who has experience in the simulatoroperation. The RO and SCO are instrumentation technicians whohave been trained in LABIHS operation for 2 years before this study,but have no previous experience in the reference plant operation.

Fig. 10. Experiment framework through ethnographic phases.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e101100

In the research setting, we have the presence of commonelements in complex environments such as large number ofdisplays featuring information with frequent changes of state,need for coordination among distributed team working to resolveissues and events taking joint decision, in real time, with consid-erable gravity and impact. The experiment was carried out byobservers in three stages: preparation, capture and analysis of dataas shown in Fig. 10. In the first two phases, the observers madeobservations, questions to the actors and recorded the informationin the mobile system to support the observation. These individualsfreely observed the actors (primary operator, secondary operatorand supervisor) and interacted with each other to clarify prob-lems, views and questions about the tasks being performed. At thecompletion of the phases, the observers met to discuss the ques-tions and points that they found relevant on the records. After thetwo phases (preparation and capture) the observers met to

Fig. 11. Participants answers to

analyze the data together and extract the relevant knowledge ofthe tasks observed in the field. The observers completed a frame-work with the main points, making a complete record of infor-mation elicited.

7. Results and discussion

At the end of the experiment, the evaluation of the strengthsand weaknesses of the mobile system was conducted througha questionnaire that assessed both the use of themobile system andthe methodology of observations. Fig. 11 shows the participants’answers to the evaluation questionnaire. The answers to thequestions are classified according to the Likert Scale: totally agree(5 points), partially agree (4 points), have no opinion (3 points),mildly disagree (2 points) and strongly disagree (1 point). Throughthe weighted average of these points were composed of the overallscores for each question. Comments were also encouraged fromobservers and experts on each question to obtain more informationand justification of each individual score.

The experiment served to its original purpose to show thatcollaborative observation can be more efficient than other non-collaborative knowledge elicitation techniques especially for usein complex environments. This assumption can be verified by thesubjective evaluation of the respondents (observers and twocognitive task analysis specialists) who indicated a general agree-ment of the benefits of the incorporation of collaborative factorthroughout the ethnographic process.

Another point found in the evaluations was the benefit of usinga system to support the observation in the field, helping observersto collect, share, organize and analyze information. This is a crucialfactor to collect relevant information in complex environments,such as interactions between individuals and artifacts, field notes,classified by type of information and automatically create a time-line containing all the events recorded in chronological order. Thereports and records delivered by the system were also important

evaluation questionnaire.

L.C. Silva Junior et al. / Progress in Nuclear Energy 55 (2012) 93e101 101

inputs for the analysis of other records used to collect informationin the field, such as audio and video recording.

The overall impression of participants were that the collabora-tion between participants stimulated the discovery of knowledge,generating greater amounts of relevant information and addingquality after filtering and improving information through furtherdiscussion. The doubts and limitations were pointed out byparticipants on their observations and can be considered as a resultof this work and input for future work.

8. Conclusions

In this work we showed how HFE programs in nuclear industrybring challenges to designers and regulators regarding HSI evalu-ation, and reviewed some important concepts in the area ofcognitive systems. We proposed a collaborative observationapproach as a more appropriate alternative for the knowledgeelicitation in complex systems such nuclear power plant controlrooms. We claim that the observation approach has many advan-tages, such as a detailed analysis of the environment where theoperator performs his/her task. When observation is combinedwith a collaborative approach that supports different viewpoints aswell as interaction and discussion among the observers, weobserved an approach that produces a more efficient and conse-quential results.

Considering that there is a need to perform ethnographicstudies in a faster way and with a more organized data collection,we identified the need of amobile support system that would allowobservers to collect the data in a ubiquitous, driven and structuredmanner, and still supported the technique of observation in itsmain aspect: the collaboration. This prototype has been specified,designed and used in the experimentation of the technique in theenvironment of a simulated nuclear reactor control room.

In general, collaborative observation proved to be an effectivetechnique for the elicitation of knowledge teams, allowingobservers in two sessions to identify large amounts of relevantinformation such as activities performed, executors, and importantinformation for decision making, interactions, artifacts problemsand difficulties in performing the task. Collaboration in the unan-imous opinion of the participants was a factor that improved thequantity and quality of records.

Acknowledgments

The author Paulo Victor R. de Carvalho gratefully acknowledgethe support of the National Council of Scientific and TechnologicalDevelopment (CNPq- Conselho Nacional de DesenvolvimentoCientífico e Tecnológico) and the support of the Rio de JaneiroResearch Support Foundation (FAPERJ e Fundação de Amparoa Pesquisa do Rio de Janeiro).

References

Bentley, R., Hughes, J., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., Sommerville, I.,1992. Ethnographically-informed systems design for air traffic control. In: Proc.Computer-Supported Cooperative Work Conference. ACM Press, New York,pp. 123e129.

Carvalho, P.V.R., 2005. Ergonomic field studies in a nuclear power plant controlroom. Progress in Nuclear Energy 48 (1), 51e69.

Carvalho, P.V.R., Oliveira, M.V., 2009. A computerized tool to evaluate the cognitivecompatibility of the emergency operational procedures task flow. Progress inNuclear Energy 51, 409e419.

Carvalho, P.V.R., Santos, I.J.A., Vidal, M.C., 2005. Nuclear power plant shift super-visor’s decision-making during micro incidents. International Journal ofIndustrial Ergonomics 35 (7), 619e644.

Carvalho, P.V.R., Santos, I.J.A., Vidal, M.C., 2006. Safety implications of some culturaland cognitive issues in nuclear power plant operation. Applied Ergonomics 37,211e223.

Carvalho, P.V.R., Vidal, M.C., Carvalho, E.F., 2007. Nuclear power plant communi-cations in normative and actual practice: a field study of control room opera-tors’ communications. Human Factors in Ergonomics and Manufacturing 17 (1),43e78.

Carvalho, P.V.R., Santos, I.J.A., Gomes, J.O., Borges, M.R., Guerlain, S., 2008. Humanfactors approach for evaluation and redesign of humanesystem interfaces ofa nuclear power plant simulator. Displays 29, 273e284.

Christel, M.G. and Kang, K.C., 1992. Issues in Requirements Elicitation, TechnicalReport, CMU/SEI-92-TR-012, ESC-TR-92e012.

Crandall, B., Klein, G., Hoffman, R., 2006. Working Minds: A Practitioner’s Guide toCognitive Task Analysis. MIT Press, Cambridge, Mass.

Guerlain, S., Shin, T., Guo, H., Calland, J., 2002. A team performance data collectionand analysis system. In: Proceedings of the 46th Annual Meeting of the HumanFactors and Ergonomics Society. HFES, Santa Monica, CA, pp. 1443e1448.

Hancock, P., Chignell, M., 1995. On human factors. In: Flach, J., Hancock, P., Caird, J.,Vicente, K.J. (Eds.), Global Perspectives on the Ecology of HumaneMachineSystems. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 14e53.

Harper, R., Hughes, J., Shapiro, D., 1990. Harmonious working and CSCW: computertechnology and air traffic control. In: Bowers, J., Benford, S. (Eds.), Studies inComputer Supported Cooperative Work: Theory, Practice and Design. North-Holland, Amsterdam.

Hollan, J., Hutchins, E., Kirsh, D., 2000. Distributed cognition: toward a new foun-dation for human-computer interaction. Research ACM Transactions onComputer-Human Interaction Vol. 7 (2), 174e196.

Hollnagel, E., Woods, D.D., 2005. Joint Cognitive Systems: Foundations of CognitiveSystems Engineering. CRC Press, Inc., Boca Raton, FL.

Hughes, J., O’Brien, J., Rodden, T., Rouncefield, M., 1997. Ethnography, Communi-cation and Support for Design. Technical Report Ref. CSEG/24/1997. LancasterUniversity, Lancaster, UK.

Hutchins, E., 1995. Cognition in the Wild. MIT Press, Cambridge, MA.Marmaras, N., Pavard, B., 1997. A methodological framework for development and

evaluation of systems supporting complex cognitive tasks. Journées Européneesdes Techniques de I`Informatique 8, 13e20.

Millen, D., 2000. Rapid Ethnography: Time Deepening Strategies for HCI Field. Proc.ACM Symposium on Designing Interactive Systems. ACM Press, New York, pp.280e286.

Nachreiner, F., Nickel, P., Meyer, I., 2006. Human factors in process control systems:the design of humanemachine interfaces. Safety Science 44, 5e26.

O’Hara, J., Higgins, J., Stubler, W., Goodman, C., Eckinrode, R., Bongarra, J., Galletti, G.,1994. Human Factors Engineering Review Program Model (NUREG-0711 Rev.1).US Nuclear Regulatory Commission, Washington.D.C.

O’Hara, J., Brown, M., 2004. Incorporation of Human Factors Engineering Analysesand Tools into the Design Process for Digital Control Room Upgrades(BNL-72801-2004-CP). Brookhaven National Laboratory, New York.

O’Hara, J., Roth, E., 2005. Operational concepts, teamwork, and technology incommercial nuclear power stations. In: Bowers, Clint, Salas, Eduardo,Jentsch, Florian (Eds.), Creating High-Tech Teams: Practical Guidance on WorkPerformance and Technology. American Psychological Association, Washington,D. C, pp. 135e159.

Parasuraman, R., Sheridan, T.B., Wickens, C.D., 2000. A model for types and levels ofhuman interaction with automation. IEEE Transactions on Systems, Man, andCybernetics. Part A: Systems and Humans 30, 286e297.

Roth, E., Christian, C., Gustafson, M., Sheridan, T., Dwyer, K., Gandhi, T., Zinner, M.,Dierks, M., 2004. Using field observations as a tool for discovery: analysingcognitive and collaborative demands in the operating room. Cognition, Tech-nology and Work 6, 148e157.

Santos, I.J.A., Teixeira, D.V., Ferraz, F.T., Carvalho, P.V.R., 2007. The use of a simulatorto include human factors issues in the interface design of a nuclear power plantcontrol room. Journal of Loss Prevention in the Process Industries 21 (3),227e238.

Sheridan, T.B., 2002. Humans and Automation System Design and Research Issues.Wiley/HFES, Santa Monica.

Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., Jenkins, D.P., 2005. HumanFactors Methods: A Practical Guide for Engineering and Design. AshgatePublishing Ltd, London.

U. S. Nuclear Regulatory Commission, NUREG 711 (Revision 1), 2002. Human FactorsEngineering Program Review Model. Office of Nuclear Regulatory ResearchWashington, DC 20555-0001.

Vicente, K., Mumaw, R., Roth, E., 1997. Cognitive Functioning of Control RoomOperators e Final Phase, AECB 96e175. Atomic Energy Canadian Bureau,Ottawa, Ontario, Canada.

Vidal, M.C., Carvalho, P.V.R., Santos, M., Santos, I.J.A., 2009. Collective work andresilience of complex systems. Journal of Loss Prevention in the ProcessIndustries 22, 537e548.