Available online at www.sciencedirect.com

Safety Science 46 (2008) 1115–1124

www.elsevier.com/locate/ssci

Predicting work performance in nuclear power plants

Sheue-Ling Hwang a,*, Yi-Jan Yau a, Yu-Ting Lin a, Jun-Hao Chen a,Tsun-Hung Huang a,b, Tzu-Chung Yenn c,1, Chong-Cheng Hsu c,1

a Department of Industrial Engineering and Engineering Management, National Tsing Hua University, 101, Section 2,

Kuang-Fu Road, Hsinchu 300, Taiwan, ROCb Department of Industrial Engineering and Management, National Chin-Yi Institute of Technology, 35, Lane 215,

Section 1, Chung-Shan Road, Taiping, Taichung 411, Taiwan, ROCc Institute of Nuclear Energy Research, Atomic Energy Council, Executive Yuan No. 1000, Wunhua Road, Jiaan Village,

Longtan Township, Taoyuan County 325, Taiwan, ROC

Received 4 December 2006; received in revised form 21 May 2007; accepted 14 June 2007

Abstract

Diagnosis and monitoring are the major tasks of an operator in main control room of nuclear power plants (NPPs). Theoperator’s mental workload influences his/her performance, and furthermore, affects the system safety and operations.This study investigated the operator’s mental workload and work performance of the NPP in Taiwan. An experimentincluding primary and secondary tasks was designed to simulate the reactor shutdown procedure of the fourth nuclearpower plant (FNPP). The performance of the secondary tasks (error rate), subjective mental workload (NASA Task LoadIndex, NASA-TLX) as well as seven physiological indices were assessed and measured. The group method of data han-dling (GMDH) was applied to integrate these physiological indices to develop a work performance predictive model.The validity of the proposed model is very well with R2 = 0.84 and its prediction capability is high (95% confidence inter-val). The proposed model is expected to provide control room operators a reference value of their work performance bygiving physiological indices. Besides NPPs, the proposed model can be applied to many other fields, e.g. aviation, air trans-portation control, driving and radar vigilance, etc.� 2007 Elsevier Ltd. All rights reserved.

Keywords: Predictive; Mental workload; Physiological; GMDH; NPPs

1. Introduction

Workload has long been considered as an important factor influencing individual performance withincomplex systems (O’Donnell and Eggemeier, 1986; Xie and Salvendy, 2000). It can be classified into physical

0925-7535/$ - see front matter � 2007 Elsevier Ltd. All rights reserved.

doi:10.1016/j.ssci.2007.06.005

* Corresponding author. Tel.: +886 3 5742694; fax: +886 3 5722685.E-mail addresses: slhwang@ie.nthu.edu.tw (S.-L. Hwang), d927814@oz.nthu.edu.tw (Y.-J. Yau), d943847@oz.nthu.edu.tw (Y.-T.

Lin), g943848@oz.nthu.edu.tw (J.-H. Chen), toby.hwang@gmail.com (T.-H. Huang), tcyenn@iner.gov.tw (T.-C. Yenn), worm@iner.gov.tw (C.-C. Hsu).

1 Tel.: +886 3 4711400; fax: +886 3 4712358.

1116 S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124

workload and mental workload. Notwithstanding the fact that there is no universally acceptable definition ofmental workload, it has been generally defined as the amount of resource difference between task demands andcapacity provision by an individual (O’Donnell and Eggemeier, 1986; Sanders and McCormick, 1993;Veltman and Gaillard, 1996; Xie and Salvendy, 2000). Modern complex systems such as nuclear power plant,air flight control systems and weapon systems very often bring heavy mental workload to their operators. Thehigh rate of information flow, complexity of the information, numerous hard decisions and task time stresscould overwhelm the operators. On the other hand, high levels of automation could contribute to low mentalworkload due to complacency or inattention (Billings, 1997; Wilson and Russell, 2003). For most operators,both excessive and low mental workloads could degrade their performance (Lysaght et al., 1989; Nachreineret al., 2006; Xie and Salvendy, 2000), and furthermore, may affect the safety and proper functioning of thesystem. Only with an appropriate level of mental workload, can the operators perform as anticipated (Moray,1988). The relationship between an operator’s performance and mental workload is similar to an upside downU shape where the best performance occurs at a level of reasonable mental workload. Therefore, for the sakeof system safety and normal functions, developing an early warning system based on mental workload topredict the operator’s performance is very important and helpful.

Various mental workload measurements have been proposed, and these measurements could be dividedinto three categories: performance measure, subjective ratings and physiological measures. Their pros andcons have been widely investigated in many studies (Charlton, 2002; Farmer and Brownson, 2003; Veltmanand Gaillard, 1996). Furthermore, numerous studies have found that the physiological measure was very suit-able for measuring real-time mental workload since it could record the continuous data, and it is less intrusiveon work activities and shows a high sensitivity to the cognitive requirements of complex tasks (Guhe et al.,2005; Miyake, 2001; Sanders and McCormick, 1993; Veltman and Gaillard, 1996; Wilson, 2002b; Wilsonand Russell, 2003). Several studies have also proved the sensitivity of physiological signals and mental work-load, but most of them focused on the interface design phase rather than on the operation phase (e.g. Nachre-iner et al., 2006; Wastell and Newman, 1996). For the purpose of a safe and efficient operation of complexsystems, it is important to monitor the operator’s work performance by measuring his/her mental workloadduring the system operation phase. However, there has been little use of real-time physiological signals todynamically measure an operator’s mental workload and develop a work performance predictive model dur-ing system operation (Chen and Vertegaal, 2004).

Miyake (2001) indicated that the responses of physiological signals to a mental task were different for eachperson and the physiological response pattern was also different from task to task. Thus, it is necessary to con-sider these situations when we try to develop a real-time work performance predictive model by dynamicallymeasuring physiological signals. One approach to solving these problems is to record several physiologicalindices and to integrate them into one synthesized index by individual differences, and then to analyze it imme-diately (Chen and Vertegaal, 2004; Guhe et al., 2005).

Numerous studies have successfully used psychophysiological signals such as electroencephalogram (EEG)to classify the state of operator’s mental workload (Chen and Vertegaal, 2004; Gevins and Smith, 1999;Wilson and Russell, 2003). Their results showed that the average rate of successful classification was over80%. However, Farmer and Brownson (2003) indicated that using EEG to classify mental workload wasnot practical since it was difficult to analyze data, had a high noise-to-signal ratio, required calibrating to eachindividual and trained person to supervise, etc. Therefore, it is not suitable for using in real life. Guhe et al.(2005) used Bayesian Network (BN) to fuse various measures such as facial features and physiological signalsto measure workload in real time. Nevertheless, only three subjects participated in their experiment and nofurther validation limited its usability. Furthermore, the conditional probability table which comprised manycritical parameters needed in BN model had individual differences, and it was not easy obtainable in real life.

In this study, we conducted an experiment to simulate the reactor shutdown task in the control room ofFNPP with a secondary task to assess the relation between work performance, mental workload and physi-ological indices. According to these relationships, we developed a real-time, non-intrusive work performancepredictive model by using group method of data handling (GMDH) to integrate seven physiological indicesinto a synthesized index. Comparing this index with a performance level set by operators/supervisor, whenthe value of the index is lower than the set performance level, the early warning system will alert the operatorsto adjust their workloads. The physiological indices used in this study were the parasympathetic/sympathetic

S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124 1117

ratio (LF/HF), heart rate, heart rate variability (HRV), diastolic pressure, systolic pressure, eye blink fre-quency and eye blink duration. The parasympathetic/sympathetic ratio (LF/HF), heart rate, diastolic and sys-tolic pressure typically increase with a higher level of mental workload. By contrast, the heart rate variabilitydecreases while the mental workload increases. The eye blink frequency and blink duration tend to decreasewith the increase of mental workload (Chen and Vertegaal, 2004; Noel et al., 2005; Veltman and Gaillard,1996, 1998; Wastell and Newman, 1996; Wilson, 2002b).

2. Method

2.1. Subjects

Fifteen paid graduate students (eight males and seven females) of National Tsing Hua University partici-pated in this experiment. The mean and standard deviation of the subject’s age was 24.4 ± 1.3 years. All ofthem had normal eyesight and good health.

2.2. Apparatus

Meditech ABPM-04 ambulatory blood pressure measuring device was used to measure the heart rate andblood pressure (systolic pressure and diastolic pressure). ANSWatch TS0411 was used to measure heart ratevariability (HRV) and parasympathetic/sympathetic ratio (LF/HF). Face lab version 4.0 was used to measurethe blink frequency and blink duration. The Personal Computer Transient Analyzer (PCTRAN) systemincluding re-circulation flow controller and rod control systems was used to simulate the shutdown reactortask. The details of PCTRAN system was illustrated in Huang et al. (2006).

2.3. Experimental tasks and variables

The PCTRAN system was used to simulate the primary task which was to shut down the reactor accordingto the procedure provided by the Institute of Nuclear Energy Research (INER). The procedure of the primarytask was to monitor the parameters of core flow and power at some reference points by turning down the re-circulation internal pump (RIP) and run-in nuclear rods by turns. Fig. 1 shows the monitor interface (power-core flow map) of the primary task. The primary task included three levels of task complexity: high, medianand low which corresponded to high, median and low core flow speeds and power speeds. Since safety was the

Fig. 1. The power-core flow map.

1118 S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124

most important issue in the task of ‘‘shutting down the reactor” and the performance of such task was not easyto measure, we introduced a secondary task to this experiment and measured its performance as a responseparameter for developing the work performance predictive model. The secondary task, a mental arithmetictask, was a comparison of two different integers with no decimal. The high, median and low task complexitylevels of the secondary task were designed by different complexity of calculations such as making a comparisonof the product of two integers or sum of two integers with another integer, or direct comparison of two indi-vidual integers respectively. The arithmetic task was developed by visual C++ and its duration of each com-plexity level was as long as the same level of primary task. No matter in high, median or low task complexitylevel, the time interval of each comparison was 15 s, and each comparison had 10 s for processing. A ‘‘beep”

was sounded while the arithmetic task appeared and a ‘‘tin” was sounded if it was not responded in 10 s. Thesubject used his/her left hand to press the Q, W or E key which represented the” >”, ‘‘=” and ‘‘ < ” to respondto the number comparison respectively.

We combined high task complexity of the primary task with that of the secondary task as the high taskcomplexity phase of this experiment, and so forth. There were three phases with high, median and low levelsof task complexity in our experiment. Each phase lasted for about 30 min and had 5 min break after eachphase. The primary and secondary tasks were presented in four monitors and their layout is shown inFig. 2. The performance of the secondary task was measured by the error rate of the number comparisons.The subjective mental workload was rated by the NASA Task Load Index (NASA-TLX) questionnaire.The physiological indices such as LF/HF ratio (X1), HRV (X2), heart rate (X3), systolic pressure (X4), diastolicpressure (X5), eye blink frequency (X6), and eye blink duration (X7) were measured before and during eachphase of the experiment.

2.4. Procedure

Before the experiment, the subject took a 15 min rest, and then wore the measurement apparatus and pro-ceeded with Face Lab adjustment. The initial physiological indices were acquired as a base line before theexperiment. After the adjustment and measurements, we explained the experimental tasks and instructedthe participants to operate the system. In order to let the subjects focus their attention on the primary task,we emphasized the importance of the primary task to the subjects and also asked them to respond to the sec-

Fig. 2. The interface layouts of primary (PCTRAN system) and secondary tasks.

S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124 1119

ondary task as quickly and accurately as possible prior to the experiment. The participants took about 30 minto practice the control procedure and familiarize themselves with the location of the information displays.During experiment, the subjects had to shut down the reactor and complete a series of mathematical opera-tions and integer comparisons simultaneously. The physiological indices were collected during each phase andthe NASA-TLX questionnaire was conducted after each phase to evaluate the subjective mental workload ofdifferent levels of task complexity.

2.5. Analysis

The error rate of the series of mathematical comparisons at each phase was collected to compare with thecorresponding scores of the NASA-TLX questionnaire. The higher score of the NASA-TLX questionnairemeans a higher level of mental workload (Charlton, 1996; Rubio, 2004; Hart and Staveland, 1988). The Cor-relation between the performance of the secondary task (error rate) and the evaluation of the NASA-TLXquestionnaire was analyzed by the Statistical Products and Services Solution (SPSS).

The proposed model is based on the group method and data handling (GMDH), which is one of the betterknown neural network methodologies. It was developed in 1971 (Ivakhnanko, 1971) and is one of the mostexact prediction methods in the last decade. Furthermore, neural networks can be regarded as a statisticalmethod and be constructed without any assumptions around the relationship between predictors and response(Stern, 1996). The GMDH algorithm has been widely used in various fields, e.g. education (Pavel and Miro-slav, 2003), business, and vehicle factory (Tomonori and Shizue, 1982).

This study investigated the relationship between seven physiological indices and work performance on dif-ferent levels of task complexity. These physiological indices (X1–X7) and the performance of the secondarytask were collected as input variables and then created a model to predict operator’s work performance.

3. Results

Each participant completed the primary task of each phase in about 30 min and successfully decreased thecore flow and power rates from 100% to 0% in the safe area (see Fig. 1), and thus differences in mental work-load would be reflected by the performance of the secondary task.

3.1. Correlation between subjective mental workload and the secondary task performance

The analysis of Pearson-product moment correlation was used to examine the relationship between tworesponse variables (NASA-TLX scores and the error rates) as shown in Table 1. It indicated that the errorrates and the subjective mental workload were positively correlated with each other. The correlation coefficientof r = 0.691 was found to be statistically significant at p < 0.01 (two-tailed). It was inferred from these resultsthat the more mistakes the subject made, the higher the score achieved on the NASA-TLX questionnaire usedto rate the subject. As a result, the increase in the number of errors in the secondary task implied that the sub-jects felt a heavier mental workload.

For the sake of providing a picture of the relationship, one may consider the trend curve in Figs. 3 and 4. Ingeneral, as shown in Figs. 3 and 4, the directions of two trend lines of error rates and the scores of NASA-TLX

Table 1The analysis of Pearson-product moment correlation

Correlations Error rate of the secondary task Score of subjective mental workload

Error rate of the secondary task Pearson Correlation 1 0.691a

Significance (two-tailed) – 0.000N 45 45

Score of subjective mental workload Pearson Correlation 0.691a 1Significance (two-tailed) 0.000 –N 45 45

a Correlation is significant at the 0.01 level (two-tailed).

Fig. 3. The scores of NASA-TLX and error rates of the secondary task of each subject.

Fig. 4. The relationship between the scores of NASA-TLX and error rates of the secondary task on different task complexity.

1120 S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124

were nearly the same. Almost all of the subjects made fewer errors and achieved a lower score on the NASAquestionnaire as the secondary task became less complex.

3.2. Model construction

3.2.1. Two levels of high and low task complexity

The hidden relationship between physiological indices (Xi) and performance (Y) needed to be found todevelop an early warning system. Seven physiological indices, including an LF/HF ratio (X1), HRV (X2), heartrate (X3), systolic pressure (X4), diastolic pressure (X5), eye blink frequency (X6), and eye blink duration (X7)were recorded in high and low levels of task complexity. To eliminate the effects of individual differences, theseseven physiological indices were transferred to the values (individual change rate) of variability ranging from�1 to 1. The accuracy (Y, %) was used to evaluate the performance of the secondary task.

S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124 1121

The physiological indices X1–X7 were used as inputs in the network system to obtain an output Y rangingfrom 0 to 100. GMDH was used to establish the work performance predictive model. Twenty-six sets of datafrom 13 subjects were used to construct the model using NeuroShell software 2.

The experimental result indicated that the physiological indices of X2, X3, X4 and X7 were the significantpredictor factors in the performance by the subject, but other indices were not. In addition, given the valuesfor X2, X3, X4 and X7, a work performance predictive model was yielded and expressed by the Eq. (1):

TableModel

No.

14

15

TableModel

No.

14

15

Y ¼ 140X 7 þ 32X 3 þ 8:4X 4 þ 87� 5:6X 2 þ 21X 22 � 290X 2

4 � 730X 27 � 9:7X 3

2 � 710X 34 þ 500X 3

7

� 82X 2X 4 þ 100X 2X 7 þ 730X 4X 7 þ 110X 23 � 1500X 3

3 ð1Þ

In Eq. (1), X2, X3, X4 and X7 respectively represented HRV, heart rate, systolic pressure and eye blink dura-tion. Also, mean square error (MSE) of the model was 1.41 and R-square (R2) of the model was 0.85.

After the predictive model was constructed, the reliability of the model needed to be further validated. Twosubjects with four sets of data were used to validate the accuracy of the proposed model. The result of thevalidation is shown in Table 2. The differences between the estimated values and real values were large,and furthermore, most of the real values were out of the 95% confidence intervals. It seemed that the modelconstructed with data of high and low levels of task complexity was not appropriate to predict the work per-formance of different mental workload. The reason might be due to the fewer physiological indices used in themodel construction. It could be possible to construct a more accurate and reliable model for work perfor-mance prediction if more physiological indices were used

3.2.2. Three levels of high, median and low task complexity

Besides the data of high and low levels of task complexity, the data of median level of task complexity wereadded to construct the predictive model. There were 39 sets of data in total from 13 subjects operating withhigh, median and low levels of task complexity. The result showed that all physiological indices of X1–X7 weresignificant predictor variables for the predicting of work performance. Also, given the values for X1–X7, awork performance predictive model was yielded and expressed by the Eq. (2):

2validation (data of high and low levels of task complexity)

Mentalworkload

X1 X2 X3 X4 X5 X6 X7 Estimativevalue(accuracy, %)

Real value(accuracy,%)

Lowbound of95% CI

Upperbound of95% CI

High 0.10 �0.61 0.03 0.05 �0.14 0.20 �0.07 91.71 81 89.214 94.206Low 0.00 �0.52 �0.05 0.15 �0.17 �0.03 0.00 94.57 95 92.074 97.066

High �0.33 �0.30 0.00 0.23 �0.04 �0.03 �0.05 58.51 76 56.014 61.006Low �0.67 0.14 �0.15 �0.15 0.01 0.25 0.04 86.35 95 83.854 88.846

3validation (data of high, median and low levels of task complexity)

Mentalworkload

X1 X2 X3 X4 X5 X6 X7 Estimativevalue(accuracy, %)

Real value(accuracy,%)

Lowbound of95% CI

Upperbound of95% CI

High 0.10 �0.61 0.03 0.05 �0.14 0.20 �0.07 83.71 81 77.491 89.929Median 0.00 �0.45 0.02 0.19 �0.12 �0.01 0.02 85.19 83 78.971 91.409Low 0.00 �0.52 �0.05 0.15 �0.17 �0.03 0.00 90.85 95 84.631 97.069

High �0.33 �0.30 0.00 0.23 �0.04 �0.03 �0.05 77.15 76 70.931 83.369Median �0.60 �0.22 �0.11 �0.10 �0.05 0.14 �0.03 95.86 91 89.641 102.079Low �0.67 0.14 �0.15 �0.15 0.01 0.25 0.04 96.44 95 90.221 102.659

1122 S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124

Y ¼ 4:5X 6 � 33X 4 þ 87� 14X 3 � 0:52X 1 þ 79X 7 þ 590X 23 � 220X 2

4 � 120X 27 � 1900X 3

3 þ 1100X 34

� 270X 3X 4 þ 840X 3X 7 þ 18X 22 � 34X 2

6 � 7:1X 32 þ 6:1X 2X 6 þ 130X 4X 7 � 39X 1X 5 ð2Þ

In Eq. (2), the MSE of the model was 9.04 and R-square (R2) of the model was 0.84. Such an equation wasexpected to provide supervisors a better predicative value of performance (Y) by giving inputs X1–X7.

For validation, six sets of data come from two subjects were put into the model. The relational informationwas described in Table 3. In this model, the six estimated values were very close to the real values and the realvalues were all in the 95% confidence intervals. Therefore, this model was more suitable and accurate to esti-mate the performance of different mental workload than the previous one.

4. Discussions

4.1. NASA-TLX mental workload assessment and the secondary task performance

The subjective mental workload assessment (NASA-TLX scores) showed a significant correlation with thedifferent levels of mental workload of the integrated task. For most of the subjects, the highest NASA-TLXscore occurred in the high task complexity phase whereas the lowest score happened in the low task com-plexity phase. This result indicated that the primary and secondary tasks used in this experiment could dis-tinguish the different levels of mental workload. On the other hand, although numerous studies have foundthat the subject’s performance was affected when the mental workload was low (Billings, 1997; Wilson andRussell, 2003), this situation did not appear in this study. It was probably because the experimental time wasnot long enough and the tasks were not boring; hence, the subjects could still maintain their good situationawareness.

4.2. Physiological indices

Compared with previous studies concerned with measuring mental workload, some physiological indiceswere significantly affected by the mental workload. In this study, we established an accurate and real-timework performance predictive model by integrating seven physiological indices. The physiological indices usedin this experiment included heart rate, heart rate variability, blood pressure (systolic pressure and diastolicpressure), parasympathetic/sympathetic ratio (LF/HF ratio), eye blink frequency and eye blink duration.The experimental result indicated that most of the participants’ heart rate and LF/HF components increasedwhen the task complexity increased. On the contrary, the heart rate variety (HRV) decreased when the taskcomplexity increased. These findings were consistent with previous studies (Veltman and Gaillard, 1996; Wil-son, 2002b). On the other hand, many participants’ blood pressure was not increased as the task complexityincreases. This was not consistent with the previous studies (Veltman and Gaillard, 1996; Wastell and New-man, 1996). The experimental result also showed that most of the participants’ eye blink duration was shorterand eye blink frequency was less during the high task complexity phase than during the low task complexityphase. The reason could be that the subjects spent more time watching the interface which therefore, shortenedthe blink duration and decreased the blink frequency. These findings were also similar to many previous stud-ies (Wilson, 2002a,b; Veltman and Gaillard, 1998; Wastell and Newman, 1996).

4.3. Neural networks

This study used GMDH to construct a model to predict the operator’s work performance on differentmental workload. Although there were only thirty-nine sets of data used in model development, all predictingvalues fell in the 95% confidence intervals. This result showed that the proposed model could predict theoperator’s work performance accurately in three task complexity levels. One important characteristic of neuralnetworks is that the more training data is available, the more accurate predictive model becomes, and which isthen more feasible to apply in real life. For future model development, more training data and physiologicalindices should be used and added to model development.

S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124 1123

4.4. Model comparison

In comparing Eq. (1) with Eq. (2), the predictive capability of Eq. (2) is better than that of Eq. (1). Even ifEq. (2) integrated seven physiological indices rather than four indices of Eq. (1), all thirty-nine data were takento construct the work performance predictive model and therefore enhance the model reliability. Wilson andRussell (2003) and Guhe et al. (2005) indicated that more physiological indices could lead to a more accurateprediction. More data training through neural networks algorithm can also enhance the accuracy and reliabil-ity of the model. As a result, the model constructed with the data of physiological indices and performance ofthe secondary task could provide a reliable reference tool to predict the work performance of operators.

5. Conclusions

Mental workload is an important factor that may affect the operating performance. The operator’s perfor-mance will be negatively affected when the mental workload is either too heavy or too low. The model devel-oped in this study can on line predict the operator’s work performance by measuring physiological indices.The validity of this model was very well with R2 = 0.84 and its prediction capability were very high (95% con-fidence interval). For further practical application in different fields such as in the military, aviation, flighttransportation control, driving and radar vigilance, etc., more data or physiological indices are necessary tobe added to this model.

Acknowledgements

This research has been supported by the Institute of National Science Council of Taiwan, Republic of Chi-na (Project No. NSC95-NU-7-007-003). The authors wish to thank for the help provided by the Institute ofNuclear Energy Research.

References

Billings, C.E., 1997. Some questions about advanced automation. In: Mouloua, M., Koonce, J.M. (Eds.), Human–automationInteraction. Erlbaum, Mahwah, NJ, pp. 314–320.

Charlton, S.G., 1996. Mental workload test and evaluation. In: O’Brein, T.G., Charlton, G. (Eds.), Handbook of Human Test andEvaluation. Lawrence Erlbraum Associates, Mahwah, NJ, pp. 181–199.

Charlton, S.G., 2002. Measurement of cognitive states in test and evaluation. In: Charlton, S.G., O’Brien, T.G. (Eds.), Handbook ofHuman Factors Testing and Evaluation. Lawrence Erlbraum Associates, Mahwah, NJ, pp. 97–126.

Chen, D., Vertegaal, R., 2004. Using mental load for managing interruptions in physiologically attentive user interfaces. In: Paperpresented at the CHI, Vienna, Austria.

Farmer E., Brownson, A., 2003. Review of workload measurement, analysis and interpretation methods (No. CARE-integra-TRS-130-02-WP2).

Gevins, A., Smith, M.E., 1999. Detecting transient cognitive impairment with EEG pattern recognition methods. Aviation, Space, andEnvironmental Medicine 70 (10), 1018–1024.

Guhe, M., Liao, W., Zhu, Z., Ji, Q., Gray, W.D., Schoelles, M.J., 2005. Non-intrusive measurement of workload in real-time. In: Paperpresented at the Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, FL.

Hart, S.G., Staveland, L.E., 1988. Development the NASA-TLX (task load index): results of empirical and theoretical research. In:Hncock, P.A., Meshkati, N. (Eds.), Human Metal Workload. North Holland Press, Amsterdam, pp. 239–250.

Huang, F., Hwang, S., Yenn, T., Yu, Y., Hsu, C., Huang, H., 2006. Evaluation and comparison of alarm reset modes in advanced controlroom of nuclear power plants. Safety Science 44, 935–946.

Ivakhnanko, A.G., 1971. Polynomial theory of complex systems. IEEE Transactions on Systems, Man, and Cybernetics SMC 1 (4), 363–378.

Lysaght, R.J., Hill, S.G., Dick, A.O., Plamondon, B.D., Linton, P.M., Wierwille, W.W., 1989. Operator workload: Comprehensive reviewand evaluation of workload methodologies: US Army Research Institute for the Behavioral and Social Sciences, Willow Grove.

Miyake, S., 2001. Multivariate workload evaluation combining physiological and subjective measures. International Journal ofPsychophysiology 40, 233–238.

Moray, N., 1988. Mental workload since 1979. International Review of Ergonomics 2, 123–150.Nachreiner, F., Nickel, P., Meyer, I., 2006. Human factors in process control systems: The design of human–machine interfaces. Safety

Science 44, 5–26.

1124 S.-L. Hwang et al. / Safety Science 46 (2008) 1115–1124

Noel, J.B., Bauer, K.W., Lanning, J.W., 2005. Improving pilot mental workload classification through feature exploitation andcombination: a feasibility study Computers & Operations Research 32, 2713–2730.

O’Donnell, R.D., Eggemeier, F.T., 1986. Workload assessment methodology. In: Boff, K., Kaufman, L., Thomas, J.P. (Eds.), Handbookof Perception and Human Performance. Wiley-Interscience, New York, pp. 42.41–42.49.

Pavel, N., Miroslav, S., 2003. Modeling of student’s quality by means of GMDH algorithms. Systems Analysis Modeling Simulation 43(10), 1415–1426.

Rubio, S., Diaz, E., Martin, J., Puente, J.M., 2004. Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, andworkload profile methods. Applied Psychology: An international review 53 (1), 61–86.

Sanders, M.S., McCormick, E.J., 1993. Human Factors in Engineering and Design. MC Graw Hill, New York.Stern, H.S., 1996. Neural networks in applied statistics (with discussion). Technometrics 38, 205–220.Tomonori, N., Shizue, S., 1982. Identification and forecasting in management systems using the GMDH method. Appl. Math. Modeling

6, 7–15.Veltman, J.A., Gaillard, A.W.K., 1996. Physiological indices of workload in a simulated flight task. Biological Psychology 42, 323–342.Veltman, J.A., Gaillard, A.W.K., 1998. Physiological workload reactions to increasing levels of task difficulty. Ergonomics 41 (5), 656–

669.Wastell, D.G., Newman, M., 1996. Stress, control and computer system design: a psychophysiological field study. Behaviour &

Information Technology 15 (3), 183–192.Wilson, G.F., 2002a. Psychophysiological test methods and procedures. In: Charlton, S.G., O’Brien, T.G. (Eds.), Handbook of Human

Factors Testing and Evaluation. Lawrence Erlbraum Associates, Mahwah, NJ, pp. 127–156.Wilson, G.F., 2002b. An analysis of mental workload in pilots during flight using multiple psychophysiological measures. The

International Journal of Aviation Psychology 12 (1), 3–18.Wilson, G.F., Russell, C.A., 2003. Real-time assessment of mental workload using psychophysiological measures and artificial neural

networks. Human Factors 45 (4), 635–643.Xie, B., Salvendy, G., 2000. Review and reappraisal of modeling and predicting mental workload in single-and multi-task environments.

Work & Stress 14 (1), 74–99.