15
ARTICLES The Role of Employee Reactions in Predicting Training Effectiveness James A. Tan, Rosalie J. Hall, Carol Boyce Reactions to training programs tend to be poor predictors of training success, yet most training programs are evaluated based solely on trainee reactions. In this study, we proposed that distinguishing between affec- tive and cognitive employee reactions may improve the prediction of trainee learning. Our results indicated that cognitive employee reactions are related to both employee learning and employee behavior. Moreover, contrary to popular notion, negative affective reactions best predicted employee learning. Implications and future research directions of the results are discussed. As the workplace undergoes sweeping changes, such as the pressure of more global competition (Cascio, 1995), the need for strategies to keep up with these changes is increasingly important. Organizational training is a method used to enhance individual productivity as well as a company’s success. An organization’s investment in human capital through the education and train- ing of its members is therefore a central component of competitive strategy (Bassi & Van Buren, 1998). Organizations often evaluate training effectiveness using one or more of Kirkpatrick’s criteria (1959a, 1959b, 1960a, 1960b). Although other approaches to training evaluation have been proposed (Day, Arthur, & Gettman, 2001; Kraiger, Ford, & Salas, 1993; Moore, Blake, Phillips, & McConaughy, 2003), training success has most often been evaluated using paper-and-pencil measures (that is, reaction and learning measures; Geber, 1995; Saari, Johnson, McLaughlin, & Zimmerle, 1988; Salas & Cannon- Bowers, 2001; Van Buren & Erskine, 2002). Research investigating the effect of various training design and workplace environment factors on workplace learning and transfer has been covered elsewhere (see Russ-Eft, 2002). In this article, we address issues concerning the relation of paper-and-pencil measures 397 HUMAN RESOURCE DEVELOPMENT QUARTERLY, vol. 14, no. 4, Winter 2003 Copyright © 2003 Wiley Periodicals, Inc.

The role of employee reactions in predicting training effectiveness

Embed Size (px)

Citation preview

Page 1: The role of employee reactions in predicting training effectiveness

A R T I C L E S

The Role of Employee Reactionsin Predicting TrainingEffectiveness

James A. Tan, Rosalie J. Hall, Carol Boyce

Reactions to training programs tend to be poor predictors of trainingsuccess, yet most training programs are evaluated based solely on traineereactions. In this study, we proposed that distinguishing between affec-tive and cognitive employee reactions may improve the prediction of traineelearning. Our results indicated that cognitive employee reactions arerelated to both employee learning and employee behavior. Moreover,contrary to popular notion, negative affective reactions best predictedemployee learning. Implications and future research directions of the resultsare discussed.

As the workplace undergoes sweeping changes, such as the pressure of moreglobal competition (Cascio, 1995), the need for strategies to keep up withthese changes is increasingly important. Organizational training is a methodused to enhance individual productivity as well as a company’s success. Anorganization’s investment in human capital through the education and train-ing of its members is therefore a central component of competitive strategy(Bassi & Van Buren, 1998).

Organizations often evaluate training effectiveness using one or moreof Kirkpatrick’s criteria (1959a, 1959b, 1960a, 1960b). Although otherapproaches to training evaluation have been proposed (Day, Arthur, &Gettman, 2001; Kraiger, Ford, & Salas, 1993; Moore, Blake, Phillips, &McConaughy, 2003), training success has most often been evaluated usingpaper-and-pencil measures (that is, reaction and learning measures; Geber,1995; Saari, Johnson, McLaughlin, & Zimmerle, 1988; Salas & Cannon-Bowers, 2001; Van Buren & Erskine, 2002). Research investigating the effectof various training design and workplace environment factors on workplacelearning and transfer has been covered elsewhere (see Russ-Eft, 2002). In thisarticle, we address issues concerning the relation of paper-and-pencil measures

397HUMAN RESOURCE DEVELOPMENT QUARTERLY, vol. 14, no. 4, Winter 2003Copyright © 2003 Wiley Periodicals, Inc.

Page 2: The role of employee reactions in predicting training effectiveness

398 Tan, Hall, Boyce

of employee reactions to training with the actual learning imparted by thetraining program.

Training Evaluation Criteria

Training evaluation is a system for measuring changes due to traininginterventions—most important, whether trainees have achieved learning out-comes (Goldstein & Ford, 2002; Kraiger et al., 1993; Sackett & Mullen, 1993).Few, if any, single measures can actually capture the complexity of training pro-grams and transfer performance. Although the issue of conceptually basedmodels to guide researchers in evaluating training effectiveness is still in itsinfancy, some advances have been made in this area recently (Noe & Colquitt,2002; Preskill & Russ-Eft, 2003; Russ-Eft & Preskill, 2001). Russ-Eft andPreskill (2001) presented alternative methods of evaluation (for example,developmental evaluation). More recently, Preskill and Russ-Eft (2003) pro-posed a model based on the philosophies, theories, and practices in variousfields such as evaluation, organizational learning, and systems thinking.

Despite these advances in training evaluation theory, the most developedand used of these models in the human resource literature focuses on definingdifferent training effectiveness criteria and their organizational implications.Common to these models is the use of multiple criteria to gather data on train-ing effectiveness. As such, training evaluators need to view criteria as multidi-mensional in nature (Campbell, McCloy, Oppler, & Sager, 1993; Kraiger et al.,1993). For example, Kraiger et al. argued that learning outcomes can bedivided into three types: cognitive (which reflect knowledge and cognitivestrategies), skill based (reflecting constructs such as automaticity and compi-lation), and affective (reflecting constructs such as attitudes and motivation).These outcomes can be seen as analogous to Kirkpatrick’s learning, behavior,and reactions criteria, respectively.

Varieties of Evaluation Criteria. Kirkpatrick (1959a, 1959b, 1960a,1960b) outlined four steps or levels of measures of the effectiveness of trainingoutcomes. Level 1, or reaction, is trainees’ feelings for and liking of a train-ing program. Reaction measures may indicate the trainee’s motivation to learn;although positive reactions may not ensure learning, negative reactionsprobably reduce the possibility that learning occurs. A focus of this articleis suggested improvements in the construction, expected relations, andinterpretations of reactions measures.

Level 2, learning, was defined as the “principles, facts, and techniquesunderstood and absorbed by the [trainees]” (Alliger & Janak, 1989, p. 331).No change in behavior can be expected unless one or more of these learningobjectives have been accomplished (Kirkpatrick, 1994). Learning is most oftenassessed by giving the trainees tests that tap declarative knowledge (Kraigeret al., 1993). Level 3, behavior, defined as transferring knowledge, skills, andattitudes learned during the training to the job (Kirkpatrick, 1994), is typically

Page 3: The role of employee reactions in predicting training effectiveness

The Role of Employee Reactions in Predicting Training Effectiveness 399

measured on the job after a particular amount of time has passed. This is mostoften assessed through performance appraisal. Level 4, results, was defined asthe final results that occurred because the participants attended the program(Kirkpatrick, 1994). These could include increased production, improvedquality, decreased costs, reduced frequency and severity of accidents, increasedsales, reduced turnover, and higher profits and return on investments.

Relations Among Different Types of Criteria. Kirkpatrick’s modelhas been misunderstood by researchers and practitioners to be hierarchical(Alliger & Janak, 1989; Russ-Eft & Preskill, 2001). Specifically, the followingassumptions arose: (1) each succeeding level of evaluation criteria is moreinformative (or “better” in terms of information obtained for the organization;Russ-Eft & Preskill, 2001) than the last, (2) each level is caused by thepreceding level, and (3) each succeeding level is correlated with the previouslevel. Using meta-analysis, Alliger and Janak (1989) clarified these misconcep-tions and found that reactions had a very weak correlation with learning(r � .07) and behavior (r � .05). They also found a correlation of .48 betweenreactions and results; however, this coefficient is based on only a single effectestimate. They found stronger relations between learning and behavior (r �.13), learning and results (r � .40), and behavior and results (r � .19).

More recently, Alliger and Tannenbaum (1995) decomposed the reactioncriteria into three subcategories: affective, utility, and a combination of the two.Alliger, Tannenbaum, Bennet, Traver, and Shotland (1997) further subdividedKirkpatrick’s learning criteria into three components: immediate posttrainingknowledge, knowledge retention, and behavior and skill demonstration. Alligeret al. (1997) found that affective and utility reactions were more strongly cor-related with each other (r � .34) than with other measures and that immedi-ate and retained learning measures were more strongly correlated with eachother (r � .35) than with other measures. Alliger et al. also indicated that reac-tion measures are easy to collect and that this type of measure could be an idealsubstitute for learning and transfer measures.

Echoing Alliger et al. (1997), Morgan and Casper (2000) argued that par-ticipant reactions can be used as predictors of more costly training effective-ness criteria: learning, behavior, and results. However, Morgan and Casperargued that constructs assessed by reaction measures may not be adequatelycaptured by Alliger et al.’s affective and utility dimensions (1997). In fact, pre-vious research on training evaluation has called for the use of multidimensionalcriteria in assessing training effectiveness (Kraiger et al., 1993).

Transfer of Training and Criticisms of Kirkpatrick’s Model. Despite thefact that organizations routinely use Kirkpatrick’s criteria to evaluate trainingsuccess (Van Buren & Erskine, 2002), this model has been criticized in the lit-erature. For example, Holton (1996) argued that Kirkpatrick’s system is noth-ing more than a taxonomy of outcomes. Researchers have consequentlyproposed evaluation models that expand the predictor space by incorporatingthe effects of individual difference variables such as motivation to learn,

Page 4: The role of employee reactions in predicting training effectiveness

400 Tan, Hall, Boyce

trainability, job attitudes, personal characteristics, self-efficacy, and transfer oftraining conditions (Campbell, 1988; Colquitt, LePine, & Noe, 2000; Gist,1987; Holton, 1996; Kanfer & Ackerman, 1989; Noe, 1986; Russ-Eft, 2002).

The inclusion of individual difference variables in models assessing train-ing effectiveness has had mixed results. Dixon (1990) found that trainee per-ceptions of job relevance, amount learned, enjoyment, and instructor’s skillwere not significantly correlated with participants’ posttest scores. Colquittet al. (2000) found that trainee motivation to learn was significantly related toboth declarative knowledge and skill acquisition. Furthermore, Tan (2002)found that learning goal orientation was significantly related with post-trainingbehavior. An individual difference variable that is promising in a training con-text is the construct of intentions in the Theory of Planned Behavior (Ajzen,1991). Intentions are assumed to capture the joint effects of motivationalfactors that have an impact on a particular behavior. Intentions, in other words,indicate how hard people are willing to try or how much of an effort they areplanning to exert in order to perform the behavior. These intentions remain asbehavioral predispositions until an appropriate time and opportunity presentsitself, at which point the individual attempts to translate intention into action(Ajzen, 1987). In a recent study, for example, Tan (2002) found that traineeintentions were strongly correlated with their posttraining behaviors (r � .73).Thus, intentions may be especially relevant to the issue of understandingtransfer of training.

Hypotheses and Overview of the Study

In this article, we suggest that reactions criteria can be usefully grouped intotwo categories. The first is an affective category, analogous to the recommen-dations of Alliger and Tannenbaum (1995) and Alliger et al. (1997). This is inline with more traditional measures of reactions, often obtained immediatelyafter the training program, that ask for the trainee’s liking or disliking of thetraining program.

The second category that we propose, however, differs from previouswork. We suggest that it might be useful to distinguish reaction items thatspecifically tap instrumentality cognitions and behavioral intentions. Exten-sive work by Ajzen and his colleagues (Ajzen, 1991; Ajzen & Fishbein, 1980;Ajzen & Madden, 1986; Madden, Ellen, & Ajzen, 1992) suggests that whilebeliefs and intentions influence a person’s likelihood of performing a givenbehavior, these remain as behavioral predispositions until an appropriate timeand opportunity presents itself on the job, after which the intention may thenbe translated into action. This model implies that items on a reaction measurethat tap the intention factor should have a stronger relation with learning andbehavior than will items that tap only the affective factor.

We propose to investigate the usefulness of the affective versus intentiondistinction for reactions criteria, as well as the relations among different types

Page 5: The role of employee reactions in predicting training effectiveness

of training criteria. We collected measures of reaction, learning, and behaviorin order to determine if the training program was effective and examine thepattern of relations among the three different types of criteria.

Specifically, we tested the following primary hypothesis:

HYPOTHESIS 1: Reaction items categorized as cognitive/intention will have a strongerrelation with learning and behavior scores than will affective items.

Also, we tested the following hypotheses to demonstrate that the trainingprogram was indeed a successful one, as well as to replicate the results of pre-vious training research (Alliger & Janak, 1989):

HYPOTHESIS 2: There will be a statistically significant increase in the scores on thelearning and behavior measures from pre- to posttest training, after controlling forrelevant covariates (that is, previous experience).

HYPOTHESIS 3: Reactions will be positively correlated with learning and behavior. How-ever, we expect that correlations of reactions with learning and with behavior will beof lower magnitude than the correlation between learning and behavior.

Method

The following paragraphs review the methods used in this study.Sample. The sample consisted of 283 automotive technicians from a large

midwestern company who were eligible and volunteered for an internallydeveloped program on brakes training. Trainees were asked to provide data tothe researchers voluntarily and were asked to sign an informed-consent formreflecting this.

The training criterion measures were collected at multiple time periodsand from multiple sources (the trainee and the trainee’s supervisor). As is com-mon with field studies, full data were available on only a subset of trainees.Unfortunately, the organization was unable to provide results data. We notethe sample sizes involved in our analyses in the tables in this article.

Procedure. The trainees underwent a two-day training program conductedby two trainers, consisting of both lectures and hands-on demonstrations. Fiftypercent of the practice was conducted after the instruction, and the other 50 per-cent was conducted on the shop floor; the trainees were also given feedback asto their performance of the task while on the shop floor. Furthermore, traineesdiagnosed automobile problems as part of their training program. They weregiven a pretest before the training program, and a posttest and the reaction mea-sure were administered after the training program. Supervisors then assessedthe trainees’ behavior six months after the program. Supervisor ratings of thetrainees’ on-the-job performance were collected by the third author. Wecontacted supervisors who did not return their ratings one month after we sentout the surveys for follow-up.

The Role of Employee Reactions in Predicting Training Effectiveness 401

Page 6: The role of employee reactions in predicting training effectiveness

402 Tan, Hall, Boyce

Measures. The following paragraphs review the measures used in thisstudy.

Reaction. Ten items assessed the affective reactions of the trainee andanother ten tapped intention/cognition; the remaining fourteen were itemsthe organization wanted to include in the measure. Responses were made ona five-point Likert scale, with 1 � strongly disagree and 5 � strongly agree.Items in the measure were either positively or negatively worded to reduceresponse sets. Internal consistency estimates for both the affective (alpha �.86) and cognitive/intention (alpha � .89) scales were adequate.

Learning. The learning measure consisted of thirty-nine multiple-choicequestions and five items that required trainees to use a guide to look upspecifications for listed vehicles in the test. The learning measure wasconstructed to tap the content domain covered in the training program toensure content validity. The learning scales tapped multiple constructs, andwe therefore expected the intercorrelations among the items to be low. Scalesthat contain items that tap multiple constructs are expected to have lowinternal consistency estimates (Murphy & Davidshoffer, 2001; Nunnally &Bernstein, 1994).

Behavior. Supervisors assessed trainee behavior six months after thetraining program with a forty-item questionnaire composed of items thatasked the frequency with which trainees performed a set of twenty relevantbehaviors before and after training. The behavior measure was alsoconstructed so that it included only behaviors that the trainees were trainedon to ensure content validity. Similar to the learning measure, the behaviormeasure tapped various constructs; thus, we expected the intercorrelationsamong the items to be low and therefore internal consistency reliabilitywould not be expected to be interpretable (Murphy & Davidshoffer, 2001;Nunnally & Bernstein, 1994).

Results

We conducted a principal components analysis with oblimin rotation on thetwenty affective and intention/cognition items. Results showed that itemsloaded on five factors. Table 1 shows a more detailed description of the factorloadings and reliability of the various subscales. Correlation analysis indicatedthat none of the demographic variables was significantly correlated with thetraining criteria at the .05 level or better (see Table 2); therefore, subsequentanalyses did not include covariates.

Hypothesis 1 stated that cognitive/intention reaction items would have astronger relation with learning and behavior scores than would affectivereaction items. An examination of the correlation matrix presented in Table 3revealed that both the Negative Evaluation (r � .62) and Improves (r � .28)dimensions positively correlated with posttraining learning. The Positive Eval-uation, Improves, and Hands-On measures were also all positively correlatedwith the post-training self-ratings of behavior.

Page 7: The role of employee reactions in predicting training effectiveness

A hierarchical regression equation was tested to see if any of the reactionsscales would add to the prediction of postlearning over prelearning. Forstep 1, the pretest was used as the predictor of posttest score. Resultsindicated that the pretest accounted for 55 percent of the variance, R2 � .55,F(1, 35) � 43.04, p � .01. For steps 2a to 2e, each reaction scale was enteredindividually to create five different two-predictor models of employee learn-ing. Regressing trainee learning on individual trainee reaction scales whilecontrolling for pretraining differences in knowledge revealed that all reactionscales except for the hands-on scale significantly predicted employee learn-ing after controlling for the pretest (see Table 4). The individual employee

The Role of Employee Reactions in Predicting Training Effectiveness 403

Table 1. Reaction Measure Subscale Items and Factor Loadings

Factor Loading Item

Positive Evaluation Scale (6 items; alpha � .89).84 I would recommend this program to other employees who have the

opportunity..70 I have an overall good feeling about how the training program was

carried out..70 I would recommend that every employee take part in this training program..59 The training program allowed me to develop specific skills that I can use on

the job..59 The training program was, overall, very effective..54 The training program is very useful.

Negative Evaluation Scale (4 items; alpha � .82).84 This training program taught me nothing I will use on the job..82 This training program was a useless waste of my and/or others’ time..80 The training program was useless for me..72 The training program was conducted poorly.

Hands-on Scale (3 items; alpha � .94)�.97 The hands-on training gave me new skills that I could take to the job.�.95 The hands-on training sharpened my current brake-repair skills.�.92 The hands-on training gave me plenty of practice so I can do my job.

Understand Scale (3 items; alpha � .76).69 The hands-on training was a good way to take the “textbook material” to the

real world..57 The training program gave me a better understanding of brake systems and

their repair..54 The training program allowed me to gain new insight into my job.

Improves Scale (4 items; alpha � .84)�.83 As a result of training, I could explain a brake repair to a non-expert.�.75 I will come out of this training with a new understanding that I can use when

talking with a potential customer.�.66 The training program will make me a better specialist.�.52 The information covered in the training program contributed to my learning.

Page 8: The role of employee reactions in predicting training effectiveness

Tab

le 2

.M

ean

s, S

tan

dar

d D

evia

tion

s, a

nd

Cor

rela

tion

s of

Dem

ogra

ph

ic, L

earn

ing,

an

d B

ehav

ior

Var

iabl

es

Beha

vior

Lear

ning

Empl

oyee

Supe

rvis

or

MSD

Pre

Post

Pre

Post

Pre

Post

Age

32.2

18.

69.2

5.1

5.3

1.1

2�

.33

�.2

0Ye

ars

empl

oyed

11.1

68.

03.3

1†.1

1.2

1.1

6�

.29

�.1

7Ye

ars

wit

h fir

m6.

475.

12.3

0†.2

3�

.03

�.2

0�

.37†

�.1

7Le

arni

ngPr

e21

.24

5.36

—.7

0**

.13

�.1

2.0

1.1

6Po

st31

.47

5.55

—.1

2�

.27

�.2

1�

.05

Beha

vior

Pree

mpl

oyee

3.65

.47

—.5

4**

.35*

*.1

6Po

stem

ploy

ee4.

04.4

4—

.40*

*.2

1Pr

esup

ervi

sor

3.43

.55

—.5

8**

Post

supe

rvis

or3.

86.5

7—

Not

e:N

for

the

corr

elat

ions

bet

wee

n Le

arni

ng a

nd D

emog

raph

ic v

aria

bles

ran

ges

from

37

to 3

9. T

he c

orre

lati

on b

etw

een

Prel

earn

ing

and

Post

lear

ning

is b

ased

on

N�

54.

Nfo

r th

e co

rrel

atio

ns o

f Be

havi

or w

ith

Dem

ogra

phic

s an

d Le

arni

ng r

ange

s fr

om 1

5 to

20.

Nfo

r th

e co

rrel

atio

ns a

mon

g Be

havi

or m

easu

res

rang

es f

rom

91to

127

.† p

�.1

0. *

*p�

.01.

Page 9: The role of employee reactions in predicting training effectiveness

reaction scales explained an additional 5 to 15 percent of the variance in theposttest score.

We tested a second regression model using the pretest and all five reactionscales to create a regression equation with six predictors of learning. Thismodel accounted for 73 percent of the variance in posttest learning, an increaseof 18 percent over the model predicting from pretest alone. However, asidefrom the unique contribution of the pretest, only negative evaluations added

The Role of Employee Reactions in Predicting Training Effectiveness 405

Table 3. Means, Standard Deviations, and Correlations of theReaction Measure Dimensions with the Posttraining

Learning and Behavior Measures

ReactionBehavior

Dimensions M SD Postlearning Postemployee Postsupervisor

AffectivePositive evaluation 4.51 .43 .26 .19† .03Negative evaluation 4.58 .65 .62** .19 .02

CognitiveUnderstand 4.40 .53 .17 .16 .04Improves 4.41 .48 .28† .21† �.01Hands-on 4.22 .55 .22 .20† �.01

Note: N for the correlations between Reactions and Learning ranges from 39 to 40. N for the corre-lations between Reactions and Postemployee behavior ranges from 73 to 76. N for the correlationsbetween Reactions and Postsupervisor behavior ratings ranges from 86 to 88.†p � .10. **p � .01.

Table 4. Hierarchical Regression of Employee Learningon Employee Reactions

IV R2 F �R2 F �

Step 1 Pretest .55 43.04** — — .74Step 2a Positive evaluation .64 30.32** .09 8.45 .30**Step 2b Negative evaluation .70 40.20** .15 17.31 .41**Step 2c Understand .62 27.97** .07 6.34 .27*Step 2d Improves .60 25.66** .05 4.28 .22*Step 2e Hands-on .60 24.46** .05 2.30 .17Step 3 Block of five reaction .73 13.23** .18 — —

dimensionsPositive evaluation — — — — .01Negative evaluation — — — — .36**Understand — — — — .06Improves — — — — .13Hands-on — — — — �.08

Note: N sizes for the regression equations ranged from 36 to 37.

*p � .05. **p � .01.

Page 10: The role of employee reactions in predicting training effectiveness

406 Tan, Hall, Boyce

uniquely to the prediction of employee learning in this model (F � 7.63,p � .01, �R2 � .16). And the sign of the regression coefficient associated withthe negative evaluation effect was positive.

We used similar hierarchical regression analyses to investigate whetheremployee reactions were good predictors of the behavior criteria. However,none of the reaction measures added significantly to the prediction of uniquevariance in behavior over the behavior pretraining measure. Thus, the specificsof these results are not presented in the article.

We also hypothesized that the training program would have the desired effectin order to eliminate the possible explanation that potential null results on otherhypothesis tests were due to ineffective training. Thus, we conducted paired-sample t tests to test hypothesis 2, which proposed significant positive changesfrom pre- to posttraining measures. Results from the t test showed that signifi-cant change did occur between the pretraining (M � 21.24) and posttraining(M � 31.47; t � 17.59, p � .01), pre- (M � 3.65) and post- (M � 4.04)employee self-ratings of behavior (t � 9.65, p � .01), and pre- (M � 3.43) andpost- (M � 3.86) supervisor behavior ratings (t � 9.06, p � .01) supportinghypothesis 2.

Hypothesis 3 stated that reactions would be positively correlated with bothlearning and behavior, although the magnitude of this correlation was expectedto be lower than the magnitude of the correlation between learning and behav-ior. Our results only partially supported hypothesis 3. Although both affectiveand cognitive/intention reaction scales did significantly correlate to a modestdegree with the learning and behavior criteria, these correlations were some-what higher than the correlations between the learning and behavior measures.For example, the correlation between negative evaluation and employee posttestscore was higher (r � .62) than the correlation between the employee andsupervisor behavior ratings (r � .21).

Discussion

Contrary to our expectations, our data showed that both affective andcognitive/intention reaction items were correlated with learning and behaviormeasures, and regression analyses showed that negative evaluations were thebest unique predictor of learning, which is contrary to previous findings. Inother words, our results showed that trainees who disliked the training pro-gram showed higher levels of learning. The data do show a positive correlationbetween pretraining knowledge and the negative evaluation dimension. A pos-sible explanation is that trainees who are already knowledgeable evaluated thetraining more harshly because it failed to live up to their high expectations.These would be the same people who would benefit more in terms of imme-diately acquired declarative knowledge, as indicated by their scores on thelearning test. This relation between negative evaluation and learning questions

Page 11: The role of employee reactions in predicting training effectiveness

the prevailing notion that positive reactions to a training program are evidencefor its success.

Our study builds on previous studies of training transfer. Dixon (1990)found that trainee perceptions were not significantly correlated with posttesttraining scores. Ruona, Leimbach, Holton, and Bates (2002) found thatparticipant utility reactions were correlated with transfer of learning factorssuch as motivation to transfer and transfer effort. Our research builds on thesestudies by examining the utility of using both affective and cognitive/intentionreaction items as possible predictors of employee learning. Our study alsobuilds on previous work in that we found correlations between employeereactions and posttraining behavior, indicating that the employees actuallytransferred what they learned to their jobs (see Table 3).

Another possible explanation for this result is that trainees’ goal orienta-tion (Dweck, 1986) might have moderated the relation between reactions andlearning. For example, mastery-oriented trainees might have perceived thetraining to be challenging and competence enhancing, while performance-oriented trainees perceived a clear link between the training content and itsapplication to their jobs (Farr, Hofmann, & Ringenbach, 1993); both groupsmight have had higher expectations about the training program. However,their high expectations about the training program might not have been met,leading to dissatisfaction. Also, performance-oriented trainees might have hadnegative reactions toward the training program in general due to the possibil-ity that they might be evaluated negatively by their supervisors; however, theseindividuals still performed well on the learning test to show their competenceand thus gain favorable judgments (Dweck & Leggett, 1988). Unfortunately,goal orientation was not a focus of this study, and therefore we were unable tocollect goal orientation data and test this hypothesis.

Another interesting result was seen when we tested hypothesis 3, whichproposed that although reaction scales might be positively correlated with bothlearning and behavior, the correlations would be of substantially lower mag-nitude than the observed relation between learning and behavior. Employeereactions did show extremely low magnitude correlations with supervisorratings of behavior, as expected. However, the magnitude of correlationbetween employee reactions and employee learning (r ranged from .17 to .62)and employee self-ratings (r ranged from .16 to .21) was somewhat higher thanthe correlations between learning and behavior (r ranged from �.27 to .12).However, because of the decreased sample size for the learning-behavior rela-tion, it is risky to overinterpret this finding. Even so, there are a couple of char-acteristics that make this study somewhat unusual among training evaluations:the time span between the measure of posttraining learning and the measuresof behavior is relatively long, and because of the heavy manual and technicalskill component of the training and subsequent job behaviors, a written learn-ing test may assess little of the knowledge that is actually relevant to future job

The Role of Employee Reactions in Predicting Training Effectiveness 407

Page 12: The role of employee reactions in predicting training effectiveness

408 Tan, Hall, Boyce

performance. Both of these factors could have contributed to higher correla-tions with reactions criteria.

Conclusions and Implications

This study showed partial support for previous findings in the literature(Alliger & Janak, 1989; Alliger & Tannenbaum, 1995; Alliger et al., 1997). Inour study, the magnitude of correlation between learning and behavior waslower than that of Alliger and colleagues. We also found mixed support forAlliger and Janak’s finding that learning and behavior had higher correlationswith each other as compared to the correlation between reactions and learn-ing or between reactions and behavior. Furthermore, our findings supportMorgan and Casper’s argument (2000) that reactions are multidimensional innature.

The most interesting finding was that negative evaluations were the bestpredictor of employee learning. This finding has implications for the trainingfield because it suggests that trainees’ feelings for the training program mightnot have that big of an impact in their learning or in their behavior after thetraining. This finding is further proof that equating training success with pos-itive trainee evaluations (as is commonly done in many organizations) may notbe the best method of evaluating a training program. It is also possible thatemployee affective reactions do not necessarily take away cognitive resourcesfrom employee goals to learn the task they are being trained on (DeShon,Brown, & Greenis, 1996).

The study found that negative evaluations significantly predicted employeelearning, with the relation showing a counterintuitive positive sign. This resultcould have occurred because trainee goal orientation moderated the relationbetween reactions and learning. Furthermore, reactions proved to have astronger relation with learning than did behavior. Of course, we would like tosee this result replicated in additional samples. It may be, for example, thattrainee characteristics, training program, and length of time between the reac-tion, learning, and behavior measures are all important in determining theexpected strength of the relation among these three criteria. We would also liketo note that our negative evaluation scale tapped affective reactions to the train-ing program and not evaluations of more objective conditions (such as theloudness of the trainer’s speaking voice).

Further research might help to establish the presence or absence of suchmoderators. Until that point, evaluators of training programs should carefullydetermine whether in their particular case negative evaluations are indeed asso-ciated with poorer training outcomes or whether these evaluations mightinstead be unrelated to outcomes or even indicative of greater learning.

One limitation of this study is the sample size. Despite the fact that therewere originally 283 participants, like most longitudinal field research, we wereunable to collect full data from all the participants. Thus, the sample sizes in

Page 13: The role of employee reactions in predicting training effectiveness

our analyses ranged from 36 to 127. We also note that we did not collectqualitative follow-up data to address the issue of why trainees responded asthey did. Future research should incorporate questions that address traineeexpectations about the program and how their expectations about the programwere met.

In sum, our results suggest that trainees whose evaluations showedevidence of negative reactions did not necessarily fail to improve learning andbehavior and in fact may show positive changes in these criteria. If these resultsare due to higher pretraining ability, knowledge, or expectations, trainingevaluations should include assessment of these variables as well to aid in theinterpretation of the evaluation results.

References

Ajzen, I. (1987). Attitudes, traits, and actions: Dispositional prediction of behavior in personal-ity and social psychology. In L. Berkowitz (Ed.), Advances in experimental social psychology(Vol. 20, pp. 1–63). Orlando, FL: Academic Press.

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human DecisionProcesses, 50, 179–211.

Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. UpperSaddle River, NJ: Prentice Hall.

Ajzen, I., & Madden, T. J. (1986). Prediction of goal-directed behavior: Attitudes, intentions, andperceived behavioral control. Journal of Experimental Social Psychology, 22, 453–474.

Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick’s levels of criteria: Thirty years later. PersonnelPsychology, 42, 331–341.

Alliger, G. M., & Tannenbaum, S. I. (1995, August). A meta-analysis on the relations amongtraining criteria. In M. Teachout (Chair), Meta-analytic investigations of training effectiveness.Symposium conducted at the annual convention of the American Psychological Association,New York.

Alliger, G. M., Tannenbaum, S. I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358.

Bassi, L. J., & Van Buren, M. E. (1998, January). State of the industry report. Training andDevelopment, 21–43.

Campbell, J. P. (1988). Training design for performance improvement. In J. P. Campbell & R. J.Campbell (Eds.), Productivity in organizations (pp. 177–216). San Francisco: Jossey-Bass.

Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of performance. InN. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 35–70). SanFrancisco: Jossey-Bass.

Cascio, W. F. (1995). Whither industrial and organizational psychology in a changing world ofwork? American Psychologist, 50, 928–939.

Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training moti-vation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85,678–707.

Day, E. A., Arthur, W. Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of acomplex skill. Journal of Applied Psychology, 85, 678–707.

DeShon, R. P., Brown, K. G., & Greenis, J. L. (1996). Does self-regulation require cognitiveresources? Evaluation of resource allocation models of goal setting. Journal of Applied Psychology,81, 595–608.

Dixon, N. M. (1990). The relationship between trainee responses on participant reaction formsand posttest scores. Human Resource Development Quarterly, 1, 129–137.

The Role of Employee Reactions in Predicting Training Effectiveness 409

Page 14: The role of employee reactions in predicting training effectiveness

410 Tan, Hall, Boyce

Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41,1040–1048.

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality.Psychological Review, 95, 256–273.

Farr, J. L., Hofmann, D. A., & Ringenbach, K. L. (1993). Goal orientation and action controltheory: Implications for industrial and organizational psychology. In C. L. Cooper & I. T.Robertson (Eds.), International review of industrial and organizational psychology (pp. 193–232).New York: Wiley.

Geber, B. (1995, March). Does training make a difference? Prove it! Training, 32, 27–34.Gist, M. E. (1987). Self-efficacy: Implications for organizational behavior and human resource

management. Academy of Management Review, 12, 472–485.Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and

evaluation (4th ed.). Belmont, CA: Wadsworth.Holton, E. F. III. (1996). The flawed four-level evaluation model. Human Resource Development

Quarterly, 7, 5–21.Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative/aptitude-

treatment interaction approach to skill acquisition [Monograph]. Journal of Applied Psychology,74, 657–690.

Kirkpatrick, D. L. (1959a). Techniques for evaluating training programs. Journal of the ASTD,13, 3–9.

Kirkpatrick, D. L. (1959b). Techniques for evaluating training programs: Part 2—Learning.Journal of the ASTD, 13, 21–26.

Kirkpatrick, D. L. (1960a). Techniques for evaluating training programs: Part 3—Behavior. Journalof the ASTD, 14, 13–18.

Kirkpatrick, D. L. (1960b). Techniques for evaluating training programs: Part 4—Results. Journalof the ASTD, 14, 28–32.

Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.

Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affectivetheories of learning outcomes to new methods of training evaluation [Monograph]. Journal ofApplied Psychology, 78, 311–328.

Madden, T. J., Ellen, P. S., & Ajzen, I. (1992). A comparison of the theory of planned behaviorand the theory of reasoned action. Personality and Social Psychology Bulletin, 18, 3–9.

Moore, R. W., Blake, D. R., Phillips, G. M., & McConaughy, D. (2003, April). Moving beyond “TheFour Levels”: A systems evaluation model for training programs. Paper presented at the 44th annualconference of the Western Academy of Management, Palm Springs, CA.

Morgan, R. B., & Casper, W. J. (2000). Examining the factor structure of participant reactionsto training: A multidimensional approach. Human Resource Development Quarterly, 11,301–317.

Murphy, K. R., & Davidshoffer, C. O. (2001). Psychological testing: Principles and applications(5th ed.). Upper Saddle River, NJ: Prentice Hall.

Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effective-ness. Academy of Management Review, 11, 736–749.

Noe, R. A., & Colquitt, J. A. (2002). Planning for training impact: Principles of trainingeffectiveness. In K. Kraiger (Ed.), Creating, implementing, and managing effective training anddevelopment: State-of-the-art lessons for practice (pp. 53–79). San Francisco: Jossey-Bass.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Preskill, H., & Russ-Eft, D. (2003). A framework for reframing HRD evaluation, practice, and

research. In A. M. Gilley, J. L. Callahan, & L. L. Bierema (Eds.), Critical issues in HRD: A newagenda for the twenty-first century (pp. 199–213). Cambridge, MA: Perseus Publishing.

Ruona, W.E.A., Leimbach, M., Holton, E. F. III., & Bates, R. (2002). The relationship betweenlearner utility reactions and predicted learning transfer among trainees. International Journalof Training and Development, 6, 218–228.

Page 15: The role of employee reactions in predicting training effectiveness

Russ-Eft, D. (2002). A typology of training design and work environment factors affecting work-place learning and transfer. Human Resource Development Review, 1, 45–65.

Russ-Eft, D., & Preskill, H. (2001). Evaluation in organizations: A systematic approach to enhancinglearning, performance, and change. Cambridge, MA: Perseus Publishing.

Saari, L. M., Johnson, T. R., McLaughlin, S. D., & Zimmerle, D. M. (1988). A survey of man-agement training and education practices in U.S. companies. Personnel Psychology, 41, 731–743.

Sackett, P. R., & Mullen, E. J. (1993). Beyond formal experimental design: Toward an expandedview of the training evaluation process. Personnel Psychology, 46, 613–627.

Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. AnnualReview of Psychology, 52, 471–499.

Tan, J. A. (2002). The role of planned behavior and goal orientation theory in a training context.Dissertation Abstracts International, 62, 3409.

Van Buren, M. E., & Erskine, W. (2002). The ASTD state of the industry report. Alexandria, VA:American Society of Training and Development.

James A. Tan is assistant professor of psychology at the University of Wisconsin-Stout.

Rosalie J. Hall is associate professor of psychology at the University of Akron.

Carol Boyce is affiliated with Alliant International University.

The Role of Employee Reactions in Predicting Training Effectiveness 411