Upload
james-a-tan
View
214
Download
0
Embed Size (px)
Citation preview
A R T I C L E S
The Role of Employee Reactionsin Predicting TrainingEffectiveness
James A. Tan, Rosalie J. Hall, Carol Boyce
Reactions to training programs tend to be poor predictors of trainingsuccess, yet most training programs are evaluated based solely on traineereactions. In this study, we proposed that distinguishing between affec-tive and cognitive employee reactions may improve the prediction of traineelearning. Our results indicated that cognitive employee reactions arerelated to both employee learning and employee behavior. Moreover,contrary to popular notion, negative affective reactions best predictedemployee learning. Implications and future research directions of the resultsare discussed.
As the workplace undergoes sweeping changes, such as the pressure of moreglobal competition (Cascio, 1995), the need for strategies to keep up withthese changes is increasingly important. Organizational training is a methodused to enhance individual productivity as well as a company’s success. Anorganization’s investment in human capital through the education and train-ing of its members is therefore a central component of competitive strategy(Bassi & Van Buren, 1998).
Organizations often evaluate training effectiveness using one or moreof Kirkpatrick’s criteria (1959a, 1959b, 1960a, 1960b). Although otherapproaches to training evaluation have been proposed (Day, Arthur, &Gettman, 2001; Kraiger, Ford, & Salas, 1993; Moore, Blake, Phillips, &McConaughy, 2003), training success has most often been evaluated usingpaper-and-pencil measures (that is, reaction and learning measures; Geber,1995; Saari, Johnson, McLaughlin, & Zimmerle, 1988; Salas & Cannon-Bowers, 2001; Van Buren & Erskine, 2002). Research investigating the effectof various training design and workplace environment factors on workplacelearning and transfer has been covered elsewhere (see Russ-Eft, 2002). In thisarticle, we address issues concerning the relation of paper-and-pencil measures
397HUMAN RESOURCE DEVELOPMENT QUARTERLY, vol. 14, no. 4, Winter 2003Copyright © 2003 Wiley Periodicals, Inc.
398 Tan, Hall, Boyce
of employee reactions to training with the actual learning imparted by thetraining program.
Training Evaluation Criteria
Training evaluation is a system for measuring changes due to traininginterventions—most important, whether trainees have achieved learning out-comes (Goldstein & Ford, 2002; Kraiger et al., 1993; Sackett & Mullen, 1993).Few, if any, single measures can actually capture the complexity of training pro-grams and transfer performance. Although the issue of conceptually basedmodels to guide researchers in evaluating training effectiveness is still in itsinfancy, some advances have been made in this area recently (Noe & Colquitt,2002; Preskill & Russ-Eft, 2003; Russ-Eft & Preskill, 2001). Russ-Eft andPreskill (2001) presented alternative methods of evaluation (for example,developmental evaluation). More recently, Preskill and Russ-Eft (2003) pro-posed a model based on the philosophies, theories, and practices in variousfields such as evaluation, organizational learning, and systems thinking.
Despite these advances in training evaluation theory, the most developedand used of these models in the human resource literature focuses on definingdifferent training effectiveness criteria and their organizational implications.Common to these models is the use of multiple criteria to gather data on train-ing effectiveness. As such, training evaluators need to view criteria as multidi-mensional in nature (Campbell, McCloy, Oppler, & Sager, 1993; Kraiger et al.,1993). For example, Kraiger et al. argued that learning outcomes can bedivided into three types: cognitive (which reflect knowledge and cognitivestrategies), skill based (reflecting constructs such as automaticity and compi-lation), and affective (reflecting constructs such as attitudes and motivation).These outcomes can be seen as analogous to Kirkpatrick’s learning, behavior,and reactions criteria, respectively.
Varieties of Evaluation Criteria. Kirkpatrick (1959a, 1959b, 1960a,1960b) outlined four steps or levels of measures of the effectiveness of trainingoutcomes. Level 1, or reaction, is trainees’ feelings for and liking of a train-ing program. Reaction measures may indicate the trainee’s motivation to learn;although positive reactions may not ensure learning, negative reactionsprobably reduce the possibility that learning occurs. A focus of this articleis suggested improvements in the construction, expected relations, andinterpretations of reactions measures.
Level 2, learning, was defined as the “principles, facts, and techniquesunderstood and absorbed by the [trainees]” (Alliger & Janak, 1989, p. 331).No change in behavior can be expected unless one or more of these learningobjectives have been accomplished (Kirkpatrick, 1994). Learning is most oftenassessed by giving the trainees tests that tap declarative knowledge (Kraigeret al., 1993). Level 3, behavior, defined as transferring knowledge, skills, andattitudes learned during the training to the job (Kirkpatrick, 1994), is typically
The Role of Employee Reactions in Predicting Training Effectiveness 399
measured on the job after a particular amount of time has passed. This is mostoften assessed through performance appraisal. Level 4, results, was defined asthe final results that occurred because the participants attended the program(Kirkpatrick, 1994). These could include increased production, improvedquality, decreased costs, reduced frequency and severity of accidents, increasedsales, reduced turnover, and higher profits and return on investments.
Relations Among Different Types of Criteria. Kirkpatrick’s modelhas been misunderstood by researchers and practitioners to be hierarchical(Alliger & Janak, 1989; Russ-Eft & Preskill, 2001). Specifically, the followingassumptions arose: (1) each succeeding level of evaluation criteria is moreinformative (or “better” in terms of information obtained for the organization;Russ-Eft & Preskill, 2001) than the last, (2) each level is caused by thepreceding level, and (3) each succeeding level is correlated with the previouslevel. Using meta-analysis, Alliger and Janak (1989) clarified these misconcep-tions and found that reactions had a very weak correlation with learning(r � .07) and behavior (r � .05). They also found a correlation of .48 betweenreactions and results; however, this coefficient is based on only a single effectestimate. They found stronger relations between learning and behavior (r �.13), learning and results (r � .40), and behavior and results (r � .19).
More recently, Alliger and Tannenbaum (1995) decomposed the reactioncriteria into three subcategories: affective, utility, and a combination of the two.Alliger, Tannenbaum, Bennet, Traver, and Shotland (1997) further subdividedKirkpatrick’s learning criteria into three components: immediate posttrainingknowledge, knowledge retention, and behavior and skill demonstration. Alligeret al. (1997) found that affective and utility reactions were more strongly cor-related with each other (r � .34) than with other measures and that immedi-ate and retained learning measures were more strongly correlated with eachother (r � .35) than with other measures. Alliger et al. also indicated that reac-tion measures are easy to collect and that this type of measure could be an idealsubstitute for learning and transfer measures.
Echoing Alliger et al. (1997), Morgan and Casper (2000) argued that par-ticipant reactions can be used as predictors of more costly training effective-ness criteria: learning, behavior, and results. However, Morgan and Casperargued that constructs assessed by reaction measures may not be adequatelycaptured by Alliger et al.’s affective and utility dimensions (1997). In fact, pre-vious research on training evaluation has called for the use of multidimensionalcriteria in assessing training effectiveness (Kraiger et al., 1993).
Transfer of Training and Criticisms of Kirkpatrick’s Model. Despite thefact that organizations routinely use Kirkpatrick’s criteria to evaluate trainingsuccess (Van Buren & Erskine, 2002), this model has been criticized in the lit-erature. For example, Holton (1996) argued that Kirkpatrick’s system is noth-ing more than a taxonomy of outcomes. Researchers have consequentlyproposed evaluation models that expand the predictor space by incorporatingthe effects of individual difference variables such as motivation to learn,
400 Tan, Hall, Boyce
trainability, job attitudes, personal characteristics, self-efficacy, and transfer oftraining conditions (Campbell, 1988; Colquitt, LePine, & Noe, 2000; Gist,1987; Holton, 1996; Kanfer & Ackerman, 1989; Noe, 1986; Russ-Eft, 2002).
The inclusion of individual difference variables in models assessing train-ing effectiveness has had mixed results. Dixon (1990) found that trainee per-ceptions of job relevance, amount learned, enjoyment, and instructor’s skillwere not significantly correlated with participants’ posttest scores. Colquittet al. (2000) found that trainee motivation to learn was significantly related toboth declarative knowledge and skill acquisition. Furthermore, Tan (2002)found that learning goal orientation was significantly related with post-trainingbehavior. An individual difference variable that is promising in a training con-text is the construct of intentions in the Theory of Planned Behavior (Ajzen,1991). Intentions are assumed to capture the joint effects of motivationalfactors that have an impact on a particular behavior. Intentions, in other words,indicate how hard people are willing to try or how much of an effort they areplanning to exert in order to perform the behavior. These intentions remain asbehavioral predispositions until an appropriate time and opportunity presentsitself, at which point the individual attempts to translate intention into action(Ajzen, 1987). In a recent study, for example, Tan (2002) found that traineeintentions were strongly correlated with their posttraining behaviors (r � .73).Thus, intentions may be especially relevant to the issue of understandingtransfer of training.
Hypotheses and Overview of the Study
In this article, we suggest that reactions criteria can be usefully grouped intotwo categories. The first is an affective category, analogous to the recommen-dations of Alliger and Tannenbaum (1995) and Alliger et al. (1997). This is inline with more traditional measures of reactions, often obtained immediatelyafter the training program, that ask for the trainee’s liking or disliking of thetraining program.
The second category that we propose, however, differs from previouswork. We suggest that it might be useful to distinguish reaction items thatspecifically tap instrumentality cognitions and behavioral intentions. Exten-sive work by Ajzen and his colleagues (Ajzen, 1991; Ajzen & Fishbein, 1980;Ajzen & Madden, 1986; Madden, Ellen, & Ajzen, 1992) suggests that whilebeliefs and intentions influence a person’s likelihood of performing a givenbehavior, these remain as behavioral predispositions until an appropriate timeand opportunity presents itself on the job, after which the intention may thenbe translated into action. This model implies that items on a reaction measurethat tap the intention factor should have a stronger relation with learning andbehavior than will items that tap only the affective factor.
We propose to investigate the usefulness of the affective versus intentiondistinction for reactions criteria, as well as the relations among different types
of training criteria. We collected measures of reaction, learning, and behaviorin order to determine if the training program was effective and examine thepattern of relations among the three different types of criteria.
Specifically, we tested the following primary hypothesis:
HYPOTHESIS 1: Reaction items categorized as cognitive/intention will have a strongerrelation with learning and behavior scores than will affective items.
Also, we tested the following hypotheses to demonstrate that the trainingprogram was indeed a successful one, as well as to replicate the results of pre-vious training research (Alliger & Janak, 1989):
HYPOTHESIS 2: There will be a statistically significant increase in the scores on thelearning and behavior measures from pre- to posttest training, after controlling forrelevant covariates (that is, previous experience).
HYPOTHESIS 3: Reactions will be positively correlated with learning and behavior. How-ever, we expect that correlations of reactions with learning and with behavior will beof lower magnitude than the correlation between learning and behavior.
Method
The following paragraphs review the methods used in this study.Sample. The sample consisted of 283 automotive technicians from a large
midwestern company who were eligible and volunteered for an internallydeveloped program on brakes training. Trainees were asked to provide data tothe researchers voluntarily and were asked to sign an informed-consent formreflecting this.
The training criterion measures were collected at multiple time periodsand from multiple sources (the trainee and the trainee’s supervisor). As is com-mon with field studies, full data were available on only a subset of trainees.Unfortunately, the organization was unable to provide results data. We notethe sample sizes involved in our analyses in the tables in this article.
Procedure. The trainees underwent a two-day training program conductedby two trainers, consisting of both lectures and hands-on demonstrations. Fiftypercent of the practice was conducted after the instruction, and the other 50 per-cent was conducted on the shop floor; the trainees were also given feedback asto their performance of the task while on the shop floor. Furthermore, traineesdiagnosed automobile problems as part of their training program. They weregiven a pretest before the training program, and a posttest and the reaction mea-sure were administered after the training program. Supervisors then assessedthe trainees’ behavior six months after the program. Supervisor ratings of thetrainees’ on-the-job performance were collected by the third author. Wecontacted supervisors who did not return their ratings one month after we sentout the surveys for follow-up.
The Role of Employee Reactions in Predicting Training Effectiveness 401
402 Tan, Hall, Boyce
Measures. The following paragraphs review the measures used in thisstudy.
Reaction. Ten items assessed the affective reactions of the trainee andanother ten tapped intention/cognition; the remaining fourteen were itemsthe organization wanted to include in the measure. Responses were made ona five-point Likert scale, with 1 � strongly disagree and 5 � strongly agree.Items in the measure were either positively or negatively worded to reduceresponse sets. Internal consistency estimates for both the affective (alpha �.86) and cognitive/intention (alpha � .89) scales were adequate.
Learning. The learning measure consisted of thirty-nine multiple-choicequestions and five items that required trainees to use a guide to look upspecifications for listed vehicles in the test. The learning measure wasconstructed to tap the content domain covered in the training program toensure content validity. The learning scales tapped multiple constructs, andwe therefore expected the intercorrelations among the items to be low. Scalesthat contain items that tap multiple constructs are expected to have lowinternal consistency estimates (Murphy & Davidshoffer, 2001; Nunnally &Bernstein, 1994).
Behavior. Supervisors assessed trainee behavior six months after thetraining program with a forty-item questionnaire composed of items thatasked the frequency with which trainees performed a set of twenty relevantbehaviors before and after training. The behavior measure was alsoconstructed so that it included only behaviors that the trainees were trainedon to ensure content validity. Similar to the learning measure, the behaviormeasure tapped various constructs; thus, we expected the intercorrelationsamong the items to be low and therefore internal consistency reliabilitywould not be expected to be interpretable (Murphy & Davidshoffer, 2001;Nunnally & Bernstein, 1994).
Results
We conducted a principal components analysis with oblimin rotation on thetwenty affective and intention/cognition items. Results showed that itemsloaded on five factors. Table 1 shows a more detailed description of the factorloadings and reliability of the various subscales. Correlation analysis indicatedthat none of the demographic variables was significantly correlated with thetraining criteria at the .05 level or better (see Table 2); therefore, subsequentanalyses did not include covariates.
Hypothesis 1 stated that cognitive/intention reaction items would have astronger relation with learning and behavior scores than would affectivereaction items. An examination of the correlation matrix presented in Table 3revealed that both the Negative Evaluation (r � .62) and Improves (r � .28)dimensions positively correlated with posttraining learning. The Positive Eval-uation, Improves, and Hands-On measures were also all positively correlatedwith the post-training self-ratings of behavior.
A hierarchical regression equation was tested to see if any of the reactionsscales would add to the prediction of postlearning over prelearning. Forstep 1, the pretest was used as the predictor of posttest score. Resultsindicated that the pretest accounted for 55 percent of the variance, R2 � .55,F(1, 35) � 43.04, p � .01. For steps 2a to 2e, each reaction scale was enteredindividually to create five different two-predictor models of employee learn-ing. Regressing trainee learning on individual trainee reaction scales whilecontrolling for pretraining differences in knowledge revealed that all reactionscales except for the hands-on scale significantly predicted employee learn-ing after controlling for the pretest (see Table 4). The individual employee
The Role of Employee Reactions in Predicting Training Effectiveness 403
Table 1. Reaction Measure Subscale Items and Factor Loadings
Factor Loading Item
Positive Evaluation Scale (6 items; alpha � .89).84 I would recommend this program to other employees who have the
opportunity..70 I have an overall good feeling about how the training program was
carried out..70 I would recommend that every employee take part in this training program..59 The training program allowed me to develop specific skills that I can use on
the job..59 The training program was, overall, very effective..54 The training program is very useful.
Negative Evaluation Scale (4 items; alpha � .82).84 This training program taught me nothing I will use on the job..82 This training program was a useless waste of my and/or others’ time..80 The training program was useless for me..72 The training program was conducted poorly.
Hands-on Scale (3 items; alpha � .94)�.97 The hands-on training gave me new skills that I could take to the job.�.95 The hands-on training sharpened my current brake-repair skills.�.92 The hands-on training gave me plenty of practice so I can do my job.
Understand Scale (3 items; alpha � .76).69 The hands-on training was a good way to take the “textbook material” to the
real world..57 The training program gave me a better understanding of brake systems and
their repair..54 The training program allowed me to gain new insight into my job.
Improves Scale (4 items; alpha � .84)�.83 As a result of training, I could explain a brake repair to a non-expert.�.75 I will come out of this training with a new understanding that I can use when
talking with a potential customer.�.66 The training program will make me a better specialist.�.52 The information covered in the training program contributed to my learning.
Tab
le 2
.M
ean
s, S
tan
dar
d D
evia
tion
s, a
nd
Cor
rela
tion
s of
Dem
ogra
ph
ic, L
earn
ing,
an
d B
ehav
ior
Var
iabl
es
Beha
vior
Lear
ning
Empl
oyee
Supe
rvis
or
MSD
Pre
Post
Pre
Post
Pre
Post
Age
32.2
18.
69.2
5.1
5.3
1.1
2�
.33
�.2
0Ye
ars
empl
oyed
11.1
68.
03.3
1†.1
1.2
1.1
6�
.29
�.1
7Ye
ars
wit
h fir
m6.
475.
12.3
0†.2
3�
.03
�.2
0�
.37†
�.1
7Le
arni
ngPr
e21
.24
5.36
—.7
0**
.13
�.1
2.0
1.1
6Po
st31
.47
5.55
—.1
2�
.27
�.2
1�
.05
Beha
vior
Pree
mpl
oyee
3.65
.47
—.5
4**
.35*
*.1
6Po
stem
ploy
ee4.
04.4
4—
.40*
*.2
1Pr
esup
ervi
sor
3.43
.55
—.5
8**
Post
supe
rvis
or3.
86.5
7—
Not
e:N
for
the
corr
elat
ions
bet
wee
n Le
arni
ng a
nd D
emog
raph
ic v
aria
bles
ran
ges
from
37
to 3
9. T
he c
orre
lati
on b
etw
een
Prel
earn
ing
and
Post
lear
ning
is b
ased
on
N�
54.
Nfo
r th
e co
rrel
atio
ns o
f Be
havi
or w
ith
Dem
ogra
phic
s an
d Le
arni
ng r
ange
s fr
om 1
5 to
20.
Nfo
r th
e co
rrel
atio
ns a
mon
g Be
havi
or m
easu
res
rang
es f
rom
91to
127
.† p
�.1
0. *
*p�
.01.
reaction scales explained an additional 5 to 15 percent of the variance in theposttest score.
We tested a second regression model using the pretest and all five reactionscales to create a regression equation with six predictors of learning. Thismodel accounted for 73 percent of the variance in posttest learning, an increaseof 18 percent over the model predicting from pretest alone. However, asidefrom the unique contribution of the pretest, only negative evaluations added
The Role of Employee Reactions in Predicting Training Effectiveness 405
Table 3. Means, Standard Deviations, and Correlations of theReaction Measure Dimensions with the Posttraining
Learning and Behavior Measures
ReactionBehavior
Dimensions M SD Postlearning Postemployee Postsupervisor
AffectivePositive evaluation 4.51 .43 .26 .19† .03Negative evaluation 4.58 .65 .62** .19 .02
CognitiveUnderstand 4.40 .53 .17 .16 .04Improves 4.41 .48 .28† .21† �.01Hands-on 4.22 .55 .22 .20† �.01
Note: N for the correlations between Reactions and Learning ranges from 39 to 40. N for the corre-lations between Reactions and Postemployee behavior ranges from 73 to 76. N for the correlationsbetween Reactions and Postsupervisor behavior ratings ranges from 86 to 88.†p � .10. **p � .01.
Table 4. Hierarchical Regression of Employee Learningon Employee Reactions
IV R2 F �R2 F �
Step 1 Pretest .55 43.04** — — .74Step 2a Positive evaluation .64 30.32** .09 8.45 .30**Step 2b Negative evaluation .70 40.20** .15 17.31 .41**Step 2c Understand .62 27.97** .07 6.34 .27*Step 2d Improves .60 25.66** .05 4.28 .22*Step 2e Hands-on .60 24.46** .05 2.30 .17Step 3 Block of five reaction .73 13.23** .18 — —
dimensionsPositive evaluation — — — — .01Negative evaluation — — — — .36**Understand — — — — .06Improves — — — — .13Hands-on — — — — �.08
Note: N sizes for the regression equations ranged from 36 to 37.
*p � .05. **p � .01.
406 Tan, Hall, Boyce
uniquely to the prediction of employee learning in this model (F � 7.63,p � .01, �R2 � .16). And the sign of the regression coefficient associated withthe negative evaluation effect was positive.
We used similar hierarchical regression analyses to investigate whetheremployee reactions were good predictors of the behavior criteria. However,none of the reaction measures added significantly to the prediction of uniquevariance in behavior over the behavior pretraining measure. Thus, the specificsof these results are not presented in the article.
We also hypothesized that the training program would have the desired effectin order to eliminate the possible explanation that potential null results on otherhypothesis tests were due to ineffective training. Thus, we conducted paired-sample t tests to test hypothesis 2, which proposed significant positive changesfrom pre- to posttraining measures. Results from the t test showed that signifi-cant change did occur between the pretraining (M � 21.24) and posttraining(M � 31.47; t � 17.59, p � .01), pre- (M � 3.65) and post- (M � 4.04)employee self-ratings of behavior (t � 9.65, p � .01), and pre- (M � 3.43) andpost- (M � 3.86) supervisor behavior ratings (t � 9.06, p � .01) supportinghypothesis 2.
Hypothesis 3 stated that reactions would be positively correlated with bothlearning and behavior, although the magnitude of this correlation was expectedto be lower than the magnitude of the correlation between learning and behav-ior. Our results only partially supported hypothesis 3. Although both affectiveand cognitive/intention reaction scales did significantly correlate to a modestdegree with the learning and behavior criteria, these correlations were some-what higher than the correlations between the learning and behavior measures.For example, the correlation between negative evaluation and employee posttestscore was higher (r � .62) than the correlation between the employee andsupervisor behavior ratings (r � .21).
Discussion
Contrary to our expectations, our data showed that both affective andcognitive/intention reaction items were correlated with learning and behaviormeasures, and regression analyses showed that negative evaluations were thebest unique predictor of learning, which is contrary to previous findings. Inother words, our results showed that trainees who disliked the training pro-gram showed higher levels of learning. The data do show a positive correlationbetween pretraining knowledge and the negative evaluation dimension. A pos-sible explanation is that trainees who are already knowledgeable evaluated thetraining more harshly because it failed to live up to their high expectations.These would be the same people who would benefit more in terms of imme-diately acquired declarative knowledge, as indicated by their scores on thelearning test. This relation between negative evaluation and learning questions
the prevailing notion that positive reactions to a training program are evidencefor its success.
Our study builds on previous studies of training transfer. Dixon (1990)found that trainee perceptions were not significantly correlated with posttesttraining scores. Ruona, Leimbach, Holton, and Bates (2002) found thatparticipant utility reactions were correlated with transfer of learning factorssuch as motivation to transfer and transfer effort. Our research builds on thesestudies by examining the utility of using both affective and cognitive/intentionreaction items as possible predictors of employee learning. Our study alsobuilds on previous work in that we found correlations between employeereactions and posttraining behavior, indicating that the employees actuallytransferred what they learned to their jobs (see Table 3).
Another possible explanation for this result is that trainees’ goal orienta-tion (Dweck, 1986) might have moderated the relation between reactions andlearning. For example, mastery-oriented trainees might have perceived thetraining to be challenging and competence enhancing, while performance-oriented trainees perceived a clear link between the training content and itsapplication to their jobs (Farr, Hofmann, & Ringenbach, 1993); both groupsmight have had higher expectations about the training program. However,their high expectations about the training program might not have been met,leading to dissatisfaction. Also, performance-oriented trainees might have hadnegative reactions toward the training program in general due to the possibil-ity that they might be evaluated negatively by their supervisors; however, theseindividuals still performed well on the learning test to show their competenceand thus gain favorable judgments (Dweck & Leggett, 1988). Unfortunately,goal orientation was not a focus of this study, and therefore we were unable tocollect goal orientation data and test this hypothesis.
Another interesting result was seen when we tested hypothesis 3, whichproposed that although reaction scales might be positively correlated with bothlearning and behavior, the correlations would be of substantially lower mag-nitude than the observed relation between learning and behavior. Employeereactions did show extremely low magnitude correlations with supervisorratings of behavior, as expected. However, the magnitude of correlationbetween employee reactions and employee learning (r ranged from .17 to .62)and employee self-ratings (r ranged from .16 to .21) was somewhat higher thanthe correlations between learning and behavior (r ranged from �.27 to .12).However, because of the decreased sample size for the learning-behavior rela-tion, it is risky to overinterpret this finding. Even so, there are a couple of char-acteristics that make this study somewhat unusual among training evaluations:the time span between the measure of posttraining learning and the measuresof behavior is relatively long, and because of the heavy manual and technicalskill component of the training and subsequent job behaviors, a written learn-ing test may assess little of the knowledge that is actually relevant to future job
The Role of Employee Reactions in Predicting Training Effectiveness 407
408 Tan, Hall, Boyce
performance. Both of these factors could have contributed to higher correla-tions with reactions criteria.
Conclusions and Implications
This study showed partial support for previous findings in the literature(Alliger & Janak, 1989; Alliger & Tannenbaum, 1995; Alliger et al., 1997). Inour study, the magnitude of correlation between learning and behavior waslower than that of Alliger and colleagues. We also found mixed support forAlliger and Janak’s finding that learning and behavior had higher correlationswith each other as compared to the correlation between reactions and learn-ing or between reactions and behavior. Furthermore, our findings supportMorgan and Casper’s argument (2000) that reactions are multidimensional innature.
The most interesting finding was that negative evaluations were the bestpredictor of employee learning. This finding has implications for the trainingfield because it suggests that trainees’ feelings for the training program mightnot have that big of an impact in their learning or in their behavior after thetraining. This finding is further proof that equating training success with pos-itive trainee evaluations (as is commonly done in many organizations) may notbe the best method of evaluating a training program. It is also possible thatemployee affective reactions do not necessarily take away cognitive resourcesfrom employee goals to learn the task they are being trained on (DeShon,Brown, & Greenis, 1996).
The study found that negative evaluations significantly predicted employeelearning, with the relation showing a counterintuitive positive sign. This resultcould have occurred because trainee goal orientation moderated the relationbetween reactions and learning. Furthermore, reactions proved to have astronger relation with learning than did behavior. Of course, we would like tosee this result replicated in additional samples. It may be, for example, thattrainee characteristics, training program, and length of time between the reac-tion, learning, and behavior measures are all important in determining theexpected strength of the relation among these three criteria. We would also liketo note that our negative evaluation scale tapped affective reactions to the train-ing program and not evaluations of more objective conditions (such as theloudness of the trainer’s speaking voice).
Further research might help to establish the presence or absence of suchmoderators. Until that point, evaluators of training programs should carefullydetermine whether in their particular case negative evaluations are indeed asso-ciated with poorer training outcomes or whether these evaluations mightinstead be unrelated to outcomes or even indicative of greater learning.
One limitation of this study is the sample size. Despite the fact that therewere originally 283 participants, like most longitudinal field research, we wereunable to collect full data from all the participants. Thus, the sample sizes in
our analyses ranged from 36 to 127. We also note that we did not collectqualitative follow-up data to address the issue of why trainees responded asthey did. Future research should incorporate questions that address traineeexpectations about the program and how their expectations about the programwere met.
In sum, our results suggest that trainees whose evaluations showedevidence of negative reactions did not necessarily fail to improve learning andbehavior and in fact may show positive changes in these criteria. If these resultsare due to higher pretraining ability, knowledge, or expectations, trainingevaluations should include assessment of these variables as well to aid in theinterpretation of the evaluation results.
References
Ajzen, I. (1987). Attitudes, traits, and actions: Dispositional prediction of behavior in personal-ity and social psychology. In L. Berkowitz (Ed.), Advances in experimental social psychology(Vol. 20, pp. 1–63). Orlando, FL: Academic Press.
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human DecisionProcesses, 50, 179–211.
Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. UpperSaddle River, NJ: Prentice Hall.
Ajzen, I., & Madden, T. J. (1986). Prediction of goal-directed behavior: Attitudes, intentions, andperceived behavioral control. Journal of Experimental Social Psychology, 22, 453–474.
Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick’s levels of criteria: Thirty years later. PersonnelPsychology, 42, 331–341.
Alliger, G. M., & Tannenbaum, S. I. (1995, August). A meta-analysis on the relations amongtraining criteria. In M. Teachout (Chair), Meta-analytic investigations of training effectiveness.Symposium conducted at the annual convention of the American Psychological Association,New York.
Alliger, G. M., Tannenbaum, S. I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358.
Bassi, L. J., & Van Buren, M. E. (1998, January). State of the industry report. Training andDevelopment, 21–43.
Campbell, J. P. (1988). Training design for performance improvement. In J. P. Campbell & R. J.Campbell (Eds.), Productivity in organizations (pp. 177–216). San Francisco: Jossey-Bass.
Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of performance. InN. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 35–70). SanFrancisco: Jossey-Bass.
Cascio, W. F. (1995). Whither industrial and organizational psychology in a changing world ofwork? American Psychologist, 50, 928–939.
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training moti-vation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85,678–707.
Day, E. A., Arthur, W. Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of acomplex skill. Journal of Applied Psychology, 85, 678–707.
DeShon, R. P., Brown, K. G., & Greenis, J. L. (1996). Does self-regulation require cognitiveresources? Evaluation of resource allocation models of goal setting. Journal of Applied Psychology,81, 595–608.
Dixon, N. M. (1990). The relationship between trainee responses on participant reaction formsand posttest scores. Human Resource Development Quarterly, 1, 129–137.
The Role of Employee Reactions in Predicting Training Effectiveness 409
410 Tan, Hall, Boyce
Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41,1040–1048.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality.Psychological Review, 95, 256–273.
Farr, J. L., Hofmann, D. A., & Ringenbach, K. L. (1993). Goal orientation and action controltheory: Implications for industrial and organizational psychology. In C. L. Cooper & I. T.Robertson (Eds.), International review of industrial and organizational psychology (pp. 193–232).New York: Wiley.
Geber, B. (1995, March). Does training make a difference? Prove it! Training, 32, 27–34.Gist, M. E. (1987). Self-efficacy: Implications for organizational behavior and human resource
management. Academy of Management Review, 12, 472–485.Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and
evaluation (4th ed.). Belmont, CA: Wadsworth.Holton, E. F. III. (1996). The flawed four-level evaluation model. Human Resource Development
Quarterly, 7, 5–21.Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative/aptitude-
treatment interaction approach to skill acquisition [Monograph]. Journal of Applied Psychology,74, 657–690.
Kirkpatrick, D. L. (1959a). Techniques for evaluating training programs. Journal of the ASTD,13, 3–9.
Kirkpatrick, D. L. (1959b). Techniques for evaluating training programs: Part 2—Learning.Journal of the ASTD, 13, 21–26.
Kirkpatrick, D. L. (1960a). Techniques for evaluating training programs: Part 3—Behavior. Journalof the ASTD, 14, 13–18.
Kirkpatrick, D. L. (1960b). Techniques for evaluating training programs: Part 4—Results. Journalof the ASTD, 14, 28–32.
Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affectivetheories of learning outcomes to new methods of training evaluation [Monograph]. Journal ofApplied Psychology, 78, 311–328.
Madden, T. J., Ellen, P. S., & Ajzen, I. (1992). A comparison of the theory of planned behaviorand the theory of reasoned action. Personality and Social Psychology Bulletin, 18, 3–9.
Moore, R. W., Blake, D. R., Phillips, G. M., & McConaughy, D. (2003, April). Moving beyond “TheFour Levels”: A systems evaluation model for training programs. Paper presented at the 44th annualconference of the Western Academy of Management, Palm Springs, CA.
Morgan, R. B., & Casper, W. J. (2000). Examining the factor structure of participant reactionsto training: A multidimensional approach. Human Resource Development Quarterly, 11,301–317.
Murphy, K. R., & Davidshoffer, C. O. (2001). Psychological testing: Principles and applications(5th ed.). Upper Saddle River, NJ: Prentice Hall.
Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effective-ness. Academy of Management Review, 11, 736–749.
Noe, R. A., & Colquitt, J. A. (2002). Planning for training impact: Principles of trainingeffectiveness. In K. Kraiger (Ed.), Creating, implementing, and managing effective training anddevelopment: State-of-the-art lessons for practice (pp. 53–79). San Francisco: Jossey-Bass.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Preskill, H., & Russ-Eft, D. (2003). A framework for reframing HRD evaluation, practice, and
research. In A. M. Gilley, J. L. Callahan, & L. L. Bierema (Eds.), Critical issues in HRD: A newagenda for the twenty-first century (pp. 199–213). Cambridge, MA: Perseus Publishing.
Ruona, W.E.A., Leimbach, M., Holton, E. F. III., & Bates, R. (2002). The relationship betweenlearner utility reactions and predicted learning transfer among trainees. International Journalof Training and Development, 6, 218–228.
Russ-Eft, D. (2002). A typology of training design and work environment factors affecting work-place learning and transfer. Human Resource Development Review, 1, 45–65.
Russ-Eft, D., & Preskill, H. (2001). Evaluation in organizations: A systematic approach to enhancinglearning, performance, and change. Cambridge, MA: Perseus Publishing.
Saari, L. M., Johnson, T. R., McLaughlin, S. D., & Zimmerle, D. M. (1988). A survey of man-agement training and education practices in U.S. companies. Personnel Psychology, 41, 731–743.
Sackett, P. R., & Mullen, E. J. (1993). Beyond formal experimental design: Toward an expandedview of the training evaluation process. Personnel Psychology, 46, 613–627.
Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. AnnualReview of Psychology, 52, 471–499.
Tan, J. A. (2002). The role of planned behavior and goal orientation theory in a training context.Dissertation Abstracts International, 62, 3409.
Van Buren, M. E., & Erskine, W. (2002). The ASTD state of the industry report. Alexandria, VA:American Society of Training and Development.
James A. Tan is assistant professor of psychology at the University of Wisconsin-Stout.
Rosalie J. Hall is associate professor of psychology at the University of Akron.
Carol Boyce is affiliated with Alliant International University.
The Role of Employee Reactions in Predicting Training Effectiveness 411