Upload
vincent-horton
View
217
Download
1
Embed Size (px)
Citation preview
1
2007 National Human Services Training
Evaluation Symposium
Fasih Ahmed, Ph.D.
2
Reaction to Training: A Few Basic Questions about the
Nature of the Construct
Is reaction to training a multi-dimensional construct with distinct components or a generalized holistic perception of training experience?
If reaction to training is a multi-dimensional concept, how the various dimensions relate to each other and how they collectively formulate the overall reaction to training?
Are the various dimensions identifiable (and measurable) as separate ‘valid’ constructs with sufficient specificity for application in evaluative analysis of training programs?
3
Some Possible Approaches and Assumptions
Most trainee satisfaction surveys assume reaction to training as a multi-faceted concept and measure satisfaction/dissatisfaction with such aspects of training as the curriculum, the trainer, and training delivery arrangements/facilities, etc.
Statistical techniques can be used to delineate and examine specific dimension of reaction to training. These can range from such sophisticated techniques as factor and logistic analyses to simple cross tabulations, depending on the level of measurement and tenability of assumptions about the underlying distributions of the specific variables being measured.
The purpose of this analysis is to determine the nature of relationship between the various dimensions of reaction, and how they individually relate to the overall reaction to training experience. Question: are individual dimensions independent of each
other, or related in some causal model, or have interactive effects on each other?
Question: What is the nature of relationships between individual dimensions with the overall reaction to training?
4
Methodology & Data: The Case of North Carolina
NC uses the Participant Satisfaction Form (PSF) to survey reaction to training and its components: curriculum (five questions); trainer(s) (six questions); training delivery arrangements (three questions), in addition to a few attitudinal questions and a question on satisfaction with the overall training experience.
Almost all questions have four response categories (strongly agree, agree, disagree, strongly disagree) which. when combined, yield dichotomous variables with two outcomes: satisfied-dissatisfied.
Our focus of this analysis are only two components: reaction to curriculum and reaction to trainer. This analysis is based on data collected over three years: 2004,2005, and 2006. Data for each year is analyzed separately to discern any patterns.
In order to aggregate data, a mean score for each component is computed from the cluster of questions pertaining to that component, which yields an interval scale continuum of satisfaction-dissatisfaction. Distribution of mean scores for reaction to trainer and reaction to curriculum are shown below.
5
6
7
8
Limitations of Analysis
Given the extremely skewed distribution of mean scores, our statistical analysis will have to be limited to non-parametric approaches, which do not depend on assumptions of normal, or approximately normal, distributions.
Mindful of the fact that the responses were originally measured as categorical variables, we have transformed the mean scores into two categories: satisfied and dissatisfied, for each of the component.
Since the scores of 3 and 4 in the original measurement indicated satisfaction, all mean scores of less than 3 were categorized as indicative of dissatisfaction, which typically includes the bottom 10% to 27% of all mean scores.
Given the need to aggregate the data as categorical variables, cross tabulation is the statistical techniques we have used for the following analysis.
Simple distribution of satisfaction/dissatisfaction with curriculum and trainer for the three years under consideration are shown below.
9
Mean Score Curriculum 2004 Frequency Percent
Satisfied 5223 72.8
Dissatisfied 1951 27.2
Total 7174 100.0
Mean Score Trainer 2004 Frequency Percent
Satisfied 10380 82.8
Dissatisfied 2150 17.2
Total 12530 100.0
Mean Score Curriculum 2005 Frequency Percent
Satisfied 6834 76.3
Dissatisfied 2128 23.7
Total 8962 100.0
Mean Score Trainer 2005 Frequency Percent
Satisfied 12676 80.8
Dissatisfied 3020 19.2
Total 15696 100.0
Mean Score Curriculum 2006 Frequency Percent
Satisfied 6442 89.3
Dissatisfied 770 10.7
Total 7212 100.0
Mean Score Trainer 2006 Frequency Percent
Satisfied 6379 88.9
Dissatisfied 800 11.1
Total 7179 100.0
10
Are Reactions to Curriculum and Trainer Independent Dimensions?
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
44.5 55.5
70.4 18.3
100
Satisfied
% within curriculum
% within trainer
7.0 93.0
29.6 81.7
100
Total 100 100
2004
2005
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
43,7 56.3
33.7 6.6
100
Satisfied
% within curriculum
% within trainer
9.8 90.2
66.3 93.4
100
Total 100 100
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
42.0 58.0
40.3 7.0
100
Satisfied
% within curriculum
% within trainer
7.5 92.5
59.8 93.0
100
Total 100 100
2006
11
Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2004
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
44.5 55.5
70.4 18.3
100
Satisfied
% within curriculum
% within trainer
7.0 93.0
29.6 81.7
100
Total 100 100
Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV.
A. Condition of dissatisfaction with trainer:44.5% of those dissatisfied with curriculum are also dissatisfied with trainer.7.0 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with
curriculum. % difference: 44.5-7.0 = 37.5.
B. Condition of satisfaction with trainer:55.5 % of those dissatisfied with curriculum are satisfied with trainer.93% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. %
difference: 93.0-55.5 = 37.5.
Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV.
C. Condition of dissatisfaction with curriculum:70.4% of those dissatisfied with trainer are dissatisfied with curriculum.18.3% of those satisfied with trainer are dissatisfied with curriculum.
Indication: dissatisfaction with curriculum is more likely to be related to dissatisfaction with trainer. % difference: 70.4-18.3 = 52.10.
D. Condition of satisfaction with curriculum:29.6% of those dissatisfied with trainer are satisfied with curriculum.81.7 of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. %
difference: 81.7-29.6 = 52.10.
Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more tenable.
12
Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2005
Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV.
A. Condition of dissatisfaction with trainer:43.7% of those dissatisfied with curriculum are also dissatisfied with trainer.9.8 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with
curriculum. % difference: 43.7-9.8 = 33..9.
B. Condition of satisfaction with trainer:56.3 % of those dissatisfied with curriculum are satisfied with trainer.90.2% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum.
% difference: 90.2-56.3 = 33.9.
Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV.
C. Condition of dissatisfaction with curriculum:33.7% of those dissatisfied with trainer are dissatisfied with curriculum.6.6% of those satisfied with trainer are dissatisfied with curriculum.
Indication: dissatisfaction with curriculum is not more likely to be related to dissatisfaction with trainer. % difference: 33.7-6.6 = 27.1.
D. Condition of satisfaction with curriculum:66.3% of those dissatisfied with trainer are satisfied with curriculum.93.4% of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is not more likely to be related to satisfaction with
trainer. % difference: 93.4-66.3 = 27.1.
Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more plausible, but only slightly so.
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
43,7 56.3
33.7 6.6
100
Satisfied
% within curriculum
% within trainer
9.8 90.2
66.3 93.4
100
Total 100 100
13
Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2006
Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV.
A. Condition of dissatisfaction with trainer:42.0% of those dissatisfied with curriculum are also dissatisfied with trainer.7.5 % of those satisfied with curriculum are dissatisfied with trainer.Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with curriculum. % difference: 42.0-7.5 = 34.5.
B. Condition of satisfaction with trainer:58.0 % of those dissatisfied with curriculum are satisfied with trainer.92.5% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. % difference: 92.5-58.0 = 34.5.
Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV.
C. Condition of dissatisfaction with curriculum:40.3% of those dissatisfied with trainer are dissatisfied with curriculum.7% of those satisfied with trainer are dissatisfied with curriculum.Indication: dissatisfaction with curriculum is more likely to be re lated to dissatisfaction with trainer. % difference: 40.3-7.0 = 33.3.
D. Condition of satisfaction with curriculum:59.8% of those dissatisfied with trainer are satisfied with curriculum.93.0% of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. % difference: 93.0-59.8 = 33.2.
Tentative Conclusion: Comparing the % differences under the two assumptions, both Assumption I and Assumption II seem equally tenable.
Curriculum Trainer Total
Dissatisfied Dissatisfied Satisfied
% within curriculum
% within trainer
42.0 58.0
40.3 7.0
100
Satisfied
% within curriculum
% within trainer
7.5 92.5
59.8 93.0
100
Total 100 100
14
Interaction Between Overall Training Experience and Reaction to Trainer 2004
Overall Experience Trainer Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within trainer
57.7 42.3
44.6 6.7
100
Satisfied
% within oa exp.
% within trainer
10.9 89.1
55.4 93.3
100
Total 100 100
Overall Experience Curriculum Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within curriculum
59.3 40.7
28.9 7.3
100
Satisfied
% within oa exp.
% within curriculum
22.2 77.8
71.1 92.7
100
Total 100 100
Interaction Between Overall Training Experience and Reaction to Curriculum 2004
PSF Data for 2004
There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum.
The effect of overall experience on reaction to trainer is a little stronger than that on reaction to curriculum, suggesting an interactive effect between overall experience, reaction to curriculum, and reaction to trainer.
One should, however, be mindful that perception of overall experience is based on only one question.
15
Interaction Between Overall Training Experience and Reaction to Trainer 2005
Overall Experience Trainer Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within trainer
51.2 48.8
32.2 4.7
100
Satisfied
% within oa exp.
% within trainer
9.8 90.2
67.8 95.3
100
Total 100 100
Interaction Between Overall Training Experience and Reaction to Curriculum 2005
Overall Experience Curriculum Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within curriculum
63.0 37.0
31.1 5.7
100
Satisfied
% within oa exp.
% within curriculum
18.5 81.5
68.9 94.3
100
Total 100 100
PSF Data for 2005
There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum.
The effect of overall experience on satisfaction with trainer is slightly more than the other way round.
The effect of reaction to curriculum on perception of overall experience is more than effect of overall experience on curriculum.
These pattern suggest an interactive effect between overall experience, satisfaction with trainer, and reaction to curriculum.
16
Interaction Between Overall Training Experience and Reaction to Trainer 2006
Overall Experience Trainer Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within trainer
45.1 54.9
33.5 5.1
100
Satisfied
% within oa exp.
% within trainer
8.0 92.0
66.5 94.9
100
Total 100 100
Interaction Between Overall Training Experience and Reaction to Curriculum 2006
Overall Experience Curriculum Total
Dissatisfied Dissatisfied Satisfied
% within oa exp.
% within curriculum
62.6 37.4
48.9 3.4
100
Satisfied
% within oa exp.
% within curriculum
5.8 94.2
51.1 96.6
100
Total 100 100
PSF Data for 2006
There is significant association between overall experience of training and reaction to trainer, the effect of overall experience on reaction to trainer being slightly more than the converse.
There is a significant positive effect of overall experience of training on
reaction to curriculum , the effect of overall experience on reaction to curriculum being slightly more than the converse.
The pattern of associations suggests interactive relationships between the overall experience of training, reaction to curriculum and reaction to trainer.
17
Inconclusive Conclusions
The three years data does not yield any consistent patternsto allow definitive answers to the questions raised initially. However, some of the interesting, though unexpected findings are:
Given the patterns of relationships between reaction to curriculum, reaction to trainer, and the overall experience with training, there is no clear evidence that specific dimensions of reaction to training are independent components of a theoretical construct.
When examining relationships between reaction to curriculum and reaction to trainers under different assumptions of a causal model, there only indications, not evidence, that satisfaction with trainer tends to play more important role in determining reaction to curriculum, relative to perception of overall experience of training.
Although analysis of reaction to training may serve some limited evaluative purpose, its utilization as a decision-making tool for management of training warrants due caution.