TrainingTrainingEvaluationEvaluation

Training evaluationTraining evaluation provides the data needed to demonstrate that training does provide benefits to the company.

What are the differences What are the differences among: among:

Training effectivenessTraining effectivenessTraining outcomesTraining outcomesTraining evaluationTraining evaluationEvaluation designEvaluation design

Types of EvaluationTypes of Evaluation

FormativeFormativeSummativeSummative

Why Evaluate Training Why Evaluate Training Programs?Programs?

Why Evaluate Training Why Evaluate Training Programs?Programs?

ObjectivesObjectivesSatisfactionSatisfactionBenefitsBenefitsComparisonComparison

Objectives = FoundationObjectives = Foundation

Terminal behaviorTerminal behaviorConditions under which Conditions under which

terminal behavior is expectedterminal behavior is expectedThe standard below which The standard below which

performance is unacceptable performance is unacceptable

--> criteria by which the --> criteria by which the trainee is judgedtrainee is judged

The Evaluation ProcessThe Evaluation Process

Conduct a Needs AnalysisConduct a Needs Analysis

Develop Measurable Learning Outcomes Develop Measurable Learning Outcomes and Analyze Transfer of Trainingand Analyze Transfer of Training

Develop Outcome MeasuresDevelop Outcome Measures

Choose an Evaluation StrategyChoose an Evaluation Strategy

Plan and Execute the EvaluationPlan and Execute the Evaluation

Training Outcomes: Training Outcomes: Kirkpatrick’s Four-Level Kirkpatrick’s Four-Level Framework of Evaluation Framework of Evaluation CriteriaCriteria

Business results achieved by traineesBusiness results achieved by traineesResultsResults44

Improvement of behavior on the job; akaImprovement of behavior on the job; akaskillsskills

BehaviorBehavior33

Acquisition of knowledge, skills, attitudes, Acquisition of knowledge, skills, attitudes, behavior; aka cognitivebehavior; aka cognitive

LearningLearning22

Trainee satisfaction; aka affectiveTrainee satisfaction; aka affectiveReactionsReactions11FocusFocusCriteriaCriteriaLevelLevel

How do you know if your How do you know if your outcomes are good?outcomes are good?

Good training outcomes Good training outcomes need to be:need to be:

RelevantRelevantReliableReliableDiscriminativeDiscriminativePracticalPractical

Good Outcomes: Good Outcomes: RelevanceRelevance

Criteria relevanceCriteria relevance – – the extent to which training the extent to which training programs are related to learned capabilities programs are related to learned capabilities emphasized in the training programemphasized in the training program

Criterion contaminationCriterion contamination – – extent that training extent that training outcomes measure inappropriate capabilities outcomes measure inappropriate capabilities or are affected by extraneous conditionsor are affected by extraneous conditions

Criterion deficiencyCriterion deficiency – – failure to measure training failure to measure training outcomes that were emphasized in the training outcomes that were emphasized in the training objectivesobjectives

Criterion deficiency, relevance, Criterion deficiency, relevance, and contamination:and contamination:

Outcomes Measured in Evaluation

Outcomes Identified by

Needs Assessment and Included in Training Objectives

Outcomes Related to Training

Objectives

Contamination Relevance Deficiency

Good Outcomes Good Outcomes (continued)(continued)

ReliabilityReliability – – degree to which outcomes can degree to which outcomes can be measured consistently over timebe measured consistently over time

DiscriminationDiscrimination – – degree to which trainee’s degree to which trainee’s performances on the outcome actually performances on the outcome actually reflect true differences in performancereflect true differences in performance

PracticalityPracticality – – refers to the ease with which refers to the ease with which the outcomes measures can be collectedthe outcomes measures can be collected

Training Evaluation Training Evaluation PracticesPractices

79%

38%

15% 9%

0%

20%

40%

60%

80%

Re

ac

tio

n

Co

gn

itiv

e

Be

ha

vio

r

Re

su

lts

Outcomes

Perc

enta

ge o

f Cou

rses

Usi

ng

Out

com

e

Evaluation ProceduresEvaluation Procedures

UtilityUtility

[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)][(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]

Ns = number of applicants selectedNs = number of applicants selectedT = tenure of selected group in years T = tenure of selected group in years r = correlation between predictor and job r = correlation between predictor and job

performance (VALIDITY)performance (VALIDITY)SDy = standard deviation of job performanceSDy = standard deviation of job performanceZs = average standard predictor score of selected Zs = average standard predictor score of selected

groupgroupN = number of applicantsN = number of applicantsC = cost per applicantC = cost per applicant

[(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)][(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]

Nc = number of trainees who complete programNc = number of trainees who complete programT = T = duration of training benefitduration of training benefit r = correlation between r = correlation between training criteriontraining criterion and job and job

performance (VALIDITY)performance (VALIDITY)SDy = standard deviation of job performanceSDy = standard deviation of job performanceZs = average standard Zs = average standard criterioncriterion score of score of traineestraineesN =N = totaltotal number of number of trainees enrolledtrainees enrolledC = cost per C = cost per traineetrainee

Training CostsTraining Costs

DirectDirectIndirectIndirectDevelopmentDevelopmentOverheadOverheadCompensation for TraineesCompensation for Trainees

50 = Ns = number of trainees who complete program 50 = Ns = number of trainees who complete program 1 = T = duration of training benefit 1 = T = duration of training benefit ..50 50 = r = correlation between training criterion and job = r = correlation between training criterion and job

performance (VALIDITY)performance (VALIDITY)4800 = SDy = standard deviation of job performance 4800 = SDy = standard deviation of job performance

(assume 40% of base pay . . . $12,000 * .40)(assume 40% of base pay . . . $12,000 * .40) .80 = Zs = average standard criterion score of trainees.80 = Zs = average standard criterion score of trainees100 = N = total number of trainees enrolled100 = N = total number of trainees enrolled 150150 = C = cost per trainee = C = cost per trainee

For On the Job Training For On the Job Training $81,000$81,000

[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)](50 * 1 * .50 * 4800 * .8) - (100 * 150)

Experimental DesignsExperimental Designs

Experimental DesignsExperimental Designs

ChoicesChoicesPretest/posttestPretest/posttestControl GroupsControl Groups

Experimental DesignsExperimental Designs

1: 1 group, posttest only1: 1 group, posttest only2: 1 group, pretest/posttest2: 1 group, pretest/posttest3: Pretest/posttest control group3: Pretest/posttest control group4: Solomon four-group 4: Solomon four-group 5: Time-series5: Time-series6: Nonequivalent control group6: Nonequivalent control group

Experimental DesignsExperimental Designs Validity Validity

InternalInternalExternalExternal

Experimental DesignsExperimental Designs Threats to Internal ValidityThreats to Internal ValidityHistoryHistory

MaturationMaturationTestingTestingInstrumentationInstrumentationRegression toward the Regression toward the

meanmeanDifferential selectionDifferential selectionExperimental mortalityExperimental mortality

InteractionsInteractionsDiffusion/imitation of Diffusion/imitation of

treatmentstreatmentsCompensatory Compensatory

equalization of equalization of treatmentstreatments

Rivalry/desirability of Rivalry/desirability of treatmentstreatments

DemoralizationDemoralization

Experimental DesignsExperimental Designs Threats to External ValidityThreats to External ValidityReactive effect of pretesting Reactive effect of pretesting Interaction of selection & Interaction of selection &

treatmenttreatmentReactive effects of Reactive effects of

experimental settingsexperimental settingsMultiple-treatment interferenceMultiple-treatment interference

Issues in Training Issues in Training ValidityValidity

Training validityTraining validityTransfer validityTransfer validityIntra-organizational validityIntra-organizational validityInter-organizational validityInter-organizational validity