Upload
creators-college
View
120
Download
0
Tags:
Embed Size (px)
Citation preview
Training on Trial
Mid New Jersey Chapter, ASTD
May 21, 2009
Presented by:
Jim Kirkpatrick , PhD
The future of training
“Training directors might be well advised to take the initiative and evaluate their programs before the day of reckoning arrives”
- From Techniques for Evaluating Training Programs, by Don Kirkpatrick - ASTD Journal, November, - 1959
What are the charges?
“You, learning professionals, are charged with incurring training and development costs that exceed the value you bring to the business”
Who exactly is the “we” who are being charged?
The training industry
Organizational Development
Learning functions
HRD
Training departments
Us
Who is charging us?
Internal business partners
External clients
The business world
How do we know we are being charged?
“We would like you to show us your ROI”
“We know you run a lot of programs and seem quite busy. Why then are so many good people leaving us?”
“We are thinking we need to put our efforts in revenue-producing activities.”
“We are going to put that on hold for awhile.”
“We have decided to make some cuts.”
What evidence is being used brought against us?
Brinkerhoff Study
- Josh Bersin and Associates, 2008
Training Application
Did not try new skills: 15%
Tried new skills and failed: 70%
Achieved sustained new behaviors: 15%
Percentage of learning transfer
1975: Percentage of formal learning that is actually applied to the job:
15%
2005: Percentage of formal learning that is actually applied to the job
15%
From Dana Robinson, ASTD ICE, 2008
Causes of “training failure”
Preparation and Readiness: 20%Learning Intervention: 10%
Application Environment: 70%
2006 ASTD Study
Typical Learning Investment
Dr. Brent Peterson, Columbia University, 2004
P re-Work 10%
L earning E vent 85%
F ollow-Up 5%
Are we guilty?
What do we have to do to be found not guilty?
We need to extend our role beyond the traditional definitions of training and learning
The increase in “informal learning”
Where Learning Takes Place
Prior to being "trained": 20%During "training": 10%On the job: 70%
- Josh Bersin and Associates, 2008
Lets get very practical here
We need to redefine and expand our role through the entire “learning and performance” management spectrum:– Our expertise– Our involvement– Our influence– Our impact– Our value
How does this relate to you
so far?
Who will take our case?
Measurement and Evaluation
Gathering data and information to answer specific questions regarding the value of learning and performance solutions; focusing on the impact of individual programs and creating overall measures of system effectiveness, leveraging findings to provide recommendations for change and to increase organizational effectiveness.
- taken from Mapping the Future, ASTD, 2004
Why Evaluate?
1. Improve the program (or know when to retire it)
2. Maximize training effectiveness by reinforcing knowledge, skills, and attitudes
3. Demonstrate the value of training
p. 3
Level 1: REACTION To what degree participants react
favorably to the training
Kirkpatrick Four Levels
p. 4
Level 2: LEARNINGTo what degree participants acquire
intended knowledge, skills, and attitudes based on participation in
learning event
Kirkpatrick Four Levels
p. 4
Level 3: BEHAVIOR To what degree participants apply what
they learned during training on the job
Kirkpatrick Four Levels
p. 4
Level 4: RESULTS To what degree targeted outcomes occur,
as a result of the learning event(s) and subsequent reinforcement
Kirkpatrick Four Levels
p. 4
Four Level Correlations
Enjoyment (L1) and learning (L2): no correlationsRelevance (L1) and immediate learning (L2): r = .26Relevance (L1) and transfer to job (L3): r = .18
Immediate knowledge (L2) and retention (L2): r = .34Immediate knowledge (L2) and skill demo (L2): r = .18Immediate knowledge (L2) and transfer to job (L3): r = .11Retained knowledge (L2) and transfer to job (L3): r = .08Skill demo (L2) and transfer to job (L3): r = .18
Kirkpatrick Four Level Evaluation major principles
• Smile sheets, pre and post-tests, and hope for the best
• Strong connection between Levels 1 and 2
• Strong connection between Levels 3 and 4
• “The Great Divide”
• “The Missing Link”
Let’s take some initial depositions . . .
“What is your job here at the hotel?”
“I am a window washer.”
“What is your job here at the resort?”
“I am part of a team that creates great experiences for our guests!”
Group Activity
What is significant about these two window washers in relation to:
Group 1: individual employees?
Group 2: training and development professionals?
Group 3: the training and development industry?
Crossing over is not easy
What does the word “charged” mean?
• Accused
• Challenged
Common Myth
“Evaluation” only occurs at the end of the instruction design process, or at the end
of a program.
Reality
An effective evaluation plan is considered at every step in the program and training development process, not JUST at the
end.
Kirkpatrick Four Level Evaluation major principles
• The end is the beginning
• ROE is the ultimate indicator of value
• Business partnership is key
• Value must be created first
• Demonstrate value through a compelling chain of evidence
Kirkpatrick Four Level Evaluation Model
Reaction
Behavior
Results
Learning
ReactionLearningBehaviorResults
ROE
Business need
identified
Measure L1 ReactionMeasure L2 Learning
Measure L3 Behavior
Present L1 Reaction findings
Present L2 Learning findings
Present L3 Behavior findings
Present L4 Results findings
Gather final four level data / information, prepare for presentation
The Kirkpatrick Model
Identify critical behaviors and key
organizational drivers
Determine required KSAs,
Learning Objectives
Negotiate stakeholder success indicators,Business outcomes
Ana
lyze
find
ings
, adj
ust,
repe
at s
teps
as
nece
ssar
yConsider necessary learning environment
and conditions
Design and build learning program and evaluation tools
Initiate ongoing reinforcement and
monitoring
Measure L4 Results
Deliver learning program
p. 22
DOT Case Example
Challenge – to increase highway work zone efficiency, effectiveness, and safety.
The Kirkpatrick Business Partnership Model
1. Identify who is on your corporate jury2. Negotiate jury (stakeholder) expectations3. Convert expectations to Level 4 outcomes and
determine metrics4a. Identify the key new participant behaviors that will be
required to bring about the desired outcomes. If L3 behaviors are not relevant, choose L2 ongoing readiness factors
4b. Identify the few key organizational drivers that will be necessary to bring about the changes in behavior
5. (Determine and implement prerequisites)
DOT Example
1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors.
2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning.
3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; increased compliment to complaint ratio.
4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures.
4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.
Airborne Warning and Control System
Activity #21. Break up into small groups as instructed.
2. Discuss and select one high impact training program from your group members to work with.
3. Develop your answers to steps 1-4. Use flip chart pages.
4. Be prepared to share your results.
Demonstrated SKA
Training-related Activities
Business Results
Did the critical behaviors contribute to a positive impact on business results?
Did the activities build sufficient capabilities to drive the critical behaviors?
Did the target audience react favorably to the activity?
Critical Behaviors
Did the target audience exhibit a sustained behavior change in the work environment?
Level 1 Level 2 Level 3 Level 4
Measurement Strategy
ActivitiesBusiness Results
CriticalBehaviors
Business Processes
Strategy
Culture Structure
Talent
HR Processes
Business Partnership
Preparing for and presenting our case
Developing a Plan to Evaluate: Critical Criteria
Evaluation methodsEvaluation Levels
Methods
1 Reaction
2 Learning
3 Behavior
4Results
Survey ● ● ● ●
Questionnaire/Interview ● ● ● ●
Focus Group ● ● ● ●
Knowledge Test/Check ●
Work Review ● ●
Skills Observation ● ●
Presentations / Teach Bk ● ●
Action Planning ● ● ●
Action Learning ●
Key Business HR Metrics ●
1
Post Session L1
Level 1
Level 2
Level 3
Level 4
Was session delivered effectively?
Post Session Reinfcmt, Coaching & L3
Expected results will occur.Build your chain of evidence & showcase
Post Session L2
Did targeted learning occur?
Was there successful transfer of learning to behavior?
Improve Reinforcement
and/or Coaching
NO
YES
NO
NONOYES
YES
YES
Improve Session 2
3
4
5
6
7
8
9
10
11
Delivery of Learning EventL1 & L2 Formative Evaluation
Was failure due to session?
Kirkpatrick Four Level EvaluationFeedback Loop to Eliminate ‘Snags’
Example: Valdez University strategic goal scorecard – goal #1
Up from last month
Same as last month
Down from last month
Help to move our
organization from
transaction-oriented to
‘trusted advisor’
Metric Actual Target Status
1. Level 1 satisfaction scores - aggregate 93% 90%
2. % courses learning objectives matched to new directive
85% 70%
3. Level 2 skills scores - aggregate 92% 90%
4. % leaders certified as trusted advisor coaches 66% 65%
5. Level 3 scores - aggregate 48% 75%
6. Gallup scores - aggregate 77% 90%
Preparing for our closing arguments
Chain of Evidence
Gather data at all four levels and show that your training delivers true value to your organization.
Level 1Reaction
Level 2Level 2LearningLearning
Level 4Results
Level 3Behavior
Data is a critical part of our Chain of Evidence
But do not underestimate the power of testimonials
What will you do as a result of what you have learned today . . .
that will lead you to the following tomorrow?
that will hopefully lead to . . .
References• Kirkpatrick, D.L. and J.D. Evaluating Training
Programs, 3rd Ed., Berrett-Koehler Publ., Inc. San Francisco, CA, 2006
• Kirkpatrick, D.L. and J.D. Transferring Learning to Behavior: Berrett-Koehler Publ., San Francisco, CA, 2005
• Kirkpatrick, D.L. and J.D. Implementing the Four Levels, Berrett-Koehler Publ., San Francisco, CA, 2007
• Kirkpatrick, J.K., and W.K. Training on Trial, AMACOM, New York, 2009
• Contact Jim at [email protected]
• www.smr-usa.com